Elsevier

Information and Software Technology

Volume 58, February 2015, Pages 272-288
Information and Software Technology

Investigating software testing and maintenance reports: Case study

https://doi.org/10.1016/j.infsof.2014.06.015Get rights and content

Abstract

Context

Although many papers have been published on software development and defect prediction techniques, problem reports in real projects quite often differ from those described in the literature. Hence, there is still a need for deeper exploration of case studies from industry.

Objective

The aim of this study is to present the impact of fine-grained problem reports on improving evaluation of testing and maintenance processes. It is targeted at projects involving several releases and complex schemes of problem handling. This is based on our experience gained while monitoring several commercial projects.

Method

Extracting certain features from detailed problem reports, we derive various measures and present analysis models which characterize and visualize the effectiveness of testing and problem resolution processes. The considered reports describe types of problems (e.g. defects), their locations in project versions and software modules, ways of their resolution, etc. The performed analysis is related to eleven projects developed in the same company. This study is an exploratory research with some explanatory features. Moreover, having identified some drawbacks, we present extensions of problem reports and their analysis which have been verified in another industrial case study project.

Results

Fine-grained (accurate) problem handling reports provide a wider scope of possible measures to assess the relevant development processes. This is helpful in controlling single projects (local perspective) as well as in managing these processes in the whole company (global perspective).

Conclusion

Detailed problem handling reports extend the space and quality of statistical analysis, they provide significant enhancement in evaluation and refinement of software development processes as well as in reliability prediction.

Introduction

Typical software lifecycle comprises such processes as: specification of requirements, architectural design, coding, testing, deployment and maintenance. Recently, in many projects these processes are performed in an evolutionary and iterative way [36], [40] covering subsequently developed versions, updates and upgrades, etc. Managing these processes, we have to take into account: product (software) quality requirements, delivery deadlines, and user satisfaction. For this purpose appropriate measurements and monitoring schemes are needed [12], [15], [45], [47]. They have been described in many papers on various software issues, e.g. verification and validation [36], testing [9], fault prediction [27], [33], software effort estimation [1], [2], [39], software process improvement factors [8], [28], failure proneness aspects [19], reliability models [32], maintainability [17], [46].

Measures are required to control and optimize software development and maintenance. Much attention has been paid to software defect prediction, complexity and control metrics [1], [13], [33], [40]. Static prediction techniques correlate defect density with software basic attributes (e.g. size, complexity, object metrics, test coverage). Component based reliability predictions [34] evaluate component reliability in function of the usage profile. Dynamic predictions correlate reliability with distribution of detected defects in the initial state of testing (software reliability growth models – SRGMs [3], [29], [32]). Control metrics relate to software processes, e.g. functional and structural test coverage, time required to repair defects, software development progress [40]. The significance of such measurements is well known and appropriate standards have been also introduced [2], [8], [40], however putting this in practice creates many problems. Most papers on software measurements and statistics are of the academic character [16]. Hence, an important issue is to collect and exchange the gained experience from real commercial (industrial) projects – case studies. This paper addresses this issue within the process oriented research and it shows the richness of the use of problem reports (repository) in managing software development process. In particular, we provide an in-depth study of handling detected problems.

Usually, case studies are targeted at analysing specific problems, they can be a product (software quality, complexity) or process oriented. Process oriented studies are targeted at studying the effectiveness of software development processes or their phases. Our research belongs to this class, however we concentrate on problem handling within testing, code improvement and deployment (including release phases) followed by initial maintenance. The classical notion of software defects (bugs) is extended to problems, which can be resolved (correction of software defects, code refinement), rejected (e.g. withdrawn due to some misunderstanding or assumption changes), qualified as duplicated (redundant) or processed in different ways. Another important issue is the input data, in our case we mainly deal with the so called problem repository which can be supplemented with test progress repository.

The process oriented case studies published in the literature (compare Section 2) are based on coarse-grained data reports (e.g. aggregated information on detected faults, modifications of requirements, etc.) and they relate to different designs and implementation. They present software measurements and conclusions addressed to different phases of software development and maintenance. We describe our experience on evaluation the effectiveness of processes related to testing, code revisions or refinements and maintenance. The considered projects relate to telecommunication applications and have been developed in a local company using a similar incremental technology. Gained experience is based on monitoring fine-grained problem handling reports for 11 commercial projects (case studies). These reports comprise many attributes characterizing occurrence of problems, descriptions of their manifestations, severity levels, advances in testing or code repairing, performed solutions and related decisions. Included information is more detailed than in other case studies from the literature. An important contribution is introducing problem handling graphs, which facilitate analysis and visualizations. Moreover, we present still another case study with improved problem reports. The presented analysis concentrates on problem tracking and it is supplemented with a discussion on reliability prediction models (SRGMs). Moreover, it can be integrated with other statistical program measures.

As opposed to other approaches usually considered in the literature, development, testing and debugging processes of newer versions are interleaved with the maintenance of the older ones. A significant part of the generated code can be based on reuse-oriented software engineering and software components provided by various vendors (integration and configuration problems). We have introduced two problem analysis perspectives: global (overall view of all handled projects) and local (restricted to a single project). An important conclusion, we draw from the work, is that fine-grained reports (correlated with project development features) are helpful in more accurate process evaluation and identification of possible imperfections and refinements. This approach is useful for project (local observation perspective) and company (global perspective) managements, for this purpose some measures have been introduced. They also can interest system testers or developers and create some feedback in discussions with the management staff.

The remainder of the paper is organized as follows. In the next section, we provide related work. Section 3 presents the context of this study combined with the research methodology. Section 4 describes the considered software lifecycle model and measurement challenges. Section 5 presents a data repository comprising monitoring reports (measurements), which characterize encountered problems and the relevant handling processes. Basing on the collected data from many projects we have derived indicators characterizing software development and maintenance quality (described in Section 6). Section 7 provides a case study of a project with an improved monitoring scheme. Section 8 discusses results, Section 9 summarizes the paper and outlines plans for the future work.

Section snippets

Related work

In this section we give a survey on case studies described in the literature, which are targeted at software process assessment and are associated with our research. This is complemented with some remarks on software reliability modelling. We also show differences (in particular coarse-grained analysis) and some gaps of the considered approaches in relevance to our experience with the analysed projects which triggered our research.

In the literature, we can find reports on various case studies

Research context and methodology

This section presents the context of the performed research and the specificity of our study. It covers a case study of projects realized within the same software company, which has been involved for many years in providing specialized software systems for telecommunication applications. The realized software projects have been developed and managed according to ITIL (http://www.itil.org/) and Prince 2 (http://www.prince2.com/prince2-methodology) standards. Our research is related to the

Software development model and measurement challenges

Developing complex software we face the problem of assuring high quality and reliability. In practice, it is not possible to predict all software usage problems during the design phase due to unforeseen factors appearing randomly such as changing software environment (e.g. due to software patches), changing user operational profiles, modifications of functionalities, load fluctuations, etc. Complex (large scale) software relates to tenths or even many hundred thousands of code lines, and is

Problem repository

This section describes problem reports, management of the problem repository and problem handling flow (processing states). Some comments on testing schemes are also included. Reports on software development, testing and maintenance are the basic data source in the process of monitoring and in analysis. The structure and information content of these reports depend upon software developers and users. In the literature this issue is neglected. In most cases the authors deal with some general

Analysis methodology and results

Exploring the problem repository and schemes of test cases we have derived several indicators which characterize the progress of developed projects, engagement of human resources, quality of software products, etc. Here, we can distinguish several observation perspectives: global and local, developers and users. The global perspective (Section 6.1) characterizes problem features and the effectiveness of the company in dealing with all projects. It also facilitates comparing projects to reveal

Improvements in problem handling reports

The problem repository considered in the previous sections changed its contents in time without detailed traces of these changes. This is typical for many software development repositories. Change traces could be deduced by on-line monitoring and recording the states of problem reports. Such information could facilitate advanced exploration of problem handling processes. We have got such possibility within project P12. In this case, the improved problem repository registers directly problem

Discussion

The goal of the study was to analyse problem handling reports in testing and early maintenance phases. This problem centred analysis is supported with measures targeted at visualization and evaluation of the related processes with respect to the ability to show the presence or absence of inefficiencies. In this research we based on the data from real projects related to telecommunication applications performed in the context specified in Section 3. We combine (integrate) the observation

Conclusion

The paper provides a holistic approach to software development based on the analysis and monitoring of problem reports. The main contribution of the paper is the analysis of industrial experience related to detailed monitoring of software testing and problem fixing processes. We present a variety of in-depth studies of problem reports associated with the considered processes. This is enhanced with a discussion of possible improvements resulting from spotted weak points. This research relates to

References (47)

  • Y-Ch. Chen, X.W. Wang, Neural network based approach on reliability prediction of software in maintenance phase, in:...
  • D. Damian et al.

    An empirical study of the complex relationships between requirements engineering processes and other processes that lead to payoffs in productivity, quality, and risk management

    IEEE Trans. Softw. Eng.

    (2006)
  • L.-O. Damm et al.

    Faults-slip-through – a concept for measuring the efficiency of the test process

    Softw. Proc. Improve. Pract.

    (2006)
  • I. E Espinosa-Curiel et al.

    A framework for evaluation and control of the factors that influence the software process improvement in small organizations

    J. Softw. Evol. Proc.

    (2013)
  • F. Ferrucci, M. Harman, J. Ren, F. Sairo, Not going to take this anymore: Multi-objective overtime planning for...
  • T. Fuji, T. Dohi, T. Fujawawa, Towards quantitative software reliability assessment in incremental development...
  • J. Gil, M. Goldstein, D. Moshkovich, An empirical investigation of changes in some software properties over time, in:...
  • Ch-Y. Huang et al.

    Estimation and analysis of some generalized multiple change-point software reliability models

    IEEE Trans. Reliab.

    (2011)
  • S. Ikemoto, T. Dohi, H. Okamura, Estimating software reliability via pseudo maximum likelihood method, in: Proceedings...
  • S.H. Kan

    Metrics and Models in Software Quality Engineering

    (2003)
  • D. Kozlov et al.

    Assessing maintainability change over multiple software releases

    J. Softw. Maint. Evol.

    (2008)
  • O. Krini, J. Borcsok, New scientific contributions to the prediction of the reliability of critical systems which base...
  • M. Król, J. Sosnowski, Multidimensional monitoring of computer systems, in: Proceedings of IEEE Symposia and Workshops...
  • Cited by (0)

    View full text