Skip to main content
Log in

Metric-based software reliability prediction approach and its application

  • Experience Report
  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

This paper proposes a software reliability prediction approach based on software metrics. Metrics measurement results are connected to quantitative reliability predictions through defect information and consideration of the operational environments. An application of the proposed approach to a safety critical software deployed in a nuclear power plant is discussed. Results show that the proposed prediction approach could be applied using a variety of software metrics at different stages of the software development life cycle and could be used as an indicator of software quality. Therefore the approach could also guide the development process and help make design decisions. Experiences and lessons learned from the application are also discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Notes

  1. M-D models for the metrics used in section 4 are given in Appendix B.

  2. The power plant control logic is comprised of three independent control units. Each unit contains 4 channels, and each channel contains one APP safety module.

References

  • (2011) Tech Salary Survey Results. [Online] http://marketing.dice.com/pdf/Dice_2010-11_TechSalarySurvey.pdf

  • Butler RW, Finelli GB (1993) The infeasibility of quantifying the reliability of life-critical real-time software. IEEE Trans Softw Eng 19(1):3–12

    Article  Google Scholar 

  • Campbell D, Stanley J (1966) Experimental and quasi-experimental designs for research. Chicago, USA

  • Card DN, Agresti WW (1988) Measuring software design complexity. J Syst Softw 8(3):185–197

    Article  Google Scholar 

  • Chapman M, Solomon D (2002) Software metrics as error predictors

  • Dougherty EM, Fragola JR (1988) Human reliability analysis: a system engineering approach with nuclear power plant applications. John Wiley & Sons

  • Embrey DE (1983) The use of performance shaping factors and quantified expert judgement in the evaluation of human reliability: an initial appraisal. In: U.S. Nuclear Regulatory Commission, NUREG/CR-2986

  • Eom HS, Kang HG, Park KH et al. (2005) A study on the quantitative assessment method of software requirement documents using software engineering measures and Bayesian belief networks. Proc KNS Autumn Meet

  • Fenton NE, Pfleeger SL (1997) Software metrics: a rigorous and practical approach, 2nd edn. Int’l Thomson Computer Pres, New York

    Google Scholar 

  • Fenton N et al (1998) Assessing dependability of safety critical systems using diverse evidence. IEEE Proceedings Software Engineering 145(1):35–39

    Article  Google Scholar 

  • Gaffney JE (1984) Estimating the number of faults in code. IEEE Trans Softw Eng 10(1):459–464

    Article  Google Scholar 

  • Garg RK, Sharma K, Nagpal CK, Garg R, Garg R, Kumar R, Sandhya (2013) Ranking of software engineering metrics by fuzzy-based matrix methodology. Software Testing, Verification and Reliability 23(2):149–168

    Article  Google Scholar 

  • Herrmann D (2000) Software safety and reliability: techniques, approaches, and standards of key industrial sectors, 1st edn. ed. Wiley-IEEE Computer Society

  • Huang C-Y, Lyu MR (2011) Estimation and analysis of some generalized multiple change-point software reliability models. IEEE Trans Reliab 60(2):498–514

    Article  Google Scholar 

  • IEEE (1988) IEEE standard dictionary of measures to produce reliable software 982.1. IEEE Computer Society

  • IEEE (2008) IEEE recommended practice on software reliability. New York, USA

  • International Function Point Users Group (2004) Function point counting practices manual (Release 4.2)

  • Ireson WG (1966) Reliability handbook. McGraw Hill

  • Jones C (1995) Measuring global software quality. Software Productivity Research, Burlington

    Google Scholar 

  • Jones C (1996) Applied software measurement: assuring productivity and quality, 2nd ed. New York, USA

  • Kapur PK, Pham H, Anand S, Yadav K (2011) A unified approach for developing software reliability growth models in the presence of imperfect debugging and error generation. IEEE Trans Reliab 60(1):331–340

    Article  Google Scholar 

  • Kapur PK, Pham H, Aggarwal AG, Kaur G (2012) Two dimensional multi-release software reliability modeling and optimal release planning. IEEE Trans Reliab 61(3):758–768

    Article  Google Scholar 

  • Kaufman L, Dugan JB, Johnson B et al. (1998) Using statistics of the extremes for software reliability analysis of safety critical systems. Proc Ninth Int Symp Software Reliab Eng 355-363

  • Kristiansen M, Winther R, Natvig B (2011) A Bayesian hypothesis testing approach for finding upper bounds for probabilities that pairs of software components fail simultaneously. International Journal of Reliability, Quality and Safety Engineeering 18(3):209–236

    Article  Google Scholar 

  • Kumar C, Yadav DK (2014) Software defects estimation using metrics of early phases of software development life cycle. International Journal of System Assurance Engineering and Management. doi:10.1007/s13198-014-0326-2

    Google Scholar 

  • Lawrence JD, Sicherman A, Person WL et al. (1998) Assessment of software reliability measurement methods for use in probabilistics risk assessment. Lawrence Livermore Nat’l Laboratory, FESSP

  • Lee W, Lee JK, Baik J et al. (2011) Software reliability prediction for open source software adoption systems based on early lifecycle measurements. 2011 I.E. 35th Ann Comput Software Applic Conf (COMPSAC) 366-371

  • Li M, Smidts C (2003) A ranking of software engineering measures based on expert opinion. IEEE Trans Softw Eng 29(9):811–824

    Article  Google Scholar 

  • Li M et al (2004) Validation of a methodology for assessing software reliability. Proc 15th Int Symp Software Reliab Eng, Saint-Malo, France, pp. 66-76

  • Lyu M, Ed.(1996) Handbook of software reliability engineering. New York, USA

  • Lyu M (2007) Software reliability engineering: a roadmap. In: Future of Software Engineering. Minneapolis, USA

  • Malaiya YK, Li N, Bieman JM et al. (1994) The relationship between test coverage and reliability. Proc 5th Int Symp Software Reliab Eng, Los Alamitos, pp. 186-195

  • McCabe T (1976) A complexity measure. IEEE Trans Softw Eng SE-2(4):308–320

    Article  MathSciNet  MATH  Google Scholar 

  • McCabe T (1982) Structured testing: a software testing methodology using the cyclomatic complexity metric. National Bureau of Standards Report NBS 500-99. Washington, DC

  • Miller KW et al (1992) Estimating the probability of failure when testing reveals no failures. IEEE Trans Softw Eng 18(1):33–43

    Article  Google Scholar 

  • Misra RB, Kumar KS (2008) An enhanced model for early software reliability predcition using software engineering metrics. 2nd Int Conf Sec Syst Integrat Reliab Improve 177-178

  • Munson J, Khoshgoftaar T (1992) The detection of fault-prone program. IEEE Trans Softw Eng 18(5):423–433

    Article  Google Scholar 

  • Murine GE (1985) On validating software quality metrics. 4th Ann IEEE Conf Software Quality 39-44

  • Musa J (1990) Software reliability: measurement, prediction, application. New York, USA

  • Musa J (1992) The operational profile in software reliability engineering: an overview. Third Int Symp Software Reliab Eng 140-154

  • NASA (2004) Standard for Software Assurance. NASA, STD 8739.8

  • Okamura H, Dohi T. (2014) A novel framework of software reliability evaluation with software reliability growth models and software metrics. 2014 I.E. 15th Int Symp High-Assurance Syst Eng (HASE 97–104

  • Pandey AK, Goyal NK (2010) Fault prediction model by fuzzy profile development of reliability relevant software metrics. Int J Comput Applic 11(6):34–41

    Article  Google Scholar 

  • Pandey AK, Goyal NK (2013) Early fault prediction model by fuzzy profile development of reliabiliy relevant software metrics. Int J Comput Applic 11(6):34–41

    Article  Google Scholar 

  • Reason J (1990) Human error. Cambridge University Press

  • RTCA (1992) Software considerations in airborne systems and equipment certification. RTCA, RTCA/DO-178

  • Sandfoss RV, Meyer SA (1997) Input requirements needed to produce an operational profile for a new telecommunications system. Eighth Int Symp Software Reliab Eng 110

  • Schneidewind N (1997) Reliability modeling for safety-critical software. IEEE Trans Reliab 46(1):88–98

    Article  Google Scholar 

  • Shi Y, Kong W, Smidts C et al. (2007) Data collection and analysis for the reliability prediction and estimation of a safety critical system. Reliab Anal Syst Failure Data 1-2

  • Shi Y, Li M, Smidts C et al (2009) On the use of extended finite state machine models for software fault propagation and software reliability estimation. 6th American Nuclear Society International Topical Meeting on Nuclear Plant Instrumentation, Controls, and Human Machine Interface Technology. Knoxville, Tennessee

  • Smidts C, Li M (2000) Software engineering measures for predicting software reliability in safety critical digital systems. USNRC, Washington DC, NUREG/GR-0019

  • Smidts CS, Li M (2004) Preliminary validation of a methodology for assessing software quality. U.S. Nuclear Regulatory Commission, Washington, DC, NUREG/CR-6848

  • Smidts C, Shi Y (2009) A test coverage-based model for predicting software fault content and location. In: Slezak D, et al. (ed) Advances in software engineering 321-329

  • Smidts C, Shi Y (2010) Predicting the types and locations of faults introduced during an imperfect repair process and their impact on reliability international journal of system assurance engineering and management. Int J Syst Assurance Eng Manag 1(1):36–43

    Article  Google Scholar 

  • Smidts CS, Shi Y, Li M, Kong W, Dai J (2011) A Large Scale Validation of A Methodology for Assessing Software Reliability. U.S. Nuclear Regulatory Commission, Washington, DC, NUREG/CR-7042.

  • Sterritt R, Bustard D (2010) Autonomic computing—a means of achieving dependability?. 10th IEEE Int Conf Workshop Eng Comput-Based Syst 247-251

  • Stutzke MA, Smidts CS (2001) A stochastic model of fault introduction and removal during software development. IEEE Trans Reliab Eng 50(2):184–193

    Article  Google Scholar 

  • Thomason MG, Whittaker JA (1999) Properties of rare fail-states and extreme values of TTF in a Markov Chain Model for Software Reliabilit. University of Tennessee, Knoxville

    Google Scholar 

  • Tukey JW (1953) The Problem of Multiple Comparisons. Technical report, Princeton University

  • Ullah N, Morisio M (2013) An empirical analysis of open source software defects data through software reliability growth models. 2013 I.E. EUROCON 460–466

  • van Manen S, Brandt E, van Ekris J, Geurts W (2013) TOPAAS: an alternative approach to software reliability quantification. Int J Quality Reliab Eng 31(2):183–191

    Article  Google Scholar 

  • Varkoi T (2013) Safety as a process quality characteristic. In: Woronowicz T, Rout T, O’Connor R, Dorling A (ed) Software process improvement and capability determination. Bremen, Germany, pp. 1-12

  • Voas JM (1992) PIE: a dynamic failure-based technique. IEEE Trans Softw Eng 18(8):717–727

    Article  Google Scholar 

  • Welker EL, Lipow M (1974) Estimating the exponential failure rate from data with no failure events. 1974 Ann Reliab Maintain Conf, Los Angeles 420-427

  • Wu H, Zhong Y, Chen Y et al. (2010) A software reliability prediction model based on benchmark measurement. 2nd IEEE Conf Inform Manag Eng (ICIME), 131-134

  • Yadav HB, Yadav DK (2014) Early software reliability analysis using reliability relevant software metrics. Int J Syst Assurance Eng Manag. doi:10.1007/s13198-014-0325-3

    Google Scholar 

  • Yadav HB, Yadav DK (2015) A fuzzy logic approach for multistage defects density analysis of software. 4th Int Conf Soft Comput Problem Solving, 122-136

  • Yadav HB, Yadav DK (2015b) A fuzzy logic based approach for phase-wise software defects prediction using software metrics. Inf Softw Technol 63(1):44–57

    Article  Google Scholar 

  • Yadav DK, Chaturvedi SK, Misra RB (2012) Early software defects prediction using fuzzy logic. Int J Perform Eng 8(4):399

    Google Scholar 

  • Zhang X, Pham H (2000) An analysis of factors affecting software reliability. J. Syst Software 50(1): 14

Download references

Acknowledgments

We would like to acknowledge the support for portions of this research through a contract with the U.S. Nuclear Regulatory Commission (NRC) (NRC-04-09-146). This paper was prepared as an account of work sponsored by an agency of the U.S. Government. Neither the U.S. Government nor any agency thereof, nor any of their employees, makes any warranty, expressed or implied, or assumes any legal liability or responsibility for any third party’s use, or the results of such use, of any information, apparatus, product, or process disclosed in this report, or represents that its use by such third party would not infringe privately owned rights. The views expressed in this paper are not necessarily those of the U.S. Nuclear Regulatory Commission.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Carol Smidts.

Additional information

Communicated by: Vittorio Cortellessa

Appendices

Appendix A. Thirteen Software Metrics used in this Study

1.1 Bugs per Line of Code (BLOC)

The goal of this measure is to give a crude estimate of the number of faults in a program module per line of code.

1.2 Cause & Effect Graphing (CEG)

A CEG is a formal translation of natural-language specification into its input conditions and expected outputs. The graph depicts a combinatorial logic network. CEG aids in identifying requirements that are incomplete and ambiguous. This measure explores the inputs and expected outputs of a program and identifies the ambiguities. Once these ambiguities are eliminated, the specifications are considered complete and consistent. The measure is computed as follows:

$$ CE\%=100\times \left(1-\frac{A_{existing}}{A_{tot}}\right) $$

Where:

A existing :

is the number of ambiguities in a program remaining to be eliminated and

A tot :

is the total number of ambiguities identified.

1.3 Software Capability Maturity Model (CMM)

CMM is a framework that describes the key elements of an effective software process. The goal of this measure is to describe the principles and practices underlying software-process maturity and to help software organizations improve the maturity of their software processes (IEEE 1988):

$$ CMM=i\kern0.36em i\in \left\{1,2,3,4,5\right\} $$

1.4 Completeness (COM)

The COM measure determines the completeness of the software requirements specifications (SRS). This measure provides a systematic guideline to identify the incompleteness defects in SRS. COM is the weighted sum of ten derivatives, D1 through D10 (IEEE 1988; Chapman and Solomon 2002):

$$ COM={\displaystyle \sum_{i=1}^{10}{w}_i{D}_i} $$

Where:

COM :

is the completeness measure

W i :

is the weight of the ith derived measure

Di :

is the i th derived measure calculated from the primitive measures B i (i = 1,…,18).

1.5 Cyclomatic Complexity (CC)

This measure determines the structural complexity of a coded module. The Cyclomatic Complexity (CC) of a module is the number of linearly independent paths through a module. The cyclomatic complexity for the i th module is originally defined by McCabe (Murine 1985; McCabe 1976) as:

$$ C{C}_i={E}_i-{N}_i+1 $$

Where:

CC i :

is the cyclomatic complexity measure of the ith module

E i :

is the number of edges of the i th module (program flows between nodes)

N i :

is the number of nodes of the i th module (sequential groups of program statements).

1.6 Coverage Factor (CF)

The coverage factor is defined as the probability that the system recovers from a fault given that a fault occurs and can be formalized as follows:

$$ c= \Pr \left(H(g)=1\left|g\in G\right.\right): $$

Where:

Pr is the probability of H(g) = 1 when g ∈ G;

H is a variable characterizing the handling of a particular faulty condition

$$ H(g)=\left\{\begin{array}{c}\hfill 1,\; if\; the\; mechanism\; correctly\; handles\; the faulty\; condition\;g;\hfill \\ {}\hfill 0,\; otherwise\hfill \end{array}\right. $$

G is the complete input space of a fault tolerance mechanism

g is a faulty condition, or a point in space G.

1.7 Defect Density (DD)

Defect density is defined as the number of defects remaining divided by the number of lines of code in the software. The defects are discovered by independent inspection. Defect Density is given as:

$$ DD=\frac{{\displaystyle {\sum}_{i=1}^I{D}_i}}{KSLOC} $$

Where:

D i :

is the number of unique defects detected during the ith design or code inspection that still remain in the code.

KSLOC :

is the number of source lines of code (LOC) in thousands.

1.8 Fault Days Number (FD)

This measure represents the number of days that faults remain in the software system from introduction to removal. The fault day measure evaluates the number of days between the time a fault is introduced into a system and until the point the fault is detected and removed (Smidts & Li 2000; McCabe 1982), such that:

$$ F{D}_i=fou{t}_i-fi{n}_i\mathrm{and}\;\mathrm{F}\mathrm{D}={\displaystyle {\sum}_{i=1}^IF{D}_i} $$

Where:

FD :

is the fault-days for the total system

FD i :

is the Fault-days for the ith fault

fin i :

is the date at which the ith fault was introduced into the system

fout i :

is the date at which the ith fault was removed from the system

1.9 Function Point (FP)

Function Point is a measure designed to determine the functional size of the software. The Function Point Counting Practices Manual is the definitive description of the Function Pointing Counting Standard. The latest version is Release 4.2, which was published in 2004 (Herrmann 2000).

1.10 Requirement Specification Change Request (RSCR)

RSCR is defined as the number of change requests that are made to the requirements specification. This measure indicates the stability and/or growth of the functional requirements. The requested changes are counted from the first release of the requirements specification document to the time when the product begins its operational life.

RSCR = Σ(requested changes to the requirements specification)

Where:

The summation is taken all requirements change requests initiated during the software development life cycle.

1.11 Requirements Traceability (RT)

According to IEEE (IEEE 1988), the requirements traceability measure aids in identifying requirements that are either missing from, or in addition to, the original requirements. Requirements traceability is defined as:

$$ RT=\frac{R_1}{R_2}\times 100\% $$

Where:

RT :

is the value of the measure requirements traceability

R 1 :

is the number of requirements met by architecture, and

R 2 :

is the number of original requirements.

1.12 System Design Complexity (SDC)

The goal of SDC is to identify the system complexity. System design complexity is a function of the average complexity of the various modules where module design complexity is a combination of structural complexity and data complexity:

$$ SDC=\frac{1}{n}{\displaystyle \sum_{i=1}^n{f}^2(i)+}\frac{1}{n}{\displaystyle \sum_{i=1}^n\frac{\vartheta (i)}{f(i)+1}} $$

Where:

n :

is the total number of the modules in the system

f(i) :

is the fanout of the i th module and

ϑ(i) :

is the number of I/O variables in the i th module

1.13 Test Coverage (TC)

As in IEEE (IEEE 1988), Test coverage (TC) is the percentage of requirement primitives implemented multiplied by the percentage of primitives executed during a set of tests. A simple interpretation of test coverage can be expressed by the following formula:

$$ TC\%=\left(\frac{IC}{RC}\right)\times \left(\frac{PRT}{TPP}\right)\times 100 $$

Where:

IC :

is the implemented capabilities

RC :

is the required capabilities

PPT :

is the tested program primitives and

TPP :

is the total program primitives.

Appendix B.The M-D Models for Each of the Thirteen RePSs

2.1 Bugs per Line of Code (BLOC)

Gaffney (Gaffney 1984) established that the number of defects remaining in the software (N G ) could be expressed empirically as a function of the number of line of codes:

$$ {N}_G={\displaystyle \sum_{i=1}^M\left(4.2+{0.0015}^3\sqrt{{S_i}^4}\right)} $$

Where:

i :

is the module index

M :

is the number of modules, and

S i :

is the number of lines of code for the ith module.

To obtain the M-D model, the next step is the partitioning of the defects based on their criticality. Jones (Jones 1996) classifies defects into four severity levels, given as: 1) Severity level 1: Critical problem (software does not operate at all); 2) Severity level 2: Significant problem (major feature disabled or incorrect); 3) Severity level 3: Minor problem (some inconvenience for the users); 4) Severity level 4: Cosmetic problem (spelling errors in messages; no effect on operations). Only defects of Severity level 1 and Severity level 2, labeled critical defects and significant defects, should be considered while estimating software reliability. Defects with Severity level 3 and level 4, called minor defects and cosmetic defects do not have an impact on the functional performance of the software system. Thus, they have no effect on reliability quantification. Using Table 13 from (Jones 1996), for “US Average Percentages for Delivered Defects by Severity Level” which provides proportions of delivered defects at different severity levels and various functional sizes and logarithmic interpolation, the percentages of delivered defects by severity levels for a software system of the functional size of APP can be obtained as.

Table 13 Percentages for delivered defects by severity level

So the number of delivered defects of interest (N) can be obtained as:

$$ {N}_{e, BLOC}^C={N}_G\times SL(FP) $$

Where:

SL(FP) is the percentage of defects introduced at the severity level of interest, i.e. at severity level 1and level 2 for a software system of functional size FP. This value for APP is equal to 0.1391 (0.0185 + 0.1206).

2.2 Cause & Effect Graphing (CEG)

CEG is a measurement of the ratio of resolved ambiguities in the SRS (see Appendix A). To determine the value of CEG, inspections of the SRS are carried out using well defined inspection rules. Detailed inspection rules are provided in (Smidts et al. 2011). As ambiguities (i.e. defects) are uncovered their descriptions, their locations and their types are recorded. Information of value to the reliability prediction process discussed in this paper relates to unresolved defects and includes Nf,CEG C = Aexisting, the number of remaining ambiguities/defects in the current version identified through CEG measurement, Li, CEG (i = 1, Nf,CEG C), location of the remaining defects/ambiguities in the current version identified through CEG measurement and Tyi,CEG (i = 1, Nf,CEG C), defect type.

2.3 Software Capability Maturity Model (CMM)

Historical industry data collected by Software Productivity Research Inc (Jones 1995) links the CMM level to the number of defects per function points. Table 14 presents this data.

Table 14 CMM Levels and average number of defects per function point

Using Table 13 from (Jones 1996) for “US Averages Percentages for Delivered Defects by Severity Level” and logarithmic interpolation, the percentages of delivered defects by severity level can be obtained.

Therefore, the number of defects based on the measurement of CMM can be expressed as:

$$ {N}_{e,CMM}^C=av{n}_{d/fp,CMM}\times FP\times SL(FP) $$

Where:

avn d/fp,CMM :

is the average number of defects for a certain CMM level

FP :

is the number of function points for the software system under study

SL(FP) :

is the percentage of defects introduced at the severity level of interest (i.e. severity level 1 and severity level 2) for a software system of functional size FP. This value for APP is equal to 0.1391 (0.0185 + 0.1206).

2.4 Completeness (COM)

COM is a measurement of the completeness of the requirements (SRS) (see Appendix A). To determine the value of COM, inspections of the SRS are carried out using well defined inspection rules. Detailed inspection rules are provided in (Smidts et al. 2011).

These inspections uncover defects such as functions which are not satisfactorily defined, data references that have no origin etc. As defects are uncovered their descriptions, their locations and their types are recorded. Information of value to the reliability prediction process discussed in this paper relate to unresolved defects and include Nf,COM C = ∑ i = 7 i = 1 B (2i + 1) + B 16 + B 18, the number of remaining defects in the current version identified through COM measurement, Li, COM C (i = 1, Nf,COM C), location of the remaining defects in the current version identified through COM measurement and Tyi,COM C (i = 1, Nf,COM C), defect type.

2.5 Cyclomatic Complexity (CC)

An empirical correlation was derived to relate cyclomatic complexity and number of defects. The correlation is based on an experimental data set composed of system software applications developed by graduate students. The empirical correlation was established by following the form proposed in (International Function Point Users Group 2004):

$$ SL{I}_{CC}=1-{\displaystyle \sum_{i=1}^9{f}_i\times {p}_i\%} $$

Where:

SLI CC :

is the SLI value of the cyclomatic complexity factor

f i :

is the failure likelihood fi used for SLI1 calculations (International Function Point Users Group 2004)

p i :

is the percentage of modules whose cyclomatic complexity belong to the ith level, i = 1, 2, …, 9.

SLI stands for Success Likelihood Index which is used to represent the likelihood of an error occurring in a particular situation depends on the combined effects of a relatively small number of performance influencing factors (PIFs). SLI was used as an index which quantifies whether a particular environment will increase the human error probability or decrease it (with respect to a “normal situation”) (Malaiya et al. 1994). SLICC is related to the likelihood that developers will err (i.e. introduce fault in the software product and/or fail to remove them) because of the cyclomatic complexity of the modules.

The number of defects predicted based on CC measurement results is:

$$ {N}_{e,CC}^C=0.036\times SIZE\times {(20)}^{1-2SL{I}_{CC}} $$

Where:

SIZE:

is the size of the delivered source code in terms of LOC (Line of Code)

2.6 Coverage Factor (CF)

The coverage factor is defined as the probability of system recovery given that a fault exists (see Appendix A). Coverage analysis specifically targets hardware failures that might be experienced by the microprocessor on which the safety critical software runs. The goal is to identify faults in the fault tolerance mechanism. The number of defects used for reliability prediction is the total number of hardware faults injected to the system while the system could not recovery from the faulty state to normal/fail-safe/recoverable state (Smidts et al. 2011), i.e. Nf,CF C = ∑ n i = 1 N i unre cov erable .

2.7 Defect Density (DD)

DD is a measurement of the ratio of defects identified in the software (see Appendix A). To determine the value of DD inspections of the SRS, SDD and code are carried out using well defined inspection rules. Detailed inspection rules are provided in (Smidts et al. 2011). These inspections uncover defects. As defects are uncovered their descriptions, their locations and their types are recorded. Information of value to the reliability prediction process discussed in this paper relate to unresolved defects in the current version and include Nf,DD C = ∑ I i = 1 D i  × u C i (where μ C i is the fraction of defects unresolved in the current version), the number of remaining defects in the current version identified through DD measurement, Li, COM (i = 1, Nf,DD C), location of the remaining defects in the current version identified through DD measurement and Tyi,DD (i = 1, Nf,DD C), defect type.

2.8 Fault Days Number (FD)

Based on the cumulative characteristic of the FD metric and by using the concepts introduced in (Stutzke & Smidts 2001), one can show that FD is related to μ U (t) by the following equation:

$$ \frac{d(FD)}{dt}={\mu}_U(t) $$

Where:

μ U (t):

is the fault count at time t

This equation shows the direct relationship between the measured real FD and the corresponding fault count. The number of faults can be obtained using this equation once FD is known (i.e. measured). However, the real FD cannot be obtained experimentally since not all the faults can be discovered during the inspection. One can only obtain the apparent FD, FDA which corresponds to faults identified through the inspection process and removed through repair. One can relate FDA to FD by:

$$ \frac{d\left(F{D}_A\right)}{dt}=\gamma \left(t;\vartheta {\mu}_H,{z}_a,{\mu}_R\right)\cdot \frac{d(FD)}{dt} $$

Where:

γ(t; ϑμ H , z a , μ R ):

is a function of ϑμ H , z a , μ R

ϑμ H :

is the estimate of the fault introduction rate

z a :

is the intensity function of per-fault detection

μ R :

is the expected change in fault count due to each repair.

Therefore, one can still obtain the fault count based on the measured apparent FD as shown by:

$$ {\mu}_U(t)=\frac{d\left(F{D}_A\right)}{dt}\cdot \frac{1}{\gamma \left(t;\vartheta {\mu}_H,{z}_a,{\mu}_R\right)} $$

The fault count at time t, μ U (t) can be related to Ne,FD C by:

$$ {N}_{e,FD}^C={\mu}_U\left({t}_c\right) $$

Where:

tC :

is the time at which development activities carried out on the current version end.

In addition to this information, defect information related to the previous version of the software is also available and can be used for reliability prediction. Information of value to the reliability prediction process discussed in this paper then includes Nf,FD P, the number of defects found in the previous version of the software, Li,FD P (i = 1, Nf,FD P), location of the defects found through testing of the previous version of the software and Tyi,FD P (i = 1, Nf,FD P), defect type.

2.9 Function Point (FP)

Jones’ empirical industry data (Jones 1996) links the FP to the number of defects per function point for different categories of applications. Table 15 (Table 3.46 in (Jones 1996)) provides the average numbers for delivered defects per function point for different types of software systems. Logarithmic interpolation can then be used for number of defects quantification.

Table 15 Averages for delivered defects per function point (extracted from (Jones 1996))

Therefore, the number of defects based on the measurement of FP can be expressed as:

$$ {N}_{e,FP}^C= ad{d}_{fp, systype}\times FP\times SL(FP) $$

Where:

add fp,systype :

is the average for delivered defects per function point obtained from Table 15 for a software system of type systype. For APP, systype=”systems”

SL(FP) :

is the percentage of defects introduced at the severity level of interest (severity level 1 and 2) for a software system of functional size FP. This value for APP is equal to 0.1391 (0.0185 + 0.1206).

2.10 Requirement Specification Change Request (RSCR)

To link requirements specification change requests to the reliability of a software system, a derived measure called REVL was used. The measure is:

$$ REVL=\frac{SIZ{E}_{changed\; due\;to\; RSCR}}{SIZ{E}_{delivered}}\times 100\% $$

Where:

REVL :

is a measure of requirements evolution and volatility

SIZE changed due to RSCR :

is the size of changed source code corresponding to RSCR, in Kilo Line of Code (KLOC)

SIZE delivered :

is the size of the delivered source code, in KLOC.

SLI :

for the REVL, denoted by SLI RSCR , is estimated using the value of REVL, as shown in Table 16 (Table 17).

Table 16 Rating scale and SLI estimation for REVL
Table 17 Acronyms and abbreviations

The number of defects predicted based on RSCR measurement results is:

$$ {N}_{e, REVL}^C=0.036\times SIZ{E}_{delivered}\times {(20)}^{1-2SL{I}_{RSCR}} $$

2.11 Requirements Traceability (RT)

The requirements traceability measure aids in identifying software implementation defects such as requirements that are either missing from, or in addition to, the original requirements (see Appendix A). To determine the value of RT inspections of the SRS, SDD and code are carried out using well defined inspection rules. Detailed inspection rules are provided in (Smidts et al. 2011). As defects are uncovered their descriptions, their locations and their types are recorded. Information of value to the reliability prediction process discussed in this paper relate to unresolved defects in the current version and include Nf,RT C , the number of remaining defects in the current version identified through RT measurement, Li, RT (i = 1, Nf,RT C), location of the remaining defects in the current version identified through RT measurement and Tyi,RT (i = 1, Nf,RT C), defect type.

2.12 System Design Complexity (SDC)

The empirical relations between SDC and defect rate are investigated in (Card and Agresti 1988) and the expected number of defect can be estimated using the following equation:

$$ {N}_{e,SDC}^C=\left(0.4124\times SDC-4.6599\right)\times SIZ{E}_{delivered}\times SL(FP) $$

Where:

SDC :

is the measured system design complexity

SIZE developed :

is the developed line of code (in KLOC) which includes all newly developed source lines of code plus 20 % of reused source lines of code and

SL(FP) :

is the percentage of defects introduced at the severity level of interest (severity level 1 and 2) for a software system of functional size FP. This value for APP is equal to 0.1391 (0.0185 + 0.1206).

2.13 Test Coverage (TC)

As noted in Appendix A, test coverage (TC) is the percentage of requirement primitives implemented multiplied by the percentage of primitives executed during a set of tests. TC is defined as:

$$ TC\%=\left(\frac{IC}{RC}\right)\times \left(\frac{PRT}{TPP}\right)\times 100 $$

If we assume that all requirements have been implemented and if we further assume that the tested program primitives are represented by the lines of code, the definition of test coverage then becomes:

$$ {C}_1=\frac{LO{C}_{Tested}}{LO{C}_{Total}}\times 100\% $$

Where:

C 1 :

is test coverage obtained through testing.

LOC Tested :

is the number of lines of code that are being executed by the test data listed in the test plan.

LOC Total :

is the total number of lines of code.

Malaiya et al. investigated the relationship between defect coverage, C0, and statement coverage, C1. In (Malaiya et al. 1994), the following relationship was proposed:

$$ {C}_o={a}_0 \ln \left(1+{a}_1{e}^{a_2{c}_{1-1}}\right) $$

Where:

a 0 , a 1 , a 2 :

are coefficients

C 1 :

is the statement coverage.

The number of defects remaining in the software then becomes N C e,TC is:

$$ {N}_{e,TC}^C=\frac{N_0}{C_0} $$

Where:

N 0 :

is the number of defects found by test cases provided in the test plan

C 0 :

is the defect coverage.

In addition to this information, defect information related to the N0 defects identified by the test plan is also available and can be used for reliability prediction. Information of value to the reliability prediction process discussed in this paper then includes Nf,TC P = N0, the number of defects found in the previous version of the software, Li, TC P (i = 1, Nf,TC P), location of the defects found through testing of the previous version of the software and Tyi,TC P (i = 1, Nf,TC P), defect type.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shi, Y., Li, M., Arndt, S. et al. Metric-based software reliability prediction approach and its application. Empir Software Eng 22, 1579–1633 (2017). https://doi.org/10.1007/s10664-016-9425-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-016-9425-9

Keyword

Navigation