Yucheng Zhang
Ali Mesbah
ABSTRACT
Code coverage is a popular test adequacy criterion in practice. Code coverage, however, remains controversial as there is a lack of coherent empirical evidence for its relation with test suite effectiveness. More recently, test suite size has been shown to be highly correlated with effectiveness. However, previous studies treat test methods as the smallest unit of interest, and ignore potential factors influencing this relationship. We propose to go beyond test suite size, by investigating test assertions inside test methods. We empirically evaluate the relationship between a test suite’s effectiveness and the (1) number of assertions, (2) assertion coverage, and (3) different types of assertions. We compose 6,700 test suites in total, using 24,000 assertions of five real-world Java projects. We find that the number of assertions in a test suite strongly correlates with its effectiveness, and this factor directly influences the relationship between test suite size and effectiveness. Our results also indicate that assertion coverage is strongly correlated with effectiveness and different types of assertions can influence the effectiveness of their containing test suites.
- Apache commons lang. http://commons.apache.org/proper/commons-lang/.Google Scholar
- Asterisk-java. https://blogs.reucon.com/asterisk-java/.Google Scholar
- Clover. https: //www.atlassian.com/software/clover/overview/.Google Scholar
- JavaParser. https://code.google.com/p/javaparser/.Google Scholar
- JavaSlicer. https://www.st.cs.uni-saarland.de/javaslicer/.Google Scholar
- JFreeChart. http://www.jfree.org/jfreechart/.Google Scholar
- Lambdaj. https://code.google.com/p/lambdaj/.Google Scholar
- Pit. http://pitest.org.Google Scholar
- SLOCCount. http://www.dwheeler.com/sloccount/.Google Scholar
- Urban airship java library. http://docs. urbanairship.com/reference/libraries/java/.Google Scholar
- J. H. Andrews, L. C. Briand, and Y. Labiche. Is mutation an appropriate tool for testing experiments? In Proceedings of the International Conference on Software Engineering, ICSE, pages 402–411. ACM, 2005. Google ScholarDigital Library
- J. H. Andrews, L. C. Briand, Y. Labiche, and A. S. Namin. Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Trans. Softw. Eng., 32:608–624, 2006. Google ScholarDigital Library
- L. Briand and D. Pfahl. Using simulation for assessing the real impact of test coverage on defect coverage. In Proceedings of the International Symposium on Software Reliability Engineering, ISSRE, pages 148–157. IEEE Computer Society, 1999. Google ScholarDigital Library
- X. Cai and M. R. Lyu. The effect of code coverage on fault detection under different testing profiles. In Proceedings of the International Workshop on Advances in Model-based Testing, A-MOST, pages 1–7. ACM, 2005. Google ScholarDigital Library
- M. Daran and P. Thévenod-Fosse. Software error analysis: A real case study involving real faults and mutations. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA, pages 158–171. ACM, 1996. Google ScholarDigital Library
- P. G. Frankl and O. Iakounenko. Further empirical studies of test effectiveness. In Proceedings of the 6th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE, pages 153–162. ACM, 1998. Google ScholarDigital Library
- P. G. Frankl and S. N. Weiss. An experimental comparison of the effectiveness of the all-uses and all-edges adequacy criteria. In Proceedings of the Symposium on Testing, Analysis, and Verification, TAV4, pages 154–164. ACM, 1991. Google ScholarDigital Library
- P. G. Frankl and S. N. Weiss. An experimental comparison of the effectiveness of branch testing and data flow testing. IEEE Trans. Softw. Eng., 19:774–787, 1993. Google ScholarDigital Library
- R. Gopinath, C. Jensen, and A. Groce. Code coverage for suite evaluation by developers. In Proceedings of the International Conference on Software Engineering, ICSE, pages 72–82. ACM, 2014. Google ScholarDigital Library
- L. Inozemtseva and R. Holmes. Coverage is not strongly correlated with test suite effectiveness. In Proceedings of the International Conference on Software Engineering, ICSE, pages 435–445. ACM, 2014. Google ScholarDigital Library
- R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser. Are mutants a valid substitute for real faults in software testing? In Proceedings of the ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE, pages 654–665. ACM, 2014. Google ScholarDigital Library
- A. S. Namin and J. H. Andrews. The influence of size and coverage on test suite effectiveness. In Proceedings of the International Symposium on Software Testing and Analysis, ISSTA, pages 57–68. ACM, 2009. Google ScholarDigital Library
- C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball. Feedback-directed random test generation. In Proceedings of the International Conference on Software Engineering (ICSE), pages 75–84. IEEE Computer Society, 2007. Google ScholarDigital Library
- D. Schuler and A. Zeller. Assessing oracle quality with checked coverage. In Proceedings of the International Conference on Software Testing, Verification and Validation, ICST, pages 90–99. IEEE Computer Society, 2011. Google ScholarDigital Library
Index Terms
- Assertions are strongly correlated with test suite effectiveness
Recommendations
Coverage is not strongly correlated with test suite effectiveness
ICSE 2014: Proceedings of the 36th International Conference on Software EngineeringThe coverage of a test suite is often used as a proxy for its ability to detect faults. However, previous studies that investigated the correlation between code coverage and test suite effectiveness have failed to reach a consensus about the nature and ...
Using coverage effectiveness to evaluate test suite prioritizations
WEASELTech '07: Proceedings of the 1st ACM international workshop on Empirical assessment of software engineering languages and technologies: held in conjunction with the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE) 2007Regression test suite prioritization techniques reorder a test suite with the goal of ensuring that the reorganized test suite finds faults faster than the initial ordering. It is challenging to empirically evaluate the effectiveness of a new test case ...
Investigating faults missed by test suites achieving high code coverage
Highlights- Studies how effectively test suites with high code coverage find 45 different faults.
AbstractCode coverage criteria are commonly used to determine the adequacy of a test suite. However, studies investigating code coverage and fault-finding capabilities have mixed results. Some studies have shown that creating test suites to ...
Comments