Ermira Daka
ABSTRACT
The name of a unit test helps developers to understand the purpose and scenario of the test, and test names support developers when navigating amongst sets of unit tests. When unit tests are generated automatically, however, they tend to be given non-descriptive names such as “test0”, which provide none of the benefits a descriptive name can give a test. The underlying challenge is that automatically generated tests typically do not represent real scenarios and have no clear purpose other than covering code, which makes naming them di cult. In this paper, we present an automated approach which generates descriptive names for automatically generated unit tests by summarizing API-level coverage goals. The tests are optimized to be short, descriptive of the test, have a clear relation to the covered code under test, and allow developers to uniquely distinguish tests in a test suite. An empirical evaluation with 47 participants shows that developers agree with the synthesized names, and the synthesized names are equally descriptive as manually written names. Study participants were even more accurate and faster at matching code and tests with synthesized names compared to manually derived names.
Supplemental Material
Available for Download
zip
issta17-mainid95-s.zip (77.5 MB)
This supplementary material includes the artifact package evaluated and accepted by the ISSTA 2017 Artifact Evaluation Committee. The artifact package aims to support the reproduction of all the experimental results presented in the paper and the internal and external replication of the empirical evaluation reported in the paper. A refined, up-to-date version of this artifact will be maintained at http://readability.evosuite.org/test-naming/.
- S. Afshan, P. McMinn, and M. Stevenson. Evolving readable string test inputs using a natural language model to reduce human oracle cost. In IEEE Int. Conference on Software Testing, Veri�cation and Validation (ICST), pages 352–361, 2013. Google Scholar
Digital Library
- Agitar One, 2016. Last visited on 30.08.2016.Google Scholar
- M. Allamanis, E. T. Barr, C. Bird, and C. Sutton. Suggesting accurate method and class names. In ACM Symposium on the Foundations of Software Engineering (FSE), pages 38–49, New York, NY, USA, 2015. ACM. Google ScholarDigital Library
- N. Alshahwan and M. Harman. Augmenting test suites e�ectiveness by increasing output diversity. In ACM/IEEE Int. Conference on Software Engineering (ICSE), pages 1345–1348. IEEE Press, 2012. Google ScholarDigital Library
- J. H. Andrews, F. C. H. Li, and T. Menzies. Nighthawk: A two-level geneticrandom unit test data generator. In IEEE/ACM Int. Conference on Automated Software Engineering (ASE), pages 144–153. ACM, 2007. Google ScholarDigital Library
- L. Baresi, P. L. Lanzi, and M. Miraz. Testful: An evolutionary test approach for java. In IEEE Int. Conference on Software Testing, Veri�cation and Validation (ICST), pages 185–194. IEEE Computer Society, 2010. Google ScholarDigital Library
- K. H. Bennett and V. T. Rajlich. Software maintenance and evolution: A roadmap. In ACM/IEEE Int. Conference on Software Engineering (ICSE), pages 73–87, New York, NY, USA, 2000. ACM. Google ScholarDigital Library
- M. Boshernitsan, R. Doong, and A. Savoia. From Daikon to Agitator: Lessons and challenges in building a commercial tool for developer testing. In ACM Int. Symposium on Software Testing and Analysis (ISSTA), pages 169–180. ACM, 2006. Google ScholarDigital Library
- J. Carver, L. Jaccheri, S. Morasca, and F. Shull. Issues in using students in empirical studies in software engineering education. In IEEE Int. Software Metrics Symposium, pages 239–249, 2003. Google ScholarDigital Library
- Codepro analytix, 2017. Last visited on 30.01.2017.Google Scholar
- C. Csallner and Y. Smaragdakis. JCrasher: An automatic robustness tester for java. Softw. Pract. Exper., 34(11):1025–1050, Sept. 2004. Google ScholarDigital Library
- E. Daka, J. Campos, G. Fraser, J. Dorn, and W. Weimer. Modeling readability to improve unit tests. In ACM Symposium on the Foundations of Software Engineering (FSE), pages 107–118. ACM, 2015. Google ScholarDigital Library
- G. Fraser and A. Arcuri. EvoSuite: Automatic test suite generation for objectoriented software. In ACM Symposium on the Foundations of Software Engineering (FSE), pages 416–419. ACM, 2011. Google ScholarDigital Library
- G. Fraser and A. Arcuri. Whole test suite generation. IEEE Transactions on Software Engineering (TSE), 39(2):276–291, 2013. Google ScholarDigital Library
- G. Fraser and A. Arcuri. A large scale evaluation of automated unit test generation using evosuite. ACM Transactions on Software Engineering and Methodology (TOSEM), 24(2):8, 2014. Google ScholarDigital Library
- G. Fraser and A. Zeller. Exploiting common object usage in test case generation. In IEEE Int. Conference on Software Testing, Veri�cation and Validation (ICST), pages 80–89, 2011. Google ScholarDigital Library
- G. Fraser and A. Zeller. Mutation-driven generation of unit tests and oracles. IEEE Transactions on Software Engineering (TSE), 38(2):278–292, 2012. Google ScholarDigital Library
- J. P. Galeotti, G. Fraser, and A. Arcuri. Improving search-based test suite generation with dynamic symbolic execution. In IEEE Int. Symposium on Software Reliability Engineering (ISSRE), pages 360–369. IEEE, 2013.Google Scholar
- E. W. Høst and B. M. Østvold. Debugging method names. In IEEE European Conference on Object-Oriented Programming (ECOOP), volume 5653 of Lecture Notes in Computer Science, pages 294–317. Springer, 2009. Google ScholarDigital Library
- M. Höst, B. Regnell, and C. Wohlin. Using students as subjects—A comparative study of students and professionals in lead-time impact assessment. Empirical Software Engineering, 5(3):201–214, 2000. Google ScholarDigital Library
- K. Inkumsah and T. Xie. Evacon: A framework for integrating evolutionary and concolic testing for object-oriented programs. In IEEE/ACM Int. Conference on Automated Software Engineering (ASE), pages 425–428. ACM, 2007. Google ScholarDigital Library
- Parasoft JTest, 2016. Last visited on 30.08.2016.Google Scholar
- R. Just. The Major mutation framework: E�cient and scalable mutation analysis for Java. In ACM Int. Symposium on Software Testing and Analysis (ISSTA), pages 433–436, 2014. Google ScholarDigital Library
- B. A. Kitchenham, S. L. P�eeger, L. M. Pickard, P. W. Jones, D. C. Hoaglin, K. E. Emam, and J. Rosenberg. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering (TSE), 28(8):721– 734, Aug. 2002. Google ScholarDigital Library
- Y. Lei and J. H. Andrews. Minimization of randomized unit test cases. In IEEE Int. Symposium on Software Reliability Engineering (ISSRE), pages 267–276. IEEE Computer Society, 2005. Google ScholarDigital Library
- A. Leitner, M. Oriol, A. Zeller, I. Ciupa, and B. Meyer. E�cient unit test case minimization. In IEEE/ACM Int. Conference on Automated Software Engineering (ASE), pages 417–420. ACM, 2007. Google ScholarDigital Library
- B. Li, C. Vendome, M. Linares-Vásquez, D. Poshyvanyk, and N. A. Kraft. Automatically documenting unit test cases. In IEEE Int. Conference on Software Testing, Veri�cation and Validation (ICST), pages 341–352, 2016.Google ScholarCross Ref
- L. Ma, C. Artho, C. Zhang, H. Sato, J. Gmeiner, and R. Ramler. GRT: An automated test generator using orchestrated program analysis. In IEEE/ACM Int. Conference on Automated Software Engineering (ASE), pages 842–847. IEEE, 2015.Google Scholar
- M. Oriol and S. Tassis. Testing .NET code with YETI. In R. Calinescu, R. F. Paige, and M. Z. Kwiatkowska, editors, ICECCS, pages 264–265. IEEE Computer Society, 2010. Google ScholarDigital Library
- R. Osherove. The Art of Unit Testing: With Examples in .Net. Manning Publications Co., Greenwich, CT, USA, 1st edition, 2009. Google ScholarDigital Library
- C. Pacheco and M. D. Ernst. Randoop: feedback-directed random testing for Java. In ACM SIGPLAN Conference on Object-oriented Programming Systems and Applications Companion (OOPSLA), pages 815–816. ACM, 2007. Google ScholarDigital Library
- F. Palomba, A. Panichella, A. Zaidman, R. Oliveto, and A. De Lucia. Automatic test case generation: what if test code quality matters? In ACM Int. Symposium on Software Testing and Analysis (ISSTA), pages 130–141. ACM, 2016. Google ScholarDigital Library
- S. Panichella, A. Panichella, M. Beller, A. Zaidman, and H. C. Gall. The impact of test case summaries on bug �xing performance: An empirical investigation. In ACM/IEEE Int. Conference on Software Engineering (ICSE), pages 547–558. ACM, 2016. Google ScholarDigital Library
- I. S. W. B. Prasetya. T3, a combinator-based random testing tool for java: Benchmarking. In T. E. J. Vos, K. Lakhotia, and S. Bauersfeld, editors, FITTEST@ICTSS, volume 8432 of Lecture Notes in Computer Science, pages 101–110. Springer, 2013.Google Scholar
- J. M. Rojas, J. Campos, M. Vivanti, G. Fraser, and A. Arcuri. Combining multiple coverage criteria in search-based unit test generation. In Int. Symposium on Search Based Software Engineering (SSBSE), volume 9275 of LNCS, pages 93–108. Springer, 2015.Google Scholar
- A. Sakti, G. Pesant, and Y.-G. Guéhéneuc. Instance generator and problem representation to improve object oriented code coverage. IEEE Transactions on Software Engineering (TSE), 41(3):294–313, 2015.Google ScholarDigital Library
- I. Salman, A. T. Misirli, and N. Juristo. Are students representatives of professionals in software engineering experiments? In ACM/IEEE Int. Conference on Software Engineering (ICSE), volume 1, pages 666–676, 2015. Google ScholarDigital Library
- S. Thummalapenta, T. Xie, N. Tillmann, J. De Halleux, and Z. Su. Synthesizing method sequences for high-coverage testing. ACM SIGPLAN Notices, 46(10):189– 206, 2011. Google ScholarDigital Library
- N. Tillmann and J. De Halleux. Pex–white box test generation for. net. In Int. Conference on Tests and Proofs (TAP), pages 134–153. Springer, 2008. Google ScholarDigital Library
- P. Tonella. Evolutionary testing of classes. SIGSOFT Softw. Eng. Notes, 29(4):119– 128, July 2004. Google ScholarDigital Library
- A. Trenk. Testing on the toilet: Writing descriptive test names. Computer, Oct. 2014.Google Scholar
- T. Xie. Augmenting automatically generated unit-test suites with regression oracle checking. In IEEE European Conference on Object-Oriented Programming (ECOOP), pages 380–403. Springer, 2006. Google ScholarDigital Library
- J. Xuan and M. Monperrus. Test case puri�cation for improving fault localization. In ACM Symposium on the Foundations of Software Engineering (FSE), pages 52–63. ACM, 2014. Google ScholarDigital Library
- B. Zhang, E. Hill, and J. Clause. Towards automatically generating descriptive names for unit tests. In IEEE/ACM Int. Conference on Automated Software Engineering (ASE), pages 625–636. ACM, 2016. Google ScholarDigital Library
- S. Zhang, C. Zhang, and M. Ernst. Automated documentation inference to explain failed tests. In IEEE/ACM Int. Conference on Automated Software Engineering (ASE), pages 63–72, 2011. Google ScholarDigital Library
Index Terms
- Generating unit tests with descriptive names or: would you name your children thing1 and thing2?
Recommendations
Towards automatically generating descriptive names for unit tests
ASE '16: Proceedings of the 31st IEEE/ACM International Conference on Automated Software EngineeringDuring maintenance, developers often need to understand the purpose of a test. One of the most potentially useful sources of information for understanding a test is its name. Ideally, test names are descriptive in that they accurately summarize both ...
Automated unit test generation during software development: a controlled experiment and think-aloud observations
ISSTA 2015: Proceedings of the 2015 International Symposium on Software Testing and AnalysisAutomated unit test generation tools can produce tests that are superior to manually written ones in terms of code coverage, but are these tests helpful to developers while they are writing code? A developer would first need to know when and how to ...
Comments