Jackson A. Prado Lima
ABSTRACT
Higher Order Mutation Testing (HOMT) uses mutants derived by applying mutation operators more than once in the program under test. This kind of test increases efficacy, but can be more expensive and lead to some challenges such as the large space of mutants. The challenges can be managed in the Search-Based Software Engineering field to seek only interesting mutants that satisfy some desired conditions regarding, for example, type of revealed faults or reduction of equivalent mutants. For this reason, we observe a growing interest in search-based HOMT and an increasing number of related studies. To better characterize such studies, this paper presents results of a mapping on search-based HOMT. We found 25 primary studies on this subject and identified the most common search-based algorithms, preferred fitness functions, addressed programming languages, and evaluation aspects. In addition to this, some trends and research opportunities are identified that allow researchers to direct future investigations.
- A. Abuljadayel and F. Wedyan. An approach for the generation of higher order mutants using genetic algorithms. International Journal of Intelligent Systems and Applications (IJISA), 10(1):34--45, 2018.Google Scholar
- E. Blanco-Muñoz, A. García-Domínguez, J. J. Domínguez-Jiménez, and I. Medina-Bulo. Towards higher-order mutant generation for WS-BPEL. In Proceedings of the International Conference on e-Business, pages 1--6, Seville, Spain, July 2011.Google Scholar
- A. S. Ghiduk. Using evolutionary algorithms for higher-order mutation testing. International Journal of Computer Science Issues, 11(2):93, 2014.Google Scholar
- M. Harman, Y. Jia, and W. B. Langdon. Strong higher order mutation-based test data generation. In Proceedings of the 19th ACM SIGSOFT Symposium on the Foundations of Software Engineering and the 13th European Software Engineering Conference, ESEC/FSE '11, pages 212--222, New York, NY, USA, 2011. ACM. Google ScholarDigital Library
- M. Harman, Y. Jia, P. R. Mateo, and M. P. Usaola. Angels and monsters: An empirical investigation of potential test effectiveness and efficiency improvement from strongly subsuming higher order mutation. In Proceedings of the 29th International Conference on Automated Software Engineering, ASE '14, pages 397--408, New York, NY, USA, 2014. ACM. Google ScholarDigital Library
- Y. Jia and M. Harman. Constructing subtle faults using higher order mutation testing. In Proceedings of the 8th International Working Conference on Source Code Analysis and Manipulation, SCAM, pages 249--258, Beijing, China, Sept 2008. IEEE.Google ScholarCross Ref
- Y. Jia and M. Harman. Higher order mutation testing. Information and Software Technology, 51(10):1379--1393, 2009. Google ScholarDigital Library
- W. B. Langdon, M. Harman, and Y. Jia. Multi objective higher order mutation testing with genetic programming. In Proceedings of the Testing: Academic and Industrial Conference - Practice and Research Techniques, TAIC-PART '09, pages 21--29, Washington, DC, USA, Sept 2009. IEEE. Google ScholarDigital Library
- W. B. Langdon, M. Harman, and Y. Jia. Efficient multi-objective higher order mutation testing with genetic programming. Journal of Systems and Software, 83(12):2416--2430, 2010. Google ScholarDigital Library
- W. B. Langdon, N. Veerapen, and G. Ochoa. Visualising the search landscape of the triangle program. In Proceedings of the 20th European Conference on Genetic Programming, volume 10196 LNCS of EuroGP, pages 96--113, Berling, Germany, 2017. Springer.Google ScholarCross Ref
- P. R. Mateo and M. P. Usaola. Bacterio: Java mutation testing tool: A framework to evaluate quality of tests cases. In Proceedings of the 28th International Conference on Software Maintenance, ICSM, pages 646--649. IEEE, Sept. 2012. Google ScholarDigital Library
- P. R. Mateo, M. P. Usaola, and J. L. Fernández-Alemán. Validating second-order mutation at system level. IEEE Transactions on Software Engineering, 39(4):570--587, April 2013. Google ScholarDigital Library
- Q. V. Nguyen and L. Madeyski. Searching for strongly subsuming higher order mutants by applying multi-objective optimization algorithm. In H. A. Le Thi, N. T. Nguyen, and T. V. Do, editors, Advanced Computational Methods for Knowledge Engineering, volume 358, pages 391--402. Springer, Metz, France, 2015.Google Scholar
- Q. V. Nguyen and L. Madeyski. Empirical evaluation of multiobjective optimization algorithms searching for higher order mutants. Cybernetics and Systems, 47(1-2):48--68, Jan. 2016.Google ScholarDigital Library
- Q. V. Nguyen and L. Madeyski. Higher order mutation testing to drive development of new test cases: An empirical comparison of three strategies. In Proceedings of the 8th Asian Conference on Intelligent Information and Database Systems, volume 9621 of ACIIDS, pages 235--244, Da Nang, Viet nam, Mar. 2016. Springer Berlin Heidelberg.Google ScholarCross Ref
- Q. V. Nguyen and L. Madeyski. On the relationship between the order of mutation testing and the properties of generated higher order mutants. In N. T. Nguyen, B. Trawiński, H. Fujita, and T.-P. Hong, editors, Proceedings of the 8th Asian Conference on Intelligent Information and Database Systems, ACIIDS, pages 245--254, Da Nang, Viet nam, Mar. 2016. Springer.Google ScholarCross Ref
- Q. V. Nguyen and L. Madeyski. Addressing mutation testing problems by applying multi-objective optimization algorithms and higher order mutation. Journal of Intelligent and Fuzzy Systems, 32(2):1173--1182, 2017.Google ScholarCross Ref
- Q. V. Nguyen and D.-T.-H. Pham. Is higher order mutant harder to kill than first order mutant? an experimental study. In N. T. Nguyen, D. H. Hoang, T.-P. Hong, H. Pham, and B. Trawiński, editors, Proceedings of the Intelligent Information and Database Systems, pages 664--673, Cham, 2018. Springer International Publishing.Google Scholar
- E. Omar, S. Ghosh, and D. Whitley. Constructing subtle higher order mutants for Java and AspectJ programs. In Proceedings of the 24th International Symposium on Software Reliability Engineering, ISSRE, pages 340--349, Pasadena, CA, United states, Nov 2013. IEEE.Google ScholarCross Ref
- E. Omar, S. Ghosh, and D. Whitley. Comparing search techniques for finding subtle higher order mutants. In Proceedings of the Annual Conference on Genetic and Evolutionary Computation, GECCO '14, pages 1271--1278, New York, NY, USA, 2014. ACM. Google ScholarDigital Library
- E. Omar, S. Ghosh, and D. Whitley. HOMAJ: A tool for higher order mutation testing in AspectJ and Java. In Proceedings of the 27th International Conference on Software Testing, Verification and Validation Workshops, ICSTW, pages 165--170, Washington, DC, USA, March 2014. IEEE. Google ScholarDigital Library
- E. Omar, S. Ghosh, and D. Whitley. Subtle higher order mutants. Information and Software Technology, 81:3--18, Jan. 2017. Google ScholarDigital Library
- J. A. Prado Lima, G. Guizzo, S. R. Vergilio, A. P. C. Silva, H. L. J. Filho, and H. V. Ehrenfried. Evaluating different strategies for reduction of mutation testing costs. In Proceedings of the 1st Brazilian Symposium on Systematic and Automated Software Testing, SAST, pages 4:1--4:10, Maringa, Brazil, 2016. ACM. Google ScholarDigital Library
- J. A. Prado Lima and S. R. Vergilio. A multi-objective optimization approach for selection of second order mutant generation strategies. In Proceedings of the 2nd Brazilian Symposium on Systematic and Automated Software Testing, SAST, pages 1--10. ACM, 2017. Google ScholarDigital Library
- F. Wu, M. Harman, Y. Jia, and J. Krinke. HOMI: Searching higher order mutants for software improvement. In F. Sarro and K. Deb, editors, Proceedings of the 8th International Symposium on Search Based Software Engineering, SSBSE, pages 18--33, Raleigh, NC, USA, Oct. 2016. Springer.Google Scholar
- {n. d.}. The RedMonk Programming Language Rankings: January 2018s. https://redmonk.com/sogrady/2018/03/07/language-rankings-1-18/. Accessed: 2018-03-20.Google Scholar
- S. Anand, E. K. Burke, T. Y. Chen, J. Clark, M. B. Cohen, W. Grieskamp, M. Harman, M. J. Harrold, and P. McMinn. 2013. An orchestrated survey of methodologies for automated software test case generation. Journal of Systems and Software 86, 8 (2013), 1978--2001. Google ScholarDigital Library
- M. Dave and R. Agrawal. 2016. Communications in Computer and Information Science. Vol. 628. Springer, Chapter Mutation Testing and Test Data Generation Approaches: A Review, 373--382.Google Scholar
- T. N. Ferreira, S. R. Vergilio, and J. T. de Souza. 2017. Incorporating user preferences in search-based software engineering: A systematic mapping study. Information and Software Technology 90 (2017), 55--69.Google Scholar
- A. S. Ghiduk, M. R. Girgis, and M. H. Shehata. 2017. Higher order mutation testing: A Systematic Literature Review. Computer Science Review 25 (2017), 29--48.Google ScholarCross Ref
- M. Harman, E. Burke, J. Clark, X. Yao, et al. 2012. Dynamic adaptive search based software engineering. In ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 1--8. Google ScholarDigital Library
- M. Harman, Y. Jia, and W. B. Langdon. 2010. A Manifesto for Higher Order Mutation Testing. In Proceedings of the 3rd International Conference on Software Testing, Verification, and Validation Workshops (ICSTW '10). IEEE, 80--89. Google ScholarDigital Library
- M. Harman, S. A. Mansouri, and Y. Zhang. 2012-12. Search-based Software Engineering: Trends, Techniques and Applications. Comput. Surveys 45, 1 (2012-12), 11:1--11:61. Google ScholarDigital Library
- Y. Jia and M. Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (2011), 649--678. Google ScholarDigital Library
- L. Madeyski, W. Orzeszyna, R. Torkar, and M. Jozala. 2014. Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation. IEEE Transactions on Software Engineering 40, 1 (Jan. 2014), 23--42. Google ScholarDigital Library
- T. Mariani and S. R. Vergilio. 2017-03. A systematic review on search-based refactoring. Information and Software Technology 83 (2017-03), 14--34. Google ScholarDigital Library
- P. McMinn. 2004. Search-based software test data generation: A survey. Software Testing Verification and Reliability 14, 2 (2004), 105--156. Google ScholarDigital Library
- J. Nishtha, S. Bharti, and R. Shweta. 2017. Systematic Literature Review on Search Based Mutation Testing. e-Informatica Software Engineering Journal 11, 1 (2017), 61--78.Google Scholar
- G. Ochoa. 2013. Search methodologies in real-world software engineering. In Proceedings of the 15th annual conference companion on Genetic and evolutionary computation. ACM, 1085--1088. Google ScholarDigital Library
- M. Papadakis, M. Kintis, J. Zhang, Y. Jia, Y. Le Traon, and M. Harman. 2017. Mutation testing advances: an analysis and survey. Advances in Computers (2017).Google Scholar
- K. Petersen and L. Vakkalanka, S. Kuzniarz. 2015. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology 64 (2015), 1--18. Google ScholarDigital Library
- O. Räihä. 2010. A survey on search-based software design. Computer Science Review 4, 4 (2010), 203--249. Google ScholarDigital Library
- R. A. Silva, S. R. S. Souza, and P. S. L. Souza. 2017. A systematic review on search based mutation testing. Information and Software Technology 81 (2017), 19--35. Google ScholarDigital Library
- F. C. M. Souza, M. Papadakis, V. H. S. Durelli, and M. E. Delamaro. 2014. Test data generation techniques for mutation testing: A systematic mapping. In Proceedings of the 17th Ibero-American Conference Software Engineering (CIBSE). 419--432.Google Scholar
- Q. Zhu, A. Panichella, and A. Zaidman. 2016-09. A systematic literature review of how mutation testing supports test activities. PeerJ Preprints 4 (2016-09).Google Scholar
Index Terms
- Search-Based Higher Order Mutation Testing: A Mapping Study
Recommendations
Predictive mutation testing
ISSTA 2016: Proceedings of the 25th International Symposium on Software Testing and AnalysisMutation testing is a powerful methodology for evaluating test suite quality. In mutation testing, a large number of mutants are generated and executed against the test suite to check the ratio of killed mutants. Therefore, mutation testing is widely ...
Search-based mutation testing to improve performance tests
GECCO '18: Proceedings of the Genetic and Evolutionary Computation Conference CompanionPerformance bugs are common and can cause a significant deterioration in the behaviour of a program, leading to costly issues. To detect them and reduce their impact, performance tests are typically applied. However, there is a lack of mechanisms to ...
Higher Order Mutation Testing
This paper introduces a new paradigm for Mutation Testing, which we call Higher Order Mutation Testing (HOM Testing). Traditional Mutation Testing considers only first order mutants, created by the injection of a single fault. Often these first order ...
Comments