Man Zhang
Skip Abstract Section
Abstract
REST web services are widely popular in industry, and search techniques have been successfully used to automatically generate system-level test cases for those systems. In this article, we propose a novel mutation operator which is designed specifically for test generation at system-level, with a particular focus on REST APIs. In REST API testing, and often in system testing in general, an individual can have a long and complex chromosome. Furthermore, there are two specific issues: (1) fitness evaluation in system testing is highly costly compared with the number of objectives (e.g., testing targets) to optimize for; and (2) a large part of the genotype might have no impact on the phenotype of the individuals (e.g., input data that has no impact on the execution flow in the tested program). Due to these issues, it might be not suitable to apply a typical low mutation rate like 1/n (where n is the number of genes in an individual), which would lead to mutating only one gene on average. Therefore, in this article, we propose an adaptive weight-based hypermutation, which is aware of the different characteristics of the mutated genes. We developed adaptive strategies that enable the selection and mutation of genes adaptively based on their fitness impact and mutation history throughout the search. To assess our novel proposed mutation operator, we implemented it in the EvoMaster tool, integrated in the MIO algorithm, and further conducted an empirical study with three artificial REST APIs and four real-world REST APIs. Results show that our novel mutation operator demonstrates noticeable improvements over the default MIO. It provides a significant improvement in performance for six out of the seven case studies, where the relative improvement is up to +12.09% for target coverage, +12.69% for line coverage, and +32.51% for branch coverage.
- S. Ali, L. C. Briand, H. Hemmati, and R. K. Panesar-Walawege. 2010. A systematic review of the application and empirical investigation of search-based test-case generation. IEEE Transactions on Software Engineering 36, 6 (2010), 742–762. Google Scholar
Digital Library
- Mohammad Alshraideh and Leonardo Bottaci. 2006. Search-based software test data generation for string data using program-specific search operators. Software Testing, Verification, and Reliability 16, 3 (2006), 175–203. DOI:https://doi.org/10.1002/stvr.v16:3 Google ScholarDigital Library
- Denis Antipov and Benjamin Doerr. 2020. Runtime analysis of a heavy-tailed (1+(λ, λ)) genetic algorithm on jump functions. In International Conference on Parallel Problem Solving from Nature. Thomas Bäck, Mike Preuss, André Deutz, Hao Wang, Carola Doerr, Michael Emmerich, and Heike Trautmann (Eds.) Springer, 545–559.Google ScholarDigital Library
- Andrea Arcuri. 2018. EvoMaster: Evolutionary multi-context automated system test generation. In IEEE 11th International Conference on Software Testing, Verification and Validation. IEEE. DOI:10.1109/ICST.2018.00046Google ScholarCross Ref
- Andrea Arcuri. 2017. An experience report on applying software testing academic results in industry: We need usable automated test generation. Empirical Software Engineering 23, 2 (2017), 1–23. Google ScholarDigital Library
- Andrea Arcuri. 2018. Test suite generation with the Many Independent Objective (MIO) algorithm. Information and Software Technology 104 (2018), 195–206. DOI:https://doi.org/10.1016/j.infsof.2018.05.003Google ScholarCross Ref
- Andrea Arcuri. 2019. RESTful API automated test case generation with EvoMaster. ACM Transactions on Software Engineering and Methodology 28, 1 (2019), 1–37. Google ScholarDigital Library
- Andrea Arcuri. 2021. Automated blackbox and whitebox testing of RESTful APIs with EvoMaster. IEEE Software 38, 3 (2021), 72–78.Google ScholarCross Ref
- A. Arcuri and L. Briand. 2011. Adaptive random testing: An illusion of effectiveness? In Proceedings of the 11th International Symposium on Software Testing and Analysis. ACM, New York, NY, 265–275. Google ScholarDigital Library
- A. Arcuri and L. Briand. 2014. A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Software Testing, Verification and Reliability 24, 3 (2014), 219–250. Google ScholarDigital Library
- Andrea Arcuri and Gordon Fraser. 2013. Parameter tuning or default values? An empirical investigation in search-based software engineering. Empirical Software Engineering 18, 3 (2013), 594–623.Google ScholarCross Ref
- Andrea Arcuri and Juan Pablo Galeotti. 2020. Handling SQL databases in automated system test generation. ACM Transactions on Software Engineering and Methodology 29, 4 (2020), 1–31. Google ScholarDigital Library
- Andrea Arcuri and Juan Pablo Galeotti. 2020. Testability transformations for existing APIs. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification. IEEE, 153–163. DOI:10.1109/ICST46399.2020.00025Google ScholarCross Ref
- Andrea Arcuri, Juan Pablo Galeotti, Bogdan Marculescu, and Man Zhang. 2020. EvoMaster: A Search-Based System Test Generation Tool. Zenodo. DOI:https://doi.org/10.5281/zenodo.4300745Google Scholar
- Andrea Arcuri, Juan Pablo Galeotti, Bogdan Marculescu, and Man Zhang. 2021. EvoMaster: A search-based system test generation tool. Journal of Open Source Software 6, 57 (2021), 2153.Google ScholarCross Ref
- Vaggelis Atlidakis, Roxana Geambasu, Patrice Godefroid, Marina Polishchuk, and Baishakhi Ray. 2020. Pythia: Grammar-based fuzzing of REST APIs with coverage-guided feedback and learning-based mutations. arXiv:2005.11498. Retrieved from https://arxiv.org/abs/2005.11498.Google Scholar
- Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2019. RESTler: Stateful REST API Fuzzing. In Proceedings of the 41st International Conference on Software Engineering. IEEE Press, 748–758. DOI:https://doi.org/10.1109/ICSE.2019.00083 Google ScholarDigital Library
- José Campos, Yan Ge, Nasser Albunian, Gordon Fraser, Marcelo Eler, and Andrea Arcuri. 2018. An empirical evaluation of evolutionary algorithms for unit test suite generation. Information and Software Technology 104 (2018), 207–235. DOI:https://doi.org/10.1016/j.infsof.2018.08.010Google ScholarCross Ref
- Leandro Nunes Castro, Leandro Nunes De Castro, and Jonathan Timmis. 2002. Artificial Immune Systems: A new Computational Intelligence Approach. Springer Science & Business Media. Google ScholarDigital Library
- Jun Chen and Mahdi Mahfouf. 2006. A population adaptive based immune algorithm for solving multi-objective optimization problems. In International Conference on Artificial Immune Systems. H. Bersini and J. Carneiro (Eds.), Springer, 280–293. Google ScholarDigital Library
- Chien-Wei Chu, Min-Der Lin, Gee-Fon Liu, and Yung-Hsing Sung. 2008. Application of immune algorithms on solving minimum-cost problem of water distribution network. Mathematical and Computer Modelling 48, 11-12 (2008), 1888–1900. Google ScholarDigital Library
- Helen G. Cobb. 1990. An Investigation into the Use of Hypermutation as an Adaptive Operator in Genetic Algorithms Having Continuous, Time-Dependent Nonstationary Environments. Naval Research Lab Washington DC.Google Scholar
- Dogan Corus, Pietro S. Oliveto, and Donya Yazdani. 2020. When hypermutations and ageing enable artificial immune systems to outperform evolutionary algorithms. Theoretical Computer Science 832 (2020), 166–185. DOI:https://doi.org/10.1016/j.tcs.2019.03.002Google ScholarCross Ref
- Vincenzo Cutello, Giuseppe Nicosia, and Mario Pavone. 2004. Exploring the capability of immune algorithms: A characterization of hypermutation operators. In Artificial Immune Systems. Giuseppe Nicosia, Vincenzo Cutello, Peter J. Bentley, and Jon Timmis (Eds.). Springer, Berlin, 263–276.Google Scholar
- Leandro N. De Castro and Fernando J. Von Zuben. 2002. Learning and optimization using the clonal selection principle. IEEE Transactions on Evolutionary Computation 6, 3 (2002), 239–251. Google ScholarDigital Library
- Benjamin Doerr and Carola Doerr. 2020. Theory of parameter control for discrete black-box optimization: Provable performance gains through dynamic parameter choices. Theory of Evolutionary Computation (2020), 271–321. DOI:10.1007/978-3-030-29414-4_6Google Scholar
- Benjamin Doerr, Carola Doerr, and Johannes Lengler. 2019. Self-adjusting mutation rates with provably optimal success rules. In Proceedings of the Genetic and Evolutionary Computation Conference. 1479–1487. Google ScholarDigital Library
- Carola Doerr and Markus Wagner. 2018. Sensitivity of parameter control mechanisms with respect to their initialization. In International Conference on Parallel Problem Solving from Nature. A. Auger, C. Fonseca, N. Lourenco, P. Machado, L. Paquete, and D. Whitley (Eds.), Springer, 360–372.Google ScholarCross Ref
- Carola Doerr and Markus Wagner. 2018. Simple on-the-fly parameter selection mechanisms for two classical discrete black-box optimization benchmark problems. In Proceedings of the Genetic and Evolutionary Computation Conference. 943–950. Google ScholarDigital Library
- S. Droste, T. Jansen, and I. Wegener. 1998. On the optimization of unimodal functions with the (1 + 1) evolutionary algorithm. In Proceedings of the International Conference on Parallel Problem Solving from Nature. 13–22. Google ScholarDigital Library
- Hamza Ed-douibi, Javier Luis Cánovas Izquierdo, and Jordi Cabot. 2018. Automatic generation of test cases for REST APIs: A specification-based approach. In 2018 IEEE 22nd International Enterprise Distributed Object Computing Conference. 181–190. DOI:10.1109/EDOC.2018.00031Google Scholar
- Á. E. Eiben, R. Hinterding, and Z. Michalewicz. 1999. Parameter control in evolutionary algorithms. IEEE Transactions on Evolutionary Computation 3, 2 (1999), 124–141. Google ScholarDigital Library
- Roy Thomas Fielding. 2000. Architectural Styles and the Design of Network-Based Software Architectures. Ph.D. Dissertation. University of California, Irvine. UMI Order Number: AAI 9980887.Google ScholarDigital Library
- Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: Automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference onFoundations of Software Engineering. 416–419. Google ScholarDigital Library
- Gordon Fraser and Andrea Arcuri. 2013. EvoSuite at the SBST 2013 Tool Competition. In 2013 IEEE 6th International Workshop on Search-Based Software Testing. 406–409. Google ScholarDigital Library
- Gordon Fraser and Andrea Arcuri. 2013. Whole test suite generation. IEEE Transactions on Software Engineering 39, 2 (2013), 276–291. Google ScholarDigital Library
- Tobias Friedrich, Andreas Göbel, Francesco Quinzan, and Markus Wagner. 2018. Heavy-tailed mutation operators in single-objective combinatorial optimization. In International Conference on Parallel Problem Solving from Nature. A. Auger, C. Fonseca, N. Lourenco, P. Machado, L. Paquete, and D. Whitley (Eds.), Springer, 134–145.Google ScholarCross Ref
- Tobias Friedrich, Francesco Quinzan, and Markus Wagner. 2018. Escaping large deceptive basins of attraction with heavy-tailed mutation operators. In Proceedings of the Genetic and Evolutionary Computation Conference. 293–300. Google ScholarDigital Library
- Patrice Godefroid, Bo-Yuan Huang, and Marina Polishchuk. 2020. Intelligent REST API Data Fuzzing. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, New York, NY, 725–736. DOI:https://doi.org/10.1145/3368089.3409719 Google ScholarDigital Library
- D. E. Goldberg. 1989. Genetic Algorithms in Search and Optimization. Addison-wesley. Google ScholarDigital Library
- Mark Harman, S. Afshin Mansouri, and Yuanyuan Zhang. 2012. Search-based software engineering: Trends, techniques and applications. ACM Computing Surveys 45, 1 (2012), 11. Google ScholarDigital Library
- Zhengxin Huang and Yuren Zhou. 2020. Runtime analysis of somatic contiguous hypermutation operators in MOEA/D Framework. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 2359–2366.Google ScholarCross Ref
- Frank Hutter, Youssef Hamadi, Holger H. Hoos, and Kevin Leyton-Brown. 2006. Performance prediction and automated tuning of randomized and parametric algorithms. In International Conference on Principles and Practice of Constraint Programming. F. Benhamou (Eds.), Springer, 213–228. Google ScholarDigital Library
- Frank Hutter, Lin Xu, Holger H. Hoos, and Kevin Leyton-Brown. 2014. Algorithm runtime prediction: Methods & evaluation. Artificial Intelligence 206 (2014), 79–111. DOI:https://doi.org/10.1016/j.artint.2013.10.003 Google ScholarDigital Library
- T. Jansen and C. Zarges. 2014. Reevaluating immune-inspired hypermutations using the fixed budget perspective. IEEE Transactions on Evolutionary Computation 18, 5 (2014), 674–688. DOI:https://doi.org/10.1109/TEVC.2014.2349160Google ScholarCross Ref
- Giorgos Karafotias, Mark Hoogendoorn, and Ágoston E. Eiben. 2014. Parameter control in evolutionary algorithms: Trends and challenges. IEEE Transactions on Evolutionary Computation 19, 2 (2014), 167–187.Google ScholarDigital Library
- Stefan Karlsson, Adnan Causevic, and Daniel Sundmark. 2020. QuickREST: Property-based test generation of OpenAPI described RESTful APIs. In IEEE 13th International Conference on Software Testing, Verification and Validation. IEEE.Google ScholarCross Ref
- S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi. 1983. Optimization by simulated annealing. Science 220, 4598 (1983), 671–680.Google Scholar
- Anton Kotelyanskii and Gregory M. Kapfhammer. 2014. Parameter tuning for search-based test-data generation revisited: Support for previous results. In 2014 14th International Conference on Quality Software. IEEE, 79–84. DOI:10.1109/QSIC.2014.43 Google ScholarDigital Library
- Johannes Lengler. 2019. A general dichotomy of evolutionary algorithms on monotone functions. IEEE Transactions on Evolutionary Computation 24, 6 (2019), 995–1009.Google ScholarDigital Library
- Kevin Leyton-Brown, Eugene Nudelman, and Yoav Shoham. 2002. Learning the empirical hardness of optimization problems: The case of combinatorial auctions. In International Conference on Principles and Practice of Constraint Programming. P. Van Hentenryck (Eds.), Springer, 556–572. Google ScholarDigital Library
- Q. Lin, J. Chen, Z. Zhan, W. Chen, C. A. C. Coello, Y. Yin, C. Lin, and J. Zhang. 2016. A hybrid evolutionary immune algorithm for multiobjective optimization problems. IEEE Transactions on Evolutionary Computation 20, 5 (2016), 711–729. DOI:https://doi.org/10.1109/TEVC.2015.2512930Google Scholar
- Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: Multi-objective automated testing for android applications. In Proceedings of the 25th International Symposium on Software Testing and Analysis. ACM, 94–105. Google ScholarDigital Library
- Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2020. RESTest: Black-box constraint-based testing of RESTful Web APIs. In International Conference on Service-Oriented Computing. E. Kafeza, B. Benatallah, F. Martinelli, H. Hacid, A. Bouguettaya, and H. Motahari (Eds.), Springer.Google ScholarDigital Library
- P. McMinn. 2004. Search-based software test data generation: A survey. Software Testing, Verification and Reliability 14, 2 (2004), 105–156. Google ScholarDigital Library
- Phil McMinn, Mark Harman, Kiran Lakhotia, Youssef Hassoun, and Joachim Wegener. 2011. Input domain reduction through irrelevant variable removal and its effect on local, global, and hybrid search-based structural test data generation. IEEE Transactions on Software Engineering 38, 2 (2011), 453–477. Google ScholarDigital Library
- Phil McMinn and Gregory M. Kapfhammer. 2016. AVMf: An open-source framework and implementation of the alternating variable method. In International Symposium on Search Based Software Engineering. F. Sarro and K. Deb (Eds.), Springer, 259–266.Google Scholar
- Vladimir Mironovich and Maxim Buzdalov. 2017. Evaluation of heavy-tailed mutation operator on maximum flow test generation problem. In Proceedings of the Genetic and Evolutionary Computation Conference Companion. 1423–1426. Google ScholarDigital Library
- Sam Newman. 2015. Building Microservices. “O’Reilly Media, Inc.”. Google ScholarDigital Library
- Annibale Panichella, Fitsum Kifetew, and Paolo Tonella. 2018. Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. IEEE Transactions on Software Engineering 44, 2 (2018), 122–158.Google ScholarCross Ref
- José Miguel Rojas, Mattia Vivanti, Andrea Arcuri, and Gordon Fraser. 2017. A detailed investigation of the effectiveness of whole test suite generation. Empirical Software Engineering 22, 2 (2017), 852–893. Google ScholarDigital Library
- Abdel Salam Sayyad, Katerina Goseva-Popstojanova, Tim Menzies, and Hany Ammar. 2013. On parameter tuning in search based software engineering: A replicated empirical study. In 2013 3rd International Workshop on Replication in Empirical Software Engineering Research. IEEE, 84–90. Google ScholarDigital Library
- Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2020. RESTTESTGEN: Automated black-box testing of RESTful APIs. In IEEE International Conference on Software Testing, Verification and Validation. IEEE. DOI:10.1109/ICST46399.2020.00024Google ScholarCross Ref
- Louis F. Williams. 1976. A modification to the half-interval search (binary search) method. In Proceedings of the 14th Annual Southeast Regional Conference.ACM, New York, NY, 95–101. DOI:https://doi.org/10.1145/503561.503582 Google ScholarDigital Library
- D. H. Wolpert and W. G. Macready. 1997. No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation 1, 1 (1997), 67–82. Google ScholarDigital Library
- Furong Ye, Carola Doerr, and Thomas Bäck. 2019. Interpolating local and global search by controlling the variance of standard bit mutation. In 2019 IEEE Congress on Evolutionary Computation. IEEE, 2292–2299.Google ScholarDigital Library
- Shayan Zamani and Hadi Hemmati. 2019. Revisiting hyper-parameter tuning for search-based test data generation. In International Symposium on Search Based Software Engineering. S. Nejati and G. Gay (Eds.), Springer, 137–152.Google ScholarDigital Library
- Man Zhang, Bogdan Marculescu, and Andrea Arcuri. 2019. Resource-based test case generation for RESTful web services. In Proceedings of the Genetic and Evolutionary Computation Conference. 1426–1434. Google ScholarDigital Library
Index Terms
- Adaptive Hypermutation for Search-Based System Test Generation: A Study on REST APIs with EvoMaster
Recommendations
On the Faults Found in REST APIs by Automated Test Generation
RESTful web services are often used for building a wide variety of enterprise applications. The diversity and increased number of applications using RESTful APIs means that increasing amounts of resources are spent developing and testing these systems. ...
Enhancing Search-based Testing with Testability Transformations for Existing APIs
Search-based software testing (SBST) has been shown to be an effective technique to generate test cases automatically. Its effectiveness strongly depends on the guidance of the fitness function. Unfortunately, a common issue in SBST is the so-called flag ...
Measuring and Maintaining Population Diversity in Search-Based Unit Test Generation
Search-Based Software EngineeringAbstractGenetic algorithms (GAs) have been demonstrated to be effective at generating unit tests. However, GAs often suffer from a loss of population diversity, which causes the search to prematurely converge, thus negatively affecting the resulting code ...
Comments