Skip to main content
SpringerLink
Log in

Applying a Genetic Algorithm for Test Suite Reduction in Industry

  • Conference paper
  • First Online:
Software Quality: Higher Software Quality through Zero Waste Development (SWQD 2023)

Part of the book series: Lecture Notes in Business Information Processing ((LNBIP,volume 472))

Included in the following conference series:

  • 317 Accesses

Abstract

Time and cost of test execution increases when regression test suites grow over time. Techniques for test suite reduction have been proposed to streamline frequent test execution in continuous integration and to optimize the set of tests without sacrificing coverage and fault detection. In this paper we report on the design of a genetic algorithm to tackle the underlying optimization problem in context of an industry project from a software company developing tools for test automation. The prototypical implementation of the algorithm has been applied to the project’s test suite containing several hundred test cases. We achieved an optimal solution with a 28% reduction of test cases. The evaluation of the reduced test suite using higher-level coverage and mutation analyses showed a minimal loss of coverage. The results demonstrated that the genetic algorithm can be successfully applied in industry and the achieved results are able to satisfy the requirements of the studied project. Nevertheless, major challenges have been identified by applying the approach in industry. They are related to the reliable collection of test execution data from previous test runs and dealing with test suites containing tests exhibiting unpredictable side-effects and flakiness.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 44.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 59.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.devmate.software.

  2. 2.

    https://openclover.org/.

  3. 3.

    https://www.jacoco.org.

  4. 4.

    https://pitest.org.

  5. 5.

    https://stackoverflow.com/questions/24369631/clover-and-jacoco-give-different-code-coverage-results.

  6. 6.

    https://stackoverflow.com/questions/63492529/why-is-jacoco-coverage-report-for-branches-saying-if-a-b-c-is-actually-6.

References

  1. Yoo, S., Harman, M.: Regression testing minimization, selection and prioritization: a survey. Softw. Test. Verif. Reliab. 22(2), 67–120 (2012)

    Article  Google Scholar 

  2. Engström, E., Runeson, P.: A qualitative survey of regression testing practices. In: Ali Babar, M., Vierimaa, M., Oivo, M. (eds.) PROFES 2010. LNCS, vol. 6156, pp. 3–16. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-13792-1_3

    Chapter  Google Scholar 

  3. Gmeiner, J., Ramler, R., Haslinger, J.: Automated testing in the continuous delivery pipeline: a case study of an online company. In: 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 1–6. IEEE (2015)

    Google Scholar 

  4. Elbaum, S., Rothermel, G., Penix, J.: Techniques for improving regression testing in continuous integration development environments. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 235–245 (2014)

    Google Scholar 

  5. Shi, A., Zhao, P., Marinov, D.: Understanding and improving regression test selection in continuous integration. In: IEEE 30th International Symposium on Software Reliability Engineering (ISSRE), vol. 2019, pp. 228–238. IEEE (2019)

    Google Scholar 

  6. Khan, S.U.R., Lee, S.P., Javaid, N., Abdul, W.: A systematic review on test suite reduction: approaches, experiment’s quality evaluation, and guidelines. IEEE Access 6, 11816–11841 (2018)

    Article  Google Scholar 

  7. Engström, E., Runeson, P., Skoglund, M.: A systematic review on regression test selection techniques. Inf. Softw. Technol. 52(1), 14–30 (2010)

    Article  Google Scholar 

  8. Khatibsyarbini, M., Isa, M.A., Jawawi, D.N., Tumeng, R.: Test case prioritization approaches in regression testing: a systematic literature review. Inf. Softw. Technol. 93, 74–93 (2018)

    Article  Google Scholar 

  9. He, Z.F., Sheng, B.K., Ye, C.Q, et al.: A genetic algorithm for test-suite reduction. In: 2005 IEEE International Conference on Systems, Man and Cybernetics, vol. 1, pp. 133–139. IEEE (2005)

    Google Scholar 

  10. Ma, X., Sheng, B., Ye, C.: Test-suite reduction using genetic algorithm. In: Cao, J., Nejdl, W., Xu, M. (eds.) APPT 2005. LNCS, vol. 3756, pp. 253–262. Springer, Heidelberg (2005). https://doi.org/10.1007/11573937_28

    Chapter  Google Scholar 

  11. Nachiyappan, S., Vimaladevi, A., SelvaLakshmi, C.: An evolutionary algorithm for regression test suite reduction. In: 2010 International Conference on Communication and Computational Intelligence (INCOCCI), pp. 503–508. IEEE (2010)

    Google Scholar 

  12. Wang, S., Ali, S., Gotlieb, A.: Minimizing test suites in software product lines using weight-based genetic algorithms. In: Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, pp. 1493–1500 (2013)

    Google Scholar 

  13. Wieringa, R.J.: Design Science Methodology for Information Systems and Software Engineering. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-43839-8

    Book  Google Scholar 

  14. Buchgeher, G., Ernstbrunner, C., Ramler, R., Lusser, M.: Towards tool-support for test case selection in manual regression testing. In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops, pp. 74–79. IEEE (2013)

    Google Scholar 

  15. Ramler, R., Salomon, C., Buchgeher, G., Lusser, M.: Tool support for change-based regression testing: an industry experience report. In: Winkler, D., Biffl, S., Bergsmann, J. (eds.) SWQD 2017. LNBIP, vol. 269, pp. 133–152. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-49421-0_10

    Chapter  Google Scholar 

  16. Goldberg, D.E.: Genetic Algorithms. Pearson Education, London (2013)

    Google Scholar 

  17. Kramer, O.: Genetic Algorithms. In: Genetic Algorithm Essentials, pp. 11–19. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-52156-5

  18. Orso, A., Shi, N., Harrold, M.J.: Scaling regression testing to large software systems. ACM SIGSOFT Softw. Eng. Notes 29(6), 241–251 (2004)

    Article  Google Scholar 

  19. Haider, A.A., Nadeem, A., Akram, S.: Safe regression test suite optimization: a review. In: 2016 International Conference on Open Source Systems & Technologies (ICOSST), pp. 7–12. IEEE (2016)

    Google Scholar 

  20. Ali, N., et al.: On the search for industry-relevant regression testing research. Empir. Softw. Eng. 24(4), 2020–2055 (2019). https://doi.org/10.1007/s10664-018-9670-1

    Article  Google Scholar 

  21. Duan, K., Fong, S., Siu, S.W., Song, W., Guan, S.S.U.: Adaptive incremental genetic algorithm for task scheduling in cloud environments. Symmetry 10(5), 168 (2018)

    Article  Google Scholar 

  22. Alemerien, K., Magel, K.: Examining the effectiveness of testing coverage tools: an empirical study. Int. J. Softw. Eng. Appl. 8(5), 139–162 (2014)

    Google Scholar 

  23. Horváth, F., Gergely, T., Beszédes, Á., Tengeri, D., Balogh, G., Gyimóthy, T.: Code coverage differences of java bytecode and source code instrumentation tools. Softw. Qual. J. 27, 79–123 (2019)

    Article  Google Scholar 

Download references

Acknowledgements

This work was partially supported by the Austrian Research Promotion Agency (FFG) in the frame of the COMET competence center SCCH [892418].

Author information

Authors and Affiliations

  1. Johannes Kepler University Linz, Linz, Austria

    Philipp Stadler & Reinhold Plösch

  2. Software Competence Center Hagenberg GmbH (SCCH), Hagenberg, Austria

    Rudolf Ramler

Authors
  1. Philipp Stadler
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Reinhold Plösch
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rudolf Ramler
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rudolf Ramler .

Editor information

Editors and Affiliations

  1. Blekinge Institute of Technology, Karlskrona, Sweden

    Daniel Mendez

  2. Austrian Center for Digital Production (CDP), SBA Research gGmbH, Vienna, Austria

    Dietmar Winkler

  3. fortiss GmbH, Munich, Germany

    Johannes Kross

  4. TU Wien, Vienna, Austria

    Stefan Biffl

  5. Software Quality Lab GmbH, Linz, Austria

    Johannes Bergsmann

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Stadler, P., Plösch, R., Ramler, R. (2023). Applying a Genetic Algorithm for Test Suite Reduction in Industry. In: Mendez, D., Winkler, D., Kross, J., Biffl, S., Bergsmann, J. (eds) Software Quality: Higher Software Quality through Zero Waste Development. SWQD 2023. Lecture Notes in Business Information Processing, vol 472. Springer, Cham. https://doi.org/10.1007/978-3-031-31488-9_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-31488-9_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-31487-2

  • Online ISBN: 978-3-031-31488-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation