Xavier Devroey
Pierre-Yves Schobbens
ABSTRACT
Mutation testing is an effective technique for either improving or generating fault-finding test suites. It creates defective or incorrect program artifacts of the program under test and evaluates the ability of test suites to reveal them. Despite being effective, mutation is costly since it requires assessing the test cases with a large number of defective artifacts. Even worse, some of these artifacts are behaviourally ``equivalent'' to the original one and hence, they unnecessarily increase the testing effort. We adopt a variability perspective on mutation analysis. We model a defective artifact as a transition system with a specific feature selected and consider it as a member of a mutant family. The mutant family is encoded as a Featured Transition System, a compact formalism initially dedicated to model-checking of software product lines. We show how to evaluate a test suite against the set of all candidate defects by using mutant families. We can evaluate all the considered defects at the same time and isolate some equivalent mutants. We can also assist the test generation process and efficiently consider higher-order mutants.
- M. Acher, A. Cleve, G. Perrouin, P. Heymans, C. Vanbeneden, P. Collet, and P. Lahire. On extracting feature models from product descriptions. In VaMoS, pages 45–54, New York, NY, USA, 2012. ACM. Google Scholar
Digital Library
- B. K. Aichernig, H. Brandl, E. Jöbstl, W. Krenn, R. Schlick, and S. Tiran. Killing strategies for model-based mutation testing. STVR, 2014.Google Scholar
- B. K. Aichernig, M. Weiglhofer, and F. Wotawa. Improving fault-based conformance testing. Electr. Notes Theor. Comput. Sci., 220(1):63–77, 2008. Google ScholarDigital Library
- P. Ammann, M. E. Delamaro, and J. Offutt. Establishing theoretical minimal sets of mutants. In ICST, pages 21–30. IEEE, 2014. Google ScholarDigital Library
- P. E. Ammann, P. E. Black, and W. Majurski. Using model checking to generate tests from specifications. In FEM, pages 46–54. IEEE, 1998. Google ScholarDigital Library
- N. Bombieri, F. Fummi, and G. Pravadelli. A mutation model for the SystemC TLM 2.0 communication interfaces. In DATE, pages 396–401. ACM, 2008. Google ScholarDigital Library
- Q. Boucher, G. Perrouin, and P. Heymans. Deriving configuration interfaces from feature models: A vision paper. In VaMoS, pages 37–44, New York, NY, USA, 2012. ACM. Google ScholarDigital Library
- A. Classen, M. Cordy, P.-Y. Schobbens, P. Heymans, A. Legay, and J.-F. Raskin. Featured transition systems: Foundations for verifying variability-intensive systems and their application to ltl model checking. IEEE TSE, 39(8):1069–1089, 2013. Google ScholarDigital Library
- M. Cohen, M. Dwyer, and J. Shi. Interaction testing of highly-configurable systems in the presence of constraints. In ISSTA, pages 129–139, 2007. Google ScholarDigital Library
- M. Cordy, A. Classen, P. Heymans, P.-Y. Schobbens, and A. Legay. Provelines: A product-line of verifiers for software product lines. In SPLC ’13 Workshops, pages 141–146. ACM, 2013. Google ScholarDigital Library
- M. Cordy, A. Classen, G. Perrouin, P.-Y. Schobbens, P. Heymans, and A. Legay. Simulation-based abstractions for software product-line model checking. In M. Glinz, G. C. Murphy, and M. Pezzè, editors, ICSE, pages 672–682. IEEE, 2012. Google ScholarDigital Library
- R. A. DeMillo and A. J. Offutt. Constraint-based automatic test data generation. IEEE TSE, 17(9):900–910, 1991. Google ScholarDigital Library
- X. Devroey, G. Perrouin, M. Cordy, P.-Y. Schobbens, A. Legay, and P. Heymans. Towards statistical prioritization for software product lines testing. In VaMoS, pages 10:1–10:7. ACM, 2014. Google ScholarDigital Library
- X. Devroey, G. Perrouin, A. Legay, M. Cordy, P.-y. Schobbens, and P. Heymans. Coverage Criteria for Behavioural Testing of Software Product Lines. In ISoLA. Springer, 2014.Google ScholarDigital Library
- S. C. P. F. Fabbri, J. C. Maldonado, P. C. Masiero, and M. E. Delamaro. Proteum/fsm: A tool to support finite state machine validation based on mutation testing. In SCCC, pages 96–104. IEEE Computer Society, 1999. Google ScholarDigital Library
- S. C. P. F. Fabbri, J. C. Maldonado, T. Sugeta, and P. C. Masiero. Mutation testing applied to validate specifications based on Statecharts. In SRE, pages 210–219. IEEE, 1999. Google ScholarDigital Library
- C. Henard, M. Papadakis, G. Perrouin, J. Klein, and Y. Le Traon. Towards automated testing and fixing of re-engineered feature models. In ICSE, pages 1245–1248. IEEE, May 2013. Google ScholarDigital Library
- C. Henard, M. Papadakis, G. Perrouin, J. Klein, and Y. L. Traon. Multi-objective test generation for software product lines. In SPLC, pages 62–71. ACM, 2013. Google ScholarDigital Library
- Y. Jia and M. Harman. MILU: A Customizable, Runtime-Optimized Higher-Order Mutation Testing Tool for the Full C Language. In TAIC-PART, pages 94–98, Washington, DC, USA, 2008. IEEE Computer Society. Google ScholarDigital Library
- Y. Jia and M. Harman. An analysis and survey of the development of mutation testing. IEEE TSE, 37(5):649–678, 2011. Google ScholarDigital Library
- M. F. Johansen, Ø. Haugen, F. Fleurey, A. G. Eldegard, and T. Syversen. Generating better partial covering arrays by modeling weights on sub-product lines. In MoDELS, pages 269–284. Springer, 2012. Google ScholarDigital Library
- K. C. Kang, S. G. Cohen, J. A. Hess, W. E. Novak, and A. Spencer Peterson. Feature-Oriented domain analysis (FODA) feasibility study. Technical report, Soft. Eng. Inst., Carnegie Mellon Univ., 1990.Google Scholar
- C. Kästner, A. von Rhein, S. Erdweg, J. Pusch, S. Apel, T. Rendel, and K. Ostermann. Toward variability-aware testing. In FOSD, pages 1–8. ACM, 2012. Google ScholarDigital Library
- C. H. P. Kim, S. Khurshid, and D. S. Batory. Shared execution for efficiently testing product lines. In ISSRE, pages 221–230. IEEE, 2012. Google ScholarDigital Library
- D. Lorenzoli, L. Mariani, and M. Pezzè. Automatic generation of software behavioral models. In ICSE, pages 501–510, New York, NY, USA, 2008. ACM. Google ScholarDigital Library
- L. Madeyski, W. Orzeszyna, R. Torkar, and M. Jozala. Overcoming the equivalent mutant problem: A systematic literature review and a comparative experiment of second order mutation. IEEE TSE, 40(1):23–42, Jan 2014.Google ScholarDigital Library
- H. V. Nguyen, C. Kästner, and T. N. Nguyen. Exploring variability-aware execution for testing plugin-based web applications. In ICSE. IEEE, 2014. Google ScholarDigital Library
- J. Offutt. A mutation carol: Past, present and future. Information & Software Technology, 53(10):1098––1107, 2011.Google ScholarCross Ref
- G. Perrouin, S. Oster, S. Sen, J. Klein, B. Baudry, and Y. L. Traon. Pairwise testing for software product lines: comparison of two approaches. Soft. Qual. Journal, 20(3-4):605–643, 2012. Google ScholarDigital Library
- R. H. Untch, A. J. Offutt, and M. J. Harrold. Mutation analysis using mutant schemata. In ISSTA, pages 139–148. ACM, 1993. Google ScholarDigital Library
Index Terms
- A variability perspective of mutation analysis
Recommendations
Predictive mutation testing
ISSTA 2016: Proceedings of the 25th International Symposium on Software Testing and AnalysisMutation testing is a powerful methodology for evaluating test suite quality. In mutation testing, a large number of mutants are generated and executed against the test suite to check the ratio of killed mutants. Therefore, mutation testing is widely ...
Faster mutation testing inspired by test prioritization and reduction
ISSTA 2013: Proceedings of the 2013 International Symposium on Software Testing and AnalysisMutation testing is a well-known but costly approach for determining test adequacy. The central idea behind the approach is to generate mutants, which are small syntactic transformations of the program under test, and then to measure for a given test ...
Mitigating the effects of flaky tests on mutation testing
ISSTA 2019: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and AnalysisMutation testing is widely used in research as a metric for evaluating the quality of test suites. Mutation testing runs the test suite on generated mutants (variants of the code under test), where a test suite kills a mutant if any of the tests fail ...
Comments