ABSTRACT
In many domains, engineers build simulation models (e.g., Simulink) before developing code to simulate the behavior of complex systems (e.g., Cyber-Physical Systems). Those models are commonly heavy to simulate which makes it difficult to execute the entire test suite. Furthermore, it is often difficult to measure white-box coverage of test cases when employing such models. In addition, the historical data related to failures might not be available. This paper proposes a cost-effective approach for test case selection that relies on black-box data related to inputs and outputs of the system. The approach defines in total five effectiveness measures and one cost measure followed by deriving in total 15 objective combinations and integrating them within Non-Dominated Sorting Genetic Algorithm-II (NSGA-II). We empirically evaluated our approach with all these 15 combinations using four case studies by employing mutation testing to assess the fault revealing capability. The results demonstrated that our approach managed to improve Random Search by 26% on average in terms of the Hypervolume quality indicator.
Supplemental Material
Available for Download
Supplemental files.
- 2018. FMI Standard. (January 2018). http://fmi-standard.org/Google Scholar
- Andrea Arcuri and Lionel Briand. 2011. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In Software Engineering (ICSE), 2011 33rd International Conference on. IEEE, 1--10. Google ScholarDigital Library
- Andrea Arcuri, Muhammad Zohaib Iqbal, and Lionel Briand. 2010. Black-box System Testing of Real-time Embedded Systems Using Random and Search-based Testing. In Proceedings of the 22Nd IFIP WG 6.1 International Conference on Testing Software and Systems (ICTSS'10). Springer-Verlag, Berlin, Heidelberg, 95--110. Google ScholarDigital Library
- Aitor Arrieta, Shuai Wang, Goiuria Sagardui, and Leire Etxeberria. 2016. Search-based Test Case Selection of Cyber-physical System Product Lines for Simulation-based Validation. In Proceedings of the 20th International Systems and Software Product Line Conference. 297--306. Google ScholarDigital Library
- Raja Ben Abdessalem, Shiva Nejati, Lionel C. Briand, and Thomas Stifter. 2016. Testing Advanced Driver Assistance Systems Using Multi-objective Search and Neural Networks. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. 63--74. Google ScholarDigital Library
- Raja Ben Abdessalem, Shiva Nejati, Lionel C. Briand, and Thomas Stifter. 2018. Testing Vision-Based Control Systems Using Learnable Evolutionary Algorithms. In Proceedings of the 40th International Conference on Software Engineering (ICSE '18). 12. Google ScholarDigital Library
- Lionel Briand, Shiva Nejati, Mehrdad Sabetzadeh, and Domenico Bianculli. 2016. Testing the Untestable: Model Testing of Complex Software-intensive Systems. In Proceedings of the 38th International Conference on Software Engineering Companion (ICSE '16). ACM, 789--792. Google ScholarDigital Library
- Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE transactions on evolutionary computation 6, 2 (2002), 182--197. Google ScholarDigital Library
- Emelie Engström, Per Runeson, and Mats Skoglund. 2010. A systematic review on regression test selection techniques. Information and Software Technology 52, 1 (2010), 14--30. Google ScholarDigital Library
- Robert Feldt, Simon M. Poulding, David Clark, and Shin Yoo. 2016. Test Set Diameter: Quantifying the Diversity of Sets of Test Cases. In 2016 IEEE International Conference on Software Testing, Verification and Validation, ICST 2016, Chicago, IL, USA, April 11--15, 2016. 223--233.Google Scholar
- Le Thi My Hanh, Nguyen Thanh Binh, and Khuat Thanh Tung. 2016. A Novel Fitness function of metaheuristic algorithms for test data generation for simulink models based on mutation analysis. Journal of Systems and Software 120, C (2016), 17--30. Google ScholarDigital Library
- Hadi Hemmati, Andrea Arcuri, and Lionel Briand. 2013. Achieving scalable model-based testing through test case diversity. ACM Transactions on Software Engineering and Methodology 22, 1 (2013), 6:1--6:42. Google ScholarDigital Library
- Christopher Henard, Mike Papadakis, Mark Harman, Yue Jia, and Yves Le Traon. 2016. Comparing white-box and black-box test prioritization. In Proceedings of the 38th International Conference on Software Engineering. ACM, 523--534. Google ScholarDigital Library
- René Just, Darioush Jalali, Laura Inozemtseva, Michael D Ernst, Reid Holmes, and Gordon Fraser. 2014. Are mutants a valid substitute for real faults in software testing?. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 654--665. Google ScholarDigital Library
- Remo Lachmann, Michael Felderer, Manuel Nieke, Sandra Schulze, Christoph Seidl, and Ina Schaefer. 2017. Multi-objective black-box test case selection for system testing. In Proceedings of the Genetic and Evolutionary Computation Conference. ACM, 1311--1318. Google ScholarDigital Library
- Reza Matinnejad, Shiva Nejati, and Lionel C. Briand. 2017. Automated Testing of Hybrid Simulink/Stateflow Controllers: Industrial Case Studies. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2017). ACM, New York, NY, USA, 938--943. Google ScholarDigital Library
- Reza Matinnejad, Shiva Nejati, Lionel C Briand, and Thomas Bruckmann. 2015. Effective test suites for mixed discrete-continuous stateflow controllers. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. ACM, 84--95. Google ScholarDigital Library
- Reza Matinnejad, Shiva Nejati, Lionel C. Briand, and Thomas Bruckmann. 2016. Automated Test Suite Generation for Time-continuous Simulink Models. In Proceedings of the 38th International Conference on Software Engineering (ICSE '16). ACM, New York, NY, USA, 595--606. Google ScholarDigital Library
- Annibale Panichella, Fitsum Kifetew, and Paolo Tonella. 2017. Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. IEEE Transactions on Software Engineering (2017).Google Scholar
- Mike Papadakis, Yue Jia, Mark Harman, and Yves Le Traon. 2015. Trivial compiler equivalence: A large scale empirical study of a simple, fast and effective equivalent mutant detection technique. In Proceedings of the 37th International Conference on Software Engineering-Volume 1. IEEE Press, 936--946. Google ScholarDigital Library
- Dipesh Pradhan, Shuai Wang, Shaukat Ali, and Tao Yue. 2016. Search-Based Cost-Effective Test Case Selection Within a Time Budget: An Empirical Study. In Proceedings of the Genetic and Evolutionary Computation Conference 2016 (GECCO '16). ACM, New York, NY, USA, 1085--1092. Google ScholarDigital Library
- Goiuria Sagardui, Leire Etxeberria, Joseba A Agirre, Aitor Arrieta, Carlos Fernando Nicolas, and Jose Maria Martin. 2017. A Configurable Validation Environment for Refactored Embedded Software: An Application to the Vertical Transport Domain. In 2017 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW). IEEE, 16--19.Google ScholarCross Ref
- T. Strathmann and J. Oehlerking. 2015. Verifying Properties of an Electro-Mechanical Braking System. In In 1st and 2nd International Workshop on Applied veRification for Continuous and Hybrid Systems. 49--56.Google Scholar
- Shuai Wang, Shaukat Ali, and Arnaud Gotlieb. 2013. Minimizing test suites in software product lines using weight-based genetic algorithms, In Proceedings of the 2013 Genetic and Evolutionary Computation Conference. GECCO 2013 - Proceedings of the 2013 Genetic and Evolutionary Computation Conference, 1493 -- 1500. Google ScholarDigital Library
- Shuai Wang, Shaukat Ali, Tao Yue, Yan Li, and Marius Liaaen. 2016. A Practical Guide to Select Quality Indicators for Assessing Pareto-based Search Algorithms in Search-based Software Engineering. In Proceedings of the 38th International Conference on Software Engineering (ICSE '16). 631--642. Google ScholarDigital Library
- Shin Yoo and Mark Harman. 2007. Pareto efficient multi-objective test case selection. In Proceedings of the 2007 international symposium on Software testing and analysis. ACM, 140--150. Google ScholarDigital Library
- Yuan Zhan and John A Clark. 2008. A search-based framework for automatic testing of MATLAB/Simulink models. Journal of Systems and Software 81, 2 (2008), 262--285. Google ScholarDigital Library
Index Terms
- Multi-objective black-box test case selection for cost-effectively testing simulation models
Recommendations
Comparing white-box and black-box test prioritization
ICSE '16: Proceedings of the 38th International Conference on Software EngineeringAlthough white-box regression test prioritization has been well-studied, the more recently introduced black-box prioritization approaches have neither been compared against each other nor against more well-established white-box techniques. We present a ...
Multi-objective black-box test case selection for system testing
GECCO '17: Proceedings of the Genetic and Evolutionary Computation ConferenceTesting is a fundamental task to ensure software quality. Regression testing aims to ensure that changes to software do not introduce new failures. As resources are often limited and testing comprises a vast amount of test cases, different regression ...
White-Box and Black-Box Test Quality Metrics for Configurable Simulation Models
SPLC '19: Proceedings of the 23rd International Systems and Software Product Line Conference - Volume BSimulation models are widely employed to model and simulate complex systems from different domains, such as automotive. These systems are becoming highly configurable to support different users' demands. Testing all of them is impracticable, and thus, ...
Comments