ABSTRACT
All engineering disciplines are founded and rely on models, although they may differ on purposes and usages of modeling. Interdisciplinary domains such as Cyber Physical Systems (CPSs) seek approaches that incorporate different modeling needs and usages. Specifically, the Simulink modeling platform greatly appeals to CPS engineers due to its seamless support for simulation and code generation. In this paper, we propose a test generation approach that is applicable to Simulink models built for both purposes of simulation and code generation. We define test inputs and outputs as signals that capture evolution of values over time. Our test generation approach is implemented as a meta-heuristic search algorithm and is guided to produce test outputs with diverse shapes according to our proposed notion of diversity. Our evaluation, performed on industrial and public domain models, demonstrates that: (1) In contrast to the existing tools for testing Simulink models that are only applicable to a subset of code generation models, our approach is applicable to both code generation and simulation Simulink models. (2) Our new notion of diversity for output signals outperforms random baseline testing and an existing notion of signal diversity in revealing faults in Simulink models. (3) The fault revealing ability of our test generation approach outperforms that of the Simulink Design Verifier, the only testing toolbox for Simulink.
- N. Alshahwan and M. Harman. Augmenting test suites effectiveness by increasing output diversity. In ICSE 2012, pages 1345--1348. IEEE Press, 2012. Google ScholarDigital Library
- N. Alshahwan and M. Harman. Coverage and fault detection of the output-uniqueness test selection criteria. In ISSTA 2014, pages 181--192. ACM, 2014. Google ScholarDigital Library
- D. Balasubramanian, C. S. Pasareanu, M. W. Whalen, G. Karsai, and M. Lowry. Polyglot: modeling and analysis for multiple statechart formalisms. In ISSTA 2011, pages 45--55. ACM, 2011. Google ScholarDigital Library
- A. Baresel, H. Pohlheim, and S. Sadeghipour. Structural and functional sequence test of dynamic and state-based software with evolutionary algorithms. In GECCO 2003, pages 2428--2441. Springer, 2003. Google ScholarDigital Library
- J. Barnat, L. Brim, J. Beran, T. Kratochvila, and I. R. Oliveira. Executing model checking counterexamples in Simulink. In TASE 2012, pages 245--248. IEEE, 2012. Google ScholarDigital Library
- E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo. The oracle problem in software testing: A survey. TSE, 41(5):507--525, 2015.Google ScholarDigital Library
- N. T. Binh et al. Mutation operators for Simulink models. In KSE 2012, pages 54--59. IEEE, 2012. Google ScholarDigital Library
- F. Bohr and R. Eschbach. SIMOTEST: A tool for automated testing of hybrid real-time Simulink models. In ETFA 2011, pages 1--4. IEEE, 2011.Google ScholarCross Ref
- A. Brillout, N. He, M. Mazzucchi, D. Kroening, M. Purandare, P. Rümmer, and G. Weissenbacher. Mutation-based test case generation for Simulink models. In FMCO 2010, pages 208--227. Springer, 2010. Google ScholarDigital Library
- C. Cadar, D. Dunbar, and D. R. Engler. Klee: Unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI 2008, volume 8, pages 209--224, 2008. Google ScholarDigital Library
- J. A. Capon. Elementary Statistics for the Social Sciences: Study Guide. Wadsworth Publishing Company, 1991.Google Scholar
- D. K. Chaturvedi. Modeling and simulation of systems using MATLAB and Simulink. CRC Press, 2009. Google ScholarDigital Library
- E. M. Clarke, Jr., O. Grumberg, and D. A. Peled. Model Checking. MIT Press, 1999. Google ScholarDigital Library
- R. Cleaveland, S. A. Smolka, and S. T. Sims. An instrumentation-based approach to controller model validation. In MDRAS 2008, pages 84--97. Springer, 2008.Google ScholarDigital Library
- J. Cohen. Statistical power analysis for the behavioral sciences (rev). Lawrence Erlbaum Associates, Inc, 1977.Google Scholar
- K. Czarnecki and U. W. Eisenecker. Generative programming. Edited by G. Goos, J. Hartmanis, and J. van Leeuwen, page 15, 2000.Google Scholar
- R. France and B. Rumpe. Model-driven development of complex software: A research roadmap. In FOSE 2007, pages 37--54. IEEE Computer Society, 2007. Google ScholarDigital Library
- G. Fraser and A. Arcuri. Whole test suite generation. TSE, 39(2):276--291, 2013. Google ScholarDigital Library
- A. A. Gadkari, A. Yeolekar, J. Suresh, S. Ramesh, S. Mohalik, and K. Shashidhar. Automotgen: Automatic model oriented test generator for embedded control systems. In CAV 2008, pages 204--208. Springer, 2008. Google ScholarDigital Library
- G. Hamon, B. Dutertre, L. Erkok, J. Matthews, D. Sheridan, D. Cok, J. Rushby, P. Bokor, S. Shukla, A. Pataricza, et al. Simulink Design Verifier - Applying Automated Formal Methods to Simulink and Stateflow. In AFM 2008. Citeseer, 2008.Google Scholar
- M. P. Heimdahl, L. Duan, A. Murugesan, and S. Rayadurgam. Modeling and requirements on the physical side of cyber-physical systems. In TwinPeaks 2013, pages 1--7. IEEE, 2013.Google ScholarCross Ref
- D. Holling, A. Pretschner, and M. Gemmar. 8Cage: lightweight fault-based test generation for Simulink. In ASE 2014, pages 859--862. ACM, 2014. Google ScholarDigital Library
- JPF. Java pathfinder tool-set. http://babelfish.arc.nasa.gov/trac/jpf. {Online; accessed 17-Aug-2015}.Google Scholar
- J. Krizan, L. Ertl, M. Bradac, M. Jasansky, and A. Andreev. Automatic code generation from MATLAB/Simulink for critical applications. In CCECE 2014, pages 1--6. IEEE, 2014.Google ScholarCross Ref
- E. A. Lee and S. A. Seshia. Introduction to embedded systems: A cyber-physical systems approach. Lee & Seshia, 2011.Google Scholar
- F. Lindlar, A. Windisch, and J. Wegener. Integrating model-based testing with evolutionary functional testing. In ICSTW 2010, pages 163--172. IEEE, 2010. Google ScholarDigital Library
- S. Luke. Essentials of metaheuristics, volume 113. Lulu Raleigh, 2009.Google Scholar
- R. Matinnejad, S. Nejati, L. Briand, and T. Bruckmann. Effective test suites for mixed discrete-continuous stateflow controllers. In ESEC/FSE 2015, 2015. Google ScholarDigital Library
- Matinnejad, Reza. The paper extra resources (technical reports and the models). https://sites.google.com/site/myicseresources/.Google Scholar
- M. Mazzolini, A. Brusaferri, and E. Carpanzano. Model-checking based verification approach for advanced industrial automation solutions. In ETFA 2010, pages 1--8. IEEE, 2010.Google ScholarCross Ref
- P. McMinn, M. Stevenson, and M. Harman. Reducing qualitative human oracle costs associated with automatically generated test data. In ISSTA 2010, pages 1--4. ACM, 2010. Google ScholarDigital Library
- S. Mohalik, A. A. Gadkari, A. Yeolekar, K. Shashidhar, and S. Ramesh. Automatic test case generation from Simulink/Stateflow models using model checking. STVR, 24(2):155--180, 2014. Google ScholarDigital Library
- P. Nardi, M. E. Delamaro, L. Baresi, et al. Specifying automated oracles for Simulink models. In RTCSA 2013, pages 330--333. IEEE, 2013.Google ScholarCross Ref
- P. A. Nardi. On test oracles for Simulink-like models. PhD thesis, Universidade de Sao Paulo, 2014.Google Scholar
- P. Peranandam, S. Raviram, M. Satpathy, A. Yeolekar, A. Gadkari, and S. Ramesh. An integrated test generation tool for enhanced coverage of Simulink/Stateflow models. In DATE 2012, pages 308--311. IEEE, 2012. Google ScholarDigital Library
- B. Porat. A course in digital signal processing, volume 1. Wiley New York, 1997. Google ScholarDigital Library
- A. Pretschner, W. Prenninger, S. Wagner, C. Kühnel, M. Baumgartner, B. Sostawa, R. Zölch, and T. Stauner. One evaluation of model-based testing and its automation. In ICSE 2005, pages 392--401. ACM, 2005. Google ScholarDigital Library
- Reactive Systems Inc. Reactis Tester. http://www.reactive-systems.com/simulink-testing-validation.html, 2010. {Online; accessed 17-Aug-2015}.Google Scholar
- Reactive Systems Inc. Reactis Validator. http://www.reactive-systems.com/simulink-testing-validation.html, 2010. {Online; accessed 17-Aug-2015}.Google Scholar
- D. J. Richardson, S. L. Aha, and T. O. O'malley. Specification-based test oracles for reactive systems. In ICSE 1992, pages 105--118. ACM, 1992. Google ScholarDigital Library
- M. Satpathy, A. Yeolekar, P. Peranandam, and S. Ramesh. Efficient coverage of parallel and hierarchical stateflow models for test case generation. STVR, 22(7):457--479, 2012. Google ScholarDigital Library
- M. Satpathy, A. Yeolekar, and S. Ramesh. Randomized directed testing (REDIRECT) for Simulink/Stateflow models. In EMSOFT 2008, pages 217--226. ACM, 2008. Google ScholarDigital Library
- S. Sims and D. C. DuVarney. Experience report: the Reactis validation tool. SIGPLAN, 42(9), 2007. Google ScholarDigital Library
- The MathWorks Inc. Building a Clutch Lock-Up Model. http://nl.mathworks.com/help/simulink/examples/building-a-clutch-lock-up-model.html?refresh=true. {Online; accessed 17-Aug-2015}.Google Scholar
- The MathWorks Inc. C Code Generation from Simulink. http://nl.mathworks.com/help/dsp/ug/generate-code-from-simulink.html. {Online; accessed 17-Aug-2015}.Google Scholar
- The MathWorks Inc. Modeling a Fault-Tolerant Fuel Control System. http://nl.mathworks.com/help/simulink/examples/modeling-a-fault-tolerant-fuel-control-system.html. {Online; accessed 17-Aug-2015}.Google Scholar
- The MathWorks Inc. Simulink. http://www.mathworks.nl/products/simulink. {Online; accessed 17-Aug-2015}.Google Scholar
- The MathWorks Inc. Simulink Deign Verifier Cruise Control Test Generation. http://nl.mathworks.com/help/sldv/examples/extending-an-existing-test-suite.html?prodcode=DV&language=en. {Online; accessed 17-Aug-2015}.Google Scholar
- The MathWorks Inc. Simulink Design Verifier. http://nl.mathworks.com/products/sldesignverifier/?refresh=true. {Online; accessed 17-Aug-2015}.Google Scholar
- The MathWorks Inc. Types of Model Coverage. http://nl.mathworks.com/help/slvnv/ug/types-of-model-coverage.html. {Online; accessed 17-Aug-2015}.Google Scholar
- R. Venkatesh, U. Shrotri, P. Darke, and P. Bokil. Test generation for large automotive models. In ICIT 2012, pages 662--667. IEEE, 2012.Google ScholarCross Ref
- B. Wilmes and A. Windisch. Considering signal constraints in search-based testing of continuous systems. In ICSTW 2010, pages 202--211. IEEE, 2010. Google ScholarDigital Library
- A. Windisch. Search-based testing of complex simulink models containing stateflow diagrams. In ICSE 2009, pages 395--398. IEEE, 2009.Google ScholarCross Ref
- A. Windisch. Search-based test data generation from stateflow statecharts. In GECCO 2010, pages 1349--1356. ACM, 2010. Google ScholarDigital Library
- A. Windisch, F. Lindlar, S. Topuz, and S. Wappler. Evolutionary functional testing of continuous control systems. In GECCO 2009, pages 1943--1944. ACM, 2009. Google ScholarDigital Library
- I. H. Witten, E. Frank, and M. A. Hall. Data Mining: Practical Machine Learning Tools and Techniques. Elsevier, 2011. Google ScholarDigital Library
- Y. F. Yin, Y. B. Zhou, and Y. R. Wang. Research and improvements on mutation operators for Simulink models. In AMM 2014, volume 687, pages 1389--1393. Trans Tech Publ, 2014.Google ScholarCross Ref
- J. Zander, I. Schieferdecker, and P. J. Mosterman. Model-based testing for embedded systems. CRC Press, 2012. Google ScholarDigital Library
- J. Zander-Nowicka. Model-based testing of real-time embedded systems in the automotive domain. Fraunhofer-IRB-Verlag, 2008.Google Scholar
- Y. Zhan and J. A. Clark. Search-based mutation testing for Simulink models. In GECCO 2005, pages 1061--1068. ACM, 2005. Google ScholarDigital Library
- Y. Zhan and J. A. Clark. A search-based framework for automatic testing of MATLAB/Simulink models. JSS, 81:262--285, 2008. Google ScholarDigital Library
Index Terms
- Automated test suite generation for time-continuous simulink models
Recommendations
SimCoTest: a test suite generation tool for simulink/stateflow controllers
ICSE '16: Proceedings of the 38th International Conference on Software Engineering CompanionWe present SimCoTest, a tool to generate small test suites with high fault revealing ability for Simulink/Stateflow controllers. SimCoTest uses meta-heuristic search to (1) maximize the likelihood of presence of specific failure patterns in output ...
Effective test suites for mixed discrete-continuous stateflow controllers
ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software EngineeringModeling mixed discrete-continuous controllers using Stateflow is common practice and has a long tradition in the embedded software system industry. Testing Stateflow models is complicated by expensive and manual test oracles that are not amenable to ...
Mutation Operators for Simulink Models
KSE '12: Proceedings of the 2012 Fourth International Conference on Knowledge and Systems EngineeringMutation testing is a fault-based technique widely used for testing software. Particularly, it allows the effectiveness of a set of test data to be evaluated in terms of the ability to reveal faults. Nowadays, many industrial complex systems are more ...
Comments