Muhammad Nouman Zafar
ABSTRACT
Adequate testing of safety-critical systems is vital to ensure correct functional and non-functional operations. Previous research has shown that testing such systems requires a lot of effort, thus automated testing techniques have found a certain degree of success. However, automated testing has not replaced the need for manual testing, rather a common industrial practice exhibits a balance between automated and manual testing. In this respect, comparing manual testing with automated testing techniques continues to be an interesting topic to investigate. The need for this investigation is most apparent at system-level testing of industrial systems, where there is a lack of results on how different testing techniques perform concerning both structural and system-level metrics such as Modified Condition/Decision Coverage (MC/DC) and requirement coverage. In addition to the coverage, the cost of these techniques will also determine their efficiency and thus practical viability. In this paper, we have developed cost models for efficiency measurement and performed an experimental evaluation of manual testing, model-based testing (MBT), and combinatorial testing (CT) in terms of MC/DC and requirement coverage. The evaluation is done in an industrial context of a safety-critical system that controls several functions on-board the passenger trains. We have reported the dominant conditions of MC/DC affected by each technique while generating MC/DC adequate test suites. Moreover, we investigated differences and overlaps of test cases generated by each of the three techniques. The results showed that all test suites achieved 100% requirement coverage except the test suite generated by the pairwise testing strategy. However, MBT-generated test suites were more MC/DC adequate and provided a higher number of both similar and unique test cases. Moreover, unique test cases generated by MBT had an observable effect on MC/DC, which will complement manual testing to increase MC/DC coverage. The least dominant MC/DC condition fulfilled by the generated test cases by all three techniques is the 'independent effect of a condition on the outcomes of a decision'. Lastly, the evaluation also showed CT as the most efficient testing technique amongst the three in terms of time required for test generation and execution, but with an added cost parameter of manual identification of expected outcomes.
- Wasif Afzal, Ahmad Nauman Ghazi, Juha Itkonen, Richard Torkar, Anneliese Andrews, and Khurram Bhatti. 2015. An experiment on the effectiveness and efficiency of exploratory testing. Empirical Software Engineering 20 (2015), 844--878.Google Scholar
Digital Library
- Saswat Anand, Edmund K Burke, Tsong Yueh Chen, John Clark, Myra B Cohen, Wolfgang Grieskamp, Mark Harman, Mary Jean Harrold, Phil McMinn, Antonia Bertolino, et al. 2013. An orchestrated survey of methodologies for automated software test case generation. Journal of Systems and Software 86, 8 (2013), 1978--2001.Google ScholarDigital Library
- Andrea Bombarda and Angelo Gargantini. 2020. An Automata-Based Generation Method for Combinatorial Sequence Testing of Finite State Machines. In 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 157--166.Google Scholar
- Josip Bozic, Bernhard Garn, Ioannis Kapsalis, Dimitris Simos, Severin Winkler, and Franz Wotawa. 2015. Attack pattern-based combinatorial testing with constraints for web security testing. In 2015 IEEE International Conference on Software Quality, Reliability and Security. IEEE, 207--212.Google ScholarDigital Library
- Jörg Brauer, Jan Peleska, and Uwe Schulze. 2012. Efficient and trustworthy tool qualification for model-based testing tools. In IFIP International Conference on Testing Software and Systems. Springer, 8--23.Google ScholarCross Ref
- Cristian Cadar, Daniel Dunbar, Dawson R Engler, et al. 2008. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI, Vol. 8. 209--224.Google ScholarDigital Library
- Yin Chen, Sven Linder, and Jonas Wigstein. 2019. An Approach of Creating Component Design Specification for Safety-Related Software in Railway. In 2019 Annual Reliability and Maintainability Symposium (RAMS). IEEE, 1--4.Google Scholar
- John Joseph Chilenski and Steven P Miller. 1994. Applicability of modified condition/decision coverage to software testing. Software Engineering Journal 9, 5 (1994), 193--200.Google ScholarCross Ref
- Myra B Cohen, Peter B Gibbons, Warwick B Mugridge, and Charles J Colbourn. 2003. Constructing test suites for interaction testing. In 25th International Conference on Software Engineering, 2003. Proceedings. IEEE, 38--48.Google ScholarCross Ref
- Siddhartha R Dalal, Ashish Jain, Nachimuthu Karunanithi, JM Leaton, Christopher M Lott, Gardner C Patton, and Bruce M Horowitz. 1999. Model-based testing in practice. In Proceedings of the 21st international conference on Software engineering. 285--294.Google ScholarDigital Library
- Arilo C. Dias-Neto and Guilherme H. Travassos. 2010. A Picture from the Model-Based Testing Area: Concepts, Techniques, and Challenges. In Advances in Computers, Marvin V. Zelkowitz (Ed.). Advances in Computers, Vol. 80. Elsevier, 45--120.Google Scholar
- Arnaud Dupuy and Nancy Leveson. 2000. An empirical evaluation of the MC/DC coverage criterion on the HETE-2 satellite software. In 19th DASC. 19th Digital Avionics Systems Conference. Proceedings (Cat. No. 00CH37126), Vol. 1. IEEE, 1B6--1.Google Scholar
- Mounia El qortobi, Amine Rahj, Jamal Bentahar, and Rachida Dssouli. 2020. Test Generation Tool for Modified Condition/Decision Coverage: Model Based Testing. In Proceedings of the 13th International Conference on Intelligent Systems: Theories and Applications. 1--6.Google Scholar
- Eduard Enoiu, Daniel Sundmark, Adnan Čaušević, and Paul Pettersson. 2017. A Comparative Study of Manual and Automated Testing for Industrial Control Software. In 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST). 412--417. Google ScholarCross Ref
- Miraldi Fifo, Eduard Enoiu, and Wasif Afzal. 2019. On measuring combinatorial coverage of manually created test cases for industrial software. In 2019 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 264--267.Google ScholarCross Ref
- Svetoslav R Ganov, Chip Killmar, Sarfraz Khurshid, and Dewayne E Perry. 2008. Test generation for graphical user interfaces based on symbolic execution. In Proceedings of the 3rd international workshop on Automation of software test. 33--40.Google ScholarDigital Library
- Gregory Gay, Ajitha Rajan, Matt Staats, Michael Whalen, and Mats PE Heimdahl. 2016. The effect of program and model structure on the effectiveness of mc/dc test adequacy coverage. ACM Transactions on Software Engineering and Methodology (TOSEM) 25, 3 (2016), 1--34.Google ScholarDigital Library
- Jon Hagar, Rick Kuhn, Raghu Kacker, and Tom Wissink. 2014. Introducing combinatorial testing in a large organization: Pilot project experience report. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops. IEEE, 153--153.Google ScholarDigital Library
- Jon Hagar, Rick Kuhn, Raghu Kacker, and Tom Wissink. 2014. Introducing Combinatorial Testing in a Large Organization: Pilot Project Experience Report. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.Google ScholarDigital Library
- Kelly J Hayhurst. 2001. A practical tutorial on modified condition/decision coverage. DIANE Publishing.Google Scholar
- Hadi Hemmati, Syed S Arefin, and Howard W Loewen. 2018. Evaluating specification-level MC/DC criterion in model-based testing of safety critical systems. In 2018 IEEE/ACM 40th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP). IEEE, 256--265.Google Scholar
- Linghuan Hu, W. Eric Wong, D. Richard Kuhn, and Raghu N. Kacker. 2020. How does combinatorial testing perform in the real world: an empirical study. Empirical Software Engineering 25 (2020), 2661--2693.Google ScholarCross Ref
- Anne Kramer and Bruno Legeard. 2016. Model-based testing essentials-guide to the ISTQB certified model-based tester: foundation level. John Wiley & Sons.Google Scholar
- Peter M. Kruse, Nelly Condori-Fernandez, Tanja E.J. Vos, Alessandra Bagnato, and Etienne Brosse. 2013. Combinatorial Testing Tool Learnability in an Industrial Environment. In 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement.Google ScholarCross Ref
- D Richard Kuhn, Raghu N Kacker, Yu Lei, et al. 2015. Combinatorial coverage as an aspect of test quality. CrossTalk 28, 2 (2015), 19--23.Google Scholar
- Rick Kuhn and Raghu Kacker. 2011. Practical combinatorial (t-way) methods for detecting complex faults in regression testing. In 2011 27th IEEE International Conference on Software Maintenance (ICSM). IEEE, 599--599.Google ScholarDigital Library
- Kiran Lakhotia, Phil McMinn, and Mark Harman. 2009. Automated Test Data Generation for Coverage: Haven't We Solved This Problem Yet?. In 2009 Testing: Academic and Industrial Conference - Practice and Research Techniques. 95--104. Google ScholarDigital Library
- Hareton KN Leung and Lee J White. 1991. A cost model to compare regression test strategies. In ICSM, Vol. 91. 201--208.Google Scholar
- Dong Li, Linghuan Hu, Ruizhi Gao, W Eric Wong, D Richard Kuhn, and Raghu N Kacker. 2017. Improving MC/DC and fault detection strength using combinatorial testing. In 2017 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C). IEEE, 297--303.Google ScholarCross Ref
- W. Li, F. Le Gall, and N. Spaseski. 2017. A survey on model-based testing tools for test case generation. In International Conference on Tools and Methods for Program Analysis. Springer.Google Scholar
- Arthur Marques, Franklin Ramalho, and Wilkerson L Andrade. 2014. Comparing model-based testing with traditional testing strategies: An empirical study. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops. IEEE, 264--273.Google ScholarDigital Library
- Ahmed Mateen, Qingsheng Zhu, and Salman Afsar. 2018. Comparitive Analysis of Manual vs Automotive Testing for Software Quality. In Proceedings of the 7th International Conference on Software Engineering and New Technologies. 1--7.Google ScholarDigital Library
- Stefan Mohacsi, Michael Felderer, and Armin Beer. 2015. Estimating the Cost and Benefit of Model-Based Testing: A Decision Support Procedure for the Application of Model-Based Testing in Industry. In 2015 41st Euromicro Conference on Software Engineering and Advanced Applications. 382--389. Google ScholarDigital Library
- Cu D Nguyen, Alessandro Marchetto, and Paolo Tonella. 2012. Combining model-based and combinatorial testing for effective test case generation. In Proceedings of the 2012International Symposium on Software Testing and Analysis. 100--110.Google ScholarDigital Library
- Sebastian Oster, Ivan Zorcic, Florian Markert, and Malte Lochau. 2011. MoSo-PoLiTe: tool support for pairwise and model-based software product line testing. In Proceedings of the 5th Workshop on Variability Modeling of Software-Intensive Systems. 79--82.Google ScholarDigital Library
- Mike Papadakis and Nicos Malevris. 2010. Automatic mutation test case generation via dynamic symbolic execution. In 2010 IEEE 21st International Symposium on Software Reliability Engineering. IEEE, 121--130.Google ScholarDigital Library
- Rudolf Ramler, Theodorich Kopetzky, and Wolfgang Platz. 2012. Combinatorial Test Design in the TOSCA Testsuite: Lessons Learned and Practical Implications. In 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation. 569--572. Google ScholarDigital Library
- Sanjai Rayadurgam and Mats Heimdahl. 2003. Generating MC/DC adequate test sequences through model checking. (2003).Google Scholar
- Mary Sánchez-Gordón, Laxmi Rijal, and Ricardo Colomo-Palacios. 2020. Beyond Technical Skills in Software Testing: Automated versus Manual Testing. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops. 161--164.Google ScholarDigital Library
- Christoph Schulze, Dharmalingam Ganesan, Mikael Lindvall, Rance Cleaveland, and Daniel Goldman. 2014. Assessing model-based testing: an empirical study conducted in industry. In Companion Proceedings of the 36th International Conference on Software Engineering. 135--144.Google ScholarDigital Library
- Ossi Taipale, Jussi Kasurinen, Katja Karhu, and Kari Smolander. 2011. Trade-off between automated and manual software testing. International Journal of System Assurance Engineering and Management 2, 2 (2011), 114--125.Google ScholarCross Ref
- M. Utting and B. Legeard. 2010. Practical model-based testing: a tools approach. Elsevier.Google Scholar
- Erik Van Veenendaal, Dorothy Graham, and Rex Black. 2008. "Foundations of Software Testing: ISTQB Certification. Cengage Learning EMEA (2008), 30.Google Scholar
- Sergiy Vilkomir, Aparna Alluri, D Richard Kuhn, and Raghu N Kacker. 2017. Combinatorial and MC/DC coverage levels of random testing. In 2017 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C). IEEE, 61--68.Google ScholarCross Ref
- Michael Wagner, Kristoffer Kleine, Dimitris E. Simos, Rick Kuhn, and Raghu Kacker. 2020. CAGEN: A fast combinatorial test generation tool with support for constraints and higher-index arrays. In 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). 191--200. Google ScholarCross Ref
- Allan L White. 2001. Comments on modified condition/decision coverage for software testing [of flight control software]. In 2001 IEEE Aerospace Conference Proceedings (Cat. No. 01TH8542), Vol. 6. IEEE, 2821--2827.Google ScholarCross Ref
- Muhammad Nouman Zafar, Wasif Afzal, and Eduard Enoiu. 2021. Towards a Workflow for Model-Based Testing of Embedded Systems. In Proceedings of the 12th International Workshop on Automating TEST Case Design, Selection, and Evaluation. Association for Computing Machinery, New York, NY, USA.Google ScholarDigital Library
- Muhammad Nouman Zafar, Wasif Afzal, Eduard Paul Enoiu, Athanasios Stratis, Aitor Arrieta, and Goiuria Sagardui. 2021. Model-Based Testing in Practice: An Industrial Case Study using GraphWalker. In Innovations in Software Engineering Conference 2021. http://www.es.mdh.se/publications/6101-Google ScholarDigital Library
- Muhammad Nouman Zafar, Wasif Afzal, Eduard Paul Enoiu, Athanasios Stratis, and Ola Sellin. 2021. A Model-Based Test Script Generation Framework for Embedded Software. In The 17th Workshop on Advances in Model Based Testing. http://www.es.mdh.se/publications/6172-Google Scholar
Index Terms
- Evaluating system-level test generation for industrial software: a comparison between manual, combinatorial and model-based testing
Recommendations
Coverage-guided test generation for continuous and hybrid systems
In this paper, we describe a formal framework for conformance testing of continuous and hybrid systems, using the international standard `Formal Methods in Conformance Testing' FMCT. We propose a novel test coverage measure for these systems, which is ...
Automatic Test Generation using Checkpoint Encoding and Antirandom Testing
ISSRE '97: Proceedings of the Eighth International Symposium on Software Reliability EngineeringThe implementation of an efficient automatic test generation scheme for black-box testing is discussed. It uses checkpoint encoding and antirandom testing schemes. Checkpoint encoding converts test generation to a binary problem. The checkpoints are ...
A visual test development environment for GUI systems
We have implemented an experimental test development environment (TDE) intended to raise the effectiveness of tests produced for GUI systems, and raise the productivity of the GUI system tester.The environment links a test designer, a test design ...
Comments