Lionel Briand
ABSTRACT
Increasingly, we are faced with systems that are untestable, meaning that traditional testing methods are expensive, time-consuming or infeasible to apply due to factors such as the systems' continuous interactions with the environment and the deep intertwining of software with hardware.
In this paper we outline our vision to enable testing of untestable systems. Our key idea is to frame testing on models rather than operational systems. We refer to such testing as model testing. Our goal is to raise the level of abstraction of testing from operational systems to models of their behaviors and properties. The models that underlie model testing are executable representations of the relevant aspects of a system and its environment, alongside the risks of system failures. Such models necessarily have uncertainties due to complex, dynamic environment behaviors and the unknowns about the system. This makes it crucial for model testing to be uncertainty-aware. We propose to synergistically combine metaheuristic search, increasingly used in traditional software testing, with system and risk models to drive the search for faults that entail the most risk.
We expect model testing to bring early and cost-effective automation to the testing of many critical systems that defy existing automation techniques, thus significantly improving the dependability of such systems.
- R. Calinescu, C. Ghezzi, M. Kwiatkowska, and R. Mirandola. Self-adaptive software needs quantitative verification at runtime. Commun. ACM, 55(9):69--77, 2012. Google Scholar
Digital Library
- T. Y. Chen, T. H. Tse, and Z. Q. Zhou. Fault-based testing without the need of oracles. Information and Software Technology, 45(1):1--9, 2003.Google ScholarDigital Library
- E. Clarke, O. Grumberg, and D. Peled. Model Checking. MIT Press, 2001.Google ScholarCross Ref
- H. Hemmati, A. Arcuri, and L. Briand. Achieving Scalable Model-Based Testing through Test Case Diversity. ACM Trans. Softw. Eng. Methodol., 22(1):6:1--6:42, 2013. Google ScholarDigital Library
- M. Z. Iqbal, A. Arcuri, and L. Briand. Empirical investigation of search algorithms for environment model-based testing of real-time embedded software. In Proc. of ISSTA 2012, pages 199--209. ACM, 2012. Google ScholarDigital Library
- MathWorks. Mathworks MATLAB Simulink. http://www.mathworks.com/products/simulink/.Google Scholar
- R. Matinnejad, S. Nejati, L. Briand, T. Bruckmann, and C. Poull. Search-based automated testing of continuous controllers: Framework, tool support, and case studies. Information and Software Technology, 57:705--722, 2015.Google ScholarCross Ref
- P. McMinn. Search-based software test data generation: A survey. Softw. Test. Verif. Reliab., 14(2):105--156, 2004. Google ScholarDigital Library
- H. Shokry and M. Hinchey. Model-based verification of embedded software. Computer, 42(4):53--59, 2009. Google ScholarDigital Library
- E. J. Weyuker. On testing non-testable programs. The Computer Journal, 25(4):465--470, 1982.Google ScholarCross Ref
- Testing the untestable: model testing of complex software-intensive systems
Recommendations
The Impacts of Untestable Defects on Transition Fault Testing
VTS '06: Proceedings of the 24th IEEE VLSI Test SymposiumIn this paper, we investigate the impacts of the untestable defects, modeled by stuck-open faults and bridging faults, on the quality of transition fault test set. The presence of those defects may make some transition faults to be tested invalidly if ...
Testing "untestable" faults in three-state circuits
VTS '96: Proceedings of the 14th IEEE VLSI Test SymposiumHigh-performance, complex CMOS designs such as microprocessors continue to gain performance by the use of "non-conventional" circuits such as tri-state, ratio or precharged logic. Such circuits are also used in noncomplementary or DC-redundant ...
Generation of High Quality Tests for Robustly Untestable Path Delay Faults
In many designs a large portion of path delay faults is not robustly testable. In this paper, we investigate testing strategies for robustly untestable faults. We show that the quality of nonrobust tests may be very poor in detecting small defects ...
Comments