Abstract
Many industrial products are based on the use of embedded computer systems. Usually, these systems have to fulfil real-time requirements, and correct system functionality depends on their logical correctness as well as on their temporal correctness. In order to verify the temporal behavior of real-time systems, previous scientific work has, to a large extent, concentrated on static analysis techniques. Although these techniques offer the possibilty of providing safe estimates of temporal behavior for certain cases, there are a number of cases in practice for which static analysis can not be easily applied. Furthermore, no commercial tools for timing analysis of real-world programs are available. Therefore, the developed systems have to be thoroughly tested in order to detect existing deficiencies in temporal behavior, as well as to strengthen the confidence in temporal correctness. An investigation of existing test methods shows that they mostly concentrate on testing for logical correctness. They are not specialised in the examination of temporal correctness which is also essential to real-time systems. For this reason, existing test procedures must be supplemented by new methods which concentrate on determining whether the system violates its specified timing constraints. Normally, a violation means that outputs are produced too early, or their computation takes too long. The task of the tester therefore is to find the input situations with the longest or shortest execution times, in order to check whether they produce a temporal error. If the search for such inputs is interpreted as a problem of optimization, evolutionary computation can be used to automatically find the inputs with the longest or shortest execution times. This automatic search for accurate test data by means of evolutionary computation is called evolutionary testing. Experiments using evolutionary testing on a number of programs with up to 1511 LOC and 5000 input parameters have successfully identified new longer and shorter execution times than had been found using other testing techniques. Evolutionary testing, therefore, seems to be a promising approach for the verification of timing constraints. A combination of evolutionary testing and systematic testing offers further opportunities to improve the test quality, and could lead to an effective test strategy for real-time systems.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Baker, J.E. 1987. “Reducing Bias and Inefficiency in the Selection Algorithm” Proceedings of the 2nd International Conference on Genetic Algorithms and their Application.
Beizer, B. 1990. “Software Testing Techniques”. New York: Van Nostrand Reinhold.
Boden, E.B., and Martino, G.F. 1996. “Testing Software Using Order-Based Genetic Algorithms” Proceedings of the 1st Annual Conference on Genetic Programming pp. 461–466.
Braberman, V., Felder, M., and Marre, M. 1997. “Testing Timing Behavior of Real-Time Software” Proceedings of the 10th International Software Quality Week '97.
Clarke, D., and Lee, I. 1995. “Testing Real-Time Constraints in a Process Algebraic Setting” Proceedings of the 17th International Conference on Software Engineering.
Dasarathy, B. 1985. “Timing Constraints of Real-Time Systems: Constructs for Expressing Them, Methods of Validating Them” IEEE Transactions on Software Engineering 11(1): 80–86.
Davis, C.G. 1979. “Testing Large, Real-Time Software Systems”. Software Testing, Infotech State of the Art Report, Vol. 2, pp. 85–105.
Davis, L. 1996. “Handbook of Genetic Algorithms”. Boston: International Thomson Computer Press.
Goldberg, D.E. 1989. “Genetic Algorithms in Search, Optimization and Machine Learning”. Reading, MA: Addison-Wesley Publishing Company, Inc.
Grimm, K. 1996. “Systematic Testing of Software-Based Systems” Proceedings of the Second Annual ENCRESS Conference ENCRESS '96.
Grochtmann, M., and Grimm, K. 1993. “Classification Trees for Partition Testing” Software Testing, Verification & Reliability 3(2): 63–82.
Grochtmann, M., and Wegener, J. 1995. “Test Case Design Using Classification Trees and the Classification-Tree Editor CTE” Proceedings of the 8th International Quality Week '95.
Healy, C.A., Whalley, D.B., and Harmon, M.G. 1995. “Integrating the Timing Analysis of Pipelining and Caching” Proceedings of the 16th IEEE Real-Time Systems Symposium, pp. 288–297.
Heath, W.S. 1991. “Real-Time Software Techniques”. New York: Van Nostrand Reinhold.
Hennell, M.A., Hedley, D., and Riddell, I.J. 1987. “Automated Testing Techniques for Real-Time Embedded Software” Proceedings of the 1st European Software Engineering Conference ESEC '87.
Hetzel, B. 1988. “The Complete Guide to Software Testing”. Wellesley, MA: QED Information Sciences.
Jones, B.F., Sthamer, H.-H., and Eyres, D.E. 1996. “Automatic Structural Testing Using Genetic Algorithms” Software Engineering Journal 11(5): 299–306.
Kim, S., Min, S.L., and Ha, R. 1996. “Efficient Worst Case Timing Analysis of Data Caching” Proceedings of the IEEE Real-Time Technology and Applications Symposium, pp. 230–240.
Kopetz, H. 1993. Foreword in (Schütz, 1993).
Li, Y.S., Malik, S., and Wolfe, A. 1996. “Cache Modeling for Real-time Software: Beyond direct mapped instruction caches” Proceedings of the 17th IEEE Real-Time Systems Symposium.
Mandrioli, D., Morasca, S., and Morzenti, A. 1992. “Functional Test Case Generation for Real-Time Systems” Proceedings of the 3rd IFIP Working Conference on Dependable Computing for Critical Applications DCCA-3, pp. 29–61.
Mühlenbein, H., Schlierkamp-Voosen, D. 1993. “Predictive Models for the Breeder Genetic Algorithm: I. Continuous Parameter Optimization” Evolutionary Computation 1(1): 25–49.
Müller, F. 1996. “Statische Analyse der maximalen Programmausführungszeiten (Static Analysis of the Maximum Program Execution Times)” Lecture given at Daimler-Benz AG, Research and Technology, Berlin, Germany.
Müller, F. 1997. “Generalizing Timing Predictions to Set-Associative Caches” Proceedings of the Euromicro Workshop on Real-Time Systems, pp. 64–71.
Müller, F., Wegener, J. 1998. “A Comparison of Static Analysis and Evolutionary Testing for the Verification of Timing Constraints” To appear in Proceedings of the 4th IEEE Real-Time Technology and Application Symposium (RTAS'98).
Myers, G.J. 1979. “The Art of Software Testing”. New York: John Wiley & Sons.
Park, C.Y. 1993. “Predicting program execution times by analyzing static and dynamic program paths” Real-Time Systems 5(1): 31–62.
Pohlheim, H. 1996. “GEATbx: Genetic and Evolutionary Algorithm Toolbox for Use with Matlab-Documentation”. Technical Report, Technical University Ilmenau, Germany.
Puschner, P., Vrchoticky, A. 1997. “Problems in Static Worst-Case Execution Time Analysis” Proceedings of the 9th ITG/GI-Conference Measurement, Modeling and Evaluation of Computational and Communication Systems, pp. 18–25.
Roper, M. 1997. “Computer Aided Software Testing using Genetic Algorithms” Proceedings of the 10th International Quality Week '97.
Schultz, A.C., Grefenstette, J.J., and De Jong, K.A. 1993. “Test and Evaluation by Genetic Algorithms” IEEE Expert 8(5): 9–14.
Schütz, W. 1993. “The Testability of Distributed Real-Time Systems”. Boston: Kluwer Academic Publishers.
Stankovic, J.A. 1988. “Misconceptions about Real-Time Computing-A Serious Problem for Next-Generation Systems” IEEE Computer 21(10): 10–19.
Sthamer, H.-H. 1996. “The Automatic Generation of Software Test Data Using Genetic Algorithms”. PhD Thesis, Department of Electronics and Information Technology, University of Glamorgan, Wales, UK.
Vössner, S. 1996. “Convergence Measures for Genetic Algorithms”. Technical Report SOL 97–5, Systems Optimization Laboratory, Department of Engineering Economic Systems and Operations Research, Stanford University, USA.
Watkins, A.E.L. 1995. “A Tool for the Automatic Generation of Test Data Using Genetic Algorithms” Proceedings of the Software Quality Conference '95.
Wegener, J., and Pitschinetz, R. 1994. “TESSY-Yet Another Computer-Aided Software Testing Tool?” Proceedings of the 2nd European International Conference on Software Testing, Analysis & Review EuroSTAR '94.
Wegener, J., Grimm, K., Grochtmann, M., Sthamer, H.-H., and Jones, B.F. 1996. “Systematic Testing of Real-Time Systems” Proceedings of the 4th European International Conference on Software Testing, Analysis & Review EuroSTAR '96.
Wegener, J., Sthamer, H.-H., Jones, B.F., and Eyres, D.E. 1997. “Testing Real-Time Systems Using Genetic Algorithms” Software Quality Journal 6(2): 127–135.
Whitley, D. 1989. “The GENITOR Algorithm and Selection Pressure: Why Rank-Based Allocation of Reproductive Trials is Best” Proceedings of the Third International Conference on Genetic Algorithms, pp. 116–121.
Xanthakis, S., Ellis, C., Skourlas, C., LeGall, A., and Katsikas, S. 1992. “Application of Genetic Algorithms to Software Testing” Proceedings of the 5th International Conference on Software Engineering.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Wegener, J., Grochtmann, M. Verifying Timing Constraints of Real-Time Systems by Means of Evolutionary Testing. Real-Time Systems 15, 275–298 (1998). https://doi.org/10.1023/A:1008096431840
Issue Date:
DOI: https://doi.org/10.1023/A:1008096431840