Skip to main content
SpringerLink
Log in

Are We There Yet? Determining the Adequacy of Formalized Requirements and Test Suites

  • Conference paper
  • First Online:
NASA Formal Methods (NFM 2015)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 9058))

Included in the following conference series:

Abstract

Structural coverage metrics have traditionally categorized code as either covered or uncovered. Recent work presents a stronger notion of coverage, checked coverage, which counts only statements whose execution contributes to an outcome checked by an oracle. While this notion of coverage addresses the adequacy of the oracle, for Model-Based Development of safety critical systems, it is still not enough; we are also interested in how much of the oracle is covered, and whether the values of program variables are masked when the oracle is evaluated. Such information can help system engineers identify missing requirements as well as missing test cases. In this work, we combine results from checked coverage with results from requirements coverage to help provide insight to engineers as to whether the requirements or the test suite need to be improved. We implement a dynamic backward slicing technique and evaluate it on several systems developed in Simulink. The results of our preliminary study show that even for systems with comprehensive test suites and good sets of requirements, our approach can identify cases where more tests or more requirements are needed to improve coverage numbers.

This work has been partially supported by NSF grants CNS-0931931 and CNS-1035715.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. IBM Rational Rhapsody (2014). http://www.ibm.com/developerworks/rational/products/rhapsody/

  2. IBM Rational Statemate (2014). http://www-03.ibm.com/software/products/en/ratistat

  3. Ammann, P., Delamaro, M.E., Offutt, J.: Establishing theoretical minimal sets of mutants. In: Proceedings of the 2014 IEEE International Conference on Software Testing, Verification, and Validation, IEEE Computer Society Washington, DC, USA (2014)

    Google Scholar 

  4. Baudin, P., Filliâtre, J.-C., Claude, M., Benjamin, M., Moy, Y., Virgile, P., Île-de France, I.S.: ANSI/ISO C specification language, ACSL (2008)

    Google Scholar 

  5. Beer, I., Ben-David, S., Eisner, C., Rodeh, Y.: Efficient detection of vacuity in ACTL formulas. In: Formal Methods in System Design, pp. 141–162 (2001)

    Google Scholar 

  6. Clause, J., Li, W., Orso, A.: Dytan: a generic dynamic taint analysis framework. In: Proceedings of the 2007 Int’l Symposium on Software Testing and Analysis, pp. 196–206 (2007)

    Google Scholar 

  7. Cuoq, P., Kirchner, F., Kosmatov, N., Prevosto, V., Signoles, J., Yakobowski, B.: Frama-c. In: Software Engineering and Formal Methods, pp. 233–247. Springer (2012)

    Google Scholar 

  8. DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Hints on test data selection: Help for the practicing programmer. Computer 11(4), 34–41 (1978)

    Article  Google Scholar 

  9. Devadas, S., Ghosh, A., Keutzer, K.: An observability-based code coverage metric for functional simulation. In: Proceedings of the 1996 IEEE/ACM Int’l Conf. on Computer-Aided Design, pp. 418–425 (1996)

    Google Scholar 

  10. Esterel-Technologies. SCADE Suite product description. http://www.esterel-technologies.com/v2/scadeSuiteForSafetyCriticalSoftwareDe

  11. Fallah, F., Devadas, S., Keutzer, K.: OCCOM-efficient computation of observability-based code coverage metrics for functional verification. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 20(8), 1003–1015 (2001)

    Article  Google Scholar 

  12. Fraser, G., Staats, M., McMinn, P., Arcuri, A., Padberg, F.: Does automated white-box test generation really help software testers? In: ISSTA 2013 Proceedings of the 2013 International Symposium on Software Testing and Analysis, pp. 291–301. ACM, New York, NY, USA (2013)

    Google Scholar 

  13. Gacek, A.: JKind - a Java implementation of the KIND model checker. https://github.com/agacek

  14. Gay, G., Staats, M., Whalen, M.W., Heimdahl, M.P.E.: Moving the goalposts: coverage satisfaction is not enough. In: Proceedings of the 7th International Workshop on Search-Based Software Testing, ACM, New York, NY, USA (2014)

    Google Scholar 

  15. Korel, B., Laski, J.: Dynamic program slicing. Information Processing Letters 29(3), 155–163 (1988)

    Article  MATH  Google Scholar 

  16. Kupferman, O., Vardi, M.Y.: Vacuity detection in temporal model checking. Journal on Software Tools for Technology Transfer 4(2), February 2003

    Google Scholar 

  17. GNUGPL License. Gcov: Gnu coverage tool. https://gcc.gnu.org

  18. Masri, W., Podgurski, A., Leon, D.: Detecting and debugging insecure information flows. In: Proceedings of the 15th Int’l Symposium on Software Reliability Engineering, pp. 198–209 (2004)

    Google Scholar 

  19. MathWorks Inc., Simulink. http://www.mathworks.com/products/simulink

  20. MathWorks Inc., Simulink Coder. http://www.mathworks.com/products/simulink-coder/

  21. MathWorks Inc., Simulink Design Verifier. http://www.mathworks.com/products/sldesignverifier

  22. Murugesan, A., Whalen, M.W., Rayadurgam, S., Heimdahl, M.P.E.: Compositional verification of a medical device system. In: ACM Int’l Conf. on High Integrity Language Technology (HILT) 2013. ACM, November 2013

    Google Scholar 

  23. Offutt, A.J., Untch, R.H.: Mutation testing for the new century. chapter Mutation 2000: Uniting the Orthogonal, pp. 34–44. Kluwer Academic Publishers, Norwell, MA, USA (2001)

    Google Scholar 

  24. Pecheur, C., Raimondi, F., Brat, G.: A formal analysis of requirements-based testing. In: Proceedings of the Eighteenth International Symposium on Software Testing and Analysis, pp. 47–56. ACM (2009)

    Google Scholar 

  25. Purandare, M., Somenzi, F.: Vacuum cleaning CTL formulae. In: Brinksma, E., Larsen, K.G. (eds.) CAV 2002. LNCS, vol. 2404, p. 485. Springer, Heidelberg (2002)

    Chapter  Google Scholar 

  26. Reactive systems inc. http://www.reactive-systems.com/index.msp

  27. RTCA/DO-178C. Software considerations in airborne systems and equipment certification

    Google Scholar 

  28. Rungta, N., Tkachuk, O., Person, S., Biatek, J., Whalen, M.W., Castle, J., Gundy-Burlet, K.: Helping system engineers bridge the peaks. In: TwinPeaks 2014 Proceedings of the 4th International Workshop on Twin Peaks of Requirements and Architecture, pp. 9–13. ACM, New York, NY, USA, (2014)

    Google Scholar 

  29. Schuler, D., Zeller, A.: Assessing oracle quality with checked coverage. In: ICST 2011 Proceedings of the 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation, pp. 90–99. IEEE Computer Society, Washington, DC, USA (2011)

    Google Scholar 

  30. Schuler, D., Zeller, A.: Checked coverage: an indicator for oracle quality. Software: Testing, Verification and Reliability 23(7), 531–551 (2013)

    Google Scholar 

  31. Staats, M., Gay, G., Whalen, M.W., Heimdahl, M.P.E.: On the danger of coverage directed test case generation. In: 15th Int’l Conf. on Fundamental Approaches to Software Engineering (FASE), April 2012

    Google Scholar 

  32. Weiser, M.: Program slicing. IEEE Transactions on Software Engineering 10(4), 352–357 (1984)

    Article  MATH  Google Scholar 

  33. Whalen, M., Gay, G., You, D., Heimdahl, M.P.E., Staats, M.: Observable modified condition/decision coverage. In: Proceedings of the 2013 Int’l Conf. on Software Engineering. ACM, May 2013

    Google Scholar 

  34. Whalen, M.W., Cofer, D.D., Miller, S.P., Krogh, B.H., Storm, W.: Integration of Formal Analysis into a Model-Based Software Development Process. In: Leue, S., Merino, P. (eds.) Formal Methods for Industrial Critical Systems. LNCS, vol. 4916, pp. 68–84. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  35. Whalen, M.W., Greve, D.A., Wagner, L.G.: Model Checking Information Flow. Springer-Verlag, Berlin Germany (2010)

    Book  Google Scholar 

  36. Whalen, M.W., Rajan, A., Heimdahl, M.P.E.: Coverage metrics for requirements-based testing. In: Proceedings of Int’l Symposium on Software Testing and Analysis, pp. 25–36. ACM, July 2006

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, University of Minnesota, 200 Union Street, Minneapolis, MN, 55455, USA

    Anitha Murugesan, Michael W. Whalen, Mats P. E. Heimdahl & Dongjiang You

  2. NASA Ames Research Center, Mountain, USA

    Neha Rungta & Oksana Tkachuk

  3. NASA Langley Research Center, Hampton, USA

    Suzette Person

Authors
  1. Anitha Murugesan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael W. Whalen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Neha Rungta
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Oksana Tkachuk
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Suzette Person
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Mats P. E. Heimdahl
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Dongjiang You
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anitha Murugesan .

Editor information

Editors and Affiliations

  1. Jet Propulsion Laboratory, Pasadena, California, USA

    Klaus Havelund

  2. Jet Propulsion Laboratory, Pasadena, California, USA

    Gerard Holzmann

  3. Jet Propulsion Laboratory, Pasadena, California, USA

    Rajeev Joshi

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Murugesan, A. et al. (2015). Are We There Yet? Determining the Adequacy of Formalized Requirements and Test Suites. In: Havelund, K., Holzmann, G., Joshi, R. (eds) NASA Formal Methods. NFM 2015. Lecture Notes in Computer Science(), vol 9058. Springer, Cham. https://doi.org/10.1007/978-3-319-17524-9_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-17524-9_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-17523-2

  • Online ISBN: 978-3-319-17524-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation