Skip to main content
SpringerLink
Log in

Test-Driven Development in scientific software: a survey

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Scientific software developers are increasingly employing various software engineering practices. Specifically, scientists are beginning to use Test-Driven Development (TDD). Even with this increasing use of TDD, the effect of TDD on scientific software development is not fully understood. To help scientific developers determine whether TDD is appropriate for their scientific projects, we surveyed scientific developers who use TDD to understand: (1) TDDs effectiveness, (2) the benefits and challenges of using TDD, and (3) the use of refactoring practices (an important part of the TDD process). Some key positive results include: (1) TDD helps scientific developers increase software quality, in particular functionality and reliability; and (2) TDD helps scientific developers reduce the number of problems in the early phase of projects. Conversely, some key challenges include: (1) TDD may not be effective for all types of scientific projects; and (2) Writing a good test is the most difficult task in TDD, particularly in a parallel computing environment. To summarize, TDD generally has a positive effect on the quality of scientific software, but it often requires a large effort investment. The results of this survey indicate the need for additional empirical evaluation of the use of TDD for the development of scientific software to help organizations make better decisions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. http://SE4Science.org/workshops.

  2. http://www.software-carpentry.org.

  3. definitions adapted from https://msdn.microsoft.com/en-us/library/aa292484(v=vs.71).aspx.

  4. http://www.cmake.org.

  5. http://www.cmake.org/cmake/help/v2.8.8/ctest.html.

  6. http://www.junit.org.

  7. https://code.google.com/p/googletest.

  8. http://nose.readthedocs.org.

  9. http://docs.python.org.

  10. http://cxxtest.com.

References

  • Abdel-Hamid, A. (2013). Refactoring as a lifeline: Lessons learned from refactoring. In Agile Conference (AGILE), 2013, pp. 129–136

  • Anselm, L. S., & Juliet, M. C. (1990). Basics of qualitative research: Rounded theory procedures and techniques. Newbury Park, CA: Sage Publications.

    Google Scholar 

  • Beck, K. (2002). Test driven development: By example. Boston, MA: Addison-Wesley Longman Publishing Co. Inc.

    Google Scholar 

  • Beck, K., & Andres, C. (2004). Extreme programming explained: Embrace change (2nd ed.). Boston, MA: Addison-Wesley Professional.

    Google Scholar 

  • Carver, J. (2011). Development of a mesh generation code with a graphical front-end: A case study. Journal of End User Computing, 23(4), 1–16.

    Article  Google Scholar 

  • Carver, J. C., Kendall, R. P., Squires, S. E., & Post, D. E. (2007). Software development environments for scientific and engineering software: A series of case studies. In The 29th international conference on software engineering (pp. 550–559). MN: Minneapolis.

  • Causevic, A., Sundmark, D., & Punnekkat, S. (2011). Factors limiting industrial adoption of test driven development: A systematic review. In The 4th international conference on software testing (pp. 337–346). Berlin: Verification and Validation.

  • Desai, C., Janzen, D., & Savage, K. (2008). A survey of evidence for Test-Driven Development in academia. SIGCSE Bulletin, 40(2), 97–101.

    Article  Google Scholar 

  • Eclipse. (2013). Photran—An integrated development environment and refactoring tool for fortran. http://www.eclipse.org/photran/. Accessed December 2013

  • Erdogmus, H., Morisio, M., & Torchiano, M. (2005). On the effectiveness of the test-first approach to programming. IEEE Transactions on Software Engineering, 31(3), 226–237. doi:10.1109/TSE.2005.37

    Article  Google Scholar 

  • Fowler, M. (1999). Refactoring: Improving the design of existing code. Boston, MA: Addison-Wesley Longman Publishing Co. Inc.

    MATH  Google Scholar 

  • Gamma, E., Helm, R., Johnson, R., & Vlissides, J. (1995). Design patterns: Elements of reusable object-oriented software. Boston, MA: Addison-Wesley Longman Publishing Co. Inc.

    MATH  Google Scholar 

  • ISO IEC. (2011). Systems and software engineering: System and software quality requirements and evaluation (SQuaRE)—System and software quality models. ISO/IEC, 25010, 2011.

  • Janzen, D., & Saiedian, H. (2005). Test-Driven Development concepts, taxonomy, and future direction. Computer, 38(9), 43–50. doi:10.1109/MC.2005.314

    Article  Google Scholar 

  • Kollanus, S. (2010). Test-Driven Development—Still a promising approach? In Proceedings of the 7th international conference on the quality of information and communications technology (pp. 403–408). Portugal: Porto.

  • Koskela, L. (2007). Test driven: Practical TDD and acceptance TDD for java developers. Greenwich, CO: Manning Publications Co.

    Google Scholar 

  • Mens, T., & Tourwé, T. (2004). A survey of software refactoring. IEEE Transactions on Software Engineering, 30(2), 126–139.

    Article  Google Scholar 

  • Nanthaamornphong, A., Morris, K., Rouson, D., & Michelsen, H. (2013). A case study: Agile development in the community laser-induced incandescence modeling environment (CLiiME). In The 5th international workshop on software engineering for computational science and engineering (pp. 9–18).

  • Nanthaamornphong, A., Carver, J., Morris, K., Michelsen, H., & Rouson, D. (2014). Building cliime via Test-Driven Development: A case study. Computing in Science Engineering, 16(3), 36–46.

    Article  Google Scholar 

  • Opdyke, W. F. (1992). Refactoring object-oriented frameworks. PhD thesis, University of Illinois at Urbana-Champaign, Champaign, Illinois, USA.

  • Orchard, D., & Rice, A. (2013). Upgrading fortran source code using automatic refactoring. Proceedings of the international workshop on refactoring tools (pp. 29–32). Indiana: Indianapolis.

    Google Scholar 

  • Overbey, J., Xanthos, S., Johnson, R., & Foote, B. (2005). Refactorings for fortran and high-performance computing. In Proceedings of the 2nd international workshop on software engineering for high performance computing system applications (pp. 37–39). Missouri: St. Louis.

  • Overbey, J. L., Negara, S., & Johnson, R. E. (2009). Refactoring and the evolution of fortran. In Proceedings of the international workshop on software engineering for computational science and engineering (pp. 28–34). British Columbia: Vancouver.

  • Rafique, Y., & Misic, V. (2013). The effects of Test-Driven Development on external quality and productivity: A meta-analysis. IEEE Transactions on Software Engineering, 39(6), 835–856.

    Article  Google Scholar 

  • Ruparelia, N. B. (2010). Software development lifecycle models. SIGSOFT Software Engineering Notes, 35(3), 8–13.

    Article  Google Scholar 

  • Sanchez, J., Williams, L., Maximilien, E. (2007). On the sustained use of a Test-Driven Development practice at ibm. In Agile conference (AGILE), 2007 (pp 5–14)

  • Sanders, R., & Kelly, D. (2008). Dealing with risk in scientific software development. IEEE Software, 25(4), 21–28.

    Article  Google Scholar 

  • Sletholt, M., Hannay, J., Pfahl, D., & Langtangen, H. (2012). What do we know about scientific software development’s agile practices? Computing in Science Engineering, 14(2), 24–37.

    Article  Google Scholar 

  • Sletholt, M. T., Hannay, J., Pfahl, D., Benestad, H. C., & Langtangen, H. P. (2011). A literature review of agile practices and their effects in scientific software development. In Proceedings of the 4th international workshop on software engineering for computational science and engineering (pp. 1–9). Hawaii: Honolulu.

Download references

Acknowledgments

The authors gratefully thank all participants in the survey for their time and contributions. Jeffrey Carver would like to acknowledge partial support from NSF Grants 1243887 and 1445344.

Author information

Authors and Affiliations

  1. Department of Information and Communication Technology, Prince of Songkla University, Phuket Campus, Phuket, Thailand

    Aziz Nanthaamornphong

  2. Department of Computer Science, University of Alabama, Tuscaloosa, AL, USA

    Jeffrey C. Carver

Authors
  1. Aziz Nanthaamornphong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jeffrey C. Carver
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jeffrey C. Carver.

Appendices

Appendix 1: Survey questions

figure cfigure cfigure c

Appendix 2: Automated testing tools

Tools

Languages

URL

Boost

C++

http://www.boost.org

Buildbot

Python

http://buildbot.net

CDash

Global

http://www.cdash.org

Clover

Java

https://www.atlassian.com/software/clover/overview

CMake

Global

http://www.cmake.org

CppUnit

C++

http://cppunit.sourceforge.net

CTest

Global

http://www.cmake.org/cmake/help/v2.8.8/ctest.html

CxxTest

C++

http://cxxtest.com

GCov

Global

http://gcc.gnu.org/onlinedocs/gcc/Gcov.html

GTest

C++

https://code.google.com/p/googletest/

Jenkins

Java

http://jenkins-ci.org

Igloo

C++

http://igloo-testing.org

JUnit

Java

http://www.junit.org

Lcov

Global

http://ltp.sourceforge.net/coverage/lcov.php

Marathon

Java

http://marathontesting.com

Nose

Python

http://nose.readthedocs.org

pFUnit

Fortran

http://pfunit.sourceforge.net

Pytest

Python

http://pytest.org

Python unit test

Python

http://docs.python.org

Qtest

Global

http://www.qasymphony.com/qtest.html

Sikuli

Global

http://www.sikuli.org

Silk TestPartner

Global

http://www.borland.com/products/silktestpartner/

Travis

Global

https://travis-ci.org

XUnit

.NET

http://xunit.codeplex.com

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nanthaamornphong, A., Carver, J.C. Test-Driven Development in scientific software: a survey. Software Qual J 25, 343–372 (2017). https://doi.org/10.1007/s11219-015-9292-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-015-9292-4

Keywords

Search

Navigation