Rudolf Ramler
ABSTRACT
Testing is a major cost factor in software development. Test automation has been proposed as one solution to reduce these costs. Test automation tools promise to increase the number of tests they run and the frequency at which they run them. So why not automate every test? In this paper we discuss the question "When should a test be automated?" and the trade-off between automated and manual testing. We reveal problems in the overly simplistic cost models commonly used to make decisions about automating testing. We introduce an alternative model based on opportunity cost and present influencing factors on the decision of whether or not to invest in test automation. Our aim is to stimulate discussion about these factors as well as their influence on the benefits and costs of automated testing in order to support researchers and practitioners reflecting on proposed automation approaches.
- Amland, S., Risk Based Testing and Metrics. 5th Int. Software Testing Analysis & Review Conference (EuroSTAR'99), Barcelona, Spain, Nov. 1999.Google Scholar
- Bach, J., Test Automation Snake Oil, 14th International Conference and Exposition on Testing Computer Software, Washington, DC, 1999.Google Scholar
- Boehm B.: Software Risk Management: Principles and Practices. IEEE Software, pp. 32--41, Jan. 1991. Google ScholarDigital Library
- Boehm, B., Value-Based Software Engineering: Overview and Agenda. In: Biffl S. et al.: Value-Based Software Engineering. Springer, 2005.Google Scholar
- Boehm, B., Basili, V.R., Software Defect Reduction Top 10 List, IEEE Computer, pp. 135--137, January 2001. Google ScholarDigital Library
- Compuware QADirector, Advanced risk-based test management. Compuware2006 (www.compuware.com/products/qacenter/qadirector.htm).Google Scholar
- Dustin, E. et. al., Automated Software Testing, Addison-Wesley, 1999.Google Scholar
- Fewster, M., Graham, D., Software Test Automation: Effective Use of Text Execution Tool, Addison-Wesley, 1999. Google ScholarDigital Library
- Harrold, M.J.: Testing: A Roadmap. In: The Future of Software Engineering, ed by Finkelstein, A., 22th International Conference on Software Engineering, Limerick, Ireland, June 2000. Google ScholarDigital Library
- Highsmith, J., Agile Software Development Ecosystems, Addison-Wesley, 2002. Google ScholarDigital Library
- Hoffman, D., Cost Benefits Analysis of Test Automation, Software Testing Analysis & Review Conference (STAR East). Orlando, FL, May 1999.Google Scholar
- Kaner, C., Bach, J., and Pettichord, B., Lessons Learned in Software Testing, Wiley, 2002. Google ScholarDigital Library
- Kaner, C., What is a Good Test Case, Software Testing Analysis & Review Conference (STAR East). May 2003.Google Scholar
- Kruchten, P., The Rational Unified Process: An Introduction, 3rd Ed., Addison-Wesley, 2003. Google ScholarDigital Library
- Link, J., Unit Testing in Java: How Tests Drive the Code, Morgan Kaufmann, 2003. Google ScholarDigital Library
- Linz, T., Daigl, M., GUI Testing Made Painless. Implementation and results of the ESSI Project Number 24306, 1998. In: Dustin, et. al., Automated Software Testing, Addison-Wesley, 1999, pp. 52.Google Scholar
- Mankiw G. W., Principles of Economics, 2nd Ed. Hartcourt College Publishers, 1998.Google Scholar
- Marick, B., When Should a Test Be Automated, Software Testing Analysis & Review Conference (STAR East). Orlando, FL, May 1999.Google Scholar
- Persson, C., Yilmaztürk, N., Establishment of Automated Regression Testing at ABB: Industiral Experience Report on Avoiding the Pitfalls, 19th IEEE International Conference on Automated Software Engineering (ASE'04), Linz, Austria, 2004. Google ScholarDigital Library
- Ramler R., Weippl E., Winterer M., Schwinger W., and Altmann J., A Quality-Driven Approach to Web-Testing. 2nd Conf. on Web Engineering (ICWE'02), Santa Fe, Argentina, Sept. 2002.Google Scholar
- Ramler R., Biffl S., Grünbacher P., Value-based Management of Software Testing. In: Biffl S. et al.: Value-Based Software Engineering. Springer, 2005.Google Scholar
- Redmill, F., Exploring risk-based testing and its implications. Software Testing, Verification and Reliability. 14, 3--15 (2004). Google ScholarDigital Library
- Rothermel, G., Elbaum, S., Putting Your Best Tests Forward. IEEE Software, 20(5), pp. 74--77, Sept/Oct 2003. Google ScholarDigital Library
- Schwaber, C., Gilpin, M., Evaluating Automated Functional Testing Tools, Forrester Research, February 2005.Google Scholar
- Voas, J. M., Morell, L. J., and Miller, K. W., Predicting where faults can hide from testing. IEEE Software, 8(2), pp. 41--48, March 1991. Google ScholarDigital Library
Index Terms
- Economic perspectives in test automation: balancing automated and manual testing with opportunity cost
Recommendations
The complementary aspect of automatically and manually generated test case sets
A-TEST 2016: Proceedings of the 7th International Workshop on Automating Test Case Design, Selection, and EvaluationThe test is a mandatory activity for software quality assurance. The knowledge about the software under testing is necessary to generate high-quality test cases, but to execute more than 80% of its source code is not an easy task, and demands an in-...
When to automate software testing? A decision-support approach based on process simulation
Software test processes are complex and costly. To reduce testing effort without compromising effectiveness and product quality, automation of test activities has been adopted as a popular approach in software industry. However, because test automation ...
When to automate software testing? decision support based on system dynamics: an industrial case study
ICSSP 2014: Proceedings of the 2014 International Conference on Software and System ProcessSoftware test processes are complex and costly. To reduce testing effort without compromising effectiveness and product quality, automation of test activities has been adopted as a popular approach in software industry. However, since test automation ...
Comments