Thanh H.D. Nguyen
Bram Adams
ABSTRACT
The goal of performance regression testing is to check for performance regressions in a new version of a software system. Performance regression testing is an important phase in the software development process. Performance regression testing is very time consuming yet there is usually little time assigned for it. A typical test run would output thousands of performance counters. Testers usually have to manually inspect these counters to identify performance regressions. In this paper, we propose an approach to analyze performance counters across test runs using a statistical process control technique called control charts. We evaluate our approach using historical data of a large software team as well as an open-source software project. The results show that our approach can accurately identify performance regressions in both software systems. Feedback from practitioners is very promising due to the simplicity and ease of explanation of the results.
- A. Avritzer and E. R. Weyuker. The automatic generation of load test suites and the assessment of the resulting software. IEEE Transactions on Software Engineering (TSE), 21(9):705--716, 1995. Google Scholar
Digital Library
- M. Y. Chen, E. Kiciman, E. Fratkin, A. Fox, and E. Brewer. Pinpoint: Problem determination in large, dynamic internet services. In International Conference on Dependable Systems and Networks (DSN), pages 595--604, 2002. Google ScholarDigital Library
- L. Cherkasova, K. Ozonat, M. Ningfang, J. Symons, and E. Smirni. Anomaly? application change? or workload change? towards automated detection of application performance anomaly and change. In International Conference on Dependable Systems and Networks (DSN), pages 452--461, 2008.Google ScholarCross Ref
- I. Cohen, M. Goldszmidt, T. Kelly, J. Symons, and J. S. Chase. Correlating instrumentation data to system states: a building block for automated diagnosis and control. In Symposium on Opearting Systems Design Implementation, pages 231--244, San Francisco, CA, 2004. USENIX Association. Google ScholarDigital Library
- Dell Inc. DVD Store Test Application, 2010. Ver. 2.1.Google Scholar
- K. C. Foo. Automated discovery of performance regressions in enterprise applications. Master's thesis, 2011.Google Scholar
- K. C. Foo, J. Zhen Ming, B. Adams, A. E. Hassan, Z. Ying, and P. Flora. Mining performance regression testing repositories for automated performance analysis. In International Conference on Quality Software (QSIC), pages 32--41, 2010. Google ScholarDigital Library
- H. W. Gunther. Websphere application server development best practices for performance and scalability. IBM WebSphere Application Server Standard and Advanced Editions - White paper, 2000.Google Scholar
- Hewlett Packard. Loadrunner, 2010.Google Scholar
- M. Jiang, M. A. Munawar, T. Reidemeister, and P. A. S. Ward. Automatic fault detection and diagnosis in complex software systems by information-theoretic monitoring. In International Conference on Dependable Systems Networks (DSN), pages 285--294, 2009.Google ScholarCross Ref
- Z. M. Jiang, A. E. Hassan, G. Hamann, and P. Flora. Automatic identification of load testing problems. In International Conference on Software Maintenance (ICSM), pages 307--316, 2008.Google ScholarCross Ref
- Z. M. Jiang, A. E. Hassan, G. Hamann, and P. Flora. Automatic performance analysis of load tests. In International Conference in Software Maintenance (ICSM), pages 125--134, Edmonton, 2009.Google Scholar
- H. Malik. A methodology to support load test analysis. In International Conference on Software Engineering (ICSE), pages 421--424, Cape Town, South Africa, 2010. ACM. Google ScholarDigital Library
- Microsoft Corp. Windows reliability and performance monitor, 2011.Google Scholar
- MySQL AB. Mysql community server, 2011. Ver. 5.5.Google Scholar
- W. Shewhart. Economic Control of Quality of Manufactured Product. American Society for Quality Control, 1931.Google Scholar
- The Apache Software Foundation. Tomcat, 2010. Ver. 5.5.Google Scholar
- I. Trubin. Capturing workload pathology by statistical exception detection system. In Computer Measurement Group (CMG), 2005.Google Scholar
- E. J. Weyuker and F. I. Vokolos. Experience with performance testing of software systems: Issues, an approach, and case study. IEEE Transactions on Software Engineering (TSE), 26(12):1147--1156, 2000. Google ScholarDigital Library
Index Terms
- Automated detection of performance regressions using statistical process control techniques
Recommendations
Automated Detection of Performance Regressions Using Regression Models on Clustered Performance Counters
ICPE '15: Proceedings of the 6th ACM/SPEC International Conference on Performance EngineeringPerformance testing is conducted before deploying system updates in order to ensure that the performance of large software systems did not degrade (i.e., no performance regressions). During such testing, thousands of performance counters are collected. ...
An industrial case study of automatically identifying performance regression-causes
MSR 2014: Proceedings of the 11th Working Conference on Mining Software RepositoriesEven the addition of a single extra field or control statement in the source code of a large-scale software system can lead to performance regressions. Such regressions can considerably degrade the user experience. Working closely with the members of a ...
Addressing Performance Regressions in DevOps: Can We Escape from System Performance Testing?
ICSE '23: Proceedings of the 45th International Conference on Software Engineering: Companion ProceedingsPerformance regression is an important type of performance issue in software systems. It indicates that the performance of the same features in the new version of the system becomes worse than that of previous versions, such as increased response time ...
Comments