Guoliang Jin
Linhai Song
Shan Lu
ABSTRACT
Developers frequently use inefficient code sequences that could be fixed by simple patches. These inefficient code sequences can cause significant performance degradation and resource waste, referred to as performance bugs. Meager increases in single threaded performance in the multi-core era and increasing emphasis on energy efficiency call for more effort in tackling performance bugs.
This paper conducts a comprehensive study of 110 real-world performance bugs that are randomly sampled from five representative software suites (Apache, Chrome, GCC, Mozilla, and MySQL). The findings of this study provide guidance for future work to avoid, expose, detect, and fix performance bugs.
Guided by our characteristics study, efficiency rules are extracted from 25 patches and are used to detect performance bugs. 332 previously unknown performance problems are found in the latest versions of MySQL, Apache, and Mozilla applications, including 219 performance problems found by applying rules across applications.
- M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds, and A. Muthitacharoen. Performance debugging for distributed systems of black boxes. In SOSP, 2003. Google Scholar
Digital Library
- M. Attariyan and J. Flinn. Automating configuration troubleshooting with dynamic information flow analysis. In OSDI, 2010. Google ScholarDigital Library
- W. Baek and T. M. Chilimbi. Green: a framework for supporting energy-conscious programming using controlled approximation. In PLDI, 2010. Google ScholarDigital Library
- C. Cadar, D. Dunbar, and D. Engler. Klee: Unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI, 2008. Google ScholarDigital Library
- V. Chipounov, V. Kuznetsov, and G. Candea. S2E: a platform for in-vivo multi-path analysis of software systems. In ASPLOS, 2011. Google ScholarDigital Library
- A. Chou, B. Chelf, D. Engler, and M. Heinrich. Using meta-level compilation to check FLASH protocol code. In ASPLOS, 2000. Google ScholarDigital Library
- A. Chou, J. Yang, B. Chelf, S. Hallem, and D. R. Engler. An empirical study of operating system errors. In SOSP, 2001. Google ScholarDigital Library
- I. Cohen, M. Goldszmidt, T. Kelly, J. Symons, and J. S. Chase. Correlating instrumentation data to system states: a building block for automated diagnosis and control. In OSDI, 2004. Google ScholarDigital Library
- M. Corporation. Performance Testing Guidance for Web Applications. Microsoft Press, 2007. Google ScholarDigital Library
- A. Diwan, M. Hauswirth, T. Mytkowicz, and P. F. Sweeney. Traceanalyzer: a system for processing performance traces. Softw. Pract. Exper., March 2011. Google ScholarDigital Library
- B. Dufour, B. G. Ryder, and G. Sevitsky. A scalable technique for characterizing the usage of temporaries in framework-intensive java applications. In FSE, 2008. Google ScholarDigital Library
- R. F. Dugan. Performance lies my professor told me: the case for teaching software performance engineering to undergraduates. In WOSP, 2004. Google ScholarDigital Library
- D. Engler, D. Y. Chen, S. Hallem, A. Chou, and B. Chelf. Bugs as deviant behavior: A general approach to inferring errors in systems code. In SOSP, 2001. Google ScholarDigital Library
- M. Ernst, A. Czeisler, W. G. Griswold, and D. Notkin. Quickly detecting relevant program invariants. In ICSE, 2000. Google ScholarDigital Library
- Fortify. HP Fortify Static Code Analyzer (SCA). https://www.fortify.com/products/hpfssc/source-code-analyzer.html.Google Scholar
- M. Gabel, J. Yang, Y. Yu, M. Goldszmidt, and Z. Su. Scalable and systematic detection of buggy inconsistencies in source code. In OOPSLA, 2010. Google ScholarDigital Library
- P. Godefroid, N. Klarlund, and K. Sen. Dart: directed automated random testing. In PLDI, 2005. Google ScholarDigital Library
- S. Gulwani, K. K. Mehra, and T. Chilimbi. Speed: precise and efficient static estimation of program computational complexity. In POPL, 2009. Google ScholarDigital Library
- M. Hauswirth, A. Diwan, P. F. Sweeney, and M. C. Mozer. Automating vertical profiling. In OOPSLA, 2005. Google ScholarDigital Library
- J. K. Hollingsworth, R. B. Irvin, and B. P. Miller. The integration of application and system based metrics in a parallel program performance tool. In PPOPP, 1991. Google ScholarDigital Library
- D. Hovemeyer and W. Pugh. Finding bugs is easy. In OOPSLA, 2004. Google ScholarDigital Library
- InfoWorld. Top 10 open source hall of famers. http://www.infoworld.com/d/open-source/top-10-open-source-hall-famers-848.Google Scholar
- R. B. Irvin and B. P. Miller. Mapping performance data for high-level and data views of parallel program performance. In ICS, 1996. Google ScholarDigital Library
- JMeter. Java desktop application designed for load tests. http://jakarta.apache.org/jmeter.Google Scholar
- M. Jovic, A. Adamoli, and M. Hauswirth. Catch me if you can: performance bug detection in the wild. In OOPSLA, 2011. Google ScholarDigital Library
- C. Killian, K. Nagaraj, S. Pervez, R. Braud, J. W. Anderson, and R. Jhala. Finding latent performance bugs in systems implementations. In FSE, 2010. Google ScholarDigital Library
- C. Lattner and V. Adve. LLVM: A compilation framework for lifelong program analysis & transformation. In CGO, 2004. Google ScholarDigital Library
- Z. Li, S. Lu, S. Myagmar, and Y. Zhou. CP-Miner: A Tool for Finding Copy-paste and Related Bugs in Operating System Code. In OSDI, 2004. Google ScholarDigital Library
- Z. Li, L. Tan, X. Wang, Y. Zhou, and C. Zhai. Have things changed now?: an empirical study of bug characteristics in modern open source software. In ASID, 2006. Google ScholarDigital Library
- Z. Li and Y. Zhou. PR-Miner: Automatically extracting implicit programming rules and detecting violations in large software code. In FSE, Sept 2005. Google ScholarDigital Library
- S. Liu, K. Pattabiraman, T. Moscibroda, and B. G. Zorn. Flikker: saving DRAM refresh-power through critical data partitioning. In ASPLOS, 2011. Google ScholarDigital Library
- T. Liu and E. D. Berger. Precise detection and automatic mitigation of false sharing. In OOPSLA, 2011. Google ScholarDigital Library
- V. B. Livshits and T. Zimmermann. Dynamine: Finding common error patterns by mining software revision histories. In FSE, 2005. Google ScholarDigital Library
- S. Lu, S. Park, E. Seo, and Y. Zhou. Learning from mistakes -- a comprehensive study of real world concurrency bug characteristics. In ASPLOS, 2008. Google ScholarDigital Library
- J. Mellor-Crummey, R. J. Fowler, G. Marin, and N. Tallent. Hpcview: A tool for top-down analysis of node performance. J. Supercomput., 23(1):81--104, 2002. Google ScholarDigital Library
- N. Meng, M. Kim, and K. S. McKinley. Systematic editing: generating program transformations from an example. In PLDI, 2011. Google ScholarDigital Library
- Microsoft. MSDN SAL annotations. http://msdn2.microsoft.com/en-us/library/ms235402.aspx.Google Scholar
- B. P. Miller, M. D. Callaghan, J. M. Cargille, J. K. Hollingsworth, R. B. Irvin, K. L. Karavanic, K. Kunchithapadam, and T. Newhall. The paradyn parallel performance measurement tool. Computer, 28(11), 1995. Google ScholarDigital Library
- N. Mitchell. The diary of a datum: an approach to modeling runtime complexity in framework-based applications. In ECOOP, 2006. Google ScholarDigital Library
- I. Molyneaux. The Art of Application Performance Testing: Help for Programmers and Quality Assurance. O'Reilly Media, 2009. Google ScholarDigital Library
- G. E. Morris. Lessons from the colorado benefits management system disaster. www.ad-mkt-review.com/publichtml/air/ai200411.html, 2004.Google Scholar
- G. Muller, Y. Padioleau, J. L. Lawall, and R. R. Hansen. Semantic patches considered helpful. Operating Systems Review, 40(3):90--92, 2006. Google ScholarDigital Library
- N. Palix, G. Thomas, S. Saha, C. Calvès, J. Lawall, and G. Muller. Faults in linux: ten years later. In ASPLOS, 2011. Google ScholarDigital Library
- S. E. Perl and W. E. Weihl. Performance assertion checking. In SOSP, 1993. Google ScholarDigital Library
- T. Richardson. 1901 census site still down after six months. http://www.theregister.co.uk/2002/07/03/1901_census_site_still_down/.Google Scholar
- C. J. Rossbach, O. S. Hofmann, and E. Witchel. Is transactional programming actually easier? In PPoPP, 2010. Google ScholarDigital Library
- A. Shankar, M. Arnold, and R. Bodik. Jolt: lightweight dynamic analysis and removal of object churn. In OOPSLA, 2008. Google ScholarDigital Library
- K. Shen, C. Stewart, C. Li, and X. Li. Reference-driven performance anomaly identification. In SIGMETRICS, 2009. Google ScholarDigital Library
- R. L. Sites. Identifying dark latency.Google Scholar
- C. U. Smith and L. G. Williams. Software performance antipatterns. In Proceedings of the 2nd international workshop on Software and performance, 2000. Google ScholarDigital Library
- Stefan Bodewig. Bug 45396: There is no hint in the javadocs. https://issues.apache.org/ bugzilla/show_bug.cgi?id=45396#c4.Google Scholar
- C. Stewart, K. Shen, A. Iyengar, and J. Yin. Entomomodel: Understanding and avoiding performance anomaly manifestations. In MASCOTS, 2010. Google ScholarDigital Library
- M. Sullivan and R. Chillarege. A comparison of software defects in database management systems and operating systems. In FTCS, 1992.Google ScholarCross Ref
- L. Torvalds. Sparse - a semantic parser for c. http://www.kernel.org/pub/software/devel/sparse/.Google Scholar
- J. S. Vetter and P. H. Worley. Asserting performance expectations. In Supercomputing, 2002. Google ScholarDigital Library
- wikipedia. Chi-squared test. http://en.wikipedia.org/wiki/Chi-squared_test.Google Scholar
- G. Xu, M. Arnold, N. Mitchell, A. Rountev, and G. Sevitsky. Go with the flow: profiling copies to find runtime bloat. In PLDI '09, 2009. Google ScholarDigital Library
- G. Xu, N. Mitchell, M. Arnold, A. Rountev, E. Schonberg, and G. Sevitsky. Finding low-utility data structures. In PLDI, 2010. Google ScholarDigital Library
- W. Xu, L. Huang, A. Fox, D. Patterson, and M. I. Jordan. Detecting large-scale system problems by mining console logs. In SOSP, 2009. Google ScholarDigital Library
- A. Zeller. Isolating cause-effect chains from computer programs. In FSE, 2002. Google ScholarDigital Library
Index Terms
- Understanding and detecting real-world performance bugs
Recommendations
Understanding and detecting real-world performance bugs
PLDI '12Developers frequently use inefficient code sequences that could be fixed by simple patches. These inefficient code sequences can cause significant performance degradation and resource waste, referred to as performance bugs. Meager increases in single ...
A qualitative study on performance bugs
MSR '12: Proceedings of the 9th IEEE Working Conference on Mining Software RepositoriesSoftware performance is one of the important qualities that makes software stand out in a competitive market. However, in earlier work we found that performance bugs take more time to fix, need to be fixed by more experienced developers and require ...
Security versus performance bugs: a case study on Firefox
MSR '11: Proceedings of the 8th Working Conference on Mining Software RepositoriesA good understanding of the impact of different types of bugs on various project aspects is essential to improve software quality research and practice. For instance, we would expect that security bugs are fixed faster than other types of bugs due to ...
Comments