Panagiotis Stratis
ABSTRACT
With the growing complexity of software, the number of test cases needed for effective validation is extremely large. Executing these large test suites is expensive, both in terms of time and energy. Cache misses are known to be one of the main factors contributing to execution time of a software. Cache misses are reduced by increasing the locality of memory references. For a single program run, compiler optimisations help improve data locality and code layout optimisations help improve spatial locality of instructions. Nevertheless, cache locality optimisations have not been proposed and explored across several program runs, which is the case when we run several test cases.
In this paper, we propose and evaluate a novel approach to improve instruction locality across test case runs. Our approach measures the distance between test case runs (number of different instructions). We then permute the test cases for execution so that the distance between neighbouring test cases is minimised. We hypothesise that test cases executed in this new order for improved instruction locality will reduce time consumed.
We conduct a preliminary evaluation with four subject programs and test suites from the SIR repository to answer the following questions, 1.~Is execution time of a test suite affected by the order in which test cases are executed? and 2.~How does time consumed in executing our permutation compare to random test case permutations? We found that the order in which test cases are executed has a definite impact on execution time. The extent of impact varies, based on program characteristics and test cases. Our approach outperformed more than 97% of random test case permutations on 3 of the 4 subject programs and did better than 93% of the random orderings on the remaining subject program. Using the optimised permutation, we saw a maximum reduction of 7.4% over average random permutation execution time and 34.7% over the worst permutation.
- Polly LLVM library. http://polly.llvm.org/index.html.Google Scholar
- Kristof Beyls and Erik D’Hollander. Reuse distance as a metric for cache behavior. In Proc. of the IASTED Conf. on Parallel and Distributed Computing and Systems, volume 14, pages 350–360, 2001.Google Scholar
- Kristof Beyls and Erik D’Hollander. Refactoring for data locality. Computer, 42(2):62–71, 2009. Google ScholarDigital Library
- Steve Carr, Kathryn S McKinley, and Chau-Wen Tseng. Compiler optimizations for improving data locality, volume 28. ACM, 1994.Google ScholarDigital Library
- J Bradley Chen and Bradley DD Leupen. Improving instruction locality with just-in-time code layout. In Proceedings of the USENIX Windows NT Workshop, pages 25–32, 1997. Google ScholarDigital Library
- Tien-Fu Chen and Jean-Loup Baer. Effective hardware-based data prefetching for high-performance processors. Computers, IEEE Transactions on, 44(5):609–623, 1995. Google ScholarDigital Library
- Peter J Denning. The locality principle. Communications of the ACM, 48(7):19–24, 2005. Google ScholarDigital Library
- Hyunsook Do, Sebastian G. Elbaum, and Gregg Rothermel. Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Empirical Software Engineering: An International Journal, 10(4):405–435, 2005. Google ScholarDigital Library
- Mats PE Heimdahl and Devaraj George. Test-suite reduction for model based tests: Effects on test quality and implications for testing. In Proceedings of the 19th IEEE international conference on Automated software engineering, pages 176–185, 2004. Google ScholarDigital Library
- W-m W Hwu and Pohua P Chang. Achieving high instruction cache performance with an optimizing compiler. In ACM SIGARCH Computer Architecture News, volume 17, pages 242–251. ACM, 1989. Google ScholarDigital Library
- Teresa L Johnson, Matthew C Merten, and Wen-Mei W Hwu. Run-time spatial locality detection and optimization. In Proceedings of the 30th annual ACM/IEEE international symposium on Microarchitecture, pages 57–64, 1997. Google ScholarDigital Library
- Sanjeev Kumar and Christopher Wilkerson. Exploiting spatial locality in data caches using spatial footprints. In ACM SIGARCH Computer Architecture News, volume 26, pages 357–368. IEEE Computer Society, 1998. Google ScholarDigital Library
- Chris Lattner and Vikram Adve. Llvm: A compilation framework for lifelong program analysis & transformation. In Code Generation and Optimization, 2004. CGO 2004. International Symposium on, pages 75–86. IEEE, 2004. Google ScholarDigital Library
- C. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser, G. Lowney, S. Wallace, V. Reddi, and K. Hazelwood. Pin: building customized program analysis tools with dynamic instrumentation. In ACM Sigplan Notices, volume 40, pages 190–200. ACM, 2005. Google ScholarDigital Library
- C. Pyo, K. Lee, H. Han, and G. Lee. Reference distance as a metric for data locality. In High Performance Computing on the Information Superhighway, 1997. HPC Asia’97, pages 151–156. IEEE, 1997. Google ScholarDigital Library
- Ajitha Rajan. Coverage metrics to measure adequacy of black-box test suites. In 21st ASE, pages 335–338. IEEE, 2006. Google ScholarDigital Library
- A. Ramirez, L. Barroso, K. Gharachorloo, R. Cohn, J. Larriba-Pey, G. Lowney, and M. Valero. Code layout optimizations for transaction processing workloads. In ACM SIGARCH Computer Architecture News, volume 29, pages 155–164. ACM, 2001. Google ScholarDigital Library
- Gabriel Rivera and Chau-Wen Tseng. Data transformations for eliminating conflict misses. In ACM SIGPLAN Notices, volume 33, pages 38–49. ACM, 1998. Google ScholarDigital Library
- Patrick J. Schroeder and Bogdan Korel. Black-box test reduction using input-output analysis. In ISSTA, pages 173–177. ACM, 2000. Google ScholarDigital Library
- V. Tiwari, S. Malik, A. Wolfe, and M.T.-C. Lee. Instruction level power analysis and optimization of software. In VLSI Design, 1996. Proceedings., Ninth International Conference on, pages 326–328, Jan 1996. Google ScholarDigital Library
- N. Vijaykrishnan, M. Kandemir, M.J. Irwin, H.S. Kim, and W. Ye. Energy-driven integrated hardware-software optimizations using simplepower. In Computer Architecture, 2000. Proceedings of the 27th International Symposium on, pages 95–106, June 2000. Google ScholarDigital Library
- Michael E Wolf and Monica S Lam. A data locality optimizing algorithm. In ACM Sigplan Notices, volume 26, pages 30–44. ACM, 1991. Google ScholarDigital Library
- Shin Yoo and Mark Harman. Regression testing minimization, selection and prioritization: a survey. 22(2):67–120, 2012. Google ScholarDigital Library
Index Terms
- Test case permutation to improve execution time
Recommendations
Improving test execution time with improved cache locality
ICSE-C '17: Proceedings of the 39th International Conference on Software Engineering CompanionAs software systems become more complex, the number of test cases required for effective testing becomes intractable. Furthermore, modern agile development methodologies [2] encourage the execution of test suites as often as possible in an attempt to ...
STREX: boosting instruction cache reuse in OLTP workloads through stratified transaction execution
ISCA '13: Proceedings of the 40th Annual International Symposium on Computer ArchitectureOnline transaction processing (OLTP) workload performance suffers from instruction stalls; the instruction footprint of a typical transaction exceeds by far the capacity of an L1 cache, leading to ongoing cache thrashing. Several proposed techniques ...
Counter-Based Cache Replacement and Bypassing Algorithms
Recent studies have shown that in highly associative caches, the performance gap between the Least Recently Used (LRU) and the theoretical optimal replacement algorithms is large, motivating the design of alternative replacement algorithms to improve ...
Comments