Test case comparison and clustering using program profiles and static execution

Selection of diverse test cases and elimination of duplicates are two major problems in product testing life cycle, especially in sustained engineering environment. In order to solve these, we introduce a framework of test case comparison metrics which will quantitatively describe the distance between any arbitrary test case pair of an existing test suite, allowing various test case analysis applications. We combine program profiles from test execution, static analysis and statistical techniques to capture various aspects of test execution and compute a specialized test case distance measurement. Using these distance metrics, we drive a customized hierarchical test suite clustering algorithm that groups similar test cases together. We present an industrial strength framework called SPIRiT that works at binary level, implementing different metrics in the form of coverage, control, data, def-use, temporal variances and does test case clustering. This is step towards integrating runtime analysis, static analysis, statistical techniques and machine learning to drive new generation of test suite analysis algorithms.