ABSTRACT
Previous techniques on concurrency testing have mainly focused on exploring the interleaving space of manually written test code to expose faulty interleavings of shared memory accesses. These techniques assume the availability of failure-inducing tests. In this paper, we present AutoConTest, a coverage-driven approach to generate effective concurrent test code that achieve high interleaving coverage. AutoConTest consists of three components. First, it computes the coverage requirements dynamically and iteratively during sequential test code generation, using a coverage metric that captures the execution context of shared memory accesses. Second, it smartly selects these sequential codes based on the computed result and assembles them for concurrent tests, achieving increased context-sensitive interleaving coverage. Third, it explores the newly covered interleavings. We have implemented AutoConTest as an automated tool and evaluated it using 6 real-world concurrent Java subjects. The results show that AutoConTest is able to generate effective concurrent tests that achieve high interleaving coverage and expose concurrency faults quickly. AutoConTest took less than 65 seconds (including program analysis, test generation and execution) to expose the faults in the program subjects.
- ASM Java Bytecode Manipulation Framework. http://asm.objectweb.org/Google Scholar
- LANG-481. https://issues.apache.org/jira/browse/LANG-481Google Scholar
- Jfreechart-278. http://sourceforge.net/p/jfreechart/bugs/278/Google Scholar
- JDK-4334376. https://bugs.openjdk.java.net/browse/JDK-4334376Google Scholar
- JDK-4791557. http://bugs.java.com/bugdatabase/view_bug.do?bug_id=4791557Google Scholar
- Guava-339. https://code.google.com/p/guava-libraries/issues/detail?id=339Google Scholar
- JDK-4779253 http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4779253Google Scholar
- ConTeGen. http://thread-safe.org/downloadGoogle Scholar
- XStream framework. http://xstream.codehaus.org/Google Scholar
- S. Artzi, M. D. Ernst, A. Kiezun, C. Pacheco, and J. H. Perkins. Finding the Needles in the Haystack: Generating Legal Test Inputs for Object-oriented Programs. MIT-CSAIL-TR-2006-056, 2006.Google Scholar
- C. Boyapati, S. Khurshid, and D. Marinov. Korat: Automated Testing Based on Java Predicates. In ISSTA, pages 123--133, 2002. Google ScholarDigital Library
- S. Burckhardt, C. Dern, M. Musuvathi, and R. Tan. Line-up: A Complete and Automatic Linearizability Checker. In PLDI, pages 330--340, 2010. Google ScholarDigital Library
- A. Carzaniga, A. Goffi, A. Gorla, A. Mattavelli, and M. Pezzè. Cross-checking Oracles from Intrinsic Software Redundancy. In ICSE, pages 931--942, 2014. Google ScholarDigital Library
- C. Csallner and Y. Smaragdakis. Jcrasher: An Automatic Robustness Tester for Java. Software Pract and Exper, 34(11):1025--1050, 2004. Google ScholarDigital Library
- A. Farzan, A. Holzer, N. Razavi, and H. Veith. Con2colic Testing. In FSE, pages 37--47, 2013. Google ScholarDigital Library
- C. Flanagan and S. N. Freund. Atomizer: A Dynamic Atomicity Checker for Multithreaded Programs. In POPL, pages 256--267, 2004. Google ScholarDigital Library
- C. Flanagan and S. N. Freund. Fasttrack: Efficient and Precise Dynamic Race Detection. In PLDI, pages 121--133, 2009. Google ScholarDigital Library
- G. Fraser and A. Arcuri. Whole Test Suite Generation. IEEE TSE, 39(2):276--291, 2013. Google ScholarDigital Library
- G. Fraser, M. Staats, P. McMinn, A. Arcuri, and F. Padberg. Does Automated White-box Test Generation Really Help Software Testers? In ISSTA, pages 291--301, 2013. Google ScholarDigital Library
- G. Fraser and F. Wotawa. Redundancy Based Test-suite Reduction. In FASE, pages 291--305, 2007. Google ScholarDigital Library
- G. Fraser and A. Arcuri. Evosuite: Automatic Test Suite Generation for Object-oriented Software. In FSE, pages 416--419, 2011. Google ScholarDigital Library
- P. Godefroid, N. Klarlund, and K. Sen. DART: Directed Automated Random Testing. In PLDI, pages 213--223, 2005. Google ScholarDigital Library
- B. Göetz, T. Peierls, J. Bloch, J. Bowbeer, D. Holmes, and D. Lea. Java Concurrency in Practice. Addison-Wesley, 2006. Google ScholarDigital Library
- M. Harder, J. Mellen, and M. D. Ernst. Improving Test Suites via Operational Abstraction. In ICSE, pages 60--71, 2003. Google ScholarDigital Library
- S. Hong, J. Ahn, S. Park, M. Kim, and M. J. Harrold. Testing Concurrent Programs to Achieve High Synchronization Coverage. In ISSTA, pages, 2012. Google ScholarDigital Library
- J. Huang and C. Zhang. Persuasive Prediction of Concurrency Access Anomalies. In ISSTA, pages 144--154, 2011. Google ScholarDigital Library
- J. C. King. Symbolic Execution and Program Testing. CACM, 19(7):385--394, 1976. Google ScholarDigital Library
- Z. Lai, S. C. Cheung, and W. K. Chan. Detecting Atomic-set Serializability Violations in Multithreaded Programs Through Active Randomized Testing. In ICSE, pages 235--244, 2010. Google ScholarDigital Library
- Z. Li, L. Tan, X. Wang, S. Lu, Y. Zhou, and C. Zhai. Have Things Changed Now?: An Empirical Study of Bug Characteristics in Modern Open Source Software. In ASID, pages 25--33, 2006. Google ScholarDigital Library
- Y. Lin and D. Dig. Check-then-act Misuse of Java Concurrent Collections. In ICST, pages 164--173, 2013. Google ScholarDigital Library
- P. Liu, J. Dolby, and C. Zhang. Finding Incorrect Compositions of Atomicity. In FSE, pages 158--168, 2013. Google ScholarDigital Library
- S. Lu, W. Jiang, and Y. Zhou. A Study of Interleaving Coverage Criteria. In FSE, pages 533--536, 2007. Google ScholarDigital Library
- S. Lu, S. Park, E. Seo, and Y. Zhou. Learning from Mistakes: A Comprehensive Study on Real World Concurrency Bug Characteristics. In ASPLOS, pages 329--339, 2008. Google ScholarDigital Library
- S. Lu, S. Park, and Y. Zhou. Finding Atomicity-violation Bugs Through Unserializable Interleaving Testing. IEEE TSE, 38(4):844--860, 2012. Google ScholarDigital Library
- R. Majumdar and K. Sen. Hybrid Concolic Testing. In ICSE, pages 416--426, 2007. Google ScholarDigital Library
- D. Marinov and S. Khurshid. Testera: A Novel Framework for Automated Testing of Java Programs. In ASE, pages 22--31, 2001. Google ScholarDigital Library
- M. Musuvathi, S. Qadeer, T. Ball, G. Basler, P. A. Nainar, and I. Neamtiu. Finding and Reproducing Heisenbugs in Concurrent Programs. In OSDI, pages 267--280, 2008. Google ScholarDigital Library
- A. Nistor, Q. Luo, M. Pradel, T. R. Gross, and D. Marinov. BALLERINA: Automatic Generation and Clustering of Efficient Random Unit Tests for Multithreaded Code. In ICSE, pages 727--737, 2012. Google ScholarDigital Library
- S. Okur and D. Dig. How Do Developers Use Parallel Libraries? In FSE, pages 54:1--54:11, 2012. Google ScholarDigital Library
- C. Pacheco, S. Lahiri, M. Ernst, and T. Ball. Feedback-directed Random Test Generation. In ICSE, pages 75--84, 2007. Google ScholarDigital Library
- S. Park, S. Lu, and Y. Zhou. CTrigger: Exposing Atomicity Violation Bugs from Their Hiding Places. In ASPLOS, pages 25--36, 2009. Google ScholarDigital Library
- F. Pastore, L. Mariani, and G. Fraser. Crowdoracles: Can the Crowd Solve the Oracle Problem? In ICST, pages 342--351, 2013. Google ScholarDigital Library
- M. Pradel and T. R. Gross. Fully Automatic and Precise Detection of Thread Safety Violations. In PLDI, pages 521--530, 2012. Google ScholarDigital Library
- G. Ramalingam. Context-sensitive Synchronization-sensitive Analysis Is Undecidable. TOPLAS, 22(2):416--430, 2000. Google ScholarDigital Library
- N. Razavi, F. Ivančić, V. Kahlon, and A. Gupta. Concurrent Test Generation Using Concolic Multi-trace Analysis. In APLAS, pages 239--255, 2012.Google ScholarCross Ref
- M. Samak and M. K. Ramanathan. Multithreaded Test Synthesis for Deadlock Detection. In OOPSLA, pages 473--489, 2014. Google ScholarDigital Library
- M. Samak and M. K. Ramanathan. Omen+: A Precise Dynamic Deadlock Detector for Multithreaded Java Libraries. In FSE, pages 735--738, 2014. Google ScholarDigital Library
- M. Samak and M. K. Ramanathan. Omen: A Tool for Synthesizing Tests for Deadlock Detection. In SPLASH, pages 37--38, 2014. Google ScholarDigital Library
- M. Samak, M. K. Ramanathan, and S. Jagannathan. Synthesizing Racy Tests. In PLDI, pages 175--185, 2015. Google ScholarDigital Library
- S. Savage, M. Burrows, G. Nelson, P. Sobalvarro, and T. Anderson. Eraser: A Dynamic Data Race Detector for Multithreaded Programs. ACM Trans. Comput. Syst., 15(4):391--411, 1997. Google ScholarDigital Library
- K. Sen and G. Agha. Cute and Jcute: Concolic Unit Testing and Explicit Path Model-checking Tools. In CAV, pages 419--423, 2006. Google ScholarDigital Library
- K. Sen and G. Agha. A Race-detection and Flipping Algorithm for Automated Testing of Multi-threaded Programs. In HVC, pages 166--182, 2006. Google ScholarDigital Library
- K. Sen, G. Roşu, and G. Agha. Detecting Errors in Multithreaded Programs by Generalized Predictive Analysis of Executions. In FMOODS, pages 211--226, 2005. Google ScholarDigital Library
- E. Sherman, M. B. Dwyer, and S. Elbaum. Saturation-based Testing of Concurrent Programs. In FSE, pages 53--62, 2009. Google ScholarDigital Library
- F. Sorrentino, A. Farzan, and P. Madhusudan. PENELOPE: Weaving Threads to Expose Atomicity violations. In FSE, pages 37--46, 2010. Google ScholarDigital Library
- S. Steenbuck and G. Fraser. Generating Unit Tests for Concurrent Classes. In ICST, pages 144--153, 2013. Google ScholarDigital Library
- L. Tan, C. Liu, Z. Li, X. Wang, Y. Zhou, and C. Zhai. Bug Characteristics in Open Source Software. ESE, 19(6):1665--1705, 2014. Google ScholarDigital Library
- S. Tasiran, M. E. Keremouglu, and K. Mucslu. Location Pairs: A Test Coverage Metric for Shared-memory Concurrent Programs. ESE, 17(3):129--165, 2012. Google ScholarDigital Library
- N. Tillmann and J. De Halleux. Pex--white Box Test Generation for Net. In TAP, pages 134--153, 2008. Google ScholarDigital Library
- P. Tonella. Evolutionary Testing of Classes. In ISSTA, pages 119--128, 2004. Google ScholarDigital Library
- M. Vaziri, F. Tip, and J. Dolby. Associating Synchronization Constraints with Data in an Object-oriented Language. In POPL, pages 334--345, 2006. Google ScholarDigital Library
- W. Visser, C. S. Păsăreanu, and R. Pelánek. Test Input Generation for Java Containers Using State Matching. In ISSTA, pages 37--48, 2006. Google ScholarDigital Library
- C. Wang and M. Ganai. Predicting Concurrency Failures in the Generalized Execution Traces of x86 Executables. In RV, pages 4--18, 2012. Google ScholarDigital Library
- C. Wang, M. Said, and A. Gupta. Coverage Guided Systematic Concurrency Testing. In ICSE, pages 221--230, 2011. Google ScholarDigital Library
- T. Xie, D. Marinov, and D. Notkin. Rostra: A Framework for Detecting Redundant Object-oriented Unit Tests. In ASE, pages 196--205, 2004. Google ScholarDigital Library
- C.-S. D. Yang, A. L. Souter, and L. L. Pollock. All-du-path Coverage for Parallel Programs. In ISSTA, pages 153--162, 1998. Google ScholarDigital Library
- J. Yu, S. Narayanasamy, C. Pereira, and G. Pokam. Maple: A Coverage-driven Testing Tool for Multithreaded Programs. In OOPSLA, pages 485--502, 2012. Google ScholarDigital Library
- T. Yu, W. Srisa-an, and G. Rothermel. An Empirical Comparison of the Fault-detection Capabilities of Internal Oracles. In ISSRE, pages 11--20, 2013.Google ScholarCross Ref
Index Terms
- Coverage-driven test code generation for concurrent classes
Recommendations
Code Coverage Aware Test Generation Using Constraint Solver
Software Engineering and Formal Methods. SEFM 2020 Collocated WorkshopsAbstractCode coverage has been used in the software testing context mostly as a metric to assess a generated test suite’s quality. Recently, code coverage analysis is used as a white-box testing technique for test optimization. Most of the research ...
The Effectiveness of T-Way Test Data Generation
SAFECOMP '08: Proceedings of the 27th international conference on Computer Safety, Reliability, and SecurityThis paper reports the results of a study comparing the effectiveness of automatically generated tests constructed using random and <em>t</em>-way combinatorial techniques on safety related industrial code using mutation adequacy criteria. A reference ...
A detailed investigation of the effectiveness of whole test suite generation
A common application of search-based software testing is to generate test cases for all goals defined by a coverage criterion (e.g., lines, branches, mutants). Rather than generating one test case at a time for each of these goals individually, whole ...
Comments