MAESTRO: Automated test generation framework for high test coverage and reduced human effort in automotive industry

Abstract

Context

The importance of automotive software has been rapidly increasing because software controls many components of motor vehicles such as smart-key system, tire pressure monitoring system, and advanced driver assistance system. Consequently, the automotive industry spends a large amount of human effort to test automotive software and is interested in automated testing techniques to ensure high-quality automotive software with reduced human effort.

Objective

Applying automated test generation techniques to automotive software is technically challenging because of false alarms caused by imprecise test drivers/stubs and lack of tool supports for symbolic analysis of bit-fields and function pointers in C. To address such challenges, we have developed an automated testing framework MAESTRO.

Method

MAESTRO automatically builds a test driver and stubs for a target task (i.e., a software unit consisting of target functions). Then, it generates test inputs to a target task with the test driver and stubs by applying concolic testing and fuzzing together in an adaptive way. In addition, MAESTRO transforms a target program that uses bit-fields into a semantically equivalent one that does not use bit-fields. Also, MAESTRO supports symbolic function pointers by identifying the candidate functions of a symbolic function pointer through static analysis.

Results

MAESTRO achieved 94.2% branch coverage and 82.3% MC/DC coverage on the four target modules (238 KLOC) developed by Hyundai Mobis. Furthermore, it significantly reduced the cost of coverage testing by reducing the manual effort for coverage testing by 58.8%.

Conclusion

By applying automated testing techniques, MAESTRO can achieve high test coverage for automotive software with significantly reduced manual testing effort.

Introduction

The automotive industry has developed automotive software to control various components in the motor vehicle, such as the body control module (BCM), smart-key system (SMK), and tire pressure monitoring system (TPMS) [1], [2]. As automotive software becomes larger and more complex with the addition of newly introduced automated features (e.g., automatic parking system (APRK) of advanced driver assistance system (ADAS)) and more sophisticated functionality (e.g., driving mode systems) [3], [4], the cost of testing automotive software is rapidly increasing. Also, it is difficult for human engineers to develop test inputs that can ensure high-quality automotive software within tight software development schedules and budgets. To resolve these problems, the automotive industry is trying to apply automated software testing/verification techniques [5], [6], [7], [8] to achieve high code quality with reduced human effort.

Concolic testing [9], [10] has been applied to automatically generate test inputs for software in various industries. Concolic testing combines dynamic concrete execution and static symbolic execution to explore all possible execution paths of a target program, which can achieve high code coverage. Concolic testing has been applied to various industrial projects (e.g., flash memory device driver [11], mobile phone software [12], [13], and large-scale embedded software [14]) and has effectively improved the quality of industrial software by increasing test coverage and detecting corner-case bugs with modest human effort. Also, fuzzing is starting to show its potential as a general automated test input generation technique, like concolic testing, although it had been originally developed to reveal security vulnerabilities of target systems.

While we were working to apply automated test generation techniques to automotive software developed by Mobis, we observed the following technical challenges that need to be resolved to successfully apply automated test generation techniques:

• We need to generate test drivers and stubs carefully to achieve high unit test coverage while avoiding generating test cases corresponding to the executions that are not feasible at the system-level. Otherwise (e.g., generating naive test drivers and stubs that provide unconstrained symbolic inputs to every function in a target program), we will waste human effort to manually filter out infeasible tests that lead to misleading high coverage and false alarms.

• Current concolic testing tools do not support symbolic bit-fields in C which are frequently used for automotive software.¹For example, automotive software uses bit-fields in message packets in the controller area network (CAN) bus to save memory and bus bandwidth. However, most concolic testing tools do not support symbolic bit-fields since a bit-field does not have a memory address (Section 3.4) and most programs running on PCs rarely use bit-fields.

• Although automotive software uses function pointers to simplify code to dynamically select a function to execute, current automatic test generation techniques and tools do not support symbolic setting for function pointers due to the limitation of SMT (Satisfiability Modulo Theories) solvers of concolic testing and input mutation technique of fuzzing.

To address the above challenges, we have developed an automated testing framework MAESTRO(Mobis Automated tESTing fRamewOrk). MAESTRO automatically generates the test driver, stubs, and test inputs for a target unit using concolic testing and fuzzing. MAESTRO achieved 94.2% branch coverage and 82.3% MC/DC coverage of the modules of the ADAS (advanced driver assistance system) and IBU (integrated body unit) software (Section 5.1). Also, MAESTRO reduced the manual testing effort by 58.8% for coverage testing of the target modules of ADAS and IBU (Section 5.3).²

The main contributions of this paper are as follows:

• We have developed MAESTRO which automatically generates the test driver, stubs, and test inputs achieving high coverage for automotive software. MAESTRO applies concolic testing and fuzzing together in an adaptive way (Section 3.5.4) to achieve high coverage.

• We have identified the technical challenges in applications of automated test input generation to automotive software and describe how MAESTRO resolves them (i.e., task-oriented driver/stub generation (Section 3.3.1), symbolic bit-field support (Section 3.4), and symbolic setting for function pointers (Section 3.3.4)). Thus, this paper can support field engineers in the automotive industry to adopt automated test generation with less trial-and-error.

• To the authors’ best knowledge, this is the first industrial study that concretely demonstrates reduced human effort (i.e., human effort reduced by 58.8%) by applying concolic testing and fuzzing together in the automotive industry (Section 5.3). Thus, this study can promote the adoption of concolic testing and fuzzing in the automotive industry.

• This paper shares lessons learned and valuable information for both field engineers in the automotive industry and researchers who develop automated testing techniques (Section 6). For example, we have found that the generation of precise test drivers and stubs is important to increase test coverage (Section 6.3) and concolic testing and fuzzing have different characteristics to achieve test coverage (Section 6.4).

This journal article is an extended version of our prior automated testing framework MAIST [18] as follows:

• We have extended MAIST [18] to MAESTRO to achieve higher test coverage as follows:

  • MAESTRO uses a hybrid technique of concolic testing and fuzzing as test input generators (Sections 3.5.3 and 3.5.4). The experiment results show that the hybrid approach achieved higher branch and MC/DC coverage than concolic testing or fuzzing alone (Section 5.7). Also, we have discussed the different characteristics of concolic testing and fuzzing (Section 6.4).

  • MAESTRO extended MAIST by generating symbolic stubs that provide more realistic contexts to a target code unit (Section 3.3.3). The experiment results show that MAESTRO’s new symbolic stub increases branch and MC/DC coverage (Section 5.8).

• We have targeted a new module (advanced driver assistance system’s automatic parking (APRK)) as well as the ones in the prior work [18]. APRK is a crucial component for safety and it is highly complex, handling multiple sensors and actuators (Sections 2.1 and 2.2).

• With the new adaptive hybrid test input generator and the precise symbolic stub generation, MAESTRO achieved 94.2% branch coverage and 82.3% MC/DC coverage for the four target modules. Compared to MAIST, MAESTRO improved 4.1% branch and 5.8% MC/DC coverage.

• We have added new sections, Sections 3.4.1–3.4.3, to describe MAESTRO’s bit-field transformation algorithms and related examples.

The rest of the paper is organized as follows. Section 2 explains the target project. Section 3 describes the MAESTRO framework. Section 4 explains how we have applied MAESTRO to the target modules. Section 5 describes the experiment results. Section 6 presents lessons learned from this industrial study. Section 7 discusses related work. Finally, Section 8 concludes this paper with future work.