Minimum/maximum delay testing of product lines with unbounded parametric real-time constraints

Highlights

• Model for product lines with unbounded configurable real-time constraints.

• Family-based test-suite generation for covering infinite configuration spaces.

• Novel coverage criterion for best-case/worst-case execution times.

• Case studies and experimental evaluation results show applicability.

Abstract

Non-functional requirements like real-time behaviors are of ever-growing interest in many application domains of software product lines. Consequently, existing modeling formalisms and analysis techniques for reasoning about time-critical behaviors have to be adapted to product-line engineering, too. Featured timed automata (FTA) extend timed automata (TA) by feature constraints to enable efficient family-based verification of real-time properties. Here, we present configurable parametric timed automata (CoPTA) to further extend expressiveness of FTA by freely configurable and a-priori unbounded timing intervals of real-time constraints. Hence, CoPTA models impose infinite configuration spaces which makes variant-by-variant analysis practically infeasible. Instead, we present a family-based test-suite generation methodology for CoPTA models ensuring symbolical location coverage for every model configuration. Furthermore, we define a novel coverage criterion, called Minimum/Maximum Delay (M/MD) coverage, requiring every location in a CoPTA model to be reached by test cases with minimum/maximum possible durations, for systematically investigating best-case/worst-case execution times. We extend our family-based test-suite generation methodology to also achieve M/MD coverage on CoPTA models. Our evaluation results, obtained from applying our CoPTA tool to a collection of subject systems, reveal efficiency improvements of family-based test-suite generation, as compared to a variant-by-variant strategy in case of finite configuration spaces.

Introduction

Many promising formalisms and corresponding quality-assurance methodologies have been proposed in the recent past for specifying and efficiently analyzing functional and non-functional properties of families of similar (software) systems (so-called software product lines Schaefer and Hähnle, 2011). Most of these approach extend existing formalisms by adding constructs for modeling behavioral variability (Benduhn et al., 2015). In addition, a majority of these works employ the notion of feature to constitute explicitly configurable system parameters which—either implicitly or explicitly—occur as explicit syntactic entities in specification- and implementation-artifacts of product-line representations. Hence, features can be used to control automated assembling of product variants for a given product configuration (i.e., a valid selection of features). Correspondingly, product-line analysis strategies address the challenge of ensuring—at least up to a reasonable degree—correctness properties for every valid product variant with respect to a given specification of the product line under consideration (Thüm et al., 2014). In particular, performing variant-based analyses runs for every valid product configuration one-by-one in separate is infeasible for realistic product lines due to the exponential growth of the possible number of configurations in the number of features. To tackle this challenge, effective product-line analysis strategies are either based on sampling (i.e., defining feature-oriented criteria for selecting a small, yet considerably sufficient subset of variants on which out-of-the-box quality-assurance tools can be applied as usual), or they pursue family-based analysis (i.e., lifting established quality-assurance techniques to become variability-aware such that they are applicable to an integrated representation of the whole product family in a single analysis run).

However, in application contexts of product lines with real-time critical behaviors, we can observe that very little support for variability modeling and analysis exists so far. Here, featured timed automata (FTA) constitute one of the most promising approaches until now (Cordy et al., 2012). FTA are an extension of timed automata (TA) (Alur, Dill, 1990, Alur, Dill, 1994, Waez, Dingel, Rudie, 2013), a well-established modeling language for software systems with discrete event/continuous time behaviors. In particular, FTA extend TA by annotation-based variability-modeling (Czarnecki and Antkiewicz, 2005) and (symbolic) family-based product-line model-checking (Classen et al., 2011), based on an underlying featured transition systems (FTS) semantics (Classen et al., 2010).

A TA is defined as a state-transition graph as usual, where states are called locations and transitions between locations are called switches. In addition, a TA consists of a set of numeric and linearly increasing variables representing synchronously elapsing, yet independently resettable clocks. Critical timing behaviors on TA runs can be enforced using clock constraints, denoted as linear inequalities over clock values expressing time intervals with constant lower and upper bounds. Based on these clock constraints, TA models may incorporate guards for switches (i.e., time intervals in which a switch is allowed to be taken), and invariants for locations (i.e., time intervals in which it is allowed to reside in a location). FTA extend TA by the possibility to annotate both kinds of clock constraints as well as entire switches with feature constraints. In this way, feature constraints denote presence conditions for relating annotated modeling entities to respective product configurations. This additional variability information in FTA models facilitates family-based model-checking tools to efficiently verify real-time properties for whole product lines in a single analysis run (Cordy et al., 2013a).

However, the modeling capabilities of FTA for expressing variable real-time modeling are, by definition, limited to Boolean feature constraints by means of configuration-specific conditions over the presence of particular guards, invariants and entire switches within product variants. Nevertheless, behavioral variability in time-critical systems may further comprise configuration-specific bounds of time intervals in product variants. If the number of possible time intervals is finite, this kind of variability can be encoded into FTA, for instance, by enumerating all configuration-specific intervals, respectively. In contrast, freely adjustable and therefore a-priori unbounded (or, open) time intervals are neither directly expressible nor encodable in FTA. This is due to the inherent limitation of FTA to product lines with Boolean variability thus only supporting finite configuration spaces with a finite number of different TA model variants.

To this end, we recently proposed Configurable Parametric Timed Automata (CoPTA) as a new modeling formalism solving the aforementioned limitations of FTA (Luthmann et al., 2017). In this way, CoPTA serve as a novel modeling foundation for behavioral variability of product lines with time-critical behaviors. In particular, CoPTA combine principles of FTA with the those of parametric timed automata (PTA), a generalization of TA by parametric instead of constant lower and upper bounds of clock constraints (Henzinger et al., 1994). CoPTA models further employ FODA (Kang et al., 1990) feature diagrams extended with non-Boolean feature attributes and respective constraints to serve as configuration models for both Boolean as well as numeric variability. In this way, CoPTA models support parametric variability in terms of freely configurable numeric values for initially open bounds of configuration-specific time intervals thus comprising a (potentially) infinite number of different TA model variants. Hence, in contrast to FTA, the resulting infinite configuration spaces of CoPTA models make variant-by-variant analysis strategies practically impossible. Instead, we adapt a family-based test-suite generation methodology (Bürdek et al., 2015) to CoPTA models, ensuring location coverage on every model configuration. To this end, we enable incremental reuse of (symbolic) location-reachability information among configurations by utilizing the language-preservation theory which has been initially developed for PTA models (André and Markey, 2015).

Based on those techniques which we have initially presented in a previous work (Luthmann et al., 2017), we now further define an entirely novel—and more meaningful—coverage criterion on CoPTA models beyond location coverage, called Minimum/Maximum Delay (M/MD) coverage, for investigating more effectively time-critical behaviors of product lines with configurable real-time constraints. To this end, M/MD coverage requires every location in a CoPTA model to be reached by at least one test case on those configurations having minimum/maximum possible durations with respect to the given parametric real-time constraints (Courcoubetis and Yannakakis, 1992). In this way, M/MD coverage systematically identifies and stimulates best-case/worst-case execution-time behaviors on whole product lines which is widely considered particularly crucial in the context of real-time critical systems (Wilhelm et al., 2008). We extend the family-based test-suite generation methodology, accordingly, to ensure M/MD coverage on a given CoPTA model. This is achieved by a novel semantic-preserving augmentation of the given CoPTA model by means of an additional parametric clock for measuring the duration of a test-case run, respectively, during reachability analysis of a particular location. Based on the information obtained from this augmentation, we are able to apply an ILP-solver to derive precise minimizing/maximizing delay values for reaching those locations. By utilizing the inverse method of PTA, we are able to derive corresponding configuration constraints for runs having those minimum/maximum delay thus serving as test configurations for M/MD test coverage.

As a new challenge, the increased expressive power of CoPTA models as compared to FTA causes almost all non-trivial decision problems to become undecidable for PTA and thus for CoPTA (including the problem of reachability of locations being essential for automated test-case generation) (André, 2016). However, experiences gained from conducting our experimental evaluation performed on a collection of case studies indicate that problematic cases barely occur in practice. In addition, we revisit non-trivial sub-classes of PTA from the literature with (semi-)decidable properties, yet being effectively more expressive than FTA (André, 2016).

The contents of this article can be summarized as follows.

• Modeling formalism CoPTA enabling freely configurable real-time behaviors of product lines, thus improving expressiveness of existing approaches by facilitating infinite configuration spaces.

• Family-based test-suite generation methodology for CoPTA models for efficient test coverage of time-critical product lines.

• Novel coverage criterion M/MD (Minimum/Maximum Delay)and a corresponding family-based test-suite generation technique for systematically deriving test cases for identifying and investigating best-case/worst-case execution times of whole product lines.

• Tool support for automated family-based test-suite generation ensuring location coverage as well as M/MD coverage on every product variant based on the PTA analysis tool IMITATOR (André et al., 2012).

• Case studies demonstrating applicability and usefulness of the CoPTA language for specifying and testing product lines with real-time critical behaviors.

• Experimental evaluation results showing feasibility of family-based test-suite generation for location coverage and M/MD coverage on CoPTA models, by handling potentially infinite configuration spaces. Furthermore, in case of finite configuration spaces, our results reveal considerable efficiency improvements using family-based test-suite generation, as compared to a variant-by-variant approach.

The modeling formalism CoPTA, family-based test-suite generation technique for simple location coverage of CoPTA models as well as accompanying tool support and an initial experimental evaluation of this technique have been already presented in our previous work in Luthmann et al. (2017). In contrast, all parts about M/MD analysis augmentation, family-based test-suite generation for M/MD coverage as well as accompanying tool support, and the additional experiments for evaluating M/MD testing are original and novel contributions not yet published elsewhere.

The remainder of this article is organized as follows. In Section 2, we first describe preliminary notions and concepts of TA and FTA using a running example. In Section 3, we introduce our novel modeling formalism CoPTA, followed by Section 4, where we describe our family-based test-suite generation approach for CoPTA models with respect to location coverage and the novel M/MD coverage criterion. In Section 5, we describe details about our tool implementation as well as experimental evaluation results gained from several case studies adapted from recent literature on TA analysis. Finally, in Section 6, we describe related work on real-time modeling and analysis for product lines and in Section 7, we conclude and give a brief outlook on future work.

We provide our tool, the collection of CoPTA models used in our experiments as well as all experimental data on a supplementary Web page: www.es.tu-darmstadt.de/copta-analysis/.