Skip to main content
SpringerLink
Log in

Sampling strategies for product lines with unbounded parametric real-time constraints

  • Foundations for Mastering Change
  • Quantitative Variability Modelling and Analysis
  • Published:
International Journal on Software Tools for Technology Transfer Aims and scope Submit manuscript

Abstract

Combinatorial interaction testing (CIT) has been successfully applied to product-line testing for selecting from a usually very large configuration space a relatively small sample of test configurations sufficiently covering critical combinations of configuration options. As most recent CIT techniques like pairwise sampling are limited to finite configuration spaces (i.e., vectors of yes/no options), they are not directly applicable to configuration parameters with a priori unbounded value domains (e.g., for adjusting unlimited resources or non-functional properties). Applying existing sampling strategies to infinite configuration spaces therefore requires further heuristics for selecting a finite subset of parameter-value combinations to be covered. Nevertheless, applying purely black-box heuristics may produce inherently ineffective test suites in which particularly critical parameter-value combinations are missed. In order to tackle this problem, we present a novel methodology for effectively sampling product lines with infinite configuration spaces by means of freely configurable real-time behaviors. To this end, we employ solution-space information obtained from our new modeling formalism, configurable parametric timed automata, to generate samples for covering critical best-case/worst-case execution-time behaviors. We also present a tool implementation which we applied to a collection of subject systems to demonstrate the applicability of our approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Abdeddaim, Y., Asarin, E., Maler, O.: Scheduling with timed automata. Theor. Comput. Sci. 354(2), 272–300 (2006). https://doi.org/10.1016/j.tcs.2005.11.018

    Article  MathSciNet  MATH  Google Scholar 

  2. Al-Hajjaji, M., Meinicke, J., Krieter, S., Schröter, R., Thüm, T., Leich, T., Saake, G.: Tool Demo: testing configurable systems with FeatureIDE. In: Proceedings of the 2016 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences, GPCE 2016, pp. 173–177. ACM (2016). https://doi.org/10.1145/2993236.2993254

  3. Alur, R., Dill, D.: Automata for modeling real-time systems. In: Paterson, M.S. (ed.) Automata, Languages and Programming, pp. 322–335. Springer, Berlin (1990). https://doi.org/10.1007/BFb0032042

    Chapter  MATH  Google Scholar 

  4. Alur, R., Henzinger, T.A., Vardi, M.Y.: Parametric real-time reasoning. In: Proceedings of the Twenty-fifth Annual ACM Symposium on Theory of Computing, pp. 592–601. ACM (1993). https://doi.org/10.1145/167088.167242

  5. Alur, R., La Torre, S., Pappas, G.J.: Optimal Paths in Weighted Timed Automata, pp. 49–62. Springer, Berlin (2001). https://doi.org/10.1007/3-540-45351-2_8

    Book  MATH  Google Scholar 

  6. André, É.: IMITATOR: A Tool for Synthesizing Constraints on Timing Bounds of Timed Automata, pp. 336–342. Springer, Berlin (2009). https://doi.org/10.1007/978-3-642-03466-4_22

    Book  Google Scholar 

  7. André, É.: What’s Decidable About Parametric Timed Automata?, pp. 52–68. Springer, Berlin (2016). https://doi.org/10.1007/978-3-319-29510-7_3

    Book  MATH  Google Scholar 

  8. André, É., Chatain, T., Fribourg, L., Encrenaz, E.: An inverse method for parametric timed automata. Int. J. Found. Comput. Sci. 20(5), 819–836 (2009). https://doi.org/10.1142/S0129054109006905

    Article  MathSciNet  MATH  Google Scholar 

  9. André, É., Fribourg, L., Kühne, U., Soulat, R.: IMITATOR 2.5: a tool for analyzing robustness in scheduling problems. In: Giannakopoulou, D., Méry, D. (eds.) FM’12, Lecture Notes in Computer Science, vol. 7436, pp. 33–36. Springer, Berlin (2012). https://doi.org/10.1007/978-3-642-32759-9_6

    Chapter  Google Scholar 

  10. André, É., Lime, D., Roux, O.H.: Decision Problems for Parametric Timed Automata, pp. 400–416. Springer, Berlin (2016). https://doi.org/10.1007/978-3-319-47846-3_25

    Book  Google Scholar 

  11. André, É., Markey, N.: Language Preservation Problems in Parametric Timed Automata, pp. 27–43. Springer, Berlin (2015). https://doi.org/10.1007/978-3-319-22975-1_3

    Book  MATH  Google Scholar 

  12. Bagnara, R., Hill, P.M., Zaffanella, E.: The parma polyhedra library: toward a complete set of numerical abstractions for the analysis and verification of hardware and software systems. Sci. Comput. Program. 72(1), 3–21 (2008). https://doi.org/10.1016/j.scico.2007.08.001

    Article  MathSciNet  Google Scholar 

  13. Baller, H., Lity, S., Lochau, M., Schaefer, I.: Multi-objective test suite optimization for incremental product family testing. In: IEEE Seventh International Conference on Software Testing, Verification and Validation, pp. 303–312 (2014). https://doi.org/10.1109/ICST.2014.43

  14. Behrmann, G., Fehnker, A., Hune, T., Larsen, K., Pettersson, P., Romijn, J., Vaandrager, F.: Minimum-cost reachability for priced timed automata. In: HSCC, vol. 1, pp. 147–161. Springer (2001)

  15. Behrmann, G., Larsen, K.G., Rasmussen, J.I.: Optimal scheduling using priced timed automata. SIGMETRICS Perform. Eval. Rev. 32(4), 34–40 (2005). https://doi.org/10.1145/1059816.1059823

    Article  Google Scholar 

  16. Bertrand, N., Pinchinat, S., Raclet, J.B.: Refinement and Consistency of Timed Modal Specifications, pp. 152–163. Springer, Berlin (2009). https://doi.org/10.1007/978-3-642-00982-2

    Book  MATH  Google Scholar 

  17. Bürdek, J., Lochau, M., Bauregger, S., Holzer, A., von Rhein, A., Apel, S., Beyer, D.: Facilitating reuse in multi-goal test-suite generation for software product lines. In: Egyed, A., Schaefer, I. (eds.) Fundamental Approaches to Software Engineering, pp. 84–99. Springer, Berlin (2015). https://doi.org/10.1007/978-3-662-46675-9_6

    Chapter  Google Scholar 

  18. Cmyrev, A., Reissing, R.: Efficient and effective testing of automotive software product lines. Appl. Sci. Eng. Prog. 7(2), 53–57 (2014). https://doi.org/10.14416/j.ijast.2014.05.001

    Article  Google Scholar 

  19. Cohen, D.M., Dalal, S.R., Fredman, M.L., Patton, G.C.: The AETG system: an approach to testing based on combinatorial design. IEEE Trans. Softw. Eng. 23(7), 437–444 (1997). https://doi.org/10.1109/32.605761

    Article  Google Scholar 

  20. Cordy, M., Classen, A., Heymans, P., Schobbens, P.Y., Legay, A.: Provelines: a product line of verifiers for software product lines. In: SPLC’13 Workshops, pp. 141–146. ACM (2013). https://doi.org/10.1145/2499777.2499781

  21. Cordy, M., Legay, A., Schobbens, P.Y., Traonouez, L.M.: A framework for the rigorous design of highly adaptive timed systems. In: FormaliSE’13, pp. 64–70 (2013). https://doi.org/10.1109/FormaliSE.2013.6612279

  22. Cordy, M., Schobbens, P.Y., Heymans, P., Legay, A.: Behavioural modelling and verification of real-time software product lines. In: Proceedings of the 16th International Software Product Line Conference, pp. 66–75. ACM (2012). https://doi.org/10.1145/2362536.2362549

  23. Cordy, M., Willemart, M., Dawagne, B., Heymans, P., Schobbens, P.Y.: An extensible platform for product-line behavioural analysis. In: SPLC’14 Workshops, pp. 102–109. ACM (2014). https://doi.org/10.1145/2647908.2655973

  24. Courcoubetis, C., Yannakakis, M.: Minimum and maximum delay problems in real-time systems. Form. Methods Syst. Des. 1(4), 385–415 (1992). https://doi.org/10.1007/BF00709157

    Article  MATH  Google Scholar 

  25. Czarnecki, K., Antkiewicz, M.: Mapping features to models: a template approach based on superimposed variants. In: Glück, R., Lowry, M. (eds.) Generative Programming and Component Engineering, pp. 422–437. Springer, Berlin (2005). https://doi.org/10.1007/11561347_28

    Chapter  Google Scholar 

  26. Dick, J., Faivre, A.: Automating the generation and sequencing of test cases from model-based specifications. In: Woodcock, J.C.P., Larsen, P.G. (eds.) FME ’93: Industrial-Strength Formal Methods, pp. 268–284. Springer, Berlin (1993). https://doi.org/10.1007/BFb0024651

    Chapter  Google Scholar 

  27. Dill, D.L.: Timing assumptions and verification of finite-state concurrent systems. In: Sifakis, J. (ed.) Automatic Verification Methods for Finite State Systems, pp. 197–212. Springer, Berlin (1990)

    Chapter  Google Scholar 

  28. Haslinger, E.N., Lopez-Herrejon, R.E., Egyed, A.: Using feature model knowledge to speed up the generation of covering arrays. In: Proceedings of the Seventh International Workshop on Variability Modelling of Software-intensive Systems, VaMoS ’13, pp. 16:1–16:6. ACM (2013). https://doi.org/10.1145/2430502.2430524

  29. Havelund, K., Skou, A., Larsen, K.G., Lund, K.: Formal modeling and analysis of an audio/video protocol: an industrial case study using UPPAAL. In: RTSS’97, pp. 2–13 (1997). REAL.1997.641264

  30. Henzinger, T.A., Nicollin, X., Sifakis, J., Yovine, S.: Symbolic model checking for real-time systems. Inf. Comput. 111(2), 193–244 (1994). https://doi.org/10.1006/inco.1994.1045

    Article  MathSciNet  MATH  Google Scholar 

  31. Jensen, H.E., Larsen, K.G., Skou, A.: Modelling and analysis of a collision avoidance protocol using spin and UPPAAL. In: DIMACS’96 (1996)

  32. Johansen, M.F., Haugen, O., Fleurey, F.: An algorithm for generating t-wise covering arrays from large feature models. In: Proceedings of the 16th International Software Product Line Conference, SPLC ’12, Vol. 1, pp. 46–55. ACM (2012). https://doi.org/10.1145/2362536.2362547

  33. Johansen, M.F., Haugen, Ø., Fleurey, F., Eldegard, A.G., Syversen, T.: Generating better partial covering arrays by modeling weights on sub-product lines. In: France, R.B., Kazmeier, J., Breu, R., Atkinson, C. (eds.) Model Driven Engineering Languages and Systems, pp. 269–284. Springer, Berlin (2012)

    Chapter  Google Scholar 

  34. Kang, K.C., Cohen, S.G., Hess, J.A., Novak, W.E., Peterson, A.S.: Feature-oriented domain analysis (FODA) feasibility study. Technical Report, DTIC Document (1990)

  35. Karataş, A.S., Oğuztüzün, H., Doğru, A.: Mapping extended feature models to constraint logic programming over finite domains. In: Bosch, J., Lee, J. (eds.) Software Product Lines: Going Beyond, pp. 286–299. Springer, Berlin (2010). https://doi.org/10.1007/978-3-642-15579-6_20

    Chapter  Google Scholar 

  36. Kim, J.H., Legay, A., Traonouez, L.M., Acher, M., Kang, S.: A formal modeling and analysis framework for software product line of preemptive real-time systems. In: SAC’16, pp. 1562–1565. ACM (2016). https://doi.org/10.1145/2851613.2851977

  37. King, A., Sokolsky, O., Lee, I.: A modal specification theory for timing variability. Technical Report, Report No. MS-CIS-13-11. Department of Computer and Information Science Technical, University of Pennsylvania (2013)

  38. Kolesnikov, S.S., Apel, S., Siegmund, N., Sobernig, S., Kästner, C., Senkaya, S.: Predicting quality attributes of software product lines using software and network measures and sampling. In: Proceedings of the Seventh International Workshop on Variability Modelling of Software-intensive Systems, VaMoS ’13, pp. 6:1–6:5. ACM (2013). https://doi.org/10.1145/2430502.2430511

  39. Krenn, W., Ničković, D., Tec, L.: Incremental language inclusion checking for networks of timed automata. In: Braberman, V., Fribourg, L. (eds.) Formal Modeling and Analysis of Timed Systems, pp. 152–167. Springer, Berlin (2013)

    Chapter  Google Scholar 

  40. Leveson, N.G., Stolzy, J.L.: Analyzing safety and fault tolerance using time petri nets. In: Ehrig, H., Floyd, C., Nivat, M., Thatcher, J. (eds.) Formal Methods and Software Development, pp. 339–355. Springer, Berlin (1985). https://doi.org/10.1007/3-540-15199-0_22

    Chapter  Google Scholar 

  41. Lindahl, M., Pettersson, P., Yi, W.: Formal design and analysis of a gear controller. STTT 3(3), 353–368 (2001). https://doi.org/10.1007/BFb0054178

    Article  MATH  Google Scholar 

  42. Lochau, M., Peldszus, S., Kowal, M., Schaefer, I.: Model-Based Testing, pp. 310–342. Springer, Berlin (2014). https://doi.org/10.1007/978-3-319-07317-0_8

    Book  MATH  Google Scholar 

  43. Luthmann, L., Stephan, A., Bürdek, J., Lochau, M.: Modeling and testing product lines with unbounded parametric real-time constraints. In: Proceedings of the 21st International Systems and Software Product Line Conference, pp. 104–113. ACM (2017). https://doi.org/10.1145/3106195.3106204

  44. Mitsching, R., Weise, C., Kowalewski, S.: Design patterns for integrating variability in timed automata. In: SSIRI-C’10, pp. 38–45 (2010). https://doi.org/10.1109/SSIRI-C.2010.21

  45. de Moura, L., Bjørner, N.: Z3: an efficient SMT solver. In: Ramakrishnan, C.R., Rehof, J. (eds.) Tools and Algorithms for the Construction and Analysis of Systems, pp. 337–340. Springer, Berlin (2008)

    Chapter  Google Scholar 

  46. Nie, C., Leung, H.: A survey of combinatorial testing. ACM Comput. Surv. 43(2), 11:1–11:29 (2011). https://doi.org/10.1145/1883612.1883618

    Article  MATH  Google Scholar 

  47. Niebert, P., Tripakis, S., Yovine, S.: Minimum-time reachability for timed automata. In: IEEE Mediteranean Control Conference (2000)

  48. Oh, J., Batory, D., Myers, M., Siegmund, N.: Finding near-optimal configurations in product lines by random sampling. In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, pp. 61–71. ACM (2017). https://doi.org/10.1145/3106237.3106273

  49. Oster, S., Markert, F., Ritter, P.: Automated incremental pairwise testing of software product lines. In: Bosch, J., Lee, J. (eds.) Software Product Lines: Going Beyond, pp. 196–210. Springer, Berlin Heidelberg (2010)

    Chapter  Google Scholar 

  50. Panek, S., Stursberg, O., Engell, S.: Optimization of Timed Automata Models Using Mixed-Integer Programming, pp. 73–87. Springer, Berlin (2004). https://doi.org/10.1007/978-3-540-40903-8_7

    Book  MATH  Google Scholar 

  51. Perrouin, G., Sen, S., Klein, J., Baudry, B., le Traon, Y.: Automated and scalable t-wise test case generation strategies for software product lines. In: Third International Conference on Software Testing, Verification and Validation, pp. 459–468. IEEE (2010). https://doi.org/10.1109/ICST.2010.43

  52. Reuling, D., Bürdek, J., Rotärmel, S., Lochau, M., Kelter, U.: Fault-based product-line testing: effective sample generation based on feature-diagram mutation. In: Proceedings of the 19th International Conference on Software Product Line, SPLC ’15, pp. 131–140. ACM (2015). https://doi.org/10.1145/2791060.2791074

  53. Sabouri, H., Jaghoori, M.M., de Boer, F., Khosravi, R.: Scheduling and analysis of real-time software families. In: COMPSAC’12 (2012). https://doi.org/10.1109/COMPSAC.2012.95

  54. Sarkar, A., Guo, J., Siegmund, N., Apel, S., Czarnecki, K.: Cost-efficient sampling for performance prediction of configurable systems (t). In: 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 342–352. IEEE (2015). https://doi.org/10.1109/ASE.2015.45

  55. Siegmund, N., Rosenmüller, M., Kuhlemann, M., Kästner, C., Apel, S., Saake, G.: SPL conqueror: toward optimization of non-functional properties in software product lines. Softw. Qual. J. 20(3), 487–517 (2012). https://doi.org/10.1007/s11219-011-9152-9

    Article  Google Scholar 

  56. Springintveld, J., Vaandrager, F., D’Argenio, P.R.: Testing timed automata. Theor. Comput. Sci. 254(1), 225–257 (2001). https://doi.org/10.1016/S0304-3975(99)00134-6

    Article  MathSciNet  MATH  Google Scholar 

  57. Utting, M., Legeard, B.: Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann, Burlington (2010)

    Google Scholar 

  58. Weiss, D.M.: The product line hall of fame. In: 12th International Software Product Line Conference, p. 395 (2008). https://doi.org/10.1109/SPLC.2008.56

Download references

Acknowledgements

This work has been supported by the German Research Foundation (DFG) in the Priority Programme SPP 1593: Design For Future—Managed Software Evolution (LO 2198/2-1). This work was funded by the Hessian LOEWE initiative within the Software-Factory 4.0 Project.

Author information

Authors and Affiliations

  1. Real-Time Systems Lab, TU Darmstadt, Darmstadt, Germany

    Lars Luthmann, Timo Gerecht & Malte Lochau

Authors
  1. Lars Luthmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Timo Gerecht
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Malte Lochau
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lars Luthmann.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Luthmann, L., Gerecht, T. & Lochau, M. Sampling strategies for product lines with unbounded parametric real-time constraints. Int J Softw Tools Technol Transfer 21, 613–633 (2019). https://doi.org/10.1007/s10009-019-00532-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10009-019-00532-4

Keywords

Search

Navigation