skip to main content
10.1145/3368089.3409685acmconferencesArticle/Chapter ViewAbstractPublication PagesfseConference Proceedingsconference-collections

Testing self-adaptive software with probabilistic guarantees on performance metrics

Published:08 November 2020Publication History

ABSTRACT

This paper discusses the problem of testing the performance of the adaptation layer in a self-adaptive system. The problem is notoriously hard, due to the high degree of uncertainty and variability inherent in an adaptive software application. In particular, providing any type of formal guarantee for this problem is extremely difficult. In this paper we propose the use of a rigorous probabilistic approach to overcome the mentioned difficulties and provide probabilistic guarantees on the software performance. We describe the set up needed for the application of a probabilistic approach. We then discuss the traditional tools from statistics that could be applied to analyse the results, highlighting their limitations and motivating why they are unsuitable for the given problem. We propose the use of a novel tool – the scenario theory – to overcome said limitations. We conclude the paper with a thorough empirical evaluation of the proposed approach, using two adaptive software applications: the Tele-Assistance Service and the Self-Adaptive Video Encoder. With the first, we empirically expose the trade-off between data collection and confidence in the testing campaign. With the second, we demonstrate how to compare different adaptation strategies.

Skip Supplemental Material Section

Supplemental Material

fse20main-p171-p-teaser.mp4

mp4

9.7 MB

fse20main-p171-p-video.mp4

mp4

303.5 MB

References

  1. Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin. 2012. Learning From Data. AMLBook.Google ScholarGoogle Scholar
  2. D.L. Applegate, R.E. Bixby, V. Chvátal, and W.J. Cook. 2011. The Traveling Salesman Problem: A Computational Study. Princeton University Press. https: //books.google.se/books?id=zfIm94nNqPoCGoogle ScholarGoogle Scholar
  3. Andrea Arcuri and Lionel Briand. 2011. Adaptive Random Testing: An Illusion of Efectiveness?. In Proceedings of the 2011 International Symposium on Software Testing and Analysis (Toronto, Ontario, Canada) ( ISSTA '11). ACM, New York, NY, USA, 265-275. https://doi.org/10.1145/2001420.2001452 Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Andrea Arcuri, Gordon Fraser, and Juan Pablo Galeotti. 2014. Automated Unit Test Generation for Classes with Environment Dependencies. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering (Vasteras, Sweden) (ASE '14). ACM, New York, NY, USA, 79-90. https://doi.org/ 10.1145/2642937.2642986 Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. R. I. Bahar, U. Karpuzcu, and S. Misailovic. 2019. Special Session: Does Approximation Make Testing Harder (or Easier)?. In 2019 IEEE 37th VLSI Test Symposium (VTS). 1-9. https://doi.org/10.1109/VTS. 2019.8758649 Google ScholarGoogle ScholarCross RefCross Ref
  6. L. Baresi, D. Bianculli, C. Ghezzi, S. Guinea, and P. Spoletini. 2007. Validation of web service compositions. IET Software 1, 6 ( December 2007 ), 219-232. https: //doi.org/10.1049/iet-sen: 20070027 Google ScholarGoogle ScholarCross RefCross Ref
  7. Antonia Bertolino and Paola Inverardi. 2019. Changing Software in a Changing World: How to Test in Presence of Variability, Adaptation and Evolution? Springer International Publishing, Cham, 56-66. https://doi.org/10.1007/978-3-030-30985-5_5 Google ScholarGoogle ScholarCross RefCross Ref
  8. Antonia Bertolino, Paola Inverardi, and Henry Muccini. 2003. Formal Methods in Testing Software Architectures. Springer Berlin Heidelberg, Berlin, Heidelberg, 122-147. https://doi.org/10.1007/978-3-540-39800-4_7 Google ScholarGoogle ScholarCross RefCross Ref
  9. Marcel Böhme. 2019. Assurance in Software Testing: A Roadmap. In Proceedings of the 41st International Conference on Software Engineering: New Ideas and Emerging Results (Montreal, Quebec, Canada) ( ICSE-NIER '19). IEEE Press, Piscataway, NJ, USA, 5-8. https://doi.org/10.1109/ICSE-NIER. 2019.00010 Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Lionel Briand, Shiva Nejati, Mehrdad Sabetzadeh, and Domenico Bianculli. 2016. Testing the Untestable: Model Testing of Complex Software-intensive Systems. In Proceedings of the 38th International Conference on Software Engineering Companion (Austin, Texas) ( ICSE '16). ACM, New York, NY, USA, 789-792. https://doi.org/10.1145/2889160.2889212 Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Giuseppe Carlo Calafiore. 2013. Direct data-driven portfolio optimization with guaranteed shortfall probability. Automatica 49, 2 ( 2013 ), 370-380. https: //doi.org/10.1016/j.automatica. 2012. 11.012 Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. G. C. Calafiore and M. C. Campi. 2006. The scenario approach to robust control design. IEEE Trans. Automat. Control 51, 5 (May 2006 ), 742-753. https://doi.org/ 10.1109/TAC. 2006.875041 Google ScholarGoogle ScholarCross RefCross Ref
  13. G. Canfora and M. Di Penta. 2006. Testing services and service-centric systems: challenges and opportunities. IT Professional 8, 2 (March 2006 ), 10-17. https: //doi.org/10.1109/MITP. 2006.51 Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Mauro Caporuscio, Rafaela Mirandola, and Catia Trubiani. 2017. Building Designtime and Run-time Knowledge for QoS-based Component Assembly. Softw. Pract. Exper. 47, 12 (Dec. 2017 ), 1905-1922. https://doi.org/10.1002/spe.2502 Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Francisco J. Cazorla, Tullio Vardanega, Eduardo Quiñones, and Jaume Abella. 2013. Upper-bounding Program Execution Time with Extreme Value Theory. In 13th International Workshop on Worst-Case Execution Time Analysis (OpenAccess Series in Informatics (OASIcs), Vol. 30 ), Claire Maiza (Ed.). Schloss Dagstuhl-LeibnizZentrum fuer Informatik, Dagstuhl, Germany, 64-76. https://doi.org/10.4230/ OASIcs.WCET. 2013.64 Google ScholarGoogle ScholarCross RefCross Ref
  16. Tsong Yueh Chen, Fei-Ching Kuo, Robert G. Merkel, and T. H. Tse. 2010. Adaptive Random Testing: The ART of Test Case Diversity. J. Syst. Softw. 83, 1 (Jan. 2010 ), 60-66. https://doi.org/10.1016/j.jss. 2009. 02.022 Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Betty H. C. Cheng, Kerstin I. Eder, Martin Gogolla, Lars Grunske, Marin Litoiu, Hausi A. Müller, Patrizio Pelliccione, Anna Perini, Nauman A. Qureshi, Bernhard Rumpe, Daniel Schneider, Frank Trollmann, and Norha M. Villegas. 2014. Using Models at Runtime to Address Assurance for Self-Adaptive Systems. Springer International Publishing, Cham, 101-136. https://doi.org/10.1007/978-3-319-08915-7_4 Google ScholarGoogle ScholarCross RefCross Ref
  18. Laurens de Haan and Ana Ferreira. 2010. Extreme Value Theory: An Introduction (Springer Series in Operations Research and Financial Engineering) (1st edition. ed.). Springer.Google ScholarGoogle Scholar
  19. Vânia de Oliveira Neves, Antonia Bertolino, Gugliemo De Angelis, and Lina Garcés. 2018. Do We Need New Strategies for Testing Systems-of-systems?. In Proceedings of the 6th International Workshop on Software Engineering for Systemsof-Systems (Gothenburg, Sweden) ( SESoS '18). ACM, New York, NY, USA, 29-32. https://doi.org/10.1145/3194754.3194758 Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Saikat Dutta, Owolabi Legunsen, Zixin Huang, and Sasa Misailovic. 2018. Testing Probabilistic Programming Systems. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Lake Buena Vista, FL, USA) (ESEC/FSE 2018 ). Association for Computing Machinery, New York, NY, USA, 574-586. https://doi.org/10.1145/3236024.3236057 Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Saikat Dutta, Wenxian Zhang, Zixin Huang, and Sasa Misailovic. 2019. Storm: Program Reduction for Testing and Debugging Probabilistic Programming Systems. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Tallinn, Estonia) (ESEC/FSE 2019 ). Association for Computing Machinery, New York, NY, USA, 729-739. https://doi.org/10.1145/3338906.3338972 Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Nicolas D'Ippolito, Víctor Braberman, Jef Kramer, Jef Magee, Daniel Sykes, and Sebastian Uchitel. 2014. Hope for the Best, Prepare for the Worst: Multi-Tier Control for Adaptive Systems. In Proceedings of the 36th International Conference on Software Engineering (Hyderabad, India) (ICSE 2014 ). Association for Computing Machinery, New York, NY, USA, 688-699. https://doi.org/10.1145/2568225. 2568264 Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Ross Edwards and Nelly Bencomo. 2018. DeSiRE: Further Understanding Nuances of Degrees of Satisfaction of Non-functional Requirements Trade-of. In Proceedings of the 13th International Conference on Software Engineering for Adaptive and Self-Managing Systems (Gothenburg, Sweden) (SEAMS '18). ACM, New York, NY, USA, 12-18. https://doi.org/10.1145/3194133.3194142 Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Paul Embrechts. 2000. Extreme Value Theory: Potential And Limitations As An Integrated Risk Management Tool. Derivatives Use, Trading and Regulation 6 ( 02 2000 ).Google ScholarGoogle Scholar
  25. Paul Embrechts, Thomas Mikosch, and Claudia Klüppelberg. 1997. Modelling Extremal Events: For Insurance and Finance. Springer-Verlag, Berlin, Heidelberg.Google ScholarGoogle Scholar
  26. Fabiano Cutigi Ferrari, Joost Noppen, Ruzanna Chitchyan, and Awais Rashid Lancaster. 2011. Investigating Testing Approaches for Dynamically Adaptive Systems Work in Progress.Google ScholarGoogle Scholar
  27. A. Filieri, C. Ghezzi, A. Leva, and M. Maggio. 2011. Self-adaptive software meets control theory: A preliminary approach supporting reliability requirements. In 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011 ). IEEE, Lawrence, KS, USA, 283-292. https://doi.org/10.1109/ASE. 2011. 6100064 Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Antonio Filieri, Henry Hofmann, and Martina Maggio. 2014. Automated Design of Self-adaptive Software with Control-theoretical Formal Guarantees. In Proceedings of the 36th International Conference on Software Engineering (Hyderabad, India) (ICSE). ACM, New York, NY, USA, 299-310. https://doi.org/10.1145/ 2568225.2568272 Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. R.A. Fisher. 1930. The Genetical Theory of Natural Selection. OUP Oxford.Google ScholarGoogle Scholar
  30. B.A. Francis and P. Khargonekar. 1995. Robust control theory. Springer-Verlag. https://books.google.se/books?id=81vvAAAAMAAJGoogle ScholarGoogle Scholar
  31. C. E. Garcia, D. M. Prett, and M. Morari. 1989. Model Predictive Control: Theory and Practice&Mdash;a Survey. Automatica 25, 3 (May 1989 ), 335-348. https: //doi.org/10.1016/ 0005-1098 ( 89 ) 90002-2 Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Carlos A. González, Mojtaba Varmazyar, Shiva Nejati, Lionel C. Briand, and Yago Isasi. 2018. Enabling Model Testing of Cyber-Physical Systems. In Proceedings of the 21th ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (Copenhagen, Denmark) ( MODELS '18). ACM, New York, NY, USA, 176-186. https://doi.org/10.1145/3239372.3239409 Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Vincenzo Gulisano, Alessandro V. Papadopoulos, Yiannis Nikolakopoulos, Marina Papatriantafilou, and Philippas Tsigas. 2017. Performance Modeling of Stream Joins. In Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems (Barcelona, Spain) (DEBS '17). ACM, New York, NY, USA, 191-202. https://doi.org/10.1145/3093742.3093923 Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Aymeric Hervieu, Benoit Baudry, and Arnaud Gotlieb. 2012. Managing Execution Environment Variability during Software Testing: An Industrial Experience. In Testing Software and Systems, Brian Nielsen and Carsten Weise (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 24-38. https://doi.org/10.1007/978-3-642-34691-0_4 Google ScholarGoogle ScholarCross RefCross Ref
  35. Robert M. Hierons and Mercedes G. Merayo. 2009. Mutation Testing from Probabilistic and Stochastic Finite State Machines. J. Syst. Softw. 82, 11 (Nov. 2009 ), 1804-1818. https://doi.org/10.1016/j.jss. 2009. 06.030 Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. San-Yih Hwang, Haojun Wang, Jian Tang, and Jaideep Srivastava. 2007. A Probabilistic Approach to Modeling and Estimating the QoS of Web-servicesbased Workflows. Inf. Sci. 177, 23 (Dec. 2007 ), 5484-5503. https://doi.org/10. 1016/j.ins. 2007. 07.011 Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. J. Hänsel, T. Vogel, and H. Giese. 2015. A Testing Scheme for Self-Adaptive Software Systems with Architectural Runtime Models. In 2015 IEEE International Conference on Self-Adaptive and Self-Organizing Systems Workshops. 134-139. https://doi.org/10.1109/SASOW. 2015.27 Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Antonio Jiménez-Martín, Alfonso Mateos, and Sixto Ríos-Insua. 2005. Monte Carlo Simulation Techniques in a Decision Support System for Group Decision Making. Group Decision and Negotiation 14 (01 2005 ), 109-130. https://doi.org/ 10.1007/s10726-005-2406-9 Google ScholarGoogle ScholarCross RefCross Ref
  39. O. Johnson. 2004. Information Theory and the Central Limit Theorem. Imperial College Press. https://books.google.se/books?id=r5XI8a0lYykCGoogle ScholarGoogle Scholar
  40. Keyur Joshi, Vimuth Fernando, and Sasa Misailovic. 2019. Statistical Algorithmic Profiling for Randomized Approximate Programs. In Proceedings of the 41st International Conference on Software Engineering (Montreal, Quebec, Canada) ( ICSE '19). IEEE Press, 608-618. https://doi.org/10.1109/ICSE. 2019.00071 Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Brian Korver. 1994. The Monte Carlo Method and Software Reliability Theory.Google ScholarGoogle Scholar
  42. Martina Maggio, Alessandro Vittorio Papadopoulos, Antonio Filieri, and Henry Hofmann. 2017. Automated Control of Multiple Software Goals Using Multiple Actuators. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (Paderborn, Germany) ( ESEC/FSE 2017). ACM, New York, NY, USA, 373-384. https://doi.org/10.1145/3106237.3106247 Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Martina Maggio, Alessandro Vittorio Papadopoulos, Antonio Filieri, and Henry Hofmann. 2017. Self-adaptive Video Encoder: Comparison of Multiple Adaptation Strategies Made Simple. In Proceedings of the 12th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (Buenos Aires, Argentina) (SEAMS '17). IEEE Press, Piscataway, NJ, USA, 123-128. https: //doi.org/10.1109/SEAMS. 2017.16 Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Claudio Mandrioli Martina Maggio. 2020. Artifact ESEC /FSE 2020. https: //doi.org/10.5281/ZENODO.3896795 Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. M. A. Mehmood, M. N. A. Khan, and W. Afzal. 2018. Automating Test Data Generation for Testing Context-Aware Applications. In 2018 IEEE 9th International Conference on Software Engineering and Service Science (ICSESS). 104-108. https: //doi.org/10.1109/ICSESS. 2018.8663920 Google ScholarGoogle ScholarCross RefCross Ref
  46. Gabriel A. Moreno, Javier Cámara, David Garlan, and Bradley Schmerl. 2015. Proactive Self-adaptation Under Uncertainty: A Probabilistic Model Checking Approach. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2015 ). ACM, Bergamo, Italy, 1-12. https://doi.org/10. 1145/2786805.2786853 Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Gabriel A. Moreno, Alessandro V. Papadopoulos, Konstantinos Angelopoulos, Javier Cámara, and Bradley Schmerl. 2017. Comparing Model-based Predictive Approaches to Self-adaptation: CobRA and PLA. In Proceedings of the 12th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (Buenos Aires, Argentina) (SEAMS '17). IEEE Press, Piscataway, NJ, USA, 42-53. https://doi.org/10.1109/SEAMS. 2017.2 Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Freddy Munoz and Benoit Baudry. 2009. Artificial table testing dynamically adaptive systems. CoRR abs/0903.0914 ( 2009 ). arXiv: 0903.0914 http://arxiv.org/ abs/0903.0914Google ScholarGoogle Scholar
  49. Yi Qin, Chang Xu, Ping Yu, and Jian Lu. 2016. SIT: Sampling-based interactive testing for self-adaptive apps. Journal of Systems and Software 120 ( 2016 ), 70-88. https://doi.org/10.1016/j.jss. 2016. 07.002 Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Federico Alessandro Ramponi and Marco C. Campi. 2018. Expected shortfall: Heuristics and certificates. European Journal of Operational Research 267, 3 ( 2018 ), 1003-1013. https://doi.org/10.1016/j.ejor. 2017. 11.022 Google ScholarGoogle ScholarCross RefCross Ref
  51. A. Reichstaller and A. Knapp. 2018. Risk-Based Testing of Self-Adaptive Systems Using Run-Time Predictions. In 2018 IEEE 12th International Conference on SelfAdaptive and Self-Organizing Systems (SASO). 80-89. https://doi.org/10.1109/ SASO. 2018.00019 Google ScholarGoogle ScholarCross RefCross Ref
  52. Christian P. Robert and George Casella. 2005. Monte Carlo Statistical Methods (Springer Texts in Statistics). Springer-Verlag, Berlin, Heidelberg.Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Christian P. Robert and George Casella. 2010. Monte Carlo Optimization. Springer New York, New York, NY, 125-165. https://doi.org/10.1007/978-1-4419-1576-4_5 Google ScholarGoogle ScholarCross RefCross Ref
  54. S. Rosario, A. Benveniste, S. Haar, and C. Jard. 2008. Probabilistic QoS and Soft Contracts for Transaction-Based Web Services Orchestrations. IEEE Transactions on Services Computing 1, 4 (Oct 2008 ), 187-200. https://doi.org/10.1109/TSC. 2008.17 Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Mazeiar Salehie and Ladan Tahvildari. 2009. Self-Adaptive Software: Landscape and Research Challenges. ACM Trans. Auton. Adapt. Syst. 4, 2, Article 14 (May 2009 ), 42 pages. https://doi.org/10.1145/1516533.1516538 Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. L. Santinelli, Jérôme Morio, Guillaume Dufour, and Damien Jacquemart. 2014. On the Sustainability of the Extreme Value Theory for WCET Estimation. OpenAccess Series in Informatics 39. https://doi.org/10.4230/OASIcs.WCET. 2014.21 Google ScholarGoogle ScholarCross RefCross Ref
  57. Ismayle de Sousa Santos. 2017. TESTDAS: Testing MEthod for Dynamically Adaptive Systems. Ph.D. Dissertation. Fortaleza, Brazil. Advisor(s) Castro Andrade, Rossana Mariade.Google ScholarGoogle Scholar
  58. Stepan Shevtsov and Danny Weyns. 2016. Keep It SIMPLEX: Satisfying Multiple Goals with Guarantees in Control-based Self-adaptive Systems. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (Seattle, WA, USA) ( FSE 2016). ACM, New York, NY, USA, 229-241. https://doi.org/10.1145/2950290.2950301 Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Harnam Singh and Preet Pal. 2013. Software Reliability Testing using Monte Carlo Methods. International Journal of Computer Applications 69 (05 2013 ), 41-44. https://doi.org/10.5120/ 11834-7554 Google ScholarGoogle ScholarCross RefCross Ref
  60. Bento Rafael Siqueira, Fabiano Cutigi Ferrari, Marcel Akira Serikawa, Ricardo Menotti, and Valter Vieira de Camargo. 2016. Characterisation of Challenges for Testing of Adaptive Systems. In Proceedings of the 1st Brazilian Symposium on Systematic and Automated Software Testing (Maringa, Parana, Brazil) (SAST). Association for Computing Machinery, New York, NY, USA, Article 11, 10 pages. https://doi.org/10.1145/2993288.2993294 Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Richard S. Sutton and Andrew G. Barto. 1998. Introduction to Reinforcement Learning (1st ed.). MIT Press, Cambridge, MA, USA.Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Porfirio Tramontana, Domenico Amalfitano, Nicola Amatucci, Atif Memon, and Anna Rita Fasolino. 2019. Developing and Evaluating Objective Termination Criteria for Random Testing. ACM Trans. Softw. Eng. Methodol. 28, 3, Article 17 ( July 2019 ), 52 pages. https://doi.org/10.1145/3339836 Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. T. H. Tse, Sik-Sang Yau, W. K. Chan, Heng Lu, and T. Y. Chen. 2004. Testing context-sensitive middleware-based software applications. In Proceedings-International Computer Software and Applications Conference, Vol. 1. 458-466.Google ScholarGoogle Scholar
  64. Huai Wang, W. K. Chan, and T. H. Tse. 2014. Improving the Efectiveness of Testing Pervasive Software via Context Diversity. ACM Trans. Auton. Adapt. Syst. 9, 2, Article 9 ( July 2014 ), 28 pages. https://doi.org/10.1145/2620000 Google ScholarGoogle ScholarDigital LibraryDigital Library
  65. Yilin Wang, Sasi Inguva, and Balu Adsumilli. 2019. YouTube UGC Dataset for Video Compression Research. arXiv: 1904. 06457 [cs.MM] https://media. withyoutube.com/Google ScholarGoogle Scholar
  66. K. Welsh and P. Sawyer. 2010. Managing Testing Complexity in Dynamically Adaptive Systems: A Model-Driven Approach. In 2010 Third International Conference on Software Testing, Verification, and Validation Workshops. 290-298. https://doi.org/10.1109/ICSTW. 2010.57 Google ScholarGoogle ScholarDigital LibraryDigital Library
  67. Danny Weyns. 2012. Towards an Integrated Approach for Validating Qualities of Self-Adaptive Systems. In Proceedings of the Ninth International Workshop on Dynamic Analysis (Minneapolis, MN, USA) ( WODA 2012 ). Association for Computing Machinery, New York, NY, USA, 24-29. https://doi.org/10.1145/ 2338966.2336803 Google ScholarGoogle ScholarDigital LibraryDigital Library
  68. Danny Weyns and Radu Calinescu. 2015. Tele Assistance: A Self-adaptive Servicebased System Examplar. In Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (Florence, Italy) (SEAMS '15). IEEE Press, Piscataway, NJ, USA, 88-92. http://dl.acm.org/citation. cfm?id= 2821357. 2821373Google ScholarGoogle ScholarDigital LibraryDigital Library
  69. Kohsuke Yatoh, Kazunori Sakamoto, Fuyuki Ishikawa, and Shinichi Honiden. 2015. Feedback-Controlled Random Test Generation. In Proceedings of the 2015 International Symposium on Software Testing and Analysis (Baltimore MD USA ) (ISSTA 2015). Association for Computing Machinery, New York, NY, USA, 316-326. https://doi.org/10.1145/2771783.2771805 Google ScholarGoogle ScholarDigital LibraryDigital Library
  70. L. Yu, W. T. Tsai, Y. Jiang, and J. Gao. 2014. Generating Test Cases for ContextAware Applications Using Bigraphs. In 2014 Eighth International Conference on Software Security and Reliability (SERE). 137-146. https://doi.org/10.1109/SERE. 2014.27 Google ScholarGoogle ScholarCross RefCross Ref
  71. Andreas Zeller, Rahul Gopinath, Marcel Böhme, Gordon Fraser, and Christian Holler. 2019. The Fuzzing Book. In The Fuzzing Book. Saarland University. https://www.fuzzingbook. org/ Retrieved 2019-09-09 16 :42: 54 + 02 : 00.Google ScholarGoogle Scholar
  72. Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (April 2004 ), 600-612. https://doi.org/10.1109/TIP. 2003. 819861 Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Testing self-adaptive software with probabilistic guarantees on performance metrics

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader