Skip to main content
SpringerLink
Log in

Validation Frameworks for Self-Driving Vehicles: A Survey

  • Chapter
  • First Online:
Smart Cities: A Data Analytics Perspective

Abstract

As a part of the digital transformation, we interact with more and more intelligent gadgets. Today, these gadgets are often mobile devices, but in the advent of smart cities, more and more infrastructure—such as traffic and buildings—in our surroundings becomes intelligent. The intelligence, however, does not emerge by itself. Instead, we need both design techniques to create intelligent systems, as well as approaches to validate their correct behavior. An example of intelligent systems that could benefit smart cities are self-driving vehicles. Self-driving vehicles are continuously becoming both commercially available and common on roads. Accidents involving self-driving vehicles, however, have raised concerns about their reliability. Due to these concerns, the safety of self-driving vehicles should be thoroughly tested before they can be released into traffic. To ensure that self-driving vehicles encounter all possible scenarios, several millions of hours of testing must be carried out; therefore, testing self-driving vehicles in the real world is impractical. There is also the issue that testing self-driving vehicles directly in the traffic poses a potential safety hazard to human drivers. To tackle this challenge, validation frameworks for testing self-driving vehicles in simulated scenarios are being developed by academia and industry. In this chapter, we briefly introduce self-driving vehicles and give an overview of validation frameworks for testing them in a simulated environment. We conclude by discussing what an ideal validation framework at the state of the art should be and what could benefit validation frameworks for self-driving vehicles in the future.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.carsim.com/.

  2. 2.

    https://www.avsimulation.fr/solutions/.

References

  1. Akhtar N, Mian A (2018) Threat of adversarial attacks on deep learning in computer vision: a survey. IEEE Access 6:14410–14430. https://doi.org/10.1109/ACCESS.2018.2807385

    Article  Google Scholar 

  2. Amoozadeh M, Raghuramu A, Chuah C, Ghosal D, Zhang HM, Rowe J, Levitt K (2015) Security vulnerabilities of connected vehicle streams and their impact on cooperative driving. IEEE Commun Mag 53(6):126–132. https://doi.org/10.1109/MCOM.2015.7120028

    Article  Google Scholar 

  3. Barrachina J, Sanguesa JA, Fogue M, Garrido P, Martinez FJ, Cano J, Calafate CT, Manzoni P (2013) V2x-d: a vehicular density estimation system that combines v2v and v2i communications. In: 2013 IFIP wireless days (WD), pp 1–6

    Google Scholar 

  4. Bento LC, Parafita R, Nunes U (2012) Intelligent traffic management at intersections supported by v2v and v2i communications. In: 2012 15th international IEEE conference on intelligent transportation systems, pp 1495–1502

    Google Scholar 

  5. Clark JO (2009) System of systems engineering and family of systems engineering from a standards, v-model, and dual-v model perspective. In: 2009 3rd annual IEEE systems conference, pp 381–387. https://doi.org/10.1109/SYSTEMS.2009.4815831

  6. Costa-Gomes MA, Crawford VP (2006) Cognition and behavior in two-person guessing games: an experimental study. Am Econ Rev 96(5):1737–1768. https://doi.org/10.1257/aer.96.5.1737

    Article  Google Scholar 

  7. Costa-Gomes MA, Crawford VP, Iriberri N (2009) Comparing models of strategic thinking in Van Huyck, Battalio, and Beil’s coordination games. J Eur Econ Assoc 7(2–3):365–376. https://doi.org/10.1162/JEEA.2009.7.2-3.365

    Article  Google Scholar 

  8. Denton EL, Chintala S, Szlam A, Fergus R (2015) Deep generative image models using a laplacian pyramid of adversarial networks. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R (eds) Advances in neural information processing systems, vol 28. Curran Associates, Inc., pp 1486–1494

    Google Scholar 

  9. Geisberger R (2014) Route planning. US Patent 14/157,913. https://patents.google.com/patent/US20140200807

  10. Geraerts R, Overmars MH (2004) A comparative study of probabilistic roadmap planners. Springer, Berlin, pp 43–57. https://doi.org/10.1007/978-3-540-45058-0_4

  11. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Ghahramani Z, Welling M, Cortes C, Lawrence ND, Weinberger KQ (eds) Advances in neural information processing systems, vol 27. Curran Associates, Inc., pp 2672–2680

    Google Scholar 

  12. Harding J, Powell G, Yoon R, Fikentscher J, Doyle C, Sade D, Lukuc M, Simons J, Wang J (2014) Vehicle-to-vehicle communications: readiness of v2v technology for application. Technical report, United States National Highway Traffic Safety Administration

    Google Scholar 

  13. Hedden T, Zhang J (2002) What do you think i think you think?: strategic reasoning in matrix games. Cognition 85(1):1–36. https://doi.org/10.1016/S0010-0277(02)00054-9

    Article  Google Scholar 

  14. Hiblot N, Gruyer D, Barreiro JS, Monnier B (2010) Pro-SiVIC and roads, a software suite for sensors simulation and virtual prototyping of ADAS. In: Proceedings of the driving simulation conference, pp 277–288

    Google Scholar 

  15. IHS Markit Ltd. (2018) Autonomous vehicle sales to surpass 33 million annually in 2040, enabling new autonomous mobility in more than 26 percent of new car sales, ihs markit says. https://technology.ihs.com/599099/autonomous-vehicle-sales-to-surpass-33-million-annually-in-2040-enabling-new-autonomous-mobility-in-more-than-26-percent-of-new-car-sales-ihs-markit-says. Accessed 17 Feb 2020

  16. Jafarnejad S, Codeca L, Bronzi W, Frank R, Engel T (2015) A car hacking experiment: When connectivity meets vulnerability. In: 2015 IEEE globecom workshops (GC Wkshps), pp 1–6. https://doi.org/10.1109/GLOCOMW.2015.7413993

  17. Jeon S, Cho J, Jung Y, Park S, Han T (2011) Automotive hardware development according to iso 26262. In: 13th international conference on advanced communication technology (ICACT2011), pp 588–592

    Google Scholar 

  18. Jha S, Banerjee S, Tsai T, Hari SKS, Sullivan MB, Kalbarczyk ZT, Keckler SW, Iyer RK (2019) Ml-based fault injection for autonomous vehicles: A case for bayesian fault injection. In: 2019 49th annual IEEE/IFIP international conference on dependable systems and networks (DSN), pp 112–124. https://doi.org/10.1109/DSN.2019.00025

  19. Jha S, Banerjee SS, Cyriac J, Kalbarczyk ZT, Iyer RK (2018) AVFI: fault injection for autonomous vehicles. In: 2018 48th annual IEEE/IFIP international conference on dependable systems and networks workshops (DSN-W), pp. 55–56. https://doi.org/10.1109/DSN-W.2018.00027

  20. Jha S, Tsai T, Hari S, Sullivan M, Kalbarczyk Z, Keckler SW, Iyer RK (2019) Kayotee: a fault injection-based system to assess the safety and reliability of autonomous vehicles to faults and errors. arXiv:1907.01024. Accessed 17 Feb 2020

  21. Jia D, Ngoduy D (2016) Enhanced cooperative car-following traffic model with the combination of v2v and v2i communication. Transp Res Part B: Methodol. 90:172–191. https://doi.org/10.1016/j.trb.2016.03.008

    Article  Google Scholar 

  22. Jo K, Kim J, Kim D, Jang C, Sunwoo M (2015) Development of autonomous car-part ii: a case study on the implementation of an autonomous driving system based on distributed architecture. IEEE Trans Ind Electron 62(8):5119–5132. https://doi.org/10.1109/TIE.2015.2410258

    Article  Google Scholar 

  23. Koenig N, Howard A (2004) Design and use paradigms for gazebo, an open-source multi-robot simulator. In: 2004 IEEE/RSJ international conference on intelligent robots and systems (IROS) (IEEE Cat. No.04CH37566), vol 3, pp 2149–2154. https://doi.org/10.1109/IROS.2004.1389727

  24. Koopman P, Ferrell U, Fratrik F, Wagner M (2019) A safety standard approach for fully autonomous vehicles. In: Romanovsky A, Troubitsyna E, Gashi I, Schoitsch E, Bitsch F (eds) Computer safety, reliability, and security. Springer International Publishing, Cham, pp 326–332

    Chapter  Google Scholar 

  25. Koopman P, Wagner M (2016) Challenges in autonomous vehicle testing and validation. SAE Int J Transp Saf 4(1):15–24

    Article  Google Scholar 

  26. Krajzewicz D, Erdmann J, Behrisch M, Bieker L (2012) Recent development and applications of SUMO - simulation of urban mobility. Int J Adv Syst Meas 5(3&4):128–138

    Google Scholar 

  27. Lattarulo R, Pérez J, Dendaluce M (2017) A complete framework for developing and testing automated driving controllers. IFAC-PapersOnLine 50(1):258–263. 20th IFAC World Congress. https://doi.org/10.1016/j.ifacol.2017.08.043

  28. Leudet J, Christophe F, Mikkonen T, Männistö T (2019) Ailivesim: an extensible virtual environment for training autonomous vehicles. In: 2019 IEEE 43rd annual computer software and applications conference (COMPSAC), vol 1. IEEE, pp 479–488

    Google Scholar 

  29. Li N, Oyler DW, Zhang M, Yildiz Y, Kolmanovsky I, Girard AR (2018) Game theoretic modeling of driver and vehicle interactions for verification and validation of autonomous vehicle control systems. IEEE Trans Control Syst Technol 26(5):1782–1797. https://doi.org/10.1109/TCST.2017.2723574

    Article  Google Scholar 

  30. Lim BS, Keoh SL, Thing VLL (2018) Autonomous vehicle ultrasonic sensor vulnerability and impact assessment. In: 2018 IEEE 4th World Forum on Internet of Things (WF-IoT), pp 231–236. https://doi.org/10.1109/WF-IoT.2018.8355132

  31. National Conference of State Legislatures (2019) Autonomous vehicles|self-driving vehicles enacted legislation. http://www.ncsl.org/research/transportation/autonomous-vehicles-self-driving-vehicles-enacted-legislation.aspx. Accessed 17 Feb 2020

  32. Nikitas A, Kougias I, Alyavina E, Njoya Tchouamou E (2017) How can autonomous and connected vehicles, electromobility, brt, hyperloop, shared use mobility and mobility-as-a-service shape transport futures for the context of smart cities? Urban Sci 1(4). https://doi.org/10.3390/urbansci1040036

  33. Pei K, Cao Y, Yang J, Jana S (2017) Deepxplore: automated whitebox testing of deep learning systems. In: Proceedings of the 26th symposium on operating systems principles, SOSP ’17. ACM, New York, NY, USA, pp 1–18. https://doi.org/10.1145/3132747.3132785

  34. Petit J, Shladover SE (2015) Potential cyberattacks on automated vehicles. IEEE Trans Intell Transp Syst 16(2):546–556. https://doi.org/10.1109/TITS.2014.2342271

    Article  Google Scholar 

  35. Rubaiyat AHM, Qin Y, Alemzadeh H (2018) Experimental resilience assessment of an open-source driving agent. In: 2018 IEEE 23rd Pacific rim international symposium on dependable computing (PRDC), pp 54–63. https://doi.org/10.1109/PRDC.2018.00016

  36. SAE international (2008) Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. Technical report, SAE International

    Google Scholar 

  37. Schnellbach A, Griessnig G (2019) Development of the iso 21448. In: Walker A, O’Connor RV, Messnarz R (eds) Systems, software and services process improvement. Springer International Publishing, Cham, pp 585–593

    Chapter  Google Scholar 

  38. Stahl DO, Wilson PW (1995) On players’ models of other players: theory and experimental evidence. Games Econ Behav 10(1):218–254. https://doi.org/10.1006/game.1995.1031

    Article  MathSciNet  MATH  Google Scholar 

  39. Tian Y, Pei K, Jana S, Ray B (2018) Deeptest: automated testing of deep-neural-network-driven autonomous cars. In: Proceedings of the 40th international conference on software engineering, ICSE ’18. ACM, New York, NY, USA, pp 303–314. https://doi.org/10.1145/3180155.3180220

  40. Voulodimos A, Doulamis N, Doulamis A, Protopapadakis E (2018) Deep learning for computer vision: a brief review. Comput Intell Neurosci 2018(7068349):1–13. https://doi.org/10.1155/2018/7068349

    Article  Google Scholar 

  41. Zhang H, Xu T, Li H, Zhang S, Wang X, Huang X, Metaxas DN (2017) Stackgan: text to photo-realistic image synthesis with stacked generative adversarial networks. In: The IEEE international conference on computer vision (ICCV)

    Google Scholar 

  42. Zhang M, Zhang Y, Zhang L, Liu C, Khurshid S (2018) Deeproad: gan-based metamorphic testing and input validation framework for autonomous driving systems. In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, ASE 2018. ACM, New York, NY, USA, pp 132–142. https://doi.org/10.1145/3238147.3238187

  43. Zheng B, Liang H, Zhu Q, Yu H, Lin C (2016) Next generation automotive architecture modeling and exploration for autonomous driving. In: 2016 IEEE computer society annual symposium on VLSI (ISVLSI), pp 53–58. https://doi.org/10.1109/ISVLSI.2016.126

  44. Zheng B, Lin C, Yu H, Liang H, Zhu Q (2016) Convince: a cross-layer modeling, exploration and validation framework for next-generation connected vehicles. In: 2016 IEEE/ACM international conference on computer-aided design (ICCAD), pp 1–8. https://doi.org/10.1145/2966986.2980078

  45. Zofka MR, Klemm S, Kuhnt F, Schamm T, Zöllner JM (2016) Testing and validating high level components for automated driving: simulation framework for traffic scenarios. In: 2016 IEEE intelligent vehicles symposium (IV), pp 144–150. https://doi.org/10.1109/IVS.2016.7535378

  46. Zofka MR, Kuhnt F, Kohlhaas R, Zöllner JM (2016) Simulation framework for the development of autonomous small scale vehicles. In: 2016 IEEE international conference on simulation, modeling, and programming for autonomous robots (SIMPAR), pp 318–324. https://doi.org/10.1109/SIMPAR.2016.7862413

Download references

Author information

Authors and Affiliations

  1. University of Helsinki, PL 64, Helsinki, Finland

    Francesco Concas

  2. University of Helsinki, PL 68, Helsinki, Finland

    Jukka K. Nurminen, Tommi Mikkonen & Sasu Tarkoma

Authors
  1. Francesco Concas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jukka K. Nurminen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tommi Mikkonen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sasu Tarkoma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Francesco Concas .

Editor information

Editors and Affiliations

  1. College of Computing and Information Technology, University of Bisha, Bisha, Saudi Arabia

    Mohammad Ayoub Khan

  2. College of Computing and Information Technology, University of Bisha, Bisha, Saudi Arabia

    Fahad Algarni

  3. College of Computing and Information Technology, University of Bisha, Bisha, Saudi Arabia

    Mohammad Tabrez Quasim

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Concas, F., Nurminen, J.K., Mikkonen, T., Tarkoma, S. (2021). Validation Frameworks for Self-Driving Vehicles: A Survey. In: Khan, M.A., Algarni, F., Quasim, M.T. (eds) Smart Cities: A Data Analytics Perspective. Lecture Notes in Intelligent Transportation and Infrastructure. Springer, Cham. https://doi.org/10.1007/978-3-030-60922-1_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-60922-1_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-60921-4

  • Online ISBN: 978-3-030-60922-1

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation