Skip to main content
SpringerLink
Log in

Multi-robot Systems, Virtual Reality and ROS: Developing a New Generation of Operator Interfaces

  • Chapter
  • First Online:
Robot Operating System (ROS)

Part of the book series: Studies in Computational Intelligence ((SCI,volume 778))

Abstract

This chapter describes a series of works developed in order to integrate ROS-based robots with Unity-based virtual reality interfaces. The main goal of this integration is to develop immersive monitoring and commanding interfaces, able to improve the operator’s situational awareness without increasing its workload. In order to achieve this, the available technologies and resources are analyzed and multiple ROS packages and Unity assets are applied, such as \(multimaster\_fkie\), \(rosbridge\_suite\), RosBridgeLib and SteamVR. Moreover, three applications are presented: an interface for monitoring a fleet of drones, another interface for commanding a robot manipulator and an integration of multiple ground and aerial robots. Finally, some experiences and lessons learned, useful for future developments, are reported.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://unity3d.com/.

  2. 2.

    https://www.unrealengine.com.

  3. 3.

    http://wiki.ros.org/multimaster_fkie.

  4. 4.

    http://wiki.ros.org/rosbridge_suite.

  5. 5.

    https://github.com/MathiasCiarlo/ROSBridgeLib.

  6. 6.

    https://steamcommunity.com/steamvr.

  7. 7.

    http://gazebosim.org/.

  8. 8.

    https://www.vive.com/.

  9. 9.

    https://www.oculus.com/.

  10. 10.

    http://wiki.ros.org/rviz.

  11. 11.

    http://wiki.ros.org/multimaster_fkie.

  12. 12.

    http://wiki.ros.org/rosbridge_suite.

  13. 13.

    https://github.com/MathiasCiarlo/ROSBridgeLib.

  14. 14.

    https://www.parrot.com.

  15. 15.

    https://www.kuka.com.

  16. 16.

    http://www.kinovarobotics.com/.

  17. 17.

    http://wiki.ros.org/Robots/SummitXL.

  18. 18.

    http://wiki.ros.org/summit_xl_sim.

  19. 19.

    http://wiki.ros.org/rotors_simulator.

References

  1. M. Garzón, J. Valente, J.J. Roldán, L. Cancar, A. Barrientos, J. Del Cerro, A multirobot system for distributed area coverage and signal searching in large outdoor scenarios. J. Field Robot. 33(8), 1087–1106 (2016)

    Article  Google Scholar 

  2. J.J. Roldán, P. Garcia-Aunon, M. Garzón, J. de León, J. del Cerro, A. Barrientos, Heterogeneous multi-robot system for mapping environmental variables of greenhouses. Sensors 16(7), 1018 (2016)

    Article  Google Scholar 

  3. M. Garzón, J. Valente, D. Zapata, A. Barrientos, An aerial-ground robotic system for navigation and obstacle mapping in large outdoor areas. Sensors 13(1), 1247–1267 (2013)

    Article  Google Scholar 

  4. J.Y. Chen, E.C. Haas, M.J. Barnes, Human performance issues and user interface design for teleoperated robots. IEEE Trans. Syst. Man Cybern. Part C (Applications and Reviews) 37(6), 1231–1245 (2007)

    Article  Google Scholar 

  5. J. Ruiz, A. Viguria, J. Martinez-de Dios, A. Ollero, Immersive displays for building spatial knowledge in multi-uav operations, in 2015 International Conference on Unmanned Aircraft Systems (ICUAS) (IEEE, 2015), pp. 1043–1048

    Google Scholar 

  6. J.T. Hansberger, Development of the next generation of adaptive interfaces. Technical report, DTIC Document (2015)

    Google Scholar 

  7. J.J. Roldán, M.A. Olivares-Méndez, J. del Cerro, A. Barrientos, Analyzing and improving multi-robot missions by means of process mining. Auton. Robot. 1(1), 1–21 (2017)

    Google Scholar 

  8. P. Ulam, Y. Endo, A. Wagner, R. Arkin, Integrated mission specication and task allocation for robot teams-part 2: Testing and evaluation. Technical report, GEORGIA INST OF TECH ATLANTA COLL OF COMPUTING (2006)

    Google Scholar 

  9. S. Tully, G. Kantor, H. Choset, Leap-frog path design for multi-robot cooperative localization, in Field and Service Robotics (Springer, Berlin, 2010), pp. 307–317

    Google Scholar 

  10. A. Janchiv, D. Batsaikhan, G. hwan Kim, S.G. Lee, Complete coverage path planning for multi-robots based on, in 2011 11th International Conference on Control, Automation and Systems (ICCAS) (IEEE, 2011), pp. 824–827

    Google Scholar 

  11. M. Lindemuth, R. Murphy, E. Steimle, W. Armitage, K. Dreger, T. Elliot, M. Hall, D. Kalyadin, J. Kramer, M. Palankar et al., Sea robot-assisted inspection. IEEE Robot. Autom. Mag. 18(2), 96–107 (2011)

    Article  Google Scholar 

  12. J. Valente, D. Sanz, A. Barrientos, Jd Cerro, Á. Ribeiro, C. Rossi, An air-ground wireless sensor network for crop monitoring. Sensors 11(6), 6088–6108 (2011)

    Article  Google Scholar 

  13. N.A. Tsokas, K.J. Kyriakopoulos, Multi-robot multiple hypothesis tracking for pedestrian tracking. Auton. Robot. 32(1), 63–79 (2012)

    Article  Google Scholar 

  14. L. Cantelli, M. Mangiameli, C.D. Melita, G. Muscato, Uav/ugv cooperation for surveying operations in humanitarian demining, in 2013 IEEE International symposium on Safety, Security, and Rescue Robotics (SSRR) (IEEE, 2013), pp. 1–6

    Google Scholar 

  15. G. De Cubber, D. Doroftei, D. Serrano, K. Chintamani, R. Sabino, S. Ourevitch, The eu-icarus project: developing assistive robotic tools for search and rescue operations, in 2013 IEEE international symposium on Safety, Security, and Rescue Robotics (SSRR) (IEEE, 2013), pp. 1–4

    Google Scholar 

  16. I. Kruijff-Korbayová, F. Colas, M. Gianni, F. Pirri, J. Greeff, K. Hindriks, M. Neerincx, P. Ögren, T. Svoboda, R. Worst, Tradr project: Long-term human-robot teaming for robot assisted disaster response. KI-Künstliche Intelligenz 29(2), 193–201 (2015)

    Article  Google Scholar 

  17. J. Gregory, J. Fink, E. Stump, J. Twigg, J. Rogers, D. Baran, N. Fung, S. Young, Application of multi-robot systems to disaster-relief scenarios with limited communication, in Field and Service Robotics (Springer, Berlin, 2016), pp. 639–653

    Google Scholar 

  18. A.C. Kapoutsis, S.A. Chatzichristofis, L. Doitsidis, J.B. de Sousa, J. Pinto, J. Braga, E.B. Kosmatopoulos, Real-time adaptive multi-robot exploration with application to underwater map construction. Auton. Robot. 40(6), 987–1015 (2016)

    Article  Google Scholar 

  19. C. Lesire, G. Infantes, T. Gateau, M. Barbier, A distributed architecture for supervision of autonomous multi-robot missions. Auton. Robot. 40(7), 1343–1362 (2016)

    Article  Google Scholar 

  20. N. Agmon, O. Maximov, A. Rosenfeld, S. Shlomai, S. Kraus, Multiple robots for multiple missions: architecture for complex collaboration

    Google Scholar 

  21. X.J. Yang, C.D. Wickens, K. Hölttä-Otto, How users adjust trust in automation: Contrast effect and hindsight bias, in Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 60 (SAGE Publications Sage CA: Los Angeles, CA, 2016), pp. 196–200

    Article  Google Scholar 

  22. C. Sampedro, H. Bavle, J.L. Sanchez-Lopez, R.A.S. Fernández, A. Rodríguez-Ramos, M. Molina, P. Campoy, A flexible and dynamic mission planning architecture for uav swarm coordination, in 2016 International Conference on Unmanned Aircraft Systems (ICUAS) (IEEE, 2016), pp. 355–363

    Google Scholar 

  23. T. Nestmeyer, P.R. Giordano, H.H. Bülthoff, A. Franchi, Decentralized simultaneous multi-target exploration using a connected network of multiple robots. Auton. Robot. 41(4), 989–1011 (2017)

    Article  Google Scholar 

  24. M. Garzón, J. Valente, J.J. Roldán, D. Garzón-Ramos, J. de León, A. Barrientos, J. del Cerro, Using ros in multi-robot systems: Experiences and lessons learned from real-world field tests, in Robot Operating System (ROS) (Springer, Berlin, 2017), pp. 449–483

    Google Scholar 

  25. J.J. Roldán, E. Peña-Tapia, A. Martín-Barrio, M.A. Olivares-Méndez, J. Del Cerro, A. Barrientos, Multi-robot interfaces and operator situational awareness: study of the impact of immersion and prediction. Sensors 17(8), 1720 (2017)

    Article  Google Scholar 

  26. M.R. Endsley, Design and evaluation for situation awareness enhancement, in Proceedings of the human factors and ergonomics society annual meeting, vol. 32 (SAGE Publications, 1988), pp. 97–101

    Article  Google Scholar 

  27. J.L. Drury, J. Scholtz, H.A. Yanco, Awareness in human-robot interactions, in IEEE International Conference on Systems, Man and Cybernetics, vol. 1 (IEEE, 2003), pp. 912–918

    Google Scholar 

  28. M.R. Endsley, Situation awareness global assessment technique (sagat), in Proceedings of the IEEE National Aerospace and Electronics Conference. NAECON (IEEE, 1988), pp. 789–795

    Google Scholar 

  29. P. Salmon, N. Stanton, G. Walker, D. Green, Situation awareness measurement: a review of applicability for c4i environments. Appl. Ergon. 37(2), 225–238 (2006)

    Article  Google Scholar 

  30. J. Scholtz, J. Young, J.L. Drury, H.A. Yanco, Evaluation of human-robot interaction awareness in search and rescue, in Proceedings of the ICRA’04 IEEE International Conference on Robotics and Automation, vol. 3 (IEEE, 2004), pp. 2327–2332

    Google Scholar 

  31. N. Li, S. Cartwright, A. Shekhar Nittala, E. Sharlin, M. Costa Sousa, Flying frustum: a spatial interface for enhancing human-uav awareness, in Proceedings of the 3rd International Conference on Human-Agent Interaction (ACM, 2015), pp. 27–31

    Google Scholar 

  32. R.J. Lysaght, S.G. Hill, A. Dick, B.D. Plamondon, P.M. Linton, Operator workload: Comprehensive review and evaluation of operator workload methodologies. Technical report, DTIC Document (1989)

    Google Scholar 

  33. N. Moray, Mental Workload: Its Theory and Measurement, vol. 8 (Springer Science & Business Media, 2013)

    Google Scholar 

  34. S.G. Hart, L.E. Staveland, Development of nasa-tlx (task load index): Results of empirical and theoretical research. Adv. Psychol. 52, 139–183 (1988)

    Article  Google Scholar 

  35. S.R. Dixon, C.D. Wickens, D. Chang, Mission control of multiple unmanned aerial vehicles: a workload analysis. Hum. Factors J. Hum. Factors Ergon. Soc. 47(3), 479–487 (2005)

    Article  Google Scholar 

  36. B. Jacobs, E. De Visser, A. Freedy, P. Scerri, Application of Intelligent Aiding to Enable Single Operator Multiple uav Supervisory Control, Association for the advancement of artificial intelligence (Palo Alto, CA, 2010)

    Google Scholar 

  37. M.L. Cummings, C. Mastracchio, K.M. Thornburg, A. Mkrtchyan, Boredom and distraction in multiple unmanned vehicle supervisory control. Interact. Comput. 25(1), 34–47 (2013)

    Article  Google Scholar 

  38. D. McDuff, S. Gontarek, R. Picard, Remote measurement of cognitive stress via heart rate variability, in 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (IEEE, 2014), pp. 2957–2960

    Google Scholar 

  39. H. Kurniawan, A.V. Maslov, M. Pechenizkiy, Stress detection from speech and galvanic skin response signals, in IEEE 26th International Symposium on Computer-Based Medical Systems (CBMS) (IEEE, 2013), pp. 209–214

    Google Scholar 

  40. E.A. Kirchner, S.K. Kim, M. Tabie, H. Wöhrle, M. Maurus, F. Kirchner, An intelligent man-machine interfacemulti-robot control adapted for task engagement based on single-trial detectability of p300. Front. Hum. Neurosci. 10(2016)

    Google Scholar 

  41. R. Parasuraman, T.B. Sheridan, C.D. Wickens, A model for types and levels of human interaction with automation. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 30(3), 286–297 (2000)

    Article  Google Scholar 

  42. B.D. Simpson, R.S. Bolia, M.H. Draper, Spatial audio display concepts supporting situation awareness for operators of unmanned aerial vehicles, Human Performance, Situation Awareness, and Automation: Current Research and Trends HPSAA II, Volumes I and II, vol. 2 (2013), p. 61

    Google Scholar 

  43. S. Scheggi, M. Aggravi, F. Morbidi, D. Prattichizzo, Cooperative human-robot haptic navigation, in IEEE International Conference on Robotics and Automation (ICRA) (IEEE, 2014), pp. 2693–2698

    Google Scholar 

  44. C.E. Lathan, M. Tracey, The effects of operator spatial perception and sensory feedback on human-robot teleoperation performance. Presence Teleoper. Virtual Environ. 11(4), 368–377 (2002)

    Article  Google Scholar 

  45. V.M. Monajjemi, S. Pourmehr, S.A. Sadat, F. Zhan, J. Wawerla, G. Mori, R. Vaughan, Integrating multi-modal interfaces to command uavs, in Proceedings of the 2014 ACM/IEEE International Conference on Human-Robot Interaction (ACM, 2014), pp. 106–106

    Google Scholar 

  46. S. Kavitha, S. Veena, R. Kumaraswamy, Development of automatic speech recognition system for voice activated ground control system, in International Conference on Trends in Automation, Communications and Computing Technology (I-TACT-15), vol. 1 (IEEE, 2015), pp. 1–5

    Google Scholar 

  47. T. Mantecón, C.R. del Blanco, F. Jaureguizar, N. García, New generation of human machine interfaces for controlling uav through depth-based gesture recognition, in SPIE Defense+ Security, International Society for Optics and Photonics (2014), pp. 90840C–90840C

    Google Scholar 

  48. J. Nagi, A. Giusti, G.A. Di Caro, L.M. Gambardella, Human control of uavs using face pose estimates and hand gestures, in Proceedings of the 2014 ACM/IEEE International Conference on Human-Robot Interaction, (ACM, 2014), pp. 252–253

    Google Scholar 

  49. M. Hou, H. Zhu, M. Zhou, G.R. Arrabito, Optimizing operator–agent interaction in intelligent adaptive interface design: a conceptual framework. IEEE Trans. Syst. Man Cybern. Part C (Applications and Reviews) 41(2), 161–178 (2011)

    Google Scholar 

  50. J.L. Drury, J. Richer, N. Rackliffe, M.A. Goodrich, Comparing situation awareness for two unmanned aerial vehicle human interface approaches. Technical report, Mitre Corp Bedford MA (2006)

    Google Scholar 

  51. K. Foit, Mixed reality as a tool supporting programming of the robot, in Advanced Materials Research, vol. 1036 (Trans Tech Publ, 2014), pp. 737–742

    Article  Google Scholar 

  52. D.C. Niehorster, L. Li, M. Lappe, The accuracy and precision of position and orientation tracking in the htc vive virtual reality system for scientific research. i-Perception 8(3), 2041669517708205 (2017)

    Google Scholar 

  53. F. Navarro, J. Fdez, M. Garzon, J.J. Roldán, A. Barrientos, Integrating 3d reconstruction and virtual reality: a new approach for immersive teleoperation, in Robot 2017: Third Iberian Robotics Conference (Springer, 2018), pp. X–Y

    Google Scholar 

  54. S.H. Juan, F.H. Cotarelo, Multi-Master Ros Systems, Institut de robotics and industrial informatics (2015)

    Google Scholar 

  55. C. Crick, G. Jay, S. Osentoski, B. Pitzer, O.C. Jenkins, Rosbridge: Ros for non-ros users, in Robotics Research (Springer, Berlin, 2017), pp. 493–504

    Google Scholar 

  56. P. Codd-Downey, A.S.H.W. Mojiri Forooshani, M. Jenkin, From ros to unity: leveraging robot and virtual environment middleware for immersive teleoperation (2014)

    Google Scholar 

  57. E. Peña-Tapia, J.J. Roldán Gómez, M. Garzón, A. Martín-Barrio, A. Barrientos-Cruz, Interfaz de control para un robot manipulador mediante realidad virtual (2017)

    Google Scholar 

  58. I.A. Sucan, S. Chitta, Moveit!. http://moveit.ros.org (2013)

  59. I.A. Şucan, M. Moll, L.E. Kavraki, The open motion planning library. IEEE Robot. Autom. Mag. 19(4), 72–82 (2012)

    Article  Google Scholar 

  60. J.J. Kuffner, S.M. LaValle, Rrt-connect: an efficient approach to single-query path planning, in Proceedings of the ICRA’00 IEEE International Conference on Robotics and Automation, vol. 2 (IEEE, 2000), pp. 995–1001

    Google Scholar 

  61. R. Guzman, R. Navarro, M. Beneto, D. Carbonell, Robotnikprofessional service robotics applications with ros, in Robot Operating System (ROS) (Springer, Berlin, 2016), pp. 253–288

    Google Scholar 

  62. R. Guzmán, R. Navarro, M. Cantero, J. Ariño, Robotnikprofessional service robotics applications with ros (2), in Robot Operating System (ROS) (Springer, Berlin, 2017), pp. 419–447

    Google Scholar 

Download references

Acknowledgements

This work is framed on SAVIER (Situational Awareness Virtual EnviRonment) Project, which is both supported and funded by Airbus Defence and Space. The research leading to these results has received funding from the RoboCity2030-III-CM project (Robótica aplicada a la mejora de la calidad de vida de los ciudadanos. Fase III; S2013/MIT-2748), funded by Programas de Actividades I\(+\)D en la Comunidad de Madrid and cofunded by Structural Funds of the EU, and from the DPI2014-56985-R project (Protección robotizada de infraestructuras críticas) funded by the Ministerio de Economía y Competitividad of Gobierno de España.

Author information

Authors and Affiliations

  1. Centro de Automática y Robótica (UPM-CSIC), Universidad Politécnica de Madrid, Calle José Gutiérrez Abascal, 2, 28006, Madrid, Spain

    Juan Jesús Roldán, Elena Peña-Tapia, Jorge de León, Mario Garzón, Jaime del Cerro & Antonio Barrientos

  2. IRIDIA, Université Libre de Bruxelles (ULB), CP194/6, Av. F. Roosevelt 50, B-1050, Brussels, Belgium

    David Garzón-Ramos

Authors
  1. Juan Jesús Roldán
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Elena Peña-Tapia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David Garzón-Ramos
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jorge de León
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mario Garzón
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jaime del Cerro
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Antonio Barrientos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Juan Jesús Roldán .

Editor information

Editors and Affiliations

  1. College of Computer Science and Information Systems, Prince Sultan University, Riyadh, Saudi Arabia

    Anis Koubaa

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer International Publishing AG, part of Springer Nature

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Roldán, J.J. et al. (2019). Multi-robot Systems, Virtual Reality and ROS: Developing a New Generation of Operator Interfaces. In: Koubaa, A. (eds) Robot Operating System (ROS). Studies in Computational Intelligence, vol 778. Springer, Cham. https://doi.org/10.1007/978-3-319-91590-6_2

Download citation

Publish with us

Policies and ethics

Search

Navigation