Skip to main content
SpringerLink
Log in

Design of a virtual reality training system for human–robot collaboration in manufacturing tasks

  • Original Paper
  • Published:
International Journal on Interactive Design and Manufacturing (IJIDeM) Aims and scope Submit manuscript

Abstract

This paper presents a highly interactive and immersive Virtual Reality Training System (VRTS) (“beWare of the Robot”) in terms of a serious game that simulates in real-time the cooperation between industrial robotic manipulators and humans, executing simple manufacturing tasks. The scenario presented refers to collaborative handling in tape-laying for building aerospace composite parts. The tools, models and techniques developed and used to build the “beWare of the Robot” application are described. System setup and configuration are presented in detail, as well as user tracking and navigation issues. Special emphasis is given to the interaction techniques used to facilitate implementation of virtual human–robot (HR) collaboration. Safety issues, such as contacts and collisions are mainly tackled through “emergencies”, i.e. warning signals in terms of visual stimuli and sound alarms. Mental safety is of utmost priority and the user is provided augmented situational awareness and enhanced perception of the robot’s motion due to immersion and real-time interaction offered by the VRTS as well as by special warning stimuli. The short-term goal of the research was to investigate users’ enhanced experience and behaviour inside the virtual world while cooperating with the robot and positive pertinent preliminary findings are presented and briefly discussed. In the longer term, the system can be used to investigate acceptability of H–R collaboration and, ultimately, serve as a platform for programming collaborative H–R manufacturing cells.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Unhelkar, V.V., Siu, H.C., Shah, J.A.: Comparative performance of human and mobile robotic assistants in collaborative fetch-and-deliver tasks. In: Proc. 2014 ACM/IEEE Int. Conf. Human–robot Interact—HRI ’14, pp. 82–89 (2014). doi:10.1145/2559636.2559655

  2. de Santos, P.G., Garcia, E., Sarria, J., Ponticelli, R., Reviejo, J.: A new manipulator structure for power-assist devices. Ind. Robot. An. Int. J. 37(5), 452–458 (2010). doi:10.1108/01439911011063272

    Article  Google Scholar 

  3. Knight, W.: Smart Robots Can Now Work Right Next to Auto Workers. http://goo.gl/EUbRNX (2013). Accessed 22 April 2014

  4. Rethink robotics: baxter robot overview. http://goo.gl/fRUqou (2014). Accessed 22 April 2014

  5. Coradeschi, S., Cesta, A., Cortellessa, G., Coraci, L., Gonzalez, J., Karlsson, L., Furfari, F., Loutfi, A., Orlandini, A., Palumbo, F., Pecora, F., von Rump, S., Stimec, A., Ullberg, J., Otslund, B.: GiraffPlus: combining social interaction and long term monitoring for promoting independent living, 2013. In: 6th Int. Conf. Hum. Syst. Interact, pp. 578–585 (2013). doi:10.1109/HSI.2013.6577883

  6. Cavallo, F., Aquilano, M., Bonaccorsi, M., Mannari, I., Carrozza, M.C., Dario, P.: Multidisciplinary approach for developing a new robotic system for domiciliary assistance to elderly people. In: Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. IEEE Eng. Med. Biol. Soc. Annu. Conf., pp. 5327–5330 (2011). doi:10.1109/HSI.2013.6577883

  7. Helander, M.G.: Ergonomics and safety considerations in the design of robotics workplaces: a review and some priorities for research. Int. J. Ind. Ergon. 6(2), 127–149 (1990). doi:10.1016/0169-8141(90)90018-W

    Article  Google Scholar 

  8. Working with robots: our friends electric. http://goo.gl/v8DhAK (2013). Accessed 22 April 2014

  9. Ogorodnikova, O.: Human Robot Interaction: The safety challenge (an integrated frame work for human safety). PhD dissertation, Budapest University of Technology and Economics, Faculty of Mechanical Engineering, Budapest (2010)

  10. Alami, R., Bicchi, A., Bischoff, R., Chatila, R., De Luca, A., De Santis, A.: Safe and dependable physical human–robot interaction in anthropic domains: State of the art and challenges. In: Bicchi, A., De Luca, A. (ed.) Procceedings IROS Workshop on pHRI—Physical Human–Robot Interaction in Anthropic Domains (2006)

  11. Hoffman, G., Breazeal, C.: Effects of anticipatory action on human-robot teamwork efficiency, fluency, and perception of team. Proceeding ACM/IEEE Int. Conf. Human-robot Interact. - HRI ’07. 1–8 (2007). doi:10.1145/1228716.1228718

  12. Wickens, C.D.: Situation awareness: review of Mica Endsley’s 1995 articles on situation awareness theory and measurement. Hum. Factors. 50(3), 397–403 (2008). doi:10.1518/001872008X288420

    Article  Google Scholar 

  13. Fuchs, P., Moreau, G., Moreau, G., Burkhardt, J.-M., Coquillart, S.: Le traité de la réalité virtuelle, vol. 2. L’interfaçage, l’immersion et l’interaction en environnement virtuel. Les Presses de l’Ecole des Mines de Paris, Paris (2006)

  14. Lin, F., Ye, L., Duffy, V., Su, C.-J.: Developing virtual environments for industrial training. Inf. Sci. (Ny) 140(1—-2), 153–170 (2002). doi:10.1016/S0020-0255(01)00185-2

    Article  MATH  Google Scholar 

  15. Shukla, C., Vazquez, M., Frank Chen, F.: Virtual manufacturing: an overview. Comput. Ind. Eng. 31(1–2), 79–82 (1996). doi:10.1016/0360-8352(96)00083-6

    Article  Google Scholar 

  16. Burdea, G., Coiffet, P.: Virtual Reality Technology, 2nd edn. Wiley-Interscience, New York (2003)

    Google Scholar 

  17. Duguleana, M., Barbuceanu, F.G., Mogan, G.: Virtual and mixed reality—new trends. In: Shumaker, R. (ed.) Virtual and mixed reality—new trends. Springer, Berlin, pp. 164–173 (2011). doi:10.1007/978-3-642-22021-0_19

  18. Zhu, J., Ong, S.K., Nee, A.Y.C.: A context-aware augmented reality system to assist the maintenance operators. Int. J. Interact. Des. Manuf. (2014). doi:10.1007/s12008-013-0199-7

    Google Scholar 

  19. Nathanael, D., Vosniakos, G.-C., Mosialos, S.: Cognitive task analysis for virtual reality training. In: Proceedings of the 28th Annual European Conference on Cognitive Ergonomics—ECCE ’10. ACM Press, New York, p. 241 (2010). doi:10.1145/1962300.1962350

  20. Burkhardt, J.-M., Bardy, B., Lourdeaux, D.: Immersion, Réalisme et Présence dans la conception et l ’ évaluation des Environnements Virtuels. Psychol. Française. 48(2), 35–42 (2003)

    Google Scholar 

  21. Pook, P.K., Ballard, D.H.: Deictic human/robot interaction. Rob. Auton. Syst. 18, 259–269 (1996). doi:10.1016/0921-8890(95)00080-1

    Article  Google Scholar 

  22. Morioka, M., Sakakibara, S.: A new cell production assembly system with human–robot cooperation. CIRP Ann. Manuf. Technol. 59, 9–12 (2010). doi:10.1016/j.cirp.2010.03.044

    Article  Google Scholar 

  23. Krüger, J., Lien, T.K., Verl, A.: Cooperation of human and machines in assembly lines. CIRP Ann. Manuf. Technol. 58, 628–646 (2009). doi:10.1016/j.cirp.2009.09.009

    Article  Google Scholar 

  24. Giesler, B., Steinhaus, P., Walther, M., Dillmann, R.: Sharing skills: using augmented reality for human–robot collaboration. In: Woods, A.J., Merritt, J.O., Benton, S.A., Bolas, M.T. (eds.) Proc. SPIE 5291, Stereoscopic Displays and Virtual Reality Systems XI, pp. 446–453 (2004). doi:10.1117/12.526689

  25. Corrales, Ja, Candelas, Fa, Torres, F.: Safe human–robot interaction based on dynamic sphere-swept line bounding volumes. Robot. Comput. Integr. Manuf. 27(1), 177–185 (2011). doi:10.1016/j.rcim.2010.07.005

    Article  Google Scholar 

  26. Arai, T., Kato, R., Fujita, M.: Assessment of operator stress induced by robot collaboration in assembly. CIRP Ann. Manuf. Technol. 59(1), 5–8 (2010). doi:10.1016/j.cirp.2010.03.043

    Article  Google Scholar 

  27. Charoenseang, S., Tonggoed, T.: Human–robot collaboration with augmented reality. In: Stephanidis, C. (ed.) HCI International 2011—Posters’ Extended Abstracts. Springer, Berlin, pp. 93–97 (2011). doi:10.1007/978-3-642-22095-1_19

  28. Fang, H.C., Ong, S.K., Nee, A.Y.C.: A novel augmented reality-based interface for robot path planning. Int. J. Interact. Des. Manuf. 8(1), 33–42 (2014). doi:10.1007/s12008-013-0191-2

    Article  Google Scholar 

  29. Du, G., Zhang, P.: Markerless human–robot interface for dual robot manipulators using Kinect sensor. Robot. Comput. Integr. Manuf. 30(2), 150–159 (2014). doi:10.1016/j.rcim.2013.09.003

    Article  Google Scholar 

  30. Wang, L., Schmidt, B., Nee, A.Y.C.: Vision-guided active collision avoidance for human–robot collaborations. Manuf. Lett. 1(1), 5–8 (2013). doi:10.1016/j.mfglet.2013.08.001

    Article  Google Scholar 

  31. Qiu, S., Fan, X., Wu, D., He, Q., Zhou, D.: Virtual human modeling for interactive assembly and disassembly operation in virtual reality environment. Int. J. Adv. Manuf. Technol. 69(9–12), 2355–2372 (2013). doi:10.1007/s00170-013-5207-3

    Article  Google Scholar 

  32. Zaeh, M., Roesel, W.: Safety aspects in a human–robot interaction scenario: a human worker is co-operating with an industrial robot. In: Kim, J.-H., et al. (eds.) Progress in robotics. Springer, Berlin, pp. 53–62 (2009). doi:10.1007/978-3-642-03986-7_7

  33. Neuhoefer, J.A., Bernhard, K., Schlick, C.M.: Embedded augmented reality training system for dynamic human–robot cooperation. NATO RTO-MP-HFM-169. RTO Human Factors and Medicine Panel (HFM) Workshop, RDECOM, pp. 1–14, Orlando, Florida, USA (2009)

  34. Weistroffer, V., Paljic, A., Callebert, L., Fuchs, P.: A methodology to assess the acceptability of human-robot collaboration using virtual reality. In: Proceedings of the 19th ACM Symposium on Virtual Reality Software and Technology—VRST ’13. ACM Press, New York, p. 39 (2013). doi:10.1145/2503713.2503726

  35. Matsas, E., Batras, D., Vosniakos, G.: Beware of the robot: a highly interactive and immersive virtual reality training application in robotic manufacturing systems. In: Emmanouilidis, C., Taisch, M., Kiritsis, D. (eds.) Advances in Production Management Systems. Competitive Manufacturing for Innovative Products and Services. Springer, Berlin, pp. 606–613 (2013). doi:10.1007/978-3-642-40352-1_76

  36. Bowman, D.A., Kruijff, E., LaViola, J.J., Poupyrev, I.: 3D User Interfaces: Theory and Practice. Addison Wesley Longman Publishing Co., Inc, Redwood City (2004)

    Google Scholar 

  37. Evolver 3D digital avatar website. www.evolver.com. Accessed 23 January 2012

  38. Usoh, M., Arthur, K., Whitton, M.C., Bastos, R., Steed, A., Slater, M., Brooks, F.P.: Walking, walking-in-place, flying, in virtual environments. In: Proc. 26th Annu. Conf. Comput. Graph. Interact. Tech.—SIGGRAPH ’99, pp. 359–364 (1999). doi:10.1145/311535.311589

  39. Unity online manual. http://goo.gl/kH1Hv8. Accessed 25 July 2014

  40. Miles, R.: Start here! Learn microsoft kinect API. O’Reilly Media Inc, Sebastopol (2012)

    Google Scholar 

  41. Vora, J., Nair, S., Gramopadhye, A.K., Duchowski, A.T., Melloy, B.J., Kanki, B.: Using virtual reality technology for aircraft visual inspection training: presence and comparison studies. Appl. Ergon. 33(6), 559–70 (2002). doi:10.1145/311535.311589

    Article  Google Scholar 

  42. Prothero, J.D., Parker, D.E., Furness III, T.A., Wells, M.J.: Towards a robust, quantitative measure for presence. In: Garland, D.J., Endsley, M.R. (eds.) Conference on Experimental Analysis and Measurement of Situational Awareness. Embry-Riddle Aeronautical University Press, Daytona Beach, pp. 359–366 (1995)

  43. Schuemie, M.J., van der Straaten, P., Krijn, M., van der Mast, C.A.: Research on presence in virtual reality: a survey. Cyberpsychol. Behav. 4(2), 183–201 (2001). doi:10.1089/109493101300117884

    Article  Google Scholar 

  44. Burkhardt, J.-M., Lourdeaux, D., Fuchs, P.: Conception d’un système de RV pour la formation des agents de conduite aux opérations en milieu ferroviaire. In: Richards, A., Grumbach, N. (eds.) Colloque Réalité Virtuelle et Cognition, pp. 123–132. ENST Paris, France (1999)

    Google Scholar 

Download references

Acknowledgments

This research has been co-financed by the European Union (European Social Fund—ESF) and Greek national funds through the Operational Program “Education and Lifelong Learning” of the National Strategic Reference Framework (NSRF)—Research Funding Program: Heracleitus II. Investing in knowledge society through the European Social Fund. The authors would also like to thank Dimitrios Batras, MSc, for his generous help in programming, his motivation and his helpful comments during the development of the application.

Author information

Authors and Affiliations

  1. School of Mechanical Engineering, National Technical University of Athens, Heroon Polytechniou 9, 15780 , Zografou-Athens, Greece

    Elias Matsas & George-Christopher Vosniakos

Authors
  1. Elias Matsas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. George-Christopher Vosniakos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to George-Christopher Vosniakos.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Matsas, E., Vosniakos, GC. Design of a virtual reality training system for human–robot collaboration in manufacturing tasks. Int J Interact Des Manuf 11, 139–153 (2017). https://doi.org/10.1007/s12008-015-0259-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12008-015-0259-2

Keywords

Search

Navigation