Skip to main content
SpringerLink
Log in

Teleoperation of industrial robot manipulators based on augmented reality

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

This research develops a novel teleoperation for robot manipulators based on augmented reality. The proposed interface is equipped with full capabilities in order to replace the classical teach pendant of the robot for carrying out teleoperation tasks. The proposed interface is based on an augmented reality headset for projecting computer-generated graphics onto the real environment and a gamepad to interact with the computer-generated graphics and provide robot commands. In order to demonstrate the benefits of the proposed method, several usability tests were conducted using a 6R industrial robot manipulator in order to compare the proposed interface and the conventional teach pendant interface for teleoperation tasks. In particular, the results of these usability tests show that the proposed approach is more intuitive, ergonomic, and easy to use. Furthermore, the comparison results also show that the proposed method clearly improves the velocity of the teleoperation task, regardless of the user’s previous experience in robotics and augmented reality technology.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. Vector \(\mathbf {q}=\begin {bmatrix} q_{1} & {\cdots } & q_{n} \end {bmatrix}^{\mathrm {T}}\) is the robot configuration, where n is the number of joints of the robot.

  2. Vector \(\mathbf {p}=\begin {bmatrix} x & y & z & \alpha & \upbeta & \gamma \end {bmatrix}^{\mathrm {T}}\) is the robot pose, where the orientation is given by roll α, pitch β, and yaw γ angles.

References

  1. ABB: Abb teach pendant (Accessed 04/24/2020)

  2. Attig C, Wessel D, Franke T (2017) Assessing personality differences in human-technology interaction: an overview of key self-report scales to predict successful interaction. In: Stephanidis C (ed) HCI International 2017 – posters’ extended abstracts. Springer International Publishing, Cham, pp 19–29

  3. Bandala M, West C, Monk S, Montazeri A, Taylor CJ (2019) Vision-based assisted tele-operation of a dual-arm hydraulically actuated robot for pipe cutting and grasping in nuclear environments. Robotics 8(2):42

    Article  Google Scholar 

  4. Bermejo C, Hui P (2017) A survey on haptic technologies for mobile augmented reality. arXiv:1709.00698

  5. Billard A, Calinon S, Dillmann R, Schaal S (2008) Robot programming by demonstration. Springer, Berlin, pp 1371–1394

    Google Scholar 

  6. Bostanci E, Kanwal N, Ehsan S, Clark AF (2013) User tracking methods for augmented reality. In: International Journal of Computer Theory and Engineering, pp 93–98

  7. Brooke J (1996) SUS-A quick and dirty usability scale Usability evaluation in industry. CRC Press, ISBN: 9780748404605

  8. Cardoso JCS (2016) Comparison of gesture, gamepad, and gaze-based locomotion for VR worlds. In: Proceedings of the 22nd ACM conference on virtual reality software and technology. Association for Computing Machinery, New York, pp 319–320

  9. Circuit 1 demonstration: https://media.upv.es/player/?id=8be7f8e0-99b4-11ea-a399-57da8aaa21ee (Accessed 05/14/2020)

  10. Circuit 2 demonstration: https://media.upv.es/player/?id=afda6530-99b4-11ea-a399-57da8aaa21ee (Accessed 05/14/2020)

  11. Circuit 3 demonstration: https://media.upv.es/player/?id=56c62d20-99b5-11ea-a399-57da8aaa21ee (Accessed 05/14/2020)

  12. Circuit 4 demonstration: https://media.upv.es/player/?id=9461f740-99b5-11ea-a399-57da8aaa21ee (Accessed 05/14/2020)

  13. Codd-Downey R, Jenkin M (2018) Wireless teleoperation of an underwater robot using li-fi. In: 2018 IEEE International conference on information and automation (ICIA), pp 859–864

  14. Craig AB (2013) Chapter 2 - augmented reality concepts. In: Craig AB (ed) Understanding augmented reality. Morgan Kaufmann, Boston, pp 39–67

  15. Fanuc: Fanuc ipendant (Accessed 04/24/2020)

  16. Ferreira A, Bastos-filho TF, Null Cheein FA, Postigo JF, Carelli R (2006) Teleoperation of an industrial manipulator through a tcp/ip channel using eeg signals. 2006 IEEE International Symposium on Industrial Electronics 4:3066–3071

  17. Fondazione Istituto Italiano di Tecnologia : force/torque sensors (2015). Accessed 04/04/2020

  18. Franke T, Attig C, Wessel D (2018) A personal resource for technology interaction: development and validation of the affinity for technology interaction (ati) scale. Int J Human-Computer Interac 0(0):1–12

    Google Scholar 

  19. Gadre SY, Rosen E, Chien G, Phillips E, Tellex S, Konidaris G (2019) End-user robot programming using mixed reality. In: 2019 International conference on robotics and automation (ICRA), pp 2707–2713

  20. Grahn I The vuforia sdk and unity3d game engine : evaluating performance on android devices

  21. Hart SG, Staveland LE (1988) Development of nasa-tlx (task load index): results of empirical and theoretical research. In: Hancock PA, Meshkati N (eds) Human mental workload, advances in psychology, vol 52, North-Holland, pp 139–183

  22. Hess R (2010) Blender foundations: the essential guide to learning blender 2.6 focal press

  23. Isop WA, Gebhardt C, Nageli T, Fraundorfer F, Hilliges O, Schmalstieg D (2019) High-level teleoperation system for aerial exploration of indoor environments. Frontiers in Robotics and AI 6:95

    Article  Google Scholar 

  24. Jackson S (2015) Unity 3D UI essentials. Packt Publishing

  25. Kitson A, Hashemian AM, Stepanova ER, Kruijff E, Riecke BE (2017) Comparing leaning-based motion cueing interfaces for virtual reality locomotion. In: 2017 IEEE Symposium on 3d user interfaces (3DUI), pp 73–82

  26. Kuka: Kuka smartpad teach pendant (Accessed 04/24/2020)

  27. Li C, Fahmy A, Sienz J (2019) An augmented reality based human-robot interaction interface using kalman filter sensor fusion. Sensors 19(20):4586

    Article  Google Scholar 

  28. Liang C, Liu C, Liu X, Cheng L, Yang C (2019) Robot teleoperation system based on mixed reality. In: 2019 IEEE 4Th international conference on advanced robotics and mechatronics (ICARM), pp 384–389

  29. Marinho MM, Adorno BV, Harada K, Deie K, Deguet A, Kazanzides P, Taylor RH, Mitsuishi M (2019) A unified framework for the teleoperation of surgical robots in constrained workspaces. In: 2019 international conference on robotics and automation (ICRA)

  30. Microsoft Hololens (2nd gen) hardware details: https://www.microsoft.com/en-us/hololens/hardware (Accessed 04/24/2020)

  31. Munawar A, Fischer G (2016) A surgical robot teleoperation framework for providing haptic feedback incorporating virtual environment-based guidance. Frontiers in Robotics and AI 3:47

    Article  Google Scholar 

  32. Muñoz A, Mahiques X, Solanes JE, Martí A, Gracia L, Tornero J (2019) Mixed reality-based user interface for quality control inspection of car body surfaces. J Manuf Syst 53:75–92

    Article  Google Scholar 

  33. Muñoz A, Martí A, Mahiques X, Gracia L, Solanes JE, Tornero J (2020) Camera 3D positioning mixed reality-based interface to improve worker safety, ergonomics and productivity. CIRP J Manuf Sci Technol 28:24–37

    Article  Google Scholar 

  34. Park DY (2017) Mrtk: Open-source building blocks for windows mixed reality experiences. https://github.com/cre8ivepark/

  35. Rosen E, Whitney D, Phillips E, Chien G, Tompkin J, Konidaris G, Tellex S (2019) Communicating and controlling robot arm motion intent through mixed-reality head-mounted displays. Int J Robot Res 38(12-13):1513–1526

    Article  Google Scholar 

  36. Siciliano B, Sciavicco L, Villani L, Oriolo G (2009) Robotics: modelling, planning and control. Springer, London

    Book  Google Scholar 

  37. Vitor R, Keller B, D’Angelo T, Azpurua H, Bianchi AGC, Delabrida S (2019) Collaborative teleoperation evaluation for drones. In: Proceedings of the 18th Brazilian symposium on human factors in computing systems, IHC ‘19. Association for Computing Machinery, New York

  38. Walker ME, Hedayati H, Szafir D (2019) Robot teleoperation with augmented reality virtual surrogates. In: 2019 14Th ACM/IEEE international conference on human-robot interaction (HRI), pp 202–210

  39. Wang D, Guo Y, Liu S, Zhang Y, Xu W, Xiao J (2019) Haptic display for virtual reality: progress and challenges. Virtual Reality & Intelligent Hardware 1(2):136–162

    Article  Google Scholar 

  40. Wei J, Ye G, Mullen T, Grundmann M, Ahmadyan A, Hou T (2019) Instant motion tracking and its applications to augmented reality

  41. Wonnacott TH, Wonnacott RJ (1990) Introductory statistics for business and economics. Wiley, New York

    MATH  Google Scholar 

  42. Xu P, Zeng Q, Zhang G, Zhu C, Zhu Z (2019) Design of control system and human-robot-interaction system of teleoperation underwater robot. In: Yu H, Liu J, Liu L, Ju Z, Liu Y, Zhou D (eds) Intelligent robotics and applications. Springer International Publishing, Cham, pp 649–660

  43. Yew AWW, Ong SK, Nee AYC (2017) Immersive augmented reality environment for the teleoperation of maintenance robots

  44. Zhao J, Allison RS (2019) Comparing head gesture, hand gesture and gamepad interfaces for answering yes/no questions in virtual environments. Virtual Reality, pp 1–9

Download references

Acknowledgments

This work was supported by the Spanish Government under the project DPI2017-87656-C2-1-R.

Author information

Authors and Affiliations

  1. Instituto IDF, Universitat Politècnica de València, Camino de Vera s/n, 46022, Valencia, Spain

    J. Ernesto Solanes, Adolfo Muñoz, Luis Gracia, Ana Martí & Josep Tornero

  2. Departament d’Enginyeria Electrònica, Universitat de València, Avda de la Universitat s/n, 46100, Burjassot, Spain

    Vicent Girbés-Juan

Authors
  1. J. Ernesto Solanes
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adolfo Muñoz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Luis Gracia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ana Martí
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Vicent Girbés-Juan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Josep Tornero
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to J. Ernesto Solanes.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Solanes, J.E., Muñoz, A., Gracia, L. et al. Teleoperation of industrial robot manipulators based on augmented reality. Int J Adv Manuf Technol 111, 1077–1097 (2020). https://doi.org/10.1007/s00170-020-05997-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-020-05997-1

Keywords

Search

Navigation