Skip to main content

Advertisement

SpringerLink
Log in

An augmented reality maintenance assistant with real-time quality inspection on handheld mobile devices

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

Advances of industry 4.0 are enabling augmented reality (AR) devices to be deployed across the manufacturing sector to enhance worker engagement and performance. This evolution of AR in the workspace is driving changes in how workers on the factory floor interact with the enterprise. AR is evolving to provide work instructions for assembly and maintenance. Recent demonstrations show the capability to allow for quality inspection to be integrally evaluated as part of the AR workflow. Previous integrations of quality include the ability for manual assessment to be logged into the enterprise as part of completing work instructions; however, this is largely based on the perception of the user to determine if they accurately performed the AR instructions. This study is designed to demonstrate the ability of an enterprise-level AR maintenance experience to go beyond manual inspection by embedding real-time quality assessment into the work instructions. Essentially, the goal is to enable the enterprise to perform a self-assessment of the quality of the work performed by maintenance personnel. Furthermore, the AR instructions enable additional instruction to empower maintenance personnel to correct their work prior to moving on to other tasks. The novelty of this work is focused on deployment on a mobile device with integrated vision inspection using the same device that is being used to deploy the AR work instructions. This work enumerates several essential qualities of mobile quality inspection tools and outlines some of the challenges associated with the development of such a system. Finally, results from testing the system demonstrate that handheld mobile devices can be used to capture inspection images while simultaneously deploying AR work instructions. Analysis of the image correction, timing for vision process of the quality check, and overall performance of the integrated system are presented.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Data availability

The authors declare that the data supporting the findings of this study are available within the article and its supplementary information files.

Code availability

The custom code that supports the findings of this study is available from the corresponding author, James Frandsen, upon reasonable request.

References

  1. LiveWorx (2021) Episode Three: digital transforms physical. 2021, PTC, Inc

  2. Chandra Sekaran S et al (2021) The implementation of virtual reality in digital factory—a comprehensive review. Int J Adv Manuf Technol 115:1349–1366

  3. Ho PT et al (2022) Study of augmented reality based manufacturing for further integration of quality control 4.0: a systematic literature review. Appl Sci 12(4):1961

    Article  Google Scholar 

  4. Tang A et al (2002) Experimental evaluation of augmented reality in object assembly task. in Proceedings. International Symposium on Mixed and Augmented Reality. IEEE Comput Soc

  5. Tang A et al (2003) Comparative effectiveness of augmented reality in object assembly. in Proceedings of the conference on Human factors in computing systems - CHI ′03. ACM Press

  6. Barbieri L and Marino E (2019) An augmented reality tool to detect design discrepancies: a comparison test with traditional methods, in lecture notes in computer science. Springer International Publishing. p. 99–110

  7. Bruno F et al (2019) An augmented reality tool to detect and annotate design variations in an Industry 4.0 approach. Int J Adv Manuf Technol 105(1–4):875–887

    Article  Google Scholar 

  8. Holm M et al (2017) Adaptive instructions to novice shop-floor operators using augmented reality. J Ind Prod Eng 34(5):362–374

    Google Scholar 

  9. Marino E et al (2021) An augmented reality inspection tool to support workers in Industry 4.0 environments. Comput Ind 127:103412

    Article  Google Scholar 

  10. Yang X et al (2020) Edge-based cover recognition and tracking method for an AR-aided aircraft inspection system. Int J Adv Manuf Technol 111(11–12):3505–3518

    Article  Google Scholar 

  11. Polvi J et al (2018) Handheld guides in inspection tasks: augmented reality versus picture. IEEE Trans Visual Comput Graphics 24(7):2118–2128

    Article  Google Scholar 

  12. Zhou J et al (2011) Applying spatial augmented reality to facilitate in-situ support for automotive spot welding inspection. in Proceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in Industry - VRCAI ′11. ACM Press

  13. Hartl AD et al (2016) Efficient verification of holograms using mobile augmented reality. IEEE Trans Visual Comput Graphics 22(7):1843–1851

    Article  Google Scholar 

  14. Runji JM and Lin C-Y (2019) Automatic optical inspection aided augmented reality-based PCBA inspection: a development. in 2019 IEEE Jordan International Joint Conference on Electrical Engineering and Information Technology (JEEIT). IEEE

  15. Runji JM, Lin C-Y (2020) Markerless cooperative augmented reality-based smart manufacturing double-check system: case of safe PCBA inspection following automatic optical inspection. Robot Comput-Integr Manuf 64:101957

    Article  Google Scholar 

  16. Odenthal B et al (2014) A comparative study of head-mounted and table-mounted augmented vision systems for assembly error detection. Hum Factors Ergon Manuf Serv Ind 24(1):105–123

    Article  Google Scholar 

  17. Yuan ML, Ong SK, Nee AYC (2008) Augmented reality for assembly guidance using a virtual interactive tool. Int J Prod Res 46(7):1745–1767

    Article  MATH  Google Scholar 

  18. Zauner J et al (2003) Authoring of a mixed reality assembly instructor for hierarchical structures. in The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings. IEEE Comput. Soc

  19. Zhang J, Ong SK, Nee AYC (2011) RFID-assisted assembly guidance system in an augmented reality environment. Int J Prod Res 49(13):3919–3938

    Article  Google Scholar 

  20. Kwon O-S, Park C-S, Lim C-R (2014) A defect management system for reinforced concrete work utilizing BIM, image-matching and augmented reality. Autom Constr 46:74–81

    Article  Google Scholar 

  21. Shin DH, Dunston PS (2009) Evaluation of augmented reality in steel column inspection. Autom Constr 18(2):118–129

    Article  Google Scholar 

  22. Ojer M et al (2020) Real-time automatic optical system to assist operators in the assembling of electronic components. Int J Adv Manuf Technol 107(5–6):2261–2275

    Article  Google Scholar 

  23. Ojer M et al (2020) Projection-based augmented reality assistance for manual electronic component assembly processes. Appl Sci 10(3):796

    Article  Google Scholar 

  24. Alves J et al (2019) Comparing spatial and mobile augmented reality for guiding assembling procedures with task validation. in 2019 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC). IEEE

  25. Alves J et al (2021) Using augmented reality and step by step verification in industrial quality control, in Human Systems Engineering and Design III. Springer International Publishing. p. 350–355

  26. Gupta A et al (2012) DuploTrack. in Proceedings of the 25th annual ACM symposium on User interface software and technology - UIST ′12. ACM Press

  27. Manuri F, Pizzigalli A, Sanna A (2019) A state validation system for augmented reality based maintenance procedures. Appl Sci 9(10):2115

    Article  Google Scholar 

  28. Wasenmuller O, Meyer M, Stricker D (2016) Augmented reality 3D discrepancy check in industrial applications. in 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE

  29. Alves JB et al (2021) Using augmented reality for industrial quality assurance: a shop floor user study. Int J Adv Manuf Technol 115(1–2):105–116

    Article  Google Scholar 

  30. Khuong BM et al (2014) The effectiveness of an AR-based context-aware assembly support system in object assembly. in 2014 IEEE Virtual Reality (VR). IEEE

  31. Nishihara A, Okamoto J (2015) Object recognition in assembly assisted by augmented reality system. in 2015 SAI Intelligent Systems Conference (IntelliSys). IEEE

  32. Pathomaree N, Charoenseang S (2005) Augmented reality for skill transfer in assembly task. in ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005. IEEE

  33. Wu L-C, Lin I-C, Tsai M-H (2016) Augmented reality instruction for object assembly based on markerless tracking. in Proceedings of the 20th ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games. ACM

  34. Okamoto J, Nishihara A (2016) Assembly assisted by augmented reality (A 3 R). Intelligent Systems and Applications. Springer, pp 281–300

    Chapter  Google Scholar 

  35. Feiner S, Macintyre B, Seligmann D (1993) Knowledge-based augmented reality. Commun ACM 36(7):53–62

    Article  Google Scholar 

  36. Liverani A, Amati G, Caligiana G (2004) A CAD-augmented reality integrated environment for assembly sequence check and interactive validation. Concurr Eng 12(1):67–77

    Article  Google Scholar 

  37. Westerfield G, Mitrovic A, Billinghurst M (2015) Intelligent augmented reality training for motherboard assembly. Int J Artif Intell Educ 25(1):157–172

    Article  Google Scholar 

  38. Alvarez H, Aguinaga I, Borro D (2011) Providing guidance for maintenance operations using automatic markerless augmented reality system. in 2011 10th IEEE International Symposium on Mixed and Augmented Reality. IEEE

  39. Dalle Mura M, Dini G (2021) An augmented reality approach for supporting panel alignment in car body assembly. J Manuf Syst 59:251–260

    Article  Google Scholar 

  40. Chi HL et al (2022) Rebar inspection integrating augmented reality and laser scanning. Autom Constr 136:104183

    Article  Google Scholar 

  41. Lee H et al (2018) A framework for process model based human-robot collaboration system using augmented reality, in Advances in Production Management Systems. Smart Manufacturing for Industry 4.0. Springer International Publishing. p. 482–489

  42. Angelopoulos J, Mourtzis D (2022) An intelligent product service system for adaptive maintenance of engineered-to-order manufacturing equipment assisted by augmented reality. Appl Sci 12(11):5349

    Article  Google Scholar 

  43. Antonelli D, Astanin S (2015) Enhancing the quality of manual spot welding through augmented reality assisted guidance. Procedia CIRP 33:556–561

    Article  Google Scholar 

  44. Hakkarainen M, Woodward C, Billinghurst M (2008) Augmented assembly using a mobile phone. in 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality. IEEE

  45. Liu C et al (2012) Evaluating the benefits of real-time feedback in mobile augmented reality with hand-held devices. in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM

  46. Mourtzis D, Xanthi F, Zogopoulos V (2019) An adaptive framework for augmented reality instructions considering workforce skill. Procedia CIRP 81:363–368

    Article  Google Scholar 

  47. Schlueter JA (2018) Remote maintenance assistance using real-time augmented reality authoring, in ProQuest Dissertations and Theses. Iowa State University: Ann Arbor. p. 72

  48. Segovia D et al (2015) Augmented reality as a tool for production and quality monitoring. Procedia Comput Sci 75:291–300

    Article  Google Scholar 

  49. Chouchene A et al (2022) Augmented reality-based framework supporting visual inspection for automotive industry. Appl Syst Innov 5(3):48

    Article  Google Scholar 

  50. Marques B et al (2022) Remote collaboration in maintenance contexts using augmented reality: insights from a participatory process. Int J Interact Design Manuf (IJIDeM) 16(1):419–438

    Article  Google Scholar 

  51. Li S, Zheng P, Zheng L (2021) An AR-assisted deep learning-based approach for automatic inspection of aviation connectors. IEEE Trans Industr Inf 17(3):1721–1731

    Article  Google Scholar 

  52. Ferraguti F et al (2019) Augmented reality based approach for on-line quality assessment of polished surfaces. Robot Comput-Integr Manuf 59:158–167

    Article  Google Scholar 

  53. Mourtzis D, Zogopoulos V, Xanthi F (2019) Augmented reality application to support the assembly of highly customized products and to adapt to production re-scheduling. Int J Adv Manuf Technol 105(9):3899–3910

    Article  Google Scholar 

  54. Danielsson O et al (2018) Operators perspective on augmented reality as a support tool in engine assembly. Procedia CIRP 72:45–50

    Article  Google Scholar 

  55. Lampen E et al (2019) Combining simulation and augmented reality methods for enhanced worker assistance in manual assembly. Procedia CIRP 81:588–593

    Article  Google Scholar 

  56. Siyaev A, Jo G-S (2021) Towards aircraft maintenance metaverse using speech interactions with virtual objects in mixed reality. Sensors 21(6):2066

    Article  Google Scholar 

  57. Laviola E et al (2023) In-situ or side-by-side? A user study on augmented reality maintenance instructions in blind areas. Comput Ind 144:103795

    Article  Google Scholar 

  58. Henderson S, Feiner S (2011) Exploring the benefits of augmented reality documentation for maintenance and repair. IEEE Trans Visual Comput Graphics 17(10):1355–1368

    Article  Google Scholar 

  59. Qeshmy DE et al (2019) Managing human errors: augmented reality systems as a tool in the quality journey. Procedia Manuf 28:24–30

    Article  Google Scholar 

  60. Schlagowski R, Merkel L, Meitinger C (2017) Design of an assistant system for industrial maintenance tasks and implementation of a prototype using augmented reality. in 2017 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM). IEEE

  61. Zhu J, Ong SK, Nee AYC (2014) A context-aware augmented reality system to assist the maintenance operators. Int J Interact Des Manuf (IJIDeM) 8(4):293–304

    Article  Google Scholar 

  62. Liu C et al (2017) Augmented reality-assisted intelligent window for cyber-physical machine tools. J Manuf Syst 44:280–286

    Article  Google Scholar 

  63. Wang Y et al (2018) Mechanical assembly assistance using marker-less augmented reality system. Assem Autom 38(1):77–87

    Article  Google Scholar 

  64. Loch F, Quint F, Brishtel I (2016) Comparing video and augmented reality assistance in manual assembly. in 2016 12th International Conference on Intelligent Environments (IE). IEEE

  65. Radkowski R, Herrema J, Oliver J (2015) Augmented reality-based manual assembly support with visual features for different degrees of difficulty. Int J Hum-Comput Interact 31(5):337–349

    Article  Google Scholar 

  66. Festo (2022) CP Lab | Festo USA. Available from: https://www.festo.com/us/en/e/technical-education/learning-systems/factory-automation-and-industry-4-0/learning-factories/cp-systems-large-scale-industry-4-0-learning-factories/cp-lab-id_36133/. Accessed 24 May 2022

  67. UR3e Technical Details (2020) @www_universal-robots_com. p. https://www.universal-robots.com/media/1802780/ur3e-32528_ur_technical_details_.pdf. Accessed 9 March 2022

  68. Lowe DG (1999) Object recognition from local scale-invariant features. in Proceedings of the Seventh IEEE International Conference on Computer Vision. IEEE

  69. Muja M, Lowe DG (2009) Fast approximate nearest neighbors with automatic algorithm configuration. VISAPP (1) 2(331–340):2

    Google Scholar 

  70. 1 in 2 visitors abandon a website that takes more than 6 seconds to load - Digital.com (2022) 16 January 2022]; Available from: https://digital.com/1-in-2-visitors-abandon-a-website-that-takes-more-than-6-seconds-to-load/. Accessed 16 January 2022

Download references

Acknowledgements

PTC, the parent company to Vuforia, ThingWorx, and Kepware, donated licenses for each of these software to the lab so they could be used in the development of the proposed system. Additionally, Festo Didactic generously supported the integration on their 4-station CP Lab, which was used for this project’s test use case.

Funding

This work was supported by Northrop Grumman Corporation via contract #4800056662.

Author information

Authors and Affiliations

  1. Department of Manufacturing Engineering, Brigham Young University, Provo, UT, 84602, USA

    James Frandsen & Yuri Hovanski

  2. Department of Engineering, Northrop Grumman Corporation, 9160 UT-83, Corinne, UT, 84307, USA

    Joe Tenny

  3. Cell and Gene Therapy, Thermo Fisher Scientific, 5781 Van Allen Way, Carlsbad, CA, 92008, USA

    Walter Frandsen Jr.

Authors
  1. James Frandsen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joe Tenny
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Walter Frandsen Jr.
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuri Hovanski
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study’s conception, design, and development. The first draft of the manuscript was written by James Frandsen and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to James Frandsen.

Ethics declarations

Ethics approval

The authors confirm that this manuscript is original and has not been published, nor is it currently under consideration for publication elsewhere.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (MOV 316983 kb)

Supplementary file2 (MOV 294366 kb)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Frandsen, J., Tenny, J., Frandsen, W. et al. An augmented reality maintenance assistant with real-time quality inspection on handheld mobile devices. Int J Adv Manuf Technol 125, 4253–4270 (2023). https://doi.org/10.1007/s00170-023-10978-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-023-10978-1

Keywords

Search

Navigation