Augmented assembly technologies based on 3D bare-hand interaction

doi:10.1016/j.cirp.2011.03.001

CIRP Annals

Volume 60, Issue 1, 2011, Pages 1-4

https://doi.org/10.1016/j.cirp.2011.03.001 Get rights and content

Abstract

Augmented reality has been applied to develop augmented assembly systems. However, most reported studies used pre-defined assembly information; AR is predominantly used to display information and interactions between users and the augmented environment are limited. This paper presents 3D bare-hand interaction in an augmented assembly environment to manipulate and assemble virtual components. A hybrid method based on constraint analysis is presented, which interprets users’ manual assembly intents robustly without the need for auxiliary CAD information. Algorithms for assembly constraint recognition, assembly tool operations and assembly sequence evaluation have been formulated. An augmented assembly system has been developed in this research.

Introduction

In recent years, virtual reality (VR) and virtual prototyping (VP) techniques [1], [2] have been widely used to simulate and evaluate assembly in the early design stage. However, the assembly planning experience is limited to a pure virtual environment (VE) due to a lack of real spatial feeling and suitable sensory feedback. A large amount of computation resources is also needed to simulate a complicated assembly process in a pure VE.

Augmented assembly (AA) is an application of augmented reality (AR) in assembly where an augmented environment (AE) is created, in which virtual objects are combined with the real environment to enhance the assembly design and planning process. An AE for product assembly design combines physical parts, real feedback and virtual contents to analyze the behaviors and properties of future products. Thus, the benefits of physical prototyping and VP can be combined. AR has been applied in manual assembly stations planning [3], product assembly guidance in product assembly [4], [5], augmenting digital virtual prototypes with physical products [6], dataglove-based virtual assembly [7] and replacing physical manuals with augmented virtual contents [8].

In many reported AA systems, assembly information, e.g., assembly features, predefined assembly constraints, etc., has to be extracted from CAD data, and this limits their impact as high preparation time is required for each application [9]. There is limited interaction between the users and the systems.

This research presents an AA system that interprets users’ manual assembly intents, supports on-line constraint recognition, and provides a robust 3D bare-hand interface to allow realistic visual feedback during assembly (Fig. 1). A method to realize interactive constraint-based AA using natural bare-hand interactions has been developed to increase interactions between users and virtual components to achieve an active participation of the users in the AA process. A bare-hand interaction augmented assembly (BHAA) system has been developed (Fig. 2).

Section snippets

3D bare-hand interaction method

Many tasks in AA systems are manipulation tasks with many degrees-of-freedom (DOFs), leading to greater difficulties in manipulating the virtual objects using traditional interaction methods, e.g., a mouse. To achieve natural and intuitive human-computer interaction (HCI), human hands can be used as interaction devices in AEs. Compared with traditional HCI devices, bare-hand is less intrusive and more convenient for the users to interact with virtual contents and explore the 3D AEs. Natural HCI

Assembly data management

A tri-layer assembly data structure (TADS) is used for assembly data management in BHAA. The first layer consists of geometric information. In the BHAA system, the environment does not need to be prepared a priori. During initialization, CAD models of the assembly parts and components are loaded into the BHAA system. The surfaces of these CAD models are enumerated and discretized into triangle tessellations to reconstruct continuous geometric entities, which are stored for display using OpenGL

Augmented assembly process

BHAA consists of a HCI interface and an AE, where virtual models are superimposed over the real scene. Users can perform assembly operations via the HCI interface with these virtual models in the AE. With the 3DNBHI interface, users can manipulate and assemble two different parts more intuitively and realistically. When these two parts are sufficiently close to each other, the user can adjust the positions and orientations of these parts easily and efficiently to trigger the assembly intent

Assembly sequence evaluation and feedback

Using bare hands in the BHAA system, the user can detect possible design faults easily. For example, when a part interferes with another part, this can be detected easily and feedback can be provided to product designers for design improvement. In traditional assembly planning, it is often hard to evaluate precisely the motions and complex manipulations needed. In the BHAA system, this can be achieved by assembling a set of virtual components using bare hands to identify the difficulties and

Implementation and case study

The BHAA system works well and consistently at about 15 frames per second for a 512 × 384 frame resolution. The accuracy of the system is determined by the accuracy of the fingertip detection method which has a RMS error of 1–2 mm in all axes.

The exploded view of the roller bracket case study is shown in Fig. 4. In Fig. 4(a), the user grasps the roller with his right hand and the left bush with his left hand and assembles these components. When the parts are colliding, the system analyzes the

Conclusion and future work

A robust and efficient system integrating natural bare-hand interaction with an AR-assisted methodology for constraint-based augmented assembly is presented. A 3D dual-handed interaction interface is provided to facilitate AA. A contribution of this research is the robust feature-based automatic constraint-based assembly recognition algorithm which can interpret a user's assembly intent robustly without the need for any a priori auxiliary CAD information. Assembly tools operations algorithms

References (13)

S.C.-Y. Lu et al.
Virtual and Augmented Reality Technologies for Product Realization
Annals of the CIRP
(1999)
K. Feldmann et al.
Development of New Concepts and Software Tools for the Optimization of Manual Assembly Systems
Annals of the CIRP
(2003)
G. Reinhart et al.
Integrating Augmented Reality in the Assembly Domain – Fundamentals, Benefits and Applications
Annals of the CIRP
(2003)
S.K. Ong et al.
Augmented Reality Aided Assembly Design and Planning
Annals of the CIRP
(2007)
R.D. Yang et al.
Virtual Assembly Technologies based on Constraint and DOF Analysis
Robotics and Computer-Integrated Manufacturing
(2007)
J. Saaski et al.
Integration of Design and Assembly using Augmented Reality
Micro-Assembly Technologies and Applications
(2008)

There are more references available in the full text version of this article.

Cited by (69)

Assessing user performance in augmented reality assembly guidance for industry 4.0 operators
2024, Computers in Industry
In the realm of smart manufacturing, Augmented Reality (AR) technology has gained increasing attention among researchers and manufacturers due to its practicality and adaptability. For this reason, it has been widely embraced in various industrial fields, especially for helping operators assemble products. Despite its widespread adoption, there is a debate in the research community about how effective AR is for improving user performance in assembly tasks, particularly when using handheld devices. These disparities can be attributed to differences in experimental approaches, such as the frequent use of qualitative methods, the inclusion of non-representative users, and the limited number of comprehensive case studies.
In response to this, the paper delved into the benefits of AR applications, with a specific focus on measuring user performance and the cognitive workload perceived by users during assembly activities. To this end, an AR assembly guidance tool has been developed to assist users during assembly tasks, running on a mobile device, specifically a tablet, for freedom of movement and high portability. Experimentation involved the assembly of a comprehensive case study and a diverse user group, allowing the comparison representative users and experienced industrial operators. The results were promising, indicating that AR technology effectively enhances user performance during assembly-guided activities compared to conventional methods, particularly when users are unfamiliar with the task at hand. This study brings valuable insights by addressing previous research limitations and providing strong evidence of AR's positive impact on user performance in real-world assembly scenarios.
Head-mounted display augmented reality in manufacturing: A systematic review
2023, Robotics and Computer-Integrated Manufacturing
Head-mounted display (HMD) augmented reality (AR) has attracted more and more attention in manufacturing activities, as it enables operators to access visual guidance in front of their view directly while freeing human's two hands. Nevertheless, HMD AR has not been widely adopted in manufacturing fields as humans expected since the release of Google Glass in 2012, and thus it is important to understand the related issues arising from the actual deployments of HMD AR on the shop floor. To the best of the authors’ knowledge, there have not been comprehensive discussions on HMD AR in manufacturing from a holistic perspective. This article aims to provide an extensive map for the distribution of HMD AR in various manufacturing activities and a systematic overview of underlying technical perspectives associated with their actual industrial applications between 2010 and 2022, involving AR visualization, tracking and registration, context awareness, human-machine interaction, as well as ergonomics and usability, which are significant for the actual AR deployments for human-centric manufacturing in Industry 5.0. It is also worth mentioning that this work presents a historical overview of the current research on the development of HMD AR, as well as a summary of the existing methods and open problems for HMD AR in manufacturing. It is helpful to understand the current technical situations of HMD AR while providing insights to deploy industrial AR applications and perform academic research in the future.
A comprehensive review of augmented reality-based instruction in manual assembly, training and repair
2022, Robotics and Computer-Integrated Manufacturing
Citation Excerpt :
The last and most important point is that, compared with conventional instructions, AR instructions have more natural, intuitive interactivity and three-dimensional performance [6,40]. Users can directly interact with them using bare-hand input [41]. Considering the above advantages and the problems in manual assembly assistance, AR instruction may be one of the most promising tools in manual assembly and repair.
In recent years, the multi-dimensional, multi-faceted, and multi-level applications of augmented reality/mixed reality have attracted more and more attention from researchers in the field of smart manufacturing. However, the disadvantage is that AR interaction cues supporting user cognition have not been fully analyzed and summarized in manual assembly, training and repair. This article reviews the current research status, projects and technical characteristics of manual operation instructions in the past 30 years, and extensively discusses these latest works. It is worth mentioning that this article provides comprehensive academic information for the development of AR-based assembly instructions, thereby providing unique insights for researchers in related fields. In short, it will help researchers designing AR instructions.
Challenges and opportunities on AR/VR technologies for manufacturing systems in the context of industry 4.0: A state of the art review
2022, Journal of Manufacturing Systems
Citation Excerpt :
Immersive technologies such as AR and VR are being used to assist workers in overcoming these obstacles and challenges by providing support, simulation, and assistance in enhancing industrial processes prior to their implementation in a production environment. This will result in cost savings in reworks and revisions to manufacturing activities such as product design, process planning, machining, and so on [5,6]. The immersive technologies such as AR and VR were first used by retailers to promote their products and services to the customers.
Manufacturing industries are currently experiencing the fourth revolution with penetration of new immersive technologies extremely focused on the flexible manufacturing and human machine interaction. The immersive technologies such as Augmented Reality (AR) and Virtual Reality (VR) have received much attention among the manufacturing industries to achieve mass production and mass customization for different product variants. This paper demonstrates the research progress and developments in the AR/VR technologies for product design and evaluation, Repair & Maintenance, Assembly, Warehouse management, Quality control, Plant layout and CNC simulation. The critical challenges in AR/VR technology implementation at hardware and software level and opportunities for the further developments are discussed.
Systematic literature review on augmented reality in smart manufacturing: Collaboration between human and computational intelligence
2021, Journal of Manufacturing Systems
Smart manufacturing offers a high level of adaptability and autonomy to meet the ever-increasing demands of product mass customization. Although digitalization has been used on the shop floor of modern factory for decades, some manufacturing operations remain manual and humans can perform these better than machines. Under such circumstances, a feasible solution is to have human operators collaborate with computational intelligence (CI) in real time through augmented reality (AR). This study conducts a systematic review of the recent literature on AR applications developed for smart manufacturing. A classification framework consisting of four facets, namely interaction device, manufacturing operation, functional approach, and intelligence source, is proposed to analyze the related studies. The analysis shows how AR has been used to facilitate various manufacturing operations with intelligence. Important findings are derived from a viewpoint different from that of the previous reviews on this subject. The perspective here is on how AR can work as a collaboration interface between human and CI. The outcome of this work is expected to provide guidelines for implementing AR assisted functions with practical applications in smart manufacturing in the near future.
Affine transformation of virtual 3D object using 2D localization of fingertips
2020, Virtual Reality and Intelligent Hardware
Interactions with virtual 3D objects in the virtual reality (VR) environment using the gesture of fingers captured in a wearable 2D camera have emerging applications in real-life.
This paper presents an approach of a two-stage convolutional neural network, one for the detection of hand and another for the fingertips. One purpose of VR environments is to transform a virtual 3D object with affine parameters by using the gesture of thumb and index fingers.
To evaluate the performance of the proposed system, one existing, and another developed egocentric fingertip databases are employed so that learning involves large variations that are common in real-life. Experimental results show that the proposed fingertip detection system outperforms the existing systems in terms of the precision of detection.
The interaction performance of the proposed system in the VR environment is higher than that of the existing systems in terms of estimation error and correlation between the ground truth and estimated affine parameters.

View all citing articles on Scopus

View full text

Augmented assembly technologies based on 3D bare-hand interaction

Abstract

Introduction

Section snippets

3D bare-hand interaction method

Assembly data management

Augmented assembly process

Assembly sequence evaluation and feedback

Implementation and case study

Conclusion and future work

Annals of the CIRP

Annals of the CIRP

Annals of the CIRP

Annals of the CIRP

Robotics and Computer-Integrated Manufacturing

Integration of Design and Assembly using Augmented Reality

Micro-Assembly Technologies and Applications