Skip to main content
SpringerLink
Log in

Gesture-based interactive augmented reality content authoring system using HMD

  • Original Article
  • Published:
Virtual Reality Aims and scope Submit manuscript

Abstract

This paper proposes an augmented reality content authoring system that enables ordinary users who do not have programming capabilities to easily apply interactive features to virtual objects on a marker via gestures. The purpose of this system is to simplify augmented reality (AR) technology usage for ordinary users, especially parents and preschool children who are unfamiliar with AR technology. The system provides an immersive AR environment with a head-mounted display and recognizes users’ gestures via an RGB-D camera. Users can freely create the AR content that they will be using without any special programming ability simply by connecting virtual objects stored in a database to the system. Following recognition of the marker via the system’s RGB-D camera worn by the user, he/she can apply various interactive features to the marker-based AR content using simple gestures. Interactive features applied to AR content can enlarge, shrink, rotate, and transfer virtual objects with hand gestures. In addition to this gesture-interactive feature, the proposed system also allows for tangible interaction using markers. The AR content that the user edits is stored in a database, and is retrieved whenever the markers are recognized. The results of comparative experiments conducted indicate that the proposed system is easier to use and has a higher interaction satisfaction level than AR environments such as fixed-monitor and touch-based interaction on mobile screens.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  • AndAR (2014) AndAR-Android augmented reality. http://code.google.com/p/andar/. Accessed 30 May 2014

  • ARToolKit (2014) ARToolKit. http://www.hitl.washington.edu/artoolkit/. Accessed 30 May 2014

  • Azuma RT (1997) A survey of augmented reality. Presence 6(4):355–385

    Article  Google Scholar 

  • Azuma R, Bishop G (1994) Improving static and dynamic registration in an optical see-through HMD. In: Proceedings of the 21st ACM annual conference on computer graphics and interactive techniques, pp 197–204

  • Azuma R, Baillot Y, Behringer R, Feiner S, Julier S, MacIntyre B (2001) Recent advances in augmented reality. IEEE Comput Graph Appl 21(6):34–47

    Article  Google Scholar 

  • Bajura M, Neumann U (1995) Dynamic registration correction in video-based augmented reality systems. IEEE Comput Graph Appl 15(5):52–60

    Article  Google Scholar 

  • Billinghurst M, Kato H, Poupyrev I (2001) The magicbook-moving seamlessly between reality and virtuality. IEEE Comput Graph Appl 21(3):6–8

    Google Scholar 

  • Billinghurst M, Hakkarainen M, Woodward C (2008) Augmented assembly using a mobile phone. In: Proceedings of the 7th ACM international conference on mobile and ubiquitous multimedia, pp 84–87

  • Buchmann V, Violich S, Billinghurst M, Cockburn A (2004) FingARtips: Gesture based direct manipulation in augmented reality. In: Proceedings of the 2nd ACM international conference on computer graphics and interactive techniques in Australasia and South East Asia, pp 212–221

  • Bunnun P, Subramanian S, Mayol-Cuevas WW (2013) In-situ interactive image-based model building for augmented reality from a handheld device. Virtual Real 17(2):137–146

    Article  Google Scholar 

  • Coquillart S, Göbel M (2004) Authoring of mixed reality applications including multi-marker calibration for mobile devices. In: Eurographics symposium on virtual environments, pp 1–9

  • Doliotis P, Athitsos V, Kosmopoulos D, Perantonis S (2012) Hand shape and 3D pose estimation using depth data from a single cluttered frame. In: Bebis G, Boyle R, Parvin B, Koracin D, Fowlkes C (eds) Advances in visual computing. Springer, Berlin, Heidelberg, pp 148–158

    Chapter  Google Scholar 

  • Dorfmuller-Ulhaas K, Schmalstieg D (2001) Finger tracking for interaction in augmented environments. In: Proceedings IEEE and ACM international symposium on augmented reality, pp 55–64

  • Grimm P, Haller M, Paelke V, Reinhold S, Reimann C, Zauner R (2002) AMIRE-authoring mixed reality. In: The first IEEE international workshop augmented reality toolkit, pp 2–pp

  • Hackenberg G, McCall R, Broll W (2011) Lightweight palm and finger tracking for real-time 3d gesture control. In: 2011 IEEE virtual reality conference (VR), pp 19–26

  • Harviainen T, Korkalo O, Woodward C (2009) Camera-based interactions for augmented reality. In: Proceedings of the ACM international conference on advances in computer entertainment technology, pp 307–310

  • Henrysson A, Billinghurst M (2007) Using a mobile phone for 6 DOF mesh editing. In: Proceedings of the 8th ACM SIGCHI New Zealand chapter’s international conference on computer-human interaction: design centered HCI, pp 9–16

  • Hoff WA, Nguyen K, Lyon T (1996) Computer-vision-based registration techniques for augmented reality. In: Photonics East’96, international society for optics and photonics, pp 538–548

  • Kato H, Billinghurst M (1999) Marker tracking and HMD calibration for a video-based augmented reality conferencing system. In: Proceedings IWAR’99, pp 85–94

  • Kato H, Billinghurst M, Poupyrev I, Imamoto K, Tachibana K (2000) Virtual object manipulation on a table-top AR environment. Proc ISAR 2000:111–119

    Google Scholar 

  • Kaufmann H, Schmalstieg D (2003) Mathematics and geometry education with collaborative augmented reality. Comput Graph 27(3):339–345

    Article  Google Scholar 

  • Langlotz T, Mooslechner S, Zollmann S, Degendorfer C, Reitmayr G, Schmalstieg D (2012) Sketching up the world: in situ authoring for mobile augmented reality. Pers Ubiquit Comput 16(6):623–630

    Article  Google Scholar 

  • Ledermann F, Schmalstieg D (2005) April: a high-level framework for creating augmented reality presentations. In: Proceedings. VR 2005, pp 187–194

  • Lee T, Hollerer T (2007) Handy AR: Markerless inspection of augmented reality objects using fingertip tracking. In: 11th IEEE international symposium on wearable computers, pp 83–90

  • Lee U, Tanaka J (2012) Hand controller: image manipulation interface using fingertips and palm tracking with Kinect depth data. In: Proceedings of Asia Pacific conference on computing human interact, pp 705–706

  • Lee GA, Nelles C, Billinghurst M, Kim GJ (2004) Immersive authoring of tangible augmented reality applications. In: Proceedings of the 3rd IEEE/ACM international symposium on mixed and augmented reality, pp 172–181

  • Li D, Weng D, Li Y, Xie J (2013) Touchablear: a new experience of augmented reality. Proc ICVRV 2013:37–42

    Google Scholar 

  • Liang H, Yuan J, Thalmann D (2012) 3D fingertip and palm tracking in depth image sequences. In: Proceedings of the 20th ACM international conference on Multimedia, pp 785–788

  • Lund A (2001) Measuring usability with the use questionnaire. Usability Interface 8(2):3–6

    MathSciNet  Google Scholar 

  • MacIntyre B, Gandy M, Dow S, Bolter JD (2004) DART: A toolkit for rapid design exploration of augmented reality experiences. In: Proceedings of the 17th annual ACM symposium on user interface software and technology, pp 197–206

  • Maier P, Tonnis M, Klinker G (2010) Designing and comparing two-handed gestures to confirm links between user controlled objects. Proc ISMAR 2010:251–252

    Google Scholar 

  • Ng L, Oon S, Ong S, Nee A (2011) GARDE: a gesture-based augmented reality design evaluation system. Int J Interact Design Manuf (IJIDeM) 5(2):85–94

    Article  Google Scholar 

  • NyARToolKit (2014) NyARToolKit. http://arforglass.org/?project=nyartoolkit/. Accessed 30 May 2014

  • Oculus VR (2014) Oculus VR developer center. http://developer.oculusvr.com/. Accessed 30 May 2014

  • Oikonomidis I, Kyriazis N, Argyros AA (2011) Efficient model-based 3D tracking of hand articulations using Kinect. Proc BMVC 2011:1–11

    Google Scholar 

  • Oikonomidis I, Kyriazis N, Argyros AA (2012) Tracking the articulated motion of two strongly interacting hands. Proc CVPR 2012:1862–1869

    Google Scholar 

  • Okuma T, Kiyokawa K, Takemura H, Yokoya N (1998) An augmented reality system using a real-time vision based registration. In: Proceedings fourteenth international conference on pattern recognition, vol 2, pp 1226–1229

  • Ong S, Wang Z (2011) Augmented assembly technologies based on 3D bare-hand interaction. CIRP Ann-Manuf Technol 60(1):1–4

    Article  Google Scholar 

  • OpenNI (2014) OpenNI. https://github.com/OpenNI/OpenNI/. Accessed 30 May 2014

  • Poupyrev I, Tan DS, Billinghurst M, Kato H, Regenbrecht H, Tetsutani N (2002) Developing a generic augmented-reality interface. Computer 35(3):44–50

    Article  Google Scholar 

  • QCAR (2014) QCAR. http://developer.qualcomm.com/dev/augmented-reality/. Accessed 30 May 2014

  • Radu I, MacIntyre B (2009) Augmented-reality scratch: a tangible programming environment for children. In: Proceedings of conference on interaction design for children, Como, Italy

  • Ren Z, Meng J, Yuan J, Zhang Z (2011a) Robust hand gesture recognition with Kinect sensor. In: Proceedings of the 19th ACM international conference on Multimedia, pp 759–760

  • Ren Z, Yuan J, Zhang Z (2011b) Robust hand gesture recognition based on finger-earth mover’s distance with a commodity depth camera. In: Proceedings of the 19th ACM international conference on Multimedia, pp 1093–1096

  • Schmalstieg D, Fuhrmann A, Hesina G, Szalavári Z, Encarnaçao LM, Gervautz M, Purgathofer W (2002) The studierstube augmented reality project. Presence Teleoper Virtual Environ 11(1):33–54

    Article  Google Scholar 

  • Seichter H, Looser J, Billinghurst M (2008) Composar: An intuitive tool for authoring AR applications. In: Proceedings of the 7th IEEE/ACM international symposium on mixed and augmented reality, pp 177–178

  • Shim J, Kong M, Yang Y, Seo J, Han T-D (2014) Interactive features based augmented reality authoring tool. Proc ICCE 2014:47–50

    Google Scholar 

  • SoftKinetic (2014) SoftKinetic: the interface is you. http://www.softkinetic.com/. Accessed 30 May 2014

  • Thomas BH, Piekarski W (2002) Glove based user interaction techniques for augmented reality in an outdoor environment. Virtual Real 6(3):167–180

    Article  Google Scholar 

  • Wang Y, Langlotz T, Billinghurst M, Bell T (2009) An authoring tool for mobile phone AR environments. Proc N Z Comput Sci Res Stud Conf 9:1–4

    Google Scholar 

  • Whitton MC, Pisano ED, Fuchs H (1996) Technologies for augmented reality systems: Realizing ultrasound-guided needle biopsies. In: Proceedings of the 23rd annual conference on computer graphics and interactive techniques, pp 439–446

  • Wilson AD (2010) Using a depth camera as a touch sensor. In: Proceedings ACM international conference on interactive tabletops and surfaces, pp 69–72

  • Wozniewski M, Warne P (2011) Towards in situ authoring of augmented reality content. In: Proceedings ISMAR 2011

  • Yao Y, Fu Y (2012) Real-time hand pose estimation from RGB-D sensor. Proc ICME 2012:705–710

    Google Scholar 

Download references

Acknowledgments

This work was supported by a National Research Foundation of Korea (NRF) Grant (No. NRF-2015R1A2A1A10055673) funded by the Korea government (MEST).

Author information

Authors and Affiliations

  1. Department of Computer Science, Yonsei University, 50, Yonsei-ro, Seodaemun-gu, Seoul, 03722, Korea

    Jinwook Shim, Yoonsik Yang, Nahyung Kang & Tack-Don Han

  2. Software Platform R&D Lab, LG Electronics Advanced Research Institute, 19, Yangjae-daero 11gil, Seocho-gu, Seoul, 06772, Korea

    Jonghoon Seo

Authors
  1. Jinwook Shim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yoonsik Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nahyung Kang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jonghoon Seo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tack-Don Han
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tack-Don Han.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shim, J., Yang, Y., Kang, N. et al. Gesture-based interactive augmented reality content authoring system using HMD. Virtual Reality 20, 57–69 (2016). https://doi.org/10.1007/s10055-016-0282-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10055-016-0282-z

Keywords

Search

Navigation