Skip to main content
SpringerLink
Log in

Touch-less interactive augmented reality game on vision-based wearable device

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

There is an increasing interest in creating pervasive games based on emerging interaction technologies. In order to develop touch-less, interactive and augmented reality games on vision-based wearable device, a touch-less motion interaction technology is designed and evaluated in this work. Users interact with the augmented reality games with dynamic hands/feet gestures in front of the camera, which triggers the interaction event to interact with the virtual object in the scene. Three primitive augmented reality games with eleven dynamic gestures are developed based on the proposed touch-less interaction technology as proof. At last, a comparing evaluation is proposed to demonstrate the social acceptability and usability of the touch-less approach, running on a hybrid wearable framework or with Google Glass, as well as workload assessment, user’s emotions and satisfaction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Alaa H, Haibo L (2013) Fingerink: turn your glass into a digital board. In: Australian computer-human interaction conference, OzCHI’13, 25–29 November 2013, Adelaide, Australia

  2. Ballagas RA, Kratz SG, Borchers J, Yu E, Walz SP, Fuhr CO, Hovestadt L, Tann M (2007) Rexplorer: a mobile, pervasive spell-casting game for tourists. In: CHI’07 extended abstracts on human factors in computing systems. ACM, pp 1929–1934

  3. Bell M, Chalmers M, Barkhuus L, Hall M, Sherwood S, Tennent P, Brown B, Rowland D, Benford S, Capra M, et al (2006) Interweaving mobile games with everyday life. In: Proceedings of the SIGCHI conference on Human Factors in computing systems. ACM, pp 417–426

  4. Benford S, Magerkurth C, Ljungstrand P (2005) Bridging the physical and digital in pervasive gaming. Commun ACM 48(3):54–57

    Article  Google Scholar 

  5. Birk M, Mandryk RL (2013) Control your game-self: effects of controller type on enjoyment, motivation, and personality in game. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ’13, 2013. ACM, New York, pp 685–694

  6. Broll W, Ohlenburg J, Lindt I, Herbst I, Braun A-K (2006) Meeting technology challenges of pervasive augmented reality games. In: Proceedings of 5th ACM SIGCOMM workshop on network and system support for games. ACM, p 28

  7. Capra M, Radenkovic M, Benford S, Oppermann L, Drozd A, Flintham M (2005) The multimedia challenges raised by pervasive games. In: Proceedings of the 13th annual ACM international conference on multimedia. ACM, pp 89–95

  8. Chan K (2011) Playing in traffic: pervasive gaming for commuters. In: Proceedings of the 7th Australasian conference on interactive entertainment. Interactive Entertainment, p 4

  9. Cheok AD, Yang X, Ying ZZ, Billinghurst M, Kato H (2002) Touch-space: mixed reality game space based on ubiquitous, tangible, and social computing. Pers Ubiquitous Comput 6(5–6):430–442

    Article  Google Scholar 

  10. Coenen T, Mostmans L, Naessens K (2013) Museus: case study of a pervasive cultural heritage serious game. J Comput Cultural Herit (JOCCH) 6(2):8

    Google Scholar 

  11. Crossan A, Brewster S, Ng A (2010) Foot tapping for mobile interaction. In: Proceedings of the 24th BCS interaction specialist group conference, BCS ’10. British Computer Society, Swinton, pp 418–422

  12. de Freitas AA, Dey AK (2015) The group context framework: an extensible toolkit for opportunistic grouping and collaboration. In: Proceedings of the 18th ACM conference on computer supported cooperative work & #38; Social Computing, CSCW ’15, 2015. ACM, New York, pp 1602–1611

  13. Duffner S, Garcia C (December 2013) Pixeltrack: a fast adaptive algorithm for tracking non-rigid objects. In: 2013 IEEE international conference on computer vision (ICCV), pp 2480–2487

  14. Ermi L, Mäyrä F (2015) Challenges for pervasive mobile game design: examining players’ emotional responses. In: Proceedings of the 2005 ACM SIGCHI international conference on advances in computer entertainment technology. ACM, pp 371–372

  15. Erol A, Bebis G, Nicolescu M (2007) Vision-based hand pose estimation: a review. Special issue on vision for human-computer interaction 108(12):52–73 Special Issue on Vision for Human-Computer Interaction

    Article  Google Scholar 

  16. Faulkner X (2002) Usability engineering. Palgrave Macmillan, New York

    Google Scholar 

  17. Felzenszwalb P, Zabih R (2011) Dynamic programming and graph algorithms in computer vision. IEEE Trans Pattern Anal Mach Intell 33(4):721–740

    Article  Google Scholar 

  18. Felzenszwalb PF, Zabih R (2011) Dynamic programming and graph algorithms in computer vision. PAMI 33(4):721–740

    Article  Google Scholar 

  19. Fitz-Walter Z, Tjondronegoro D, Koh D, Zrobok M (2012) Mystery at the library: encouraging library exploration using a pervasive mobile game. In: Proceedings of the 24th Australian computer-human interaction conference. ACM, pp 142–145

  20. Gentes A, Guyot-Mbodji A, Demeure I (2010) Gaming on the move: urban experience as a new paradigm for mobile pervasive game design. Multimed Syst 16(1):43–55

    Article  Google Scholar 

  21. Godec M, Roth P, Bischof H (2013) Hough-based tracking of non-rigid objects. Comput Vis Image Underst 117(10):1245–1256

    Article  Google Scholar 

  22. Guo B, Yu Z, Zhang D, He H, Tian J, Zhou X (2014) Toward a group-aware smartphone sensing system. Pervasive Comput IEEE 13(4):80–88

    Article  Google Scholar 

  23. Herbst I, Braun A-K, McCall R, Broll W (2008) Timewarp: interactive time travel with a mobile mixed reality game. In: Proceedings of the 10th international conference on human computer interaction with mobile devices and services. ACM, pp 235–244

  24. Jiang D, Xu Z, Li W, Chen Z (2015) Network coding-based energy-efficient multicast routing algorithm for multi-hop wireless networks. J Syst Softw 104:152–165

    Article  Google Scholar 

  25. Jiang D, Xu Z, Zhang P Zhu T (2014) A transform domain-based anomaly detection approach to network-wide traffic. J Netw Comput Appl

  26. Jonsson S, Waern A (2008) The art of game-mastering pervasive games. In: Proceedings of the 2008 international conference on advances in computer entertainment technology. ACM, pp 224–231

  27. Kalal Z, Mikolajczyk K, Matas J (2012) Tracking-learning-detection. IEEE Trans Pattern Anal Mach Intell 34(7):1409–1422

    Article  Google Scholar 

  28. Kasapakis V, Gavalas D (2014) Blending history and fiction in a pervasive game prototype. In: Proceedings of the 13th international conference on mobile and ubiquitous multimedia, MUM ’14. ACM, New York, pp 116–122

  29. Kasapakis V, Gavalas D, Bubaris N (2013) Pervasive games research: a design aspects-based state of the art report. In: Proceedings of the 17th panhellenic conference on informatics. ACM, pp 152–157

  30. Kasapakis V, Gavalas D, Chatzidimitris T (2015) Evaluation of pervasive games: recruitment of qualified participants through preparatory game phases. arXiv preprint. arXiv:1501.02661

  31. Lehner U, Baldauf M, Eranti V, Reitberger W, Fröhlich P (2014) Civic engagement meets pervasive gaming: towards long-term mobile participation. In: CHI’14 extended abstracts on human factors in computing systems. ACM, pp 1483–1488

  32. Li X, Lv Z, Hu J, Zhang B, Yin L, Zhong C, Wang W, Feng S (2015) Traffic management and forecasting system based on 3d gis. In: 15th IEEE/ACM international conference on cluster, cloud and grid computing (CCGrid), 2015. IEEE

  33. Li X, Lv Z, Zhang B, Wang W, Feng S, Hu J (2015) Webvrgis based city bigdata 3d visualization and analysis. In: Visualization symposium (PacificVis), 2015 IEEE Pacific. IEEE

  34. Lindt I, Ohlenburg J, Pankoke-Babatz U, Ghellal S (2007) A report on the crossmedia game epidemic menace. Comput Entertain (CIE) 5(1):8

    Article  Google Scholar 

  35. Lv Z (2013) Wearable smartphone: wearable hybrid framework for hand and foot gesture interaction on smartphone. In: proceedings of the 2013 IEEE international conference on computer vision workshops, ICCVW ’13. IEEE, Sydney

  36. Lv Z, Chen G, Zhong C, Han Y, Qi YY (2012) A framework for multi-dimensional webgis based interactive online virtual community. Adv Sci Lett 7(1):215–219

    Article  Google Scholar 

  37. Lv Z, Esteve C, Chirivella J, Gagliardo P (2015) A game based assistive tool for rehabilitation of dysphonic patients. In: 3rd workshop on virtual and augmented assistive technology (VAAT), 2015. IEEE

  38. Lv Z, Esteve C, Chirivella J, Gagliardo P (2015) Serious game based dysphonic rehabilitation tool. In 2015 international conference on virtual rehabilitation (ICVR). IEEE

  39. Lv Z et al (2013) Finger in air: touch-less interaction on smartphone. In: Proceedings of the 12th ACM international conference on mobile and ubiquitous multimedia, MUM ’13, 2013. ACM, Luleå

  40. Lv Z, Feng L, Feng S, Li H (2015) Extending touch-less interaction on vision based wearable device. In: Virtual reality (VR), 2015 IEEE. IEEE

  41. Lv Z, Feng L, Li H, Feng S (2014) Hand-free motion interaction on Google glass. In: SIGGRAPH Asia 2014 mobile graphics and interactive applications. ACM

  42. Lv Z, Halawani A, Feng S, Li H, Ur Réhman S (2014) Multimodal hand and foot gesture interaction for handheld devices. ACM Trans Multimed Comput Commun Appl (TOMM) 11(1s):1–19

  43. Lv Z, Khan MSL, Réhman S (2013) Hand and foot gesture interaction for handheld devices. In: Proceedings of the 21st ACM international conference on multimedia, MM ’13, 2013. ACM, Barcelona

  44. Lv Z, Li H (2015) Imagining in-air interaction for hemiplegia sufferer. In: 2015 international conference on virtual rehabilitation (ICVR). IEEE

  45. Lv Z, Li X, Zhang B, Wang W, Feng S, Hu J (2015) Big city 3d visual analysis. In: Eurographics 2015

  46. Lv Z, Su T (2014) 3d seabed modeling and visualization on ubiquitous context. In: SIGGRAPH Asia 2014 posters. ACM, p 33

  47. Lv Z, Tek A, Da Silva F, Empereur-mot C, Chavent M, Baaden M (2013) Game on, science-how video game technology may help biologists tackle visualization challenges. PloS one 8(3):e57990

    Article  Google Scholar 

  48. Lv Z, Yin T, Han Y, Chen Y, Chen G (2011) Webvr-web virtual reality engine based on p2p network. J Netw 6(7):990–998

    Google Scholar 

  49. Magnusson C, Waern A, Gröhn KR, Bjernryd Å, Bernhardsson H, Jakobsson A, Salo J, Wallon M, Hedvall P-O (2011) Navigating the world and learning to like it: mobility training through a pervasive game. In: Proceedings of the 13th international conference on human computer interaction with mobile devices and services . ACM, pp 285–294

  50. Mayo MJ (2007) Games for science and engineering education. Commun ACM 50(7):30–35

    Article  Google Scholar 

  51. Mistry P, Maes P, Chang L (2009) Wuw-wear ur world: a wearable gestural interface. In: CHI ’09 extended abstracts on human factors in computing systems, CHI EA ’09. ACM, New York, pp 4111–4116

  52. Mittal A, Zisserman A, Torr PHS (2011) Hand detection using multiple proposals. In: British machine vision conference

  53. Montola M, Stenros J, Waern A (2009) Pervasive games: theory and design. Morgan Kaufmann Publishers Inc., San Francisco

    Google Scholar 

  54. Nohre R (1996) Deformed template matching by the viterbi algorithm. Pattern Recogn Lett 17(14):1423–1428

    Article  Google Scholar 

  55. Oliveira T, Carvalho L, Ferreira E (2013) Ectodiegesis as immersive effect in pervasive games. In: Proceedings of international conference on making sense of converging media. ACM, p 281

  56. Paelke V, Reimann C, Stichling D (2004) Foot-based mobile interaction with games. In: Proceedings of the 2004 ACM SIGCHI international conference on advances in computer entertainment technology, ACE ’04. ACM, New York, pp 321–324

  57. Petrakis EGM, Diplaros A, Milios E (2002) Matching and retrieval of distorted and occluded shapes using dynamic programming. IEEE Trans Pattern Anal Mach Intell 24(11):1501–1516

    Article  Google Scholar 

  58. Qian C, Sun X, Wei Y, Tang X, Sun J (2014) Realtime and robust hand tracking from depth. In: IEEE conference on computer vision and pattern recognition (CVPR)

  59. Réhman S, Khan A, Li H (2012) Interactive feet for mobile immersive interaction. In: ACM international, workshop MobiVis Workshop at MobileHCI

  60. Rico J, Brewster S (2010) Usable gestures for mobile interfaces: evaluating social acceptability. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ’10. ACM, New York, pp 887–896

  61. Ross J (2011) Pervasive negabehavior games for environmental sustainability. In: CHI’11 extended abstracts on human factors in computing systems. ACM, pp 1085–1088

  62. Scherer K (January 2005) What are emotions? And how can they be measured? Social science information

  63. Simon J, Jahn M, Al-Akkad A (2012) Saving energy at work: the design of a pervasive game for office spaces. In: Proceedings of the 11th international conference on mobile and ubiquitous multimedia. ACM, p 9

  64. Su T, Lv Z, Gao S, Li X, Lv H (2014) 3d seabed: 3d modeling and visualization platform for the seabed. In: 2014 IEEE international conference on multimedia and expo workshops (ICMEW). IEEE, pp 1–6

  65. Tan J, Fan X, Ren Y (2014) Methodology for geographical data evolution: three-dimensional particle-based real-time snow simulation with remote-sensing data. J Appl Remote Sens 8(1):084598–084598

    Article  Google Scholar 

  66. Tek A, Laurent B, Piuzzi M, Lu Z, Chavent M, Baaden M, Delalande O, Martin C, Piccinali L, Katz B, et al (2012) Advances in human-protein interaction-interactive and immersive molecular simulations. In: Biochemistry, genetics and molecular biology ‘protein–protein interactions-computational and experimental tools’, pp 27–65

  67. Tutzschke J-P, Zukunft O (2009) Frap: a framework for pervasive games. In: Proceedings of the 1st ACM SIGCHI symposium on engineering interactive computing systems. ACM, pp 133–142

  68. Wachs JP, Kölsch M, Stern H, Edan Y (2011) Vision-based hand-gesture applications. Commun ACM 54(2):60–71

    Article  Google Scholar 

  69. Waern A, Ahmet Z, Sundström D (2009) An in-game reporting tool for pervasive games. In: Proceedings of the international conference on advances in computer entertainment technology. ACM, pp 240–248

  70. Waern A, Montola M, Stenros J (2009) The three-sixty illusion: designing for immersion in pervasive games. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 1549–1558

  71. Walther BK (2005) Atomic actions-molecular experience: theory of pervasive gaming. Comput Entertain (CIE) 3(3):4–4

    Article  Google Scholar 

  72. Wang Y, Nandi A, Agrawal G (2014) Saga: array storage as a db with support for structural aggregations. In: SSDBM ’14. ACM, New York

  73. Wigdor D, Forlines C, Baudisch P, Barnwell J, Shen C (2007) Lucid touch: a see-through mobile device. In: Proceedings of the 20th annual ACM symposium on user interface software and technology, UIST ’07. ACM, New York, pp 269–278

  74. Wilcoxon F (1945) Individual comparisons by ranking methods. Biom Bull 1(6):80–83

    Article  Google Scholar 

  75. Xu C, Cheng L (2013) Efficient hand pose estimation from a single depth image. In: International conference on computer vision (ICCV)

  76. Yang J, Ding Z, Guo F, Wang H (2014) Multiview image rectification algorithm for parallel camera arrays. J Electron Imaging 23(3):033001–033001

    Article  Google Scholar 

  77. Yang J, Liu Y, Meng Q, Chu R (2015) Objective evaluation criteria for stereo camera shooting quality under different shooting parameters and shooting distances. IEEE Sens J (99):1. doi:10.1109/JSEN.2015.2421518

  78. Yousefi S, Abedan Kondori F, Li H (2013) Experiencing real 3d gestural interaction with mobile devices. Pattern Recogn Lett 34(8):912–921

    Article  Google Scholar 

  79. Zhang M, Lv Z, Zhang X, Chen G, Zhang K (2009) Research and application of the 3d virtual community based on WEBVR and RIA. Comput Inform Sci 2(1):P84

    Google Scholar 

  80. Zhang S, Jing H (2014) Fast log-gabor-based nonlocal means image denoising methods. In: 2014 IEEE international conference on image processing (ICIP). IEEE, pp 2724–2728

  81. Zou S, Xiao H, Wan H, Zhou X (2009) Vision-based hand interaction and its application in pervasive games. In: Proceedings of the 8th international conference on virtual reality continuum and its applications in industry. ACM, pp 157–162

Download references

Acknowledgments

The authors are thankful to Muhammad Sikandar Lal Khan for the preliminary hardware device support, to Liangbing Feng for kind help at SIAT and to our friends for their fruitful discussions and code sharing.

Author information

Authors and Affiliations

  1. SIAT, Chinese Academy of Science, Shenzhen, China

    Zhihan Lv & Shengzhong Feng

  2. Department of Applied Physics & Electronics, Umeå University, Umeå, Sweden

    Alaa Halawani & Shafiq ur Réhman

  3. Royal Institute of Technology (KTH), Stockholm, Sweden

    Haibo Li

  4. Computer Engineering and Science Department, Palestine Polytechnic University, Hebron, Palestine

    Alaa Halawani

Authors
  1. Zhihan Lv
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alaa Halawani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shengzhong Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shafiq ur Réhman
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Haibo Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Zhihan Lv, Alaa Halawani, Shengzhong Feng, Shafiq ur Réhman or Haibo Li.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lv, Z., Halawani, A., Feng, S. et al. Touch-less interactive augmented reality game on vision-based wearable device. Pers Ubiquit Comput 19, 551–567 (2015). https://doi.org/10.1007/s00779-015-0844-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-015-0844-1

Keywords

Search

Navigation