Skip to main content
SpringerLink
Log in

Non-visual identification, localization, and selection of entities of interest in a 3D environment

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

This paper addresses the use of audio and haptics as a mean to reduce the load of the visual channel in interaction tasks within virtual environments. An examination is made regarding the exploitation of audio and/or haptic interactions for the acquisition of a target of interest in an environment containing multiple and obscured distractors. A first study compares means for identifying and locating a specified target among others employing either audio, haptic, or both sensori-motor channels activated simultaneously. Following an analysis of the results and subject comments, an improved multimodal approach is proposed and evaluated in a second study, combining advantages offered by each sensory channel. Results confirm the efficiency and effectiveness of the proposed multimodal approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. http://www.freesound.org.

References

  1. Afonso A, Blum A, Katz B, Tarroux P, Borst G, Denis M (2010) Structural properties of spatial representations in blind people: scanning images constructed from haptic exploration or from locomotion in a 3-D audio virtual environment. Memory Cognit 38:591–604. doi:10.3758/MC.38.5.591

    Article  Google Scholar 

  2. Ammi M, Katz B (2011) Design of haptic stimuli for audio-haptic concurrent coupling. In: IEEE international symposium on haptic audio-visual environments and games. Hebei, China, pp 74–80

  3. Ammi M, Katz B (2012) Audio-haptic intermodal coupling for comparative search tasks. In: IEEE haptics symposium (HAPTICS 2012). Vancouver, Canada, pp 307–313

  4. Aussal M, Alouges F, Katz BF (2012) ITD interpolation and personalization for binaural synthesis using spherical harmonics. In: Audio engineering society UK conference. New York, UK, pp 04

  5. de Berg M, van Kreveld M, Overmars M, Schwarzkopf O (2000) Computational geometry. In: Algorithm. Chapter 7: Voronoi diagrams, 2nd revised edn. Springer

  6. Bowman D, Kruijff E, LaViola J, Poupyrev I (2004) 3D User interfaces: theory and practice. Addison-Wesley

  7. Brown L, Brewster S, Purchase H (2005) A first investigation into the effectiveness of tactons. In: Proceedings first joint Eurohaptics conference and symposium on haptic interfaces for virtual environment and teleoperator systems world haptics 2005, pp 167–176. doi:10.1109/WHC.2005.6

  8. Carpendale MST, Cowperthwaite DJ, Fracchia FD (1997) Extending distortion viewing from 2d to 3d. IEEE Compu Graph Appl 17(4):42–51

    Article  Google Scholar 

  9. Cockburn A, Brewster S (2005) Multimodal feedback for the acquisition of small targets. Ergonomics 48(9):1129–1150

    Article  Google Scholar 

  10. Dennerlein J, Yang M (1999) Perceived musculoskeletal loading during use of a force-feedback computer mouse. In: Human factors and ergonomics sociecty annual conference, Houston

  11. Dennerlein J, Yang MC (2001) Haptic force feedback devices for the office computer: performance and musculoskeletal loading issues. Hum Factors 43(2):278–286

    Article  Google Scholar 

  12. Elmqvist N, Tsigas P (2006) View projection animation for occlusion reduction. In: AVI 06: Proceedings of the working conference on advanced visual interfaces. ACM, NY, pp 471–475. doi:10.1145/1133265.1133361

  13. Enriquez MJ, MacLean KE (2003) The hapticon editor: a tool in support of haptic communication research. In: HAPTICS’ 03. IEEE Computer Society, Los Angeles, p 356

  14. Flasar J, Sochor J (2007) Manipulating objects behind obstacles. In: Lecture notes in computer science,vol 4563. pp 32–41

  15. Friedes D (1974) Human information processing and sensory modality: cross-modal functions, information complexity, memory and deficit. Psychol Bull 81:284–310

    Article  Google Scholar 

  16. Grosjean J, Coquillart S (1999) The magic mirror: a metaphor for assisting the exploration of virtual worlds. In: Proceedings spring conference on computer graphics

  17. Hasser C, Goldenberg A (1998) User performance in a gui pointing task with a low-cost force-feedback computer mouse. In: Seventh annual symposium on haptic interfaces. International mechanical engineering congress and exposition, Anaheim, CA

  18. Hawley LM, Litovsky RY, Colburn HS (1999) Speech intelligibility and localization in a multi-source environment. J Acoust Soc Am 105(6):3236–3448

    Google Scholar 

  19. Hermann T, Ritter H (1999) Listen to your data: model-based sonification for data analysis. In: Advances in intelligent computing and multimedia systems, G. E. Lasker, Baden-Baden, pp 189–194

  20. Hwang F, Keates S, Langdon P, Clarkson PJ (2003) Multiple haptic targets for motion-impaired computer users. In: CHI ’03: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, New York, pp 41–48

  21. Katz B, Parseihian G (2012) Perceptually based head-related transfer function database optimization. J Acoust Soc Am 131(2):EL99-EL105

    Google Scholar 

  22. Katz B, Rio E, Picinali L (2010) LIMSI spatialisation engine. In: International deposit digital number IDDN.FR.001.340014. 000.S.P.2010.000.31235.

  23. Katz BF, Kammoun S, Parseihian G, Gutierrez O, Brilhault A, Auvray M, Truillet P, Denis M, Thorpe S, Jouffrais C (2012) NAVIG: augmented reality guidance system for the visually impaired. Virtual Real 16(4):253–269. doi:10.1007/s10055-012-0213-6

    Article  Google Scholar 

  24. Kim SC, Kwon DS (2007) Haptic and sound grid for enhanced positioning in a 3-d virtual environment. In: Proceedings of haptic and audio interaction design. Second international workshop, HAID vol 2007 no. 1. pp 98–109

  25. Kuznetsov S, Dey AK, Hudson SE (2009) The effectiveness of haptic cues as an assistive technology for human memory. In: Proceedings of the 7th international conference on pervasive computing. Pervasive ’09, Springer, Berlin, pp 168–175

  26. Maclean K, Enriquez M (2003) Perceptual design of haptic icons. In: Proceedings of Eurohaptics, pp 351–363

  27. Magnusson C, Danielsson H, Rassmus-Gröhn K (2006) Non visual haptic audio tools for virtual environments. In: Proceedings of the first international conference on haptic and audio interaction design, HAID’06, pp 111–120

  28. Menelas B, Ammi M, Bourdot P (2008) A flexible method for haptic rendering of isosurface from volumetric data. In: Proceedings of the 6th international conference on Haptics: perception. Devices and scenarios, EuroHaptics ’08, Springer, Berlin, pp 687–693

  29. Menelas B, Ammi M, Pastur L, Bourdot P (2009) Haptical exploration of an unsteady flow. In: Proceedings and symposium on haptic interfaces for virtual environment and teleoperator systems EuroHaptics conference world haptics 2009. Third Joint, pp 232–237

  30. Menelas B, Picinali L, Katz B, Bourdot P (2010) Audio haptic feedbacks in a task of targets acquisition. In: IEEE symposium on 3D user interfaces (3DUI 2010). Waltham, pp 51–54. doi:10.1109/3DUI.2010.5444722

  31. Menelas B, Picinali L, Katz B, Bourdot P, Ammi M (2009) Haptic audio guidance for target selection in a virtual environment. In: 4th Intl. haptic and auditory interaction design workshop (HAID’09), Dresden, pp 12–13

  32. Menelas BAJ (2013) Innovative approaches of data visualization and visual analytics, chap. In: Virtual reality technologies (visual, haptics and audio) in large datasets analysis. IGI Global, pp 225–250

  33. Menelas BAJ (2013) Interactive analysis of cavity-flows in a virtual environment. In: Proceedings of the 28th spring conference on computer graphics. SCCG ’12, ACM, NY, pp 31–37, doi:10.1145/2448531.2448535

  34. Menelas, BAJ, Otis MJD (2012) Design of a serious game for learning vibrotactile messages. In: Haptic audio visual environments and games. IEEE, pp 124–129

  35. Moller H, Sorensen M, Hammershoi D, Jensen C (1995) Head-related transfer functions of human subjects. J Audio Eng Soc 45(5):300–321

    Google Scholar 

  36. Oakley I, Brewster S, Gray P (2001) Solving multi-target haptic problems in menu interaction. In: CHI ’01: CHI ’01 extended abstracts on human factors in computing systems, pp 357–358

  37. Oakley I, McGee MR, Brewster S, Gray P (2000) Putting the feel in ‘look and feel‘. In: CHI ’00: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, NY, pp 415–422

  38. Picinali L, Menelas B, Katz B, Bourdot P (2010) Evaluation of a haptic / audio system for 3-D targeting tasks. In: 128th Conv. audio engineering society. London, pp 1–11

  39. Picon F, Ammi M, Bourdot P (2008) Force model for CAD selection. In: Feiner S, Thalmann D, Guitton P, Frhlich B, Kruijff E, Hachet M (eds) VRST, ACM, pp 283–284. doi:10.1145/1450579.1450654

  40. Simpson BD, Brungart DS, Nandini I (2009) Sound localization in multisource environments. In: Tech. rep., Pentagon report, human effectiveness directorate Wright-Patterson AFB OH 711 human performance wing

  41. Swapp D, Pawar V, Loscos C (2006) Interaction with co-located haptic feedback in virtual reality. Virtual Real 10(1):24–30

    Article  Google Scholar 

  42. Vanacken L, Grossman T, Coninx K (2009) Multimodal selection techniques for dense and occluded 3d virtual environments. Int J Hum Comput Stud 67(3):237–255

    Article  Google Scholar 

  43. Vanacken L, Raymaekers C, Coninx K (2006) Evaluating the influence of multimodal feedback on egocentric selection metaphors in virtual environments. In: Proceedings of the first international conference on haptic and audio interaction design. HAID’06, Springer, Berlin, pp 12–23

  44. Wall SA, Paynter K, Shillito AM, Wright M, Scali S (2002) The effect of haptic feedback and stereo graphics in a 3d target acquisition. In: Proceedings of eurohaptics, Edinburgh

  45. Yamada T, Tsubouchi D, Ogi T, Hirose M (2002) Desk-sized immersive workplace using force feedback grid interface. In: Proceedings IEEE. Virtual Reality, pp 135–142

Download references

Author information

Authors and Affiliations

  1. Mathematics and Computer Science Department, University of Quebec at Chicoutimi, 555, boulevard de l’Université, Chicoutimi, Canada

    Bob-Antoine J. Menelas

  2. Faculty of Technology, De Montfort University, The Gateway, Leicester, LE1 9BH, UK

    Lorenzo Picinali

  3. LIMSI-CNRS, BP 133, Orsay, France

    Patrick Bourdot & Brian F. G. Katz

Authors
  1. Bob-Antoine J. Menelas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lorenzo Picinali
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Patrick Bourdot
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Brian F. G. Katz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bob-Antoine J. Menelas.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Menelas, BA.J., Picinali, L., Bourdot, P. et al. Non-visual identification, localization, and selection of entities of interest in a 3D environment. J Multimodal User Interfaces 8, 243–256 (2014). https://doi.org/10.1007/s12193-014-0148-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-014-0148-1

Keywords

Search

Navigation