Skip to main content

Combining BCI with Virtual Reality: Towards New Applications and Improved BCI

  • Chapter
  • First Online:
Towards Practical Brain-Computer Interfaces

Abstract

Brain–Computer Interfaces (BCI) are communication systems which can convey messages through brain activity alone. Recently BCIs were gaining interest among the virtual reality (VR) community since they have appeared as promising interaction devices for virtual environments (VEs). Especially these implicit interaction techniques are of great interest for the VR community, e.g., you are imaging the movement of your hand and the virtual hand is moving, or you can navigate through houses or museums by your thoughts alone or just by looking at some highlighted objects. Furthermore, VE can provide an excellent testing ground for procedures that could be adapted to real world scenarios, especially patients with disabilities can learn to control their movements or perform specific tasks in a VE. Several studies will highlight these interactions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    See, for instance, the work achieved as part of the BrainAble project: http://www.brainable.org/

  2. 2.

    http://openvibe.inria.fr/

  3. 3.

    http://www.ogre3d.org

  4. 4.

    http://qt.nokia.com/

  5. 5.

    http://www.gtec.at

  6. 6.

    http://www.diana.uma.es

  7. 7.

    http://www.inria.fr/en/

  8. 8.

    http://www.sm4all-project.eu

References

  1. Allison, B.Z., McFarland, D.J., Schalk, G., Zheng, S.D., Jackson, M.M., Wolpaw, J.R.: Towards an independent brain–computer interface using steady state visual evoked potentials. Clin. Neurophysiol. 119(2), 399–408 (2008)

    Google Scholar 

  2. Ammon, K., Gandevia, S.C.: Transcranial magnetic stimulation can influence the selection of motor programmes. J. Neurol. Neurosurg. Psychiatry 53(8), 705–707 (1990)

    Google Scholar 

  3. Bayliss, J.D.: Use of the evoked potential P3 component for control in a virtual apartment. IEEE Trans. Neural Syst. Rehabil. Eng. 11(2), 113–116 (2003)

    Google Scholar 

  4. Bayliss, J.D., Ballard, D.H.: A virtual reality testbed for brain–computer interface research. IEEE Trans. Rehabil. Eng. 8(2), 188–90 (2000)

    Google Scholar 

  5. Birbaumer, N., Ghanayim, N., Hinterberger, T., Iversen, I., Kotchoubey, B., Kübler, A., Perelmouter, J., Taub, E., Flor, H.: A spelling device for the paralysed. Nature 398, 297–298 (1999)

    Google Scholar 

  6. Bowman, D., Kruijff, E., Jr, J.L., Poupyrev, I.: 3D User Interfaces: Theory and Practice. Addison-Wesley/Pearson Education, Redwood, USA (2005)

    Google Scholar 

  7. Burdea, G.: Force and touch feedback for virtual reality. Wiley, New York, USA (1996)

    Google Scholar 

  8. Burdea, G., Coiffet, P.: Virtual Reality Technology. Wiley, New York, USA (2003)

    Google Scholar 

  9. Clark, A.: Supersizing the mind: Embodiment, action, and cognitive extension. Oxford University Press, USA (2008)

    Google Scholar 

  10. Congedo, M., Goyat, M., Tarrin, N., Varnet, L., Rivet, B., Ionescu, G., Jrad, N., Phlypo, R., Acquadro, M., Jutten, C.: “Brain Invaders”: a prototype of an open-source P300-based video game working with the OpenViBE platform. In: 5th International BCI Conference (2011)

    Google Scholar 

  11. Cruz-Neira, C., Sandin, D., Defanti, T., Kentyon, R., Hart, J.: The CAVE : audio visual experience automatic virtual environment. Commun. ACM 35(6), 64–72 (1992)

    Google Scholar 

  12. Damen, E.J., Brunia, C.H.: Changes in heart rate and slow brain potentials related to motor preparation and stimulus anticipation in a time estimation task. Psychophysiology 24(6), 700–713 (1987)

    Google Scholar 

  13. Decety, J., Jeannerod, M., Germain, M., Pastene, J.: Vegetative response during imagined movement is proportional to mental effort. Behav. Brain Res. 42, 1–5 (1991)

    Google Scholar 

  14. Donchin, E., Spencer, K.M., Wijesinghe, R.: The mental prosthesis: assessing the speed of a P300-based brain–computer interface. IEEE Trans. Neural Syst. Rehabil. Eng. 8, 174–179 (2000)

    Google Scholar 

  15. Edlinger, G., Holzner, C., Groenegress, C., Guger, C., Slater, M.: Goal-oriented control with brain–computer interface. In: Lecture Notes in Computer Science, Springer, Berlin/Heidelberg, vol. 5638, pp. 732–740 (2009)

    Google Scholar 

  16. Edlinger, G., Holzner, C., Guger, C.: A hybrid brain–computer interface for smart home control. In: Human Computer Interface Conference, Springer, Berlin/Heidelberg, 417–425 (2011)

    Google Scholar 

  17. Elshout, J., Molina, G.G.: Review of brain–computer interfaces based on the P300 evoked potential. Tech. Rep. PR-TN 2009/00066, Koninklijke Philips Electronics (2009)

    Google Scholar 

  18. Faller, J., Allison, B., Brunner, C., Schmalstieg, D., Pfurtscheller, G.: A software SSVEP BCI integrating stimuli within motivating and immersive virtual and augmented reality environments. In: Real Actions in Virtual Environments (RAVE) conference, Barcelona, Spain (2010a)

    Google Scholar 

  19. Faller, J., Leeb, R., Pfurtscheller, G., Scherer, R.: Avatar navigation in virtual and augmented reality environments using an SSVEP BCI. In: International Conference on Applied Bionics and Biomechanics (ICABB) 2010, Venice, Italy (2010b)

    Google Scholar 

  20. Faller, J., Müller-Putz, G.R., Schmalstieg, D., Pfurtscheller, G.: An application framework for controlling an avatar in a desktop based virtual environment via a software SSVEP brain–computer interface. Presence (Camb.) 19(1), 25–34 (2010c)

    Google Scholar 

  21. Fellner, D., Havemann, S., Hopp, A.: Dave - eine neue technologie zur preiswerten und hochqualitativen immersiven 3d-darstellung. In: Möller, R. (ed.) Proc. 8. Workshop: Sichtsysteme - Visualisierung in der Simulationstechnik, pp. 77–83. Shaker Verlag, Bremen (2003)

    Google Scholar 

  22. Friedman, D., Donenfeld, A., Zafran, E.: Neurophysiology-based art in immersive virtual reality. Int. J. Arts Technol. 2(4), 331–347 (2009)

    Google Scholar 

  23. George, L., Lécuyer, A.: An overview of research on passive brain–computer interfaces for implicit human–computer interaction. In: International Conference on Applied Bionics and Biomechanics (2010)

    Google Scholar 

  24. Groenegress, C., Holzner, C., Guger, C., Slater, M.: Effects of P300-based BCI use on reported presence in a virtual environment. Presence (Camb.) 19(1), 1–11 (2010)

    Google Scholar 

  25. Grychtol, B., Lakany, H., Valsan, G., Conway, B.A.: Human behavior integration improves classification rates in real-time BCI. IEEE Trans. Neural Syst. Rehabil. Eng. 18(4), 362–368 (2010)

    Google Scholar 

  26. Guger, C., Schlögl, A., Neuper, C., Walterspacher, D., Strein, T., Pfurtscheller, G.: Rapid prototyping of an EEG-based brain–computer interface (BCI). IEEE Trans. Rehab. Eng. 9(1), 49–58 (2001)

    Google Scholar 

  27. Guger, C., Holzner, C., Groenegress, C., Edlinger, G., Slater, M.: Control of a smart home with a brain–computer interface. In: 4th International Brain–Computer Interface Workshop, pp. 339–342 (2008)

    Google Scholar 

  28. Guger, C., Daban, S., Sellers, E., Holzner, C., Krausz, G., Carabalona, R., Gramatica, F., Edlinger, G.: How many people are able to control a P300-based brain–computer interface (BCI)? Neurosci. Lett. 462(1), 94–98 (2009a)

    Google Scholar 

  29. Guger, C., Holzner, C., Groenegress, C., Edlinger, G., Slater, M.: Brain–computer interface for virtual reality control. In: Proceedings of ESANN 2009, pp. 443–448 (2009b)

    Google Scholar 

  30. Guger, C., Edlinger, G., Krausz, G.: Recent Advances in Brain–Computer Interface Systems, InTech, chap. Hardware/Software Components and Applications of BCIs, Rijeka, Croatia, 1–24 (2011)

    Google Scholar 

  31. Haggard, P.: Conscious intention and motor cognition. Trends Cogn. Sci. 9(6), 290–295 (2005)

    Google Scholar 

  32. Kronegg, J., Chanel, G., Voloshynovskiy, S., Pun, T.: EEG-based synchronized brain–computer interfaces: A model for optimizing the number of mental tasks. IEEE Trans. Neural Syst. Rehabil. Eng. 15(1), 50–58 (2007)

    Google Scholar 

  33. Krusienski, D., Sellers, E., Cabestaing, F., Bayoudh, S., McFarland, D., Vaughan, T., Wolpaw, J.: A comparison of classification techniques for the P300 speller. J. Neural Eng. 3, 299–305 (2006)

    Google Scholar 

  34. Kuebler, A., Nijboer, F., Mellinger, J., Vaughan, T.M., Pawelzik, H., Schalk, G., McFarland, D.J., Birbaumer, N., Wolpaw, J.R.: Patients with ALS can use sensorimotor rhythms to operate a brain–computer interface. Neurology 64(10), 1775–1777 (2005)

    Google Scholar 

  35. Lalor, E., Kelly, S., Finucane, C., Burke, R., Smith, R., Reilly, R.B., McDarby, G.: Steady-state vep-based brain computer interface control in an immersive 3-d gaming environment. EURASIP J. Appl. Signal Process. 19, 3156–3164 (2005)

    Google Scholar 

  36. Lécuyer, A.: Using eyes, hands, and brain for 3D interaction with virtual environments: A perception-based approach. Tech. rep., Habilitation thesis (2010)

    Google Scholar 

  37. Lécuyer, A., Lotte, F., Reilly, R., Leeb, R., Hirose, M., Slater, M.: Brain–computer interfaces, virtual reality and videogames. IEEE Computer 41(10), 66–72 (2008)

    Google Scholar 

  38. Leeb, R.: Brain-computer communication: the motivation, aim, and impact of virtual feedback. PhD thesis, Graz University of Technology (2008)

    Google Scholar 

  39. Leeb, R., Keinrath, C., Friedman, D., Guger, C., Scherer, R., Neuper, C., Garau, M., Antley, A., Steed, A., Slater, M., Pfurtscheller, G.: Walking by thinking: the brainwaves are crucial, not the muscles! Presence (Camb.) 15, 500–514 (2006)

    Google Scholar 

  40. Leeb, R., Friedman, D., Müller-Putz, G.R., Scherer, R., Slater, M., Pfurtscheller, G.: Self-paced (asynchronous) BCI control of a wheelchair in virtual environments: a case study with a tetraplegics. Comput. Intell. Neurosci. 2007, 79,642 (2007a)

    Google Scholar 

  41. Leeb, R., Lee, F., Keinrath, C., Scherer, R., Bischof, H., Pfurtscheller, G.: Brain-computer communication: motivation, aim and impact of exploring a virtual apartment. IEEE Trans. Neural Syst. Rehabil. Eng. 15, 473–482 (2007b)

    Google Scholar 

  42. Leeb, R., Scherer, R., Friedman, D., Lee, F.Y., Keinrath, C., Bischof, H., Slater, M., Pfurtscheller, G.: Combining BCI and virtual reality: scouting virtual worlds. In: Dornhege, G., Millán, J., Hinterberger, T., McFarland, D.J., Müller, K.R. (eds.) Toward brain–computer interfacing, chap 23, pp. 393–408. MIT Press, Cambridge/London (2007c)

    Google Scholar 

  43. Leeb, R., Settgast, V., Fellner, D.W., Pfurtscheller, G.: Self-paced exploring of the Austrian National Library through thoughts. Int. J. Bioelectromagn. 9, 237–244 (2007d)

    Google Scholar 

  44. Legény, J., Viciana-Abad, R., Lécuyer, A.: Navigating in virtual worlds using a self-paced SSVEP-based brain–computer interface with integrated stimulation and real-time feedback. Presence – Teleoperators and Virtual Environments, vol 20(6), 529–544, 2011

    Google Scholar 

  45. Lotte, F.: Brain–computer interfaces for 3D games: Hype or hope? In: Foundations of Digital Games, pp. 325–327 (2011)

    Google Scholar 

  46. Lotte, F., Renard, Y., Lécuyer, A.: Self-paced brain–computer interaction with virtual worlds: a qualitative and quantitative study “out-of-the-lab.” In: 4th International Brain–Computer Interface Workshop and Training Course, pp. 373–378 (2008)

    Google Scholar 

  47. Lotte, F., Langhenhove, A.V., Lamarche, F., Ernest, T., Renard, Y., Arnaldi, B., Lécuyer, A.: Exploring large virtual environments by thoughts using a brain–computer interface based on motor imagery and high-level commands. Presence (Camb.) 19(1), 54–70 (2010)

    Google Scholar 

  48. Mason, S., Kronegg, J., Huggins, J., Fatourechi, M., Schloegl, A.: Evaluating the performance of self-paced BCI technology. Tech. rep., Neil Squire Society (2006)

    Google Scholar 

  49. Millán JdR., Rupp, R., Müller-Putz, G., Murray-Smith, R., Giugliemma, C., Tangermann, M., Kübler, A., Leeb, R., Neuper, C., Müller, K.R., Mattia, D.: Combining brain-computer interfaces and assistive technologies: state-of-the-art and challenges. Front. Neuroprosthetics, 4(161), 1–15 (2010)

    Google Scholar 

  50. Neuper, C., Scherer, R., Wriessnegger, S., Pfurtscheller, G.: Motor imagery and action observation: modulation of sensorimotor brain rhythms during mental control of a brain–computer interface. Clin. Neurophysiol. 120(2), 239–247 (2009)

    Google Scholar 

  51. Nijholt, A., Tan, D., Pfurtscheller, G., Brunner, C., del R Millán, J., Allison, B., Graimann, B., Popescu, F., Blankertz, B., Müller, K.R.: Brain–computer interfacing for intelligent systems. IEEE Intell. Syst. 23, 72–79 (2008)

    Google Scholar 

  52. Nijholt, A., Bos, D.P.O., Reuderink, B.: Turning shortcomings into challenges: Brain–computer interfaces for games. Entertain. Comput. 1(2), 85–94 (2009)

    Google Scholar 

  53. Oishi, K., Kasai, T., Maeshima, T.: Autonomic response specificity during motor imagery. J. Physiol. Anthropol. Appl. Human Sci. 19(6), 255–261 (2000)

    Google Scholar 

  54. Pfurtscheller, G., Lopes da Silva, F.H.: Event-related EEG/MEG synchronization and desynchronization: basic principles. Clin. Neurophysiol. 110, 1842–1857 (1999)

    Google Scholar 

  55. Pfurtscheller, G., Neuper, C.: Motor imagery and direct brain–computer communication. Proc. IEEE 89, 1123–1134 (2001)

    Google Scholar 

  56. Pfurtscheller, G., Leeb, R., Keinrath, C., Friedman, D., Neuper, C., Guger, C., Slater, M.: Walking from thought. Brain Res. 1071(1), 145–152 (2006a)

    Google Scholar 

  57. Pfurtscheller, G., Leeb, R., Slater, M.: Cardiac responses induced during thought-based control of a virtual environment. Int. J. Psychophysiol. 62, 134–140 (2006b)

    Google Scholar 

  58. Pfurtscheller, G., Müller-Putz, G.R., Schlögl, A., Graimann, B., Scherer, R., Leeb, R., Brunner, C., Keinrath, C., Lee, F., Townsend, G., Vidaurre, C., Neuper, C.: 15 years of BCI research at Graz University of Technology: current projects. IEEE Trans. Neural Syst. Rehabil. Eng. 14, 205–210 (2006c)

    Google Scholar 

  59. Pfurtscheller, G., Leeb, R., Friedman, D., Slater, M.: Centrally controlled heart rate changes during mental practice in immersive virtual environment: a case study with a tetraplegic. Int. J. Psychophysiol. 68, 1–5 (2008)

    Google Scholar 

  60. Pfurtscheller, G., Allison, B., Bauernfeind, G., Brunner, C., Solis Escalante, T., Scherer, R., Zander, T., Müller-Putz, G., Neuper, C., Birbaumer, N.: The hybrid BCI. Front. Neurosci. 4, 42 (2010)

    Google Scholar 

  61. Pineda, J.A., Silverman, D.S., Vankov, A., Hestenes, J.: Learning to control brain rhythms: making a brain–computer interface possible. IEEE Trans. Neural Syst. Rehabil. Eng. 11, 181–184 (2003)

    Google Scholar 

  62. Plass-Oude Bos, D., Duvinage, M., Oktay, O., Delgado Saa, J., Guruler, H., Istanbullu, A., Van Vliet, M., Van de Laar, B., Poel, M., Roijendijk, L., Tonin, L., Bahramisharif, A., Reuderink, B.: Looking around with your brain in a virtual world. In: IEEE Symposium on Computational Intelligence, Cognitive Algorithms, Mind, and Brain (SSCI’2011 CCMB) (2011)

    Google Scholar 

  63. Renard, Y., Lotte, F., Gibert, G., Congedo, M., Maby, E., Delannoy, V., Bertrand, O., Lécuyer, A.: OpenViBE: An open-source software platform to design, test and use brain–computer interfaces in real and virtual environments. Presence (Camb.) 19(1), 35–53 (2010)

    Google Scholar 

  64. Ron-Angevin, R., Diaz-Estrella, A.: Brain–computer interface: Changes in performance using virtual reality technique. Neurosci. Lett. 449(2), 123–127 (2009)

    Google Scholar 

  65. Ron-Angevin, R., Diaz-Estrella, A., Velasco-Alvarez, F.: A two-class brain computer interface to freely navigate through virtual worlds. Biomedizinische Biomed. Tech. (Berl.) 54(3), 126–133 (2009)

    Google Scholar 

  66. Scherer, R., Schlögl, A., Lee, F., Bischof, H., Jansa, J., Pfurtscheller, G.: The self-paced graz brain–computer interface: methods and applications. Comput. Intell. Neurosci. 2007, 1–9 (2007) (Article ID 79826)

    Google Scholar 

  67. Scherer, R., Lee, F., Schlögl, A., Leeb, R., Bischof, H., Pfurtscheller, G.: Towards self-paced brain–computer communication: Navigation through virtual worlds. IEEE Trans. Biomed. Eng. 55(2), 675–682 (2008)

    Google Scholar 

  68. Slater, M., Usoh, M., Steed, A.: Taking steps: the influence of a walking technique on presence in virtual reality. ACM Trans. Comput. Hum. Interact. 2(3), 201–219 (1995)

    Google Scholar 

  69. Taylor, R.M., Hudson, T.C., Seeger, A., Weber, H., Juliano, J., Helser, A.: VRPN: A device-independent, network-transparent VR peripheral system. In: VRST ’01, Proceedings of the ACM symposium on Virtual reality software and technology, pp. 55–61. ACM, New York, NY, USA (2001)

    Google Scholar 

  70. Touyama, H.: Advances in Human Computer Interaction, InTech Education and Publishing, chap Brain-CAVE Interface Based on Steady-State Visual Evoked Potential, pp. 437–450 (2008). No. 26 in ISBN 978-953-7619-15-2

    Google Scholar 

  71. Vaughan, T.M., Wolpaw, J.R., Donchin, E.: EEG-based communication: prospects and problems. IEEE Trans. Rehabil. Eng. 4, 425–430 (1996)

    Google Scholar 

  72. Velasco-Álvarez, F., Ron-Angevin, R.: Free virtual navigation using motor imagery through an asynchronous brain–computer interface. Presence (Camb.) 19(1), 71–81 (2010)

    Google Scholar 

  73. Vialatte, F., Maurice, M., Dauwels, J., Cichocki, A.: Steady-state visually evoked potentials: Focus on essential paradigms and future perspectives. Prog. Neurobiol. 90, 418–438 (2010)

    Google Scholar 

  74. Williamson, J., Murray-Smith, R., Blankertz, B., Krauledat, M., Müller, K.: Designing for uncertain, asymmetric control: Interaction design for brain–computer Int. J. Hum. Comput. Stud. 67(10), 827–841 (2009)

    Google Scholar 

  75. Wolpaw, J., Birbaumer, N., McFarland, D., Pfurtscheller, G., Vaughan, T.: Brain–computer interfaces for communication and control. Clin. Neurophysiol. 113(6), 767–791 (2002)

    Google Scholar 

  76. Zander, T., Kothe, C.: Towards passive brain–computer interfaces: applying brain–computer interface technology to human-machine systems in general. J. Neural Eng. 8(2), 025005 (2011)

    Google Scholar 

  77. Zickler, C., Donna, V.D., Kaiser, V., Al-Khodairy, A., Kleih, S., Kuebler, A., Malavasi, M., Mattia, D., Mongardi, S., Neuper, C., Rohm, M., Rupp, R.: Brain computer interaction applications for people with disabilities: Defining user needs and user requirements. In: AAATE (2009)

    Google Scholar 

Download references

Acknowledgements

This work was supported by the European Union projects PRESENCIA (IST-2001-37927) and PRESENCCIA (IST-2006-27731), furthermore by the French National Research Agency projects OpenViBE (ANR-05-RNTL01601) and OpenViBE2 (ANR-09- CORD-017).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Fabien Lotte .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2012 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Lotte, F. et al. (2012). Combining BCI with Virtual Reality: Towards New Applications and Improved BCI. In: Allison, B., Dunne, S., Leeb, R., Del R. Millán, J., Nijholt, A. (eds) Towards Practical Brain-Computer Interfaces. Biological and Medical Physics, Biomedical Engineering. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-29746-5_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-29746-5_10

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-29745-8

  • Online ISBN: 978-3-642-29746-5

  • eBook Packages: Physics and AstronomyPhysics and Astronomy (R0)

Publish with us

Policies and ethics