Chapter 11 - 3D graphics, virtual reality, and motion-onset visual evoked potentials in neurogaming

https://doi.org/10.1016/bs.pbr.2016.06.006Get rights and content

Abstract

A brain–computer interface (BCI) offers movement-free control of a computer application and is achieved by reading and translating the cortical activity of the brain into semantic control signals. Motion-onset visual evoked potentials (mVEP) are neural potentials employed in BCIs and occur when motion-related stimuli are attended visually. mVEP dynamics are correlated with the position and timing of the moving stimuli. To investigate the feasibility of utilizing the mVEP paradigm with video games of various graphical complexities including those of commercial quality, we conducted three studies over four separate sessions comparing the performance of classifying five mVEP responses with variations in graphical complexity and style, in-game distractions, and display parameters surrounding mVEP stimuli. To investigate the feasibility of utilizing contemporary presentation modalities in neurogaming, one of the studies compared mVEP classification performance when stimuli were presented using the oculus rift virtual reality headset. Results from 31 independent subjects were analyzed offline. The results show classification performances ranging up to 90% with variations in conditions in graphical complexity having limited effect on mVEP performance; thus, demonstrating the feasibility of using the mVEP paradigm within BCI-based neurogaming.

Introduction

Brain–computer interfaces (BCIs) offer an alternative method of human–computer interaction and may replace input devices that require muscle control such as mice, keyboards, and gaming joysticks. A BCI-controlled computer system translates electrophysiological activity of the brain, measured either invasively or noninvasively that is modulated by a user as they sense stimuli presented visually (Farwell and Donchin, 1988), auditorily (Sellers and Donchin, 2006), tactilely (Müller-Putz et al., 2006), or if they intentionally modulate brain activity using mental imagery (Wolpaw et al., 2002). Due to its high temporal resolution, low inter- and intrasubject variability, ease of use, low cost, and user safety (Chancellor, 2009), electroencephalography (EEG) is the most widely used method to measure cortical activity. EEG is recorded for analysis by placing electrodes on the scalp of the user. BCIs have been researched for a number of decades (Wolpaw et al., 2000) with a major focus on clinical and assistive technology applications; however, recent advances in computing technology have broadened both its application domain and user base. BCI training paradigms have traditionally used gameplay elements in video games to challenge and motivate users. Brain–computer game interaction (BCGI) represents an obvious target application domain for BCI and research in the area has grown significantly in recent years (Coyle et al., 2015, Marshall et al., 2013).

Video gamers are traditionally early adopters of new and novel interaction modalities such as the Nintendo Wii (Nintendo, 2006), Microsoft Kinect (Microsoft, 2010), and PlayStation Move (Sony, 2010). The oculus rift (OCR) (Oculus, 2014b) is a technological breakthrough and represents the state of the art in virtual reality (VR) technology, spurring the development of similar VR platforms such as the HTC VIVE (HTC, 2016) and Samsung Gear VR (Samsung, 2015). The OCR is a virtual reality head-mounted display and by using the concept of stereopsis (Hubel, 1995) can provide the user with the sense of depth in a virtual scene. By combining information from the right and left eyes concerning the right and left fields of view, the human visual processing system provides us with stereoscopic depth perception. The OCR operates on the same principle to reproduce a stereoscopic view of a virtual world using two separate images rendered onto two different LCD screens—one for each eye (Oculus, 2014a). Aside from its technological underpinnings, the financial and commercial strengths of the video gaming industry may help to propel BCI technology to commercial success (Nijholt, 2008).

Most BCI studies involving the use of video games have been used as a means of training the user on how to use a BCI system (Marshall et al., 2013), to keep them motivated and challenged while they learn BCI control and served mainly as an interesting or novel training environment in order to cloak the mundane nature of training to use a BCI paradigm. Inevitably, a trade-off exists between user-centered and data-centered BCI design which refers to the fact that BCI designers sometimes sacrifice gameplay and esthetics to focus on data gathering. As a consequence, most BCI games employ rudimentary graphics and can therefore lack immersion. In the interests of promoting BCI technology to the wider gaming population, it is necessary to improve their esthetics, functional, and technological appeal.

There are a number of brain signatures that can be used for BCI and BCGI often referred to as “BCI paradigms.” The motor imagery paradigm involves intentional modulation of sensorimotor oscillations or rhythms (SMRs) by imagining various limb movements (Allison et al., 2010, Pfurtscheller, 2001, Pfurtscheller and Lopes da Silva, 1999). In Coyle et al. (2015), SMRs are used to control three different BCI action games. In Asensio-Cubero et al. (2015), SMRs are employed to control left and right movements of a game character in a BCI action game. The authors in Rao et al. (2014) developed and tested a cooperative two-player BCI action game based on SMR control. Although SMRs offer an interesting control strategy that can be learned and improved over time, lengthy training periods are common before reliable control can be acquired. Also, there appears to be a nonnegligible portion of the population that cannot gain reliable control in this paradigm (Nijholt et al., 2009, Sannelli et al., 2008, Vidaurre and Blankertz, 2010). Other techniques for BCI control which subvert the requirement for user training and BCI learning problem involve evoking a response through visual, auditory, or tactile stimulation.

For example, steady-state visual evoked potentials (SSVEPs) utilize flashing visual stimuli presented on a screen or light array via a light source such as light-emitting diodes or filament bulbs which are presented to the user to invoke a response in the cortical activity and readable when EEG sensors are focused around the visual cortex (Bin et al., 2009, Cheng et al., 2002, Gao et al., 2003). Each of the stimuli flashes at a different but fixed frequency and each can be related to a command for the BCI system to process. The fundamental blinking frequency of the users’ target stimulus is observed in the EEG signal. Recent BCI-gaming research which has exploited the SSVEP paradigm includes giving user control over a virtual cart in order to collect monetary rewards (Wong et al., 2015) where four SSVEP stimuli offer control representing Up, Down, Left, and Right motion of the cart. In Koo et al. (2015) the authors utilize SSVEPs to control a maze game. SSVEP control of a game character in a popular video game was shown in Kapeller et al. (2012). In Lalor et al. (2005) users were able to control the left- and right-balanced motion of a computer character in an immersive action game. To investigate the feasibility of utilizing the OCR within an SSVEP-based BCI, Koo and Choi (2015) demonstrated that lower frequencies (those < 10 Hz) were able to provide better SSVEP control than the use of higher frequencies (> 10 Hz). SSVEP use within an action game was investigated in Martinez et al. (2007) where the user navigates a racing car around a racing track using four chequer-board patterns representing Up, Down, Left, and Right commands. Although SSVEP BCIs are capable of providing high information transfer rates and a large number of stimuli can be presented within the field of view of the user, the use of flashing stimuli can cause visual fatigue for users, especially after long-term use (Punsawad and Wongsawat, 2013).

The P300 BCI paradigm also utilizes flashing stimuli to evoke a response in the EEG. A positive peak is produced in the EEG around 300 ms after the flash of a stimulus the BCI user attends to (the oddball stimulus) among frequent stimuli which the user ignores (Bos et al., 2009, Finke et al., 2009, Kathner et al., 2015, Pires et al., 2011). The user gazes at their required target among multiple on-screen stimuli, each relating to a specific command for the BCI system to execute. In Korczowski et al. (2015) a P300 version of the popular arcade game “Space Invaders” was controlled by flashing random groups of six on-screen targets and the role of the player was to select the single target item out of the 36 items available. A P300-based puzzle control paradigm employed in a game targeting treatment of children with attention deficit hyperactive disorder was investigated in Rohani et al. (2014) where users also have to contend with other visual distractions in a complete 3D scene of a school classroom. A P300-based game was presented in Finke et al. (2009) where users attempted to control an avatar between flashing trees in the shortest possible route. P300 use has been successful in BCI-gaming studies, and the presentation of multiple stimuli is possible within the field of view of the user but, as with the SSVEP paradigm, P300 depends on continuously flashing stimuli to evoke responses in the EEG which can be visually fatiguing for the user and perhaps may have limited appeal and performance in games that are graphically complex with fast-paced graphical changes. An alternative to these paradigms is the motion-onset visual evoked potential (mVEP) paradigm.

An mVEP response is evoked by motion-related stimuli (Guo et al., 2008a, Guo et al., 2008b, Marshall et al., 2015a, Marshall et al., 2015b). The saliency of a sudden movement of an object or stimuli produces responses in the dorsal pathway of the brain. The dorsal pathway extends from the primary visual cortex to the parietal cortex and is known as the “where” pathway; it processes actions such as motion, spatial location, shape, and orientation (Hebart and Hesselmann, 2012). The mVEP response comprises three main peaks. The “P100” peak, a positive deflection in the EEG at between 70 and 110 ms poststimulus followed by the motion-specific “N200” peak, a negative deflection between 160 and 200 ms, and finally the “P200” peak, a positive deflection which occurs between 240 and 500 ms after the evoking stimulus whose amplitude can be increased with more complex stimuli (Guo et al., 2008a, Guo et al., 2008b). Fig. 1 depicts the typical mVEP response (target vs nontarget).

Clear and robust features such as N200 and P200 constitute the neural response exploited in the mVEP-based BCI paradigm. mVEP stimuli may consist of a black rectangle containing a white center and a moving red vertical line placed within the center subtending a visual field of 0.76 degree height × 1.24 degree width (Guo et al., 2008a, Guo et al., 2008b, Marshall et al., 2015a, Marshall et al., 2015b). The moving red line is 0.66 degree in height as shown in Fig. 2. The red vertical line begins motion (motion-onset) starting on the right-hand side of the vacant rectangle and continues to the left-hand side in one continuous motion lasting 140 ms (motion-offset). A number of stimuli can be delineated by positioning at different locations on the screen.

The use of motion rather than flashing stimuli renders mVEP a more elegant visual paradigm than SSVEP or P300, and as a consequence, mVEP does not induce visual fatigue making it apposite for use in visually rich BCI video games. Also, due to the natural ability of the brain to capture the onset of motion (Abrams and Christ, 2003), mVEP can be used without the need for long training sessions. It is also possible to present more compact visual stimuli onto the game environment allowing more of the screen space to be used for the game.

As the mVEP is a relatively new BCI paradigm, it has been relatively understudied, especially in BCI-controlled games. mVEP, however, may offer beneficial characteristics which are suitable for use within visually rich video games. In a previous study by Marshall et al., 2015a, Marshall et al., 2015b, the mVEP paradigm was evaluated with three games belonging to the sports, puzzle, and action genres, each with five control options, ie, five stimuli to select from. Their findings demonstrated when users trained within a dedicated (plain) training level, average offline, and online accuracies of 74% and 66%, respectively, were possible across all three game levels. In a further study by Marshall et al., 2015a, Marshall et al., 2015b, they investigated the use of online mVEP control within the same three game genres with stimuli presented on a heads-up-display as opposed to overlaid onto the game graphics when users were trained within the same level as they were tested in. The study concluded that the average online accuracy had improved to 85% compared to the findings of the previous study (74%).

Here, we present three studies which investigate the influence on the mVEP paradigm using various components of video games namely graphical variations, visual parameters, and presentation modalities.

  • Study 1: The aim of study 1 is to investigate the effects on mVEP system accuracies on video game presentations when using graphics and in-game distractions of various complexities. mVEP stimuli were placed overlying the game scene to test the effects of in-game distractions surrounding the stimuli. Rudimentary game graphics based on the action genre were utilized.

  • Study 2: As an extension to study 1, we utilized the same mVEP system setup, but instead of the rudimentary graphics, we tested popular commercially available video game presentations of increasingly advanced graphical fidelities. In order to test if mVEP accuracies were affected by in-game distractions, the stimuli were placed within a dedicated controller area consisting of a plain white background.

  • Study 3: Using the findings gained from studies 1 and 2, we developed two game level presentations, one with basic and one with complex graphics both based on the action genre. We also compared the use of two different presentation environments for the mVEP stimuli—one where the stimuli were presented within a plain white background (dedicated stimuli area) and one where the stimuli were located within the game scene. Moreover, to investigate the effects of utilizing VR technology with mVEP-based BCI games and prove the concept of mVEP stimuli in VR for the first time, we compared the use of two display modalities namely the OCR and LCD computer screen.

Section snippets

Data Acquisition

All data were recorded in an electrostatically shielded, acoustic noise insulated room, and participants were seated in a comfortable chair 50 cm in front of a 56-cm (width 47.7 cm and height 29.8 cm) LCD screen (except in study 3 when using OCR as the display modality). During the course of each game level, each of five mVEP stimuli was active 60 times yielding data from 300 trials, ie, 60 target trials for each stimulus per game level. In order to avoid habituation, during each trial the stimuli

Data Preprocessing

Data from all studies were analyzed offline. The data for each stimulus were epoched at 1200 ms, triggered 200 ms prior to motion-onset. Using the 200 ms preceding motion-onset of the five stimuli, the mean voltage was baseline corrected. Using a low-pass Butterworth filter (order 5, cutoff at 10 Hz) the data were digitally filtered and then subsequently resampled at 20 Hz. The most reactive mVEP components, ie, P100, N200, and P200, appear in the EEG between 70 and 500 ms; therefore, features were

Study 1—Comparing Graphical Complexity (Basic Games)

Results for study 1 are presented in Fig. 21. Here, we compare the basic game level 1 (white background—no graphics) with each of the other levels. Across all the tests there is a drop in accuracy between the basic level game graphic (Game 1) to the Game 2 level graphics; however, these drops are only significant for the 2-class results (p < 0.05). For all tests, there is no difference between the Game 1 and Game 3 or 4, however, comparing the results of the most basic vs most complex Game levels

Discussion

We have studied the effects on classification accuracies using various graphical properties in games based on the mVEP and compared the use of two different approaches for presenting the mVEP stimuli namely the dedicated “controller area” (W) vs the stimuli overlaid onto the game scene (NW). Results suggest that graphics containing various complexities including high-end graphics as seen in popular commercial video games can be used in an mVEP-based BCI environment while still maintaining

Conclusion

Considering the vast range of video games currently available which span many genres, platforms, demographic groups, and complexity, it is clear that video games do not wholly depend on rich 3D, realistic or state-of-the-art graphics to be attractive. Other properties of video games are just as important such as but not limited to gameplay mechanics, rewarding gameplay social interaction, and novel interaction technology. However, for BCI technology to be shown in the best possible light and

References (63)

  • J. Asensio-Cubero et al.

    Multiresolution analysis over graphs for a motor imagery based online BCI game

    Comput. Biol. Med.

    (2015)
  • G. Bin

    An online multi-channel SSVEP-based brain-computer interface using a canonical correlation analysis method

    J. Neural Eng.

    (2009)
  • D.P. Bos

    Bacteria hunt: a multimodal, multiparadigm BCI game

  • A. Chancellor

    Electroencephalography: maturing gracefully

    Pract. Neurol.

    (2009)
  • M. Cheng

    Design and implementation of a brain-computer interface with high transfer rates

    IEEE Trans. Biomed. Eng.

    (2002)
  • D. Coyle

    Action games, motor imagery, and control strategies: toward a multi-button controller

  • Easycap

    Easycap EEG Cap

  • g.Tec

    g.tec—g.BSamp Biosignal Amplifier

    (2016)
  • g.Tec

    g.Tec—g.GAMMAsys

    (2016)
  • X. Gao

    A BCI-based environmental controller for the motion-disabled

    IEEE Trans. Neural Syst. Rehabil. Eng.

    (2003)
  • F. Guo

    A brain computer interface based on motion-onset VEPs

  • F. Guo

    A brain–computer interface using motion-onset visual evoked potential

    J. Neural Eng.

    (2008)
  • M.N. Hebart et al.

    What visual information is processed in the human dorsal stream?

    J. Neurosci.

    (2012)
  • H.J. Heinze

    Combined spatial and temporal imaging of brain activity during visual selective attention in humans

    Nature

    (1994)
  • HTC

    HTC VIVE

  • D.H. Hubel

    Eye, Brain and Vision

    (1995)
  • C. Kapeller et al.

    Augmented control of an avatar using an SSVEP based BCI

  • I. Kathner et al.

    Rapid P300 brain-computer interface communication with a head-mounted display

    Front. Neurosci.

    (2015)
  • B. Koo et al.

    SSVEP response on oculus rift

  • B. Koo

    Immersive BCI with SSVEP in VR head-mounted display

  • L. Korczowski

    Single-trial classification of multi-user P300-based brain-computer interface using Riemannian geometry

    37TH Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)

    (2015)
  • Cited by (13)

    • Effects of fatigue on steady state motion visual evoked potentials: Optimised stimulus parameters for a zoom motion-based brain-computer interface

      2020, Computer Methods and Programs in Biomedicine
      Citation Excerpt :

      The most common SSVEP-BCI stimulation paradigms is based on flicker which realized by graphic pattern reversal [2–5], the number of target is related to the refresh rate of the screen. Recent studies realized the flicker-based stimulation by changing the luminance flux of the graphic in a sinusoidal mode [6,7]. This method has become the main visual stimulation paradigm for BCIs as it produces precise frequencies and achieves high pattern recognition accuracies and information transmission rates (ITR) [8–10].

    • A multiplayer online car racing virtual-reality game based on internet of brains

      2018, Journal of Systems Architecture
      Citation Excerpt :

      For example, Leeb et al. [7] showed that it is possible to use an EEG sensorimotor rhythm (SMR)-based BCI to control a penguin in a projection display-based VE (i.e., CAVE VR), where the SMR feature at the vertex was induced by actively performing a foot motor imagery. Reactive BCI-VR utilizes the brain signals (e.g., P300 [1], SSVEP [12], motion-onset VEP [13]) elicited by external stimuli to control objects in VEs [14–16], where VR plays the role of stimulus generation and feedback. State-of-the-art BCI-VR technologies have shown the feasibility of developing a safe BCI-VR training platform or designing interactive BCI-VR games (see [17] for a review).

    • Online 3D Motion Decoder BCI for Embodied Virtual Reality Upper Limb Control: A Pilot Study

      2022, 2022 IEEE International Workshop on Metrology for Extended Reality, Artificial Intelligence and Neural Engineering, MetroXRAINE 2022 - Proceedings
    View all citing articles on Scopus
    View full text