Review and Classification of Emotion Recognition Based on EEG Brain-Computer Interface System Research: A Systematic Review

Abstract

Recent developments and studies in brain-computer interface (BCI) technologies have facilitated emotion detection and classification. Many BCI studies have sought to investigate, detect, and recognize participants’ emotional affective states. The applied domains for these studies are varied, and include such fields as communication, education, entertainment, and medicine. To understand trends in electroencephalography (EEG)-based emotion recognition system research and to provide practitioners and researchers with insights into and future directions for emotion recognition systems, this study set out to review published articles on emotion detection, recognition, and classification. The study also reviews current and future trends and discusses how these trends may impact researchers and practitioners alike. We reviewed 285 articles, of which 160 were refereed journal articles that were published since the inception of affective computing research. The articles were classified based on a scheme consisting of two categories: research orientation and domains/applications. Our results show considerable growth of EEG-based emotion detection journal publications. This growth reflects an increased research interest in EEG-based emotion detection as a salient and legitimate research area. Such factors as the proliferation of wireless EEG devices, advances in computational intelligence techniques, and machine learning spurred this growth.

1. Introduction

A Brain Computer Interface (BCI) is a system that takes a biosignal, measured from a person, and predicts (in real-time) certain aspects of the person’s cognitive state [1,2]. At the outset, BCIs started as assistive technological solutions for individuals with significant speech anomalies. However, the research was rooted in a subject’s desire to communicate through either speech or writing or to control his or her immediate environment. BCI systems have also used computer-based recreational activities to stimulate a subject’s innate ability to overcome physical disabilities.

Today, BCI-based research has been expanded to include people with and without physical disabilities. The entire system underscores how adaptive systems can enhance analytic methods and application areas. This assistive ability has created a widespread awareness among potential users and researchers alike. In the past 15 years, the increasing numbers of BCI research groups, peer-reviewed journals, conference abstracts, and attendees at relevant conferences are indicators of the rapid growth of interest in this field. Aside from these indicators, numerous companies are collaborating with research groups to develop BCI-related technologies. These companies have further defined clear roadmaps for BCI-related technologies.

There are a number of annual conferences in human-computer interaction (HCI) and BCI fields, which bring together prominent researchers to present their research projects, such as the ACM Conference on Human Factors in Computing Systems (CHI), the ACM International Conference on Multimodal Interaction (ICMI), the International Conference on Intelligent User Interfaces (IUI), Computer Supported Cooperative Work (CSCW), and the IEEE/ACM International Conference on Computer-Aided Design (ICCAD). These conferences hold many Workshops, case studies, and courses that are conducted by industry experts, practitioners, and researchers.

The phenomenal growth of BCI research is aligned with an influx of researchers from diverse disciplines, including clinical neurology and neurosurgery, rehabilitative engineering, neurobiology, engineering, psychology, computer science, mathematics, medical physics, and biomedical engineering. The interdisciplinary nature of BCI research has resulted in the development of BCI systems with different target applications.

Research on BCIs has shown that brain activity can be used as an active or passive control modality [3]. In an active BCI, the user controls a device using brain signals and patterns through a direct and conscious generation of commands that are wired to external applications. In contrast, passive BCIs are systems wherein brain signals yield outputs without any voluntary control. Emotional states, such as levels of meditation, engagement, frustration, excitement, and stress, are examples of affective and cognitive feedback in passive BCIs.

The concept of passive BCI has been applied in various fields, such as affective computing, which aims to improve the communication between individuals and machines by recognizing human emotions, and to develop applications that adapt to changes in user state, and thereby enrich the interaction. The entire communication process enriches the interaction, leading to a natural and effective user experience. Motivated by a new understanding of brain functions and advances in computer interface devices, countless research studies on emotion detection in real-time procedures for patients and clinicians have been undertaken (e.g., [4,5,6]). Furthermore, similar advancements are under development for additional cognitive mental states, such as attention and workload. These cognitive mental states correspond to affective states. These and many other advances in BCI technologies have piqued scientific interest in BCI technology and its application in different contexts.

According to the Gartner’s 2016 Hype Cycle report on trending research topics, both Brain-Computer Interface and Affective Computing are at the Innovation Trigger stage. These researchers predict that mainstream adoption will occur in 5–10 years for Affective Computing, and in more than 10 years for BCI research. This phenomenon is captured in Figure 1.

The volume of studies, research, and publications on BCI emotion-based recognition systems has surged in recent years. A plethora of studies with varied research methodologies has led to a broad range of results, depending on the datasets, recording protocol, emotion elicitation technique, detected features, temporal window, classifiers, involved modality, number of participants, and emotion models.

While BCI research encompasses a wide spectrum of applied domains, our research interest is specifically focused on electroencephalography (EEG)-based emotion detection, although objective measures of emotion can also be acquired from physiological cues derived from the physiology theories of emotion. Instruments that measure blood pressure responses, skin responses, pupillary responses, brain waves, heart responses, facial recognition, speech, and posture are often used as objective measures in affective computing [7]. This review will seek to understand EEG-based emotion recognition trends. The review will examine published literature with the aim of providing insights for future emotion recognition systems to practitioners and researchers.

During our literature review, we observed that articles on emotion recognition cut across various disciplines, including clinical neurology, rehabilitation, neurobiology, engineering, psychology, computer science, medical physics, and biomedical engineering. Hence, conducting a comparative analysis of articles is difficult since different journals and different scientific domains have different research focuses and methodologies.

Accordingly, the main objective of this review is to classify and summarize research that is relevant to emotion recognition systems and to provide conceptual frameworks for integrating and classifying emotion recognition articles. This system of classification will be useful for literature reviews on emotion recognition research.

The following sections illustrate our proposed classification framework for emotion recognition literature reviews based on research articles. In Section 2, we outline our research methodology. Section 3 describes our proposed classification framework for emotion-recognition-based literature reviews. Section 4 presents our discussion. In Section 5, we provide insights for future research and discuss the challenges and trends in EEG-based emotion recognition. Finally, in Section 6, we present the study’s conclusions.

2. Research Methodology

Articles on emotion recognition systems are scattered across journals of various disciplines and were found in both medical and non-medical journal publications, including clinical neurology, rehabilitation engineering, neurobiology, engineering, psychology, computer science, medical physics, and biomedical engineering.

We searched Web of Science (WoS), https://webofknowledge.com, to obtain a comprehensive bibliography of the academic literature on emotion-recognition-based BCI. The Web of Science Core Collection database provides us with quick, powerful access to the world's leading citation databases, such as Science Direct (Elsevier), IEEE/IEE Library, ACM Digital Library, Springer Link Online Libraries, and Taylor & Francis.

The following subsections describe the procedure we followed in extracting articles, along with our article selection criteria and filtering processes.

2.1. Data Sources and Procedures for the Extraction of Articles

We searched our selected database (Web of Science Core Collection database) for articles over a span of eleven years, 2005–2016. We used Basic Search to look for topics that fall into our research scope. Using basic search settings, we input search terms and phrases, such as: affective or emotion; emotion detection or recognition; EEG or Electroencephalography; and, Brain-computer interface, Passive BCI or BCI. According to Web of Science search result templates, auto-generated search terms are a result of searches covering articles, meeting abstracts, book chapter(s), and proceedings papers.

The initial search resulted in 766 articles.

Table A1. Distribution of the results according to the Web of Science categorization.

Range	Field/Domain	Number of Publications
>100	Neurosciences	202
10–100	Psychology Experimental	64
	Psychology	62
	Psychiatry	44
	Clinical Neurology	43
	Physiology	40
	Computer Science Artificial Intelligence	37
	Behavioral Sciences	34
	Psychology Biological	25
	Neuroimaging	24
	Radiology Nuclear Medicine Medical Imaging, Multidisciplinary Sciences	23
	Psychology Multidisciplinary	18
	Computer Science Cybernetics	17
	Psychology Developmental, Engineering Biomedical	14
	Engineering Electrical Electronic	13
	Computer Science Information Systems	10
1–9	Mathematical Computational Biology, Computer Science Interdisciplinary Applications	9
	Rehabilitation	7
	Psychology Clinical	6
	Robotics, Psychology Social, Pharmacology Pharmacy, Medical Informatics, Audiology Speech-Language Pathology	5
	Operations Research Management Science, Medicine Research Experimental, Linguistics, Ergonomics, Computer Science Theory Methods, Computer Science Software Engineering	4
	Public Environmental Occupational Health, Pediatrics, Instruments Instrumentation, Engineering Multidisciplinary, Education Special	3
	Telecommunications, Medical Laboratory Technology, Electrochemistry, Computer Science Hardware Architecture, Chemistry Analytical, Biology, Automation Control Systems, Anesthesiology	2
	Substance Abuse, Sport Sciences, Social Issues, Psychology Psychoanalysis, Optics Otorhinolaryngology, Medicine General Internal, Materials Science Multidisciplinary, Materials Science Biomaterials, Integrative Complementary Medicine, Ethics, Endocrinology Metabolism, Geriatrics Gerontology, Genetics Heredity, Imaging Science Photographic Technology, Health Care Sciences Services, Education Educational Research, Biotechnology Applied Microbiology, Chemistry Medicinal, Acoustics	1

in Appendix A shows a broad categorization and distribution of our search results. We further refined our search based on some predetermined criteria. The following section illustrates our selection criteria.

2.2. Selection Criteria

Three criteria were used to select and accept emotion recognition articles for further review. Articles were excluded if they did not meet the following selection criteria:

• Articles must address one of the Gartner Hype Cycle 2016 trending research topics. To meet this criterion, they must be relatively current. In this regard, we chose articles that were published between 2005 and 2016. This 11-year period could be considered to correspond to the main research period of emotion recognition systems.
• We excluded meeting abstracts, book chapters, conference proceedings, workshop descriptions, masters and doctoral dissertations, and non-English articles. Notably, the number of conference papers in this domain was 322, the number of book chapters was 3, and the number of meeting abstracts was 6.
• We also ensured that only peer-reviewed journal articles were included. The logic behind this is that practitioners and academics frequently use journals to both obtain and spread research findings. Thus, journal articles contain the highest level of research.

2.3. Filtering/Reviewing Process

In this step, each article was manually reviewed in three rounds. This was done to eliminate non-emotion-recognition-based works and non-BCI-based works. After that step, we classified the filtered articles according to our classification scheme.

During our first round of review, we excluded articles based on our predetermined selection criteria. We excluded all articles that did not address EEG-based emotion recognition systems. After applying our selection criteria, we were left with 435 articles. We directly imported all of the articles into an online EndNote database that facilitated, managed, assessed, and reviewed our result articles. The remaining articles were then scanned and filtered, as described in the next subsection.

Our second round of review involved the manual scanning of titles, abstracts, authors, keywords, and conclusions. This round of review excluded articles in which the central theme of the discussion was centered on subjects other than emotion recognition systems, with emotion recognition systems only being mentioned in passing. By the end of this round, we were left with 340 articles.

The final round involved reading the full texts and analyzing each article according to our classification scheme. This scheme is described in the next section. The infographic in Figure 2 shows the procedure used to filter and extract articles that meet our predetermined criteria.

3. Classification Method

To systematically reveal and examine research insights on EEG-based emotion recognition, a literature classification scheme was developed. This classification scheme was based on categorizing the research focuses of the 285 selected articles that remained after the filtering processes. A graphical representation of these categories and subcategories and their relationships is presented in Figure 3.

3.1. Application Domain and Field

EEG-based emotion detection applications can be categorized into two broad domains: Medical and Non-medical. They are further described in the following subsections.

3.1.1. Medical Context

EEG emotion recognition is medically contextual if the system is designed to provide assistance, enhancement, monitoring, assessment, and diagnosis of human psychiatric and neurological diseases.

3.1.2. Non-Medical Context

Includes emotion recognition systems based on EEG signals that are designed to entertain, educate, monitor, or play games.

4. Results and Discussion

We extracted 285 articles on emotion recognition systems from 29 online databases with 160 different journals. Each article was reviewed and classified according to our classification scheme. Although the extensity of our search was limited, it offers a comprehensive insight into EEG-based emotion recognition system research. Furthermore, we provided a descriptive overview of temporal trends of these types of publications, along with a description of publication domains (e.g., publication research area, online database and journal) in Appendix B.

4.1. Classification of Articles by Paper Orientation

When we classified journal articles according to paper orientation, we found 20 review articles and 265 experimental design innovation articles.

4.1.1. Review Paper

We found twenty review papers. Seventeen of the articles were narrative reviews, and three were systematic reviews. We classify these papers according to their focus: general/background, signal processing, classification, application, and others. We found seven general reviews, including neuroimaging techniques, emotion modeling, and applications. We also found three signal processing reviews, one classification method, seven application reviews, and two reviews that covered the training protocols, validity in the EEG-neurofeedback optimal performance field, and evidence of neurofeedback learning.

Table 1. Review article classifications.

Focus	Description		References
General/background	Review recent studies that investigate the recognition of affective states from EEG signals. Aim to present a general discussion on one or more aspects, such as the neuroimaging techniques, emotion representation models, physiological signals, stimulus, feature extraction, and classification, and discuss their current achievements and applications.		Liberati et al., 2015 [7], Verma et al., 2014 [8], Rule et al., 2013 [9], Keysers and Fadiga, 2008 [10], Grossmann and Johnson, 2007 [11], Muthukumaraswamy and Johnson, 2007 [12], Schupp et al., 2006 [3]
Signal Processing	Survey the recent developments in the field of EEG signal processing, including filtering and artifact processing, signal enhancement methods, feature extraction methods, and channel selection methods.		Alotaiby et al., 2015 [13], Jenke et al., 2014 [14], Knyazev, 2007 [15]
Classification	Survey the recent developments in the field of machine learning, including the classification methods, performance evaluation approaches, and post-processing methods.		Kim et al., 2013 [16]
Application	Medical	Schizophrenia	Isaac et al., 2016 [17], Campos et al., 2016 [18]
		Disorders of consciousness	Harrison et al., 2013 [19]
		Depression	Acharya et al., 2015 [20]
		Autism	Bhat et al., 2014 [21]
	Non-medical	Games	Bontchev and Boyan, 2016 [22]
		Traffic Safety	Reyes-Munoz et al., 2016 [23]
Other	Include a review of the training protocols, validity in the EEG-neurofeedback optimal performance field, and evidence of neurofeedback learning.		Gruzelier et al., 2014 [24] Harmon-Jones et al., 2009 [25]

shows the review article classifications.

The objectives of the earliest reviews (2007 and 2008) were to understand the mirror neurons; cover the EEG correlates of emotions, brain rhythms, and functions; and, present a framework and suggestions for a research program [3,12]. We also found two published review articles that aim mainly to highlight an emerging field, such as the neuroscience of culture in [9] and social neuroscience in [10].

Other review articles were published in medical fields. These articles focused on EEG applications and how to use EEGs to diagnose and assess medical conditions. These articles also explored the relationships between symptoms and affective states, such as schizophrenia [17,18], depression [20], disorders of consciousness [19], and autism [21].

4.1.2. Design Innovation (Experimental) Paper

A number of design innovation papers (265) were published in the period of 2005–2016. Technical aspects of design and design implications are important, and there are still many technological challenges and obstacles to the growth of EEG-based emotion detection systems. Therefore, information was extracted from each study on the following aspects: affective states investigated, emotion elicitation method, number of participants, acquisition technique, and emotion recognition techniques, including feature extraction, type of classifier, performance of the classifier, and online (real-time) vs. offline classification.

Emotion model: Emotions are traditionally classified on the basis of two models: the discrete and dimensional models of emotion. Dimensional models of emotion propose that emotional states can be accurately represented as combinations of several psychological dimensions. Most dimensional models incorporate valence and arousal. Valence refers to the degree of ‘pleasantness’ that is associated with an emotion, whereas arousal refers to the strength of the experienced emotion. Discrete theories of emotion propose the existence of small numbers of separate emotions, as characterized by coordinated response patterns in physiology, neural anatomy, and morphological expressions. Six basic emotions that are frequently specified in research papers are happiness, sadness, anger, disgust, fear, and surprise.

Another emotion model is the appraisal model. The appraisal model of emotion is based on the evaluation of currently remembered or imagined circumstances. Basically, the appraisal model proposes that thought precedes emotion and that emotion precedes behavior. The majority of the papers (172 articles, 64.91%) clearly specify that they used a dimensional model, whereas (34 articles, 12.83%) used a discrete model; and, 21.89% of articles used a different model or did not specify which emotion model they used.

Emotion elicitation technique: Different emotion elicitation techniques have been developed and reported. Critical examples of emotion elicitation techniques include standardized emotional stimuli (e.g., pictures, films, and audio), imagination techniques (e.g., guided imagery and autobiographic recall), present social interactions (e.g., games), and directed facial action tasks. We found that researchers have used different methods to elicit target emotions. In addition, other researchers have found that the use of multiple stimuli is more effective in eliciting emotions than the use of single stimuli.

Table 2. Emotion elicitation techniques.

Technique	Number of Articles	Domain (Medical, Non-Medical)
Visual-based elicitation using images	88	26% , 73.9%
Prepared task	43	25.6%, 47.4%
Audio-visual elicitation using short film video clips	38	18.4%, 81.6%
Audio-based elicitation using music	29	17.2%, 82.8%
Multiple techniques	19	26.3%, 73.9%
Other	17	11.7%, 88.2%
Imagination techniques/memory recall	10	20%, 80%
Social interactions	4	25%, 75%

shows the numbers of articles that used a combination of different emotion elicitation techniques.

When using elicitation techniques to evoke emotions, the emotional stimuli (image, audio, and video) are usually taken from reputable sources, such as the International Affective Picture System (IAPS) database and the International Affective Digitized Sounds (IADS). In addition, other databases and video clips can be collected from various resources on the internet (e.g., YouTube and Facebook). Other modalities, such as the recall paradigm, where the subject is asked to repeatedly recall emotional instances from their life, and dyadic interaction, where a facilitator helps to induce various emotions, are also used by researchers.

Although the affective information from image, video, and audio stimuli has been extensively studied, olfactory stimuli [26], written words [27,28,29,30,31,32], food stimuli (enriched by emotional stimuli) [33], and games have been used as elicitation methods in a number of studies as ways to assess human emotional state by investigating physiological signals [34,35,36,37].

Single/multimodality: Recently, many studies have shown that combinations of modalities can significantly enhance the emotion recognition accuracy in comparison with single modalities. For example, combining eye movements, which are measured using an eye tracking method, and EEG can considerably improve the performance of emotion recognition systems [38,39]. Moreover, in [40], the researchers proposed a multi-modal emotion recognition system using four sensing methods: EEG, heart inter-beat interval, galvanic skin response, and stressor level lever. The purpose of this study was to measure the human stress response during using a powered wheelchair. It also provided a comprehensive background for multi-modal emotional state and proposed a frame work and discussed the feasibility and utility of using the multi-modal approach. In [8], the researchers used multimodal physiological signals EEG, Galvanic skin response, blood volume pressure, respiration pattern, skin temperature, electromyogram, and electrooculogram to classify and predict depression. They proved the potential of the multimodal approach and achieved 85.46% accuracy using support vector machine (SVM). The work in [41] proposes an approach for multi-modal video-induced emotion recognition, based on facial expression and EEG technologies. In [42], the researchers proposed an emotion recognition system that used a large number of modalities: galvanic skin response, heart rate, blood volume pulse, electrocardiogram, respiration, facial electromyograph, and EEG, while the subjects were watching the affective eliciting materials to distinguish six different kinds of emotions (joy, surprise, disgust, grief, anger, and fear). They stated that the integration of these methods facilitates a more detailed assessment of human affective state, which also improved accuracy and robustness. In [43], the researchers examined the role of emotional arousal on subsequent memory in school-age children. They used EEG, heart rate, and respiration. Their findings endorsed the value of combining multiple methods to assess emotion and memory in development.

Most previous research into emotion recognition used either a single modality or multiple modalities of different physiological signals. The former method allows for the limited enhancement of accuracy, and the latter has the disadvantage that its performance can be affected by head or body movements. Furthermore, the latter causes inconvenience to the user due to the sensors that are attached to the body. In our review, we found that the majority of the papers (218 articles, 82.3%) used a single modality, namely, EEG signals, as an objective method in their studies, whereas (45 articles, 17%) used multiple modalities.

Recognition model: The analysis and classification of the EEG signal can be performed either online or offline. The performance time begins when features are extracted and ends when classification has been completed. The number of electrodes that are used during experimentation in emotion detection situations imposes time constraints on the algorithms. For example, in [44], the authors built a system that detects current user affective states and obtained a classification accuracy of 65%. In [45,46], they tested their method online and offline. We observed that the majority of the 187 articles mentioned that they were using offline analysis, whereas only 15 articles used online analysis.

EEG device: A number of EEG devices have been used in EEG-based BCI research in medical and academic settings. We found up to 48 different devices from different companies all over the world. In

Table 3. Electroencephalography (EEG) devices.

EEG Device	Number of Articles	References
Quik-cap, NuAmps (Compumedics NeuroScan Inc., El Paso, TX, USA)	33	[4,28,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77]
Active-electrodes (BioSemi Inc., Amsterdam, Netherlands)	28	[5,6,34,35,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101]
EPOC (Emotiv Inc., San Francisco, CA, USA)	24	[2,39,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123]
Geodesic Sensor Net (Electrical Geodesics Inc., Eugene, OR, USA)	22	[26,30,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143]
actiCAP, EASYCAP, BrainCap (Brain Products Inc., Munich, Germany)	22	[29,31,44,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162]
EasyCap (FMS, Herrsching-Breitbrunn, Germany)	15	[27,37,163,164,165,166,167,168,169,170,171,172,173,174,175]
Electro-Cap (Electro-Cap International Inc., Eaton, OH, USA)	9	[176,177,178,179,180,181,182,183,184]
g.MOBIlab, g.EEGcap (g.tec Guger Technologies Inc., Graz, Austria)	7	[33,45,185,186,187,188,189]

, we specify the most commercially available and widespread devices that have been used in more than five articles.

Electrodes: When considering the number of electrodes, time interval required to set up an EEG device, comfort level of subjects, system usability, and number of features to be processed, it is advised, from this standpoint, that fewer electrodes be utilized; for example, five channels were used in [176,190]. Nonetheless, most current EEG devices still require a relatively large number of electrodes; for example, 64 channels were used in [93,95], and 32 channels were used in [43,71,191]. We found that the maximum number of electrodes that were used when recording EEG signals was 257 in [141], whereas the minimum number was one in [192]. In this study, the electrode was placed at the Fpz according to the international 10–20 system, and two reference electrodes were located on the left and right ear lobes. Notably, other physiological signals were recorded in this study, including facial electromyogram, electroencephalography, skin conductivity, and respiration data.

Benchmark EEG emotional databases: Only a small number of benchmark emotional EEG databases with categorized emotions are publicly available for use and to test a new system. These datasets are available to the academic community via a web-based system and have been used in a number of research studies to test and explore proposed systems.

Table 4. Benchmark EEG emotional databases.

Dataset	Description	References
DEAP	It is a multimodal dataset for the analysis of human affective states. The electroencephalogram (EEG) and peripheral physiological signals of 32 participants were recorded as each watched 40 one-minute-long excerpts of music videos. Participants rated each video in terms of the levels of arousal, valence, like/dislike, dominance, and familiarity. For 22 of the 32 participants, frontal face video was also recorded. The database is described in [92] and is available online at http://www.eecs.qmul.ac.uk/mmv/datasets/deap/.	[92,194,195,196,197,198,199,200,201,202,203,204,205]
SEED	The EEG signals of 15 subjects were recorded while they were watching emotional film clips. For the feedback, participants were told to report their emotional reactions to each film clip by completing a questionnaire immediately after watching each clip. To investigate neural signatures and stable patterns across sessions and individuals, each subject was required to perform the experiments for three sessions. The time interval between two sessions was one week or longer. Facial videos and EEG data were recorded simultaneously. The database is available online at http://bcmi.sjtu.edu.cn/~seed/download.html.	[206]
MAHNOB	It is a multimodal database for emotion recognition and implicit tagging. It includes the physiological signals from 27 participants in response to 20 stimulus videos. Subjects’ emotional self-assessments are nine-scale evaluations, from 1 to 9, for both valence and arousal. The database is described in [101] and is available online at http://www.ibug.doc.ic.ac.uk/resources/mahnob-hci-tagging-database/.	[41,207,208]
eNTERFACE06_EMOBRAIN	It is a multimodal database for emotion recognition. It contains physiological signals from both the peripheral (galvanic skin response, respiration, and blood volume pressure) and central (EEG and frontal fNIRS) nervous systems from five subjects in response to picture stimuli. The database is available online at http://www.enterface.net/enterface06/docs/results/databases/eNTERFACE06_EMOBRAIN.	[209]

lists the benchmark datasets, along with descriptions and references to the articles that have used them. However, most of the reviewed articles (239 articles, 88.7%) recorded their own EEG signals and used them during their experiments.

Moreover, in [193], the authors evaluated their approach on three benchmark databases: DEAP, MAHNOB-HCI, and USTC-ERVS. USTC-ERVS is an EEG dataset that contains 197 EEG responses to 92 video stimuli from 28 users, along with users’ emotional self-assessments. However, the USTC-ERVS dataset is no longer available online. Either its page was moved or the URL was completely expunged from the initial database location.

Participants: The number of participants can vary based on the experiment type and field. In our review, we found that the minimum number of participants was one. In [137], an EEG investigation was carried out on a patient with complete cortical blindness who presented with affective blindness. The maximum number of participants was 300 in [42]. In this study, EEG-based physiological signals of Chinese college students were recorded to establish a large effective physiological signal database.

Moreover, different age groups ranging from infants [132,143,169], children [6,43,75,84,89,103,125,126,128,129,130,140,183,184,210,211,212,213], adolescents [59,65,70,135,139], and elderly [105,214] in different experiments were considered as study samples. Some studies were conducted on a single gender, such as those investigating women's emotions during motherhood and parental status [133,134,164] and a study of men [215]. Some studies targeted a specific type of subject, such as healthy people or patients, to investigate and observe the differences in emotions between two different groups, such as control and healthy groups in [76,107,135], groups of women and men in [46,97,177], young adults versus older adults in [87,216], or children versus adults in [126,143].

EEG correlates of emotion (signals): Numerous research studies have examined neural correlates of emotion in humans. Frontal EEG asymmetry, event-related desynchronization/synchronization, event-related potentials, and steady-state visually evoked potentials have been found to be associated with emotional states. We found that the majority of the 130 articles used event-related potentials, whereas 48 articles used Frontal EEG asymmetry in their analysis, six articles used event-related desynchronization/synchronization, and four articles used steady-state visually evoked potentials.

Emotion types and numbers: Human feelings and emotions can be quantified for various emotional states. However, only certain types of emotions can be recognized using EEG. Moreover, finding key emotional states to be recognized is mandatory; for example, six emotions were detected in [42,45,106,120,122,192], whereas in [108], a real-time EEG signal to classify happy and unhappy emotions was proposed, and in [113], a fear evaluation system was proposed. In our review, we found that most of the articles aim to detect unpleasant, pleasant, and neutral emotions, such as in [105,217], or positive, negative, and neutral emotions that are based on the valence-arousal dimensional emotion model, as in [159,206].

Computational methods to estimate affective state: Various techniques and approaches have been proposed in the literature in the processing steps to estimate the emotional state from the acquisition signals. Reviewing recent publications and comparing computation methods and results was conducted on a sample of the papers collected. The selection was based on the year 2015–2016 as a timeframe for publication to reflect recent trends and methodologies for emotion detection. The computational methods to extract and classify emotional features from EEG are summarized in

Table 5. Computational methods to extract and classify emotional features from EEG.

Method		Number of Articles	References
Feature extraction	Frequency domain: power spectral density, band power >> using Fourier Transform	29	[26,49,63,74,104,105,107,116,123,149,161,169,176,193,197,200,203,204,207,208,216,218,219,220,221,222,223,224,225]
	Time domain: Activity, mobility and complexity >> using Hjorth Parameters, Fractal dimension >> using Higuchi Method	11	[107,117,200,203,204,206,213,216,220,222,224]
	Wavelet domain: Entropy, Energy >> using Wavelet Transform	7	[186,201,203,213,216,217,226]
	Statistical features: Median, Standard deviation, Kurtosis symmetry, etc.	6	[6,104,117,200,204,226]
Classification	Support Vector Machine (SVM)	24	[49,104,106,107,116,117,157,176,186,190,193,196,197,202,203,204,213,216,218,220,225,226,227,228]
	K-Nearest Neighbor (k-NN)	10	[49,104,107,190,204,207,213,216,218,228]
	Linear Discriminant Analysis (LDA)	4	[26,123,176,227]
	Artificial Neural Network (ANN)	7	[105,176,190,204,216,223,227]

.

It is noteworthy to mention that a single feature extraction technique is not optimal across all of the applications. Besides, existing signals are not enough for high accuracy feature extraction. Several approaches introduce more features in different analysis domains to capture extra information about the state of the brain [107,117,200,203,213,216,224]. Consequently, feature extraction is one of the major challenges in designing BCI systems; it is determined based on the features and on the appropriate transformation. Although the answer to what are the most emotion-relevant EEG features is still under investigation, power features from different frequency bands are still the most popular in the context of emotion recognition. Studies [26,197,218] have shown that power spectral density (PSD) extracted from EEG signals performs well on distinguishing affective states.

Several machine learning algorithms have been used as emotion classifiers, such as support vector machine (SVM), K-nearest neighbors (K-NN), linear discriminant analysis (LDA), random forest, Naïve Bayes (NB) and Artificial Neural Network (ANN). In general, therefore, the choice of which classification algorithm can be used when designing a BCI depends largely on both the type of brain signal being recoded and the type of application that is being controlled. However, SVM based on frequency domain features as power spectral density (PSD) is shown as the most commonly used method.

Recently, Deep learning methods have been applied to the EEG-based emotion recognition filed. In [49], they proposed a deep belief network (DBN) classifier to recognize three categories of emotions (positive, neutral, and negative). Their experimental results show that the DBN models obtain higher accuracy than SVM and K-NN methods. Also, recurrent neural networks have been used in [74,208,229]. In [206], Chai et al. proposed auto-encoder-based deep learning method to investigate emotion recognition on the SEED dataset.

4.2. Classification of Articles by Application Domain and Field

We further classified the 285 articles that we extracted into areas of real-life application. Notably, we classified them into medical and non-medical fields of application. The following criteria were used to classify the articles under the medical field:

• the paper discussed a medical condition (disorder/disease), such as a psychiatric or neurological case;
• the participants of the experiment were patients or it involved two groups: one consisting of healthy people, the other of patients;
• the experiment was conducted in a clinical setting; and/or,
• the paper was directed toward the medical community and suggested a new method for assistance, enhancement, monitoring, or diagnosis using emotion-based recognition.

After evaluating all 285 articles according to the medical classification criteria listed above, we found that 67 articles, or 23.51%, were medical articles. The remaining 218 articles, or 76.5%, were non-medical. Distributions of medical and non-medical articles according to year of publication are presented in Figure 4. We further classified the articles into areas of application within each domain. Figure 5 shows a treemap of the 285 articles, classified according to field and application domain.

4.2.1. Medical Applications

In the medical field, EEG-based emotion detection systems are used for assisting, monitoring, enhancing, or diagnosing patients’ affective states. These medically adapted EEG systems are also used to analyze different types of neurodevelopmental disorders. Some of these neurodevelopmental disorders affect the memory, emotion, learning ability, behavior, and communication of persons suffering from these conditions. The most common cases are Schizophrenia, Autism, Depression, Huntington’s disease (HD), and a myriad of psychiatric and neurological diseases.

Table 6. Medical application classification.

Domain	Description	References
Assessment	These articles assess the performance in terms of health and non-health. The data from these articles can be used in evaluating severity level and monitoring progress and attainment. Moreover, information from these studies is used to develop a scoring system to assess emotionality.	[4,47,72,79,94,106,118,119,120,133,135,145,153,167,212,213,230,231,232,233,234,235]
Assistance	These articles provide insight into technologies/resources for a patient with a disorder/learning problem by means of assistive tools. Assistance occurs after the emotion detection methods have been used to identify skills and limitations of potential users.	[6,33,90,103,125,140,146,165,168,180,221,236,237]
Diagnosis	These articles describe how doctors use EEG in the interpretation of medical conditions. EEG is used by clinicians as a diagnostic tool for patients with psychiatric and neurological disorders. EEG studies produce conclusive results as per the symptoms experienced by the patients. Most of these symptoms are emotional; hence, they are difficult to diagnose using subjective means. The goal of these systems is to objectively detect medical anomalies in a patient’s emotional affective state.	[20,62,66,76,137,183,226,228]
Monitoring	These articles outline the performances of patients who are monitored during emotion processing to understand the deficits in emotion and cognition. The data are used to understand neural activity and correlates of emotions in patients with different conditions/disorders using different stimulus types.	[5,52,54,73,75,81,98,99,107,114,121,122,124,126,156,181,211,238,239,240]
Other	These are review studies that report previous research efforts. They serve as avenues for increasing the awareness of EEG emotional response studies. They also increase the awareness of how EEG may be used in clinical practice to uncover potential neurophysiologic abnormalities.	[17,18,19,21]

shows a classification of medical articles into areas of application, including Assessment (22 articles, 32.8%), Assistance (13 articles, 19.4%), Diagnosis (8 articles, 11.9%), Monitoring (20 articles, 29.9%), and Other (4 articles, 6%).

Generally, neurological studies seek to understand how defects in neurobiological processes result in problems associated with neural functioning. These studies are conducted with the goal of understanding how individual differences in brain structure and function influence affective, cognitive, and behavioral outcomes.

Different approaches have been proposed, and several research groups have developed EEG-based BCI systems that aim to detect these affective states. Examples of medical applications are hereunder identified.

Depression is a mental disorder that is related to a state of sadness and dejection. It affects the emotional and physical state of a person. Using EEG-based emotion recognition as a diagnostic test for depression produces conclusive results. A number of studies are based on the automated classification of normal and depression-related EEG signals. This proposed automatic classification system could serve as a useful diagnostic and monitoring tool for the detection of depression [20,79,133,135,226,228,231].

Persons with schizophrenia exhibit drab facial expressions and rarely show positive emotions. These abnormal personality traits may impact social functioning and communication. Whether these deficits reflect an aberrant sensory anomaly, an inability to retain information in their memory, or a dysfunctional integration of these two functions remains unclear. However, studies have shown that sensory processing and memory functioning may be affected in schizophrenic patients [5,17,18,54,62,66,72,73,90,94,98,180,233,236]. These studies’ experimental protocols were intended to clarify patient deficits in processing emotional faces.

Persons with Parkinson's disease have shown deficits in emotional recognition abilities. Research findings on the reason for this are inconclusive. Nine articles discussed the idea of using EEG-based emotion detection to provide assistance, monitoring, assessment, and diagnosis of Parkinson’s disease in patients [47,106,107,114,118,119,120,121,122].

EEG-based emotion detection systems for cochlear implant patients have been proposed in [156,211,238]. EEG signals are used to estimate the ability of these patients to recognize emotional sounds by using EEG signals and different stimuli. Three studies compared the emotional approaches of two groups of study participants. In one group, which was comprised of children with normal hearing, the participants displayed a withdrawal/approach model, whereas the cochlear implant users did not.

Other medical cases include autism [6,21,75,140,212,213], bipolar disorder [124,145], epilepsy [230,234,235], attention-deficit/hyperactivity disorder [103,165], bulimia nervosa [33,168], borderline personality disorder [167], pervasive developmental disorder [81], and eye movement desensitization and reprocessing [125].

4.2.2. Non-Medical Applications

EEGs have been utilized in numerous non-medical applications. These applications are employed for both healthy and physically challenged individuals. Non-medical fields where EEGs have been applied include entertainment, education, monitoring, and gaming.

Table 7. Non-medical application classification.

Domain	Description		References
Monitoring	These articles explored the effects of different types of stimuli, test emotions, and elicitation methods. They also investigated how different types of stimuli induce specific emotional reactions. They identified efforts in brain lateralization, which aims to define regions of the brain and the functioning of specific behaviors and cognitive skills. They also compared emotional responses between genders and during human developmental stages.		[26,27,28,29,30,31,32,35,43,45,46,48,51,55,56,59,60,64,65,67,68,69,70,74,77,78,80,82,83,84,85,86,87,89,91,93,96,97,113,127,128,129,132,134,138,139,141,142,143,144,147,148,150,155,158,159,161,163,164,166,169,170,171,172,175,177,178,179,182,184,187,192,210,215,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261]
New method	These articles proposed approaches for detecting affective states using single/multi-modality signal processing methods, including feature extraction and selection, machine learning and pattern recognition methods. These proposed systems aim to explore or improve EEG-based emotion recognition systems.		[2,39,41,42,49,50,57,61,63,92,104,108,109,117,131,136,149,152,157,173,174,185,186,189,191,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,217,219,223,224,225,229,262,263,264,265,266,267,268,269,270,271,272,273,274,275]
Entertainment	These articles observed relationships between multimedia data (music/video) and human emotions. For example, they explored the effects of different types of music on subjects of different ages or genders. In gaming research, some articles sought to detect gamers’ affective states to adapt to specific game features, such as the levels of difficulty, punishment, and encouragement. All of these were investigated using EEG-based emotion recognition systems.		[34,37,44,53,58,71,88,100,101,110,111,112,115,116,151,154,190,193,216,220,222,276,277,278,279]
Marketing	These articles sought to understand consumer responses toward marketing stimuli by using imaging techniques and recognition of neurophysiological parameters.		[102,123,218,280]
Education	These articles tracked students’ engagement and learning.		[38,281]
Assistance	These articles explored how assistive technologies or learning resources were provided to individuals. Thereafter, they were used to identify skills, user experience, and limitations of potential users. They were used to improve behavior, cognition, and emotion regulation.		[1,40,95,105,139,194,214,227,282,283]
Others	Other articles explored different aspects, such as	workload;	[176,188,284]
		social interaction and cultural differences;	[9,36,130,160,162]
		review studies that reported previous research efforts.	[3,7,8,10,11,12,13,14,15,16,22,23,24,25]

shows a classification of non-medical articles into areas of application: Monitoring (95 articles, 43.6%), New method (60 articles, 27.5%), Entertainment (25 articles, 11.5%), Marketing (4 articles, 1.8%), Education (2 articles, 0.9%), Assistance (10 articles, 4.6%), and Other (22 articles, 6.4%).

Different approaches have been proposed, and several research groups have developed EEG-based BCI systems that aim to detect affective state. Examples of non-medical applications are hereafter identified.

Recently, a new research area appeared in the marketing field: neuro-marketing. The goal of this new area is to understand consumer responses toward marketing stimuli by using imaging techniques and the recognition of physiological parameters. Because customer feelings in sales areas are strongly influenced by the perception of the surroundings, recognition of emotional responses can reveal true consumer preferences and improve and assist in the (buying) process. Four articles have presented this idea in different applications [102,123,218,280].

Similarly, an EEG-based experimental study [144] is also used to identify the temporal point at which smokers’ responses to health warnings begin to differ. Basically, they aim to determine the effects of graphic pictorial cigarette package health warnings by assessing the selective attentional orientation and measuring emotional processing; they reported that smokers are less sensitive to the emotional content of cigarette health warnings. Therefore, future health warning development should focus on increasing the emotional salience of the pictorial health warning content among smokers.

Three research groups have explored EEG emotion detection systems for emotion (stress) monitoring during self-driving of a powered wheelchair [40,227,285]. These studies aim to investigate the ability to assist and enhance BCI-based wheelchairs by integrating emotion detection while controlling a wheelchair.

The relationship between music genres and human emotions has been investigated in several recent BCI studies. In these studies, brain signals were recorded using an EEG headset while the subject listens to music [44,53,58,100,110,112,115,116,151,154,190,205,216,220,222,235,276,279]. Moreover, the subjects’ emotions were recognized as displayed by EEG signals. These signals were then used to tag multimedia data [71,101,193,277]. These studies also investigated methods for implicit tagging, wherein users’ responses to interactions with the multimedia content are analyzed to generate descriptive tags [71].

Recent BCI research from different disciplines approaches EEG emotion detection and recognition via diverse methods. Some of these methods include feature extraction and selection, machine learning, and pattern recognition methods. These different methods are used to build EEG-based emotion recognition systems. Several research groups (63 articles, 21%) have proposed novel/effective methods for building improved EEG-based emotion recognition systems. Three articles have proposed and described EEG emotion database benchmarks [42,92,101].

Another non-medical application of EEG-based emotion recognition is word processing. Because individual emotional differences impact word processing, differences in interpreting a string of words may elicit different emotional responses. These varying emotional responses are caused by involuntary (implicit) semantic processing, lexical decision tasks (LDTs), and interpretations of perceived positive or negative emotional words [29,31,32,150,160].

Several research groups have explored EEG emotion detection systems as means for monitoring levels of attention and measuring workload [176,188]. One important use is to monitor the level of alertness in security-critical tasks such as driving and surveillance. For example, the Air Force Multi-Attribute Task Battery was used as a realistic, ecologically valid multitask environment, in which a participant's workload could be varied [188].

Recently, EEG-based emotion recognition was proposed as a technique that can be used to support a classification task, such as the EEG-based emotional state clustering task [50], image classification [219], and Odor Pleasantness Classification Using Brain and Peripheral Signals [26].

Other non-medical applications of EEG-based emotion recognition include, lie detection applications, security-critical tasks, and driving. For instance, EEG emotion applications are used in untangling criminal cases in legal proceedings and to tell whether an individual is telling the truth [97,139,274,284].

Two articles discussed how EEG-based emotion detection systems could be used to monitor the alertness of humans when performing security-critical tasks. The researchers proposed a real-time low-level attention detection application that can measure a driver’s degree of attention [23]. Likewise, a driver’s emotional and stress levels can be monitored under different conditions [23,283].

5. Challenges and Future Directions

One challenge with regard to the detection and modeling of emotions in the context of human-computer interaction (HCI), is that it remains complex and requires further exploration. In this context, future research on EEG-based emotion recognition will be explored using the BCI design. From our perspective, Figure 6 provides insights into future research, challenges, and trends in EEG-based emotion recognition.

In general, there are many challenges that are associated with the BCI system. These challenges can be classified as technology related and/or user related. There are differences between the two; however, it is noted that technology-related challenges include, impedance with sensors, system usability, and real-time constraints. When dealing with the device, there are other things to consider, such as the perceived obtrusiveness, information transfer rate, and high error rates. When dealing with user-related challenges, one needs to consider the unfamiliarity of the participating subjects or patients with BCI technologies, discrepancies between ratings, and the duration of setup and preparation. Testing requires assistance from the facilitator in applying the electrodes, which makes the training phase time-consuming.

The BCI design requires multidisciplinary skills from such fields as neuroscience, engineering, computer science, psychology and clinical rehabilitation to achieve the goal of developing an alternative communication and control medium. The number of EEG-based emotion recognition research studies in recent years has been increasing, yet EEG-based emotion recognition is still a new area of research. The effectiveness and efficiency of these algorithms are somewhat limited. Computational methods are being used to estimate emotional state; nevertheless, they can be further improved with technological advancements in order to increase the effectiveness of these algorithms. Some examples of limitations in current algorithms and approaches involve time constraints, accuracy, the number of electrodes, the number of recognized emotions, and benchmark EEG affective databases.

Another limitation is the accuracy and reliability of sensory interfacing and translation algorithms in BCI systems. These factors generally limit the usage of these technologies in clinical settings. Other fields have limitations and challenges as well. For example, engineering challenges focus on the low signal-to-noise ratio in noninvasive electroencephalography (EEG) signals. Moreover, computational challenges include the optimal placement of a reduced number of electrodes and the robustness of BCI algorithms to a smaller set of recording sites.

As we mentioned previously, the challenges currently facing EEG-based BCI systems are two-pronged: technological and user related. We anticipate that some of these challenges will be resolved in the future. For the convenience of description, we have categorized these challenges according to the time that we predict it will take to resolve them: near-term (2–5 years), mid-term (6–9 years), and long-term (10+ years). Although we have categorized these challenges, possible trends and future solutions into three time phases, it is worth noting that some of these anticipated future trends might come earlier, overshoot their time frame, be delayed, or never be achieved.

We anticipate that some of the challenges we have identified may be addressed in 2 to 5 years’ time. Some challenges that may be addressed include familiarity of BCI electronic gadgets to study participants or patients; the proliferation of mobile devices’ mood and emotion apps; increased multidisciplinary cooperation in BCI systems; and, formulating more accurate and precise definition of the emotion processing in human and neurophysiological studies in EEG correlates of emotion as noted by Kim et al. in [16].

In addition to the above-stated trends, we anticipate extensive advancement in body sensors, cameras, and head-mounted devices. Moreover, improving technology sensors is advancing at a rapid rate, as noted by Alotaiby et al., in [13]. We estimate that these will contribute to the development of rudimentary affective computing systems in approximately six to nine years’ time. In ten+ years’ time, we anticipate better machine learning and pattern recognition algorithms, in addition to improved accuracy, speed, and elimination of latency in BCI systems. Advances in computational methods have been noted by researchers to reach a reasonable rate of diffusion and serve as key elements for practical online emotion detection systems, as noted by Kim et al. in [16] and by Jenke et al. in [14].

Moreover, we believe that, there is potential for using hybrid approaches in classification and feature extraction methods. Another aspect is sharing datasets and making them accessible to researchers for further testing. Third, optimizing the number of electrodes is increasingly recognized as important for computation. Also, combining different modalities with EEG. Together, these contribute towards facilitating the work of future researchers in this domain.

6. Conclusions

Recent developments and studies in BCI technologies have facilitated emotion detection and classification. These BCI studies set out to investigate, detect, and recognize a participant’s emotional affective state. These studies also sought the application of research findings in varied contexts, including communication, education, entertainment, and medical settings. However, increasing numbers of BCI researchers are conducting novel experiments, such as considering different responses in various frequency bands, different ways of eliciting emotions, and various models of affective states. These various approaches have been employed in gauging the emotional states of participants/patients from BCI acquisition signals.

This study set out to review published articles on emotion detection, recognition, classification, and current and future trends. Thereafter, it sought to provide insight into how current and future trends will impact researchers and practitioners alike. To achieve this, we mined 29 online databases, selected 160 journals, and extracted 285 articles on emotion recognition systems. Each article was reviewed, analyzed, and categorized according to publication year, research methods, primary contribution, and publication outlet.

Our classification and descriptive review provides quality reference sources for academics, researchers, and practitioners that are working in the field of emotion detection and recognition. This study also contributes to our understanding of various applied concepts of emotion recognition using BCI in different contextual fields. Our results showed an explosive growth of the number of EEG-based emotion detection journal publications. This reflects increased research interest in EEG-based emotion detection as a salient and legitimate research area. Factors such as the proliferation of wireless EEG devices, advanced computational intelligence techniques, and machine learning, spurred this growth. In general, we expect exponential growth in the amount of EEG-based emotion detection research in the near future. As would be anticipated of any new research field, the field of EEG-based emotion detection and recognition is fraught with challenges. Our results show that the challenges raised herein may be resolved in the near future, thereby causing further growth and increased research interest in BCI systems.

As noted in Section 5, EEG-based emotion detection involves multiple dimensions that need to be considered in research methodologies. Researchers would need to take into account the usability of devices, accessibility of datasets, optimization of computational methods, and combining different modalities in the design of their EEG based studies.

Regarding computational methods that can be used in feature extraction and classification phases, it appears from our review that there is no particular feature extraction nor classification technique that emerges as the single best choice for all of the applications. The choice depends on the specific system paradigm and task.

It has been recommended to consider as many algorithms as possible to determine the validity of the proposed process, including preprocessing and synchronization. In most cases, one should compare performance with a range of features and techniques before settling on a choice that yields adequate performance for the given application.