Next Article in Journal
Machine Learning Methodology in a System Applying the Adaptive Strategy for Teaching Human Motions
Next Article in Special Issue
Crowd of Oz: A Crowd-Powered Social Robotics System for Stress Management
Previous Article in Journal
Easing Power Consumption of Wearable Activity Monitoring with Change Point Detection
Previous Article in Special Issue
General Mental Health Is Associated with Gait Asymmetry
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sensors
Volume 20
Issue 1
10.3390/s20010313
sensors-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Brain and Body Emotional Responses: Multimodal Approximation for Valence Classification

by
Jennifer Sorinas
1,2,*,
Jose Manuel Ferrández
2 and
Eduardo Fernandez
1,*
1
The Institute of Bioengineering, University Miguel Hernandez, 03202 Elche, Spain
2
Department of Electronics and Computer Technology, University of Cartagena, 30202 Cartagena, Spain
*
Authors to whom correspondence should be addressed.
Sensors 2020, 20(1), 313; https://doi.org/10.3390/s20010313
Submission received: 28 November 2019 / Revised: 2 January 2020 / Accepted: 3 January 2020 / Published: 6 January 2020
(This article belongs to the Special Issue Sensors for Affective Computing and Sentiment Analysis)
Browse Figures
Versions Notes

Abstract

:
In order to develop more precise and functional affective applications, it is necessary to achieve a balance between the psychology and the engineering applied to emotions. Signals from the central and peripheral nervous systems have been used for emotion recognition purposes, however, their operation and the relationship between them remains unknown. In this context, in the present work, we have tried to approach the study of the psychobiology of both systems in order to generate a computational model for the recognition of emotions in the dimension of valence. To this end, the electroencephalography (EEG) signal, electrocardiography (ECG) signal and skin temperature of 24 subjects have been studied. Each methodology has been evaluated individually, finding characteristic patterns of positive and negative emotions in each of them. After feature selection of each methodology, the results of the classification showed that, although the classification of emotions is possible at both central and peripheral levels, the multimodal approach did not improve the results obtained through the EEG alone. In addition, differences have been observed between cerebral and peripheral responses in the processing of emotions by separating the sample by sex; though, the differences between men and women were only notable at the peripheral nervous system level.

1. Introduction

Emotions are understood as a complex set of neural and hormonal interactions that can give rise to affective experiences (bodily sensations); generate cognitive processes (feelings, the conscious emotions); imply physiological adjustments to adapt to them; and lead to adaptive behaviors and/or decision making [1]. Emotions are an important evolutionary factor that allow survival and breeding through adaptation to the environment. However, the mechanisms of emotional processes and the modeling of human emotions are still fairly unknown. Many efforts have been made to unmask the psychobiology of emotions, since a century and a half ago Darwin proposed the first theory that tried to explain its origin [2]. Still, this is a complicated task by the fact that, even today, there is no consensus regarding the functioning, structure and classification of emotions. One of the most spread theories, Dimensional model of emotions [3,4], sustains that emotions can be explained mainly by two dimensions, valence (pleasure/displeasure) and arousal (calm/excited). Depending on the level of activation and polarity of this biphasic dimensions, motivational systems of approach (survival and pleasure) and withdrawal (fight or flight responses) are activated with the intention of adapting behavior to an emotional stimulus [5]. This theory has increased its popularity due to the affective computing research and applications [6]. While affective neuro-science’s main objective has focused on the study of the neurobiology of emotions, the affective computing branch has been much more pragmatic, leaving the biology behind and concentrating on its recognition and classification. Nevertheless, neglecting the psychological theories of emotions has filled the affective computing studies with assumptions that undermine its own credibility and effectiveness. Therefore, for the development of functional affective interfaces, it is necessary to contextualize engineering goals based on psychological principles [7].
In order to achieve a successful adaptation to the environment, emotions are integrated in the central nervous system (CNS)—what we will call the brain response; and in the autonomic nervous system (ANS)—the bodily response; leading to goal-directed behaviors. The fact that emotions have specific brain and body responses, has motivated the study of the different signals that intervene both in their processing and in their response in order to try to find patterns that allow them to be identified.
At CNS level, the functional Magnetic Resonance Imaging (fMRI) technique, due to its spatial resolution, is the most used—for the study of the neural substrates that underlie emotions; however, the low temporal resolution, high cost and the impossibility of using it in normal life environments and situations, separate it from the picture of affective computing applications [8]. On the other hand, the temporal resolution, usability, low-cost and wireless nature of the EEG, make it the suitable technique for emotion recognition applications [9]. In order to characterize the neural structures implied in the emotional process and describe the way in which they work and interact, specific features of the EEG frequency spectrum, electrode location and temporal window have been evaluated. However, the lack of theoretical consensus on the emotional process complicates this task due to the high variability between studies and, therefore, the difficulty of comparing results. Nevertheless, different EEG asymmetry patterns across hemispheres have been observed, left hemisphere activation over frontal and prefrontal regions is linked with positive affect experience and therefore to the approach motivational system; and contrary the right hemisphere presents higher activity when processing negative affective or withdrawal stimuli [10].
Regarding the emotional body response, the main structure that regulates and controls the vegetative auto-regulatory processes in order to meet behavioral demands is the ANS [11], which is closely linked with the CNS emotional part [12]; and it is believed to be involved in the generation of the physiological arousal of the emotional episode. It is thought to be related with the arousal dimension, but not with the valence scale [13,14]. The ANS has two branches, the sympathetic nervous system (SNS), which becomes dominant, increasing the physiological arousal, when either psychological or physical stress is taking place and, the parasympathetic nervous system (PNS), which dominates during periods of rest or safety, maintaining a low degree of physiological arousal. These systems are related with the approach and withdrawal motivational systems since they are responsible for the body’s response [15]; however, the degree and functionality of the process is still diffuse. The traditional view of the ANS emotional response stands for specific patterns of activation regarding emotional component of the stimulus [16]. Nevertheless, more evidences supports the undifferentiated arousal theory [17], suggesting that all emotions present the same or at least similar ANS activation pattern when high arousal stimuli take place. However, this theory does not explain the different responses showed with stimuli presenting the same value in the arousal scale but opposite valence scores. Several techniques have been used for emotion recognition based on bodily responses, such as heart rate, galvanic skin response, respiration, skin temperature and behavioral measures [7]. Interest in behavioral measures such as facial expressions, voice, and body language emerged because of similarities between cultures found in emotional expression [18], however these methods have a high cost and require long training times and modelling of the subject [19]; therefore, physiological measures are preferred. The SNS mediated responses for negative emotions and PNS for positives have been described by several authors and measures as heart rate [20,21], skin temperature [22,23] and galvanic skin response [24]. However, the same problem regarding the comparison of the EEG studies apply for the ANS emotional patterns.
Each modality or psychophysiological signal used for emotion recognition has its own pros and cons and an extensive literature behind. Calvo et al. [7] have pro-posed a set of factors to evaluate the effectivity of a modality to serve as a way for affective computing interfaces. First, the validity of the signal to represent the emotional process. Brain signals are preferred to bodily signals since the latter can be consciously modified and are more unspecific. Second, the reliability of the signal in real-life applications. In general, brain signals have obtained better classification results at the valence scale, whereas bodily signal did for the arousal [25,26]; suggesting that both types of signals measure different, but complementary aspects of the emotional state and therefore bringing up the idea of combining the modalities for better performances. Finally, the time resolution, cost and user invasiveness also have to be taken into account. Although EEG technology is now advancing in the development of more accessible and user-friendly devices, its complexity and discomfort is greater than that needed to measure bodily signals, which can be acquired at the time through a simple bracelet [27].
In previous studies [28,29], we have evaluated some of the technical parameters necessary for emotion recognition based on the EEG signal, but without delving deeper into its biological implications. In the present work, we wanted to study the response of the ANS and its contribution to emotion recognition on the valence scale. Moreover, we would like to test the idea of the synergy between brain and body responses for emotion recognition applications. For this end, we have recorded the EEG, ECG and skin temperature signals of 24 subjects during stimulation using videos with positive and negative emotional content. Each biological modality has been studied individually, in a subject dependent (SD) and independent (SI) way, to finally perform a multimodal classification and correlation analysis in order to see the relationships between body and brain signals, and the emotional subjective ratings provided by the participants.

2. Materials and Methods

2.1. Experimental Procedure and Data Analysis

A total of 24 subjects (mean age: 23.12; range: 19–37; sixteen men and eight women) were emotionally stimulated while EEG brain activity; cardiac activity and skin temperature were recorded. The stimuli consisted of 14 videos, 7 of them labeled as positive and the other 7 as negatives depending on the emotional content. The audiovisual stimuli were rated by the subjects in both valence and arousal scales, obtaining average valence scores of 7.51 (s.d. 1.6) and 2.91 (s.d. 0.98) for the positive and negative categories, respectively. In the arousal scale, positive stimuli were rated with a mean arousal of 3.76 (s.d. 1.62) and 5.47 (s.d. 1.35) for the negatives. The clips, with durations between 43 and 78 s, were selected from the internet, edited with the software Camtasia Studio 8 and presented randomly and counterbalanced to the subjects, in alternation with a 30 s black screen that also served as the initial baseline. The subjects were right-handed, had no personal history of psychiatric or neurological disorders, normal or corrected vision and audition, and were not under medication during the development of the study. They also provided their written consent, supervised by the Ethics Committee of the University Miguel Hernandez.

2.1.1. EEG

64 Ag-AgCl electrode cap was used for the electrical brain recording according to the International 10/10 System [30]. Data were amplified and registered through a NeuroScan SynAmps EEG amplifier (Compumedics, Charlotte, NC, USA), keeping the impedance for every electrode under 25 kΩ [31], and with a sampling rate of 1000 Hz. Data were filtered through a high-pass and low-pass filters, 0.5 Hz and 45 Hz respectively, and electrodes were re-referenced to a Common Average Reference (CAR), by Curry 7 software (Compumedics, Charlotte, NC, USA). For artifact rejection corresponding to electrical non-brain origin signals as eye-blinking, heart rate and muscle activity, the Matlab (The MathWorks Inc., Natick, MA, USA) toolbox EEGLAB [32] was used; artifacts were selected by means of visual inspection on the basis of Independent Component Analysis (ICA) [33]. Detailed data collection and pre-processing steps could be found in previous work [28].
In our previous work [29], a set of 20 features were selected as the most informative in terms of positive and negative emotion classification; however, few could be said about the theoretical interpretation of this result. Therefore, in the present work, the study of cerebral asymmetries was performed. From the 20 frequency-location variables corresponding to Alpha, Beta1, Beta2 and Gamma frequencies at both PreFrontal (PF) left and right locations; Beta 1 and Gamma at Frontal midline (Fm); Alpha and Beta1 at Central midline (Cm); Beta1 and Beta2 at Parieto-Occipital midline (POm); Gamma at both Central (C) left and right hemispheres; Gamma at Parietal (P) right; and Beta1, Beta2 and Gamma at Occipital (O) right; only those which localize in one or both of the hemispheres were used to study cerebral asymmetries through the classical method [34]. The asymmetry index (AI) is calculated for the spectral power of a specific bandwidth on homologous hemispheric regions, according to the Formula (1).
A I = r i g h t − l e f t r i g h t + l e f t
Pairs corresponding to midline regions were not included in the analysis; therefore, 9 combinations of frequencies and locations were evaluated. The AI was calculated for every frequency-location feature in each subject. However, in order to look for significant differences between hemispheres and emotional categories, data from all subjects were assessed together. Mann-Whitney test was used for the statistical analysis [35] between the AI of the positive and negative emotions. The same test was applied at the power spectral values themselves for the comparison between homologous hemispheric regions with either positive and negative categories. The entire EEG asymmetry analysis was performed in the Matlab environment.

2.1.2. ECG

Two electrodes were placed, one on the right side of the sternum and the other on the intercostal space between fifth and sixth rib, using lead II configuration in order to record ECG activity. The recording sampling rate was 1000 Hz and, as for the EEG signal, was performed with Curry 7 technology (Compumedics, Charlotte, NC, USA). After recording, ECG data were resampled to 256 Hz for further analysis. Data from subjects 1, 9, 17, 19 and 22 were not properly recorder, therefore these subjects were excluded from the analysis. Freely available stand-alone Artiifact 2.09 software [36] was used to accomplish the heart rate variability (HRV) analysis. HRV analysis was performed for each of the 24 subjects individually. A high-pass filter with the cut-off frequency at 10 Hz was applied to ECG data in order to extract the inter-beat interval data from every trial. Artifact detection and elimination was carried out through cubic spline interpolation. Finally, we performed time (Mean RR, Median RR, SDNN, RMSSD, NN50, pNN50) and frequency (VLF, LF, HF, LF/HF; frequency bands 0–0.04, 0.04–0.15, 0.15–0.4, respectively. Values for the different band-widths were obtained as a percentage, absolute values and normalized units only for the LF and HF measures) domain HRV measures. On the other hand, Matlab environment was used to assess non-lineal methods as the Poincare plot (SD1, SD2, SD1/SD2) [37,38,39,40]. See Table 1 for a detailed explanation of each HRV measure. Geometrical methods have not been included in the analysis because a minimum time period of 20 minutes’ recording is necessary to ensure feasible results [41]. As expected on healthy young people, no abnormalities on the ECG signal as tachycardia, arrhythmias or bundle branch block were found.
Each of the 19 variables evaluated was z-scored and tested for normal distribution with the One-sample Kolmogorov-Smirnov test, resulting in non-normal data distribution. Mann-Whitney test was carried out to test if positive and negative data came from the same distribution at both SD and SI approximations. We have also analyzed the data separating it by gender. Finally, simulated annealing optimization method was used to select the most informative features with K-nearest neighbors (KNN) with 5 neighbors and quadratic discriminant analysis (QDA) classifiers, at the SI approach.

2.1.3. Skin Temperature

Skin temperature signal was recorded at the right wrist of the volunteers through the ActTrust bracelet (Condor Instruments Ltda., São Paulo, Brazil). Due to problems during the experimentation, the data of subjects 1 and 15 could not be included in the analysis, therefore 22 subjects from the total of 24 enter into the analysis. Data were segmented in trials of 28 s lengths, corresponding to every video clip. Values more than three scaled median absolute deviations from the median were consider as outliers and replaced by the mean value. One-sample Kolmogorov-Smirnov test was used for evaluating data normal distribution, resulting as non-normal distributed data. In order to look for differences between the two categories of emotions and the neutral state, the statistical Mann-Whitney test was performed [35], in the SD and SI approximations. Gender differences were also assessed with the same statistical test.

2.1.4. Multimodal Approximation

In previous work [29], we proposed an EEG-based model for the classification of positive and negative emotions. In the present work we have evaluated the relevance that peripheral physiological signals could have in classification performance associated to our EEG-based model. In order to obtain same length segments of each type of signal, temperature data was down-sampled and ECG data was up-sampled to match EEG data. Moreover, a total of 6 subjects (numbers 1, 9, 15, 17, 19 and 22) were eliminated from the analysis due to they were missing at least one of the data inputs. Final classification stage was performed with the 20 variables coming from the EEG analysis, 1 variable corresponding to skin temperature, and the significant variable resulting from the ECG analysis. KNN with 5 neighbors, and QDA classifiers were selected based on previous work, and applied in a SD and SI approximations. As there is evidence of gender differences in bodily signals, we also performed gender-based emotional classification (11 men, 6 women). Experimental procedure could be overviewed on Figure 1.
Finally, in order to assess the relationship between the emotional subjective ratings of the stimuli and the 22 different features that conform the model (20 EEG, 1 T, 1 ECG), we performed both a SD and a SI correlation analysis.

3. Results

3.1. ECG

Looking at the statistical differences between positive and negative emotions at the studied HRV variables, no significant differences were found at the SI level. Contrary, significant differences appeared at the SD level for some of the variables and subjects (Figure 2); however, no clear general patterns were found that would allow conclusions to be drawn about the population. As HRV gender differences are demonstrated in young people, we have assessed differences at the SI level for women and men. Significant differences were found between positive and negative emotions at the SD2 variable (p-value = 0.0498) for women; and at the NN50 variable (p-value = 0.0382) for men. SD2 index showed higher values during the processing of negative emotional stimuli, suggesting greater long term variability during the experience of negative emotions for women. On the contrary, higher values of NN50 variable were presented during the processing of positive emotions in comparison with negative emotions, i.e., there is greater variability between contiguous beats in the positive condition for men.
Statistical analyses have revealed that making common inferences for the population based on differences in HRV measures for positive and negative emotions is not an easy task. When classifying positive and negative emotions based on all the HRV measures, performances below the value of chance were obtained for both QDA and KNN classifiers, with f1 scores equal to 0.355 ± 0.161; and 0.497 ± 0.153, respectively. After simulated annealing optimization, performance improves up to 0.57 ± 0.17 for the QDA classifier, and 0.616 ± 0.125 for KNN; through using 5 (Mean RR, RMSSD, NN50, pNN50 and VLF) and 6 (Median RR, SDNN, NN50, LF [absolute values], HF [absolute values] and SD2) HRV variables, respectively, as inputs for the sorter. Although after feature selection, classification performance improved, the selected features were not shared by the classifiers, with the exception of the NN50 index. In this context, selecting those variables that best allow discrimination between positive and negative emotions seems complicated, since the results obtained from classification, although favorable, left too much error margin. However, since our objective was to perform a multimodal classification to evaluate the contribution of the central and peripheral nervous systems in the recognition of emotions in the valence dimension, we have selected the NN50 variable to form part of the final classification stage, since it was the only common variable in both optimization groups, although it has only shown significant differences in the men group.

3.2. Skin Temperature

Results comparing positive and negative recordings resulted in significant differences (p-value under 0.05) for all subjects in the SD approximation. Moreover, skin temperature data corresponding either to positive and negative emotional conditions, was also significantly different than the baseline period; except for subject 14 at both comparisons positive vs. baseline and negative vs. baseline. At the SI approximation, there were also significant differences between the means of positive vs. negative (p-value = 1.56 × 10−31), positive vs. baseline (p-value = 3.79 × 10−6) and negative vs. baseline (p-value = 3.79 × 10−6) emotional groups. Mean values corresponding to positive, negative and baseline skin temperature were 28.767 °C (s.d. 1.515), 28.847 °C (s.d. 1.486), and 27.087 °C (s.d. 1.552), respectively. Temperature values for negative emotions were slightly higher than for the positive, reflecting an opposite pattern of the expected vasoconstrictor sympathetic regulation in front of adverse stimuli [23].
Looking at gender differences, skin temperature values for positive emotions showed no significant differences between women and men samples (p-value = 0.1502), but negative emotions did (p-value = 0.0017) (Figure 3). If we disaggregate the population sample by gender, significant differences were presented in the women positive vs negative emotions (p-value = 1.7237 × 10−42). Contrary, no differences were found for men (p-value = 0.7738).

3.3. EEG Asymmetries

No differences were found between genders when classifying emotions based on EEG data (Table 2), therefore asymmetry studies were performed with the whole population. In order to have a better overview of the positive and negative emotional brain processes underlying the proposed model, we have studied the interhemispheric asymmetries of the 20 frequency-location pairs. This analysis was done with either SD and SI approaches. With respect to the SD approximation, none of the AI measures resulted as having significant differences between the positive and negative conditions in all subjects, this fact demonstrates once more the high inter-subject brain variability present in the emotional processing. Therefore, in this case, the evaluation of the EEG asymmetries between interhemispheric frequency-location pairs would be more fruitful with a SI approach, trying to focus only on the commonalities between subjects instead of the differences.
When comparing the AI of positive and negative emotional conditions, we have found differences in 5 of the 9 interhemispheric studied variables: PF-alpha (lateralized towards the left hemisphere at positive emotions and toward right on negative), PF-beta2 (presented the same lateralization pattern as PF-alpha), P-gamma (at both conditions, lateralization occurs towards the left hemisphere, however for positive emotions, power spectral density is higher than for negatives), O-beta1 (positive emotions lateralized towards the right hemisphere and negatives towards the left) and O-gamma (same lateralization pattern as the presented on the PF region). Conversely, if we look at the interhemispheric differences of each emotional category, we found significant differences in positive emotions at the PF-beta1 pair, lateralizing towards the left hemisphere; the PF-beta2, lateralizing towards the left too; and O-beta2 pair, lateralizing this time towards de right hemisphere. For the negative emotions, only O-beta1 and O-beta2 pairs presented significant differences, both lateralizing towards the left hemisphere. Figure 4 represents the AI between the positive and negative emotional categories at the frequency-location features of study.

3.4. Multimodal Approximation

After the assessment of the differences between positive and negative emotional categories at the ECG and skin temperature signals, we wanted to evaluate their contribution to the classification of emotions, assessing if it is better to take into account the central nervous system or autonomic nervous system by themselves or if a synergy between brain and body exists. To this end, we have performed a SD and SI classification with the KNN and QDA classifiers using different inputs. Figure 5 shows the f1 scores for the SD classification, and Figure 6 the SI results.
Regarding SD classification, T + ECG condition presented the worst performance at both classifiers (0.602 s.d. 0.123 QDA; 0.943 s.d. 0.03 KNN). This result suggested that, although autonomic nervous system data per se could be used to differentiate positive and negative emotions, it is less effective than using central nervous system data alone (0.98 s.d. 0.015 QDA; 0.988 s.d. 0.014 KNN) or in combination with it (0.97 s.d. 0.038 QDA; 0.989 s.d. 0.014 KNN). All other classifications showed similar results on both classifiers. As the performance obtained with the EEG data alone, was almost perfect, it is difficult to say if the addition of the autonomic nervous system data improves the recognition of emotions, but at least we could conclude that it does not worsen performance.
In the case of the SI classification, we continued having the same inter-subject variability problem observed in our previous work [29]. So that, possible inferences from the results are not conclusive.
Although EEG data did not show differences between genders, peripheral physiological signals did, therefore SI classification regarding gender was performed with all variables. The results obtained for the QDA classifier did not improve the ones obtained for the whole population (0.523 s.d. 0.076), the f1 scores were 0.492 (s.d. 0.156) for women and 0.51 (s.d. 0.097) for men. Contrary, classification performance improved when segregating by gender with the KNN classifier which reached f1 scores of 0.529 (s.d. 0.101) for the entire population, 0.585 (s.d. 0.056) for women and 0.533 (s.d. 0.109) for men. Although mean classification performance improved, the differences between groups were not significant.
Once we have evaluated the performance of the model taking into account brain and body features together and independently, we wanted to study the correlation between those features and the emotional subjective rating of the stimuli. Figure 7 represents the correlation coefficients of the 22 variables at every subject in an SD analysis and for the whole sample in the SI analysis. In the SD analysis, no clear pattern among subjects was found suggesting positively or negatively correlated specific features with the emotional valence scale. Contrary, each subject presents its own brain and body response in front of subjectively rated positive and negative stimuli. We found the three types of possible responses occurring in all features, (a) increased activity with positive stimuli, (b) increased activity with negative stimuli and (c) same activation with both positive and negative emotional stimuli; suggesting that although all the studied features have relevance in the processing of emotions in the valence scale, their response is not constrained by the polarity of the scale. Moreover, we could also infer in three types of subjects, those who, in general, respond more actively to positive stimuli (e.g., subjects 5, 11, 14, 15, 16), to negative stimuli (e.g., subjects 2, 3, 4, 7, 18), and subjects whose features respond in the same way in front of both positive and negative stimuli (e.g., subjects 1, 13, 17). On the other hand, when performing the correlation analysis with the SI approach, the variability observed in the SD results was not present. Although some variables presented slight polarizations, the result reflects that all variables are activated in the same way in both conditions of emotional valence, however, this result could be explained by the high variability presented between subjects. This result could be also reflecting that the process of consciously evaluating the emotional stimuli is not the same as the processing of the emotion.

4. Discussion

Understanding the psychophysiology of emotional processes, i.e., the relationship between body and mind, is key to the design of effective and reliable affective brain computer interface applications. Knowing which are and how the activation patterns of the neuronal substrates are involved in the processing of emotions would allow to design more precise computational models and reduce the preparation and training times of the subjects. At the same time, being able to distinguish the emotional peripheral physiological responses and understand the performance of the ANS mechanisms responsible for them would allow a more complete emotional approach and the possibility of developing simpler and more accessible systems.
The belief that emotions are encoded in subcortical and limbic structures, whereas, cognition is encoded in the cortical level has been dismissed, as new evidence, coming from affective neuroscience studies, has supported the statement that emotion and cognition display or overlap along the same cortical nets [46]. In the meta-analysis conducted by Kober et al. [47], they tried to identify patterns of co-activation of brain regions and its functional organization in emotional neuroimaging studies without labeling the underlying emotions, i.e., without semantically defining the emotional category, thus overcoming the problem of lack of consensus on emotional theory. They defined six functional groups; lateral occipital or visual association group, medial posterior group, cognitive/motor group, lateral para-limbic group, medial pre-frontal cortex group and the core limbic group; of which the prefrontal, occipital and central-motor cortical lobes should be highlighted. Most of these functional groups were found as relevant regions in our emotional model, thus relating the emotional process with a whole brain network, more than specific isolated areas. Moreover, another important fact revealed by the meta-analysis of Kober et al. was that all cortical structures involved in emotional processing showed co-activation with subcortical structures as the limbic system and the brainstem. The EEG only allows us to assess cortical brain activity, and therefore, it is important to note that we are trying to classify emotions by missing an important part of the puzzle. It is thus interesting and necessary to know the body-mind relationships to have a more complete vision of the process and therefore, define emotions more accurately.
Based on our results, in the case of the frequency domain aspect of emotions, at positive ones, Alpha and both Beta frequencies seemed relevant at the left prefrontal hemisphere; and both Beta1 and Beta2 bandwidths increase its activation at the occipital right hemisphere. On the other hand, negative emotions presented a lateralization pattern towards the right prefrontal cortex and highlight the presence of Beta1 and Beta2 frequencies towards the left hemisphere over occipital regions. These results seem to point to a reverted lateralization pattern of frontal and posterior cortex when processing emotions. Our results agree with the frontal EEG asymmetry theory, described by Davidson et al. [5,10], reflecting the activation of motivational systems of approach and withdrawal, also verified by other authors [48,49,50,51]. In essence, we can conclude that there are interhemispheric differences in the processing of emotions; that this lateralization is also different depending on the emotional category; and that the processing of emotions not only falls on the prefrontal cortex, but rather there seems to be a neural network that expands along the entire cortex [52], and at all spectrum, excluding the low frequencies of the EEG [29,52,53]. The similarity on the scalp distribution of the different frequencies involved in the processing of positive and negative emotions suggests that, at least at the cortical level, there are no separate neural pathways for processing positive and negative emotions, but there is a network of cortical structures in charge of processing the valence, whose activity varies depending on the polarity of the emotion. Yet, a more in depth study of the relationships between regions and emotional conditions is necessary in order to draw meaningful conclusions about the brain emotional net.
Peripheral psychophysiological reactions constitute an important source of emotional information, therefore, researchers have focused on the different ANS measures. However, if the PNS and SNS are linked with the positive/approach and negative/withdrawal responses [54], respectively, is more diffuse. As for HRV measures, both our results and those of other authors, showed differences in their response to either emotional dimensions [20,55] and discrete emotions [21,56,57]. These differences are present in all levels of the HRV analysis, the time domain, the frequency domain and the Poincare plot. Nevertheless, and as it is customary in the study of emotions, there is no consensus as to which are the HRV variables most representative of the emotional state. In our case, no significant differences were found in any variable with the SI approach, and although there were differences at SD level, they did not show clear patterns across individuals. Moreover, classification results achieved using all HRV measures to differentiate positive and negative emotions were not different than chance; and although after feature selection, precisions of around 60% were reached, these values are far from the percentages achieved by Guo et al. [55] or Goshvarpour et al. [20], of 71.4% and 100%, respectively. Differences between our study and the mentioned studies, could be explained by the different methodologies used. Guo et al. applied principal component analysis in order to select the features used for the classification; therefore, not using the “real” HRV measures. On the other hand, Goshvarpour et al. used a neural network method, that although obtained better performance, it doesn’t allow inferences to be made at the biological level as the interaction of the variables with each other and their weight in the classification is not known. Nevertheless, when separating the sample by gender, significant differences were found between positive and negative emotions in the Poincare variable SD2 in women and in the time variable NN50 in men, corroborating the existence of gender differences in young subjects [58,59]. The results suggest that in men, the differences are more evident in the short term, responding with greater intensity to positive stimuli; and on the contrary, women respond with greater intensity to negative stimuli, although the difference in this case is observable in the longer term. This may indicate a greater readiness of men for immediate response and greater adaptability to emotional stimulation in women. The variables found as informative, SD2 and NN50, are both regulated by both components of the ANS, so conjectures about the involvement or role that each division plays in the emotional response are not possible.
On the other hand, it has been proven that skin temperature measure could be used for positive and negative emotion discrimination [22,23]. The accepted explanation of the role of the skin temperature in the emotional process point out to vasoconstriction responses in order to mobilize blood into the muscular system to allow reaction to an aversive stimulus. Therefore, it seems that the dichotomy between the activation of the approach and withdrawal systems apply at the skin temperature level, however, both vasoconstriction and cooling and vasodilation and warming responses, are mediated by the SNS [23], pointing to the arousal scale. Therefore, although there are different patterns of response when it comes to processing positive and negative emotions, the functional organization of the activity of the ANS components remains unclear [14]. Nevertheless, our results showed an opposite pattern of the commonly accepted response [60], obtaining higher temperatures for negative than for positive emotions. Regarding gender, although there are differences in the thermoregulation of women and men [61,62], our results indicate that there are no gender differences at the skin blood volume regulation response when processing positive emotions, but, differences exist while negative stimuli occur; suggesting that women react more intensely to negative emotions than men.
Although the responses of the CNS and ANS systems to emotional stimuli and the relationships that exist between them are not known exactly; our results suggest that in both systems, independently, it is able to differentiate between positive and negative emotions. Koelstra et al. [26] used the detection of facial expressions and the EEG signal to classify emotions on the valence and arousal scales, demonstrating that classification performance improved when the two signals were combined, reaching percentages of 67.1% and 71.5%, respectively. Torres et al. [25] evaluated the combination of several biosignals for the detection of emotions on the valence and arousal scales. In the arousal scale, the best classification, 75%, was obtained after the combination of the EEG with physiological signals (heart rate, galnavic skin response (GSR), respiration and skin temperature); however, on the valence scale, the results of the combination of modalities did not improve the percentage achieved by the EEG alone, 58.75%. Using other theories of emotions, Verma and Tiwary [63] showed the best accuracy obtained in the literature when applying the multimodal approach (CNS: EEG; PNS: GSR, respiration, blood volume pressure and skin temperature) for the classification of discrete emotions, with an average value of 81.45% with a support vector machine. On the other hand, using the same database as Verma and Tiwary, but analyzing the emotional valence dimension, Chen et al. [64] obtained an accuracy of 83.98%. Our results improve classification performances obtained by other authors and conclude that the CNS per se (0.988 f1-score classification result for the KNN classifier) is most informative than the ANS data (HRV + skin temperature, 0.943 f1 score) in order to classify emotions regarding the valence dimension; and that the combination of the modalities (0.989 f1 score) does not significantly improve the results reached by the EEG alone. Authors like Jatupaiboon et al. [65] and Torres-Valencia et al. [66], also concluded, through other methodologies and emotional induction methods but addressing the effectiveness of multimodal emotional approach, that the EEG is the signal with the best capacity of emotional discrimination following the dimensional model of emotions. Therefore, based in our results we can discard the undifferentiated arousal theory that supports that the emotional response in the valence dimension is only reflected at the brain level and not at the peripheral/body level. Moreover, regarding the traditional idea that links the motivational systems of approach and withdrawal with the PNS and SNS systems, respectively, we cannot conclude for or against. Nevertheless, our results make even more evident the incompatibility of having a population-trained model that can be used for particular individuals, due to the high variability observed among subjects. Therefore, in order to make an accurate classification of positive and negative emotions it is necessary to train the computational model for each subject. However, it is possible to specify the most relevant features for the classification of emotions in the dimension of valence at the population level; thus decreasing the dimensionality of the model, the complexity of the system and the temporal cost of classification.
Regarding the gender factor in the classification of emotions, we found differences in the peripheral nervous system response, but not at the CNS. This suggests that emotions are processed in the same way for men and women at the brain level, but the body response is different [67]. However, one of the main drawbacks of our study is that women and men samples were not balanced, being considerably less number of women than men, and moreover, although our sample of 24 subjects is more than acceptable for this kind of studies, when splitting it into gender, the sample size is not representative of the population for SI analysis regarding the minimum size of 15 subjects stablished for proper classifications [68]. Therefore, although our results are encouraging, it would be necessary to enlarge the sample size in order to obtain more reliable results in terms of gender differences.
At the peripheral level, it seems that there were differences between the responses to opposing emotional valence stimuli. However, they did not seem to follow the expected pattern of ‘fight or flight’ or ‘calm or safety’ associated with the motivational systems of approach and withdrawal, that are believed to act at the level of the prefrontal EEG asymmetries. The components of the ANS are not activated in an ‘all-or-none’ fashion, rather each tissue is innervated differently by the sympathetic and parasympathetic pathways, which act independently of each other [14]. It is therefore difficult to attribute approaching or rejecting responses to specific components of the ANS. At this point, it is worth asking if motivational systems represent the same as the dimension of affective valence or if, on the contrary, they are different processes that do not always go hand in hand [69]. Conversely, it is probable that the arousal is influencing the physiological response, since although there were no significant differences in the arousal rating in the population, polarity existed in specific individuals.

5. Conclusions

We have assessed the activation states of the CNS, through EEG data, and ANS, through HRV and skin temperature measures, at positive and negative categories of the dimensional valence of emotion. Population differences were found at the frequency domain of electrical cortical signals showing a lateralization pattern towards the left hemisphere for the positive emotions and towards the right hemisphere for negative at anterior regions, and the inverse pattern at posterior regions. Peripheral bodily differences were also found at the skin temperature response, suggesting that valence dichotomy is also present at the ANS level. Moreover, gender differences presented at both ANS measures but not at the CNS suggest distinct mechanisms at the central and peripheral nervous systems and different gender predisposition. However, the multimodal classification approach did not seem to benefit emotion recognition in comparison with the existing EEG computational models. Our results bring more clarity to the debate between the theory-psychology and practice-engineering of emotions; but, more efforts are needed to finish solving the riddle of the psychobiology of emotional processes.

Author Contributions

J.S. developed the theoretical formalism, performed the experimentation and the analytic calculations, and wrote the manuscript. J.M.F. and E.F. encourage the investigation and supervised the project. This manuscript has been released as a Pre-Print at bioRxiv: https://doi.org/10.1101/638239. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported in part by a grant from the Ministry of Education of Spain (FPU grant AP2013/01842), Grant RTI2018-098969-B-100 from the Spanish Ministerio de Ciencia Innovación y Universidades and PROMETEO/2019/119 from the Generalitat Valenciana.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Kleinginna, P.R.; Kleinginna, A.M. A categorized list of motivation definitions, with a suggestion for a consensual definition. Motiv. Emot. 1981, 5, 263–291. [Google Scholar] [CrossRef]
  2. Darwin, C.; Prodger, P. The Expression of the Emotions in Man and Animals; Oxford University Press: Oxford, UK, 1988. [Google Scholar]
  3. Russell, J. A circumplex model of affect. J. Personal. Soc. Psychol. 1980, 39, 1161. [Google Scholar] [CrossRef]
  4. Wundt, W. Lectures on Human and Animal Psychology; Swan Sonnenschein & Co.: London, UK, 1894. [Google Scholar]
  5. Davidson, R.J.; Ekman, P.; Saron, C.D.; Senulis, J.A.; Friesen, W.V. Approach-withdrawal and cerebral asymmetry: Emotional expression and brain physiology: I. J. Personal. Soc. Psychol. 1990, 58, 330–341. [Google Scholar] [CrossRef]
  6. Picard, R.W. Affective Computing; The MIT Press: Cambridge, MA, USA, 1997. [Google Scholar]
  7. Calvo, R.A.; Member, S.; Mello, S.D.; Society, I.C. Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications. IEEE Trans. Affect. Comput. 2010, 1, 18–37. [Google Scholar] [CrossRef]
  8. Ruiz, S.; Buyukturkoglu, K.; Rana, M.; Birbaumer, N.; Sitaram, R. Real-time fMRI brain computer interfaces: Self-regulation of single brain regions to networks. Biol. Psychol. 2014, 95, 4–20. [Google Scholar] [CrossRef]
  9. Mauss, I.B.; Robinson, M.D. Measures of emotion: A review. Cogn. Emot. 2009, 23, 209–237. [Google Scholar] [CrossRef]
  10. Tomarken, A.J.; Davidson, R.J.; Wheeler, R.E.; Doss, R.C. Individual Differences in Anterior Brain Asymmetry and Fundamental Dimensions of Emotion. J. Personal. Soc. Psychol. 1992, 62, 676–687. [Google Scholar] [CrossRef]
  11. Ioannou, S.; Gallese, V.; Merla, A. Thermal infrared imaging in psychophysiology: Potentialities and limits. Psychophysiology 2014, 51, 951–963. [Google Scholar] [CrossRef] [Green Version]
  12. Hagemann, D.; Waldstein, S.R.; Thayer, J.F. Central and autonomic nervous system integration in emotion. Brain Cogn. 2003, 52, 79–87. [Google Scholar] [CrossRef]
  13. Lang, P.; Greenwald, M.; Bradley, M.; Hamm, A. Looking at picture: Affective, facial, visceral, and behavioral reactions. Psychophysiology 1993, 30, 261–273. [Google Scholar] [CrossRef]
  14. Kreibig, S.D. Autonomic nervous system activity in emotion: A review. Biol. Psychol. 2010, 84, 394–421. [Google Scholar] [CrossRef]
  15. Levenson, R.W. Blood, Sweat, and Fears the Autonomic Architecture of Emotion. Ann. N. Y. Acad. Sci. 2003, 1000, 348–366. [Google Scholar] [CrossRef] [PubMed]
  16. James, W. What is an Emotion? Mind 1884, 9, 188–205. [Google Scholar] [CrossRef]
  17. Schachter, S.; Singer, J.E. Cognitive, social, and physiological determinants of emotional state. Psychol. Rev. 1962, 69, 379–399. [Google Scholar] [CrossRef] [PubMed]
  18. Eckman, P. Universals and cultural differences in facial expressions of emotion. Neb. Symp. Motiv. 1972, 19, 207–284. [Google Scholar] [CrossRef]
  19. Levenson, R. Emotion and the autonomic nervous system: A prospectus for research on autonomic specificity. In Social Psychophysiology and Emotion: Theory and Clinical Applications; Wagner, H.L., Ed.; John Wiley & Sons Ltd: Hoboken, NJ, USA, 1988. [Google Scholar]
  20. Goshvarpour, A.; Abbasi, A.; Goshvarpour, A. An accurate emotion recognition system using ECG and GSR signals and matching pursuit method. Biomed. J. 2017, 40, 355–368. [Google Scholar] [CrossRef]
  21. Valderas, T.; Bolea, J.; Laguna, P.; Ieee, S.M. Human Emotion Recognition Using Heart Rate Variability Analysis with Spectral Bands Based on Respiration. In Proceedings of the 2015 37th IEEE Engineering in Medicine and Biology Society, Milan, Italy, 25–29 August 2015; pp. 6134–6137. [Google Scholar]
  22. McFarland, R.A. Relationship of Skin Temperature Changes to the Emotions Accompanying Music. Biofeedback Self-Regul. 1985, 10, 255–267. [Google Scholar] [CrossRef] [PubMed]
  23. Rimm-kaufman, S.E.; Kagan, J. The Psychological Significance of Changes in Skin Temperature. Motiv. Emot. 1996, 20, 63–78. [Google Scholar] [CrossRef]
  24. Wu, G.; Liu, G.; Hao, M. The analysis of emotion recognition from GSR based on PSO. In Proceedings of the 2010 International Symposium on Intelligence Information Processing and Trusted Computing, Huanggang, China, 28–29 October 2010; pp. 360–363. [Google Scholar]
  25. Torres, C.A.; Orozco, A.; Alvarez, M.A. Feature Selection for Multimodal Emotion Recognition in the Arousal-Valence Space. In Proceedings of the 35th Annual International Conference of the IEEE EMBS, Osaka, Japan, 3–7 July 2013; pp. 4330–4333. [Google Scholar]
  26. Koelstra, S.; Patras, I. Fusion of facial expressions and EEG for implicit affective tagging. Image Vis. Comput. 2013, 31, 164–174. [Google Scholar] [CrossRef]
  27. Li, L.; Chen, J.H. Emotion recognition using physiological signals. In Proceedings of the International Conference on Artificial Reality and Telexistence, Hangzhou, China, 29 November–1 December 2006; pp. 437–446. [Google Scholar]
  28. Sorinas, J.; Grima, M.D.; Ferrandez, J.M. Identifying Suitable Brain Regions and Trial Size Segmentation for Positive/Negative Emotion Recognition. Int. J. Neural Syst. 2018, 29. [Google Scholar] [CrossRef]
  29. Sorinas, J.; Fernandez-Troyano, J.C.; Calvo, M.V.; Ferrandez, J.M.; Fernandez, E. A new model for the implementation of positive and negative emotion recognition. arXiv 2019, arXiv:1905.00230. [Google Scholar]
  30. Chatrian, G.E.; Lettich, E.; Nelson, P.L. Ten percent electrode system for topographic studies of spontaneous and evoked EEG activities. Am. J. EEG Technol. 1985, 25, 83–92. [Google Scholar] [CrossRef]
  31. Ferree, T.C.; Luu, P.; Russell, G.S.; Tucker, D.M. Scalp electrode impedance, infection risk, and EEG data quality. Clin. Neurophysiol. 2001, 112, 536–544. [Google Scholar] [CrossRef]
  32. Delorme, A.; Makeig, S. EEGLAB: An Open Source Toolbox for Analysis of Sin-gle-trail EEG Dynamics Including Independent Compo-nent Anlaysis. J. Neurosci. Methods 2004, 134, 9–21. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  33. Jung, T.P.; Humphries, C.; Lee, T.W.; Makeig, S.; McKeown, M.J.; Iragui, V.; Sejnowski, T.J. Extended ICA removes artifacts from electroencephalographic recordings. Adv. Neural Inf. Process. Syst. 1998, 10, 894–900. [Google Scholar]
  34. Davidson, R.J. EEG Measures of Cerebral Asymmetry: Conceptual and Methodological Issues. Int. J. Neurosci. 1988, 39, 71–89. [Google Scholar] [CrossRef]
  35. Hollander, M.; Wolfe, D.A. Nonparametric Statistical Methods; Hoboken, N.J., Ed.; John Wiley & Sons Inc: Hoboken, NJ, USA, 1999. [Google Scholar]
  36. Kaufmann, T.; Sütterlin, S.; Schulz, S.M.; Vögele, C. ARTiiFACT: A tool for heart rate artifact processing and heart rate variability analysis. Behav. Res. Methods 2011, 43, 1161–1170. [Google Scholar] [CrossRef]
  37. Khandoker, A.H.; Karmakar, C.; Brennan, M.; Voss, A.; Palaniswami, M. Quantitative Poincaré Plot. In Poincaré Plot Methods for Heart Rate Variability Analysis; Springer: Boston, MA, USA, 2013. [Google Scholar]
  38. Hoshi, R.A.; Pastre, C.M.; Vanderlei, L.C.M.; Godoi, M.F. Poincaré plot indexes of heart rate variability: Relationships with other nonlinear variables. Auton. Neurosci. Basic Clin. 2013, 177, 271–274. [Google Scholar] [CrossRef]
  39. Woo, M.A.; Stevenson, W.G.; Moser, D.K.; Trelease, R.B.; Harper, R.M. Patterns of beat-to-beat heart rate variability in advanced heart failure. Am. Heart J. 1992, 123, 704–710. [Google Scholar] [CrossRef]
  40. Tulppo, M.P.; Makikallio, T.H.; Takala, T.E.; Seppanen, T.; Huikuri, H.V. Quantitative beat-to-beat analysis of heart rate dynamics during exercise. Am. J. Physiol. 1996, 271, 244–252. [Google Scholar] [CrossRef]
  41. Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology. Heart rate variability: Standards of measurement, physiological interpretation, and clinical use. Eur. Heart J. 1996, 17, 354–381. [Google Scholar] [CrossRef] [Green Version]
  42. Malliani, A.; Pagani, M.; Lombardi, F.; Cerutti, S. Cardiovascular neural regulation explored in the frequency domain. Circulation 1991, 84, 482–492. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  43. del Paso, G.A.R.; Langewitz, W.; Mulder, L.M.; van Roon, A.; Duschek, S. The utility of low frequency heart rate variability as an index of sympathetic cardiac tone: A review with emphasis on a reanalysis of previous studies. Psychophysiology 2013, 50, 477–487. [Google Scholar] [CrossRef] [PubMed]
  44. Eckberg, D. Sympathovagal balance: A critical appraisal. Circulation 1997, 96, 3224–3232. [Google Scholar] [CrossRef] [PubMed]
  45. Mourot, L.; Bouhaddi, M.; Perrey, S.; Rouillon, J.-D.; Regnard, J. Quantitative Poincare plot analysis of heart rate variability: Effect of endurance training. Eur. J. Appl. Physiol. 2004, 91, 79–87. [Google Scholar] [CrossRef] [PubMed]
  46. Storbeck, J.; Clore, G.L. On the interdependence of cognition and emotion. Cogn. Emot. 2007, 21, 1212–1237. [Google Scholar] [CrossRef] [Green Version]
  47. Kober, H.; Barrett, F.; Joseph, J.; Bliss-moreau, E.; Lindquist, K.; Wager, T.D. Functional grouping and cortical—Subcortical interactions in emotion: A meta-analysis of neuroimaging studies. Neuroimage 2008, 42, 998–1031. [Google Scholar] [CrossRef] [Green Version]
  48. Schmidt, L.A.; Trainor, L.J. Frontal brain electrical activity (EEG) distinguishes valence and intensity of musical emotions valence and intensity of musical emotions. Cogn. Emot. 2015, 15, 487–500. [Google Scholar] [CrossRef]
  49. Murphy, F.C.; Nimmo-smith, I.A.N.; Lawrence, A.D. Functional neuroanatomy of emotions: A meta-analysis. Cogn. Affect. Behav. Neurosci. 2003, 3, 207–233. [Google Scholar]
  50. Coan, J.A.; Allen, J.J.B. Frontal EEG asymmetry as a moderator and mediator of emotion. Biol. Psychol. 2004, 67, 7–49. [Google Scholar] [CrossRef]
  51. Vecchiato, G.; Toppi, J.; Astolfi, L.; De Vico Fallani, F.; Cincotti, F.; Mattia, D.; Bez, F.; Babiloni, F. Spectral EEG frontal asymmetries correlate with the experienced pleasantness of TV commercial advertisements. Med. Biol. Eng. Comput. 2011, 49, 579–583. [Google Scholar] [CrossRef]
  52. Nie, D.; Wang, X.; Shi, L.; Member, B.L.S. EEG-based emotion recognition during watching movies. In Proceedings of the 5th International IEEE EMBS Conference on Neural Engineering, Cancun, Mexico, 27 April–1 May 2011. [Google Scholar]
  53. Muller, M.; Keil, A.; Gruber, T.; Elbert, T. Processing of affective pictures modulates right-hemispheric gamma band EEG activity. Clin. Neurophysiol. 1999, 110, 1913–1920. [Google Scholar] [CrossRef] [Green Version]
  54. Porges, S. The polyvagal theory: New insights into adaptive reactions of the autonomic nervous system. Clevel. Clin. J. Med. 2009, 76, 86–90. [Google Scholar] [CrossRef]
  55. Guo, H.; Huang, Y. Heart Rate Variability Signal Features for Emotion Recognition by using Principal Component Analysis and Support Vectors Machine. In Proceedings of the 2016 IEEE 16th International Conference on Bioinformatics and Bioengineering, Taichung, Taiwan, 31 October–2 November 2016; pp. 2–5. [Google Scholar]
  56. McCraty, R.; Atkinson, M.; Tiller, W.A.; Rein, G.; Watkins, A.D. The effects of emotions on short-term power spectrum analysis of heart rate variability. Am. J. Cardiol. 1995, 76, 1089–1093. [Google Scholar] [CrossRef]
  57. Murugappan, M.; Murugappan, S.; Zheng, B. Frequency Band Analysis of Electrocardiogram (ECG) Signals for Human Emotional State Classification Using Discrete Wavelet Transform(DWT). J. Phys. Ther. Sci. 2013, 25, 753–759. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  58. Umetani, K.E.N.; Singer, D.H.; Craty, R.M.C.; Atkinson, M. Twenty-Four Hour Time Domain Heart Rate Variability and Heart Rate: Relations to Age and Gender Over Nine Decades. J. Am. Coll. Cardiol. 1998, 31, 593–601. [Google Scholar] [CrossRef]
  59. Moodithaya, S.; Avadhany, S.T. Gender Differences in Age-Related Changes in Cardiac Autonomic Nervous Function. J. Aging Res. 2012. [Google Scholar] [CrossRef] [Green Version]
  60. Nater, U.M.; Abbruzzese, E.; Krebs, M.; Ehlert, U. Sex differences in emotional and psychophysiological responses to musical stimuli. Int. J. Psychophysiol. 2006, 62, 300–308. [Google Scholar] [CrossRef]
  61. Hardy, J.; Bois, E.D. Differences between men and women in their response to heat and cold. Physiology 1940, 26, 389–398. [Google Scholar] [CrossRef] [Green Version]
  62. Marchand, I.; Johnson, D.; Montgomery, D.; Brisson, G.; Perrault, H. Gender differences in temperature and vascular characteristics during exercise recovery. Can. J. Appl. Physiol. 2001, 26, 425–441. [Google Scholar] [CrossRef]
  63. Verma, G.K.; Tiwary, U.S. Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals. NeuroImage 2014, 102, 162–172. [Google Scholar] [CrossRef] [PubMed]
  64. Chen, J.; Hu, B.; Xu, L.; Moore, P.; Su, Y. Feature-level fusion of multimodal physiological signals for emotion recognition. In Proceedings of the 2015 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Washington, DC, USA, 9–12 November 2015; pp. 395–399. [Google Scholar]
  65. Jatupaiboon, N.; Pan-Ngum, S.; Israsena, P. Subject-dependent and subject-independent emotion classification using unimodal and multimodal physiological signals. J. Med. Imaging Health Inform. 2015, 5, 1020–1027. [Google Scholar] [CrossRef]
  66. Torres-Valencia, C.; Álvarez-López, M.; Orozco-Gutiérrez, Á. SVM-based feature selection methods for emotion recognition from multimodal data. J. Multimodal User Interfaces 2017, 11, 9–23. [Google Scholar] [CrossRef]
  67. Bailenson, J.N.; Pontikakis, E.D.; Mauss, I.B.; Gross, J.J.; Jabon, M.E.; Hutcherson, C.A.; Nass, C.; John, O. Real-time classification of evoked emotions using facial feature tracking and physiological responses. Int. J. Hum. Comput. Stud. 2008, 66, 303–317. [Google Scholar] [CrossRef]
  68. Hua, J.; Xiong, Z.; Lowey, J.; Suh, E.; Dougherty, E.R. Optimal number of features as a function of sample size for various classification rules. Bioinformatics 2005, 21, 1509–1515. [Google Scholar] [CrossRef] [Green Version]
  69. Harmon-Jones, E. Clarifying the emotive functions of asymmetrical frontal cortical activity. Psychophysiology 2003, 40, 838–848. [Google Scholar] [CrossRef]
Sensors 20 00313 g001 550
Figure 1. Experimental setup and multimodal approach for emotion classification. We performed the final multimodal classification stage with a total of 22 features (20 EEG frequency-location pairs, 1 HRV (heart rate variability) measure and skin temperature). Tº, skin temperature; IBIs, inter-beat intervals.
Figure 1. Experimental setup and multimodal approach for emotion classification. We performed the final multimodal classification stage with a total of 22 features (20 EEG frequency-location pairs, 1 HRV (heart rate variability) measure and skin temperature). Tº, skin temperature; IBIs, inter-beat intervals.
Sensors 20 00313 g001
Sensors 20 00313 g002 550
Figure 2. p-value results for the differences between positive and negative emotions at all the HRV measures and for all subjects.
Figure 2. p-value results for the differences between positive and negative emotions at all the HRV measures and for all subjects.
Sensors 20 00313 g002
Sensors 20 00313 g003 550
Figure 3. Mean temperature values for every positive and negative audiovisual stimulus used. Videos are ranked from lowest to highest score on the valence scale.
Figure 3. Mean temperature values for every positive and negative audiovisual stimulus used. Videos are ranked from lowest to highest score on the valence scale.
Sensors 20 00313 g003
Sensors 20 00313 g004 550
Figure 4. AI for the nine studied frequency-location pairs at positive and negative emotions for all subjects. The bracket with the asterisk above, which joins two bars of the histogram, indicates those pairs that have shown significant differences (p-value < 0.05) between the positive-negative conditions. Asterisks on the bars indicate significant differences (p-value < 0.05) between interhemispheric, intra-condition (positive-positive or negative-negative).
Figure 4. AI for the nine studied frequency-location pairs at positive and negative emotions for all subjects. The bracket with the asterisk above, which joins two bars of the histogram, indicates those pairs that have shown significant differences (p-value < 0.05) between the positive-negative conditions. Asterisks on the bars indicate significant differences (p-value < 0.05) between interhemispheric, intra-condition (positive-positive or negative-negative).
Sensors 20 00313 g004
Sensors 20 00313 g005 550
Figure 5. F1 scores for the SD approach at the different multimodal classifications performed with the KNN and QDA classifiers.
Figure 5. F1 scores for the SD approach at the different multimodal classifications performed with the KNN and QDA classifiers.
Sensors 20 00313 g005
Sensors 20 00313 g006 550
Figure 6. F1 scores for the SI approach at the different multi-modal classifications performed with the KNN and QDA classifiers. The black dotted line corresponds to value of chance.
Figure 6. F1 scores for the SI approach at the different multi-modal classifications performed with the KNN and QDA classifiers. The black dotted line corresponds to value of chance.
Sensors 20 00313 g006
Sensors 20 00313 g007 550
Figure 7. SD and SI correlation of the 22 features of the computational model of study and the emotional valence dimension. PFr: Pre-Frontal right. PFl: Pre-Frontal left. Fm = Frontal midline. Cm = Central midline. Cl = Central left. Cr = Central right. Pr = Parietal right. Or = Occipital right. POm = Parietal-occipital midline.
Figure 7. SD and SI correlation of the 22 features of the computational model of study and the emotional valence dimension. PFr: Pre-Frontal right. PFl: Pre-Frontal left. Fm = Frontal midline. Cm = Central midline. Cl = Central left. Cr = Central right. Pr = Parietal right. Or = Occipital right. POm = Parietal-occipital midline.
Sensors 20 00313 g007
Table
Table 1. Heart rate variability variables.
Table 1. Heart rate variability variables.
Time DomainMean RR: Mean Inter-Beat Interval
SDNN: standard deviation on NN (normal-to-normal) intervals
RMSSD: square of the root of MSSD (mean square difference of successive NN intervals)
NN50: the number of pairs of adjacent NN intervals differing by more than 50 ms
pNN50: the proportion derived by dividing the NN50 by the total number of NN intervals
RMSSD, NN50, and pNN50 are thought to represent parasympathetic mediated HRV [41].
Frequency DomainVLF: very-low-frequency component (0.003–0.04 Hz)
LF: low-frequency component (0.04–0.15 Hz). There is controversy on whether the LF component reflects SNS activity, is a product of both SNS and PNS [41,42] or instead it is also mainly determined by the PNS [43].
HF: high-frequency component occurs at the frequency of adult respiration (0.15–0.4 Hz), primarily reflects cardiac parasympathetic influence due to respiratory sinus arrhythmia.
LF/HF ratio: This rate is interpreted as an index of sympathovagal balance [44].
Poincare PlotSD1: standard deviation of the instantaneous (short-term) beat-to-beat RR interval variability. As vagal regulation over the sinus node are known to be faster than the sympathetically mediated effects, SD1 is considered a parasympathetic index [45].
SD2: standard deviation of the continuous long-term RR interval variability. There is evidence of both parasympathetic and sympathetic tones influenced on this index [38].
SD1/SD2 ratio: ratio between the short and long interval variation.
Table
Table 2. SI classification F1 scores based on our EEG model *.
Table 2. SI classification F1 scores based on our EEG model *.
All PopulationWomenMen
QDA0.5240.4630.483
classifier(s.d. 0.082)(s.d. 0.082)(s.d. 0.076)
KNN0.5220.4990.474
classifier(s.d. 0.092)(s.d. 0.041)(s.d. 0.065)
* Data coming from previous study [29].

Share and Cite

MDPI and ACS Style

Sorinas, J.; Ferrández, J.M.; Fernandez, E. Brain and Body Emotional Responses: Multimodal Approximation for Valence Classification. Sensors 2020, 20, 313. https://doi.org/10.3390/s20010313

AMA Style

Sorinas J, Ferrández JM, Fernandez E. Brain and Body Emotional Responses: Multimodal Approximation for Valence Classification. Sensors. 2020; 20(1):313. https://doi.org/10.3390/s20010313

Chicago/Turabian Style

Sorinas, Jennifer, Jose Manuel Ferrández, and Eduardo Fernandez. 2020. "Brain and Body Emotional Responses: Multimodal Approximation for Valence Classification" Sensors 20, no. 1: 313. https://doi.org/10.3390/s20010313

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop