Skip to main content
SpringerLink
Log in

EEG-based Emotion Recognition Using Multiple Kernel Learning

  • Research Article
  • Published:
Machine Intelligence Research Aims and scope Submit manuscript

Abstract

Emotion recognition based on electroencephalography (EEG) has a wide range of applications and has great potential value, so it has received increasing attention from academia and industry in recent years. Meanwhile, multiple kernel learning (MKL) has also been favored by researchers for its data-driven convenience and high accuracy. However, there is little research on MKL in EEG-based emotion recognition. Therefore, this paper is dedicated to exploring the application of MKL methods in the field of EEG emotion recognition and promoting the application of MKL methods in EEG emotion recognition. Thus, we proposed a support vector machine (SVM) classifier based on the MKL algorithm EasyMKL to investigate the feasibility of MKL algorithms in EEG-based emotion recognition problems. We designed two data partition methods, random division to verify the validity of the MKL method and sequential division to simulate practical applications. Then, tri-categorization experiments were performed for neutral, negative and positive emotions based on a commonly used dataset, the Shanghai Jiao Tong University emotional EEG dataset (SEED). The average classification accuracies for random division and sequential division were 92.25% and 74.37%, respectively, which shows better classification performance than the traditional single kernel SVM. The final results show that the MKL method is obviously effective, and the application of MKL in EEG emotion recognition is worthy of further study. Through the analysis of the experimental results, we discovered that the simple mathematical operations of the features on the symmetrical electrodes could not effectively integrate the spatial information of the EEG signals to obtain better performance. It is also confirmed that higher frequency band information is more correlated with emotional state and contributes more to emotion recognition. In summary, this paper explores research on MKL methods in the field of EEG emotion recognition and provides a new way of thinking for EEG-based emotion recognition research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, J. G. Taylor. Emotion recognition in human-computer interaction. IEEE Signal Processing Magazine, vol. 18, no. 1, pp. 32–80, 2001. DOI: https://doi.org/10.1109/79.911197.

    Article  Google Scholar 

  2. N. Mehendale. Facial emotion recognition using convolutional neural networks (FERC). SN Applied Sciences, vol. 2, no. 3, Article number 446, 2020. DOI: https://doi.org/10.1007/s42452-020-2234-1.

  3. S. Minaee, M. Minaei, A. Abdolrashidi. Deep-emotion: Facial expression recognition using attentional convolutional network. Sensors, vol. 21, no. 9, Article number 3046, 2021. DOI: https://doi.org/10.3390/s21093046.

  4. D. M. Schuller, B. W. Schuller. A review on five recent and near-future developments in computational processing of emotion in the human voice. Emotion Review, vol. 13, no. 1, pp. 44–50, 2021. DOI: https://doi.org/10.1177/1754073919898526.

    Article  Google Scholar 

  5. Y. M. Huang, K. X. Tian, A. Wu, G. B. Zhang. Feature fusion methods research based on deep belief networks for speech emotion recognition under noise condition. Journal of Ambient Intelligence and Humanized Computing, vol. 10, no. 5, pp. 1787–1798, 2009. DOI: https://doi.org/10.1007/s12652-017-0644-8.

    Article  Google Scholar 

  6. J. H. Tao, J. Huang, Y. Li, Z. Lian, M. Y. Niu. Correction to: Semi-supervised ladder networks for speech emotion recognition. International Journal of Automation and Computing, vol. 18, no. 4, Article number 680, 2021. DOI: https://doi.org/10.1007/s11633-019-1215-6.

  7. Y. Q. Yin, X. W. Zheng, B. Hu, Y. Zhang, X. C. Cui. EEG emotion recognition using fusion model of graph convolutional neural networks and LSTM. Applied Soft Computing, vol. 100, Article number 106954, 2021. DOI: https://doi.org/10.1016/j.asoc.2020.106954.

  8. P. Sarkar, A. Etemad. Self-supervised learning for ecg-based emotion recognition. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, IEEE, Barcelona, Spain, pp. 3217–3221, 2020. DOI: https://doi.org/10.1109/ICASSP40776.2020.9053985.

    Google Scholar 

  9. V. Kehri, R. Ingle, S. Patil, R. N. Awale. Analysis of facial EMG signal for emotion recognition using wavelet packet transform and SVM. Machine Intelligence and Signal Analysis, M. Tanveer, R. B. Pachori, Eds., Singapore: Springer, pp. 247–257, 2019. DOI: https://doi.org/10.1007/978-981-13-0923-6_21.

    Chapter  Google Scholar 

  10. Q. Zhang, X. X. Chen, Q. Y. Zhan, T. Yang, S. H. Xia. Respiration-based emotion recognition with deep learning. Computers in Industry, vol. 92–93, pp. 84–90, 2017. DOI: https://doi.org/10.1016/j.compind.2017.04.005.

    Article  Google Scholar 

  11. L. Shu, Y. Yu, W. Z. Chen, H. Q. Hua, Q. Li, J. X. Jin, X. M. Xu. Wearable emotion recognition using heart rate data from a smart bracelet. Sensors, vol. 20, no. 3, Article number 718, 2020. DOI: https://doi.org/10.3390/s20030718.

  12. D. Ayata, Y. Yaslan, M. Kamaşak. Emotion recognition via random forest and galvanic skin response: Comparison of time based feature sets, window sizes and wavelet approaches. In Proceedings of Medical Technologies National Congress, IEEE, Antalya, Turkey, 2016. DOI: https://doi.org/10.1109/TIPTEKNO.2016.7863130.

    Google Scholar 

  13. S. Huang, W. Shao, M. L. Wang, D. Q. Zhang. fMRI-based decoding of visual information from human brain activity: A brief review. International Journal of Automation and Computing, vol. 18, no. 2, pp. 170–184, 2021. DOI: https://doi.org/10.1007/s11633-020-1263-y.

    Article  Google Scholar 

  14. L. M. Ward. Synchronous neural oscillations and cognitive processes. Trends in Cognitive Sciences, vol. 7, no. 12, pp. 553–559, 2003. DOI: https://doi.org/10.1016/j.tics.2003.10.012.

    Article  Google Scholar 

  15. J. A. Coan, J. J. B. Allen. Frontal EEG asymmetry as a moderator and mediator of emotion. Biological Psychology, vol. 67, no. 1–2, pp. 7–50, 2004. DOI: https://doi.org/10.1016/j.biopsycho.2004.03.002.

    Article  Google Scholar 

  16. J. Jin, Z. M. Chen, R. Xu, Y. Y. Miao, X. Y. Wang, T. P. Jung. Developing a novel tactile P300 brain-computer interface with a cheeks-stim paradigm. IEEE Transactions on Biomedical Engineering, vol. 67, no. 9, pp. 2585–2593, 2020. DOI: https://doi.org/10.1109/TBME.2020.2965178.

    Article  Google Scholar 

  17. Y. Yu, Y. D. Liu, E. W. Yin, J. Jiang, Z. T. Zhou, D. W. Hu. An asynchronous hybrid spelling approach based on EEG-EOG signals for Chinese character input. IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 27, no. 6, pp. 1292–1302, 2019. DOI: https://doi.org/10.1109/TNSRE.2019.2914916.

    Article  Google Scholar 

  18. Z. Mohammadi, J. Frounchi, M. Amiri. Wavelet-based emotion recognition system using EEG signal. Neural Computing and Applications, vol. 28, no. 8, pp. 1985–1990, 2017. DOI: https://doi.org/10.1007/s00521-015-2149-8.

    Article  Google Scholar 

  19. M. Degirmenci, M. A. Ozdemir, R. Sadighzadeh, A. Akan. Emotion recognition from EEG signals by using empirical mode decomposition. In Proceedings of Medical Technologies National Congress, IEEE, Magusa, Cyprus, 2018. DOI: https://doi.org/10.1109/TIPTEKNO.2018.8597061.

    Book  Google Scholar 

  20. Z. Yin, L. Liu, J. N. Chen, B. X. Zhao, Y. X. Wang. Locally robust EEG feature selection for individual-independent emotion recognition. Expert Systems with Applications, vol. 162, Article number 113768, 2020. DOI: https://doi.org/10.1016/j.eswa.2020.113768.

  21. T. F. Song, W. M. Zheng, P. Song, Z. Cui. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Transactions on Affective Computing, vol. 11, no. 3, pp. 532–541, 2020. DOI: https://doi.org/10.1109/TAFFC.2018.2817622.

    Article  Google Scholar 

  22. H. Chao, L. Dong, Y. L. Liu, B. Y. Lu. Emotion recognition from multiband EEG signals using CapsNet. Sensors, vol. 19, no. 9, Article number 2212, 2019. DOI: https://doi.org/10.3390/s19092212.

  23. Y. Cimtay, E. Ekmekcioglu. Investigating the use of pre-trained convolutional neural network on cross-subject and cross-dataset EEG emotion recognition. Sensors, vol. 20, no. 7, Article number 2034, 2020. DOI: https://doi.org/10.3390/s20072034.

  24. E. Iáñez, J. M. Azorín, A. Úbeda, E. Fernández, J. L. Sirvent. LDA-based classifiers for a mental tasks-based brain-computer interface. In Proceedings of IEEE International Conference on Systems, Man and Cybernetics, Istanbul, Turkey, pp. 546–551, 2010. DOI: https://doi.org/10.1109/ICSMC.2010.5642018.

  25. L. Guo, Y. X. Wu, L. Zhao, T. Cao, W. L. Yan, X. Q. Shen. Classification of mental task from EEG signals using immune feature weighted support vector machines. IEEE Transactions on Magnetics, vol. 47, no. 5, pp. 866–869, 2011. DOI: https://doi.org/10.1109/TMAG.2010.2072775.

    Article  Google Scholar 

  26. M. J. Abdi, S. M. Hosseini, M. Rezghi. A novel weighted support vector machine based on particle swarm optimization for gene selection and tumor classification. Computational and Mathematical Methods in Medicine, vol. 2012, Article number 320698, 2012. DOI: https://doi.org/10.1155/2012/320698.

  27. A. Rakotomamonjy, F. R. Bach, S. Canu, Y. Grandvalet. SimpleMKL. Journal of Machine Learning Research, vol. 9, pp. 2491–2521, 2008.

    MathSciNet  MATH  Google Scholar 

  28. M. Varma, B. R. Babu. More generality in efficient multiple kernel learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ACM, Montreal, Canada, pp. 1065–1072, 2009. DOI: https://doi.org/10.1145/1553374.1553510.

    Google Scholar 

  29. M. Kloft, U. Brefeld, S. Sonnenburg, A. Zien. Non-sparse regularization and efficient training with multiple kernels. [Online], Available: http://arxiv.org/abs/1003.0079v1, 2010.

  30. Z. L. Xu, R. Jin, H. Q. Yang, I. King, M. R. Lyu. Simple and efficient multiple kernel learning by group lasso. In Proceedings of the 27th International Conference on International Conference on Machine Learning, Haifa, Israel, pp. 1175–1182, 2010.

  31. F. Aiolli, M. Donini. EasyMKL: A scalable multiple kernel learning algorithm. Neurocomputing, vol. 169, pp. 215–224, 2015. DOI: https://doi.org/10.1016/j.neucom.2014.11.078.

    Article  Google Scholar 

  32. W. Samek, A. Binder, K. R. Müller. Multiple kernel learning for brain-computer interfacing. In Proceedings of the 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, IEEE, Osaka, Japan, pp. 7048–7051, 2013. DOI: https://doi.org/10.1109/EMBC.2013.6611181.

    Google Scholar 

  33. X. O. Li, X. Chen, Y. N. Yan, W. S. Wei, Z. J. Wang. Classification of EEG signals using a multiple kernel learning support vector machine. Sensors, vol. 14, no. 7, pp. 12784–12802, 2014. DOI: https://doi.org/10.3390/s140712784.

    Article  Google Scholar 

  34. C. A. Frantzidis, C. Bratsas, C. L. Papadelis, E. Konstantinidis, C. Pappas, P. D. Bamidis. Toward emotion aware computing: An integrated approach using multichannel neurophysiological recordings and affective visual stimuli. IEEE Transactions on Information Technology in Biomedicine, vol. 14, no. 3, pp. 589–597, 2010. DOI: https://doi.org/10.1109/TITB.2010.2041553.

    Article  Google Scholar 

  35. P. J. Lang, M. M. Bradley, B. N. Cuthbert. International Affective Picture System (IAPS): Technical Manual and Affective Ratings, Technical Report No. 3, NIMH Center for the Study of Emotion and Attention, USA, pp. 39–58, 1997.

    Google Scholar 

  36. R. N. Duan, J. Y. Zhu, B. L. Lu. Differential entropy feature for EEG-based emotion classification. In Proceedings of the 6th International IEEE/EMBS Conference on Neural Engineering, IEEE, San Diego, USA, pp. 81–84, 2013. DOI: https://doi.org/10.1109/NER.2013.6695876.

    Google Scholar 

  37. W. L. Zheng, B. L. Lu. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Transactions on Autonomous Mental Development, vol. 7, no. 3, pp. 162–175, 2015. DOI: https://doi.org/10.1109/TAMD.2015.2431497.

    Article  Google Scholar 

  38. S. Koelstra, C. Muhl, M. Soleymani, J. S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, I. Patras. DEAP: A database for emotion analysis; using physiological signals. IEEE Transactions on Affective Computing, vol. 3, no. 1, pp. 18–31, 2012. DOI: https://doi.org/10.1109/T-AFFC.2011.15.

    Article  Google Scholar 

  39. X. Chen, X. Y. Xu, A. P. Liu, S. Lee, X. Chen, X. Zhang, M. J. McKeown, Z. J. Wang. Removal of muscle artifacts from the EEG: A review and recommendations. IEEE Sensors Journal, vol. 19, no. 14, pp. 5353–5368, 2019. DOI: https://doi.org/10.1109/JSEN.2019.2906572.

    Article  Google Scholar 

  40. R. Jenke, A. Peer, M. Buss. Feature extraction and selection for emotion recognition from EEG. IEEE Transactions on Affective Computing, vol. 5, no. 3, pp. 327–339, 2014. DOI: https://doi.org/10.1109/TAFFC.2014.2339834.

    Article  Google Scholar 

  41. C. Zhang, H. Wang, R. R. Fu. Automated detection of driver fatigue based on entropy and complexity measures. IEEE Transactions on Intelligent Transportation Systems, vol. 15, no. 1, pp. 168–177, 2014. DOI: https://doi.org/10.1109/TITS.2013.2275192.

    Article  Google Scholar 

  42. V. S. Vijith, J. E. Jacob, T. Iype, K. Gopakumar, D. G. Yohannan. Epileptic seizure detection using non linear analysis of EEG. In Proceedings of International Conference on Inventive Computation Technologies, IEEE, Coimbatore, India, 2016. DOI: https://doi.org/10.1109/INVENTIVE.2016.7830193.

    Book  Google Scholar 

  43. R. C. Guido. A tutorial review on entropy-based handcrafted feature extraction for information fusion. Information Fusion, vol. 41, pp. 161–175, 2018. DOI: https://doi.org/10.1016/j.inffus.2017.09.006.

    Article  Google Scholar 

  44. K. H. Kim, S. W. Bang, S. R. Kim. Emotion recognition system using short-term monitoring of physiological signals. Medical and Biological Engineering and Computing, vol. 42, no. 3, pp. 419–427, 2004. DOI: https://doi.org/10.1007/BF02344719.

    Article  Google Scholar 

  45. X. W. Wang, D. Nie, B. L. Lu. Emotional state classification from EEG data using machine learning approach. Neurocomputing, vol. 129, pp. 94–106, 2014. DOI: https://doi.org/10.1016/j.neucom.2013.06.046.

    Article  Google Scholar 

  46. K. R. Muller, S. Mika, G. Ratsch, K. Tsuda, B. Scholkopf. An introduction to kernel-based learning algorithms. IEEE Transactions on Neural Networks, vol. 12, no. 2, pp. 181–201, 2001. DOI: https://doi.org/10.1109/72.914517.

    Article  Google Scholar 

  47. X. Chen, Z. J. Wang. Pattern recognition of number gestures based on a wireless surface EMG system. Biomedical Signal Processing and Control, vol. 8, no. 2, pp. 184–192, 2013. DOI: https://doi.org/10.1016/j.bspc.2012.08.005.

    Article  Google Scholar 

  48. C. M. Qing, R. Qiao, X. M. Xu, Y. Q. Cheng. Interpretable emotion recognition using EEG signals. IEEE Access, vol. 7, pp. 94160–94170, 2019. DOI: https://doi.org/10.1109/ACCESS.2019.2928691.

    Article  Google Scholar 

  49. L. C. Shi, Y. Y. Jiao, B. L. Lu. Differential entropy feature for EEG-based vigilance estimation. In Proceedings of the 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, IEEE, Osaka, Japan, pp. 6627–6630, 2013. DOI: https://doi.org/10.1109/EMBC.2013.6611075.

    Google Scholar 

  50. M. Soleymani, M. Pantic, T. Pun. Multimodal emotion recognition in response to videos. IEEE Transactions on Affective Computing, vol. 3, no. 2, pp. 211–223, 2012. DOI: https://doi.org/10.1109/T-AFFC.2011.37.

    Article  Google Scholar 

  51. W. L. Zheng, J. Y. Zhu, B. L. Lu. Identifying stable patterns over time for emotion recognition from EEG. IEEE Transactions on Affective Computing, vol. 10, no. 3, pp. 417–429, 2019. DOI: https://doi.org/10.1109/TAFFC.2017.2712143.

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (No. 62176054), and University Synergy Innovation Program of Anhui Province, China (No. GXXT-2020-015).

Author information

Authors and Affiliations

  1. School of Statistics and Data Science, Nanjing Audit University, Nanjing, 211815, China

    Qian Cai

  2. Key Laboratory of Child Development and Learning Science of Ministry of Education, School of Biological Science & Medical Engineering, Southeast University, Nanjing, 210096, China

    Guo-Chong Cui & Hai-Xian Wang

  3. Institute of Artificial Intelligence of Hefei Comprehensive National Science Center, Hefei, 230094, China

    Hai-Xian Wang

Authors
  1. Qian Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guo-Chong Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hai-Xian Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Guo-Chong Cui or Hai-Xian Wang.

Additional information

Colored figures are available in the online version at https://link.springer.com/journal/11633

Qian Cai received the B. Sc. and M. Sc. degrees in mathematics from Anhui University, China in 2000 and 2003, respectively. Currently, she is with School of Statistics and Data Science, Nanjing Audit University, China.

Her research interests include statistical pattern recognition and data science.

Guo-Chong Cui received the B. Sc. degree in biomedical engineering from Yanshan University, China in 2019. He is a master student in biomedical engineering at Department of Biomedical Engineering, School of Biological Science & Medical Engineering, Southeast University, China.

His research interests include EEG-based emotion recognition, machine learning and neural information processing.

Hai-Xian Wang received the B. Sc. and M. Sc. degrees in statistics and the Ph. D. degree in computer science from Anhui University, China in 1999, 2002 and 2005, respectively. Currently, he is with Key Laboratory of Child Development and Learning Science of Ministry of Education, School of Biological Science & Medical Engineering, Southeast University, China.

His research interests include biomedical signal processing, brain-computer interfaces, and machine learning.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cai, Q., Cui, GC. & Wang, HX. EEG-based Emotion Recognition Using Multiple Kernel Learning. Mach. Intell. Res. 19, 472–484 (2022). https://doi.org/10.1007/s11633-022-1352-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11633-022-1352-1

Keywords

Search

Navigation