Rafael Zimmer
ABSTRACT
This paper focuses on the use of emotion recognition techniques to assist psychologists in performing children’s therapy through remotely robot operated sessions. In the field of psychology, the use of agent-mediated therapy is growing increasingly given recent advances in robotics and computer science. Specifically, the use of Embodied Conversational Agents (ECA) as an intermediary tool can help professionals connect with children who face social challenges such as Attention Deficit Hyperactivity Disorder (ADHD), Autism Spectrum Disorder (ASD) or even who are physically unavailable due to being in regions of armed conflict, natural disasters, or other circumstances. In this context, emotion recognition represents an important feedback for the psychotherapist. In this article, we initially present the result of a bibliographical research associated with emotion recognition in children. This research revealed an initial overview on algorithms and datasets widely used by the community. Then, based on the analysis carried out on the results of the bibliographical research, we used the technique of dense optical flow features to improve the ability of identifying emotions in children in uncontrolled environments. From the output of a hybrid model of Convolutional Neural Network, two intermediary features are fused before being processed by a final classifier. The proposed architecture was called HybridCNNFusion. Finally, we present the initial results achieved in the recognition of children’s emotions using a dataset of Brazilian children.
- Lisa Feldman Barrett, Ralph Adolphs, Stacy Marsella, Aleix M. Martinez, and Seth D. Pollak. 2019. Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements. PSYCHOLOGICAL SCIENCE IN THE PUBLIC INTEREST 20, 1 (7 2019), 1–68. https://doi.org/10.1177/1529100619832930Google Scholar
Cross Ref
- De’Aira Bryant and Ayanna Howard. 2019. A Comparative Analysis of Emotion-Detecting Al Systems with Respect to Algorithm Performance and Dataset Diversity. In AIES ‘19: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. 377–382. https://doi.org/10.1145/3306618.3314284Google ScholarDigital Library
- M. Catalina Camacho, Helmet T. Karim, and Susan B. Perlman. 2019. Neural architecture supporting active emotion processing in children: A multivariate approach. NEUROIMAGE 188 (3 2019), 171–180. https://doi.org/10.1016/j.neuroimage.2018.12.013Google ScholarCross Ref
- L. I. Cuadrado, M. R. Angeles, and F. P. Lopez. 2019. FER in Primary School Children for Affective Robot Tutors. In FROM BIOINSPIRED SYSTEMS AND BIOMEDICAL APPLICATIONS TO MACHINE LEARNING, PT II(Lecture Notes in Computer Science, Vol. 11487). Spanish CYTED; Red Nacl Computac Nat & Artificial, Programa Grupos Excelencia Fundac Seneca & Apliquem Microones 21 s l, 461–471. https://doi.org/10.1007/978-3-030-19651-6_45Google ScholarCross Ref
- A. Ohman D. E.Lundqvist, A. Flykt. 1998. The Karolinska Directed Emotional Face. https://www.kdef.se/Google Scholar
- Arnaud Dapogny, Charline Grossard, Stephanie Hun, Sylvie Serret, Ouriel Grynszpan, Severine Dubuisson, David Cohen, and Kevin Bailly. 2019. On Automatically Assessing Children’s Facial Expressions Quality: A Study, Database, and Protocol. 1 (10 2019). https://doi.org/10.3389/fcomp.2019.00005Google ScholarCross Ref
- Cynthia Borges de Moura and M.R.Z.S. Azevedo. 2000. Estratégias lúdicas para uso em terapia comportamental infantil. In Sobre comportamento e cognição: questionando e ampliando a teoria e as intervenções clínicas e em outros contextos, R. C. Wielenska (Ed.). Vol. 6. Santo André, 163–170.Google Scholar
- P. Ekman and W.V. Friesen. 1978. Facial Action Coding System. Number v. 1. Consulting Psychologists Press. https://books.google.com.br/books?id=08l6wgEACAAJGoogle Scholar
- P. Ekman and K. Scherer. 1984. Expression and the Nature of Emotion. lawrence Erlbaum Associates. https://www.paulekman.com/wp-content/uploads/2013/07/Expression-And-The-Nature-Of-Emotion.pdfGoogle Scholar
- Gunnar Farnebäck. 2003. Two-Frame Motion Estimation Based on Polynomial Expansion., 363–370 pages.Google Scholar
- Christiane Goulart, Carlos Valadao, Denis Delisle-Rodriguez, Douglas Funayama, Alvaro Favarato, Guilherme Baldo, Vinicius Binotte, Eliete Caldeira, and Teodiano Bastos-Filho. 2019. Visual and Thermal Image Processing for Facial Specific Landmark Detection to Infer Emotions in a Child-Robot Interaction. SENSORS 19, 13 (7 2019). https://doi.org/10.3390/s19132844Google ScholarCross Ref
- M. I. U. Haque and D. Valles. 2018. A Facial Expression Recognition Approach Using DCNN for Autistic Children to Identify Emotions, S Chakrabarti and HN Saha (Eds.). Inst Engn & Management; IEEE Vancouver Sect; UBC; Univ Engn & Management, IEEE, 546–551.Google Scholar
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition. arxiv:1512.03385 [cs.CV]Google Scholar
- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735–1780.Google ScholarDigital Library
- Jiuk Hong, Chaehyeon Lee, and Heechul Jung. 2022. Late Fusion-Based Video Transformer for Facial Micro-Expression Recognition. APPLIED SCIENCES-BASEL 12, 3 (2 2022). https://doi.org/10.3390/app12031169Google ScholarCross Ref
- Asha Jaison and C. Deepa. 2021. A Review on Facial Emotion Recognition and Classification Analysis with Deep Learning. BIOSCIENCE BIOTECHNOLOGY RESEARCH COMMUNICATIONS 14, 5, SI (2021), 154–161. https://doi.org/10.21786/bbrc/14.5/29Google ScholarCross Ref
- Salma Kammoun Jarraya, Marwa Masmoudi, and Mohamed Hammami. 2020. Compound Emotion Recognition of Autistic Children During Meltdown Crisis Based on Deep Spatio-Temporal Analysis of Facial Geometric Features. IEEE ACCESS 8 (2020), 69311–69326. https://doi.org/10.1109/ACCESS.2020.2986654Google ScholarCross Ref
- Samira Ebrahimi Kahou, Vincent Michalski, Kishore Konda, Roland Memisevic, and Christopher Pal. 2015. Recurrent Neural Networks for Emotion Recognition in Video. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. ACM. https://doi.org/10.1145/2818346.2830596Google ScholarDigital Library
- Haik Kalantarian, Khaled Jedoui, Kaitlyn Dunlap, Jessey Schwartz, Peter Washington, Arman Husic, Qandeel Tariq, Michael Ning, Aaron Kline, and Dennis Paul Wall. 2020. The Performance of Emotion Classifiers for Children With Parent-Reported Autism: Quantitative Feasibility Study. JMIR MENTAL HEALTH 7, 4 (4 2020). https://doi.org/10.2196/13174Google ScholarCross Ref
- Haik Kalantarian, Khaled Jedoui, Kaitlyn Dunlap, Jessey Schwartz, Peter Washington, Arman Husic, Qandeel Tariq, Michael Ning, Aaron Kline, and Dennis Paul Wall. 2020. The Performance of Emotion Classifiers for Children With Parent-Reported Autism: Quantitative Feasibility Study. JMIR MENTAL HEALTH 7, 4 (4 2020). https://doi.org/10.2196/13174Google ScholarCross Ref
- Akhilesh Kumar and Awadhesh Kumar. 2022. Analysis of Machine Learning Algorithms for Facial Expression Recognition. In ADVANCED NETWORK TECHNOLOGIES AND INTELLIGENT COMPUTING, ANTIC 2021, Vol. 1534. 730–750. https://doi.org/10.1007/978-3-030-96040-7_55Google ScholarCross Ref
- S. Li, W. Zheng, Y. Zong, C. Lu, C. Tang, X. Jiang, J. Liu, and W. Xia. 2019. Bi-modality Fusion for Emotion Recognition in the Wild. ASSOC COMPUTING MACHINERY, 1601 Broadway, 10th Floor, NEW YORK, NY, UNITED STATES, 589–594. https://doi.org/10.1145/3340555.3355719Google ScholarDigital Library
- Xiaohong Li. 2022. Expression Recognition of Classroom Children’s Game Video Based on Improved Convolutional Neural Network. SCIENTIFIC PROGRAMMING 2022 (4 2022). https://doi.org/10.1155/2022/5203022Google ScholarCross Ref
- Jose Luis Espinosa-Aranda, Noelia Vallez, Jose Maria Rico-Saavedra, Javier Parra-Patino, Gloria Bueno, Matteo Sorci, David Moloney, Dexmont Pena, and Oscar Deniz. 2018. Smart Doll: Emotion Recognition Using Embedded Deep Learning. SYMMETRY-BASEL 10, 9 (9 2018). https://doi.org/10.3390/sym10090387Google ScholarCross Ref
- Aleix M. Martinez. 2019. The Promises and Perils of Automated Facial Action Coding in Studying Children’s Emotions. DEVELOPMENTAL PSYCHOLOGY 55, 9, SI (9 2019), 1965–1981. https://doi.org/10.1037/dev0000728Google ScholarCross Ref
- Juliana Gioia Negrão, Ana Alexandra Caldas Osorio, Rinaldo Focaccia Siciliano, Vivian Renne Gerber Lederman, Elisa Harumi Kozasa, Maria Eloisa Famá D’Antino, Anderson Tamborim, Vitor Santos, David Leonardo Barsand de Leucas, Paulo Sergio Camargo, Daniel C. Mograbi, Tatiana Pontrelli Mecca, and José Salomão Schwartzman. 2021. The Child Emotion Facial Expression Set: A Database for Emotion Recognition in Children. Frontiers in Psychology 12 (2021). https://doi.org/10.3389/fpsyg.2021.666245Google ScholarCross Ref
- Jean Piaget. 1952. The Origins of Intelligence in Children. International Universities Press.Google Scholar
- Simon Provoost, Ho Ming Lau, Jeroen Ruwaard, and Heleen Riper. 2017. Embodied Conversational Agents in Clinical Psychology: A Scoping Review. Journal of Medical Internet Research 19, 5 (05 2017), e151. https://doi.org/10.2196/jmir.6553Google ScholarCross Ref
- Sergio Pulido-Castro, Nubia Palacios-Quecan, Michelle P. Ballen-Cardenas, Sandra Cancino-Suarez, Alejandra Rizo-Arevalo, and Juan M. Lopez Lopez. 2021. Ensemble of Machine Learning Models for an Improved Facial Emotion Recognition. In 2021 IEEE URUCON. IEEE, 345 E 47TH ST, NEW YORK, NY 10017 USA, 512–516. https://doi.org/10.1109/URUCON53396.2021.9647375 IEEE URUCON Conference (IEEE URUCON), Montevideo, URUGUAY, NOV 24-26, 2021.Google ScholarCross Ref
- Manas Sambare. 2022. "FER-2013 Learn facial expressions from an image. https://www.kaggle.com/datasets/msambare/fer2013. accessed on 15 fev 2023.Google Scholar
- Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. arxiv:1409.1556 [cs.CV]Google Scholar
- Ninu Preetha Nirmala Sreedharan, Brammya Ganesan, Ramya Raveendran, Praveena Sarala, Binu Dennis, and Rajakumar R. Boothalingam. 2018. Grey Wolf optimisation-based feature selection and classification for facial emotion recognition. IET BIOMETRICS 7, 5 (9 2018), 490–499. https://doi.org/10.1049/iet-bmt.2017.0160Google ScholarCross Ref
- Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2014. Going Deeper with Convolutions. arxiv:1409.4842 [cs.CV]Google Scholar
- Yaniv Taigman, Ming Yang, Marc'Aurelio Ranzato, and Lior Wolf. 2014. DeepFace: Closing the Gap to Human-Level Performance in Face Verification. In 2014 IEEE Conference on Computer Vision and Pattern Recognition. IEEE. https://doi.org/10.1109/cvpr.2014.220Google ScholarDigital Library
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. arxiv:1706.03762 [cs.CL]Google Scholar
- Paul Viola and Michael Jones. 2001. Rapid Object Detection using a Boosted Cascade of Simple Features. CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (2001).Google Scholar
- Web of Science. 2022. "Web of Science platform.bit.ly/3McZko4. accessed on 08 may 2022.Google Scholar
- John R. Weisz and Alan E. Kazdin. 2010. Evidence-Based Psychotherapies for Children and Adolescents. Guilford Press.Google Scholar
- Guiping Yu. 2021. Emotion Monitoring for Preschool Children Based on Face Recognition and Emotion Recognition Algorithms. COMPLEXITY 2021 (3 2021). https://doi.org/10.1155/2021/6654455Google ScholarCross Ref
- Zimmer, R. and Sobral, M. and Azevedo, H.2023. Spreadsheet with Reference Classification Groups. https://tinyurl.com/hybridmodelsbibliography. accessed on 08 may 2023.Google Scholar
Index Terms
- Hybrid Models for Facial Emotion Recognition in Children
Index terms have been assigned to the content through auto-classification.
Recommendations
A computer activity to encourage facial expression recognition for children with Autism Spectrum Disorders
iConference '12: Proceedings of the 2012 iConferenceChildren with Autism Spectrum Disorders (ASD) do not recognize emotions in context as quickly as children without ASD. This research explores the use of Information and Communication Technology (ICT) in the area of social skills education. The project ...
A gamified app on emotion recognition and anger management for pre-school children
AbstractAnger is a common reason for child mental-health referrals. Therefore, children need to develop emotion recognition and anger management skills from an early age. Gamified apps can deliver mental health interventions for children, but ...
Comments