Hong-Xia Xie
ABSTRACT
Micro-expressions (MEs) are important clues for reflecting the real feelings of humans, and micro-expression recognition (MER) can thus be applied in various real-world applications. However, it is difficult to perceive and interpret MEs correctly. With the advance of deep learning technologies, the accuracy of micro-expression recognition is improved but still limited by the lack of large-scale datasets. In this paper, we propose a novel micro-expression recognition approach by combining Action Units (AUs) and emotion category labels. Specifically, based on facial muscle movements, we model different AUs based on relational information and integrate the AUs recognition task with MER. Besides, to overcome the shortcomings of limited and imbalanced training samples, we propose a data augmentation method that can generate nearly indistinguishable image sequences with AU intensity of real-world micro-expression images, which effectively improve the performance and are compatible with other micro-expression recognition methods. Experimental results on three mainstream micro-expression datasets, i.e., CASME II, SAMM, and SMIC, manifest that our approach outperforms other state-of-the-art methods on both single database and cross-database micro-expression recognition.
Supplemental Material
Available for Download
zip
mmfp1205aux.zip (1.1 MB)
Due to the space limit, the supplemental file contains Implementation Details of Backbone Architecture and Synthetic Results from SAMM.
- Iman Abbasnejad, Sridha Sridharan, Dung Nguyen, Simon Denman, Clinton Fookes, and Simon Lucey. 2017. Using Synthetic Data to Improve Facial Expression Analysis with 3D Convolutional Networks. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 1609--1618.Google Scholar
Cross Ref
- Jie Cai, Zibo Meng, Ahmed Shehab Khan, Zhiyuan Li, James O'Reilly, and Yan Tong. 2019. Identity-Free Facial Expression Recognition using conditional Generative Adversarial Network. arXiv:1903.08051 (2019).Google Scholar
- Cua tua lina Cangea, Petar Velivc ković, Nikola Jovanović, Thomas Kipf, and Pietro Liò. 2018. Towards sparse hierarchical graph classifiers. In Proceedings of the Workshop on Relational Representation Learning (R2L) at Conference on Neural Information Processing Systems.Google Scholar
- Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. 2019. Multi-label image recognition with graph convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5177--5186.Google ScholarCross Ref
- Wen-Sheng Chu, Fernando De la Torre, and Jeffrey F Cohn. 2017. Learning spatial and temporal cues for multi-label facial action unit detection. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition. 25--32.Google ScholarCross Ref
- Adrian Davison, Cliff Lansley, Nicholas Costen, Kevin Tan, and Moi Hoon Yap. 2016. SAMM: A Spontaneous Micro-Facial Movement Dataset. IEEE Transactions on Affective Computing, Vol. PP (06 2016), 1--1.Google Scholar
- Paul Ekman and Wallace V Friesen. 1969. Nonverbal leakage and clues to deception. Psychiatry, Vol. 32, 1 (1969), 88--106.Google ScholarCross Ref
- Itir Onal Ertugrul, Jeffrey F Cohn, László A Jeni, Zheng Zhang, Lijun Yin, and Qiang Ji. 2019. Cross-domain au detection: Domains, learning approaches, and measures. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition. 1--8.Google ScholarCross Ref
- Mark Frank, Malgorzata Herbasz, Kang Sinuk, A Keller, and Courtney Nolan. 2009. I see how you feel: Training laypeople and professionals to recognize fleeting emotions. In The Annual Meeting of the International Communication Association.Google Scholar
- E Friesen and Paul Ekman. 1978. Facial action coding system: a technique for the measurement of facial movement. Palo Alto, Vol. 3 (1978).Google Scholar
- Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. 2017. Improved training of wasserstein gans. In Advances in neural information processing systems. 5767--5777.Google Scholar
- Yanjun Guo, Yantao Tian, Xu Gao, and Xuange Zhang. 2014. Micro-expression recognition based on local binary patterns from three orthogonal planes and nearest neighbor method. In Proceedings of the IEEE International Joint Conference on Neural Networks. 3473--3479.Google ScholarCross Ref
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.Google ScholarCross Ref
- Chia-Wei Hsieh, Chieh-Yun Chen, Chien-Lung Chou, Hong-Han Shuai, Jiaying Liu, and Wen-Huang Cheng. 2019. FashionOn: Semantic-guided Image-based Virtual Try-on with Detailed Human and Clothing Information. In Proceedings of the ACM International Conference on Multimedia. 275--283.Google ScholarDigital Library
- Heechul Jung, Sihaeng Lee, Junho Yim, Sunjeong Park, and Junmo Kim. 2015. Joint Fine-Tuning in Deep Neural Networks for Facial Expression Recognition. In Proceedings of the IEEE International Conference on Computer Vision. 2983--2991.Google ScholarDigital Library
- Huai-Qian Khor, John See, Raphael Chung Wei Phan, and Weiyao Lin. 2018. Enriched long-term recurrent convolutional network for facial micro-expression recognition. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition. 667--674.Google Scholar
- Dae Hoe Kim, Wissam J Baddar, and Yong Man Ro. 2016. Micro-expression recognition with expression-state constrained spatio-temporal feature representations. In Proceedings of the ACM international conference on Multimedia. 382--386.Google ScholarDigital Library
- Thomas N Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In Proceedings of the International Conference on Learning Representations.Google Scholar
- Ying-Hsiu Lai and Shang-Hong Lai. 2018. Emotion-Preserving Representation Learning via Generative Adversarial Network for Multi-View Facial Expression Recognition. In Proceedings of the IEEE International Conference on Automatic Face Gesture Recognition. 263--270.Google ScholarCross Ref
- Junhyun Lee, Inyeop Lee, and Jaewoo Kang. 2019. Self-attention graph pooling. In Proceedings of the International Conference on Machine Learning.Google Scholar
- Guanbin Li, Xin Zhu, Yirui Zeng, Qing Wang, and Liang Lin. 2019. Semantic relationships guided representation learning for facial action unit recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 8594--8601.Google ScholarDigital Library
- Wei Li, Farnaz Abtahi, and Zhigang Zhu. 2017a. Action unit detection with region adaptation, multi-labeling learning and optimal temporal fusing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1841--1850.Google ScholarCross Ref
- Wei Li, Farnaz Abtahi, Zhigang Zhu, and Lijun Yin. 2017b. Eac-net: A region-based deep enhancing and cropping approach for facial action unit detection. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition. 103--110.Google ScholarCross Ref
- Xiaobai Li, Tomas Pfister, Xiaohua Huang, Guoying Zhao, and Matti Pietikainen. 2013. A Spontaneous Micro-expression Database: Inducement, collection and baseline. Proceedings of the IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, 1--6.Google ScholarCross Ref
- Feng Lin, Richang Hong, Wengang Zhou, and Houqiang Li. 2018. Facial Expression Recognition with Data Augmentation and Compact Feature Learning. In Proceedings of the IEEE International Conference on Image Processing. 1957--1961.Google ScholarCross Ref
- Zhilei Liu, Jiahui Dong, Cuicui Zhang, Longbiao Wang, and Jianwu Dang. 2020. Relation Modeling with Graph Convolutional Networks for Facial Action Unit Detection. In Proceedings of the International Conference on Multimedia Modeling. Springer, 489--501.Google ScholarCross Ref
- Ling Lo, Hong-Xia Xie, Hong-Han Shuai, and Wen-Huang Cheng. 2020. MER-GCN: Micro Expression Recognition Based on Relation Modeling with Graph Convolutional Network. arXiv:2004.08915 (2020).Google Scholar
- Andre Lopes, Edilson Aguiar, Alberto De Souza, and Thiago Oliveira-Santos. 2016. Facial Expression Recognition with Convolutional Neural Networks: Coping with Few Data and the Training Sample Order. Pattern Recognition, Vol. 61 (07 2016).Google Scholar
- Chen Ma, Li Chen, and Junhai Yong. 2019 a. AU R-CNN: Encoding expert prior knowledge into R-CNN for action unit detection. Neurocomputing, Vol. 355 (2019), 35--47.Google ScholarCross Ref
- Jiaxin Ma, Hao Tang, Wei-Long Zheng, and Bao-Liang Lu. 2019 b. Emotion Recognition using Multimodal Residual LSTM Network. In Proceedings of the ACM International Conference on Multimedia. 176--183.Google ScholarDigital Library
- Walied Merghani, Adrian K Davison, and Moi Hoon Yap. 2018. A review on facial micro-expressions analysis: datasets, features and metrics. arXiv:1805.02397 (2018).Google Scholar
- Mehdi Mirza and Simon Osindero. 2014. Conditional generative adversarial nets. arXiv:1411.1784 (2014).Google Scholar
- Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor. 2017. Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, Vol. 10, 1 (2017), 18--31.Google ScholarDigital Library
- Albert Pumarola, Antonio Agudo, Aleix M Martinez, Alberto Sanfeliu, and Francesc Moreno-Noguer. 2018. GANimation: Anatomically-aware Facial Animation from a Single Image. In Proceedings of the The European Conference on Computer Vision.Google ScholarCross Ref
- Zhaofan Qiu, Ting Yao, and Tao Mei. 2017. Learning spatio-temporal representation with pseudo-3d residual networks. In proceedings of the IEEE International Conference on Computer Vision. 5533--5541.Google ScholarCross Ref
- Sai Prasanna Teja Reddy, Surya Teja Karri, Shiv Ram Dubey, and Snehasis Mukherjee. 2019. Spontaneous facial micro-expression recognition using 3D spatiotemporal convolutional neural networks. In Proceedings of the IEEE International Joint Conference on Neural Networks. 1--8.Google ScholarCross Ref
- Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Proceedings of the Advances in neural information processing systems. 91--99.Google Scholar
- John See, Moi Hoon Yap, Jingting Li, Xiaopeng Hong, and Su-Jing Wang. 2018. Facial Micro-Expressions Grand Challenge 2018 Summary. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition. IEEE, 675--678.Google ScholarCross Ref
- Madhumita Takalkar, Min Xu, Qiang Wu, and Zenon Chaczko. 2018. A survey: facial micro-expression recognition. Multimedia Tools and Applications, Vol. 77, 15 (2018), 19301--19325.Google ScholarDigital Library
- Madhumita A Takalkar and Min Xu. 2017. Image Based Facial Micro-Expression Recognition Using Deep Learning on Small Datasets. In Proceedings of the International Conference on Digital Image Computing: Techniques and Applications. 1--7.Google ScholarCross Ref
- Y-I Tian, Takeo Kanade, and Jeffrey F Cohn. 2001. Recognizing action units for facial expression analysis. IEEE Transactions on pattern analysis and machine intelligence, Vol. 23, 2 (2001), 97--115.Google ScholarDigital Library
- Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. 2015. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision. 4489--4497.Google ScholarDigital Library
- Nguyen Van Quang, Jinhee Chun, and Takeshi Tokuyama. 2019. CapsuleNet for micro-expression recognition. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition. IEEE, 1--7.Google ScholarCross Ref
- Monu Verma, Santosh Kumar Vipparthi, Girdhari Singh, and Subrahmanyam Murala. 2020. LEARNet: Dynamic Imaging Network for Micro Expression Recognition. IEEE Transactions on Image Processing, Vol. 29 (2020), 1618--1627.Google ScholarCross Ref
- Chongyang Wang, Min Peng, Tao Bi, and Tong Chen. 2018. Micro-Attention for Micro-Expression Recognition. arXiv:1811.02360 (2018).Google Scholar
- S. Wang, W. Yan, X. Li, G. Zhao, C. Zhou, X. Fu, M. Yang, and J. Tao. 2015. Micro-Expression Recognition Using Color Spaces. IEEE Transactions on Image Processing, Vol. 24, 12 (2015), 6034--6047.Google ScholarDigital Library
- Wen-Jing Yan, Xiaobai Li, Su-Jing Wang, Guoying Zhao, Yong-Jin Liu, Yu-Hsin Chen, and Xiaolan Fu. 2014. CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation. PLOS ONE, Vol. 9, 1 (2014), e86041.Google ScholarCross Ref
- Wen-Jing Yan, Qi Wu, Yong-Jin Liu, Su-Jing Wang, and Xiaolan Fu. 2013. CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces. In Proceedings of the IEEE international conference and workshops on automatic face and gesture recognition. 1--7.Google Scholar
- Zhenbo Yu, Qinshan Liu, and Guangcan Liu. 2017. Deeper cascaded peak-piloted network for weak expression recognition. The Visual Computer (09 2017).Google Scholar
- Marcus Vinicius Zavarez, Rodrigo F Berriel, and Thiago Oliveira-Santos. 2017. Cross-Database Facial Expression Recognition Based on Fine-Tuned Deep Convolutional Network. In Proceedings of the SIBGRAPI Conference on Graphics, Patterns and Images. 405--412.Google ScholarCross Ref
- Feifei Zhang, Tianzhu Zhang, Qirong Mao, and Changsheng Xu. 2018. Joint Pose and Expression Modeling for Facial Expression Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3359--3368.Google ScholarCross Ref
- Guoying Zhao and Matti Pietikainen. 2007. Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE transactions on pattern analysis and machine intelligence, Vol. 29, 6 (2007), 915--928.Google Scholar
- Kaili Zhao, Wen-Sheng Chu, Fernando De la Torre, Jeffrey F Cohn, and Honggang Zhang. 2016. Joint patch and multi-label learning for facial action unit and holistic expression recognition. IEEE Transactions on Image Processing, Vol. 25, 8 (2016), 3931--3946.Google ScholarCross Ref
- Chuanlin Zhu, Xinyun Chen, Jianxin Zhang, Zhiying Liu, Zhen Tang, Yuting Xu, Didi Zhang, and Dianzhi Liu. 2017. Comparison of ecological micro-expression recognition in patients with depression and healthy individuals. Frontiers in behavioral neuroscience, Vol. 11 (2017), 199.Google Scholar
- Yaochen Zhu, Zhenzhong Chen, and Feng Wu. 2019. Multimodal Deep Denoise Framework for Affective Video Content Analysis. In Proceedings of the ACM International Conference on Multimedia. 130--138.Google ScholarDigital Library
Index Terms
- AU-assisted Graph Attention Convolutional Network for Micro-Expression Recognition
Recommendations
Micro-Expression Recognition with Expression-State Constrained Spatio-Temporal Feature Representations
MM '16: Proceedings of the 24th ACM international conference on MultimediaRecognizing spontaneous micro-expression in video sequences is a challenging problem. In this paper, we propose a new method of small scale spatio-temporal feature learning. The proposed learning method consists of two parts. First, the spatial features ...
Micro-expression recognition based on 3D flow convolutional neural network
AbstractMicro-expression recognition (MER) is a growing field of research which is currently in its early stage of development. Unlike conventional macro-expressions, micro-expressions occur at a very short duration and are elicited in a spontaneous ...
Analysis of Eyebrow Motion for Micro-Expression Recognition
ICIGP '22: Proceedings of the 2022 5th International Conference on Image and Graphics ProcessingFacial expression recognition is a field of emotion recognition that has received considerable attention over the last decades. However, most of the research has been focused on the recognition of macro-expressions that are generally posed and easy to ...
Comments