Abstract
In recent years, facial expression recognition (FER) has becoming a growing topic in computer vision with promising applications on virtual reality and human–robot interaction. Due to the influence of illumination, individual differences, attitude variation, etc., facial expression recognition with robust accuracy in complex environment is still an unsolved problem. Meanwhile, with the wide use of social communication, massive data are uploaded to the Internet; the effective utilization of those data is still a challenge due to noisy label phenomenon in the study of FER. To resolve the above-mentioned problems, firstly, a double active layer-based CNN is established to recognize the facial expression with high accuracy by learning robust and discriminative features from the data, which could enhance the robustness of network. Secondly, an active incremental learning method was utilized to tackle the problem of using Internet data. During the training phase, a two-stage transfer learning method is explored to transfer the relative information from face recognition to FER task to alleviate the inadequate training data in deep convolution network. Besides, in order to make better use of facial expression data from Web site and further improve the FER accuracy, Unconstrained Facial Expression Database from Web site database is built in this paper. Extensive experiments performed on two public facial expression recognition databases FER 2013 and SFEW 2.0 have demonstrated that the proposed scheme outperforms the state-of-the-art methods, which could achieve 67.08% and 51.90%, respectively.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ahmed A (2016) Facial expression recognition using Gabor wavelet and artificial neural networks. Dissertation Sudan University of Science and Technology
Alsamirat MA, Al-Alem F, Al-Ayoub M, Jararweh Y, Gupta B (2019) Impact of digital fingerprint image quality on the fingerprint recognition accuracy. Multimed Tools Appl 78(3):3649–3688
Burkert P, Trier F, Afzal M (2015) Dexpression: deep convolutional neural network for expression recognition. arXiv preprint
Chen YS (2019) Performance identification in large-scale class data from advanced facets of computational intelligence and soft computing techniques. Int J High Perform Comput Netw 13(3):283–293
Chen J, Liu X, Tu P (2012) Person-specific expression recognition with transfer learning. In: Proceedings of IEEE international conference on image processing, pp 2621–2624
Ding H, Zhou S, Chellappa R (2017) Facenet2expnet: regularizing a deep face recognition net for expression recognition. In: Proceedings of the 12th IEEE international conference on automatic face and gesture recognition, pp 118–126
El-Latif AA, Abd-El-Atty B, Hossain MS, Rahman MA, Alamri A, Gupta BB (2018) Efficient quantum information hiding for remote medical image sharing. IEEE Access 6:21075–21083
Frénay B, Verleysen M (2014) Classification in the presence of label noise: a survey. IEEE Trans Neural Netw Learn Syst 25(5):845–869
Hai T, Le H, Thuy N (2015) Facial expression classification using artificial neural network and K-nearest neighbor. Int J Inf Technol Comput Sci 7(3):27–32
Jing W, Jiang T, Zhang X, Zhu L (2019) The optimization of speech recognition based on convolutional neural network. IJHPCN 13(2):222–231
Kumar P, Happy S, Routray A (2017) A real-time robust facial expression recognition system using HOG features. In: Proceedings of international conference on computing, analytics and security trends, pp 289–293
Kung S (2016) Facial expression recognition using optical flow and 3D HMM and human action recognition using cuboid and topic models. Oakland University, Oakland
Lecun Y, Boser B, Denker J (2008) Backpropagation applied to handwritten zip code recognition. Neural Comput 1(4):541–551
Levi G, Hassner T (2015) Emotion recognition in the wild via convolutional neural networks and mapped binary patterns. In: Proceedings of ACM on international conference on multimodal interaction, 2015, pp 503–510
Li J, Ban X, Yang G, Li Y, Wang Y (2019) Real-time human action recognition using depth motion maps and convolutional neural networks. Int J High Perform Comput Netw 13(3):312–320
Liu P, Han S, Meng Z (2014) Facial expression recognition via a boosted deep belief network. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 1805–1812
Liu M, Li S, Shan S (2015) AU-inspired deep networks for facial expression feature learning. Neurocomputing 159(2):126–136
Liu Y, Cao Y, Li Y (2016a) Facial expression recognition with PCA and LBP features extracting from active facial patches. In: Proceedings of IEEE international conference on real-time computing and robotics (RCAR), pp 368–373
Liu S, Chen X, Fan D (2016b) 3D smiling facial expression recognition based on SVM. In: Proceedings of IEEE international conference on mechatronics and automation, pp 1661–1666
Mohammadi M, Fatemizadeh E, Mahoor M (2014) PCA-based dictionary building for accurate facial expression recognition via sparse representation. J Vis Commun Image Represent 25(5):1082–1092
Mollahosseini A, Chan D, Mahoor MH (2015) Going deeper in facial expression recognition using deep neural networks. In: IEEE winter conference on applications of computer vision (WACV), pp 1–10. https://doi.org/10.1109/WACV.2016.7477450
Ng H, Nguyen V, Vonikakis V (2015) Deep learning for emotion recognition on small datasets using transfer learning. In: Proceedings of ACM international conference on multimodal interaction, 2015, pp 443–449
Pan S, Yang Q (2010) A survey on transfer learning. Proc IEEE Trans Knowl Data Eng 22(10):1345–1359
Sukhbaatar S, Fergus R (2015) Learning from noisy labels with deep neural networks. In: 3rd International conference on learning representations (ICLR) Workshop Track Proceedings, San Diego, CA, USA, May 7–9, 2015. https://dblp.org/rec/bib/journals/corr/SukhbaatarF14
Sumeet S, Singh S, Saini R (2016) Hardware accelerator for facial expression classification using linear SVM. In: Advances in signal processing and intelligent recognition systems, pp 39–50
Sun N, Chen Z, Day R (2016) Facial expression recognition using digitalized facial features based on active shape model. In: Proceedings of international conference on computer science engineering and applications, pp 39–46
Viola P, Jones M (2004) Robust real-time face detection. In: Eighth IEEE international conference on IEEE Xplore, p 747
Wang X, Liu A, Zhang S (2015) New facial expression recognition based on FSVM and KNN. Int J Light Electron Opt 126(21):3132–3134
Wang F, Xiang X, Liu C (2017) Transferring face verification nets to pain and expression regression. arXiv preprint
Wu X, He R, Sun Z (2015) A lightened CNN for deep face representation. Comput Sci 111–118. https://arxiv.org/abs/1511.02683
Yu Z, Zhang C (2015) Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on international conference on multimodal interaction, pp 435–442
Yu K, Wang Z, Zhuo L et al (2010) Harvesting web images for realistic facial expression recognition. In: International conference on digital image computing: techniques and applications, pp 516–521
Zhai Y, Liu J, Zeng J et al (2017) Deep convolutional neural network for facial expression recognition. In: International conference on image and graphics, pp 211–223
Zhang X (2015) Facial expression analysis via transfer learning. Dissertation University of Denver
Zhang Y, Zong Y, Wang H (2008) Facial expression recognition based on C-means and K-nearest neighbor algorithms. Caai Trans Intell Syst 3(1):57–61 (in Chinese)
Zhang B, Liu G, Xie G (2017) Facial expression recognition using LBP and LPQ based on Gabor wavelet transform. In: Proceedings of IEEE international conference on computer and communications, pp 365–369
Zhao Q, Zhang D, Lu H (2005) Supervised LLE in ICA space for facial expression recognition. In: Proceedings of international conference on neural networks and brain, pp 1970–1975
Zhou Y, Xu T, Zheng W et al (2017) Classification and recognition approaches of tomato main organs based on DCNN. Trans Chin Soc Agric Eng 33(15):219–226
Zhou Z, Shin J, Gurudu S et al (2018) AFT*: integrating active learning and transfer learning to reduce annotation efforts. arXiv preprint 2018
Zhu R, Zhang T, Zhao Q (2015) A transfer learning approach to cross-database facial expression recognition. In: International conference on biometrics, pp 293–298
Acknowledgements
This work is supported by National Natural Science Foundation of China (No. 61771347); Characteristic Innovation Project of Guangdong Province (No. 2017KTSCX181); Young Innovative Talents Project of Guangdong Province (2017KQNCX206); Jiangmen Science and Technology Project ([2017] No. 268); 2017 Guangdong Science and Technology Plan Project (No. 2017A010101019); Youth Foundation of Wuyi University (No. 2015zk11); the Opening Project of GuangDong Province Key Laboratory of Information Security Technology (Grant No. 2017B030314131); the 2018 Opening Project of GuangDong Province Key Laboratory of Digital Signal and Image Processing.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
All authors declare that he/she has no conflict of interest.
Ethical approval
This article does not contain any studies with human participants or animals performed by any of the authors.
Additional information
Communicated by B. B. Gupta.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Xu, Y., Liu, J., Zhai, Y. et al. Weakly supervised facial expression recognition via transferred DAL-CNN and active incremental learning. Soft Comput 24, 5971–5985 (2020). https://doi.org/10.1007/s00500-019-04530-1
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-019-04530-1