Skip to main content
SpringerLink
Log in

Micro-expression recognition based on 3D flow convolutional neural network

  • Original Article
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

Micro-expression recognition (MER) is a growing field of research which is currently in its early stage of development. Unlike conventional macro-expressions, micro-expressions occur at a very short duration and are elicited in a spontaneous manner from emotional stimuli. While existing methods for solving MER are largely non-deep-learning-based methods, deep convolutional neural network (CNN) has shown to work very well on such as face recognition, facial expression recognition, and action recognition. In this article, we propose applying the 3D flow-based CNNs model for video-based micro-expression recognition, which extracts deeply learned features that are able to characterize fine motion flow arising from minute facial movements. Results from comprehensive experiments on three benchmark datasets—SMIC, CASME/CASME II, showed a marked improvement over state-of-the-art methods, hence proving the effectiveness of our fairly easy CNN model as the deep learning benchmark for facial MER.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Boureau YL, Ponce J, LeCun Y (2010) A theoretical analysis of feature pooling in visual recognition. In: Proceedings of the 27th international conference on machine learning, pp 111–118

  2. Byeon YH, Kwak KC (2014) Facial expression recognition using 3d convolutional neural network. Int J Adv Comput Sci Appl 5(12):107–112

    Google Scholar 

  3. Ekman P, Friesen WV (1969) Nonverbal leakage and clues to deception. Psychiatry 32(1):88–106

    Article  Google Scholar 

  4. Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. Aistats 9:249–256

    Google Scholar 

  5. Glorot X, Bordes A, Bengio Y (2011) Deep sparse rectifier neural networks. In: Aistats, vol 15, p 275

  6. Goswami B, Chan C.H, Kittler J, Christmas B (2010) Local ordinal contrast pattern histograms for spatiotemporal, lip-based speaker authentication. In: 2010 fourth IEEE international conference on biometrics: theory applications and systems (BTAS). IEEE, pp 1–6

  7. Haggard E.A, Isaacs K.S (1966) Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. In: Methods of research in psychotherapy. Springer, pp 154–165

  8. He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385

  9. Hinton GE, Osindero S, Teh YW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554

    Article  MathSciNet  MATH  Google Scholar 

  10. Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinov RR (2012) Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580

  11. Huang X, Wang SJ, Zhao G, Piteikainen M (2015) Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection. In: Proceedings of the IEEE international conference on computer vision workshops, pp. 1–9

  12. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167

  13. Jaiswal S, Valstar M (2016) Deep learning the dynamic appearance and shape of facial action units. In: IEEE winter conference on applications of computer vision (WACV), pp 1–8

  14. Ji S, Xu W, Yang M, Yu K (2013) 3d convolutional neural networks for human action recognition. IEEE Trans Pattern Anal Mach Intell 35(1):221–231

    Article  Google Scholar 

  15. Jung H, Lee S, Yim J, Park S, Kim J (2015) Joint fine-tuning in deep neural networks for facial expression recognition. In: IEEE international conference on computer vision (ICCV)

  16. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  17. Le Ngo A.C, Phan R.C.W, See J (2014) Spontaneous subtle expression recognition: imbalanced databases and solutions. In: Asian conference on computer vision. Springer, pp 33–48

  18. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  19. Li X, Pfister T, Huang X, Zhao G, Pietikainen M A (2013) spontaneous micro-expression database: Inducement, collection and baseline. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG). IEEE, pp 1–6

  20. Liong S.T, See J, Phan R.C.W, Le Ngo A.C, Oh Y.H, Wong K (2014) Subtle expression recognition using optical strain weighted features. In: Asian conference on computer vision workshops, pp 644–657

  21. Oh YH, Le Ngo AC, Phari RCW, See J, Ling HC (2016) Intrinsic two-dimensional local structures for micro-expression recognition. In: IEEE international conference on acoustics, speech, and signal processing (ICASSP), pp 1851–1855

  22. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  23. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

  24. Wang Y, See J, Oh YH, Phan RCW, Rahulamathavan Y, Ling HC, Tan SW, Li X (2016) Effective recognition of facial micro-expressions with video motion magnification. Multimed Tools Appl. https://doi.org/10.1007/s11042-016-4079-6

  25. Wang Y, See J, Phan RCW, Oh YH (2015) Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition. PLoS ONE 10(5):e0124–674

    Google Scholar 

  26. Wang Y, See J, Phan RCW, Oh YH (2015) LBP with six intersection points: reducing redundant information in lbp-top for micro-expression recognition. In: Asian conference on computer vision, pp 525–537

  27. Xu F, Zhang J, Wang J (2016) Microexpression identification and categorization using a facial dynamics map. IEEE Trans Affect Comput 8(2):254–267

    Article  Google Scholar 

  28. Yan WJ, Li X, Wang SJ, Zhao G, Liu YJ, Chen YH, Fu X (2014) Casme II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9(1):e86041

    Article  Google Scholar 

  29. Yan WJ, Wu Q, Liu YJ, Wang SJ, Fu X (2013) Casme database: a dataset of spontaneous micro-expressions collected from neutralized faces. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG). IEEE, pp 1–7

  30. Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: Computer vision–ECCV 2014. Springer, pp 818–833

  31. Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928

    Article  Google Scholar 

  32. Zhou Z, Zhao G, Guo Y, Pietikainen M (2012) An image-based visual speech animation system. IEEE Trans Circ Syst Video Technol 22(10):1420–1432

    Article  Google Scholar 

Download references

Acknowledgements

We gratefully acknowledge the support of NVIDIA Corporation for the donation of a Quadro K5200 GPU used in this work.

Author information

Authors and Affiliations

  1. Faculty of Computer Science and Technology, Wenzhou University, Wenzhou, 325035, Zhejiang, China

    Jing Li, Yandan Wang & Wenbin Liu

  2. Faculty of Computing and Informatics, Multimedia University, Persiaran Multimedia, 63100, Cyberjaya, Selangor, Malaysia

    John See

Authors
  1. Jing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yandan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. John See
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wenbin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yandan Wang.

Additional information

This work is funded in part by the Zhejiang Provincial Natural Science Foundation of China (Grants Nos. LQ17F020002, R1110261) and the National Science Foundation of China (Grants Nos. 61572367, 61272018)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, J., Wang, Y., See, J. et al. Micro-expression recognition based on 3D flow convolutional neural network. Pattern Anal Applic 22, 1331–1339 (2019). https://doi.org/10.1007/s10044-018-0757-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-018-0757-5

Keywords

Search

Navigation