Skip to main content
SpringerLink
Log in

A Video-Based Facial Motion Tracking and Expression Recognition System

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

We proposed a facial motion tracking and expression recognition system based on video data. By a 3D deformable facial model, the online statistical model (OSM) and cylinder head model (CHM) were combined to track 3D facial motion in the framework of particle filtering. For facial expression recognition, a fast and efficient algorithm and a robust and precise algorithm were developed. With the first, facial animation and facial expression were retrieved sequentially. After that facial animation was obtained, facial expression was recognized by static facial expression knowledge learned from anatomical analysis. With the second, facial animation and facial expression were simultaneously retrieved to increase the reliability and robustness with noisy input data. Facial expression was recognized by fusing static and dynamic facial expression knowledge, the latter of which was learned by training a multi-class expressional Markov process using a video database. The experiments showed that facial motion tracking by OSM+CHM is more pose robust than that by OSM, and the facial expression score of the robust and precise algorithm is higher than those of other state-of-the-art facial expression recognition methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Ahlberg J (2002) Model-based coding: extraction, coding, and evaluation of face model parameters. Linköping University, Sweden

    Google Scholar 

  2. Anderson K, McOwan PW (2006) A real-time automated system for recognition of human facial expressions. IEEE Trans Syst Man Cybern B 36(1):96–105

    Article  Google Scholar 

  3. Bartlett M, Littlewort G, Lainscsek C (2004) Machine learning methods for fully automatic recognition of facial expressions and facial actions. ICSMC, 145-152

  4. Black MJ et al (1997) Recognizing facial expressions in image sequences using local parameterized models of image motion. Int J Comput Vis 25(1):23–28

    Article  Google Scholar 

  5. Blake A, Isard M (2000) Active contours. Springer, Berlin

    Google Scholar 

  6. Blanz V, Vetter T (2003) Face recognition based on fitting a 3D morphable model. IEEE Trans Pattern Anal Mach Intell 15(4):253–263

    Google Scholar 

  7. Cao C, Lin Y, Lin WS, Zhou K (2013) 3D shape regression for real-time facial animation. ACM Trans Graph 32(4):149–158

    Article  MATH  Google Scholar 

  8. Cascia ML, Sclaroff S, Athitsos V (2000) Fast, reliable head tracking under varying illumination: an approach based on registration of texture mapped 3D models. IEEE Trans Pattern Anal Mach Intell 22(4):322–336

    Article  Google Scholar 

  9. Chang Y, Hu C, Feris R, Turk M (2006) Manifold based analysis of facial expression. JIVC 24(6):605–614

    Google Scholar 

  10. Chang Y, Hu C, Turk M (2004) Probabilistic expression analysis on manifolds. CVPR, 520-527

  11. Chang Y, Vieira M, Turk M, Velho L (2005) Automatic 3D facial expression analysis in videos. IWAMFG, 293-307

  12. Cohen L, Sebe N, Garg A, Chen L, Huang T (2003) Facial expression recognition from video sequences: temporal and static modeling. Comput Vis Image Underst 91(1–2):160–187

    Article  Google Scholar 

  13. Cohn JF, Reed LI, Ambadar Z, Xiao J, Moriyama T (2004) Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior. ICSMC, 610-616

  14. Cohn JF, Schmidt KL (2004) The timing of facial motion in posed and spontaneous smiles. IJWMIP 2:1–12

    Google Scholar 

  15. Dornaika F, Davoine F (2006) On appearance based face and facial action tracking. IEEE Trans Circuits Syst Video Technol 16(9):1107–1124

    Article  Google Scholar 

  16. Dornaika F, Davoine F (2008) Simultaneous facial action tracking and expression recognition in the presence of head motion. Int J Comput Vis 76(3):257–281

    Article  Google Scholar 

  17. Ekman P, Friesen W, Hager J (2002) Facial action coding system: research nexus. Network Research Information, Salt Lake City

    Google Scholar 

  18. Fang T, Zhao X, Ocegueda O, Shah S, Kakadiaris I (2011) 3d facial expression recognition: a perspective on promises and challenges. ICAFGR, 603-610

  19. Fidaleo D et al (2005) An investigation of model bias in 3D face tracking. ICAMFG, 125-129

  20. Gokturk S et al (2001) A data-driven model for monocular face tracking. ICCV, 701–708

  21. Grassberger P (1997) The pruned-enriched rosenbluth method: simulations of theta polymers of chain length up to 1, 000, 000. PRE 56:3682–3693

    Article  Google Scholar 

  22. Guo G, Dyer CR (2005) Learning from examples in the small sample case: face expression recognition. IEEE Trans Syst Man Cybern B 35(3):477–488

    Article  Google Scholar 

  23. Hammal Z, Couvreur L, Caplier A (2007) Facial expression classification: an approach based on the fusion of facial deformations using the transferable belief model. IJAR, 542-567

  24. Hayat M, Bennamoun M, An S (2014) Learning non-linear reconstruction models for image set classification. CVPR, 1915-1922

  25. Hu YK, Wang ZF (2006) A low-dimensional illumination space representation of human faces for arbitrary lighting conditions. ICPR, 1147-1150

  26. Hu YK, Zheng Y, Wang ZF (2005) Reconstruction of 3D face from a single 2D image for face recognition. IWVSPETS, 217-222

  27. Jepson AD, Fleet DJ, El-Maraghi TF (2003) Robust online appearance models for visual tracking. IEEE Trans Pattern Anal Mach Intell 25(10):1296–1311

    Article  Google Scholar 

  28. Kotsia I, Pitas I (2007) Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans Image Process 16(1):172–187

    Article  MathSciNet  Google Scholar 

  29. Lee C, Elgammal A (2005) Facial expression analysis using nonlinear decomposable generative models. IWAMFG, 958-963

  30. Liao WK, Fidaleo D, Medioni G (2007) Integrating multiple visual cues for robust real-time 3d face tracking. AMFG, 109-123

  31. Liao WK, Medioni G (2008) 3D face tracking and expression inference from a 2D sequence using manifold learning. CVPR, 3597-3604

  32. Littlewort GC, Bartlett MS, Lee K (2007) Faces of pain: automated measurement of spontaneous facial expressions of genuine and posed pain. ICMI, 15-21

  33. Liu SC, Wang YT et al (2012) Video stabilization with a depth camera. CVPR, 89-95

  34. Lucey S, Ashraf AB, Cohn JF (2007) Investigating spontaneous facial action recognition through AAM representations of the face. In: Delac K, Grgic M (eds) Face recognition. I-Tech Education and Publishing, p 275-286

  35. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended Cohn-Kande dataset (CK+): a complete facial expression dataset for action unit and emotion-specified expression. Workshop on CVPR, 217-224

  36. Lui YM, Beveridge JR, Whitley LD (2010) Adaptive appearance model and condensation algorithm for robust face tracking. IEEE Trans Syst Man Cybern A 40(3):437–448

    Article  Google Scholar 

  37. Marks TK, Hershey JR, Movellan JR (2010) Tracking motion, deformation, and texture using conditionally gaussian processes. IEEE Trans Pattern Anal Mach Intell 32(2):348–363

    Article  Google Scholar 

  38. Matthews I, Xiao J, Baker S (2007) 2D vs. 3D deformable face models: representational power, construction, and real-time fitting. Int J Comput Vis 75(1):93–113

    Article  Google Scholar 

  39. Nordstrøm MM, Larsen M, Sierakowski J et al (2004) The IMM face database - an annotated dataset of 240 face images. Institute of Informatics and Mathematical Modelling, Technical University of Denmark, DTU, Technical Report

  40. North B, Blake A, Isard M, Rittscher J (2000) Learning and classification of complex dynamics. IEEE Trans Pattern Anal Mach Intell 22(9):1016–1034

    Article  Google Scholar 

  41. Pantic M, Bartlett MS (2007) Machine analysis of facial expressions. In: Delac K, Grgic M (eds). Face recognition. I-Tech Education and Publishing, p 377-416

  42. Pantic M, Patras I (2006) Dynamics of facial expression: recognition of facial actions and their temporal segments form face profile image sequences. IEEE Trans Syst Man Cybern B 36(2):433–449

    Article  Google Scholar 

  43. Pantic M, Rothkrantz LJM (2004) Facial action recognition for facial expression analysis from static face images. IEEE Trans Syst Man Cybern B 34(3):1449–1461

    Article  Google Scholar 

  44. Sandbach G, Zafeiriou S, Pantic M, Yin L (2012) Static and dynamic 3d facial expression recognition: a comprehensive survey. Image Vis Comput 30(10):683–697

    Article  Google Scholar 

  45. Schmidt K, Cohn J (2001) Dynamics of facial expression: normative characteristics and individual differences. ICME, 728-731

  46. Sebe N, Lew MS, Cohen I, Sun Y, Gevers T, Huang TS (2004) Authentic facial expression analysis. ICAFGR, 701-708

  47. Strom J (2002) Model-based head tracking and coding. Linköping University, Sweden

    MATH  Google Scholar 

  48. Sung J, Kanade T, Kim D (2008) Pose robust face tracking by combining active appearance models and cylinder head models. Int J Comput Vis 80(2):260–274

    Article  Google Scholar 

  49. Tang FQ, Deng B (2007) Facial expression recognition using AAM and local facial feature. ICNC, 632-635

  50. Tao H, Huang TS (1999) Explanation-based facial motion tracking using a piecewise Bezier volume deformation mode. CVPR, 611-617

  51. Tian Y, Kanade T, Cohn JF (2001) Recognizing action units for facial expression analysis. IEEE Trans Pattern Anal Mach Intell 23:97–115

    Article  Google Scholar 

  52. Tian YL, Kanade T, Cohn JF (2005) Facial expression analysis. In: Li SZ, Jain AK (Eds) Handbook of face recognition. Springer, p 247-276

  53. Vacchetti L, Lepetit V, Fua P (2004) Stable real-time 3D tracking using online and offline information. IEEE Trans Pattern Anal Mach Intell 26(10):1385–1391

    Article  Google Scholar 

  54. Valstar MF, Gunes H, Pantic M (2007) How to distinguish posed from spontaneous smiles using geometric features. ICMI, 38-45

  55. Valstar M, Pantic M, Ambadar Z, Cohn JF (2006) Spontaneous versus posed facial behavior: automatic analysis of brow actions. ICMI, 162-170

  56. Wang H, Ahuja N (2003) Facial expression decomposition. ICCV, 958-963

  57. Wang Y, Ai H, Wu B, Huang C (2004) Real time facial expression recognition with Adaboost. ICPR, 307-314

  58. Wang J, Yin L, Wei X, Sun Y (2006) 3D Facial expression recognition based on primitive surface feature distribution. CVPR, 1399-1406

  59. Wang Q, Zhang W, Tang X, Shum HY (2006) Real-time bayesian 3-D pose tracking. IEEE Trans Circuits Syst Video Technol 16(12):1533–1541

    Article  Google Scholar 

  60. Wen Z, Huang T (2003) Capturing subtle facial motions in 3d face tracking. ICCV, 1343-1350

  61. Whitehill J, Omlin CW (2006) Haar features for FACS AU recognition. ICAFGR, 217-222

  62. Xiao J, Moriyama T, Kanade T, Cohn JF (2003) Robust full motion recovery of head by dynamic templates and re-registration techniques. Int J Imaging Syst Technol 13(1):85–94

    Article  Google Scholar 

  63. Xiao J et al (2003) Robust full-motion recovery of head by dynamic templates and registration techniques. Int J Imaging Syst Technol 13(2):85–94

    Article  Google Scholar 

  64. Yin L, Chen X, Sun Y, Worm T, Reale M (2008) A high-resolution 3D dynamic facial expression database. ICAFGR, 958-963

  65. Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) A 3D facial expression database for facial behavior research. ICAFGR, 211-216

  66. Yu J, Wang Z-f (2014) 3D facial motion tracking by combining online appearance model and cylinder head model in particle filtering. Sci China Inf Sci 57(2):274–280

    Article  Google Scholar 

  67. Zeng Z, Fu Y, Roisman GI, Wen Z, Hu Y, Huang TS (2006) Spontaneous emotional facial expression detection. J Multimed 1(5):1–8

    Article  Google Scholar 

  68. Zeng ZH, Pantic MJ, Roisman GI, Huang TS (2009) A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans Pattern Anal Mach Intell 31(1):31–58

    Google Scholar 

  69. Zhang Y, Ji Q (2005) Active and dynamic information fusion for facial expression understanding from image sequences. IEEE Trans Pattern Anal Mach Intell 27(5):699–714

    Article  Google Scholar 

  70. Zhang W, Wang Q, Tang XO (2008) Real time feature based 3-D deformable face tracking. ECCV, 720-732

  71. Zhou S, Chellappa R, Mogghaddam B (2004) Visual tracking and recognition using appearance-adaptive models in particle filters. IEEE Trans Image Process 13(11):1491–1506

    Article  Google Scholar 

  72. Zhou S, Krueger V, Chellappa R (2003) Probabilistic recognition of human faces from video. Comput Vis Image Underst 91(1–2):214–245

    Article  Google Scholar 

  73. Zhu Z, Ji Q (2006) Robust pose invariant facial feature detection and tracking in real time. ICPR, 1092-1095

  74. Zhu Z, Ji Q (2006) Robust real-time face pose and facial expression recovery. CVPR, 681-688

Download references

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No.61572450 and No.61303150), the Open Project Program of the State KeyLab of CAD&CG, Zhejiang University (No.A1501), the Fundamental Research Funds for the Central Universities (WK2350000002), the Open Funding Project of State Key Laboratory of Virtual Reality Technology and Systems, Beihang University (No. BUAA-VR-16KF-12).

Author information

Authors and Affiliations

  1. National Laboratory of Speech and Language Information Processing, University of Science and Technology of China, Hefei, 230027, China

    Jun Yu & Zengfu Wang

  2. State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, People’s Republic of China

    Jun Yu & Zengfu Wang

Authors
  1. Jun Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zengfu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jun Yu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yu, J., Wang, Z. A Video-Based Facial Motion Tracking and Expression Recognition System. Multimed Tools Appl 76, 14653–14672 (2017). https://doi.org/10.1007/s11042-016-3883-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-016-3883-3

Keywords

Search

Navigation