Skip to main content
SpringerLink
Log in

3D facial expression recognition using SIFT descriptors of automatically detected keypoints

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Methods to recognize humans’ facial expressions have been proposed mainly focusing on 2D still images and videos. In this paper, the problem of person-independent facial expression recognition is addressed using the 3D geometry information extracted from the 3D shape of the face. To this end, a completely automatic approach is proposed that relies on identifying a set of facial keypoints, computing SIFT feature descriptors of depth images of the face around sample points defined starting from the facial keypoints, and selecting the subset of features with maximum relevance. Training a Support Vector Machine (SVM) for each facial expression to be recognized, and combining them to form a multi-class classifier, an average recognition rate of 78.43% on the BU-3DFE database has been obtained. Comparison with competitor approaches using a common experimental setting on the BU-3DFE database shows that our solution is capable of obtaining state of the art results. The same 3D face representation framework and testing database have been also used to perform 3D facial expression retrieval (i.e., retrieve 3D scans with the same facial expression as shown by a target subject), with results proving the viability of the proposed solution.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Berretti, S., Ben Amor, B., Daoudi, M., del Bimbo, A.: Person-independent 3D facial expression recognition by a selected ensemble of SIFT descriptors. In: Proceedings of the 3rd Eurographics/ACM SIGGRAPH Symposium on 3D Object Retrieval, Norrköping, Sweden, pp. 47–54 (2010)

    Google Scholar 

  2. Berretti, S., Del Bimbo, A., Pala, P.: 3D face recognition using iso-geodesic stripes. IEEE Trans. Pattern Anal. Mach. Intell. 32(12), 2162–2177 (2010)

    Article  Google Scholar 

  3. Chalechale, A.: Content-based retrieval from image databases using sketched queries. Ph.D. thesis (2005)

  4. Colombo, A., Cusano, C., Schettini, R.: 3D face detection using curvature analysis. Pattern Recognit. 39(3), 444–455 (2006)

    Article  MATH  Google Scholar 

  5. Do Carmo, M.P.: Differential Geometry of Curves and Surfaces. Prentice-Hall, Englewood Cliffs (1976)

    MATH  Google Scholar 

  6. Ekman, P.: Universals and cultural differences in facial expressions of emotion. In: Proceedings of the Nebraska Symposium on Motivation, Lincoln, NE, vol. 19, pp. 207–283 (1972)

    Google Scholar 

  7. Ekman, P., Friesen, W.V.: Manual for the Facial Action Coding System. Consulting Psychologist Press, Palo Alto (1977)

    Google Scholar 

  8. Farkas, L.G.: Anthropometry of the Head and Face. Raven Press, New York (1994)

    Google Scholar 

  9. Farkas, L.G., Munro, I.R.: Anthropometric Facial Proportions in Medicine. Thomas Books, Springfield (1987)

    Google Scholar 

  10. Gong, B., Wang, Y., Liu, J., Tang, X.: Automatic facial expression recognition on a single 3D face by exploring shape deformation. In: Proceedings of the ACM International Conference on Multimedia, Beijing, China, pp. 569–572 (2009)

    Google Scholar 

  11. Gupta, S., Markey, M.K., Bovik, A.C.: Anthropometric 3D face recognition. Int. J. Comput. Vis. 90(3), 331–349 (2010)

    Article  Google Scholar 

  12. Kakadiaris, I.A., Passalis, G., Toderici, G., Murtuza, N., Lu, Y., Karampatziakis, N., Theoharis, T.: Three-dimensional face recognition in the presence of facial expressions: an annotated deformable approach. IEEE Trans. Pattern Anal. Mach. Intell. 29(4), 640–649 (2007)

    Article  Google Scholar 

  13. Lowe, D.: Distinctive image features from scale-invariant key points. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  14. Maalej, A., Ben Amor, B., Daoudi, M., Srivastava, A., Berretti, S.: Local 3D shape analysis for facial expression recognition. In: Proceedings of the 20th International Conference on Pattern Recognition, Istanbul, Turkey, pp. 4129–4132 (2010)

    Chapter  Google Scholar 

  15. Maalej, A., Ben Amor, B., Daoudi, M., Srivastava, A., Berretti, S.: Shape analysis of local facial patches for 3D facial expression recognition. Pattern Recognit. 44(8), 1581–1589 (2011)

    Article  Google Scholar 

  16. Mayo, M., Zhang, E.: 3D face recognition using multiview key point matching. In: IEEE International Conference on Advanced Video and Signal Based Surveillance, Genoa, Italy, pp. 290–295 (2009)

    Chapter  Google Scholar 

  17. Mian, A.S., Bennamoun, M., Owens, R.: An efficient multimodal 2D–3D hybrid approach to automatic face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 29(11), 1927–1943 (2007)

    Article  Google Scholar 

  18. Mian, A.S., Bennamoun, M., Owens, R.: Keypoint detection and local feature matching for textured 3D face recognition. Int. J. Comput. Vis. 79(1), 1–12 (2008)

    Article  Google Scholar 

  19. Mpiperis, I., Malassiotis, S., Petridis, V., Strintzis, M.G.: 3D facial expression recognition using swarm intelligence. In: Proceedings of the IEEE International Conference on Acoustic, Speech, and Signal Processing, pp. 2133–2136 (2008)

    Chapter  Google Scholar 

  20. Mpiperis, I., Malassiotis, S., Strintzis, M.G.: Bilinear models for 3-D face and facial expression recognition. IEEE Trans. Inf. Forensics Secur. 3(3), 498–511 (2008)

    Article  Google Scholar 

  21. Ohbuchi, R., Furuya, T.: Scale-weighted dense bag of visual features for 3D model retrieval from a partial view 3D model. In: Proceedings of the Workshop on Search in 3D and Video, Kyoto, Japan (2009)

    Google Scholar 

  22. Pandzic, I., Forchheimer, R.: MPEG-4 Facial Animation: The Standard, Implementation and Applications. Wiley, New York (2005)

    Google Scholar 

  23. Peng, H., Long, F., Ding, C.: Feature selection based on mutual information: criteria of max-dependency max-relevance and min-redundancy. IEEE Trans. Pattern Anal. Mach. Intell. 27(8), 1226–1238 (2005)

    Article  Google Scholar 

  24. Queirolo, C.C., Silva, L., Bellon, O.R., Segundo, M.P.: 3D face recognition using simulated annealing and the surface interpenetration measure. IEEE Trans. Pattern Anal. Mach. Intell. 32(2), 206–219 (2010)

    Article  Google Scholar 

  25. Ramanathan, S., Kassim, A., Venkatesh, Y.V., Wah, W.S.: Human facial expression recognition using a 3D morphable model. In: Proceedings of the IEEE International Conference on Image Processing, Atlanta, GA, pp. 661–664 (2006)

    Google Scholar 

  26. Rodriguez, J.J., Aggarwal, J.K.: Matching aerial images to 3-D terrain maps. IEEE Trans. Pattern Anal. Mach. Intell. 12(12), 1138–1149 (1990)

    Article  Google Scholar 

  27. Samir, C., Srivastava, A., Daoudi, M., Klassen, E.: An intrinsic framework for analysis of facial surfaces. Int. J. Comput. Vis. 82(1), 80–95 (2009)

    Article  Google Scholar 

  28. Savran, A., Alyüz, N., Dibeklioǧlu, H., Çeliktutan, O., Gökberk, B., Sankur, B., Akarun, L.: Bosphorus database for 3D face analysis. In: Proceedings of the First COST 2101 Workshop on Biometrics and Identity Management (2008)

    Google Scholar 

  29. Soyel, H., Demirel, H.: Facial expression recognition using 3D facial feature distances. In: Proceedings of the International Conference on Image Analysis and Recognition, pp. 831–838 (2007)

    Google Scholar 

  30. Tang, H., Huang, T.S.: 3D facial expression recognition based on automatically selected features. In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Anchorage, AK, pp. 1–8 (2008)

    Google Scholar 

  31. Vedaldi, A., Fulkerson, B.: VLFeat: an open and portable library of computer vision algorithms (2008). http://www.vlfeat.org/

  32. Venkatesh, Y.V., Kassim, A.A., Murthy, O.V.R.: A novel approach to classification of facial expressions from 3D-mesh data sets using modified PCA. Pattern Recognit. Lett. 30(12), 1128–1137 (2009)

    Article  Google Scholar 

  33. Wang, J., Yin, L., Wei, X., Sun, Y.: 3D facial expression recognition based on primitive surface feature distribution. In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 1399–1406 (2006)

    Google Scholar 

  34. Yin, L., Wei, X., Sun, Y., Wang, J., Rosato, M.: A 3D facial expression database for facial behavior research. In: Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, Southampton, UK, pp. 211–216 (2006)

    Google Scholar 

  35. Zheng, W., Tang, H., Lin, Z., Huang, T.S.: A novel approach to expression recognition from non-frontal face images. In: Proceedings of the IEEE International Conference on Computer Vision, Kyoto, Japan, pp. 1901–1908 (2009)

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dipartimento di Sistemi e Informatica, University of Firenze, Firenze, Italy

    Stefano Berretti & Alberto del Bimbo

  2. Institut TELECOM, TELECOM Lille 1, LIFL (UMR 8022), Lille, France

    Boulbaba Ben Amor & Mohamed Daoudi

Authors
  1. Stefano Berretti
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Boulbaba Ben Amor
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohamed Daoudi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alberto del Bimbo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefano Berretti.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Berretti, S., Ben Amor, B., Daoudi, M. et al. 3D facial expression recognition using SIFT descriptors of automatically detected keypoints. Vis Comput 27, 1021–1036 (2011). https://doi.org/10.1007/s00371-011-0611-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-011-0611-x

Keywords

Search

Navigation