Skip to main content
Log in

Benchmarking commercial emotion detection systems using realistic distortions of facial image datasets

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Currently, there are several widely used commercial cloud-based services that attempt to recognize an individual’s emotions based on their facial expressions. Most research into facial emotion recognition has used high-resolution, front-oriented, full-face images. However, when images are collected in naturalistic settings (e.g., using smartphone’s frontal camera), these images are likely to be far from ideal due to camera positioning, lighting conditions, and camera shake. The impact these conditions have on the accuracy of commercial emotion recognition services has not been studied in full detail. To fill this gap, we selected five prominent commercial emotion recognition systems—Amazon Rekognition, Baidu Research, Face++, Microsoft Azure, and Affectiva—and evaluated their performance via two experiments. In Experiment 1, we compared the systems’ accuracy at classifying images drawn from three standardized facial expression databases. In Experiment 2, we first identified several common scenarios (e.g., partially visible face) that can lead to poor-quality pictures during smartphone use, and manipulated the same set of images used in Experiment 1 to simulate these scenarios. We used the manipulated images to again compare the systems’ classification performance, finding that the systems varied in how well they handled manipulated images that simulate realistic image distortion. Based on our findings, we offer recommendations for developers and researchers who would like to use commercial facial emotion recognition technologies in their applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. https://aice.uva.nl/research-tools/adfes-stimulus-set/adfes-stimulus-set.html.

  2. http://www.rafd.nl/.

  3. http://www.emotional-face.org/.

References

  1. Affectiva: Home—Affectiva : Affectiva. https://www.affectiva.com/ (2019)

  2. Albohn, D.N., Adams Jr., R.B.: Social vision: at the intersection of vision and person perception. In: Cloutier, J., Absher, J.R. (eds.) Neuroimaging Personality, Social Cognition, and Character, pp. 159–186. Elsevier, Amsterdam (2016)

    Google Scholar 

  3. Alm, C.O., Roth, D., Sproat, R.: Emotions from text: machine learning for text-based emotion prediction. In: Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, pp. 579–586. Association for Computational Linguistics (2005)

  4. AlZoubi, O., Calvo, R.A., Stevens, R.H.: Classification of EEG for affect recognition: an adaptive approach. In: Australasian Joint Conference on Artificial Intelligence, pp. 52–61. Springer, Berlin (2009)

  5. Amazon: Amazon Rekognition-Video and Image-aws. https://aws.amazon.com/rekognition/?nc1=h_ls (2019)

  6. Bachmann, T.: Identification of spatially quantised tachistoscopic images of faces: how many pixels does it take to carry identity? Eur. J. Cogn. Psychol. 3(1), 87–103 (1991)

    Google Scholar 

  7. Baidu: Baidu ai. https://ai.baidu.com/docs#/Face-Detect-V3/top (2019)

  8. Barrett, L.F., Adolphs, R., Marsella, S., Martinez, A.M., Pollak, S.D.: Emotional expressions reconsidered: challenges to inferring emotion from human facial movements. Psychol. Sci. Public Interest 20(1), 1–68 (2019)

    Google Scholar 

  9. Bartlett, M.S., Littlewort, G., Fasel, I., Movellan, J.R.: Real time face detection and facial expression recognition: Development and applications to human computer interaction. In: 2003 Conference on Computer Vision and Pattern Recognition Workshop, vol. 5, pp. 53–53. IEEE (2003)

  10. Bernin, A., Müller, L., Ghose, S., von Luck, K., Grecos, C., Wang, Q., Vogt, F.: Towards more robust automatic facial expression recognition in smart environments. In: Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments, pp. 37–44. ACM (2017)

  11. Berretti, S., Amor, B.B., Daoudi, M., Del Bimbo, A.: 3d facial expression recognition using sift descriptors of automatically detected keypoints. Vis. Comput. 27(11), 1021 (2011)

    Google Scholar 

  12. Berretti, S., Del Bimbo, A., Pala, P.: Automatic facial expression recognition in real-time from dynamic sequences of 3d face scans. Vis. Comput. 29(12), 1333–1350 (2013)

    Google Scholar 

  13. Bettadapura, V.: Face expression recognition and analysis: the state of the art. arXiv preprint arXiv:1203.6722 (2012)

  14. Bourel, F., Chibelushi, C.C., Low, A.A.: Robust facial expression recognition using a state-based model of spatially-localised facial dynamics. In: Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition, pp. 113–118. IEEE (2002)

  15. Boyat, A.K., Joshi, B.K.: A review paper: noise models in digital image processing. arXiv preprint arXiv:1505.03489 (2015)

  16. Bröhl, C., Mertens, A., Ziefle, M.: How do users interact with mobile devices? an analysis of handheld positions for different technology generations. In: International Conference on Human Aspects of IT for the Aged Population, pp. 3–16. Springer, Berlin (2017)

  17. Bryant, D., Howard, A.: A comparative analysis of emotion-detecting AI systems with respect to algorithm performance and dataset diversity. In: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 377–382. ACM (2019)

  18. Calvo, R.A., D’Mello, S.: Affect detection: an interdisciplinary review of models, methods, and their applications. IEEE Trans. Affect. Comput. 1(1), 18–37 (2010)

    Google Scholar 

  19. Carlotta Olivetti, E., Violante, M.G., Vezzetti, E., Marcolin, F., Eynard, B.: Engagement evaluation in a virtual learning environment via facial expression recognition and self-reports: a preliminary approach. Appl. Sci. 10(1), 314 (2020)

    Google Scholar 

  20. Carroll, J.M., Russell, J.A.: Do facial expressions signal specific emotions? judging emotion from the face in context. J. Personal. Soc. Psychol. 70(2), 205 (1996)

    Google Scholar 

  21. Celma, O.: Music recommendation. In: Gerstner, R. (ed.) Music Recommendation and Discovery, pp. 43–85. Springer, Berlin (2010)

  22. Chang, W.Y., Hsu, S.H., Chien, J.H.: Fatauva-net: an integrated deep learning framework for facial attribute recognition, action unit detection, and valence-arousal estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 17–25 (2017)

  23. Chen, Y., McBain, R., Norton, D.: Specific vulnerability of face perception to noise: a similar effect in schizophrenia patients and healthy individuals. Psychiatry Res. 225(3), 619–624 (2015)

    Google Scholar 

  24. Cheng, Y., Jiang, B., Jia, K.: A deep structure for facial expression recognition under partial occlusion. In: 2014 Tenth International Conference on Intelligent Information Hiding and Multimedia Signal Processing, pp. 211–214. IEEE (2014)

  25. Coulson, M.: Attributing emotion to static body postures: recognition accuracy, confusions, and viewpoint dependence. J. Nonverbal Behav. 28(2), 117–139 (2004)

    MathSciNet  Google Scholar 

  26. Darwin, C.: The Expression of the Emotions in Man and Animals, Anniversary edn. Harper Perennial, London (1872). (P. Ekman, ed)

    Google Scholar 

  27. Dodge, S., Karam, L.: Understanding how image quality affects deep neural networks. In: 2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX), pp. 1–6. IEEE (2016)

  28. Dodge, S., Karam, L.: Can the early human visual system compete with deep neural networks? In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2798–2804 (2017)

  29. Dodge, S., Karam, L.: A study and comparison of human and deep learning recognition performance under visual distortions. In: 2017 26th International Conference on Computer Communication and Networks (ICCCN), pp. 1–7. IEEE (2017)

  30. Dupré, D., Andelic, N., Morrison, G., McKeown, G.: Accuracy of three commercial automatic emotion recognition systems across different individuals and their facial expressions. In: 2018 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 627–632. IEEE (2018)

  31. Ekman, P.: Methods for measuring facial action. In: Scherer, K.R., Ekman, P. (eds.) Handbook of Methods in Nonverbal Behavior Research, pp. 45–90. Cambridge University Press, Cambridge (1982)

    Google Scholar 

  32. Ekman, P.: Expression and the nature of emotion. Approaches Emot. 3, 19–344 (1984)

    Google Scholar 

  33. Ekman, P.: An argument for basic emotions. Cogn. Emot. 6(3–4), 169–200 (1992)

    Google Scholar 

  34. Ekman, P.: Basic emotions. In: Dalgleish, T., Power, M.J. (eds.) Handbook of Cognition and Emotion, vol. 98, pp. 45–60. Wiley, New York (1999)

    Google Scholar 

  35. Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Personal. Soc. Psychol. 17(2), 124 (1971)

    Google Scholar 

  36. Ekman, P., Friesen, W.V.: Unmasking the Face: A Guide to Recognizing Emotions from Facial Clues. ISHK, Los Altos (2003)

    Google Scholar 

  37. Ekman, P., Friesen, W.V., Ellsworth, P.: Emotion in the Human Face: Guide-Lines for Research and an Integration of Findings: Guidelines for Research and an Integration of Findings. Pergamon, Oxford (1972)

    Google Scholar 

  38. Ekman, P., Friesen, W.V., Hager, J.C.: Facial Action Coding System: The Manual on CD ROM, pp. 77–254. A Human Face, Salt Lake City (2002)

    Google Scholar 

  39. Ekman, P., Friesen, W.V., O’sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., Krause, R., LeCompte, W.A., Pitcairn, T., Ricci-Bitti, P.E., et al.: Universals and cultural differences in the judgments of facial expressions of emotion. J. Personal. Soc. Psychol. 53(4), 712 (1987)

    Google Scholar 

  40. El Ayadi, M., Kamel, M.S., Karray, F.: Survey on speech emotion recognition: features, classification schemes, and databases. Pattern Recognit. 44(3), 572–587 (2011)

    MATH  Google Scholar 

  41. Fridlund, A.J.: Evolution and facial action in reflex, social motive, and paralanguage. Biol. Psychol. 32(1), 3–100 (1991)

    Google Scholar 

  42. Friesen, E., Ekman, P.: Facial Action Coding System: A Technique for the Measurement of Facial Movement, vol. 3. Consulting Psychologists Press, Palo Alto (1978)

    Google Scholar 

  43. Friesen, W.V., Ekman, P., et al.: Emfacs-7: emotional facial action coding system, vol. 2, no. 36, p. 1. Unpublished manuscript, University of California at San Francisco (1983)

  44. Gedraite, E.S., Hadad, M.: Investigation on the effect of a Gaussian blur in image filtering and segmentation. In: Proceedings ELMAR-2011, pp. 393–396. IEEE (2011)

  45. Gellman, M.D.: Behavioral Medicine. Springer, Berlin (2013)

    Google Scholar 

  46. Goeleven, E., De Raedt, R., Leyman, L., Verschuere, B.: The Karolinska directed emotional faces: a validation study. Cogn. Emot. 22(6), 1094–1118 (2008)

    Google Scholar 

  47. Goncalves, J., Pandab, P., Ferreira, D., Ghahramani, M., Zhao, G., Kostakos, V.: Projective testing of diurnal collective emotion. In: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp’ 14, pp. 487–497. New York, NY, USA (2014)

  48. Gong, B., Wang, Y., Liu, J., Tang, X.: Automatic facial expression recognition on a single 3d face by exploring shape deformation. In: Proceedings of the 17th ACM International Conference on Multimedia, pp. 569–572 (2009)

  49. Google: Vision AI-derive image insights via ml-cloud vision api-google cloud. https://cloud.google.com/vision/ (2019)

  50. Gross, J.J.: Emotion regulation: past, present, future. Cogn. Emot. 13(5), 551–573 (1999)

    Google Scholar 

  51. Gu, Y., Li, X., Huang, K., Fu, S., Yang, K., Chen, S., Zhou, M., Marsic, I.: Human conversation analysis using attentive multimodal networks with hierarchical encoder–decoder. In: 2018 ACM Multimedia Conference on Multimedia Conference, pp. 537–545. ACM (2018)

  52. Gu, Y., Yang, K., Fu, S., Chen, S., Li, X., Marsic, I.: Hybrid attention based multimodal network for spoken language classification. In: Proceedings of the Conference. Association for Computational Linguistics. Meeting, vol. 2018, pp. 2379–2390. NIH Public Access (2018)

  53. Gu, Y., Yang, K., Fu, S., Chen, S., Li, X., Marsic, I.: Multimodal affective analysis using hierarchical attention strategy with word-level alignment. arXiv preprint arXiv:1805.08660 (2018)

  54. Heraz, A., Frasson, C.: Predicting the three major dimensions of the learner’s emotions from brainwaves. Int. J. Comput. Sci. 2(3), 187–193 (2007)

    Google Scholar 

  55. Hou, L., Ji, H., Shen, Z.: Recovering over-/underexposed regions in photographs. SIAM J. Imaging Sci. 6(4), 2213–2235 (2013)

    MathSciNet  MATH  Google Scholar 

  56. Howard, A., Zhang, C., Horvitz, E.: Addressing bias in machine learning algorithms: a pilot study on emotion recognition for intelligent systems. In: 2017 IEEE Workshop on Advanced Robotics and Its Social Impacts (ARSO), pp. 1–7. IEEE (2017)

  57. Huang, D., De la Torre, F.: Bilinear kernel reduced rank regression for facial expression synthesis. In: European Conference on Computer Vision, pp. 364–377. Springer, Berlin (2010)

  58. Izard, C.E.: The Face of Emotion. Appleton-Century Crofts, New York (1971)

    Google Scholar 

  59. Jack, R.E., Garrod, O.G., Yu, H., Caldara, R., Schyns, P.G.: Facial expressions of emotion are not culturally universal. Proc. Natl. Acad. Sci. 109(19), 7241–7244 (2012)

    Google Scholar 

  60. Keltner, D., Ekman, P., Gonzaga, G., Beer, J.: Facial Expression of Emotion. Guilford Publications, New York (2000)

    Google Scholar 

  61. Khamis, M., Baier, A., Henze, N., Alt, F., Bulling, A.: Understanding face and eye visibility in front-facing cameras of smartphones used in the wild. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, p. 280. ACM (2018)

  62. Kheradpisheh, S.R., Ghodrati, M., Ganjtabesh, M., Masquelier, T.: Deep networks can resemble human feed-forward vision in invariant object recognition. Sci. Rep. 6, 32672 (2016)

    Google Scholar 

  63. Kim, Y., Lee, H., Provost, E.M.: Deep learning for robust feature generation in audiovisual emotion recognition. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 3687–3691. IEEE (2013)

  64. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

  65. Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D.H., Hawk, S.T., Van Knippenberg, A.: Presentation and validation of the radboud faces database. Cogn. Emot. 24(8), 1377–1388 (2010)

    Google Scholar 

  66. Le, H.V., Mayer, S., Wolf, K., Henze, N.: Finger placement and hand grasp during smartphone interaction. In: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pp. 2576–2584. ACM (2016)

  67. Lewinski, P., den Uyl, T.M., Butler, C.: Automated facial coding: validation of basic emotions and FACS AUs in facereader. J. Neurosci. Psychol. Econ. 7(4), 227 (2014)

    Google Scholar 

  68. Lewis, M., Haviland-Jones, J.M., Barrett, L.F.: Handbook of Emotions. Guilford Press, New York (2010)

    Google Scholar 

  69. Litman, D.J., Forbes-Riley, K.: Predicting student emotions in computer-human tutoring dialogues. In: Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, p. 351. Association for Computational Linguistics (2004)

  70. Liu, C., Freeman, W.T., Szeliski, R., Kang, S.B.: Noise estimation from a single image. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), vol. 1, pp. 901–908. IEEE (2006)

  71. Ma, C., Osherenko, A., Prendinger, H., Ishizuka, M.: A chat system based on emotion estimation from text and embodied conversational messengers. In: Proceedings of the 2005 International Conference on Active Media Technology, 2005. (AMT 2005), pp. 546–548. IEEE (2005)

  72. Maalej, A., Amor, B.B., Daoudi, M., Srivastava, A., Berretti, S.: Local 3d shape analysis for facial expression recognition. In: 2010 20th International Conference on Pattern Recognition, pp. 4129–4132. IEEE (2010)

  73. Mao, X., Xue, Y., Li, Z., Huang, K., Lv, S.: Robust facial expression recognition based on RPCA and AdaBoost. In: 2009 10th Workshop on Image Analysis for Multimedia Interactive Services, pp. 113–116. IEEE (2009)

  74. Matsumoto, D., Ekman, P.: Facial expression analysis. Scholarpedia 3(5), 4237 (2008)

    Google Scholar 

  75. Matsumoto, D., Keltner, D., Shiota, M.N., O’Sullivan, M., Frank, M.: Facial expressions of emotion. In: Lewis, M., Haviland-Jones, J.M., Barrett, L.F. (eds.) Handbook of Emotions, vol. 3, pp. 211–234. Guilford Press, New York (2008)

    Google Scholar 

  76. Matthews, O., Sarsenbayeva, Z., Jiang, W., Newn, J., Velloso, E., Clinch, S., Goncalves, J.: Inferring the mood of a community from their walking speed: a preliminary study. In: Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing, UbiComp’ 18, pp. 1144–1149 (2018)

  77. McDaniel, B., D’Mello, S., King, B., Chipman, P., Tapp, K., Graesser, A.: Facial features for affective state detection in learning environments. In: Proceedings of the Annual Meeting of the Cognitive Science Society, vol. 29 (2007)

  78. Microsoft: Face API—facial recognition software–microsoft azure. https://azure.microsoft.com/en-au/services/cognitive-services/face/ (2019)

  79. Narwekar, A.A., Girju, R.: Uiuc at semeval-2018 task 1: recognizing affect with ensemble models. In: Proceedings of The 12th International Workshop on Semantic Evaluation, pp. 377–384 (2018)

  80. Nelson, N.L., Russell, J.A.: Universality revisited. Emot. Rev. 5(1), 8–15 (2013)

    Google Scholar 

  81. Olszanowski, M., Pochwatko, G., Kuklinski, K., Scibor-Rylski, M., Lewinski, P., Ohme, R.K.: Warsaw set of emotional facial expression pictures: a validation study of facial display photographs. Front. Psychol. 5, 1516 (2015)

    Google Scholar 

  82. Opencv: Cascade Classifier Training. https://docs.opencv.org/3.1.0/dc/d88/tutorial_traincascade.html#gsc.tab=0

  83. Panigrahi, S.K., Gupta, S., Sahu, P.K.: Phases under Gaussian additive noise. In: 2016 International Conference on Communication and Signal Processing (ICCSP), pp. 1771–1776. IEEE (2016)

  84. Patton, R.: Software Testing. Pearson Education India, New Delhi (2006)

    Google Scholar 

  85. Poria, S., Cambria, E., Bajpai, R., Hussain, A.: A review of affective computing: from unimodal analysis to multimodal fusion. Inf. Fusion 37, 98–125 (2017)

    Google Scholar 

  86. Poria, S., Cambria, E., Gelbukh, A.: Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 2539–2544 (2015)

  87. Poria, S., Chaturvedi, I., Cambria, E., Hussain, A.: Convolutional MKL based multimodal emotion recognition and sentiment analysis. In: 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 439–448. IEEE (2016)

  88. Rodner, E., Simon, M., Fisher, R.B., Denzler, J.: Fine-grained recognition in the noisy wild: sensitivity analysis of convolutional neural networks approaches. arXiv preprint arXiv:1610.06756 (2016)

  89. Rodriguez, P., Cucurull, G., Gonzàlez, J., Gonfaus, J.M., Nasrollahi, K., Moeslund, T.B., Roca, F.X.: Deep pain: Exploiting long short-term memory networks for facial expression classification. IEEE Trans. Cybern. 2017. https://doi.org/10.1109/TCYB.2017.2662199

  90. Russell, J.A.: Is there universal recognition of emotion from facial expression? a review of the cross-cultural studies. Psychol. Bull. 115(1), 102 (1994)

    Google Scholar 

  91. Sander, D., Scherer, K.: Oxford Companion to Emotion and the Affective Sciences. Oxford University Press, Oxford (2014)

    Google Scholar 

  92. Sarsenbayeva, Z., Ferreira, D., van Berkel, N., Luo, C., Vaisanen, M., Kostakos, V., Goncalves, J.: Vision-based happiness inference: a feasibility case-study. In: Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp ’17, pp. 494–499. ACM, New York, NY, USA (2017)

  93. Sarsenbayeva, Z., Marini, G., van Berkel, N., Luo, C., Jiang, W., Yang, K., Wadley, G., Dingler, T., Kostakos, V., Goncalves, J.: Does smartphone use drive our emotions or vice versa? a causal analysis. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI’ 20, pp. 1–15. New York, NY, USA (2020)

  94. Schuller, B., Stadermann, J., Rigoll, G.: Affect-robust speech recognition by dynamic emotional adaptation. In: Proceedings of Speech Prosody 2006, Dresden (2006)

  95. Sharma, P., Esengönül, M., Khanal, S.R., Khanal, T.T., Filipe, V., Reis, M.J.: Student concentration evaluation index in an e-learning context using facial emotion analysis. In: International Conference on Technology and Innovation in Learning, Teaching and Education, pp. 529–538. Springer, Berlin (2018)

  96. Stöckli, S., Schulte-Mecklenbeck, M., Borer, S., Samson, A.C.: Facial expression analysis with affdex and facet: a validation study. Behav. Res. Methods 50(4), 1446–1460 (2018)

    Google Scholar 

  97. Swinton, R., El Kaliouby, R.: Measuring emotions through a mobile device across borders, ages, genders and more. In: Proceedings of the ESOMAR Congress, Atlanta, pp. 1–12 (2012)

  98. Tan, X., Triggs, W.: Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Trans. Image Process. 19(6), 1635–1650 (2010)

    MathSciNet  MATH  Google Scholar 

  99. Technology, M.: Emotion recognition—face++ AI open platform. https://www.faceplusplus.com/emotion-recognition/ (2019)

  100. Torralba, A., Fergus, R., Freeman, W.T.: 80 million tiny images: a large data set for nonparametric object and scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 30(11), 1958–1970 (2008)

    Google Scholar 

  101. Towner, H., Slater, M.: Reconstruction and recognition of occluded facial expressions using PCA. In: International Conference on Affective Computing and Intelligent Interaction, pp. 36–47. Springer, Berlin (2007)

  102. Useche, O., El-Sheikh, E.: An intelligent system framework for measuring attention levels of students in online course environments. In: Proceedings on the International Conference on Artificial Intelligence (ICAI), p. 452. The Steering Committee of The World Congress in Computer Science, Computer... (2015)

  103. Valstar, M.F., Jiang, B., Mehu, M., Pantic, M., Scherer, K.: The first facial expression recognition and analysis challenge. In: Face and Gesture 2011, pp. 921–926. IEEE (2011)

  104. van Berkel, N., Goncalves, J., Koval, P., Hosio, S., Dingler, T., Ferreira, D., Kostakos, V.: Context-informed scheduling and analysis: improving accuracy of mobile self-reports. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI’ 19 (2019)

  105. van der Haar, D.T.: Student emotion recognition in computer science education: a blessing or curse? In: International Conference on Human–Computer Interaction, pp. 301–311. Springer, Berlin (2019)

  106. Van Der Schalk, J., Hawk, S.T., Fischer, A.H., Doosje, B.: Moving faces, looking places: validation of the Amsterdam dynamic facial expression set (ADFES). Emotion 11(4), 907 (2011)

    Google Scholar 

  107. Violante, M.G., Marcolin, F., Vezzetti, E., Ulrich, L., Billia, G., Di Grazia, L.: 3d facial expression recognition for defining users’ inner requirements-an emotional design case study. Appl. Sci. 9(11), 2218 (2019)

    Google Scholar 

  108. Visuri, A., Sarsenbayeva, Z., Goncalves, J., Karapanos, E., Jones, S.: Impact of mood changes on application selection. In: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct, pp. 535–540. ACM (2016)

  109. Yuan, L., Sun, J., Quan, L., Shum, H.Y.: Image deblurring with blurred/noisy image pairs. ACM Trans. Graph. (TOG) 26(3), 1 (2007)

    Google Scholar 

  110. Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31(1), 39–58 (2008)

    Google Scholar 

  111. Zhang, L., Verma, B., Tjondronegoro, D., Chandran, V.: Facial expression analysis under partial occlusion: a survey. ACM Comput. Surv. (CSUR) 51(2), 25 (2018)

    Google Scholar 

  112. Zhang, Y., Chen, M., Huang, D., Wu, D., Li, Y.: idoctor: personalized and professionalized medical recommendations based on hybrid matrix factorization. Future Gener. Comput. Syst. 66, 30–35 (2017)

    Google Scholar 

Download references

Funding

This work is supported by the Australian Research Council (DP190102627).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Kangning Yang.

Ethics declarations

Conflict of interest

Authors, Kangning Yang, Chaofan Wang, Zhanna Sarsenbayeva, Benjamin Tag, Tilman Dingler, Greg Wadley, and Jorge Goncalves, declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, K., Wang, C., Sarsenbayeva, Z. et al. Benchmarking commercial emotion detection systems using realistic distortions of facial image datasets. Vis Comput 37, 1447–1466 (2021). https://doi.org/10.1007/s00371-020-01881-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-020-01881-x

Keywords

Navigation