Skip to main content
SpringerLink
Log in

Strategies of Face Recognition by Humans and Machines

  • Chapter
  • First Online:
Deep Learning-Based Face Analytics

Part of the book series: Advances in Computer Vision and Pattern Recognition ((ACVPR))

Abstract

Face recognition by machines has improved markedly over the last decade. Machines now perform some face recognition tasks at the level of untrained humans and forensic face identification experts. In this chapter, first we review recent work on human and machine performance on face recognition tasks. Second, we consider the benefits of statistically fusing human and machine responses to improve performance. Third, we review strategic differences in how humans with various levels of expertise approach face identification tasks. We conclude by considering the challenging problem of human and machine performance on recognition of faces of different races. Understanding how humans and machines perform these tasks can lead to more effective and accurate face recognition in applied settings.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Jeckeln G, Hahn CA, Noyes E, Cavazos JG, O’Toole AJ (2018) Wisdom of the social versus non-social crowd in face identification. B J Psychol

    Google Scholar 

  2. White D, Burton AM, Kemp RI, Jenkins R (2013) Crowd effects in unfamiliar face matching. Appl Cogn Psychol 27(6):769–777

    Google Scholar 

  3. White D, Phillips PJ, Hahn CA, Hill M, O’Toole AJ (2015) Perceptual expertise in forensic facial image comparison. Proc R Soc B 282(1814):20151292

    Article  Google Scholar 

  4. Phillips PJ, Yates AN, Hu Y, Hahn CA, Noyes E, Jackson K, Cavazos JG, Jeckeln G, Ranjan R, Sankaranarayanan S et al (2018) Face recognition accuracy of forensic examiners, superrecognizers, and face recognition algorithms. In: Proceedings of the National Academy of Sciences, p 201721355

    Google Scholar 

  5. O’Toole AJ, Abdi H, Jiang F, Phillips PJ (2007) Fusing face-verification algorithms and humans. IEEE Trans Syst Man Cybern Part B (Cybern) 37(5):1149–1155

    Article  Google Scholar 

  6. Furl N, Phillips PJ, O’Toole AJ (2002) Face recognition algorithms and the other-race effect: computational mechanisms for a developmental contact hypothesis. Cogn Sci 26(6):797–815

    Article  Google Scholar 

  7. Damasio AR, Damasio H, Van Hoesen GW (1982) Prosopagnosia anatomic basis and behavioral mechanisms. Neurology 32(4):331

    Article  Google Scholar 

  8. Russell R, Duchaine B, Nakayama K (2009) Super-recognizers: people with extraordinary face recognition ability. Psychon Bull Rev 16(2):252–257

    Article  Google Scholar 

  9. Young AW, Burton AM (2017) Are we face experts? trends in cognitive sciences

    Google Scholar 

  10. Johnston RA, Edmonds AJ (2009) Familiar and unfamiliar face recognition: a review. Memory 17(5):577–596

    Article  Google Scholar 

  11. Carl Metzgar CSP, A (2012) Strengthening forensic science in the us: a path forward. Prof Saf 57(1):32

    Google Scholar 

  12. FI, SW, Group et al (2012) Guidelines for facial comparison methods

    Google Scholar 

  13. Phillips PJ, Moon H, Rizvi SA, Rauss PJ (2000) The feret evaluation methodology for face-recognition algorithms. IEEE Trans Pattern Anal Mach Intell 22(10):1090–1104

    Article  Google Scholar 

  14. Phillips PJ, Flynn PJ, Scruggs T, Bowyer KW, Chang J, Hoffman K, Marques J, Min J, Worek W (2005) Overview of the face recognition grand challenge. In: IEEE computer society conference on Computer vision and pattern recognition CVPR 2005, vol 1. IEEE, pp 947–954

    Google Scholar 

  15. Phillips PJ, Scruggs WT, O’Toole AJ, Flynn PJ, Bowyer KW, Schott CL, Sharpe M (2010) Frvt 2006 and ice 2006 large-scale experimental results. IEEE Trans Pattern Anal Mach Intell 32(5):831–846

    Article  Google Scholar 

  16. Phillips PJ, Beveridge JR, Draper BA, Givens G, O’Toole AJ, Bolme D, Dunlop J, Lui YM, Sahibzada H, Weimer S (2012) The good, the bad, and the ugly face challenge problem. Image Vis Comput 30(3):177–185

    Article  Google Scholar 

  17. Phillips PJ, O’Toole AJ (2014) Comparison of human and computer performance across face recognition experiments. Image Vis Comput 32(1):74–85

    Google Scholar 

  18. Phillips PJ, Jiang F, Ayyad J, Pénard N et al (2007) Face recognition algorithms surpass humans matching faces over changes in illumination. IEEE Trans Pattern Anal Mach Intell 9:1642–1646

    Google Scholar 

  19. O’Toole AJ, An X, Dunlop J, Natu V, Phillips PJ (2012) Comparing face recognition algorithms to humans on challenging tasks. ACM Trans Appl Percept (TAP) 9(4):16

    Google Scholar 

  20. Sun Y, Wang X, Tang X (2014) Deep learning face representation from predicting 10,000 classes. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1891–1898

    Google Scholar 

  21. Sun Y, Wang X, Tang X (2013) Hybrid deep learning for face verification. In: Proceedings of the IEEE international conference on computer vision, pp 1489–1496

    Google Scholar 

  22. Parkhi OM, Vedaldi A, Zisserman A et al. (2015) Deep face recognition. BMVC 1(3):6

    Google Scholar 

  23. Chen J-C, Ranjan R, Kumar A, Chen C-H, Patel VM, Chellappa R (2015) An end-to-end system for unconstrained face verification with deep convolutional neural networks. In: Proceedings of the IEEE international conference on computer vision workshops, pp 118–126

    Google Scholar 

  24. Sankaranarayanan S, Alavi A, Castillo C, Chellappa R (2016) Triplet probabilistic embedding for face verification and clustering. arXiv:1604.05417

  25. Taigman Y, Yang M, Ranzato M, Wolf L (2014) Deepface: closing the gap to human-level performance in face verification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1701–1708

    Google Scholar 

  26. Schroff F, Kalenichenko D, Philbin J (2015) Facenet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 815–823

    Google Scholar 

  27. Huang GB, Mattar M, Berg T, Learned-Miller E (2008) Labeled faces in the wild: a database forstudying face recognition in unconstrained environments. In: Workshop on faces in ‘Real-Life’ images: detection, alignment, and recognition

    Google Scholar 

  28. Kumar N, Berg AC, Belhumeur PN, Nayar SK (2009) Attribute and simile classifiers for face verification. In: 2009 IEEE 12th international conference on computer vision, pp 365–372. IEEE

    Google Scholar 

  29. Whitelam C, Taborsky E, Blanton A, Maze B, Adams JC, Miller T, Kalka ND, Jain AK, Duncan JA, Allen K et al (2017) Iarpa janus benchmark-b face dataset. In: CVPR workshops, pp 592–600

    Google Scholar 

  30. Miller D, Brossard E, Seitz S, Kemelmacher-Shlizerman I (2015) Megaface: a million faces for recognition at scale. arXiv:1505.02108

  31. Kittler J (1998) Combining classifiers: a theoretical framework. Pattern Anal Appl 1(1):18–27

    Article  MathSciNet  Google Scholar 

  32. Condorcet MD (1785) Essay on the application of analysis to the probability of majority decisions. Paris: Imprimerie Royale

    Google Scholar 

  33. Dowsett AJ, Burton AM (2015) Unfamiliar face matching: pairs out-perform individuals and provide a route to training. B J Psychol 106(3):433–445

    Article  Google Scholar 

  34. Rice A, Phillips PJ, Natu V, An X, O’Toole AJ (2013) Unaware person recognition from the body when face identification fails. Psychol Sci 24(11):2235–2243

    Google Scholar 

  35. Hu Y, Jackson K, Yates A, White D, Phillips PJ, O’Toole AJ (2017) Person recognition: qualitative differences in how forensic face examiners and untrained people rely on the face versus the body for identification. Vis Cognit 25(4–6):492–506

    Article  Google Scholar 

  36. Czyz J, Kittler J, Vandendorpe L (2002) Combining face verification experts. In: 16th International conference on pattern recognition, 2002. Proceedings, vol 2. IEEE, pp. 28–31

    Google Scholar 

  37. Maurer D, Le Grand R, Mondloch CJ (2002) The many faces of configural processing. Trends Cognit Sci 6(6):255–260

    Article  Google Scholar 

  38. Yin RK (1969) Looking at upside-down faces. J Exp Psycholy 81(1):141

    Article  Google Scholar 

  39. Norell K, Läthén KB, Bergström P, Rice A, Natu V, O’Toole A (2015) The effect of image quality and forensic expertise in facial image comparisons. J Forensic Sci 60(2):331–340

    Article  Google Scholar 

  40. Malpass RS, Kravitz J (1969) Recognition for faces of own and other race. J Pers Soc Psychol 13(4):330

    Article  Google Scholar 

  41. Meissner CA, Brigham JC (2001) Thirty years of investigating the own-race bias in memory for faces: a meta-analytic review. Psychol Public Policy Law 7(1):3

    Article  Google Scholar 

  42. Sporer SL (2001) The cross-race effect: beyond recognition of faces in the laboratory. Psychol Public Policy Law 7(1):170

    Article  Google Scholar 

  43. Wells GL, Olson EA (2001) The other-race effect in eyewitness identification: what do we do about it? Psychol Public Policy Law 7(1):230

    Article  Google Scholar 

  44. Wilson JP, Hugenberg K, Bernstein MJ (2013) The cross-race effect and eyewitness identification: how to improve recognition and reduce decision errors in eyewitness situations. Soc Issues Policy Rev 7(1):83–113

    Article  Google Scholar 

  45. Pezdek K, Blandon-Gitlin I, Moore C (2003) Children’s face recognition memory: more evidence for the cross-race effect. J Appl Psychol 88(4):760

    Article  Google Scholar 

  46. Anzures G, Kelly DJ, Pascalis O, Quinn PC, Slater AM, De Viviés X, Lee K (2014) Own-and other-race face identity recognition in children: the effects of pose and feature composition. Dev Psychol 50(2):469

    Article  Google Scholar 

  47. Tham DSY, Bremner JG, Hay D (2017) The other-race effect in children from a multiracial population: a cross-cultural comparison. J Exp Psychol 155:128–137

    Google Scholar 

  48. Sangrigoli S, De Schonen S (2004) Recognition of own-race and other-race faces by three-month-old infants. J Child Psychol Psychiatry 45(7):1219–1227

    Article  Google Scholar 

  49. Kelly DJ, Quinn PC, Slater AM, Lee K, Ge L, Pascalis O (2007) The other-race effect develops during infancy: evidence of perceptual narrowing. Psychol Sci 18(12):1084–1089

    Article  Google Scholar 

  50. Pinkham AE, Sasson NJ, Calkins ME, Richard J, Hughett P, Gur RE, Gur RC (2008) The other-race effect in face processing among African American and Caucasian individuals with schizophrenia. Am J Psychiatry 165(5):639–645

    Article  Google Scholar 

  51. Wilson CE, Palermo R, Burton AM, Brock J (2011) Recognition of own-and other-race faces in autism spectrum disorders. Q J Exp Psychol 64(10):1939–1954

    Article  Google Scholar 

  52. Yi L, Quinn PC, Feng C, Li J, Ding H, Lee K (2015) Do individuals with autism spectrum disorder process own-and other-race faces differently? Vis Res 107:124–132

    Article  Google Scholar 

  53. Megreya AM, White D, Burton AM (2011) The other-race effect does not rely on memory: evidence from a matching task. Q J Exp Psychol 64(8):1473–1483

    Article  Google Scholar 

  54. Evans JR, Marcon JL, Meissner CA (2009) Cross-racial lineup identification: assessing the potential benefits of context reinstatement. Psychol Crime Law 15(1):19–28

    Article  Google Scholar 

  55. Jackiw LB, Arbuthnott KD, Pfeifer JE, Marcon JL, Meissner CA (2008) Examining the cross-race effect in lineup identification using caucasian and first nations samples. Canadian Journal of Behavioural Science/Revue canadienne des sciences du comportement 40(1):52

    Article  Google Scholar 

  56. Hugenberg K, Miller J, Claypool HM (2007) Categorization and individuation in the cross-race recognition deficit: toward a solution to an insidious problem. J Exp Soc Psychol 43(2):334–340

    Article  Google Scholar 

  57. Hayward WG, Favelle SK, Oxner M, Chu MH, Lam SM (2017) The other-race effect in face learning: using naturalistic images to investigate face ethnicity effects in a learning paradigm. Q J Exp Psychol 70(5):890–896

    Article  Google Scholar 

  58. Brigham JC, Ready DJ (1985) Own-race bias in lineup construction. Law Hum Behav 9(4):415–424

    Article  Google Scholar 

  59. Rodríguez J, Bortfeld H, Gutiérrez-Osuna R (2008) Reducing the other-race effect through caricatures. In: 8th IEEE international conference on automatic face & gesture recognition, 2008. FG’08. IEEE, pp. 1–5

    Google Scholar 

  60. Heron-Delaney M, Anzures G, Herbert JS, Quinn PC, Slater AM, Tanaka JW, Lee K, Pascalis O (2011) Perceptual training prevents the emergence of the other race effect during infancy. PLoS One 6(5):e19858

    Article  Google Scholar 

  61. Xiao WS, Fu G, Quinn PC, Qin J, Tanaka JW, Pascalis O, Lee K (2015) Individuation training with other-race faces reduces preschoolers-implicit racial bias: A link between perceptual and social representation of faces in children. Dev Sci 18(4):655–663

    Google Scholar 

  62. Tanaka JW, Pierce LJ (2009) The neural plasticity of other-race face recognition. Cognit Affect Behav Neurosci 9(1):122–131

    Article  Google Scholar 

  63. Matthews CM, Mondloch CJ (2018) Improving identity matching of newly encountered faces: effects of multi-image training. J Appl Res Mem Cognit 7(2):280–290

    Article  Google Scholar 

  64. Cavazos JG, Noyes E, O’Toole AJ (2018) Learning context and the other-race effect: strategies for improving face recognition. Vis Res

    Google Scholar 

  65. Sporer SL (2001) Recognizing faces of other ethnic groups: an integration of theories. Psychol Public Policy Law 7(1):36

    Article  Google Scholar 

  66. Hugenberg K, Young SG, Bernstein MJ, Sacco DF (2010) The categorization-individuation model: an integrative account of the other-race recognition deficit. Psychol Rev 117(4):1168

    Article  Google Scholar 

  67. Young SG, Hugenberg K, Bernstein MJ, Sacco DF (2012) Perception and motivation in face recognition: a critical review of theories of the cross-race effect. Person Soc Psychol Rev 16(2):116–142

    Article  Google Scholar 

  68. Diamond R, Carey S (1986) Why faces are and are not special: an effect of expertise. J Exp Psychol: Gen 115(2):107

    Article  Google Scholar 

  69. Rhodes G, Brake S, Taylor K, Tan S (1989) Expertise and configural coding in face recognition. B J Psychol 80(3):313–331

    Article  Google Scholar 

  70. Rhodes G, Hayward WG, Winkler C (2006) Expert face coding: configural and component coding of own-race and other-race faces. Psychon Bull Rev 13(3):499–505

    Article  Google Scholar 

  71. Tanaka JW, Kiefer M, Bukach CM (2004) A holistic account of the own-race effect in face recognition: evidence from a cross-cultural study. Cognition 93(1):B1–B9

    Article  Google Scholar 

  72. Bukach CM, Cottle J, Ubiwa J, Miller J (2012) Individuation experience predicts other-race effects in holistic processing for both caucasian and black participants. Cognition 123(2):319–324

    Article  Google Scholar 

  73. Sangrigoli S, Pallier C, Argenti A-M, Ventureyra V, de Schonen S (2005) Reversibility of the other-race effect in face recognition during childhood. Psychol Sci 16(6):440–444

    Google Scholar 

  74. Bodenhausen GV, Macrae CN, Hugenberg K (2003) Activating and inhibiting social identities: implications for perceiving the self and others. In: Foundations of social cognition: a festschrift in honor of Robert S. Jr. Erlbaum, Wyer

    Google Scholar 

  75. Millon T, Lerner MJ, Weiner IB (2003) Handbook of psychology: personality and social psychology. Wiley, New Jersey

    Google Scholar 

  76. O’Toole A. J, Deffenbacher K, Abdi H, Bartlett JC (1991) Simulating the “other-race effect” as a problem in perceptual learning. Connect Sci 3(2):163–178

    Google Scholar 

  77. Phillips PJ, Jiang F, Narvekar A, Ayyad J, O’Toole AJ (2011) An other-race effect for face recognition algorithms. ACM Trans Appl Percept (TAP) 8(2):14

    Google Scholar 

  78. Klare BF, Burge MJ, Klontz JC, Bruegge RW, Jain AK (2012) Face recognition performance: role of demographic information. IEEE Trans Inf Forensics Sec 7(6):1789–1801

    Google Scholar 

  79. Grother P, Ngan M, Hanaoka K (2019) Face recognition vendor test (frvt) part 3: demographic effects. National Institute of Standards and Technology

    Google Scholar 

  80. Cavazos JG, Phillips PJ, Castillo CD, O’Toole AJ (2020) Accuracy comparison across face recognition algorithms: where are we on measuring race bias? IEEE transactions on biometrics, behavior, and identity science

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. The University of Texas at Dallas, Dallas, USA

    Jacqueline G. Cavazos, Géraldine Jeckeln, Ying Hu & Alice J. O’Toole

Authors
  1. Jacqueline G. Cavazos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Géraldine Jeckeln
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ying Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alice J. O’Toole
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alice J. O’Toole .

Editor information

Editors and Affiliations

  1. Department of Computer Science and Engineering, University at Buffalo-SUNY, Buffalo, NY, USA

    Nalini K Ratha

  2. Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA

    Vishal M. Patel

  3. Departments of Electrical and Computer Engineering (Whiting School of Engineering) and Biomedical Engineering (School of Medicine), Johns Hopkins University, Baltimore, MD, USA

    Rama Chellappa

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Cavazos, J.G., Jeckeln, G., Hu, Y., O’Toole, A.J. (2021). Strategies of Face Recognition by Humans and Machines. In: Ratha, N.K., Patel, V.M., Chellappa, R. (eds) Deep Learning-Based Face Analytics. Advances in Computer Vision and Pattern Recognition. Springer, Cham. https://doi.org/10.1007/978-3-030-74697-1_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-74697-1_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-74696-4

  • Online ISBN: 978-3-030-74697-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation