Elsevier

Cognition

Volume 121, Issue 3, December 2011, Pages 313-323
Cognition

Variability in photos of the same face

https://doi.org/10.1016/j.cognition.2011.08.001Get rights and content

Abstract

Psychological studies of face recognition have typically ignored within-person variation in appearance, instead emphasising differences between individuals. Studies typically assume that a photograph adequately captures a person’s appearance, and for that reason most studies use just one, or a small number of photos per person. Here we show that photographs are not consistent indicators of facial appearance because they are blind to within-person variability. Crucially, this within-person variability is often very large compared to the differences between people. To investigate variability in photos of the same face, we collected images from the internet to sample a realistic range for each individual. In Experiments 1 and 2, unfamiliar viewers perceived images of the same person as being different individuals, while familiar viewers perfectly identified the same photos. In Experiment 3, multiple photographs of any individual formed a continuum of good to bad likeness, which was highly sensitive to familiarity. Finally, in Experiment 4, we found that within-person variability exceeded between-person variability in attractiveness. These observations are critical to our understanding of face processing, because they suggest that a key component of face processing has been ignored. As well as its theoretical significance, this scale of variability has important practical implications. For example, our findings suggest that face photographs are unsuitable as proof of identity.

Highlights

► The same person can look very different in different photographs. ► Photos of the same person often look more different than photos of different people. ► This is true for judgements involving identity and judgements of attractiveness. ► No currently theories can account for within-person variability on this scale. ► Our findings suggest that photographs are not suitable as proof of identity.

Introduction

Theories of face recognition are based almost entirely on studies of photo recognition. In such studies, a person’s face may be represented by a single photograph that is repeated throughout the experiment (e.g. Dyer et al., 2006, Golarai et al., 2007, Gupta and Srinivasan, 2008, Mehl and Buchner, 2008, Tsukiura and Cabeza, 2011), or by a matched pair or set of photos that differ only in one respect, such as facial expression or viewpoint (e.g. D’Argembeau et al., 2003, Turati et al., 2008, Winston et al., 2004). Here we argue that equating photographs with faces perpetuates a serious misconstrual of the face recognition problem, leading to spurious findings and theorising that misses the core issue. By the same token, recasting the problem illuminates a clear remedial path. In the discussion we outline a promising approach to this.

The problem of face recognition is often presented as a problem of telling people apart. Given that all human faces share the same basic template (two eyes above a nose above a mouth), how are we able to distinguish among many thousands of individuals? This question is often addressed in the context of within-category discrimination (e.g. Bukach et al., 2006, McKone et al., 2007). Since this perspective emphasises sensitivity to differences between individuals, it encourages the traditional focus on between-person variability. Experimentally, this often reduces to between-photo variability, where each person is represented by a single photo. This substitution of photos for faces implies that a photograph adequately captures a person’s appearance, such that exposure to the snapshot is interchangeable with exposure to the face. The purpose of the present study is to challenge this idea. We show that a photograph is not a reliable indicator of facial appearance because it is blind to within-person variability. Crucially, this within-person variability is large compared with between-person variability. This is a transformative observation, not only for cognitive theories of face recognition, but also for face recognition in applied settings.

Face photographs sample three interacting layers of variation: The face itself undergoes non-rigid deformations – on the millisecond scale during muscular movement, and on the decade scale over ageing. Surface reflectance properties of the face are also affected by many factors, including cardiovascular activity in the short term, and general health in the longer term. Superimposed upon these face changes are lighting and other atmospheric changes, which vary with the ambient environment. Finally, image parameters such as resolution and depth of contrast depend on the characteristics of the camera. The interplay between these variables guarantees that no two photos of any face are the same. In practice, different photos of an individual vary greatly (see Fig. 1).

The photographs in Fig. 1 were not chosen to be especially variable. Indeed four of them are from current photo-identification documents. Notice that even this relatively modest range of variability is rarely admitted to the laboratory. The experimental convention is to minimise image variability, treating it as ‘noise’ that merely obscures the problem of interest. This creates a fundamental disjoint between the situation that we would like to understand and the situation that is studied in the lab. Within-person variability pervades face recognition in the real world, because no face casts the same image twice. The only exception to this is repetition of photographs, yet a great deal of experimental work is based solely on this artificial and anomalous case. Conversely, within-person variability has been almost entirely overlooked, and has never been examined in its own right.

It is worth considering some possible reasons why within-person variability has been so comprehensively ignored. Certainly, there is the pragmatic reason that it is much easier to present photographs in experiments than to present faces (and also somewhat easier to present one photograph of each face than to present more than one photograph of each face). However, previous face recognition research suggests a more psychologically interesting reason: perhaps within-person variability has never been directly addressed because we are simply unaware of its scale. Familiar face recognition is surprisingly robust, in the sense that we can recognise familiar faces over an enormously wide range of viewing conditions (e.g. Bruce, 1982, Burton et al., 1999). In cognitive terms, this corresponds to a many-to-one mapping of diverse input images onto a more abstractive representation of the individual’s face (e.g. a Face Recognition Unit in Bruce & Young’s 1986 framework). It is possible that this funnel-like connectivity attenuates sensitivity to variation in input, leading to underestimation of within-person variability in familiar faces. We return to this issue in the discussion.

In contrast to familiar face recognition, unfamiliar face recognition is surprisingly fragile. It can be disrupted by even superficial changes in the input image (Bruce, 1982, Burton et al., 1999, Megreya and Burton, 2006, Megreya and Burton, 2008). Perhaps less intuitively, this too may lead within-person variability to be underestimated. Outside of the psychology experiments, we seldom receive feedback on recognition errors. So if we encounter an unfamiliar person on one day, and then fail to recognise the same person on a later day, we can simply assume that the second sighting was of a different person. This is a reasonable interpretation in the absence of feedback, but it is an error arising from a narrow view of within-person variability. The data presented below highlight the very large discrepancy between the expected range of this variability and the actual range.

Interestingly, a number of recent studies have begun to uncover large variability in the face recognition ability of observers. Duchaine and Nakayama (2006), and Russell, Duchaine, and Nakayama (2009) have described groups of individuals at opposite ends of this spectrum. ‘Developmental prosopagnosics’ (Duchaine & Nakayama, 2006) have profound difficulty with face recognition, despite having otherwise intact visual abilities and no history of brain damage. In contrast, ‘Super-recognizers’ perform exceptionally well on a range of face recognition tasks (Russell et al., 2009). Megreya and Burton (2006) have reported large and stable individual differences for a number of face processing tasks, and recently Burton, White, and McNeill (2010) developed the Glasgow Face Matching Test (GFMT) as an instrument for assessing subjects’ ability to match unfamiliar faces. All of these studies point to substantial variability among perceivers. However, no theory yet addresses variability in the person perceived. We hope to persuade readers that within-person variability must be built into our theorising if the problem of face recognition is to be properly understood.

We begin in Studies 1 and 2 by using a photo sorting task to compare actual within-person variability with the expectations of naïve observers. In Study 3 we address the everyday notion of ‘good likeness’ and ‘bad likeness’ photographs by examining the distribution of likeness ratings both within individuals and between individuals. Finally, in Study 4 we turn to within-person variability in facial attractiveness. The overall message from these studies is that photographs are not stable representations of facial appearance. This is true for forensically important judgements of identity. It is also true for socially important judgements of attractiveness.

Section snippets

Experiment 1

The purpose of this experiment was to examine face matching in the context of realistic within-person variability. Our main interest was observers’ tolerance to this variability when matching photographs for identity. To investigate this, we developed a new sorting task using multiple photographs of different faces. In this task, participants are simply asked to group the photographs according to identity, so that different photos of the same person are gathered together. Participants are not

Experiment 2

In view of the very poor performance in Experiment 1, we next sought to rule out the possibility that the photo sets were inherently difficult for participants to process, perhaps due to poor image quality or biased sampling. To this end, we recruited 20 Dutch participants who were familiar with both of the faces shown in the task. If the images are identifiable in principle, then participants who are familiar with the faces should have no trouble sorting them correctly. On the other hand, if

Experiment 3

In the preceding experiments, ambient photos of an individual face were thought to depict different people, unless the observer was familiar with the face concerned. Given the image variability associated with each person, we next asked whether some photos capture identity better than others. To investigate this formally, we collected multiple images for a set of well-known celebrities, and asked participants to rate each photo for likeness (i.e., degree of resemblance to the depicted person).

Experiment 4

The preceding experiments demonstrate that photographs are not stable representations of facial identity. However, identity is just one of many signals that we read from the face. In this final experiment, we examined within-person variability for another socially significant signal – facial attractiveness. Previous studies of facial attractiveness have typically focused on biological variation between individuals (e.g., Perrett et al., 1998, Thornhill and Gangestad, 1999, Roberts et al., 2004,

General discussion

Four studies using ambient face photographs revealed unexpected within-person variability in appearance. In Experiment 1, photos of the same face were seen as different people. Experiment 2 confirmed that this was not due to unrepresentative photographs. In Experiment 3, multiple photographs of any individual formed a continuum of likeness, which was highly sensitive to familiarity level. Finally, in Experiment 4, within-person variability exceeded between-person variability in attractiveness:

Acknowledgements

This research was supported by an ESRC grant to Jenkins & Burton (RES-062-23-0549), and an ESRC grant to Burton & Jenkins (RES-000-22-2519). We thank Rachael Main for assistance in collecting data for Experiment 4. Our thanks also to three anonymous reviewers for helpful comments on an earlier version of this paper.

References (53)

  • A. Todorov et al.

    Understanding evaluation of faces on social dimensions

    Trends in Cognitive Sciences

    (2008)
  • T. Tsukiura et al.

    Remembering beauty: Roles of orbitofrontal and hippocampal regions in successful memory encoding of attractive faces

    NeuroImage

    (2011)
  • C. Turati et al.

    Newborns’ face recognition over changes in viewpoint

    Cognition

    (2008)
  • Blanz, V., & Vetter, T. (1999). A morphable model for the synthesis of 3D faces. In Proceedings of the 26th annual...
  • R.F. Bornstein

    Exposure and affect: Overview and metaanalysis of research, 1968–1987

    Psychological Bulletin

    (1989)
  • V. Bruce

    Changing faces: Visual and non-visual coding processes in face recognition

    British Journal of Psychology

    (1982)
  • V. Bruce

    Stability from variation: The case of face recognition the MD Vernon memorial lecture

    Quarterly Journal of Experimental Psychology

    (1994)
  • V. Bruce et al.

    Understanding face recognition

    British Journal of Psychology

    (1986)
  • Burton, A. M., Jenkins, R., & Schweinberger, S. R. (in press). Mental representations of familiar faces. British...
  • A.M. Burton et al.

    The Glasgow Face Matching Test

    Behavior Research Methods

    (2010)
  • A.M. Burton et al.

    Face recognition in poor quality video: Evidence from security surveillance

    Psychological Science

    (1999)
  • R. Clutterbuck et al.

    Exploring levels of face familiarity by using an indirect face-matching measure

    Perception

    (2002)
  • R. Clutterbuck et al.

    Matching as an index of face familiarity

    Visual Cognition

    (2004)
  • R. Clutterbuck et al.

    Demonstrating how unfamiliar faces become familiar using a face matching task

    European Journal of Cognitive Psychology

    (2005)
  • A. D’Argembeau et al.

    The effects of happy and angry expressions on identity and expression memory for unfamiliar faces

    Cognition & Emotion

    (2003)
  • K. Dion et al.

    What is beautiful is good

    Journal of Personality and Social Psychology

    (1972)
  • Cited by (452)

    View all citing articles on Scopus
    View full text