Multiple-image arrays in face matching tasks with and without memory

Abstract

Previous research has shown that exposure to within-person variability facilitates face learning. A different body of work has examined potential benefits of providing multiple images in face matching tasks. Viewers are asked to judge whether a target face matches a single face image (as when checking photo-ID) or multiple face images of the same person. The evidence here is less clear, with some studies finding a small multiple-image benefit, and others finding no advantage. In four experiments, we address this discrepancy in the benefits of multiple images from learning and matching studies. We show that multiple-image arrays only facilitate face matching when arrays precede targets. Unlike simultaneous face matching tasks, sequential matching and learning tasks involve memory and require abstraction of a stable representation of the face from the array, for subsequent comparison with a target. Our results show that benefits from multiple-image arrays occur only when this abstraction is required, and not when array and target images are available at once. These studies reconcile apparent differences between face learning and face matching and provide a theoretical framework for the study of within-person variability in face perception.

Introduction

We rely on faces to verify identity in a variety of situations ranging from buying alcohol to crossing borders. It is, therefore, important to understand how accurate we are at determining whether a photo-ID shows the person using it, and to identify potential ways to improve our performance in such tasks.

A large body of literature suggests that recognising familiar and unfamiliar faces entail some qualitatively different processes (Johnston & Edmonds, 2009; Megreya & Burton, 2006) and this could have serious practical implications. On the one hand, we are very good at recognising images of familiar identities even when these images are heavily distorted or degraded (e.g. Bruce, 1982, Bruce, 1986; Burton, Wilson, Cowan, & Bruce, 1999). On the other hand, recognition of unfamiliar identities is much poorer even with images taken on the same day or in the same session (e.g. Bruce et al., 1999; Burton, White, & McNeill, 2010). This stark contrast between familiar and unfamiliar faces has been demonstrated using many different tasks and paradigms including face memory, search and sorting tasks (Jenkins, White, Van Montfort, & Burton, 2011; Klatzky & Forrest, 1984; Kramer, Hardy, & Ritchie, 2020) as well as face matching tasks where typically two images are presented side-by-side on a computer screen, and participants are asked to judge whether the photos show the same person or different people (Bruce et al., 1999; Bruce, Henderson, Newman, & Burton, 2001; Clutterbuck and Johnston, 2002, Clutterbuck and Johnston, 2004; Megreya & Burton, 2008; Ritchie et al., 2015). While matching tasks have been generally used to approximate the process of checking photo-ID, the effect of familiarity has also been documented outside the lab with findings of poor performance when matching a live unfamiliar person to a photograph (Davis & Valentine, 2009; Kemp, Towell, & Pike, 1997; Megreya & Burton, 2008; Ritchie, Mireku, & Kramer, 2020). Moreover, all of this is true for many people who are employed to check photo-ID such as checkout assistants (Kemp et al., 1997), passport officers (White, Kemp, Jenkins, Matheson, & Burton, 2014) and police officers (Burton et al., 1999).

The difference between recognising familiar and unfamiliar faces has been attributed to the types of processing involved. We have seen the faces of familiar identities in a variety of contexts, situations and conditions, providing us with rich information about the many ways a single person might look. This way, we are able to isolate everything that is diagnostic of the person and discard any superficial image differences, leading to a more abstracted and image-independent processing for familiar faces. In Bruce and Young's influential model (Bruce and Young, 1986), familiar recognition is conceptualised through the use of Face Recognition Units (FRUs) which code structural information about known faces. FRUs must therefore store an abstracted, stable representation of a familiar person that is not influenced by simple image properties such as changes in head angle or expression.

Bruce (1994) first introduced the notion of stability from variation as a key familiarisation mechanism. Since then, a number of behavioural and computer modelling studies have shown that we can create and store stable representations of faces through exposure to within-person variability – that is multiple exposures to the same person showing naturally-occurring changes in their appearance. However, the same natural within-person variability that aids the recognition of familiar faces, can be detrimental to unfamiliar recognition which relies to a much greater extent on superficial image properties. This means that irrelevant differences in the physical properties of images or simple changes in clothing or accessories can be mistakenly regarded as evidence for differences in identity (Bindemann & Sandford, 2011; Graham & Ritchie, 2019; Kramer & Ritchie, 2016). In fact, recent research has suggested that the difference between familiar and unfamiliar face recognition may be due to our ability to use or tolerate within-person variability for familiar people (Burton, 2013; Burton, Jenkins, & Schweinberger, 2011; Burton, Kramer, Ritchie, & Jenkins, 2016; Jenkins et al., 2011). It is therefore possible that exposure to this variability can help unfamiliar viewers to switch from image-based to a more abstracted processing by aggregating the variability information into a single identity representation.

A growing body of research has shown that exposure to within-person variability helps when learning a new identity and this has been supported by work using both behavioural and computer modelling data (Dowsett, Sandford, & Burton, 2016; Jones, Dwyer, & Lewis, 2017; Kramer, Young, & Burton, 2018; Longmore et al., 2017; Longmore, Liu, & Young, 2008; Matthews, Davis, & Mondloch, 2018; Murphy, Ipser, Gaigg, & Cook, 2015; Ritchie & Burton, 2017; Robins, Susilo, Ritchie, & Devue, 2018). The benefits from access to multiple images of the same identity have been shown in adults' as well as in children's face learning (Matthews et al., 2018), with some evidence that children aged 6–13 need more variability than adults to learn a new person from video footage (Baker, Laurence, & Mondloch, 2017).

The amount of within-person variability is also an important factor in face learning. Ritchie and Burton (2017), for example, showed participants photos that were either high in variability (displaying changes in head angle, lighting, camera, age, hair style, etc) from a Google Images search, or photos that were low in variability, taken from a video of a single event (changes only in head angle and expression). After learning the identities from these images, participants' performance was tested with a name-verification and a face matching task using novel images of the same identities. In both cases, participants who had learned from the high variability image set outperformed those who learned from the low variability set. These results suggest that exposure to variability is key to learning or abstracting a stable representation of a person.

Research on the benefits of within-person variability for face matching has been less consistent and conclusive. Unlike face learning, this is a purely perceptual task with no demands on memory. Some studies suggest that multiple images may help to improve performance on face matching. White, Burton, Jenkins, and Kemp (2014) presented participants with arrays of two, three, or four images of the same person and asked them to match another image to the array. The multiple-image arrays gave rise to better performance than matching to a single image. In a different paradigm, participants were presented with a physical photograph of a target and asked to search through a pile of photos to find another image of the same person. On successive trials, participants were given an additional image of the same identity and their accuracy improved as the number of target images increased (Dowsett et al., 2016). Other recent studies, however, have failed the replicate these results with no benefits reported from exposure to arrays comprising a frontal and a profile view image (Kramer & Reynolds, 2018) or when matching a live person to a four-image array compared to a single image (Ritchie et al., 2020).

Therefore, when it comes to the key role of within-person variability for successful recognition, face learning and face matching tasks present somewhat dissimilar results. Exposure to variability helps learning a new identity, whereas results with matching are unclear. One possible explanation for this difference is that learning paradigms require the face to be memorised whereas matching paradigms present all stimuli simultaneously, without a memory component to the task. It is thus possible that exposure to variability, or multiple images, is only helpful when the task requires that a representation of the face be abstracted in order to be held in memory to make subsequent comparisons. This is supported by evidence for the benefits of within-person variability in face matching when images are presented one after the other, rather than simultaneously (Menon, White, & Kemp, 2015b).

Here, we compare face recognition accuracy in a purely perceptual simultaneous matching task and a memory-dependent sequential matching task. In a series of four studies, we manipulate the amount of within-person variability available, and the presentation order of multiple image arrays and comparison images, allowing us to determine why variability seems to be consistently aiding face learning but not face matching performance. It is possible that differences in results between previous studies are due to a difference in the amount of within-person variability shown in the arrays, with studies that have found a multiple-image benefit (e.g. White, Burton, et al., 2014) perhaps displaying more variability in the arrays than those that have not found that effect (e.g. Ritchie et al., 2020). However, if the difference in the utility of variability between face learning and matching is due to the memory component of learning tasks, then we would expect variability to facilitate performance in only sequential matching tasks. Like learning tasks, sequential matching tasks may require variability to be incorporated into a stable identity representation.

Experiment 1 investigates the effect of array variability on face matching performance in a simultaneous task. Experiment 2 compares performance in simultaneous versus sequential matching tasks. Finally, Experiments 3 and 4 compare performance on two different sequential tasks – one that allows for variability to be integrated into a single mental representation and one that does not.