Simulationist models of face-based emotion recognition
Introduction
Mindreading is the capacity to identify the mental states of others, for example, their beliefs, desires, intentions, goals, experiences, sensations and also emotion states. One approach to mindreading holds that mental-state attributors deploy a naı̈ve psychological theory to infer mental states in others from their behavior, the environment, and/or their other mental states. According to different versions of this “theory-theory” (TT), the naı̈ve psychological theory is either a component of an innate, dedicated module or is acquired by domain-general learning. A second approach holds that people typically execute mindreading by a different sort of process, a simulation process. Roughly, according to simulation theory (ST), an attributor arrives at a mental attribution by simulating, replicating, or reproducing in his own mind the same state as the target's, or by attempting to do so. For example, the attributor would pretend to be in initial states thought to correspond to those of the target, feed these states into parts of his own cognitive equipment (e.g. a decision-making mechanism), which would operate on them to produce an output state that is imputed to the target.
Mindreading has been studied in many disciplines, and both TT and ST have had proponents in each of them. In developmental psychology TT has been endorsed by Gopnik and Meltzoff, 1997, Gopnik and Wellman, 1992, Gopnik and Wellman, 1994, Leslie, 1994, Perner, 1991, Premack and Woodruff, 1978, Wellman, 1990, whereas ST has been defended by Harris, 1991, Harris, 1992. In philosophy ST has been endorsed by Currie and Ravenscroft, 2002, Goldman, 1989, Goldman, 1992a, Goldman, 1992b, Goldman, 2000, Gordon, 1986, Gordon, 1992, Gordon, 1996, Heal, 1986, Heal, 1996, Heal, 1998, whereas TT has been defended as an explicit approach to the execution of mindreading by Fodor, 1992, Nichols et al., 1996, Stich and Nichols, 1992, or as a theory of how the folk conceptualize mental states by Armstrong, 1968, Lewis, 1972, Sellars, 1956, Shoemaker, 1975. Studies of mentalizing in neuroscience (e.g. Fletcher et al., 1995, Frith and Frith, 1999, McCabe et al., 2001) typically ignore the TT–ST controversy but work with ‘theory of mind’ terminology, which is suggestive of TT, and cite the TT-leaning literature. On the other hand, much recent neuroscientific work is quite receptive to simulationist ideas (Blakemore and Decety, 2001, Carr et al., 2003, Chaminade et al., 2001, Gallese, 2001, Gallese, 2003, Gallese and Goldman, 1998, Iacoboni et al., 1999, Jeannerod, 2001), although the majority of this research is addressed less to mindreading per se than to related topics such as simulation of action, imitation, or empathy. In recent years a number of researchers have moved away from pure forms of TT or ST in the direction of some sort of TT/ST hybrid (Adolphs, 2002, Goldman (in preparation), Nichols and Stich, 2003, Perner, 1996), though the exact nature of the hybrid is rather fluid. In light of this continuing controversy, any research that provides substantial evidence for either ST or TT, even in a single subdomain of mindreading, deserves close attention.
In this paper we review a body of neuropsychological research that, we shall argue, supports ST for a certain circumscribed mindreading task. This is the task of attributing emotion states to others based on their facial expressions. This task is different from those usually studied in the mindreading literature, in part because the attributed mental states differ from the usual ones. The vast majority of the literature is devoted to propositional attitudes such as desires and beliefs, almost entirely ignoring emotion states like fear, anger, disgust, or happiness. There is no good reason to exclude these mental states, which are routinely attributed to others in daily life. So it is time to extend research and theory into this subdomain of the mental. At the same time, it cannot be assumed that the style of mindreading in this subdomain is the same as the style that characterizes other subdomains. So we make no attempt to generalize from the type of mental state ascriptions studied here to mindreading tout court.
There are at least two reasons why the properties of face-based emotion recognition (FaBER) might not be shared by methods of mindreading in other subdomains. First, the recognition or classification of propositional attitude contents may introduce a level of complexity that goes beyond the task of classifying emotion types. Second, the reading of emotions, especially basic emotions, may have unique survival value, so it is conceivable that specialized programs have evolved for the recognition of emotions, and these specialized programs may not operate in other mindreading tasks. Because of these differences between FaBER and other types of mindreading, it cannot be assumed that the processes characteristic of FaBER can be extrapolated to other types of mindreading.
We begin by reviewing existing findings, some clinical and some experimental, that display a striking pattern of paired deficits between emotion production and face-based recognition (attribution). These findings have not previously been brought together with the explicit intent of examining them in the context of the TT–ST controversy. Next we argue that this pattern readily lends itself to a simulationist explanation, whereas existing data do not fit with a theory-based explanation. Finally, the core project of the paper is to formulate and evaluate four specific models of how normal mindreaders could use simulation to arrive at emotion classifications.
Section snippets
Paired deficits in emotion production and face-based recognition
In early studies, Ralph Adolphs and colleagues investigated whether damage to the amygdala affects face-based emotion recognition (Adolphs, 1995, Adolphs et al., 1994). These studies were motivated by the well-known role of the amygdala in mediating fear, including its prominent role in fear-conditioning and the storage of fear-related emotion memories (LeDoux, 1993, LeDoux, 2000). One patient studied by Adolphs et al. was SM, a 30-year-old woman with Urbach-Wiethe disease, which resulted in
Emotion mindreading by theory versus simulation
Let us further clarify and expound the two basic theoretical positions towards mindreading, which have loomed large in the literature. There are numerous ways of developing the TT idea, but the main idea is that the mindreader selects a mental state for attribution to a target based on inference from other information about the target. According to one popular version of TT, such an inference is guided by folk psychological generalizations concerning relationships or transitions between
Explaining the emotion recognition data by TT versus ST
The central claim presented in Section 2 was that there is substantial evidence concerning three emotions indicating that deficits in the production (experience) of an emotion and deficits in the face-based recognition of that emotion reliably co-occur. This strongly suggests that the same neural mechanisms subserve both the experience and the recognition of an emotion. In addition, the Wicker et al. (2003) study found that, in normals, the same neural regions were implicated in both the
Possible simulationist models
Although the foregoing case for a simulational approach to face-based emotion recognition strikes us as compelling, it leaves open the question of how the simulational process proceeds. Those skeptical of our case for simulation may remain skeptical as long as no plausible, sufficiently detailed story of the simulation process in these cases is forthcoming. We get little help on this question from the existing literature. Articles describing paired deficits often contain conclusions hinting at
Why simulation?
In the preceding sections, we marshaled evidence that FaBER proceeds by simulation rather than theory-based mechanisms. At any rate, simulation is the fundamental or primitive method of recognizing emotion from faces, although theorizing might also be used, for example, as a compensatory strategy. The evidence is much less clear-cut, however, with regard to distinguishing possible simulationist models, and much further study is warranted. Our hope is that the specific models we have formulated
Acknowledgements
The authors wish to thank Andrew Lawrence, Vittorio Gallese, Giacomo Rizzolatti, Christian Keysers, Ralph Adolphs, Daniel Osherson, and three anonymous reviewers for valuable comments on earlier drafts of this paper.
References (95)
- et al.
Dissociable neural systems for recognizing emotions
Brain and Cognition
(2003) - et al.
Recognition of facial emotion in nine individuals with bilateral amygdala damage
Neuropsychologia
(1999) - et al.
Cognitve representations of hand posture in ideomotor apraxia
Neuropsychologia
(2003) - et al.
A principal component analysis of facial expressions
Vision Research
(2001) - et al.
Other minds in the brain: A functional imaging study of ‘theory of mind’ in story comprehension
Cognition
(1995) A theory of the child's theory of mind
Cognition
(1992)- et al.
Mirror neurons and the simulation theory of mind-reading
Trends in Cognitive Sciences
(1998) - et al.
Recognition and imitation of pantomimed motor acts after unilateral parietal and premotor lesions: A perspective on apraxia
Neuropsychologia
(2001) - et al.
Facial mimicry and emotional contagion to dynamic facial expressions and their influence on decoding accuracy
International Journal of Psychophysiology
(2001) Neural simulation of action. A unifying mechanism for motor cognition
NeuroImage
(2001)
Emotion memory systems in the brain
Behavioral Brain Research
Pretending and believing: Issues in the theory of ToMM
Cognition
Premotor cortex and the recognition of motor actions
Cognitive Brain Research
Dissociation of neural representation of intensity and affective valuation in human gustation
Neuron
Both of us disgusted in my insula: The common neural basis of seeing and feeling disgust
Neuron
Facial efference and the experience of emotion
Annual Review of Psychology
Fear and the human amygdala
Journal of Neuroscience
Recognizing emotion from facial expressions: Psychological and neurological mechanisms
Behavioral and Cognitive Neuroscience Reviews
A role for the somatosensory cortices in the visual recognition of emotion as revealed by three-dimensional lesion mapping
Journal of Neuroscience
Impaired recognition of emotion in facial expressions following bilateral damage to the amygdala
Nature
A materialist theory of the mind
The architecture of the colour centre in the human visual brain: New results and a review
European Journal of Neuroscience
Double dissociation of conditioning and declarative knowledge relative to the amygdala and hippocampus in humans
Science
Contributions to neuropsychological assessment
From the perception of action to the understanding of intention
Nature Reviews Neuroscience
Facial expression recognition by people with Mobius syndrome
Cognitive Neuropsychology
Impaired recognition and experience of disgust following brain injury
Nature Reviews Neuroscience
Neuropsychology of fear and loathing
Nature Reviews Neuroscience
Neural mechanisms of empathy in humans: A relay from neural systems for imitation to limbic areas
Proceedings of the National Academy of Science USA
Is perceptual anticipation a motor simulation? A PET study
NeuroReport
Recreative minds
Descartes' error
The feeling of what happens
Rapid facial reactions to emotional facial expressions
Scandinavian Journal of Psychology
Are there basic emotions?
Psychological Review
Motor facilitation during action observation: A magnetic stimulation study
Journal of Neurophysiology
Interacting minds—a biological basis
Science
MR functional imaging of regional brain responses to pleasant and unpleasant odors
American Journal of Neuroradiology
The ‘shared manifold’ hypothesis: From mirror neurons to empathy
Journal of Consciousness Studies
The manifold nature of interpersonal relations: The quest for a common mechanism
Philosophical Transactions of the Royal Society of London, B
Action recognition in the premotor cortex
Brain
Interpretation psychologized
Mind and Language
In defense of the simulation theory
Mind and Language
Empathy, mind and morals
Proceedings and Addresses of the American Philosophical Association
The mentalizing folk
Words, thoughts, and theories
Cited by (481)
Effects of square attractiveness on emotional perception, cognitive performance, and neurophysiology
2023, Frontiers of Architectural ResearchNonverbal communication
2023, Encyclopedia of Mental Health, Third Edition: Volume 1-3Facial emotion recognition deficits are associated with hypomimia and related brain correlates in Parkinson’s disease
2024, Journal of Neural TransmissionBodily expressions as gestalts. An argument for grounding direct perception theories
2024, Philosophical Psychology