Abstract

Introduction

During a speech in the winter of 2004, photographers captured a picture of Howard Dean looking enraged; this picture cost him his political party’s endorsement to run for President of the United States. Reporters who saw Dean in context noted that he seemed happily engaged with the animated, cheering crowd. Such mistakes are easy to make. The man in Figure 1a looks angry. But look again, this time at Figure 1b. You see an elated Jim Webb celebrating the 2007 electoral victory that returned control of the United States Senate to the Democratic National Party. Or consider the fact that 60%–75% of the time, people see facial portrayals of fear as ‘angry’ when the images are paired with contextual information typically associated with anger [1]. You can imagine the consequences when, in war, a soldier enters a house and sees a civilian as angry instead of fearful (or vice versa). These examples illustrate the importance of context in emotion perception. Descriptions of the social situation [2], body postures, voices, scenes [3] or other emotional faces [4] each influence how emotion is seen in the face of another person.

Open in a separate window

Figure 1

The role of context in emotion perception (Doug Mills/The New York Times/Redux). Look at United States Senator Jim Webb in (a). Taken out of context, he looks agitated and aggressive. Yet look at him again in (b). When situated, he appears happy and excited. Without context, and with only the structural information from the face as a guide, it is easy to mistake the emotion that you see in another person. A similar error in perception was said to have cost Howard Dean the opportunity to run for President of the United States in 2004.

Context refers not only to the external surroundings in which facial actions take place but also to parallel brain processes that dynamically constrain or shape how structural information from a face is processed. In this opinion piece, we focus on one such process, language, by exploring the idea that emotion words (implicitly or explicitly) serve as an internal context to constrain the meaning of a face during an instance of emotion perception.

We begin by suggesting that the psychological phenomena referred to by the English words ‘anger,’ ‘sadness,’ ‘fear,’ ‘disgust,’ ‘surprise’ and ‘happiness’ are not expressed as fixed patterns of facial behaviors, even though studies of emotion perception employ pictures of posed, highly stereotyped configurations of facial actions (or caricatures; see Box 1). In everyday life, the available structural information in a face is considerably more variable (and ambiguous) than scientists normally assume (and certainly more ambiguous than the structural information that is presented to perceivers in the typical emotion-perception experiment). We then consider psychological and neuroscience investigations that are broadly consistent with the idea that language serves as a context to reduce the ambiguity of this information, even when caricatured faces are being used as perceptual targets. Finally, we end by suggesting that the language-as-context hypothesis reframes the linguistic-relativity debate into the more interesting question of how far down into perception language can reach.

The great emotions debate

The ‘basic emotion’ approach

Faces appear to display emotional information for you to read, like a word on a page. If you take your alacrity in seeing anger (Figure 1a) or excitement (Figure 1b) as evidence that reading emotion in faces is natural and intrinsic, then you are in good company. The ‘basic emotion’ approach is grounded in the belief that certain emotion categories are universal biological states that are (i) triggered by dedicated, evolutionarily preserved neural circuits (or affect programs), (ii) expressed as clear and unambiguous biobehavioral signals involving configuration of facial muscle activity (or ‘facial expressions’), physiological activity, instrumental behavior (or the tendency to produce a behavior) and distinctive phenomenological experience (Figure 2), and (iii) recognized by mental machinery that is innately hardwired, reflexive and universal, so that all people everywhere (barring organic disturbance) are born in possession of five or six perceptually grounded emotion categories. (An alternative view might be that people are not born in possession of these categories but instead develop them as they inductively learn the statistical regularities in emotional responses.)

Open in a separate window

Figure 2

The natural-kind model of emotion (adapted from [2] with permission). A natural-kind model of emotion states that emotions are triggered by an event and are expressed as a recognizable signature consisting of behavioral and physiological outputs that are coordinated in time and correlated in intensity [54–56]. Presumably, these patterns allow people (including scientists) to know an emotion when they see it by merely looking at the structural features of the emoter’s face.

According to the basic emotion view, “the face, as a transmitter, evolved to send expression signals that have low correlations with one another . . . the brain, as a decoder, further de-correlates these signals” [5]. The face is presumed to encode anger (or sadness, fear, disgust etc.) in a consistent and unambiguous way so that structural information on the face is sufficient for communicating a person’s emotional state. As a consequence, experimental studies of emotion perception often rely on a set of fixed, exaggerated facial portrayals of emotion that were designed for maximum discriminability (Box 1).

Evidence for the role of language in emotion perception

Some studies are consistent with, but not necessarily direct evidence for, the language-as-context hypothesis. For example, a recent meta-analysis of neuroimaging studies [10] found that inferior frontal gyrus (IFG), extending from the pars opercularis (Broca’s area, BA 44) through pars triangularis (BA 45) and pars orbitalis on the inferior frontal convexity (BA 47/12 l), is part of the distributed neural system that supports emotion perception. IFG is broadly implicated in a host of cognitive processes, including language [11] and the goal-related retrieval of conceptual knowledge [12]. The act of providing an emotional label to caricatured emotional faces (as opposed to a gender label) increases neural activity in right IFG and produces a corresponding decrease in amgydala response[13]. This reduction in amgydala response can be thought of as reflecting a reduced ambiguity in the meaning of the structural information from the face.

Other studies offer evidence that more directly supports the language-as-context hypothesis, even when people view caricatured portrayals. Failure to provide perceivers with a small set of emotion labels to choose from when judging caricatures (i.e. requiring participants to free label) significantly reduces ‘recognition accuracy’ [14], leading to the conclusion that emotion words (when they are offered) are constraining people’s perceptual choices. A similar effect can be observed in event-related potential (ERP) studies of emotion perception. Early ERPs resulting from structural analysis of a face (as early as 80 ms, but typically between 120 and 180 ms after stimulus onset, depending on whether the face is presented fovially or parafoveally) do not distinguish caricatured portrayals of discrete emotions from one another but instead reflect the categorization of the face as a face (vs a non-face), as generally affective (neutral vs valenced), as valenced (e.g. happy vs sad), or as portraying some degree of arousal (for reviews, see [15–17]). Yet when participants explicitly categorize caricatures as ‘anger’ or ‘fear’, P1 and N170 ERPs are differentially sensitive to anger and fear faces that were incongruously paired with fear and anger body postures, suggesting that these components distinguished between the two emotion categories. Presumably, participants would need to perceive that the faces and bodies were associated with different emotion categories to see them as incongruous [18].

In addition, emotion words cause a perceptual shift in the way that faces are seen. Morphed faces depicting an equal blend of happiness and anger are encoded as angrier when those faces are paired with the word ‘angry’, and they are encoded as even angrier when participants are asked to explain why those faces are angry [19]. In addition, the pattern of neural activity associated with judging a neutral face as fearful or disgusted is similar (although not identical) to the pattern associated with looking at caricatured fearful and disgusted faces (Figure S2 in online supplementary materials for [20]).

Possibly the most direct experimental evidence for the language-as-context hypothesis comes from studies that manipulate language and look at the resulting effects on emotion perception. Verbalizing words disrupts the ability to make correct perceptual judgments about faces, presumably because it interferes with access to judgment-necessary language [21]. A temporary reduction in the accessibility of an emotion word’s meaning (via a semantic satiation procedure, Figure 3) leads to slower and less accurate perceptions of an emotion, even when participants are not required to verbally label the target faces [22].

Open in a separate window

Figure 3

Semantic-satiation paradigm. Participants in [22] performed a number of trials in which they repeated an emotion word such as ‘anger’ aloud either three times (temporarily increasing its accessibility) or 30 times (temporarily reducing its accessibility), after which they were asked to judge whether two faces matched or did not match in emotional content. Participants were slower and less accurate to correctly judge emotional faces (e.g. two anger faces) as matching when they had repeated the relevant emotion word (e.g. ‘anger’) 30 times (i.e. when the meaning of the word was made temporarily inaccessible). By examining response times and accuracy rates for various trial types, researchers were able to rule out fatigue as an alternative explanation for the observed effects (e.g. emotion perception was similarly encumbered when participants repeated an irrelevant emotion word either three or 30 times, whereas fatigue would have caused a decrease only when the word was repeated 30 times).

Implications

In this paper, we have suggested that people usually go beyond the information given on the face when perceiving emotion in another person. Emotion perception is shaped by the external context that a face inhabits and by the internal context that exists in the mind of the perceiver during an instance of perception. Language’s role in emotion perception, however unexpected, is consistent with emerging evidence of its role in color perception [23], the visualization of spatial locations [24], time perception [25] and abstract inference [26]. In our view, language is linked to conceptual knowledge about the world that is derived from prior experience and that is re-enacted during perception [27]. It may be that all context influences emotion perception via such conceptual knowledge, but that remains to be seen.

Outstanding questions remain regarding the role of language in perception of emotion (see Box 3). From our view, the language-as-context hypothesis is generative because it moves past the debate between the strong version of linguistic relativity (which is untenable) and the weak version (which some consider less interesting) into a more interesting question of process: how far down into perceptual processing does language reach?

Acknowledgments

References