1 Introduction

It is a common view that mental states are private entities. Since perception is awareness of objects in the environment, how can we see something that is as private as an emotion? In what follows I argue that we can see what emotions others have when two conditions obtain. One is that expressions carry information about the emotions that they express. The other is that the visual system functions to detect these states and it does so by extracting the information that expressions carry. I should note that my approach will focus on early stages of visual processing and on the brain structures that underpin it. Nothing of what I will say is incompatible with the fact that mature visual experiences of other people’s emotions—the experiences that we have when we see others as being sad or angry—are the result of the activity of two systems: the perceptual and the cognitive. My goal in this paper is to provide a partial account of how the visual system works in producing such experiences. Finally, I should note that my approach is entirely compatible with the view that simulation and empathy play an important role in our ability to see other people’s emotions.

2 Theory of Mind and Emotion Perception

Expressions are patterns of changes in the face.Footnote 1 It is a common philosophical view that our ability to make sense of these (and other) changes depends on having a theory of mind. This is known as the ‘theory-theory’ approach to mental states (Armstrong 1968; Lewis 1972). In this view, there is a theory of mind in much the same way as there is a theory of gases. The thermodynamic theory of gases posits states—called temperature, pressure, and volume—that stand in certain relationships approximating those expressed in the ideal gas laws (‘approximating’ because no real gas is ideal). These states and relationships enable us to explain and predict the behaviour of gases. In the same way, with the theory-theory approach to the mind, we posit psychological properties of subjects; these properties stand in various relationships to one another and to the behaviour of subjects. We suppose, for example, that subjects have beliefs and desires, and that they behave in such a way that, if their beliefs are true, their desires are satisfied. In other words, there is a theory sometimes known as folk-psychology that connects mental states and behaviour. If this approach to the mind is right, how do we arrive at justified beliefs about other people’s mental states? By a simple application of hypothetico-deduction. We note a person’s behaviour within its circumstances and look for the hypothesis about that person’s mental states which best explains her behaviour. This exercise is an inference from the person’s behaviour to the mental states that best explain it.

How do these considerations bear upon the claim that we see what others feel by looking at their facial expressions? To answer this question we need to consider two things. One is the theory that links patterns of changes in the face with emotions. The other is the role that this theory plays in our ability to see what emotions others have. The theory says that there is a correlation between certain patterns of changes and emotions. For example, there is a correlation between happiness and smiling such that it is likely that a person smiles when she is happy. It is plausible to think that a theory of mind will include a number of beliefs about the relationship between patterns of changes in the body and emotions.

Now, what role do these beliefs play in the ability to read what others feel? It is commonly accepted that knowledge and belief influence the way we see the world. For example, it is when I am familiar with a specific convention about the measurement of time with devices like clocks that I am able to tell the time simply by looking at the position of a clock’s hands on a dial. In this case my beliefs about clocks enable me to see what time it is. The enabling character of belief can be described in terms of an unconscious inference that I make as I look at the clock’s dial, and which allows me to infer the time from the position of the hands. The same argument applies to the experience of looking at people’s facial expressions. Beliefs about the correlation between emotions and patterns of bodily changes enable me to see the changes as expressions of emotions. This view provides an explanation for the observation that we can often tell what others feel by looking at their expressions.

In spite of its good explanatory power, the view has at least one serious drawback. We are all familiar with the observation that infants respond to facial expressions of emotions in others. This everyday observation is corroborated by empirical evidence (Nelson 1987) showing that infants are more attracted to expressions of positive emotions, while they tend to avoid and respond with fear to expressions of negative emotions. Prima facie, the evidence shows that facial expressions prompt positive and negative responses in infants. This, however, is not the only possible explanation. Consider the case of an infant avoiding an expression of anger. Since avoidance is a way of reacting to threats, we may suppose that the expression presents the infant with some sort of threat. But exactly what is it that makes the expression threatening? If we take expressions to be ways of signalling emotions, we can say that the threat consists in the emotion that the expression shows, and not in the expression alone. After all, it is the person’s emotion, not her expression, that constitutes a threat. On this reading of the evidence, infants react to the emotions that are shown in the expressions, not just to the patterns of changes that constitute the expressions. If this account is correct, it is possible to argue that infants can see emotions in others even though they lack the sort of knowledge that, in the theory of mind view, is necessary to see patterns of changes in the face as expressions of emotions. This, of course, does not rule out the possibility that infants may have a theory of mind—one that is hardwired and, possibly, innate—which they use to make sub-personal inferences about the mental states of others. These inferences would still be significantly different from the full-blown deductions (or rational reconstructions) that, on certain versions of the theory of mind approach, lead to judgments about what others feel.

In light of these considerations, one may ask the following question: Is there a way of explaining our ability to see emotions that does not rely entirely on background knowledge? This in turn raises a more general question: Is there any situation in which our ability to see a certain object or property is not determined by what we know about it? Philosophers accept that knowledge influences perception, although they disagree on the extent to which knowledge determines the content of perception (Peacocke 1992; McDowell 1994). If we want to explain the observation from the case of infants, we need to consider those visual experiences that can be had independently of knowledge (implicit or otherwise). These are experiences that can be had solely by virtue of how the visual system works. Research into the visual system (Adams et al. 1994; Kaufmann 1995) has shown that the ability to see specific properties, such as colour and motion, is already present at birth and therefore does not require prior knowledge of colour or motion. This is to say that an experience of, say, something red can be had even though the perceiver does not know anything about the colour red, the things that normally look red, or what an experience of red is like. It is indeed one of the functions of the visual system that it detects colours. Similarly, we may suppose that the ability to see what emotions others have also depends on the working of the visual system.

Visual perception is the result of physical properties, such as colours and shapes, impinging upon the visual system. Since facial expressions play an important role in our ability to tell what others feel, is there a way of understanding the role of expressions that helps us make better sense of the intuition that we do, indeed, see what emotions others have? Expressions are physical events consisting of patterns of changes in the face. As such, they are things that we can see. More precisely, they are things that can affect the visual system and consequently produce visual experiences. This fact, however, does not explain how visual experiences of expressions can lead to visual experiences of emotions, as one may perceive physical changes in the face without perceiving the emotions that they express. To explain how we can see emotions we need to suppose that expressions carry information about them—that is, we need to suppose that seeing other people’s facial expressions is a way of acquiring visual information about the emotions they have. I will examine this view in the next section.

3 Expressions and Information

Expressions tell us all sorts of things about emotions. They can tell us what kinds of emotions others have. For example, simply by looking at someone’s face, I can tell if she is sad or happy. They can also tell us how intense those emotions are and how long they seem to last. Thus I may be able to tell if someone is slightly, as opposed to intensely, annoyed. In addition, I can say for how long that emotion lasts. In some cases, expressions can also tell us how certain emotions arise in relation to objects and events. Consider the case of someone who sees a child being attacked by a dog. If you are looking at the person’s expression while she witnesses the attack, you will learn that something terrible is happening in the vicinity, something that is responsible for the person’s sudden horror. In summary, expressions carry information about several features of emotions. They can tell us what kinds of emotions others have, how intense they are, how long they last and in relation to what objects or events they occur.

At this stage, one may wonder how expressions get to carry information about emotions. I will answer this question by considering the relationship between expressions, on one hand, and emotions, on the other. There are different views about this relationship and they roughly coincide with theories of emotions. One view claims that emotions are patterns of changes in the body (Damasio 1994). Another claims that emotions are experiences of these patterns (James 1884; Prinz 2004). Both views say that expressions are the outwardly observable part of the bodily changes that constitute or occur in correspondence with emotions. The fact that expressions carry information about emotions is thus explained in terms of expressions being part of what emotions are or supervene on. Another view is that emotions bring about patterns of changes in the body, including the face (Lyons 1980; Roberts 2003). This view implies that there is a dependence relationship between emotions and expressions such that, in normal circumstances, expressions are produced by emotions and other similar affective states, pain and pleasure, for example. The fact that expressions normally occur as a result of emotions makes it more likely that when one observes a pattern of changes in the face one is exposed to something that has been produced by an emotion. This explains how expressions can carry information about emotions.

The view that expressions carry information about emotions (and other affective states) is corroborated by empirical evidence. In a series of seminal studies, Paul Ekman and colleagues (Ekman et al. 1969; Ekman and Friesen 1971; Izard 1971) showed that some specific emotions, which they named basic emotions, appear to be expressed in the same way in every human culture where this has been tested. In particular, they found that basic emotions produce the same patterns of changes in the face. These findings turn out to be surprising if we adopt the following reasoning. It is a fact that different cultures develop different expressive tools. For example, different cultures have different languages and communication systems. If we take expressions to be signs for emotions, it is natural to expect that different cultures have different ways of expressing emotions. While this is true of some expressions (Lutz and White 1986), it is not true of expressions of basic emotions, for basic emotions seem to have the same expressions in all human cultures that have been studied so far. Ekman and colleagues concluded that there is a connection between particular facial configurations and specific emotions, and that such a connection is universal. They supposed that this universal connection became established by natural selection. Although the evidence has been widely challenged in recent years, especially on methodological grounds (Russell 1994; Naab and Russell 2007), it is clear that there is a sense in which some emotions, disgust for example, produce the same patterns of facial changes in most humans. This means that, in all human cultures, a person who feels disgust is likely to undergo the same, or roughly the same, pattern of changes in the face. As a result, a specific pattern of changes is a reliable indicator for the occurrence of disgust in others. This consideration lends support to the idea that expressions carry information at least about basic emotions.

4 Some Objections

I should note that work in psychology challenges the approach that I have been outlining so far. The psychologist Alan Fridlund (1994), for example, argues that the function of expressions is not to show or manifest inner states like emotions, but to signal social intentions and motives. A smile, for example, is not an expression of happiness; it is an invitation for others to draw near. This view challenges a key claim of my approach, namely the idea that expressions carry information about emotions. Studies on emotion perception (Aviezer et al. 2008; Barrett and Kensinger 2010) challenge another key claim, the one according to which visual experiences of emotions can be had independently of background knowledge. There is indeed evidence that such experiences encode information about contextual variables, including the situation in which the emotional expressions occur (Barrett and Kensinger 2010) and the perceiver’s emotional vocabulary (Lindquist et al. 2006; Barrett et al. 2007).

One may wonder if and how my view can withstand these objections. The answer depends on how one constructs the opposing views. Consider Fridlund’s approach. If one takes him to say that there is no relationship between emotions and expressions, then my view is clearly incompatible with his approach. However, it seems unlikely that Fridlund would want to deny any relationship between emotions and expressions. If, by contrast, one takes Fridlund to say that signalling emotions is not the primary function of expressions, then his approach is compatible with mine. By saying that expressions carry information about emotions, my view allows for the possibility that expressions may have other functions—conveying social motives, for example—besides the one I described.

How about the evidence showing that background and contextual factors play an important role in visual experiences of other people’s emotions? Most philosophical theories of perception acknowledge the influence of these factors. Some philosophers (Dretske 1981; Peacocke 1983), however, think that, by carrying information about the environment, visual experience encodes not only information about background and contextual factors, it also encodes information about basic features of the environment, such as colour and motion, and that this information would be acquired in any case, even if the perceiver did not know anything about the world. Consider the case of visual information about motion. I can see that something is moving even when I do not know what it is that is moving. Thus, for example, I can see four bright dots moving in the sky without knowing that it is an airplane. In this case information about motion is acquired independently of information about the identity of the moving object. Of course, this is not to say that background and contextual information plays no role in perception. If I knew that I was looking at an airplane, I would probably see the four dots as forming a familiar shape – that of an airplane with a light on the tail, a light on each wing, and another light at the front. This is because the content of perception comprises both visual information—information about light and motion in my example—and collateral information like knowledge about airplanes.

A similar argument carries over to the case of emotion perception. As will be clear in the next section, I believe that the visual system functions to extract the information that expressions carry about emotions and that it does so without relying on background knowledge. In other words, part of what we see when we see other people’s emotions is acquired in a way that depends not so much on background knowledge, but rather on the working of the visual system. This view is entirely compatible with the observation that background and contextual factors can enter the content of perception, as they commonly do, and determine how we see what we see. To put it in other terms, emotion perception is a complex process that combines information about the source of the stimulus, that is, the emotion and the emotional expression, and information about background and contextual variables. In this paper I describe what may happen in early stages of vision, when the stimulus that reaches the eyes originates from a physical event—a facial expression—that has an emotion as its underlying cause. Of course, this is only part of the story and complex variables, including knowledge and context, may influence the way emotion perception works. The fact remains that if the hypothesis that the visual system functions to detect emotions is correct, there really is something like seeing another person’s emotion.

5 Emotions and the Visual System

What I have so far observed shows that expressions carry information about emotions. This phenomenon, however, is not sufficient for us to conclude that we see emotions. Consider the case of infrared light. It carries information about the temperature of things. We can say that a creature is able to see an object’s temperature only if it is endowed with a visual system sensitive to infrared light, that is, a visual system able to extract the information about the object’s temperature and make it available in the creature’s visual experience. Similar considerations apply to emotions, for we can say that a creature sees emotions only if its visual system is able to extract the information that expressions carry and make it available as part of the content of the creature’s visual experience.

The question now is, How are we to determine whether the human visual system extracts the information that expressions carry and makes it available in the content of experience? This question ultimately concerns the working of the human visual system in response to presentations of facial expressions of emotions. We know from the study of vision that a reliable way of learning about the working of the visual system is to consider the relationship between damage to the system and acquired visual deficits. For example, evidence that the visual system functions to detect motion comes from the observation that damage to specific areas of the visual cortex results in a specific form of blindness that makes people unable to see motion (Zihl et al. 1983). In line with this approach, it is plausible to think that if the visual system functions to detect emotions, we should be able to observe a correspondence between damage to the central nervous system and a visual deficit that makes people unable to see what emotions others have.

Studies have shown that acquired damage to the amygdala results in impaired recognition of negative emotions when they are expressed in the face. Ralph Adolphs and colleagues (Adolphs et al. 1994; Adolphs 1995), for example, studied SM, a 30-year-old woman with damage to both amygdalae. During a series of tests, SM was shown photographs and video clips of facial expressions. She was then asked to identify the emotions to which the expressions corresponded. SM showed a significant impairment in the recognition of fear. Initially it was thought that the deficit compromised SM’s ability to recall information about the ways in which fear is expressed in the face (Adolphs 2002). This hypothesis was then abandoned when researchers noticed that SM did not spontaneously attend to the eye region of faces (Adolphs et al. 2005). Kennedy and Adolphs (2010) have recently replicated their study and found that SM’s reduced focus on the eye region is particularly pronounced during the first fixation to the face and is almost normal during subsequent fixations.

A possible interpretation of the evidence gathered in these and various other studies is that the amygdala plays a key role in directing attention to specific features of expressions, features that carry a particular weight in terms of the information they provide about specific emotions, fear in particular. Fear is an emotion that mainly produces changes in the eye region and upper side of the face. The fact that SM did not spontaneously attend to this area during her first fixation to the face explains why she was not immediately able to recognize fear. One may speculate that the visual experiences that SM formed during the first saccade lacked information about changes in the eye region, and, consequently lacked information about emotions, such as fear, that typically produce changes in that area of the face. One may further speculate that SM’s visual experience is analogous to the experience that a healthy person has when she looks at someone who is wearing large sunglasses. In this case, the person’s experience lacks information about what goes on in the eye region, and consequently provides no visual information about the observed person’s mental state.

These considerations suggest that there is a visual mechanism involved in perception of specific emotions in others. Although it is not yet clear how the mechanism extracts the information that expressions carry about emotions, it is evident that damage to parts of the physiological structures underpinning the mechanism can compromise a person’s ability to see what emotions others have. The evidence I have examined suggests that the amygdala is a key component of this mechanism. Of course, other brain structures are likely to be involved in the same mechanism as well as in the process of acquiring information about other types of emotions. Evidence from studies of subjects with Huntington’s disease (Sprengelmeyer et al. 1996), for example, show that the basal ganglia play an important role in people’s ability to see disgust in others.

6 Conclusions

I opened my discussion with the following question: How is it that we can tell what emotions others have by looking at their facial expressions? A natural answer is that we see their emotions. This answer, however, raises a further question. Since emotions are not physical objects, how can we possibly see them? I have argued that expressions can be viewed as physical events that carry information about emotions. This is either because expressions are part of what emotions are or, on a different view, because they are part of the overall changes that emotions produce when they affect the body. I have then examined evidence from studies showing that damage to specific brain areas can result in a deficit characterized by the inability to see and consequently recognize fear when it is expressed in the face. I have used this evidence to show that there is something like a visual mechanism to detect fear.