There are many ways to assess face perception skills. In this study, we describe a novel task battery FEAST (Facial Expression Action Stimulus Test) developed to test recognition of identity and expressions of human faces as well as stimulus control categories. The FEAST consists of a neutral and emotional face memory task, a face and object identity matching task, a face and house part-to-whole matching task, and a human and animal facial expression matching task. The identity (...) and part-to-whole matching tasks contain both upright and inverted conditions. The results provide reference data of a healthy sample of controls in two age groups for future users of the FEAST. (shrink)

This paper proposes that human expression of pain in the presence or absence of caregivers, and the detection of pain by observers, arises from evolved propensities. The function of pain is to demand attention and prioritise escape, recovery, and healing; where others can help achieve these goals, effective communication of pain is required. Evidence is reviewed of a distinct and specific facial expression of pain from infancy to old age, consistent across stimuli, and recognizable as pain by (...) observers. Voluntary control over amplitude is incomplete, and observers can better detect pain that the individual attempts to suppress rather than amplify or simulate. In many clinical and experimental settings, the facial expression of pain is incorporated with verbal and nonverbal vocal activity, posture, and movement in an overall category of pain behaviour. This is assumed by clinicians to be under operant control of social contingencies such as sympathy, caregiving, and practical help; thus, strong facial expression is presumed to constitute an attempt to manipulate these contingencies by amplification of the normal expression. Operant formulations support skepticism about the presence or extent of pain, judgments of malingering, and sometimes the withholding of caregiving and help. To the extent that pain expression is influenced by environmental contingencies, however, could equally plausibly constitute the release of suppression according to evolved contingent propensities that guide behaviour. Pain has been largely neglected in the evolutionary literature and the literature on expression of emotion, but an evolutionary account can generate improved assessment of pain and reactions to it. (shrink)

Research on facial emotion expression has mostly focused on emotion recognition, assuming that a small number of discrete emotions is elicited and expressed via prototypical facial muscle configurations as captured in still photographs. These are expected to be recognized by observers, presumably via template matching. In contrast, appraisal theories of emotion propose a more dynamic approach, suggesting that specific elements of facial expressions are directly produced by the result of certain appraisals and predicting the facial (...) patterns to be expected for certain appraisal configurations. This approach has recently been extended to emotion perception, claiming that observers first infer individual appraisals and only then make categorical emotion judgments based on the estimated appraisal patterns, using semantic inference rules. Here we report two related studies to empirically investigate the facial action unit configurations that are used by actors to convey specific emotions in short affect bursts and to examine to what extent observers can infer a person’s emotions from the predicted facial expression configurations. The results show that 1) professional actors use many of the predicted facial action unit patterns to enact systematically specified appraisal outcomes in a realistic scenario setting, and 2) naïve observers infer the respective emotions based on highly similar facial movement configurations with a degree of accuracy comparable to earlier research findings. Based on estimates of underlying appraisal criteria for the different emotions we conclude that the patterns of facial action units identified in this research correspond largely to prior predictions and encourage further research on appraisal-driven expression and inference. (shrink)

In this article, we review empirical evidence regarding the relationship between facial expression and emotion during infancy. We focus on differential emotions theory’s view of this relationship because of its theoretical and methodological prominence. We conclude that current evidence fails to support its proposal regarding a set of pre-specified facial expressions that invariably reflect a corresponding set of discrete emotions in infants. Instead, the relationship between facial expression and emotion appears to be more complex. Some (...) facial expressions may have different meanings in infants than in children and adults. In addition, nonemotion factors may sometimes lead to the production of “emotional” facial expressions. We consider alternative perspectives on the nature of emotion and emotional expression in infancy with particular focus on differentiation and dynamical systems approaches. (shrink)

Following Yik and Russell (1999) a judgement paradigm was used to examine to what extent differential accuracy of recognition of facial expressions allows evaluation of the well-foundedness of different theoretical views on emotional expression. Observers judged photos showing facial expressions of seven emotions on the basis of: (1) discrete emotion categories; (2) social message types; (3) appraisal results; or (4) action tendencies, and rated their confidence in making choices. Emotion categories and appraisals were judged significantly more accurately (...) and confidently than messages or action tendencies. These results do not support claims of primacy for message or action tendency views of facial expression. Based on a componential model of emotion it is suggested that judges can infer components from categories and vice versa. (shrink)

Facial expression recognition plays a significant part in artificial intelligence and computer vision. However, most of facial expression recognition methods have not obtained satisfactory results based on low-level features. The existed methods used in facial expression recognition encountered the major issues of linear inseparability, large computational burden, and data redundancy. To obtain satisfactory results, we propose an innovative deep learning model using the kernel entropy component analysis network and directed acyclic graph support vector machine. (...) We use the KECANet in the feature extraction stage. In the stage of output, binary hashing and blockwise histograms are adopted. We sent the final output features to the DAGSVM classifier for expression recognition. We test the performance of our proposed method on three databases of CK+, JAFFE, and CMU Multi-PIE. According to the experiment results, the proposed method can learn high-level features and provide more recognition information in the stage of training, obtaining a higher recognition rate. (shrink)

Facial expression is a mode of close‐proximity non‐vocal communication used by primates and is produced by mimetic/facial musculature. Arguably, primates make the most‐intricate facial displays and have some of the most‐complex facial musculature of all mammals. Most of the earlier ideas of primate mimetic musculature, involving its function in facial displays and its evolution, were essentially linear “scala natural” models of increasing complexity. More‐recent work has challenged these ideas, suggesting that ecological factors and social (...) systems have played a much larger role in explaining the diversity of structures than previously believed. The present review synthesizes the evidence from gross muscular, microanatomical, behavioral and neurobiological studies in order to provide a preliminary analysis of the factors responsible for the evolution of primate facial musculature with comparisons to general mammals. In addition, the unique structure, function and evolution of human mimetic musculature are discussed, along with the potential influential roles of human speech and eye gaze. BioEssays 30:212–225, 2008. © 2008 Wiley Periodicals, Inc. (shrink)

This article aims to determine to what extent photographic practices in psychology, psychiatry and physiology contributed to the definition of the external bodily signs of passions and emotions in the second half of the 19th century in France. Bridging the gap between recent research in the history of emotions and photographic history, the following analyses focus on the photographic production of scientists and photographers who made significant contributions to the study of expressions and gestures, namely Duchenne de Boulogne, Charles Darwin, (...) Paul Richer and Albert Londe. This article argues that photography became a key technology in their works due to the adequateness of the exposure time of different cameras to the duration of the bodily manifestations to be recorded, and that these uses constituted facial expressions and bodily gestures as particular objects for the scientific study. (shrink)

Many nonhuman animals produce facial expressions which sometimes bear clear resemblance to the facial expressions seen in humans. An understanding of this evolutionary continuity between species, and how this relates to social and ecological variables, can help elucidate the meaning, function, and evolution of facial expression. This aim, however, requires researchers to overcome the theoretical and methodological differences in how human and nonhuman facial expressions are approached. Here, we review the literature relating to nonhuman (...) class='Hi'>facial expressions and suggest future directions that could facilitate a better understanding of facial expression within an evolutionary context. (shrink)

We review evidence that language is involved in the establishment and maintenance of adult categories of facial expressions of emotion. We argue that individual and group differences in facial expression interpretation are too great for a fully specified system of categories to be universal and hardwired. Variations in expression categorization, across individuals and groups, favor a model in which an initial “core” system recognizes only the grouping of positive versus negative emotional expressions. The subsequent development of (...) a rich representational structure may require the integration of a verbal categorization system with a perceptual processing system that is category-agnostic. Such a model may reconcile many strands of apparently conflicting behavioral, physiological, and neuroscience evidence concerning our understanding of facial expressions of emotion. (shrink)

Facial expressions are a vital way for humans to show their perceived emotions. It is convenient for detecting and recognizing expressions or micro-expressions by annotating a lot of data in deep learning. However, the study of video-based expressions or micro-expressions requires that coders have professional knowledge and be familiar with action unit coding, leading to considerable difficulties. This paper aims to alleviate this situation. We deconstruct facial muscle movements from the motor cortex and systematically sort out the relationship (...) among facial muscles, AU, and emotion to make more people understand coding from the basic principles: We derived the relationship between AU and emotion based on a data-driven analysis of 5,000 images from the RAF-AU database, along with the experience of professional coders.We discussed the complex facial motor cortical network system that generates facial movement properties, detailing the facial nucleus and the motor system associated with facial expressions.The supporting physiological theory for AU labeling of emotions is obtained by adding facial muscle movements patterns.We present the detailed process of emotion labeling and the detection and recognition of AU.Based on the above research, the video's coding of spontaneous expressions and micro-expressions is concluded and prospected. (shrink)

We discuss here one of our projects, aimed at developing an automatic facial expression interpreter, mainly in terms of signaled emotions. We present some of the relevant findings on facial expressions from cognitive science and psychology that can be understood by and be useful to researchers in Human-Computer Interaction and Artificial Intelligence. We then give an overview of HCI applications involving automated facial expression recognition, we survey some of the latest progresses in this area reached (...) by various approaches in computer vision, and we describe the design of our facial expression recognizer. We also give some background knowledge about our motivation for understanding facial expressions and we propose an architecture for a multimodal intelligent interface capable of recognizing and adapting to computer users' affective states. Finally, we discuss current interdisciplinary issues and research questions which will need to be addressed for further progress to be made in the promising area of computational facial expression recognition. (shrink)

Facial expressions of pain may be best conceptualized as an example of an evolved propensity to communicate distress, rather than as a distinct category of facial expression. The operant model goes beyond the evolutionary account, as it can explain how the (facial) expression of pain can become maladaptive as a result of its capability to elicit attention and caring behavior in the observer.

This article reports results from a program that produces high‐quality animation of facial expressions and head movements as automatically as possible in conjunction with meaning‐based speech synthesis, including spoken intonation. The goal of the research is as much to test and define our theories of the formal semantics for such gestures, as to produce convincing animation. Towards this end, we have produced a high‐level programming language for three‐dimensional (3‐D) animation of facial expressions. We have been concerned primarily with (...) expressions conveying information correlated with the intonation of the voice: This includes the differences of timing, pitch, and emphasis that are related to such semantic distinctions of discourse as “focus,”“topic,” and “comment,”“theme” and “rheme,” or “given” and “new” information. We are also interested in the relation of affect or emotion to facial expression. Until now, systems have not embodied such rule‐governed translation from spoken utterance meaning to facial expressions. Our system embodies rules that describe and coordinate these relations: intonation/information, intonation/affect, and facial expressions/affect. A meaning representation includes discourse information: What is contrastive/background information in the given context, and what is the “topic” or “theme” of the discourse? The system maps the meaning representation into how accents and their placement are chosen, how they are conveyed over facial expression, and how speech and facial expressions are coordinated. This determines a sequence of functional groups: lip shapes, conversational signals, punctuators, regulators, and manipulators. Our algorithms then impose synchrony, create coarticulation effects, and determine affectual signals, eye and head movements. The lowest level representation is the Facial Action Coding System (FACS), which makes the generation system portable to other facial models. (shrink)

Automatic facial expression recognition has now advanced to the point that we are able to apply it to spontaneous expressions. The automated tools enable new research activity in cognitive neuroscience, psychiatry, education, human–machine communication, and human social dynamics. Moreover, automated facial expression analysis enables investigations into facial expression dynamics that were previously intractable by human coding because of the time required to code intensity changes. This article provides an overview on the state of the (...) art in computer vision approaches to facial expression recognition, including methods for characterizing expression dynamics. It then discusses behavioral studies that have employed automatic facial expression recognition to learn new information about the relationships of facial expression to internal state, and reviews the first generation of applications in learning and education that take advantage of the real-time expression signal. (shrink)

That facial expressions are universal emotion signals has been supported by observers agreeing on the emotion mimed by actors. We show that actors can mime a diverse range of states: emotions, cognitions, physical states, and actions. English, Hindi, and Malayalam speakers viewed 25 video clips and indicated the state conveyed. Within each language, at least 23 of the 25 clips were recognised above chance and base rate. Facial expressions of emotions are not special in their recognisability, and it (...) is miming that may be the universal human ability. (shrink)

Facial responses to pain might be the result of evolution but Williams' interesting “Just So” story provides no convincing evidence for her hypothesis. Contrary to her hope, casting facial action in an evolutionary perspective will probably not reduce the common practice of health care professionals blaming patients for their problems; instead, it may discourage appropriate treatment.

Emotional facial expressions can inform researchers about an individual's emotional state. Recent technological advances open up new avenues to automatic Facial Expression Recognition. Based on machine learning, such technology can tremendously increase the amount of processed data. FER is now easily accessible and has been validated for the classification of standardized prototypical facial expressions. However, applicability to more naturalistic facial expressions still remains uncertain. Hence, we test and compare performance of three different FER systems with (...) human emotion recognition for standardized posed facial expressions and for non-standardized acted facial expressions. For the standardized images, all three systems classify basic emotions accurately and they are mostly on par with human raters. For the non-standardized stimuli, performance drops remarkably for all three systems, but Azure still performs similarly to humans. In addition, all systems and humans alike tend to misclassify some of the non-standardized emotional facial expressions as neutral. In sum, emotion recognition by automated facial expression recognition can be an attractive alternative to human emotion recognition for standardized and non-standardized emotional facial expressions. However, we also found limitations in accuracy for specific facial expressions; clearly there is need for thorough empirical evaluation to guide future developments in computer vision of emotional facial expressions. (shrink)

The proposal that there are specific adaptations for the expression and detection of pain appears premature on both conceptual and empirical grounds. We discuss criteria for the validation of a pain facial expression. We also describe recent findings from our lab on coping styles and pain expression, which illustrate the importance of considering individual differences when proposing evolutionary explanations.

Honest signalling theory suggests that humans and chimpanzees can extract socially relevant information relating to personality from the faces of their conspecifics. Humans are also able to extract information from chimpanzees’ faces. Here, we examine whether personality characteristics of chimpanzees, including measures of psychopathy, can be discerned based purely on facial morphology in photographs. Twenty-one chimpanzees were given naïve and expert personality ratings on the Ten Item Personality Inventory (TIPI) and the Chimpanzee Triarchic Model of Psychopathy (CHMP-Tri) before and (...) following behavioural observations. Characteristics relating to openness, conscientiousness, extraversion, and disinhibition could be distinguished from the faces of chimpanzees. Individuals higher on disinhibition have lower scores on conscientiousness and emotional stability and higher scores on extraversion, while those higher on meanness have lower conscientiousness and agreeableness. Facial expressions are linked to personality traits present in the TIPI and CHMP-Tri models: the Relaxed Face and the Grooming Face were displayed more by chimpanzees higher on agreeableness, whereas the Compressed Lips Face was observed more in those individuals higher on boldness, and the Full Open Grin was displayed more by chimpanzees higher on extraversion but lower on emotional stability and conscientiousness. Facial expressions were also found to be associated with particular behavioural contexts, namely the Grooming Face in affiliative contexts and the Relaxed and Relaxed Open Mouth Faces in neutral contexts. Dominant chimpanzees display higher levels of boldness and more Compressed Lips Faces, Relaxed Open Mouth Faces, and Grooming Faces than subordinate individuals. These findings support and extend evidence for an honest signalling system and a personality structure shared between humans and chimpanzees. Future research could further explore how personality is conveyed through the face, perhaps through more than just singular aspects of character, and maybe reflecting what chimpanzees themselves are able to do. (shrink)

Evidence on the coherence between emotion and facial expression in adults from laboratory experiments is reviewed. High coherence has been found in several studies between amusement and smiling; low to moderate coherence between other positive emotions and smiling. The available evidence for surprise and disgust suggests that these emotions are accompanied by their “traditional” facial expressions, and even components of these expressions, only in a minority of cases. Evidence concerning sadness, anger, and fear is very limited. For (...) sadness, one study suggests that high emotion–expression coherence may exist in specific situations, whereas for anger and fear, the evidence points to low coherence. Insufficient emotion intensity and inhibition of facial expressions seem unable to account for the observed dissociations between emotion and facial expression. (shrink)

This paper proposes that human expression of pain in the presence or absence of caregivers, and the detection of pain by observers, arises from evolved propensities. The function of pain is to demand attention and prioritise escape, recovery, and healing; where others can help achieve these goals, effective communication of pain is required. Evidence is reviewed of a distinct and specific facial expression of pain from infancy to old age, consistent across stimuli, and recognizable as pain by (...) observers. Voluntary control over amplitude is incomplete, and observers can better detect pain that the individual attempts to suppress rather than amplify or simulate. In many clinical and experimental settings, the facial expression of pain is incorporated with verbal and nonverbal vocal activity, posture, and movement in an overall category of pain behaviour. This is assumed by clinicians to be under operant control of social contingencies such as sympathy, caregiving, and practical help; thus, strong facial expression is presumed to constitute an attempt to manipulate these contingencies by amplification of the normal expression. Operant formulations support skepticism about the presence or extent of pain, judgments of malingering, and sometimes the withholding of caregiving and help. To the extent that pain expression is influenced by environmental contingencies, however, “amplification” could equally plausibly constitute the release of suppression according to evolved contingent propensities that guide behaviour. Pain has been largely neglected in the evolutionary literature and the literature on expression of emotion, but an evolutionary account can generate improved assessment of pain and reactions to it. Key Words: adaptation; evolutionary psychology; facial expression; pain. (shrink)

The experience of pain appears to be associated, from early infancy and across pain stimuli, with a consistent facial expression in humans. A social function is proposed for this: the communication of pain and the need for help to observers, to whom information about danger is of value, and who may provide help within a kin or cooperative relationship. Some commentators have asserted that the evidence is insufficient to account for the consistency of the face, as judged by (...) technical means or in the perceptions of observers, or that facial expression is epiphenomenal to a gross behavioural defensive response to pain. The major criticism is that it is unnecessary to invoke evolutionary mechanisms beyond the emergence of an unconditioned facial response to pain in neonates, subsequently shaped and maintained by instrumental and social reinforcement throughout life. These criticisms are addressed, acknowledging the need for further data to address some, and elaborating the areas in which evolutionary and operant mechanisms would predict different behavioural interactions, rather than acting synergistically. Several supportive commentaries propose extending evolutionarily-based hypotheses to sex differences, the complexities of others' responses within the social relationship, and the role of empathy. A number of commentators provided valuable suggestions for experimental paradigms or methodological issues. Overall, addressing these issues indicates the need for further conceptual development and for collection of data specifically in relation to these hypotheses. (shrink)

The experience of pain appears to be associated, from early infancy and across pain stimuli, with a consistent facial expression in humans. A social function is proposed for this: the communication of pain and the need for help to observers, to whom information about danger is of value, and who may provide help within a kin or cooperative relationship. Some commentators have asserted that the evidence is insufficient to account for the consistency of the face, as judged by (...) technical means or in the perceptions of observers, or that facial expression is epiphenomenal to a gross behavioural defensive response to pain. The major criticism is that it is unnecessary to invoke evolutionary mechanisms beyond the emergence of an unconditioned facial response to pain in neonates, subsequently shaped and maintained by instrumental and social reinforcement throughout life. These criticisms are addressed, acknowledging the need for further data to address some, and elaborating the areas in which evolutionary and operant mechanisms would predict different behavioural interactions, rather than acting synergistically. Several supportive commentaries propose extending evolutionarily-based hypotheses to sex differences, the complexities of others' responses within the social relationship, and the role of empathy. A number of commentators provided valuable suggestions for experimental paradigms or methodological issues. Overall, addressing these issues indicates the need for further conceptual development and for collection of data specifically in relation to these hypotheses. (shrink)

ABSTRACTUsing faces as the priming stimuli, the present study explored the influence of facial expressions on the activation of gender stereotypes using a lexical decision paradigm. Experiment 1 explored the activation of gender stereotypes when the facial primes contained only gender information. The results showed that gender stereotypes were activated. In Experiment 2, the facial primes contained both gender category and expression information. The results indicated that gender stereotypes were not activated. Experiment 3 required the participants (...) to make emotion, gender, or impression decisions concerning the facial primes before the lexical decision task. The results showed that gender stereotypes were not activated in emotion and impression decisions conditions, whereas stereotypes were activated in gender decisions condition. These finding suggest that facial expressions can inhibit automatic activation of gender stereotypes, unless the perceivers perform gender categorization processing to prime f... (shrink)

With a few yet increasing number of exceptions, the cognitive sciences enthusiastically endorsed the idea that there are basic facial expressions of emotions that are created by specific configurations of facial muscles. We review evidence that suggests an inherent role for context in emotion perception. Context does not merely change emotion perception at the edges; it leads to radical categorical changes. The reviewed findings suggest that configurations of facial muscles are inherently ambiguous, and they call for a (...) different approach towards the understanding of facial expressions of emotions. Prices of sticking with the modal view, and advantages of an expanded view, are succinctly reviewed. (shrink)

We discuss here one of our projects, aimed at developing an automatic facial expression interpreter, mainly in terms of signaled emotions. We present some of the relevant findings on facial expressions from cognitive science and psychology that can be understood by and be useful to researchers in Human-Computer Interaction and Artificial Intelligence. We then give an overview of HCI applications involving automated facial expression recognition, we survey some of the latest progresses in this area reached (...) by various approaches in computer vision, and we describe the design of our facial expression recognizer. We also give some background knowledge about our motivation for understanding facial expressions and we propose an architecture for a multimodal intelligent interface capable of recognizing and adapting to computer users¿ affective states. Finally, we discuss current interdisciplinary issues and research questions which will need to be addressed for further progress to be made in the promising area of computational facial expression recognition. (shrink)

We used multiple methods to examine two questions about emotion and culture: (1) Which facial expressions are recognised cross-culturally; and (2) does the “forced-choice” method lead to spurious findings of universality? Forty participants in the US and 40 in India were shown 14 facial expressions and asked to say what had happened to cause the person to make the face. Analyses of the social situations given and of the affect words spontaneously used showed high levels of recognition for (...) most of the expressions. A subsequent forced-choice task using the same faces confirmed these findings. Analysis of the pattern of magnitude, discreteness, and similarity of responses across cultures and expressions led to the conclusion that there is no neat distinction between cross-culturally recognisable and nonrecognisable expressions. Results are better described as a gradient of recognition. (shrink)

A substantial body of research has established that even when we are not consciously aware of the faces of others we are nevertheless sensitive to, and impacted by their facial expression. In this paper, we consider this body of research from a new perspective by examining the functions of unconscious perception revealed by these studies. A consideration of the literature from this perspective highlights that existing research methods are limited when it comes to revealing possible functions of unconscious (...) perception. The critical shortcoming is that in all of the methods, the perceived facial expression remains outside of awareness. This is a problem because there are good reasons to believe that one important function of unconsciously perceived negative faces is to attract attention so that they are consciously perceived; such conscious perception, however, is never allowed with existing methodologies. We discuss recent studies of emotional face perception under conditions of visual search that address this issue directly. Further, we suggest that methodologies that do not examine cognitive processes as they occur in more natural settings may result in fundamental misunderstandings of human cognition. (shrink)

According to a common sense theory, facial expressions signal specific emotions to people of all ages and therefore provide children easy access to the emotions of those around them. The evidence, however, does not support that account. Instead, children’s understanding of facial expressions is poor and changes qualitatively and slowly over the course of development. Initially, children divide facial expressions into two simple categories (feels good, feels bad). These broad categories are then gradually differentiated until an adult (...) system of discrete categories is achieved, likely in the teen years. Children’s understanding of most specific emotions begins not with facial expressions, but with their understanding of the emotion’s antecedents and behavioral consequences. (shrink)