How do people learn multisensory, or amodal, representations, and what consequences do these representations have for perceptual performance? We address this question by performing a rational analysis of the problem of learning multisensory representations. This analysis makes use of a Bayesian nonparametric model that acquires latent multisensory features that optimally explain the unisensory features arising in individual sensory modalities. The model qualitatively accounts for several important aspects of multisensory perception: (a) it integrates information from multiple sensory sources in such a (...) way that it leads to superior performances in, for example, categorization tasks; (b) its performances suggest that multisensory training leads to better learning than unisensory training, even when testing is conducted in unisensory conditions; (c) its multisensory representations are modality invariant; and (d) it predicts ‘‘missing” sensory representations in modalities when the input to those modalities is absent. Our rational analysis indicates that all of these aspects emerge as part of the optimal solution to the problem of learning to represent complex multisensory environments. (shrink)

ABSTRACTIt has recently been demonstrated that the reported tastes/flavours of food/beverages can be modulated by means of external visual and auditory stimuli such as typeface, shapes, and music. The present study was designed to assess the role of the emotional valence of the product-extrinsic stimuli in such crossmodal modulations of taste. Participants evaluated samples of mixed fruit juice whilst simultaneously being presented with auditory or visual stimuli having either positive or negative valence. The soundtracks had either been harmonised with consonant (...) or dissonant musical intervals. The visual stimuli consisted of images of emotional faces from the International Affective Picture System with valence ratings matched to the soundtracks. Each juice sample was rated on two computer-based scales: One anchored with the words sour and sweet, while the other scale required hedonic ratings. Those participants who tasted the juice sample while presented with... (shrink)

Imagine a scenario where you are cooking and suddenly, the contents of the pot start to come out, and the oven bell rings. You would have to stop what you are doing and start responding to the changing demands, switching between different objects, operations and mental sets. This ability is known as cognitive flexibility. Now, add to this scenario a strong emotional atmosphere that invades you as you spontaneously recall a difficult situation you had that morning. How would you behave? (...) Recent studies suggest that emotional states do modulate cognitive flexibility, but these findings are still controversial. Moreover, there is a lack of evidence regarding the underlying brain processes. The purpose of the present study was, therefore, to examine such interaction while monitoring changes in ongoing cortical activity using EEG. In order to answer this question, we used two musical stimuli to induce emotional states (positive/high arousal/open stance and negative/high arousal/closed stance). Twenty-nine participants performed two blocks of the Madrid Card Sorting Task (MCST) in a neutral silence condition and then four blocks while listening to the counterbalanced musical stimuli. To explore this interaction, we used a combination of first-person (micro-phenomenological interview) and third-person (behavior and EEG) approaches. Our results show that compared to the positive stimuli and silence condition, negative stimuli decrease reaction times (RTs) for the shift signal. Our data show that the valance of the first emotional block is determinant in the RTs of the subsequent blocks. Additionally, the analysis of the micro-phenomenological interview and the integration of first- and third-person data show that the emotional disposition generated by the music could facilitate task performance for some participants or hamper it for others, independently of its emotional valence. When the emotional disposition hampered task execution, RTs were slower, and the P300 potential showed a reduced amplitude compared to the facilitated condition. These findings show that the interaction between emotion and cognitive flexibility is more complex than previously thought and points to a new way of understanding the underlying mechanisms by incorporating an in-depth analysis of individual subjective experience. (shrink)

Prompted by the development of the Theory of Challenge and Threat States in Athletes (Jones et al, 2009), recent years has witnessed a considerable increase in research examining challenge and threat in sport. This manuscript provides a critical review of the literature examining challenge and threat in sport, tracing its historical development and some of the current empirical ambiguities. In an attempt to reconcile some of these ambiguities, and utilising neurobiological evidence associated with approach- and avoidance-motivation (cf. Elliot & Covington, (...) 2001), this paper draws upon the Evaluative Space Model (ESM: Cacioppo et al., 1997) and considers the implications for understanding challenge and threat in sport. For example, rather than see challenge and threat as opposite ends of a single bipolar continuum, the ESM implies that individuals could be (1) challenged, (2) threatened, (3) challenged and threatened, or (4) neither challenged or threatened by a particular stimulus. From this perspective it could be argued that the appraisal of some sport situations as both challenging and threatening could be advantageous, whereas the current literature seems to imply that the appraisal of stress as a threat is maladaptive for performance. In drawing upon the ESM, we outline the Evaluative Space Approach to Challenge and Threat and provide several testable hypotheses for advancing understanding of challenge and threat (in sport). We also outline a number of cardiovascular and experiential measures that can be used to examine these hypotheses. In sum, this paper provides a significant theoretical, empirical and practical contribution to our understanding of challenge and threat (in sport). (shrink)

We examined whether facial expressions of performers influence the emotional connotations of sung materials, and whether attention is implicated in audio-visual integration of affective cues. In Experiment 1, participants judged the emotional valence of audio-visual presentations of sung intervals. Performances were edited such that auditory and visual information conveyed congruent or incongruent affective connotations. In the single-task condition, participants judged the emotional connotation of sung intervals. In the dual-task condition, participants judged the emotional connotation of intervals while performing a secondary (...) task. Judgements were influenced by melodic cues and facial expressions and the effects were undiminished by the secondary task. Experiment 2 involved identical conditions but participants were instructed to base judgements on auditory information alone. Again, facial expressions influenced judgements and the effect was undiminished by the secondary task. The results suggest that visual aspects of music performance are automatically and preattentively registered and integrated with auditory cues. (shrink)

Social-rank cues communicate social status or social power within and between groups. Information about social-rank is fluently processed in both visual and auditory modalities. So far, the investigation on the processing of social-rank cues has been limited to studies in which information from a single modality was assessed or manipulated. Yet, in everyday communication, multiple information channels are used to express and understand social-rank. We sought to examine the voluntary nature of processing of facial and vocal signals of social-rank using (...) a cross-modal Stroop task. In two experiments, participants were presented with face-voice pairs that were either congruent or incongruent in social-rank. Participants’ task was to label face social dominance while ignoring the voice, or label voice social dominance while ignoring the face. In both experiments, we found that face-voice incongruent stimuli were processed more slowly and less accurately than were the congruent stimuli in the face-attend and the voice-attend tasks, exhibiting classical Stroop-like effects. These findings are consistent with the functioning of a social-rank bio-behavioural system which consistently and automatically monitors one’s social standing in relation to others and uses that information to guide behaviour. (shrink)

This article explores interpersonal functions of emotional mimicry under the absence versus the presence of visual contact between the interacting partners. We review relevant literature and stress that previous studies on the role of emotional mimicry were focused on imitative responses to facial displays. We also show that the rules explaining why people mimic facial expressions may be inapplicable when visual signals are unavailable (e.g., people attending an online meeting have their cameras off). Overall, our review suggests that emotional mimicry (...) functionally adapts to whether the perceiver and the expresser can see each other. We, therefore, argue that blocking visual contact between them may provide insight into emotional mimicry's social functions, thereby clarifying its role in fostering affiliation and emotional understanding. (shrink)

Facial emotional recognition is something used often in our daily lives. How does the brain process the face search? Can taste modify such a process? This study employed two tastes to investigate the cross-modal interaction between taste and emotional face recognition. The behavior responses and the event-related potential were applied to analyze the interaction between taste and face processing. Behavior data showed that when detecting a negative target face with a positive face as a distractor, the participants perform the task (...) faster with an acidic taste than with sweet. No interaction effect was observed with correct response ratio analysis. The early and mid-stage [early posterior negativity ] components have shown that sweet and acidic tastes modified the ERP components with the affective face search process in the ERP results. No interaction effect was observed in the late-stage component. Our data have extended the understanding of the cross-modal mechanism and provided electrophysiological evidence that affective facial processing could be influenced by sweet and acidic tastes. (shrink)

The conflicting findings from the few studies conducted with regard to gender differences in the recognition of vocal expressions of emotion have left the exact nature of these differences unclear. Several investigators have argued that a comprehensive understanding of gender differences in vocal emotion recognition can only be achieved by replicating these studies while accounting for influential factors such as stimulus type, gender-balanced samples, number of encoders, decoders and emotional categories. This study aimed to account for these factors by investigating (...) whether emotion recognition from vocal expressions differs as a function of both listeners’ and speakers’ gender. A total of N = 290 participants were randomly and equally allocated to two groups. One group listened to words and pseudo-words, while the other group listened to sentences and affect bursts. Participants were asked to categorize the stimuli with respect to the expressed emotions in a fixed-choice response format. Overall, females were more accurate than males when decoding vocal emotions, however, when testing for specific emotions these differences were small in magnitude. Speakers’ gender had a significant impact on how listeners’ judged emotions from the voice. The group listening to words and pseudo-words had higher identification rates for emotions spoken by male than by female actors, whereas in the group listening to sentences and affect bursts the identification rates were higher when emotions were uttered by female than male actors. The mixed pattern for emotion-specific effects, however, indicates that, in the vocal channel, the reliability of emotion judgments is not systematically influenced by speakers’ gender and the related stereotypes of emotional expressivity. Together, these results extend previous findings by showing effects of listeners’ and speakers’ gender on the recognition of vocal emotions. They stress the importance of distinguishing these factors to explain recognition ability in the processing of emotional prosody. (shrink)

This study explored a dimensional approach to vocal expression of emotion. Actors vocally portrayed emotions (anger, disgust, fear, happiness, sadness) with weak and strong emotion intensity. Listeners (30 university students and 6 speech experts) rated each portrayal on four emotion dimensions (activation, valence, potency, emotion intensity). The portrayals were also acoustically analysed with respect to 20 vocal cues (e.g., speech rate, voice intensity, fundamental frequency, spectral energy distribution). The results showed that: (a) there were distinct patterns of ratings of activation, (...) valence, and potency for the different emotions; (b) all four emotion dimensions were correlated with several vocal cues; (c) listeners' ratings could be successfully predicted from the vocal cues for all dimensions except valence; and (d) the intensity dimension was positively correlated with the activation dimension in the listeners' ratings. (shrink)

Results from studies on gender differences in emotion recognition vary, depending on the types of emotion and the sensory modalities used for stimulus presentation. This makes comparability between different studies problematic. This study investigated emotion recognition of healthy participants (N = 84; 40 males; ages 20 to 70 years), using dynamic stimuli, displayed by two genders in three different sensory modalities (auditory, visual, audio-visual) and five emotional categories. The participants were asked to categorise the stimuli on the basis of their (...) nonverbal emotional content (happy, alluring, neutral, angry, and disgusted). Hit rates and category selection biases were analysed. Women were found to be more accurate in recognition of emotional prosody. This effect was partially mediated by hearing loss for the frequency of 8,000 Hz. Moreover, there was a gender-specific selection bias for alluring stimuli: Men, as compared to women, chose “alluring” more often when a stimulus was presented by a woman as compared to a man. (shrink)

Two experiments examined how affective values from visual and auditory modalities are integrated. Experiment 1 paired music and videos drawn from three levels of valence while holding arousal constant. Experiment 2 included a parallel combination of three levels of arousal while holding valence constant. In each experiment, participants rated their affective states after unimodal and multimodal presentations. Experiment 1 revealed a congruency effect in which stimulus combinations of the same extreme valence resulted in more extreme state ratings than component stimuli (...) presented in isolation. An interaction between music and video valence reflected the greater influence of negative affect. Video valence was found to have a significantly greater effect on combined ratings than music valence. The pattern of data was explained by a five parameter differential weight averaging model that attributed greater weight to the visual modality and increased weight with decreasing values of valence. Experiment 2 revealed a congruency effect only for high arousal combinations and no interaction effects. This pattern was explained by a three parameter constant weight averaging model with greater weight for the auditory modality and a very low arousal value for the initial state. These results demonstrate key differences in audiovisual integration between valence and arousal. (shrink)

A substantial body of research has established that even when we are not consciously aware of the faces of others we are nevertheless sensitive to, and impacted by their facial expression. In this paper, we consider this body of research from a new perspective by examining the functions of unconscious perception revealed by these studies. A consideration of the literature from this perspective highlights that existing research methods are limited when it comes to revealing possible functions of unconscious perception. The (...) critical shortcoming is that in all of the methods, the perceived facial expression remains outside of awareness. This is a problem because there are good reasons to believe that one important function of unconsciously perceived negative faces is to attract attention so that they are consciously perceived; such conscious perception, however, is never allowed with existing methodologies. We discuss recent studies of emotional face perception under conditions of visual search that address this issue directly. Further, we suggest that methodologies that do not examine cognitive processes as they occur in more natural settings may result in fundamental misunderstandings of human cognition. (shrink)

Social interactions in daily life necessitate the integration of social signals from different sensory modalities. In the aging literature, it is well established that the recognition of emotion in facial expressions declines with advancing age, and this also occurs with vocal expressions. By contrast, crossmodal integration processing in healthy aging individuals is less documented. Here, we investigated the age-related effects on emotion recognition when faces and voices were presented alone or simultaneously, allowing for crossmodal integration. In this study, 31 young (...) adults (M = 25.8 years) and 31 older adults (M = 67.2 years) were instructed to identify several basic emotions (happiness, sadness, anger, fear, disgust) and a neutral expression, which were displayed as visual (facial expressions), auditory (nonverbal affective vocalizations) or crossmodal (simultaneous, congruent facial and vocal affective expressions) stimuli. The results showed that older adults performed slower and worse than younger adults at recognizing negative emotions from isolated faces and voices. In the crossmodal condition, although slower, older adults were as accurate as younger except for anger. Importantly, additional analyses using the ‘race model’ demonstrate that older adults benefited to the same extent as younger adults from the combination of facial and vocal emotional stimuli. These results help explain some conflicting results in the literature and may clarify emotional abilities related to daily life that are partially spared among older adults. (shrink)