According to the shared signal hypothesis the impact of facial expressions on emotion processing partially depends on whether the gaze is directed toward or away from the observer. In autism spectrum disorder several aspects of face processing have been found to be atypical, including attention to eye gaze and the identification of emotional expressions. However, there is little research on how gaze direction affects emotional expression processing in typically developing individuals and in those with ASD. This question is investigated here (...) in two multimodal experiments. Experiment 1 required processing eye gaze direction while faces differed in emotional expression. Forty-seven children participated. Their Autism Diagnostic Observation Schedule scores ranged from 0 to 6 in the experiment. Event-related potentials were sensitive to gaze direction and emotion, but emotion processing did not depend on gaze direction. However, for angry faces the gaze direction effect on the N170 amplitude, as typically observed in TD individuals, diminished with increasing ADOS score. For neutral expressions this correlation was not significant. Experiment 2 required explicit emotion classifications in a facial emotion composite task while eye gaze was manipulated incidentally. A group of 22 children with ASD was compared to a propensity score-matched group of TD children. The same comparison was carried out for a subgroup of nine children with ASD who were less trained in social cognition, according to clinician’s report. The ASD group performed overall worse in emotion recognition than the TD group, independently of emotion or gaze direction. However, for disgust expressions, eye tracking data revealed that TD children fixated relatively longer on the eyes of the stimulus face with a direct gaze as compared with averted gaze. In children with ASD we observed no such modulation of fixation behavior as a function of gaze direction. Overall, the present findings from ERPs and eye tracking confirm the hypothesis of an impaired sensitivity to gaze direction in children with ASD or elevated autistic traits, at least for specific emotions. Therefore, we conclude that multimodal investigations of the interaction between emotional processing and stimulus gaze direction are promising to understand the characteristics of individuals differing along the autism trait dimension. (shrink)

Recent advances in the field of statistical learning have established that learners are able to track regularities of multimodal stimuli, yet it is unknown whether the statistical computations are performed on integrated representations or on separate, unimodal representations. In the present study, we investigated the ability of adults to integrate audio and visual input during statistical learning. We presented learners with a speech stream synchronized with a video of a speaker’s face. In the critical condition, the visual (e.g. /gi/) (...) and auditory (e.g. /mi/) signals were occasionally incongruent, which we predicted would produce the McGurk illusion, resulting in the perception of an audiovisual syllable (e.g. /ni/). In this way, we used the McGurk illusion to manipulate the underlying statistical structure of the speech streams, such that perception of these illusory syllables facilitated participants’ ability to segment the speech stream. Our results therefore demonstrate that participants can integrate audio and visual input to perceive the McGurk illusion during statistical learning. We interpret our findings as support for modality-interactive accounts of statistical learning. (shrink)

One way to characterize the special relation that one has to one's own body is to say that only one's body appears to one from the inside. Although widely accepted, the nature of this specific experiential mode of presentation of the body is rarely spelled out. Most definitions amount to little more than lists of the various body senses (including senses of posture, movement, heat, pressure, and balance). It is true that body senses provide a kind of informational access to (...) one's own body, which one has to no other bodies, by contrast to external senses like vision, which can take many bodies as their object. But a theory of bodily awareness needs to take into account recent empirical evidence that indicates that bodily awareness is infected by a plague of multisensory effects, regardless of any dichotomy between body senses and external senses. Here I will argue in favour of a multimodal conception of bodily awareness. I will show that the body senses fail to fully account for the content of bodily experiences. I will then propose that vision helps compensate for the insufficiencies of the body senses in people who can see. I will finally argue that the multimodality of bodily experiences does not prevent privileged access to one's body. (shrink)

This paper proposes a methodological synthesis in order to study multimodal media discourse and argumentation in the context of the so-called ‘refugee crisis’ in Greece. It follows the framework of Multimodal Critical Discourse Analysis, integrating this with argumentation studies, with a particular emphasis on the analysis of inference. Our data come from the Greek newspapers Kathimerini and Ta Nea. We contend that the proposed methodological synergy enables scrutiny of (a) racist conceptualizations cultivated by the representation of migrants and (...) refugees in headlines and photographs in newspaper articles along with (b) the argumentative potential that is implicitly sustained in these multimodal representations. (shrink)

As humans interact in the world, they often orient one another's attention to objects through the use of spoken demonstrative expressions and head and/or hand movements to point to the objects. Although indicating behaviors have frequently been studied in lab settings, we know surprisingly little about how demonstratives and pointing are used to coordinate attention in large-scale space and in natural contexts. This study investigates how speakers of Quiahije Chatino, an indigenous language of Mexico, use demonstratives and pointing to give (...) directions to named places in large-scale space across multiple scales. The results show that the use and coordination of demonstratives and pointing change as the scale of search space for the target grows. At larger scales, demonstratives and pointing are more likely to occur together, and the two signals appear to manage different aspects of the search for the target: demonstratives orient attention primarily to the gesturing body, while pointing provides cues for narrowing the search space. These findings underscore the distinct contributions of speech and gesture to the linguistic composite, while illustrating the dynamic nature of their interplay.s in Spanish and Quiahije Chatino are provided as appendices.Se incluyen como apéndices resúmenes en español y en el chatino de San Juan Quiahije. SonGktyiCreCinH, ngyaqCskaEktyiCnoEndaHsonBnaFngaJnoIngyaqCloEktyiCreC, ngyaqCranFchaqExlyaKqoEchaqFjnyaJnoAndywiqArenqAKchinAKyqyaC. (shrink)

The learning style myth is a commonly held myth that matching instruction to a student's “learning style” will result in improved learning, while providing mismatched instruction will result in suboptimal learning. The present study used a short online reasoning exercise about the efficacy of multimodal instruction to investigate the nature of learning styles beliefs. We aimed to: understand how learning style beliefs interact with beliefs about multimodal learning; characterize the potential complexity of learning style beliefs and understand how (...) this short exercise might influence endorsements of learning styles. Many participants who believed in the learning style myth supported the efficacy of multimodal learning, and many were willing to revise their belief in the myth after the exercise. Personal experiences and worldviews were commonly cited as reasons for maintaining beliefs in learning styles. Findings reveal the complexity of learning style beliefs, and how they interact with evidence in previously undocumented ways. (shrink)

By the time they reach early adulthood, English speakers are familiar with the meaning of thousands of words. In the last decades, computational simulations known as distributional semantic models have demonstrated that it is possible to induce word meaning representations solely from word co-occurrence statistics extracted from a large amount of text. However, while these models learn in batch mode from large corpora, human word learning proceeds incrementally after minimal exposure to new words. In this study, we run a set (...) of experiments investigating whether minimal distributional evidence from very short passages suffices to trigger successful word learning in subjects, testing their linguistic and visual intuitions about the concepts associated with new words. After confirming that subjects are indeed very efficient distributional learners even from small amounts of evidence, we test a DSM on the same multimodal task, finding that it behaves in a remarkable human-like way. We conclude that DSMs provide a convincing computational account of word learning even at the early stages in which a word is first encountered, and the way they build meaning representations can offer new insights into human language acquisition. (shrink)

Pantomime has long been considered distinct from co-speech gesture. It has therefore been argued that pantomime cannot be part of gesture-speech integration. We examine pantomime as distinct from silent gesture, focusing on non-co-speech gestures that occur in the midst of children’s spoken narratives. We propose that gestures with features of pantomime are an infrequent but meaningful component of a multimodal communicative strategy. We examined spontaneous non-co-speech representational gesture production in the narratives of 30 monolingual English-speaking children between the ages (...) of 8- and 11-years. We compared the use of co-speech and non-co-speech gestures in both autobiographical and fictional narratives and examined viewpoint and the use of non-manual articulators, as well as the length of responses and narrative quality. The use of non-co-speech gestures was associated with longer narratives of equal or higher quality than those using only co-speech gestures. Non-co-speech gestures were most likely to adopt character-viewpoint and use non-manual articulators. The present study supports a deeper understanding of the term pantomime and its multimodal use by children in the integration of speech and gesture. (shrink)

From the perspective of distributed cognition I will stress how abduction is essentially multimodal, in that both data and hypotheses can have a full range of verbal and sensory representations, involving words, sights, images, smells, etc., but also kinesthetic – related to the ability to sense the position and location and orientation and movement of the body and its parts – and motor experiences and other feelings such as pain, and thus all sensory modalities. The presence of kinesthetic and (...) motor aspects demonstrates that abductive reasoning can be manipulative. We can also see, in this regard, how implicit factors take part in the abductive procedure, which consequently acquires the character of a kind of “thinking through doing”. This paper further describes 1) the fact that hypotheses in science can be built through different cognitive mediators and so they can also model the same cognitive aspect in different ways; how they can be carriers/ producers of knowledge in a multimodal way; 2) the problem of the possible non-explanatory and instrumental nature of abductive reasoning and the analysis of the consequences for induction; 3) the role of manipulative abduction in building new evidence/ experiments and how they trigger smart inductive inferences. (shrink)

A persistent controversy in language evolution research has been whether language emerged in the gestural-visual or in the vocal-auditory modality. A “dialectic” solution to this age-old debate has now been gaining ground: language was fully multimodal from the start and remains so to this day. In this paper, we show this solution to be too simplistic and outline a more specific theoretical proposal, which we designate as pantomime-first. To decide between the multimodal-first and pantomime-first alternatives, we review several (...) lines of interdisciplinary evidence and complement it with a cognitive-semiotic experiment. In the study, the participants saw – and then matched to hand-drawn images – recordings of short transitive events enacted by 4 actors in two conditions: visual and multimodal. Significantly, the matching accuracy was greater in the visual than the multimodal condition, though a follow-up experiment revealed that the emotional profiles of the events enacted in the multimodal condition could be reliably detected from the sound alone. We see these results as supporting the proposed pantomime-first scenario. (shrink)

One of the main limitations of natural language‐based approaches to meaning is that they do not incorporate multimodal representations the way humans do. In this study, we evaluate how well different kinds of models account for people's representations of both concrete and abstract concepts. The models we compare include unimodal distributional linguistic models as well as multimodal models which combine linguistic with perceptual or affective information. There are two types of linguistic models: those based on text corpora and (...) those derived from word association data. We present two new studies and a reanalysis of a series of previous studies. The studies demonstrate that both visual and affective multimodal models better capture behavior that reflects human representations than unimodal linguistic models. The size of the multimodal advantage depends on the nature of semantic representations involved, and it is especially pronounced for basic‐level concepts that belong to the same superordinate category. Additional visual and affective features improve the accuracy of linguistic models based on text corpora more than those based on word associations; this suggests systematic qualitative differences between what information is encoded in natural language versus what information is reflected in word associations. Altogether, our work presents new evidence that multimodal information is important for capturing both abstract and concrete words and that fully representing word meaning requires more than purely linguistic information. Implications for both embodied and distributional views of semantic representation are discussed. (shrink)

Dancers and musicians are experts in spatial and temporal processing which allows them to coordinate movement with music. This high-level processing has been associated with structural and functional adaptation of the brain for high performance sensorimotor integration. For these integration processes, adaptation does not only take place in primary and secondary sensory and motor areas but tertiary brain areas, such as the lateral prefrontal cortex (lPFC) and the intraparietal sulcus (IPS), also provide vital resources for highly specialized performance. Here, we (...) review evidence for the role of these brain areas in multimodal training protocols and integrate these findings into a new model of sensorimotor processing in complex motor learning. (shrink)

The human experience is shaped by information from different perceptual channels, but it is still debated whether and how differential experience influences language use. To address this, we compared congenitally blind, blindfolded, and sighted people's descriptions of the same motion events experienced auditorily by all participants (i.e., via sound alone) and conveyed in speech and gesture. Comparison of blind and sighted participants to blindfolded participants helped us disentangle the effects of a lifetime experience of being blind versus the task-specific effects (...) of experiencing a motion event by sound alone. Compared to sighted people, blind people's speech focused more on path and less on manner of motion, and encoded paths in a more segmented fashion using more landmarks and path verbs. Gestures followed the speech, such that blind people pointed to landmarks more and depicted manner less than sighted people. This suggests that visual experience affects how people express spatial events in the multimodal language and that blindness may enhance sensitivity to paths of motion due to changes in event construal. These findings have implications for the claims that language processes are deeply rooted in our sensory experiences. (shrink)

Recent studies of naturalistic face‐to‐face communication have demonstrated coordination patterns such as the temporal matching of verbal and non‐verbal behavior, which provides evidence for the proposal that verbal and non‐verbal communicative control derives from one system. In this study, we argue that the observed relationship between verbal and non‐verbal behaviors depends on the level of analysis. In a reanalysis of a corpus of naturalistic multimodal communication (Louwerse, Dale, Bard, & Jeuniaux, ), we focus on measuring the temporal patterns (...) of specific communicative behaviors in terms of their burstiness. We examined burstiness estimates across different roles of the speaker and different communicative modalities. We observed more burstiness for verbal versus non‐verbal channels, and for more versus less informative language subchannels. Using this new method for analyzing temporal patterns in communicative behaviors, we show that there is a complex relationship between verbal and non‐verbal channels. We propose a “temporal heterogeneity” hypothesis to explain how the language system adapts to the demands of dialog. (shrink)

This study explores multimodal metaphors and metonymies in Faust, a German Expressionist silent fiction movie by Murnau. The article combines principles of psychocinematics, an interdisciplinary scientific field of enquiry, with the multimodal metaphor and expressive movement model, which looks into the temporal dynamics of metaphoric meaning-making by movie watchers. It is shown that interrelating both film-analytic approaches provides a deeper and more comprehensive insight into how figurative thought influences psycho-cognitive processes in the moviegoer’s mind as they dynamically unfold (...) in their cinematic contexts. Evidence is provided that Kappelhoff and co-workers’ proposal can be complemented and enriched by psychocinematics findings on the allocation of visual attention and eye fixation, motion and image frame sensing, and mental activity underlying emotional and diagetic experiences. (shrink)

The concordance of results that are “robust” across multiple scientific modalities is widely considered to play a critical role in the epistemology of science. But what should we make of those cases where such multimodal evidence is discordant? Jacob Stegenga has recently argued that robustness is “worse than useless” in these cases, suggesting that “different kinds of evidence cannot be combined in a coherent way.” In this article I respond to this critique and illustrate the critical methodological (...) role that robustness plays as an aim of scientific inquiry. (shrink)

An increasing number of research teams are investigating the efficacy of brain-computer interface (BCI)-mediated interventions for promoting motor recovery following stroke. A growing body of evidence suggests that of the various BCI designs, most effective are those that deliver functional electrical stimulation (FES) of upper extremity (UE) muscles contingent on movement intent. More specifically, BCI-FES interventions utilize algorithms that isolate motor signals—user-generated intent-to-move neural activity recorded from cerebral cortical motor areas—to drive electrical stimulation of individual muscles or muscle synergies. (...) BCI-FES interventions aim to recover sensorimotor function of an impaired extremity by facilitating and/or inducing long-term motor learning-related neuroplastic changes in appropriate control circuitry. We developed a non-invasive, electroencephalogram (EEG)-based BCI-FES system that delivers closed-loop neural activity-triggered electrical stimulation of targeted distal muscles while providing the user with multimodal sensory feedback. This BCI-FES system consists of three components: (1) EEG acquisition and signal processing to extract real-time volitional and task-dependent neural command signals from cerebral cortical motor areas, (2) FES of muscles of the impaired hand contingent on the motor cortical neural command signals, and (3) multimodal sensory feedback associated with performance of the behavioral task, including visual information, linked activation of somatosensory afferents through intact sensorimotor circuits, and electro-tactile stimulation of the tongue. In this report, we describe device parameters and intervention protocols of our BCI-FES system which, combined with standard physical rehabilitation approaches, has proven efficacious in treating UE motor impairment in stroke survivors, regardless of level of impairment and chronicity. (shrink)

A variety of theoretical frameworks predict the resemblance of behaviors between two people engaged in communication, in the form of coordination, mimicry, or alignment. However, little is known about the time course of the behavior matching, even though there is evidence that dyads synchronize oscillatory motions (e.g., postural sway). This study examined the temporal structure of nonoscillatory actions—language, facial, and gestural behaviors—produced during a route communication task. The focus was the temporal relationship between matching behaviors in the interlocutors (e.g., (...) facial behavior in one interlocutor vs. the same facial behavior in the other interlocutor). Cross-recurrence analysis revealed that within each category tested (language, facial, gestural), interlocutors synchronized matching behaviors, at temporal lags short enough to provide imitation of one interlocutor by the other, from one conversational turn to the next. Both social and cognitive variables predicted the degree of temporal organization. These findings suggest that the temporal structure of matching behaviors provides low-level and low-cost resources for human interaction. (shrink)

Humans typically combine linguistic and nonlinguistic information to comprehend emotions. We adopted an emotion identification Stroop task to investigate how different channels interact in emotion communication. In experiment 1, synonyms of “happy” and “sad” were spoken with happy and sad prosody. Participants had more difficulty ignoring prosody than ignoring verbal content. In experiment 2, synonyms of “happy” and “sad” were spoken with happy and sad prosody, while happy or sad faces were displayed. Accuracy was lower when two channels expressed an (...) emotion that was incongruent with the channel participants had to focus on, compared with the cross-channel congruence condition. When participants were required to focus on verbal content, accuracy was significantly lower also when prosody was incongruent with verbal content and face. This suggests that prosody biases emotional verbal content processing, even when conflicting with verbal content and face simultaneously. Implications for multimodal communication and language evolution studies are discussed. (shrink)

Place-based policies tackle grand socio-economic challenges through differentiated, context-sensitive interventions. However, they often run the risk of under- or mis-performing. This work studies how grand challenges translate from policy to cross-sector partnerships through place. By focusing on the place-based policy of regional smart specialization (RIS3), I investigate how the setup of science and technology parks mediates the practices of the actors in the translation chain: a transnational policymaker (macro), a regional broker (meso), and a local partnership which served as prototype (...) for the regional policy (micro level). I document two types of practices—emplacement and space configuration—enacted at each level, and show how their interplay transformed the grand challenge from a cautious ideal at the macro level which balances risk and responsibility, to an optimistic and risk-prone approach at the partnership level. The study contributes to the policy and cross-sector partnerships literatures by documenting a two-sided effect of place-based policy and a consequent risk of ethical reversal, from an early attractor bringing partners together to a later accomplice keeping the partners together despite evident signs of failure. By adopting a strong multimodal approach, I also distinguish four types of multimodal outcomes—ideal type, prototype, virtual model, and lived artifact—which perform the two-sided effect and bring about the risk of reversal. Practical implications include a redistribution of the burden of failure from the CSPs implementing the grand challenges to the institutional fields in which these are bred. (shrink)

Multimodal discourse analysis offers a novel lens for the study of legal discourse. Within the closing arguments of a trial, prosecution and defense attorneys utilize various multimodal resources to present evidence, articulate opinions, and adopt stances to achieve their communicative goals. This research focuses on the closing arguments in the criminal trial concerning the death of George Floyd to analyze the multimodal representation of the closing arguments delivered by both prosecution and defense attorneys. It employs the (...) analytical framework of the attitude system and paralinguistic typology proposed by Martin and colleagues, and uses the UAM Corpus Tool and Praat to qualitatively annotate and quantify the linguistic and paralinguistic information. The findings reveal that these attorneys employ different multimodal resources, including language and paralanguage, to convey their attitudes towards the case’s participants. This leads to the construction of competing narratives, shedding light on how justice is seen to be done. (shrink)

This article examines the interaction between grammar and multimodal resources by analyzing reported speech in Korean multiparty face-to-face interaction. The operation of two relevancy rules — minimization and recognition in interaction — is examined together with how the absence or presence of grammar is complemented by multimodal resources of various sorts. For the analysis, three categories are posited depending on who the quoted character is in the talk. In quoting oneself or co-participants in the talk, syntactic resources, prosody, (...) sequential organization and embodied participant frameworks collaborate in the interactional management of the quote without overt realization of grammatical resources traditionally proposed for building quotations in Korean. The multimodal character of quoted speech is most weakly realized when quoting a non-present character, not only because of the limitation in body and spatial resources available in the local context to recover various referential terms, but more importantly, because the interactional project that the parties are engaged in makes a distancing device such as the quotative particle -ko relevant. This study provides evidence for the intricate interaction between multimodal resources and grammar through investigation of an important realm of human interaction, that of reported speech. (shrink)

Although taste and smell seem hard to imagine, some people nevertheless report vivid imagery in these sensory modalities. We investigate whether experts are better able to imagine smells and tastes because they have learned the ability, or whether they are better imaginers in the first place, and so become experts. To test this, we first compared a group of wine experts to yoked novices using a battery of questionnaires. We show for the first time that experts report greater vividness of (...) wine imagery, with no difference in vividness across sensory modalities. In contrast, novices had more vivid color imagery than taste or odor imagery for wines. Experts and novices did not differ on other vividness of imagery measures, suggesting a domain‐specific effect of expertise. Critically, in a second study, we followed a group of students commencing a wine course and a group of matched control participants. Students and controls did not differ before the course, but after the wine course students reported more vivid wine imagery. We provide evidence that expertise improves imagery, exemplifying the extent of plasticity of cognition underlying the chemical senses. (shrink)

Intersemiotic projection is one of the most common configurations in the knowledge construction process of academic comics. Although previous studies address some general features of intersemiotic projection, further research on interdependency relations of intersemiotic projection is needed in order to map out the whole system. This study, based on the social-semiotic approach to multimodal studies, proposes a systemic framework of image-text paratactic and hypotactic projection in academic comics. This framework identifies three sub-categories of paratactic projection and hypotactic projection, respectively: (...) Emergent, Adjoined, Sequential, Compound, Subsumed, and Fused. The use of these projection configurations contributes to the realization of the communicative purposes of academic comics to evidence, exemplify, emphasize, explain, elaborate, and visualize the discussed knowledge. This research attempts not only to contribute to the theoretical development of multimodality from the social-semiotic perspective by refining and enlarging the multimodal hypotactic projection framework but also to provide an effective and applicable tool for comics analysis. (shrink)

Research over the past few decades has shown the positive influence that cognitive, social, and physical activities have on older adults’ cognitive and affective health. Especially interventions in health-related behaviors, such as cognitive activation, physical activity, social activity, nutrition, mindfulness, and creativity, have shown to be particularly beneficial. Whereas most intervention studies apply unimodal interventions, such as cognitive training, this study investigates the potential to foster cognitive and affective health factors of older adults by means of an autonomy-supportive multimodal (...) intervention. The intervention integrates everyday life recommendations for six evidence-based areas combined with psychoeducational information. This randomized controlled trial study compares the effects of a MMI and CT on those of a waiting control group on cognitive and affective factors, everyday life memory performance, and activity in everyday life. Three groups, including a total of 119 adults aged 65–86 years, attended a 5- or 10-week intervention. Specifically, one group completed a 10-week MMI, the second group completed 5-week of computer-based CT followed by a 5-week MMI, whereas the third group paused before completing the MMI for the last 5 weeks. All participants completed online surveys and cognitive tests at three test points. The findings showed an increase in the number and variability of activities in the everyday lives of all participants. Post hoc analysis on cognitive performance of MMI to CT indicate similar or better effects. Furthermore, results on far transfer variables showed interesting trends in favor of the MMI, such as increased well-being and attitude toward the aging brain. Also, the MMI group showed the biggest perceived improvements out of all groups for all self-reported personal variables. The results implicate a positive trend toward MMI on cognitive and affective factors of older adults. These tendencies show the potential of a multimodal approach compared to training a specific cognitive function. Moreover, the findings suggest that information about MMI motivates participants to increase activity variability and frequency in everyday life. Finally, the results could also have implications for the primary prevention of neurocognitive deficits and degenerative diseases. (shrink)

It is commonly argued that values “fill the logical gap” of underdetermination of theory by evidence, namely, values affect our choice between two or more theories that fit the same evidence. The underdetermination model, however, does not exhaust the roles values play in evidential reasoning. I introduce WAVE – a novel account of the logical relations between values and evidence. WAVE states that values influence evidential reasoning by adjusting evidential weights. I argue that the weight-adjusting role of (...) values is distinct from their underdetermination gap-filling role. Values adjust weights in three ways. First, values affect our trust in the testimony of others. Second, values influence the evidential thresholds required for justified epistemic judgments. Third, values influence the relative weight of a certain type of evidence within a body of multimodal discordant evidence. WAVE explains, from an epistemic perspective, rather than psychological, how smokers, for example, can find the same evidence about the dangers of smoking less persuasive than non-smokers. WAVE allows for a wider effect of values on our accepted scientific theories and beliefs than the effect for which the underdetermination model allows alone; therefore, science studies scholars must consider WAVE in their research and analysis of evidential case studies. (shrink)

Disorders of consciousness (DoC)—that is, unresponsive wakefulness syndrome/vegetative state and minimally conscious state—are debilitating conditions for which no reliable markers of consciousness recovery have yet been identified. Evidence points to the GABAergic system being altered in DoC, making it a potential target as such a marker.

The rise in Study Abroad participation among college students has increased interest among educationalists wondering about the impact of SA on students, particularly when students return home without evidence of deep engagement and understanding of other cultures and people. The purpose of this case study was to locate one potential source of the meanings ascribed to the SA experience, through analysis of multimodal representations on the institutional website of a popular SA program provider. In this study, Kress’ model (...) of multimodal social semiotic communication was employed in the analysis of the ensemble of modes of communication on the website, and their role in the production and dissemination of discourses of SA. Findings indicate that discourses of tourism prevail over discourses of education, and the representations enacted on the institutional website are mirrored in the discursive practices of students. (shrink)

Firstly, this paper aims to analyze how verbal and visual modes contribute to build two basic components of cartoons: the referenced situation and the fictional situation. Secondly, it aims to unravel the semiotics of this discursive genre by offering a fine-grained picture of modal variations and continuities of the rhetorical resources deployed, by means of which the fictional situation is displayed. The corpus is composed of 50 multimodal cartoons chosen by cartoonists of 22 nationalities as the most representative of (...) their work in the frame of a wide questionnaire. Category analysis and non-parametric statistics were applied. We identified and fleshed out four different types of modal functioning of resources: visual preference, verbal preference, modal flexibility, and genuine multimodal functioning. Even if visual and multimodal metaphor were often in charge of transferring meaning from fiction to the referenced situation, results show that other resources such as antithesis, irony, and syllepsis can become the key resources of cartoons. Findings highlight the multilayered functioning of cartoons, among which plastic features can play a significant role, and provide further evidence in favor of the hyperdetermination of humor hypothesis proposed by Attardo. (shrink)

The first premise of Schellenberg’s particularity argument reads, “If a subject S perceives a particular α, then S discriminates and singles out α” (2018: 25). But this is false if seeing a ganzfeld is possible (i.e., a homogeneous field without any particulars to discriminate). In response, Schellenberg argues that seeing a ganzfeld is impossible by appealing to the ganzfeld effect (viz. hallucinatory experiences caused by ganzfeld exposure) exclusively as a ‘sense of blindness’. I present two challenges for this line of (...) reasoning. The first concerns the need to account for the long delay before blindness is reported (if it is reported at all) and the second concerns the need to be consistent with potential causes of ganzfeld hallucinations (in general, not exclusive to the sense of blindness). I show how Schellenberg’s particularity argument can meet the first challenge but struggles to meet the second. I present evidence based on considerations related to sensory deprivation and that follow from the predictive processing theory of cognition, which describes seeing a ganzfeld as the cause of ganzfeld hallucinations. If correct, this means that you can see a homogeneous field, and thus we ought to be skeptical of the soundness of Schellenberg’s particularity argument. I conclude with a possible strategy for rehabilitating the particularity argument that reconceptualizes perceptual experience as fundamentally multimodal, constituted by both exteroceptive and interoceptive sensation. (shrink)

This article addresses the issue of how team identity is constructed between two people during a series of regular meetings of a work group in Serbia. Using conversation analysis to investigate social actions, this study looks at the recurrent construction of an implicit team identity by focusing on management of speaking rights and co-construction of units, and displays of knowledge and accountability. With its longitudinal perspective, the article contributes to the existing body of research on teams in interaction in general, (...) as it builds upon previous research on interactional parties and conjoined participation. The results are especially relevant for the investigations of teams in meetings, as they provide evidence of how formal features of interaction are recurrently employed to display institutionally relevant epistemics and accountability. (shrink)

Many philosophers take experience to be an essential aspect of perceptual justification. I argue against a specific variety of such an experientialist view, namely, the Looks View of perceptual justification, according to which our visual beliefs are mediately justified by beliefs about the way things look. I describe three types of cases that put pressure on the idea that perceptual justification is always related to looks-related reasons: unsophisticated cognizers, multimodal identification, and amodal completion. I then provide a tentative diagnosis (...) of what goes wrong in the Looks View: it ascribes a specific epistemic role to beliefs about looks that is actually fulfilled by subpersonal perceptual processes. (shrink)

A multimodal person representation contains information about what a person looks like and what a person sounds like. However, little is known about how children form these face-voice mappings. Here, we explored the possibility that two cognitive tools that guide word learning, a one-to-one mapping bias and fast mapping, also guide children’s learning about faces and voices. We taught 4- and 5-year-olds mappings between three individual faces and voices, then presented them with new faces and voices. In Experiment 1, (...) we found that children rapidly learned face-voice mappings from just a few exposures, and furthermore spontaneously mapped novel faces to novel voices using a one-to-one mapping bias (that each face can produce only one voice). In Experiment 2, we found that children’s face-voice representations are abstract, generalizing to novel tokens of a person. In Experiment 3, we found that children retained in memory the face-voice mappings that they had generated via inference (i.e., they showed evidence of fast mapping), and used these newly formed representations to generate further mappings between new faces and voices. These findings suggest that preschoolers’ rapid learning about faces and voices may be aided by biases that are similar to those that support word learning. (shrink)

In the past few decades there has been a growing interest and debate amongst historians of education surrounding issues of visuality, materiality, spatiality, transfer, and circulation. This collection of essays – with its focus on the interaction between ideas, images, objects, and/or spaces that contain an educational dimension – is a contribution to this ongoing debate. The contributors address how meaning is created, conveyed, and transformed through multiple modes of communication, representation, and interaction; through movement across spaces; through media and (...) technologies; and through collective memory- and identity-making. The collection demonstrates that meaning is mobilized through ‘multimodality’, ‘translocation’, ‘technology’, and ‘heritage’, and that it assumes different qualities which need to be reflected upon in the history of education in particular and in education research in general. This book was originally published as a special issue of _Paedagogica Historica. _. (shrink)

Journal Name: Semiotica Issue: Ahead of print.

Executive function is essential to child development, with associated skills beginning to emerge in the first few years of life and continuing to develop into adolescence and adulthood. The prefrontal cortex, which follows a neurodevelopmental timeline similar to EF, plays an important role in the development of EF. However, limited research has examined prefrontal function in young children due to limitations of currently available neuroimaging techniques such as functional resonance magnetic imaging. The current study developed and applied a multimodal (...) Go/NoGo task to examine the EF component of inhibitory control in children 4–10 years of age. Cortical activity was measured using a non-invasive and child-friendly neuroimaging technique – functional near-infrared spectroscopy. Children’s response accuracy and reaction times were captured during the fNIRS session and compared with responses obtained using the standardized assessments from NIH Toolbox cognition battery. Results showed significant correlations between the behavioral measures during the fNIRS session and the standardized EF assessments, in line with our expectations. Results from fNIRS measures demonstrated a significant, age-independent effect of inhibitory control in the right PFC, and an age-dependent effect in the left orbitofrontal cortex, consistent with results in previous studies using fNIRS and fMRI. Thus, the new task designed for fNIRS was suitable for examining IC in young children, and results showed that fNIRS measures can reveal prefrontal IC function. (shrink)

Cognitive semantic studies have shown that our conceptualization of morality is at least partially metaphorical and that our moral cognition is grounded in some fundamental contrastive categories of our embodied experience in the physical environment. It is argued that our moral cognition is built on a moral metaphor system. Within the framework of conceptual metaphor theory, this study aims to examine the spatial subsystem of moral metaphors in English. We set out with five pairs of moral metaphors that involve the (...) following spatial source concepts: HIGH and LOW, UPRIGHT and TILTED, LEVEL and UNLEVEL, STRAIGHT and CROOKED, and BIG and SMALL. These metaphors were found to constitute the spatial subsystem of moral metaphors in Chinese. Our primary goal is to find out if the five pairs of moral–spatial metaphors are manifested in English as well. To that end we collected linguistic data from the Corpus of Contemporary American English and searched Google Images for multimodal evidence. Our finding is that the five pairs of moral–spatial metaphors are applicable in English as they are in Chinese. We also discuss issues related to conceptual metaphor theory. (shrink)

Consensus conferences are social techniques which involve bringing together a group of scientific experts, and sometimes also non-experts, in order to increase the public role in science and related policy, to amalgamate diverse and often contradictory evidence for a hypothesis of interest, and to achieve scientific consensus or at least the appearance of consensus among scientists. For consensus conferences that set out to amalgamate evidence, I propose three desiderata: Inclusivity, Constraint, and Evidential Complexity. Two examples suggest that consensus (...) conferences can readily satisfy Inclusivity and Evidential Complexity, but consensus conferences do not as easily satisfy Constraint. I end by discussing the relation between social inclusivity and the three desiderata. (shrink)

Abstract: What grounds the experience of our body as our own? Can we rationally doubt that this is our own body when we feel sensations in it? This article shows how recent empirical evidence can shed light on issues on the body and the self, such as the grounds of the sense of body ownership and the immunity to error through misidentification of bodily self-ascriptions. In particular, it discusses how bodily illusions (e.g., the Rubber Hand Illusion), bodily disruptions (e.g., (...) somatoparaphrenia), and the multimodal nature of bodily self-knowledge challenge a classic view of ownership and immunity that puts bodily sensations at its core. (shrink)

In the case of ventriloquism, seeing the movement of the ventriloquist dummy’s mouth changes your experience of the auditory location of the vocals. Some have argued that cases like ventriloquism provide evidence for the view that at least some of the content of perception is fundamentally multimodal. In the ventriloquism case, this would mean your experience has constitutively audio-visual content (not just a conjunction of an audio content and visual content). In this paper, I argue that cases like (...) ventriloquism do not in fact warrant that conclusion. I then try to make sense of crossmodal cases without appealing to fundamentally multimodal content. (shrink)

This article presents a multimodal video corpus with the principal aim to model and predict the effects of aging in Mild Cognitive Impairment situation on pragmatic and communicative skills. We take as observable variables the verbal pragmatic markers and non-verbal pragmatic markers. This approach, at the interface of the psycholinguistics, cognitive sciences and rehabilitation medicine (speech-language pathology and therapy) is part of a longitudinal research process in an ecological situation (interviews conducted by close intimate of the elderly).In the first (...) part of the article we present the linguistic, cognitive and social characteristics of aging in its continuum up to mild cognitive impairment and pathological disorders such as Alzheimer’s disease. In the second part, we develop a multimodal approach, in particular to inform and enrich speech and language therapy knowledge. Finally, we present our experimental design and preliminary results on two female participants over 75 years of age with mild cognitive impairment.Our general findings indicate that with aging, verbal pragmatic markers acquire an interactive function that allows people with Mild Cognitive Impairment to maintain intersubjective relationships with their interlocutor. In addition, at the non-verbal level, gestural manifestations are increasingly mobilized over time with a preference for non-verbal pragmatic markers with a referential function and an interactive function. One such non-verbal manifestation compensates for naming deficits, planning difficulties, discursive hitches; while another optimizes and maintains the interaction with the interlocutor.Clinicians have a duty to develop their professional practice through an evidence-based clinical approach whose main objective is to reconcile clinical practice with the best evidence from research (Dollaghan 2007). In the case of speech-language pathology, clinicians consider themselves very limited in this approach (Lof 2011; McCurtin 2011), especially for patients with Mild Cognitive Impairment (Mungas et al. 2010; Hopper 2013; Morello 2017) and more specifically when it comes to assessing or supporting language functions (Cummings 2014).The studies focusing on Mild Cognitive Impairment require longitudinal corpora i) to understand the naturally occurring evolutions in subjects, ii) the implication of the cognitive reserve in each individual, and iii) to take advantage of these parameters as evidence for research and earlier rehabilitation. We aim to show the benefits of linguistic and interactional scientific investigation methods through fragile aging, for health professionals and everyday caregivers. (shrink)

In conversation, individuals work together to achieve communicative goals, complementing and aligning language and body with each other. An important emerging question is whether interlocutors entrain with one another equally across linguistic levels (e.g., lexical, syntactic, and semantic) and modalities (i.e., speech and gesture), or whether there are complementary patterns of behaviors, with some levels or modalities diverging and others converging in coordinated fashions. This study assesses how kinematic and linguistic entrainment interact with one another across levels of measurement, and (...) according to communicative context. We analyzed data from two matched corpora of dyadic interaction between—respectively—Danish and Norwegian native speakers engaged in affiliative conversations and task‐oriented conversations. We assessed linguistic entrainment at the lexical, syntactic, and semantic level, and kinetic alignment of the head and hands using video‐based motion tracking and dynamic time warping. We tested whether—across the two languages—linguistic alignment correlates with kinetic alignment, and whether these kinetic‐linguistic associations are modulated either by the type of conversation or by the language spoken. We found that kinetic entrainment was positively associated with low‐level linguistic (i.e., lexical) entrainment, while negatively associated with high‐level linguistic (i.e., semantic) entrainment, in a cross‐linguistically robust way. Our findings suggest that conversation makes use of a dynamic coordination of similarity and complementarity both between individuals as well as between different communicative modalities, and provides evidence for a multimodal, interpersonal synergy account of interaction. (shrink)

Overwhelming evidence points to the existence of separate sensory channels in the nervous system. The power of this type of parallel organization is that information is first processed in neurons specialized to code it most efficiently. However, sensory pathways are convergent and divergent at each level as well, as is necessary to interpret multimodal and conflicting information.

There is converging evidence that high doses of hallucinogenic drugs can produce significant alterations of self-experience, described as the dissolution of the sense of self and the loss of boundaries between self and world. This article discusses the relevance of this phenomenon, known as “drug-induced ego dissolution (DIED)”, for cognitive neuroscience, psychology and philosophy of mind. Data from self-report questionnaires suggest that three neuropharmacological classes of drugs can induce ego dissolution: classical psychedelics, dissociative anesthetics and agonists of the kappa (...) opioid receptor (KOR). While these substances act on different neurotransmitter receptors, they all produce strong subjective effects that can be compared to the symptoms of acute psychosis, including ego dissolution. It has been suggested that neuroimaging of DIED can indirectly shed light on the neural correlates of the self. While this line of inquiry is promising, its results must be interpreted with caution. First, neural correlates of ego dissolution might reveal the necessary neurophysiological conditions for the maintenance of the sense of self, but it is more doubtful that this method can reveal its minimally sufficient conditions. Second, it is necessary to define the relevant notion of self at play in the phenomenon of DIED. This article suggests that DIED consists in the disruption of subpersonal processes underlying the “minimal” or “embodied” self, i.e., the basic experience of being a self rooted in multimodal integration of self-related stimuli. This hypothesis is consistent with Bayesian models of phenomenal selfhood, according to which the subjective structure of conscious experience ultimately results from the optimization of predictions in perception and action. Finally, it is argued that DIED is also of particular interest for philosophy of mind. On the one hand, it challenges theories according to which consciousness always involves self-awareness. On the other hand, it suggests that ordinary conscious experience might involve a minimal kind of self-awareness rooted in multisensory processing, which is what appears to fade away during DIED. (shrink)

During face‐to‐face conversation, transitions between speaker turns are incredibly fast. These fast turn exchanges seem to involve next speakers predicting upcoming semantic information, such that next turn planning can begin before a current turn is complete. Given that face‐to‐face conversation also involves the use of communicative bodily signals, an important question is how bodily signals such as co‐speech hand gestures play into these processes of prediction and fast responding. In this corpus study, we found that hand gestures that depict or (...) refer to semantic information started before the corresponding information in speech, which held both for the onset of the gesture as a whole, as well as the onset of the stroke (the most meaningful part of the gesture). This early timing potentially allows listeners to use the gestural information to predict the corresponding semantic information to be conveyed in speech. Moreover, we provided further evidence that questions with gestures got faster responses than questions without gestures. However, we found no evidence for the idea that how much a gesture precedes its lexical affiliate (i.e., its predictive potential) relates to how fast responses were given. The findings presented here highlight the importance of the temporal relation between speech and gesture and help to illuminate the potential mechanisms underpinning multimodal language processing during face‐to‐face conversation. (shrink)

This chapter contains sections titled: Pluralistic Ignorance Modal Operator Epistemology Agents and Inquiry Methods Multimodal Systems Knowledge Transmissibility Knowledge Transmissibility and Public Announcement Knowledge Transmissibility and Pluralistic Ignorance References.

TThis paper aims to evidence the inherently metonymic nature of co-speech gestures. Arguing that motivation in gesture involves iconicity (similarity), indexicality (contiguity), and habit (conventionality) to varying degrees, it demonstrates how a set of metonymic principles may lend a certain systematicity to experientially grounded processes of gestural abstraction and enaction. Introducing visuo-kinetic signs as an umbrella term for co-speech gestures and signed languages, the paper shows how a frame-based approach to gesture may integrate different cognitive/functional linguistic and semiotic accounts (...) of metonymy (e.g., experiential domains, frame metonymy, contiguity, and pragmatic inferencing). The guiding assumption is that gestures metonymically profile deeply embodied, routinized aspects of familiar scenes, that is, the motivating context of frames. The discussion shows how gestures may evoke frame structures exhibiting varying degrees of groundedness, complexity, and schematicity: basic physical action and object frames; more complex frames; and highly abstract, complex frame structures. It thereby provides gestural evidence for the idea that metonymy is more basic and more directly experientially grounded than metaphor and thus often feeds into correlated metaphoric processes. Furthermore, the paper offers some initial insights into how metonymy also seems to induce the emergence of schematic patterns in gesture which may result from action-based and discourse-driven processes of habituation and conventionalization. It exemplifies how these forces may engender grammaticalization of a basic physical action into a gestural marker that shows strong metonymic form reduction, decreased transitivity, and interacting pragmatic functions. Finally, addressing basic metonymic operations in signed lexemes elucidates certain similarities regarding sign constitution in gesture and sign. English and German multimodal discourse data as well as German Sign Language (DGS) are drawn upon to illustrate the theoretical points of the paper. Overall, this paper presents a unified account of metonymy’s role in underpinning forms, functions, and patterns in visuo-kinetic signs. (shrink)