Gestural viewpoint research suggests that several dimensions determine which perspective a narrator takes, including properties of the event described. Events can evoke gestures from the point of view of a character , an observer , or both perspectives. CVPT and OVPT gestures have been compared to constructed action and classifiers in signed languages. We ask how CA and CL, as represented in ASL productions, compare to previous results for CVPT and OVPT from English-speaking co-speech gesturers. Ten ASL signers described (...) cartoon stimuli from Parrill . Events shown by Parrill to elicit a particular gestural strategy were coded for signers' instances of CA and CL. CA was divided into three categories: CA-torso, CA-affect, and CA-handling. Signers used CA-handling the most when gesturers used CVPT exclusively. Additionally, signers used CL the most when gesturers used OVPT exclusively and CL the least when gesturers used CVPT exclusively. (shrink)

Sighted speakers of different languages vary systematically in how they package and order components of a motion event in speech. These differences influence how semantic elements are organized in gesture, but only when those gestures are produced with speech, not without speech. We ask whether the cross-linguistic similarity in silent gesture is driven by the visuospatial structure of the event. We compared 40 congenitally blind adult native speakers of English or Turkish to 80 sighted adult (...) speakers as they described three-dimensional motion scenes. We found an effect of language on co-speech gesture, not on silent gesture—blind speakers of both languages organized their silent gestures as sighted speakers do. Humans may have a natural semantic organization that they impose on events when conveying them in gesture without language—an organization that relies on neither visuospatial cues nor language structure. (shrink)

Establishing and maintaining reference is a crucial part of discourse. In spoken languages, differential linguistic devices mark referents occurring in different referential contexts, that is, introduction, maintenance, and re-introduction contexts. Speakers using gestures as well as users of sign languages have also been shown to mark referents differentially depending on the referential context. This article investigates the modality-specific contribution of the visual modality in marking referential context by providing a direct comparison between sign language and co-speech gesture with (...) speech in elicited narratives. Across all forms of expression, we find that referents in subject position are referred to with more marking material in re-introduction contexts compared to maintenance contexts. Furthermore, we find that spatial modification is used as a modality-specific strategy in both DGS and German co-speech gesture, and that the configuration of referent locations in sign space and gesture space corresponds in an iconic and consistent way to the locations of referents in the narrated event. However, we find that spatial modification is used in different ways for marking re-introduction and maintenance contexts in DGS and German co-speech gesture. The findings are discussed in relation to the unique contribution of the visual modality to reference tracking in discourse when it is used in a unimodal system with full linguistic structure versus in a bimodal system that is a composite of speech and gesture. (shrink)

This study concerned the role of gestures that accompany discourse in deep learning processes. We assumed that co‐speech gestures favor the construction of a complete mental representation of the discourse content, and we tested the predictions that a discourse accompanied by gestures, as compared with a discourse not accompanied by gestures, should result in better recollection of conceptual information, a greater number of discourse‐based inferences drawn from the information explicitly stated in the discourse, and poorer recognition of verbatim of (...) the discourse. The results of three experiments confirmed these predictions. (shrink)

In general, we know that gesture accompanying spoken instruction can help children learn. The present study was conducted to better understand how gesture can support children’s comprehension of spoken instruction and whether the benefit of teaching though speech and gesture over spoken instruction alone depends on differences in cognitive profile – prior knowledge children have that is related to a to-be-learned concept. To answer this question, we explored the impact of gesture instruction on children’s analogical (...) reasoning ability. Children between the ages of 4 and 11 years solved scene analogy problems before or after speech alone or speech and gesture instruction while their visual attention was monitored. Our behavioral results suggest a marginal benefit of gesture instruction over speech alone, but only 5-year-old children showed a distinct advantage from speech+gesture instruction when solving the post-instruction trial, suggesting that at this age, children have the cognitive profile in place to utilize the added support of gesture. Furthermore, while speech+gesture instruction facilitated effective visual attention during instruction, directing attention away from featural matches, and towards relational information, was pivotal for younger children’s success post instruction. We consider how these results contribute to the gesture-for-learning literature and consider how the nuanced impact of gesture is informative for educators teaching tasks of analogy in the classroom. (shrink)

Co-speech gestures are closely connected to speech, but little attention has been paid to the associations between gesture and L2 speech performance. This study explored the associations between four types of co-speech gestures and the meaning, form, and discourse dimensions of L2 speech performance. Gesture and speech data were collected by asking 61 lower-intermediate English learners whose first language is Chinese to retell a cartoon clip. Results showed that all the four types (...) of co-speech gestures had positive associations with meaning and discourse L2 speech measures but no association with form-related speech measures, except the positive association between metaphorics and the percentage of error-free clauses. The findings suggest that co-speech gestures may have a tighter connection with meaning construction in producing L2 speech. (shrink)

Speakers of different languages (e.g., English vs. Turkish) show a binary split in how they package and order components of a motion event in speech and co‐speech gesture but not in silent gesture. In this study, we focused on Mandarin Chinese, a language that does not follow the binary split in its expression of motion in speech, and asked whether adult Chinese speakers would follow the language‐specific speech patterns in co‐speech but not silent (...) gesture, thus showing a pattern akin to Turkish and English adult speakers in their description of animated motion events. Our results provided evidence for this pattern, with Chinese—as well as English and Turkish—speakers following language‐specific patterns in speech and co‐speech gesture but not in silent gesture. Our results provide support for the “thinking‐for‐speaking” account, namely that language influences thought only during online, but not offline, production of speech. (shrink)

In previous analyses of the influence of language on cognition, speech has been the main channel examined. In studies conducted among Yucatec Mayas, efforts to determine the preferred frame of reference in use in this community have failed to reach an agreement (Bohnemeyer & Stolz, 2006; Levinson, 2003 vs. Le Guen, 2006, 2009). This paper argues for a multimodal analysis of language that encompasses gesture as well as speech, and shows that the preferred frame of reference in (...) Yucatec Maya is only detectable through the analysis of co-speech gesture and not through speech alone. A series of experiments compares knowledge of the semantics of spatial terms, performance on nonlinguistic tasks and gestures produced by men and women. The results show a striking gender difference in the knowledge of the semantics of spatial terms, but an equal preference for a geocentric frame of reference in nonverbal tasks. In a localization task, participants used a variety of strategies in their speech, but they all exhibited a systematic preference for a geocentric frame of reference in their gestures. (shrink)

During face‐to‐face conversation, transitions between speaker turns are incredibly fast. These fast turn exchanges seem to involve next speakers predicting upcoming semantic information, such that next turn planning can begin before a current turn is complete. Given that face‐to‐face conversation also involves the use of communicative bodily signals, an important question is how bodily signals such as co‐speech hand gestures play into these processes of prediction and fast responding. In this corpus study, we found that hand gestures that depict (...) or refer to semantic information started before the corresponding information in speech, which held both for the onset of the gesture as a whole, as well as the onset of the stroke (the most meaningful part of the gesture). This early timing potentially allows listeners to use the gestural information to predict the corresponding semantic information to be conveyed in speech. Moreover, we provided further evidence that questions with gestures got faster responses than questions without gestures. However, we found no evidence for the idea that how much a gesture precedes its lexical affiliate (i.e., its predictive potential) relates to how fast responses were given. The findings presented here highlight the importance of the temporal relation between speech and gesture and help to illuminate the potential mechanisms underpinning multimodal language processing during face‐to‐face conversation. (shrink)

This study investigates whether there is a relation between the semantics of linguistic expressions that indicate temporal distance and the spatial properties of their co-speech gestures. To this date, research on time gestures has focused on features such as gesture axis, direction, and shape. Here we focus on a gesture property that has been overlooked so far: the distance of the gesture in relation to the body. To achieve this, we investigate two types of temporal linguistic (...) expressions are addressed: proximal (e.g., near future, near past) and distal (e.g., distant past, distant future). Data was obtained through the NewsScape library, a multimodal corpus of television news. A total of 121 co-speech gestures were collected and divided into the two categories. The gestures were later annotated in terms of gesture space and classified in three categories: (i) center, (ii) periphery, and (iii) extreme periphery. Our results suggest that gesture and language are coherent in the expression of temporal distance: when speakers locate an event far from them, they tend to gesture further from their body; similarly, when locating an event close to them, they gesture closer to their body. These results thus reveal how co-speech gestures also reflect a space-time mapping in the dimension of distance. (shrink)

Past research has sought to elucidate how speakers and addressees establish common ground in conversation, yet few studies have focused on how visual cues such as co-speech gestures contribute to this process. Likewise, the effect of cognitive constraints on multimodal grounding remains to be established. This study addresses the relationship between the verbal and gestural modalities during grounding in referential communication. We report data from a collaborative task where repeated references were elicited, and a time constraint was imposed to (...) increase cognitive load. Our results reveal no differential effects of repetition or cognitive load on the semantic-based gesture rate, suggesting that representational gestures and speech are closely coordinated during grounding. However, gestures and speech differed in their execution, especially under time pressure. We argue that speech and gesture are two complementary streams that might be planned in conjunction but that unfold independently in later stages of language production, with speakers emphasizing the form of their gestures, but not of their words, to better meet the goals of the collaborative task. (shrink)

In his paper ‘Gesture Projection and Cosuppositions,’ Philippe Schlenker argues that co-verbal gestures convey not at-issue content by default and in particular, that they trigger conditional presuppositions. In this commentary, I take issue with both of these claims. Conditional presuppositions do not supply a systematic means for capturing the semantic contribution of a co-verbal gesture. Some gestures appear to contribute content inside of a negation when their associated speech content is likewise embedded; in other cases, co-verbal gestures (...) arguably contribute unconditional content to the global level. When this happens, we can infer what might look like a conditional presupposition, but this inference follows naturally from general principles already at work in purely verbal discourse and does not justify the claim that gesture content is contributed to a conditional presupposition. Problems exposed in the discussion of conditional presuppositions show that we are not yet in a position to make a general claim about the at-issue status of co-verbal gestures. (shrink)

In dynamic theories of presupposition, a trigger pp′ with presupposition p and at-issue component p′ comes with a requirement that p should be entailed by the local context of pp′. We argue that some co-speech gestures should be analyzed within a presuppositional framework, but with a twist: an expression p co-occurring with a co-speech gesture G with content g comes with the requirement that the local context of p should guarantee that p entails g; we call such (...) assertion-dependent presuppositions ‘cosuppositions’. We show that this analysis can be combined with earlier theories of local contexts to account for complex patterns of gesture projection in quantified and in attitudinal contexts, and we compare our account to two potential alternatives: one based on supervaluations, and one, due to Cornelia Ebert, that treats co-speech gestures as supplements. We argue that the latter is correct, but for ‘post-speech’ gestures, rather than for co-speech gestures. (shrink)

The framework of depicting put forward byClark (2016)offers a schematic vantage point from which to examine iconic language use. Confronting the framework with empirical data, we consider some of its key theoretical notions. Crucially, by reconceptualizing the typology of depictions, we identify an overlooked domain in the literature: “speech-embedded nonverbal depictions,” namely cases where meaning is communicated iconically, nonverbally, and without simultaneously co-occurring speech. In addition to contextualizing the phenomenon in relation to existing research, we demonstrate, with examples (...) from American TV talk shows, how such depictions function in real-life language use, offering a brief sketch of their complexities and arguing also for their theoretical significance. (shrink)

Reasoning about bedrock abstract concepts such as time, number, and valence relies on spatial metaphor and often on multiple spatial metaphors for a single concept. Previous research has documented, for instance, both future-in-front and future-to-right metaphors for time in English speakers. It is often assumed that these metaphors, which appear to have distinct experiential bases, remain distinct in online temporal reasoning. In two studies we demonstrate that, contra this assumption, people systematically combine these metaphors. Evidence for this combination was found (...) in both directly elicited and spontaneous co-speech gestures about time. These results provide first support for the hypothesis that the metaphorical representation of time, and perhaps other abstract domains as well, involves the continuous co-activation of multiple metaphors rather than the selection of only one. (shrink)

Most language learners have difficulties acquiring the phonemes of a second language (L2). Unfortunately, they are often judged on their L2 pronunciation, and segmental inaccuracies contribute to miscommunication. Therefore, we aim to determine how to facilitate phoneme acquisition. Given the close relationship between speech and co-speech gesture, previous work unsurprisingly reports that gestures can benefit language acquisition, e.g., in (L2) word learning. However, gesture studies on L2 phoneme acquisition present contradictory results, implying that both specific properties (...) of gestures and phonemes used in training, and their combination, may be relevant. We investigated the effect of phoneme and gesture complexity on L2 phoneme acquisition. In a production study, Dutch natives received instruction on the pronunciation of two Spanish phonemes, /u/ and /θ/. Both are typically difficult to produce for Dutch natives because their orthographic representation differs between both languages. Moreover, /θ/ is considered more complex than /u/, since the Dutch phoneme inventory contains /u/ but not /θ/. The instruction participants received contained Spanish examples presented either via audio-only, audio-visually without gesture, audio-visually with a simple, pointing gesture, or audio-visually with a more complex, iconic gesture representing the relevant speech articulator(s). Preceding and following training, participants read aloud Spanish sentences containing the target phonemes. In a perception study, Spanish natives rated the target words from the production study on accentedness and comprehensibility. Our results show that combining gesture and speech in L2 phoneme training can lead to significant improvement in L2 phoneme production, but both gesture and phoneme complexity affect successful learning: Significant learning only occurred for the less complex phoneme /u/ after seeing the more complex iconic gesture, whereas for the more complex phoneme /θ/, seeing the more complex gesture actually hindered acquisition. The perception results confirm the production findings and show that items containing /θ/ produced after receiving training with a less complex pointing gesture are considered less foreign-accented and more easily comprehensible as compared to the same items after audio-only training. This shows that gesture can facilitate task performance in L2 phonology acquisition, yet complexity affects whether certain gestures work better for certain phonemes than others. (shrink)

The ubiquitous human practice of spontaneously gesturing while speaking demonstrates the embodiment, embeddedness, and sociality of cognition. The present essay takes gestural practice to be a paradigmatic example of a more general claim: human cognition is social insofar as our embedded, intelligent, and interacting bodies select and construct meaning in a way that is intersubjectively constrained and defeasible. Spontaneous co-speech gesture is markedly interesting because it at once confirms embodied aspects of linguistic meaning-making that formalist and linguistic turn-type (...) philosophical approaches fail to appreciate, and it also forefronts intersubjectivity as an inherent and inherently normative dimension of communicative action. Co-speech hand gestures, as linguistically meaningful speech acts, demonstrate both sedimentation and spontaneity (in the sense of Maurice Merleau-Ponty’s dialectic of linguistic expression ( 2002 )), or features of convention and nonconvention in a Gricean sense ( 1989 ). Yet neither pragmatic nor classic phenomenological approaches to communication can accommodate the practice of co-speech hand gesturing without some rehabilitation and reorientation. Pragmatic criteria of intersubjectivity, normativity, and rationality need to confront the non-propositional and nonverbal meaning-making of embodied encounters. Phenomenological treatments of expression and intersubjectivity must consider the normative nature of high-order social practices like language use. Reciprocally critical exchanges between these traditions and gesture studies yield an improved philosophy that treats language as a multi-modal medium for collaborative meaning achievement. The proper paradigm for these discussions is found in enactive approaches to social cognition. Co-speech hand gestures are first and foremost emergent elements of social interaction, not the external whirring of an isolated internal consciousness. In contrast to current literature that frequently presents gestures as uncontrollable bodily upsurge or infallible imagistic phenomenon that drives and dances with verbal or “linguistic” convention (McNeill 1992 , 2005 ), I suggest that we study gestures as dynamic, embodied, and shared tools for collaborative sense-making. (shrink)

When young children learn to use language, they start to use their hands in co‐verbal gesturing. There are, however, considerable differences between children, and it is not completely understood what these individual differences are due to. We studied how children at 4 years of age employ speech and iconic gestures to convey meaning in different kinds of spatial event descriptions, and how this relates to their cognitive abilities. Focusing on spontaneous illustrations of actions, we applied a semantic feature (SF) (...) analysis to characterize combinations of speech and gesture meaning and related them to the child's visual‐spatial abilities or abstract/concrete reasoning abilities (measured using the standardized SON‐R test). Results show that children with higher cognitive abilities convey significantly more meaning via gesture and less solely via speech. These findings suggest that young children's use of cospeech representational gesturing is positively related to their mental representation and reasoning abilities. (shrink)

The temporal‐focus hypothesis claims that whether people conceptualize the past or the future as in front of them depends on their cultural attitudes toward time; such conceptualizations can be independent from the space–time metaphors expressed through language. In this paper, we study how Chinese people conceptualize time on the sagittal axis to find out the respective influences of language and culture on mental space–time mappings. An examination of Mandarin speakers' co‐speech gestures shows that some Chinese spontaneously perform past‐in‐front/future‐at‐back (besides (...) future‐in‐front/past‐at‐back) gestures, especially when gestures are accompanying past‐in‐front/future‐at‐back space–time metaphors (Exp. 1). Using a temporal performance task, the study confirms that Chinese can conceptualize the future as behind and the past as in front of them, and that such space–time mappings are affected by the different expressions of Mandarin space–time metaphors (Exp. 2). Additionally, a survey on cultural attitudes toward time shows that Chinese tend to focus slightly more on the future than on the past (Exp. 3). Within the Chinese sample, we did not find evidence for the effect of participants' cultural temporal attitudes on space–time mappings, but a cross‐cultural comparison of space–time mappings between Chinese, Moroccans, and Spaniards provides strong support for the temporal‐focus hypothesis. Furthermore, the results of Exp. 2 are replicated even after controlling for factors such as cultural temporal attitudes and age (Exp. 3), which implies that linguistic sagittal temporal metaphors can indeed influence Mandarin speakers' space–time mappings. The findings not only contribute to a better understanding of Chinese people's sagittal temporal orientation, but also have additional implications for theories on the mental space–time mappings and the relationship between language and thought. (shrink)

We examine the embodiment of one foundational aspect of human cognition, language, through its bodily association with the gestures that accompany its expression in speech. Gesture is a universal feature of human communication. Gestures are produced by all speakers in every culture . They are tightly timed with speech . Gestures convey important communicative information to the listener, but even blind speakers gesture while talking to blind listeners , so the mutual co-occurrence of speech and (...) gesture reflects a deep association between the two modes that transcends the intentions of the speaker to communicate. Indeed, we believe that this linkage of the vocal expression of language and the arm movements produced with it are a manifestation of the embodiment of thought: that human mental activities arise through bodily interactions with the world and remain linked with them throughout the lifespan. In particular, we propose that speech and gesture have their developmental origins in early hand-mouth linkages, such that as oral activities become gradually used for meaningful speech, these linkages are maintained and strengthened. Both hand and mouth are tightly coupled in the mutual cognitive activity of language. In short, it is the initial sensorimotor linkages of these systems that form the bases for their later cognitive interdependence. (shrink)

Do English speakers think about time the way they talk about it? In spoken English, time appears to flow along the sagittal axis (front/back): the future is ahead and the past is behind us. Here we show that when asked to gesture about past and future events deliberately, English speakers often use the sagittal axis, as language suggests they should. By contrast, when producing co-speech gestures spontaneously, they use the lateral axis (left/right) overwhelmingly more often, gesturing leftward for (...) earlier times and rightward for later times. This left-right mapping of time is consistent with the flow of time on calendars and graphs in English-speaking cultures, but is completely absent from conventional spoken metaphors. English speakers gesture on the lateral axis even when they are using front/back metaphors in their co-occurring speech. This speech-gesture dissociation is not due to any lack of lexical or constructional resources to spatialize time laterally in language, nor to any lack of physical resources to spatialize time sagittally in gesture. We propose that when speakers are describing sequences of events, they often use neither the Moving Ego nor Moving Time perspectives. Rather, they adopt a “Moving Attention” perspective, which is grounded in patterns of interaction with cultural artifacts, not in patterns of interaction with the natural environment. We suggest possible pragmatic, kinematic, and mnemonic motivations for the use of a lateral mental timeline in gesture and in thought. Gestures reveal an implicit spatial conceptualization of time that cannot be inferred from language. (shrink)

Pantomime has long been considered distinct from co-speech gesture. It has therefore been argued that pantomime cannot be part of gesture-speech integration. We examine pantomime as distinct from silent gesture, focusing on non-co-speech gestures that occur in the midst of children’s spoken narratives. We propose that gestures with features of pantomime are an infrequent but meaningful component of a multimodal communicative strategy. We examined spontaneous non-co-speech representational gesture production in the narratives of (...) 30 monolingual English-speaking children between the ages of 8- and 11-years. We compared the use of co-speech and non-co-speech gestures in both autobiographical and fictional narratives and examined viewpoint and the use of non-manual articulators, as well as the length of responses and narrative quality. The use of non-co-speech gestures was associated with longer narratives of equal or higher quality than those using only co-speech gestures. Non-co-speech gestures were most likely to adopt character-viewpoint and use non-manual articulators. The present study supports a deeper understanding of the term pantomime and its multimodal use by children in the integration of speech and gesture. (shrink)

Drawing on Arabic data sets, this study examines reported speech in naturally occurring conversations. Building on earlier work in discourse analysis, the study demonstrates how reported speech is a multiparty social field in which much of the reporting involves not only speech but also intricate forms of voice patterns and embodied reenactments. The study argues that speakers create an integrated complex of reporting, including multimodal utterances that go beyond the stream of speech to include relevant nonlinguistic (...) sounds and embodied gestures. Analysis also shows that participants engage in verbal and nonverbal forms of stance toward the reported activities. The speaker makes certain these stance displays are available to the addressees to achieve co-participation. In contrast to studies which restrict analysis to linguistic phenomena, this study approaches reported speech as a fine performance in which different kinds of semiotic resources are brought together. (shrink)

Speakers of many languages prefer allocentric frames of reference (FoRs) when talking about small‐scale space, using words like “east” or “downhill.” Ethnographic work has suggested that this preference is also reflected in how such speakers gesture. Here, we investigate this possibility with a field experiment in Juchitán, Mexico. In Juchitán, a preferentially allocentric language (Isthmus Zapotec) coexists with a preferentially egocentric one (Spanish). Using a novel task, we elicited spontaneous co‐speech gestures about small‐scale motion events (e.g., toppling blocks) (...) in Zapotec‐dominant speakers and in balanced Zapotec‐Spanish bilinguals. Consistent with prior claims, speakers’ spontaneous gestures reliably reflected either an egocentric or allocentric FoR. The use of the egocentric FoR was predicted—not by speakers’ dominant language or the language they used in the task—but by mastery of words for “right” and “left,” as well as by properties of the event they were describing. Additionally, use of the egocentric FoR in gesture predicted its use in a separate nonlinguistic memory task, suggesting a cohesive cognitive style. Our results show that the use of spatial FoRs in gesture is pervasive, systematic, and shaped by several factors. Spatial gestures, like other forms of spatial conceptualization, are thus best understood within broader ecologies of communication and cognition. (shrink)

For humans, the ability to communicate and use language is instantiated not only in the vocal modality but also in the visual modality. The main examples of this are sign languages and gestures. Sign languages, the natural languages of Deaf communities, use systematic and conventionalized movements of the hands, face, and body for linguistic expression. Co-speech gestures, though non-linguistic, are produced in tight semantic and temporal integration with speech and constitute an integral part of language together with (...) class='Hi'>speech. The articles in this issue explore and document how gestures and sign languages are similar or different and how communicative expression in the visual modality can change from being gestural to grammatical in nature through processes of conventionalization. As such, this issue contributes to our understanding of how the visual modality shapes language and the emergence of linguistic structure in newly developing systems. Studying the relationship between signs and gestures provides a new window onto the human ability to recruit multiple levels of representation in the service of using or creating conventionalized communicative systems. (shrink)

TThis paper aims to evidence the inherently metonymic nature of co-speech gestures. Arguing that motivation in gesture involves iconicity (similarity), indexicality (contiguity), and habit (conventionality) to varying degrees, it demonstrates how a set of metonymic principles may lend a certain systematicity to experientially grounded processes of gestural abstraction and enaction. Introducing visuo-kinetic signs as an umbrella term for co-speech gestures and signed languages, the paper shows how a frame-based approach to gesture may integrate different cognitive/functional linguistic (...) and semiotic accounts of metonymy (e.g., experiential domains, frame metonymy, contiguity, and pragmatic inferencing). The guiding assumption is that gestures metonymically profile deeply embodied, routinized aspects of familiar scenes, that is, the motivating context of frames. The discussion shows how gestures may evoke frame structures exhibiting varying degrees of groundedness, complexity, and schematicity: basic physical action and object frames; more complex frames; and highly abstract, complex frame structures. It thereby provides gestural evidence for the idea that metonymy is more basic and more directly experientially grounded than metaphor and thus often feeds into correlated metaphoric processes. Furthermore, the paper offers some initial insights into how metonymy also seems to induce the emergence of schematic patterns in gesture which may result from action-based and discourse-driven processes of habituation and conventionalization. It exemplifies how these forces may engender grammaticalization of a basic physical action into a gestural marker that shows strong metonymic form reduction, decreased transitivity, and interacting pragmatic functions. Finally, addressing basic metonymic operations in signed lexemes elucidates certain similarities regarding sign constitution in gesture and sign. English and German multimodal discourse data as well as German Sign Language (DGS) are drawn upon to illustrate the theoretical points of the paper. Overall, this paper presents a unified account of metonymy’s role in underpinning forms, functions, and patterns in visuo-kinetic signs. (shrink)

Researchers in the fields of sign language and gesture studies frequently present their participants with video stimuli showing actors performing linguistic signs or co-speech gestures. Up to now, such video stimuli have been mostly controlled only for some of the technical aspects of the video material, leaving open the possibility that systematic differences in video stimulus materials may be concealed in the actual motion properties of the actor’s movements. Computer vision methods such as OpenPose enable the fitting of (...) body-pose models to the consecutive frames of a video clip and thereby make it possible to recover the movements performed by the actor in a particular video clip without the use of a point-based or markerless motion-tracking system during recording. The OpenPoseR package provides a straightforward and reproducible way of working with these body-pose model data extracted from video clips using OpenPose, allowing researchers in the fields of sign language and gesture studies to quantify the amount of motion pertaining only to the movements performed by the actor in a video clip. These quantitative measures can be used for controlling differences in the movements of an actor in stimulus video clips or, for example, between different conditions of an experiment. In addition, the package also provides a set of functions for generating plots for data visualization, as well as an easy-to-use way of automatically extracting metadata from large sets of video files. (shrink)

When faced with an ambiguous pronoun, comprehenders use both multimodal cues and linguistic cues to identify the antecedent. While research has shown that gestures facilitate language comprehension, improve reference tracking, and influence the interpretation of ambiguous pronouns, literature on reference resolution suggests that a wide set of linguistic constraints influences the successful resolution of ambiguous pronouns and that linguistic cues are more powerful than some multimodal cues. To address the outstanding question of the importance of gesture as a cue (...) in reference resolution relative to cues in the speech signal, we have previously investigated the comprehension of contrastive gestures that indexed abstract referents – in this case expressions of personal preference – and found that such gestures did facilitate the resolution of ambiguous statements of preference. In this study, we extend this work to investigate whether the effect of gesture on resolution is diminished when the gesture indexes a statement that is less likely to be interpreted as the correct referent. Participants watched videos in which a speaker contrasted two ideas that were either neutral or moral acceptable). A gesture to the left or right side co-occurred with speech expressing each position. In gesture-disambiguating trials, an ambiguous phrase was accompanied by a gesture to one side or the other. In gesture non-disambiguating trials, no third gesture occurred with the ambiguous phrase. Participants were more likely to choose the idea accompanied by gesture as the stimulus speaker’s preference. We found no effect of scenario type. Regardless of whether the linguistic cue expressed a view that was morally charged or neutral, observers used gesture to understand the speaker’s opinion. This finding contributes to our understanding of the strength and range of cues, both linguistic and multimodal, that listeners use to resolve ambiguous references. (shrink)

This paper addresses the question of how emojis are integrated into the text that they occur with. I use the typology of gestural iconic enrichments proposed by Schlenker (2018a, 2018b) to investigate the hypothesis that emojis denoting objects (e.g., ????) and activities (e.g., ????) project (i.e., interact with logical operators) when co-occurring with text in a similar way as gestures do with speech. In particular, I claim that [i.] emojis generate co-suppositions, i.e., assertion-dependent presuppositions, when immediately following text (e.g., (...) the sentence “John didn’t train today ????” gives rise to the inference that “if John had trained today, weightlifting would have been involved”); [ii.] emojis generate supplements (in the sense of Potts, 2005), just like appositive relative clauses, when they are separated from the accompanying text by a pause (e.g., the sentence “John trained today...????” gives rise to the inference that “John trained today, which involved weightlifting”); [iii.] when used as replacements for words (e.g., “Yesterday, John didn’t ???? ”), emojis have an at-issue semantics and can trigger standard presuppositions. (shrink)

Height-pitch associations are claimed to be universal and independent of language, but this claim remains controversial. The present study sheds new light on this debate with a multimodal analysis of individual sound and melody descriptions obtained in an interactive communication paradigm with speakers of Dutch and Farsi. The findings reveal that, in contrast to Dutch speakers, Farsi speakers do not use a height-pitch metaphor consistently in speech. Both Dutch and Farsi speakers’ co-speech gestures did reveal a mapping of (...) higher pitches to higher space and lower pitches to lower space, and this gesture space-pitch mapping tended to co-occur with corresponding spatial words (high-low). However, this mapping was much weaker in Farsi speakers than Dutch speakers. This suggests that cross-linguistic differences shape the conceptualization of pitch and further calls into question the universality of height-pitch associations. (shrink)

The canonical history of mathematics suggests that the late 19th-century “arithmetization” of calculus marked a shift away from spatial-dynamic intuitions, grounding concepts in static, rigorous definitions. Instead, we argue that mathematicians, both historically and currently, rely on dynamic conceptualizations of mathematical concepts like continuity, limits, and functions. In this article, we present two studies of the role of dynamic conceptual systems in expert proof. The first is an analysis of co-speech gesture produced by mathematics graduate students while proving (...) a theorem, which reveals a reliance on dynamic conceptual resources. The second is a cognitive-historical case study of an incident in 19th-century mathematics that suggests a functional role for such dynamism in the reasoning of the renowned mathematician Augustin Cauchy. Taken together, these two studies indicate that essential concepts in calculus that have been defined entirely in abstract, static terms are nevertheless conceptualized dynamically, in both contemporary and historical practice. (shrink)

Negation can be considered a shared social action that develops since early infancy with very basic acts of refusals or rejection. Inspired by an approach to the embodiment of concepts known as Multiple Representation Theories, the present paper explores negation as an embodied action that relies on both sensorimotor and linguistic/social information. Despite the different variants, MRT accounts share the basic ideas that both linguistic/social and sensorimotor information concur to the processes of concepts formation and representation and that the balance (...) between these components depends on the kind of concept, the context, or the performed task. In the present research we will apply the MRT framework for exploring negation in Italian sign language. The nature of negation in LIS has been explored in continuity with the co-speech gesture where negative elements are encoded through differentiated prosodic and gestural strategies across languages. Data have been collected in naturalistic settings that may allow a much wider understanding of negation both in speech and in spoken language with a semi-structured interview. Five LIS participants with age range 30–80 were recruited and interviewed with the aim of understanding the continuity between gesture and sign in negation. Results highlight that negation utterances mirror the functions of rejection, non-existence and denial that have been described in language acquisition both in deaf and hearing children. These different steps of acquisition of negation show a different balance between sensorimotor, linguistic and social information in the construction of negative meaning that the MRT is able to enlighten. (shrink)

As part of a recent attempt to extend the methods of formal semantics beyond language, it has been claimed that music has an abstract truth-conditional semantics, albeit one that has more in common with iconic semantics than with standard compositional semantics. After summarizing this approach and addressing a common objection, we argue that music semantics should be enriched in three directions by incorporating insights of other areas of Super Semantics. First, it has been claimed by Abusch 2013 that visual narratives (...) make use of discourse referents akin to those we find in language. We argue that a similar conclusion extends to music, and we highlight it by investigating ways in which orchestration and dance may make cross-referential dependencies more explicit. Second, we show that by bringing music semantics closer to the semantics of visual narratives, we can give an account of the semantics of mixed visual and musical sequences. Third, it has been claimed that co-speech gestures trigger characteristic conditionalized presuppositions, called ‘cosuppositions’, and that their semantic status derives from their parasitic character relative to words. We argue that the same conclusion extends to some instances of film and cartoon music: it may trigger cosuppositions that can be revealed by embedding film excerpts or gifs in sentences so as to test presupposition projection. We further argue that under special discourse conditions, pro-speech gestures and pro-speech music alike can trigger cosuppositions as well. These results suggest that new insights can be gained not just by extending the methods of semantics to new objects, but also by drawing new connections among them. (shrink)

Human communication can be either monosemiotic or polysemiotic, depending on whether it combines ensembles of representations from one or more semiotic systems such as language, gesture and depiction. Each semiotic system has its unique storytelling potentials, which makes intersemiotic translation from one system to another challenging. We investigated the influence of the source semiotic system, realised in speech and a sequence of pictures, respectively, on the way the same story was retold using speech and co-speech gestures. (...) The story was the content of the picture book Frog, Where Are You?. A group of Finnish speakers saw the story in pictures, and another group heard it in matched oral narration. Each participant retold the story to an addressee and all narrations were video-recorded and analysed for both speech and gestures. Given the high degree of iconicity in depiction, we expected more iconic gestures (especially enactments) in the narratives translated from pictures than in those translated from speech. Conversely, we expected greater narrative coherence in the narratives translated from speech. The results showed that more iconic gestures were produced in the narratives translated from speech, but these were primarily not from the enactment subtype. As expected, iconic enactments were more frequent in the narratives translated from the story presented in pictures. The narratives produced by participants who had only heard the story did not have a greater variety of connective devices, yet the type of devices differed slightly between the groups. Together with some additional differences between the groups that had not been anticipated, the results indicate that a story presented in different semiotic systems tends to be translated into different polysemiotic narratives. (shrink)

The present study claims that the use of spontaneous gestures can contribute to accomplishing coherence in sequential conversational exchanges. Three types of speech-accompanying gestures were analyzed in Chinese conversation. Based on the syntactic-semantic relation with the co-occurring utterance, the first type is associated with words that do not convey explicit meaning; the second type is associated with covert constituents; the third type does not have any linguistic affiliates. They suggest different ways in which gesture adds information to the (...) propositional content of the associated utterance or to the content of the speech event under discussion. They also play a role in achieving conversational coherence: on the part of the speaker, it is only by using the various types of gesture that a current speaker can make the message to be conveyed in the utterance complete. On the part of the next speaker, the recognition and negotiation with the former speaker of the gestured information is a way to acknowledge or confirm mutual knowledge. The collaborative efforts contribute to the coherence of the talk in the subsequent interaction. (shrink)

In this paper, I argue that an important component of the language-ready brain is the ability to recognize and conceptualize events. By ‘event’, I mean any situation or activity in the world or our mental life, that we find salient enough to individuate as a thought or word. While this may sound either trivial or non-unique to humans, I hope to show that abstracting away events and their participants from the embodied flow of experience is a characteristic unique to humans. (...) This ability is enabled, I will argue, by two critical competencies that act as scaffolds for language-ready thought in the prehuman brain. The first, as argued by Arbib (2006,2012,2016) and others, is a sophisticated system of gesture production and understanding in prehumans, which provided a template for schema-like sequencing and slot-filling of information units. The second involves the integration of multiple modalities of expression in the communicative act, in particular, the alignment of co-gestural speech and co-speech gesture. With such computational facilities, action-based gestures can be abstracted away from their associated objects and become full event representations. This view supports the MSH argument for the emergence of more complex linguistic expressions from initially holophrastic units. In particular, actions can be thought of as protoverbs, which through this process are abstracted to full event descriptions, i.e., verbs. (shrink)

When people are engaged in social interaction, they can repeat aspects of each other’s communicative behavior, such as words or gestures. This kind of behavioral alignment has been studied across a wide range of disciplines and has been accounted for by diverging theories. In this paper, we review various operationalizations of lexical and gestural alignment. We reveal that scholars have fundamentally different takes on when and how behavior is considered to be aligned, which makes it difficult to compare findings and (...) draw uniform conclusions. Furthermore, we show that scholars tend to focus on one particular dimension of alignment (traditionally, whether two instances of behavior overlap in form), while other dimensions remain understudied. This hampers theory testing and building, which requires a well‐defined account of the factors that are central to or might enhance alignment. To capture the complex nature of alignment, we identify five key dimensions to formalize the relationship between any pair of behavior: time, sequence, meaning, form, and modality. We show how assumptions regarding the underlying mechanism of alignment (placed along the continuum of priming vs. grounding) pattern together with operationalizations in terms of the five dimensions. This integrative framework can help researchers in the field of alignment and related phenomena (including behavior matching, mimicry, entrainment, and accommodation) to formulate their hypotheses and operationalizations in a more transparent and systematic manner. The framework also enables us to discover unexplored research avenues and derive new hypotheses regarding alignment. (shrink)

The human experience is shaped by information from different perceptual channels, but it is still debated whether and how differential experience influences language use. To address this, we compared congenitally blind, blindfolded, and sighted people's descriptions of the same motion events experienced auditorily by all participants (i.e., via sound alone) and conveyed in speech and gesture. Comparison of blind and sighted participants to blindfolded participants helped us disentangle the effects of a lifetime experience of being blind versus the (...) task-specific effects of experiencing a motion event by sound alone. Compared to sighted people, blind people's speech focused more on path and less on manner of motion, and encoded paths in a more segmented fashion using more landmarks and path verbs. Gestures followed the speech, such that blind people pointed to landmarks more and depicted manner less than sighted people. This suggests that visual experience affects how people express spatial events in the multimodal language and that blindness may enhance sensitivity to paths of motion due to changes in event construal. These findings have implications for the claims that language processes are deeply rooted in our sensory experiences. (shrink)

Challenges for extending the mirror system hypothesis include mechanisms supporting planning, conversation, motivation, theory of mind, and prosody. Modeling remains relevant. Co-speech gestures show how manual gesture and speech intertwine, but more attention is needed to the auditory system and phonology. The holophrastic view of protolanguage is debated, along with semantics and the cultural basis of grammars. Anatomically separated regions may share an evolutionary history.