This paper presents a proposal for the description of gesture phases derived from articulatory characteristics observable in their execution. Based on the results of an explorative study examining the execution of gesture phases of ten German speakers, the paper presents two sets of articulatory features, i.e., distinctive and additional features by which gesture phases are characterized from a context-independent and context-sensitive point of view. It will be shown that gesture phases show a particular distribution of the features, thus distinguishing one (...) phase from another. Furthermore, changes in the execution of phases in linear successions can be described by means of features. Contrary to other accounts, whose focus on gesture phases is primarily in relation to speech and/or adjacent phases, this proposal concentrates on the visible physical characteristics of gesture phases. (shrink)

This study investigates whether there is a relation between the semantics of linguistic expressions that indicate temporal distance and the spatial properties of their co-speech gestures. To this date, research on time gestures has focused on features such as gesture axis, direction, and shape. Here we focus on a gesture property that has been overlooked so far: the distance of the gesture in relation to the body. To achieve this, we investigate two types of temporal linguistic expressions are addressed: proximal (...) (e.g., near future, near past) and distal (e.g., distant past, distant future). Data was obtained through the NewsScape library, a multimodal corpus of television news. A total of 121 co-speech gestures were collected and divided into the two categories. The gestures were later annotated in terms of gesture space and classified in three categories: (i) center, (ii) periphery, and (iii) extreme periphery. Our results suggest that gesture and language are coherent in the expression of temporal distance: when speakers locate an event far from them, they tend to gesture further from their body; similarly, when locating an event close to them, they gesture closer to their body. These results thus reveal how co-speech gestures also reflect a space-time mapping in the dimension of distance. (shrink)

Two eye-tracking experiments investigated how and when pointing gestures and location descriptions affect target identification. The experiments investigated the effect of gestures and referring expressions on the time course of fixations to the target, using videos of human gestures and human voice, and animated gestures and synthesized speech. Ambiguous, yet informative pointing gestures elicited attention and facilitated target identification, akin to verbal location descriptions. Moreover, target identification was superior when both pointing gestures and verbal location descriptions were used. These findings (...) suggest that gesture not only operates as a context to verbal descriptions, or that verbal descriptions operate as a context to gesture, but that they complement one another in reference resolution. (shrink)

Visual information conveyed by iconic hand gestures and visible speech can enhance speech comprehension under adverse listening conditions for both native and non‐native listeners. However, how a listener allocates visual attention to these articulators during speech comprehension is unknown. We used eye‐tracking to investigate whether and how native and highly proficient non‐native listeners of Dutch allocated overt eye gaze to visible speech and gestures during clear and degraded speech comprehension. Participants watched video clips of an actress uttering a clear or (...) degraded (6‐band noise‐vocoded) action verb while performing a gesture or not, and were asked to indicate the word they heard in a cued‐recall task. Gestural enhancement was the largest (i.e., a relative reduction in reaction time cost) when speech was degraded for all listeners, but it was stronger for native listeners. Both native and non‐native listeners mostly gazed at the face during comprehension, but non‐native listeners gazed more often at gestures than native listeners. However, only native but not non‐native listeners' gaze allocation to gestures predicted gestural benefit during degraded speech comprehension. We conclude that non‐native listeners might gaze at gesture more as it might be more challenging for non‐native listeners to resolve the degraded auditory cues and couple those cues to phonological information that is conveyed by visible speech. This diminished phonological knowledge might hinder the use of semantic information that is conveyed by gestures for non‐native compared to native listeners. Our results demonstrate that the degree of language experience impacts overt visual attention to visual articulators, resulting in different visual benefits for native versus non‐native listeners. (shrink)