Elsevier

Cognitive Science

Volume 28, Issue 4, July–August 2004, Pages 611-619
Cognitive Science

Moving words: dynamic representations in language comprehension

https://doi.org/10.1016/j.cogsci.2004.03.004Get rights and content

Abstract

Eighty-two participants listened to sentences and then judged whether two sequentially presented visual objects were the same. On critical trials, participants heard a sentence describe the motion of a ball toward or away from the observer (e.g., “The pitcher hurled the softball to you”). Seven hundred and fifty milliseconds after the offset of the sentence, a picture of an object was presented for 500 ms, followed by another picture. On critical trials, the two pictures depicted the kind of ball mentioned in the sentence. The second picture was displayed 175 ms after the first. Crucially, it was either slightly larger or smaller than the first picture, thus suggesting movement of the ball toward or away from the observer. Participants responded more quickly when the implied movement of the balls matched the movement described in the sentence. This result provides support for the view that language comprehension involves dynamic perceptual simulations.

Introduction

In a famous experiment, Loftus and Palmer (1974) showed participants short traffic safety films of car accidents. They subsequently asked participants in different conditions how fast the cars were going when they ‘contacted,’ ‘bumped into,’ ‘collided with,’ ‘smashed into,’ or ‘hit’ each other. The speed implied by the verb in the question affected participants’ judgments, such that more intense verbs led to higher estimates than less intense verbs. This finding has most often been used to demonstrate the unreliability of eyewitness testimony, or its malleability by post-event questions, but there is another intriguing aspect of this finding that has not received much attention. The finding suggests that language comprehension may affect the visual representation of a motion event. In this article, we examine whether language comprehension routinely involves the activation of visual motion representations. This question is warranted by findings in two domains: imagery and language comprehension. Specifically, there is empirical evidence that cognition involves dynamic visual representations, and there is evidence that language comprehension routinely involves the activation of (static) visual representations. These bodies of findings are discussed in the next two sections.

Research on what has become known as ‘representational momentum,’ has shown that mental representations can be dynamic (e.g., Freyd & Finke, 1984; Hubbard & Barucha, 1998; see Shepard & Metzler, 1971, for a seminal study on dynamic mental representations). For example, in the prototypical paradigm, participants are presented with a sequence of three pictures of a rectangle, with each rectangle slightly rotated relative to the previous one to suggest continuous rotation. Participants are then shown a recognition probe presented 250 ms after the last of the three-picture sequence. They generally false alarm to probes depicting the rectangle as slightly more rotated in the direction of the implied rotation than the last seen shape actually was (Freyd & Finke, 1984). Findings such as these are explained by assuming that the rectangle continued its rotation in the participants’ mental representation after the last picture was shown, hence the term representational momentum. The extent to which representational momentum occurs appears to depend on context. For example, a sequence of three pointed shapes moving upward on a computer screen produces more representational momentum when the participant is told the shape represents a rocket ship than when the participant is told it represents a steeple (Reed & Vinson, 1996; Vinson & Reed, 2002). Brain-imaging studies have provided converging evidence for dynamic mental representations. Still pictures of animate entities in motion generate more activity in the medial temporal cortex, a brain region involved in motion perception, than do pictures of those same entities in a stationary position (Kourtzi & Kanwisher, 2000). There currently exists no unified theoretical explanation for findings such as these (Thornton & Hubbard, 2002), but the evidence is commonly interpreted as supporting the notion of dynamic mental representations (e.g., Freyd, 1987; Wallis & Bülthoff, 1999). In accordance with Wallis and Bülthoff, among others, we assume that dynamic mental object representations are the result of spatio-temporal associations between visual patterns acquired during experience of our environment.

The assumption that language-like abstract representations, such as propositions, form the building blocks of cognition (e.g., Pylyshyn, 1986) has recently been challenged by research demonstrating that people routinely activate perceptual representations during language comprehension—the cognitive skill that intuitively would seem to be the one most likely to involve propositional representations—(Richardson, Spivey, Barsalou, & McRae, 2003; Spivey & Geng, 2001; Stanfield & Zwaan, 2001; Zwaan, Stanfield, & Yaxley, 2002; Zwaan and Yaxley, 2003a, Zwaan and Yaxley, 2003b). Importantly, these studies employed implicit tasks, such as recognition or naming, thus demonstrating that perceptual information is routinely activated, even when doing so does not facilitate task performance.

These findings are consistent with the general idea that language comprehension is a perceptual simulation of the described situation (Barsalou, 1999, Glenberg, 1997, Zwaan, 2004). During our interaction with the world, we store traces in memory of perceptions and actions filtered through selective attention. These traces become associated with words (themselves traces of perceiving or producing sound or visual patterns). During language comprehension, these traces are reactivated to produce a perceptual simulation of the described situation. This line of reasoning provides a straightforward explanation for the findings of Zwaan et al. (2002). Participants in that study read sentences from a computer screen and were then shown a picture, which they had to recognize (Experiment 1) or name (Experiment 2). On experimental trials, the picture always showed an object or animal that was mentioned in the sentence. However, the shape of this entity was manipulated to match or mismatch the shape implied by the sentence. Thus, a picture of an eagle could follow a sentence such as “He saw an eagle in the sky” with its wings outstretched (match) or with its wings drawn in (mismatch). Both recognition and naming responses were significantly faster in the match than in the mismatch condition. This finding is not predicted by amodal theories of cognition (e.g., Pylyshyn, 1986). These theories represent the eagle as an argument node in a propositional network, or as a list of features, thus failing to capture the fact that its shape may change according to the location that it is in. In contrast, the perceptual-simulation hypothesis has a natural account for this finding. Visual traces of soaring and perched eagles are stored in memory. These traces are reactivated during comprehension, with the most contextually consistent traces receiving the most activation and thus being the most likely to be incorporated in the simulation. Seeing the picture during the experiment produces a new visual trace. In the match condition, this trace will be more similar to the trace activated by the sentence than in the mismatch condition, so that a comparison can be made more rapidly, thus producing faster recognition and naming responses.

Section snippets

The present study

In this study, we combined and extended the logic behind theories of representational momentum and theories of language comprehension as perceptual simulation. We assume that dynamic mental representations are perceptual traces that are stored as temporal patterns of activation that unfold over time corresponding to a certain perceptual experience. Extending the logic behind the Stanfield and Zwaan (2001) and the Zwaan et al. (2002) experiments, we predicted that comprehension of a sentence

Conclusion

The findings reported in this article add to the growing body of evidence that language comprehension routinely involves the activation of perceptual representations (Pecher, Zeelenberg, & Barsalou, 2003; Richardson et al., 2003; Stanfield & Zwaan, 2001; Zwaan et al., 2002; Zwaan and Yaxley, 2003a, Zwaan and Yaxley, 2003b). However, the current findings constitute an advance over this earlier research in several ways. Most importantly, they demonstrate that language comprehension may involve

Acknowledgements

We thank Meredith Lynam, Michelle Peruche, Raymond Britton, and Greg Smith for assistance with data collection. This research was supported by grant MH-63972 to R.A.Z.

References (23)

  • D. Pecher et al.

    Verifying properties from different modalities for concepts produces switching costs

    Psychological Science

    (2003)
  • Cited by (192)

    View all citing articles on Scopus

    Supplementary data associated with this article can be found, in the online version, at doi:10.1016/j.cogsci.2004.03.004.

    View full text