Moving words: dynamic representations in language comprehension☆
Introduction
In a famous experiment, Loftus and Palmer (1974) showed participants short traffic safety films of car accidents. They subsequently asked participants in different conditions how fast the cars were going when they ‘contacted,’ ‘bumped into,’ ‘collided with,’ ‘smashed into,’ or ‘hit’ each other. The speed implied by the verb in the question affected participants’ judgments, such that more intense verbs led to higher estimates than less intense verbs. This finding has most often been used to demonstrate the unreliability of eyewitness testimony, or its malleability by post-event questions, but there is another intriguing aspect of this finding that has not received much attention. The finding suggests that language comprehension may affect the visual representation of a motion event. In this article, we examine whether language comprehension routinely involves the activation of visual motion representations. This question is warranted by findings in two domains: imagery and language comprehension. Specifically, there is empirical evidence that cognition involves dynamic visual representations, and there is evidence that language comprehension routinely involves the activation of (static) visual representations. These bodies of findings are discussed in the next two sections.
Research on what has become known as ‘representational momentum,’ has shown that mental representations can be dynamic (e.g., Freyd & Finke, 1984; Hubbard & Barucha, 1998; see Shepard & Metzler, 1971, for a seminal study on dynamic mental representations). For example, in the prototypical paradigm, participants are presented with a sequence of three pictures of a rectangle, with each rectangle slightly rotated relative to the previous one to suggest continuous rotation. Participants are then shown a recognition probe presented 250 ms after the last of the three-picture sequence. They generally false alarm to probes depicting the rectangle as slightly more rotated in the direction of the implied rotation than the last seen shape actually was (Freyd & Finke, 1984). Findings such as these are explained by assuming that the rectangle continued its rotation in the participants’ mental representation after the last picture was shown, hence the term representational momentum. The extent to which representational momentum occurs appears to depend on context. For example, a sequence of three pointed shapes moving upward on a computer screen produces more representational momentum when the participant is told the shape represents a rocket ship than when the participant is told it represents a steeple (Reed & Vinson, 1996; Vinson & Reed, 2002). Brain-imaging studies have provided converging evidence for dynamic mental representations. Still pictures of animate entities in motion generate more activity in the medial temporal cortex, a brain region involved in motion perception, than do pictures of those same entities in a stationary position (Kourtzi & Kanwisher, 2000). There currently exists no unified theoretical explanation for findings such as these (Thornton & Hubbard, 2002), but the evidence is commonly interpreted as supporting the notion of dynamic mental representations (e.g., Freyd, 1987; Wallis & Bülthoff, 1999). In accordance with Wallis and Bülthoff, among others, we assume that dynamic mental object representations are the result of spatio-temporal associations between visual patterns acquired during experience of our environment.
The assumption that language-like abstract representations, such as propositions, form the building blocks of cognition (e.g., Pylyshyn, 1986) has recently been challenged by research demonstrating that people routinely activate perceptual representations during language comprehension—the cognitive skill that intuitively would seem to be the one most likely to involve propositional representations—(Richardson, Spivey, Barsalou, & McRae, 2003; Spivey & Geng, 2001; Stanfield & Zwaan, 2001; Zwaan, Stanfield, & Yaxley, 2002; Zwaan and Yaxley, 2003a, Zwaan and Yaxley, 2003b). Importantly, these studies employed implicit tasks, such as recognition or naming, thus demonstrating that perceptual information is routinely activated, even when doing so does not facilitate task performance.
These findings are consistent with the general idea that language comprehension is a perceptual simulation of the described situation (Barsalou, 1999, Glenberg, 1997, Zwaan, 2004). During our interaction with the world, we store traces in memory of perceptions and actions filtered through selective attention. These traces become associated with words (themselves traces of perceiving or producing sound or visual patterns). During language comprehension, these traces are reactivated to produce a perceptual simulation of the described situation. This line of reasoning provides a straightforward explanation for the findings of Zwaan et al. (2002). Participants in that study read sentences from a computer screen and were then shown a picture, which they had to recognize (Experiment 1) or name (Experiment 2). On experimental trials, the picture always showed an object or animal that was mentioned in the sentence. However, the shape of this entity was manipulated to match or mismatch the shape implied by the sentence. Thus, a picture of an eagle could follow a sentence such as “He saw an eagle in the sky” with its wings outstretched (match) or with its wings drawn in (mismatch). Both recognition and naming responses were significantly faster in the match than in the mismatch condition. This finding is not predicted by amodal theories of cognition (e.g., Pylyshyn, 1986). These theories represent the eagle as an argument node in a propositional network, or as a list of features, thus failing to capture the fact that its shape may change according to the location that it is in. In contrast, the perceptual-simulation hypothesis has a natural account for this finding. Visual traces of soaring and perched eagles are stored in memory. These traces are reactivated during comprehension, with the most contextually consistent traces receiving the most activation and thus being the most likely to be incorporated in the simulation. Seeing the picture during the experiment produces a new visual trace. In the match condition, this trace will be more similar to the trace activated by the sentence than in the mismatch condition, so that a comparison can be made more rapidly, thus producing faster recognition and naming responses.
Section snippets
The present study
In this study, we combined and extended the logic behind theories of representational momentum and theories of language comprehension as perceptual simulation. We assume that dynamic mental representations are perceptual traces that are stored as temporal patterns of activation that unfold over time corresponding to a certain perceptual experience. Extending the logic behind the Stanfield and Zwaan (2001) and the Zwaan et al. (2002) experiments, we predicted that comprehension of a sentence
Conclusion
The findings reported in this article add to the growing body of evidence that language comprehension routinely involves the activation of perceptual representations (Pecher, Zeelenberg, & Barsalou, 2003; Richardson et al., 2003; Stanfield & Zwaan, 2001; Zwaan et al., 2002; Zwaan and Yaxley, 2003a, Zwaan and Yaxley, 2003b). However, the current findings constitute an advance over this earlier research in several ways. Most importantly, they demonstrate that language comprehension may involve
Acknowledgements
We thank Meredith Lynam, Michelle Peruche, Raymond Britton, and Greg Smith for assistance with data collection. This research was supported by grant MH-63972 to R.A.Z.
References (23)
- et al.
Reconstruction of automobile destruction: An example of the interaction between language and memory
Journal of Verbal Learning & Verbal Behavior
(1974) - et al.
Spatial representations activated during real-time comprehension of verbs
Cognitive Science
(2003) - et al.
Learning to recognize objects
Trends in Cognitive Sciences
(1999) - et al.
Hemispheric differences in semantic-relatedness judgments
Cognition
(2003) Perceptual symbol systems
Behavioral and Brain Sciences
(1999)Dynamic mental representations
Psychological Review
(1987)- et al.
Representational momentum
Journal of Experimental Psychology: Learning, Memory, & Cognition
(1984) What memory is for
Behavioral & Brain Sciences
(1997)- et al.
Judged displacement in apparent vertical and horizontal motion
Perception & Psychophysics
(1998) - et al.
Activation in human MT/MST by static images with implied motion
Journal of Cognitive Neuroscience
(2000)
Verifying properties from different modalities for concepts produces switching costs
Psychological Science
Cited by (192)
Demystifying wine expertise through the lens of imagination: Descriptions and imagery vividness across sensory modalities
2024, Food Research InternationalThe role of visual imagery in story reading: Evidence from aphantasia
2024, Consciousness and CognitionThe activation of object-state representations during online language comprehension
2020, Acta PsychologicaAnalysis of activation mechanisms of facial expressions of emotions with a priming paradigm
2020, Psychologie FrancaiseNodding and shaking of the head as simulated approach and avoidance responses
2020, Acta Psychologica
- ☆
Supplementary data associated with this article can be found, in the online version, at doi:10.1016/j.cogsci.2004.03.004.