Reading-time evidence for automaticity of event segmentation

Studies of the pace of reading during narrative text comprehension indicate that readers slow down at event boundaries. These studies arise mostly from discourse comprehension theories, which propose that readers construct a series of mental models of a situation described by a narrative. In the Structure Building Framework [12], new mental models are initiated when the text refers to a new set of people, places or things, when the action in the text changes its temporal or spatial location, or when a new causal sequence is initiated. In the Event Indexing Model [13], new mental models are initiated when there is a change in space, time, protagonist, objects, goals or causes. When the changes identified by these models occur, readers have been found to read more slowly. This is consistent with the processing load hypothesis put forward by the Event Indexing Model, which states that changes in situational features increase the difficulty of integrating newly encountered information into the current mental model. In one such study [14], participants read two literary narratives one sentence at a time on a computer screen, and reading time was recorded. The texts were coded to identify changes in time, space and causal contingency. Reading time increased consistently at changes in time and cause; reading time sometimes increased at spatial changes, but this depended on the previous knowledge of the reader and task goals. More recent work has found that reading time increases at shifts in characters and their goals [15,16]. Such results support the notion that reading times increase at event boundaries because, as noted previously, event boundaries are associated with changes in time, space, causes, characters and goals. Although most reading-time studies have not measured event segmentation, two recent studies have directly compared event segmentation and narrative reading time. In one [17], cues to event boundaries were experimentally manipulated by changing the temporal contiguity of events (Figure 1). Phrases marking a temporal discontinuity (‘an hour later’) increased the likelihood that a clause would be identified as an event boundary during segmentation, and slowed reading during self-paced comprehension. The second study looked at the relation between event segmentation and reading time correlationally [8]. Two groups of participants read narratives about a young boy. One group segmented them into events; the other read them one clause at a time while reading time was recorded. Clauses in which more readers identified event boundaries produced longer reading times. These results support the view that when comprehenders encounter boundaries between events they perform extra processing operations.

Open in a separate window

Figure 1

Temporal changes are perceived as event boundaries, and this affects working memory [17]. Participants read narratives containing sentences (marked here in italics) that either indicated a significant interval of time had passed ('an hour later') or not ('a moment later'). These temporal reference sentences often were identified as event boundaries – particularly in the 'hour later' condition. Subsequent experiments assessed memory for objects mentioned before the temporal shift (creek, in this example). Memory was tested in two ways: by measuring the time it took to read an anaphoric reference to the previously mentioned object, or by asking directly whether the participant recognized the word as having been read. Bottom left: Sentences in the 'hour later' condition were read more slowly than the nearly identical 'moment later' versions. Bottom middle: Anaphoric references to previously mentioned objects were read more slowly in the 'hour later' condition. Bottom right: Recognition memory for previously mentioned objects was less accurate in the 'hour later' condition. Such results indicate that working memory is updated at event boundaries. Error bars are standard errors.

Neurophysiological evidence for automaticity of event segmentation

Converging evidence for the automaticity of event segmentation comes from noninvasive neurophysiological measures, including functional magnetic resonance imaging (fMRI) and electroencephalography (EEG). These studies have been motivated by the hypothesis that if the brain is undertaking processing that relates to event boundaries, transient changes in brain activity should be observed at those points in time corresponding to event boundaries – whether or not you are attending to event segmentation. In one study [18], participants passively viewed short movies of everyday activities while their brain activity was recorded with fMRI. During the initial viewing and fMRI data recording, participants were asked simply to watch the movies and try to remember as much as possible. In the second phase of the experiment these participants segmented the movies into events. Event boundaries were associated with increases in brain activity during passive viewing in bilateral posterior occipital, temporal and parietal cortex and right lateral frontal cortex. The posterior activation included the MT complex [11], an area associated with the processing of motion [19]. A subsequent study using simple animations of geometric objects found that activity in the MT complex was correlated with the speed of motion of objects, and with the presence of event boundaries [20].

Whereas most of the neurophysiological research on event perception involves visual events, a new study by Sridharan et al. [21] investigated the perception of event structure in music. This study examined the extent to which musically untrained listeners use transitions between movements to segment classical pieces into coarse-grained events. A particular advantage in studying musical movements is that they provide objective, normative events. As Figure 2 illustrates, two dissociable networks in the right hemisphere were selectively responsive at transitions between movements: an early-responding ventral network that included the ventrolateral prefrontal cortex and posterior temporal cortex, and a late-responding dorsal network that included the dorsolateral prefrontal cortex and posterior parietal cortex.

Open in a separate window

Figure 2

The boundaries between musical movements elicited increased activity at a network of brain regions. (a) In Western concert music, symphonies are made up of movements. The sound wave is plotted, and the breaks between movements are illustrated with red lines. In [21], musically untrained participants listened to two 8–10-min segments of symphonies by William Boyce, consisting of movements lasting an average of 1 min and 10 s, while their brain activity was recorded using fMRI. (b) A ventral network including the ventrolateral prefrontal cortex (VLPFC) and posterior temporal cortex (TEMPORAL) increased in activity first at movement boundaries. (c) A dorsal network including the dorsolateral prefrontal cortex (DLPFC) and posterior parietal (PPC) increased in activity slightly later. Both networks were lateralized to the right hemisphere. The authors interpreted these data by suggesting that the response of the ventral network reflected the processing of violations of musical expectancy, whereas the response of the dorsal network reflected consequent top-down modulation of the processing of new musical information. Reproduced, with permission, from [21].

fMRI data also have provided evidence for automatic event segmentation during reading. Speer et al. [22] got participants to read narrative texts, one word at a time, while brain activity was recorded with fMRI. Participants subsequently segmented the texts into events. Brain activity increased at the later-identified event boundaries in a set of regions that corresponded substantially with the regions that increased at event boundaries in movies; these included bilateral regions in medial posterior temporal, occipital and parietal cortex, in the lateral temporal-parietal and anterior temporal cortex, and in the right posterior dorsal frontal cortex.

Converging with the fMRI data, recent experiments using EEG indicate that perceptual processing is modulated at event boundaries on an ongoing basis. In one set of experiments [23], participants viewed movies of goal-directed activities while undergoing EEG recording, after which they segmented the movies into events. Evoked responses were detected at frontal and parietal electrode sites, and these responses were modulated by whether or not participants were familiar with the movies themselves or with the activities they depicted.

Finally, measurements of pupil diameter provide further evidence that information processing increases transiently after the perception of an event boundary. Pupil diameter provides an online measure of cognitive processing load, as demonstrated in studies using a variety of motor and cognitive paradigms [24]. In one study [25], participants viewed movies while their pupil diameters and eye movements were recorded. They then segmented the movies into events. Pupil diameter transiently increased following those points that were later identified as event boundaries. (Saccades also became more frequent around event boundaries, which might reflect the reorienting of viewers at the beginning of a new event.)

Together, the reading-time, neurophysiological and oculomotor data strongly suggest that there are transient changes in brain activity correlated with the subjective experience that one event has ended and another has begun. We can be confident these effects are independent of task demands such as the conscious intention to segment activity or covert attention to the location of event boundaries, because for all the studies discussed here the crucial reading-time or physiological data were collected before participants were introduced to the event segmentation task. An important question for ongoing research is: to what extent do such effects reflect neural processing that has a causal role in event segmentation, and to what extent do they reflect processing that is correlated with the presence of event boundaries but not a cause or consequence of event segmentation as such?

Working memory for events in text

Studies of memory access during text reading generally have not directly measured event boundaries, but have found evidence that those cues associated with event boundaries reduce memory access. Memory for recently mentioned objects is poorer after reading sentences that indicate a shift in time or space [37–41]. As noted previously, one study [17] experimentally manipulated the presence of event boundaries in narratives by varying the passage of time indicated in an auxiliary phrase. As is shown in Figure 1, implicit and explicit measures of memory indicated that information became less accessible following a time shift.

Working memory for events in pictures

Gernsbacher [42] got participants to identify episode boundaries in a picture story. New participants then viewed the picture story, and from time to time were probed to discriminate pictures that had recently been presented in the story from pictures that were left–right reversed. Recognition was better for images from within the current episode than for images from the previous episode, indicating that some of the surface information in the pictures was less available once a new event had begun.

Working memory for events in cinema and virtual reality

Converging with the results from text and picture stories, experiments looking at recognition memory for objects in cinema and virtual reality indicate that working memory is updated at event boundaries. In one recent study [43], participants navigated in a virtual reality environment, and memory for recently appearing objects was tested. Memory was reduced after walking through a door into an adjacent room – a probable event boundary. In another recent set of studies [44], participants viewed excerpts from movies that were occasionally interrupted by recognition memory tests for objects that had been on the screen five seconds previously. The information that participants could retrieve about these objects differed systematically depending on whether an event boundary had occurred during those five seconds. Further, neuroimaging data suggested that the basis for responding also differed: retrieval of information about objects from within the current event selectively activated brain areas including the bilateral occipital and lateral temporal cortices and the right inferior lateral frontal cortex, whereas retrieval of information about objects from the previous event selectively activated brain areas including the medial temporal cortex and medial parietal cortex. The medial temporal regions correspond to the hippocampal formation, which is known to have a key role in long-term memory storage and retrieval [45]. This is consistent with the hypothesis that participants depended preferentially on working memory for within-event retrieval and shifted to long-term memory for across-event retrieval – although the information to be retrieved was only a few seconds old.

Preparation of this article was supported in part by grants R01-MH070674 and T32 AG000030–31 from the National Institutes of Health. We thank Devarajan Sridharan for providing figure materials, and thank the following colleagues for thoughtful comments on the manuscript: Dare Baldwin, Joe Magliano, G.A. Radvansky, Stephan Schwan, Ric Sharp, Khena Swallow and Barbara Tversky.