Abstract

1. Introduction

Determining how attention operates at the level of neuronal circuitry remains one of the central unresolved problems of cognitive neuroscience. In order to determine the role of attention in neuronal information processing, the basic spatiotemporal characteristics of attention must first be mapped in detail. Perhaps the most basic task is to characterize how the spatial distribution of attention changes over time. Surprisingly, this has only been done in a rudimentary way by past researchers. Previous psychophysical mappings of the distribution of visual attention during a task have sampled only a few points in the visual field (e.g., Hughes & Zimba, 1985; the sole exception being Bennett & Pratt, 2001). This may in part account for the lack of consensus regarding how attention is spatially distributed and how that distribution changes over time. Because past maps of the spatiotemporal dynamics of attention have been so low in spatial and temporal resolution, numerous models of attention have arisen that are consistent with this poor data, but inconsistent with each other. Thus, researchers debate over whether attention should be modeled as a “spotlight,” “window,” or “zoom lens,” and they debate whether it moves in an analog or quantal fashion (e.g., Eriksen & St. James, 1986; Posner, Snyder, & Davidson, 1980). They debate whether attention can operate on features other than the locations of visual space, such as motion, shape, or color (e.g., Driver & Baylis, 1989). And they debate whether attention adheres primarily to moving objects or their locations (Duncan, 1984). My coworkers and I (Tse, Sheinberg, & Logothetis, 2003) have recently introduced a new method for mapping attention with high spatial and temporal resolution that may help settle some of these old debates. This method exploits the fact that attention has to be allocated to the location of a change in order for that change to be detected in a change blindness paradigm (Rensink, O’Regan, & Clark, 1997). The data described here have been described elsewhere (Tse et al., 2003), but will be summarized again here in order to clarify the method. In this paper I focus on the advantages, limitations, and possible future applications of this new method for mapping visual attention. This new method allows researchers to carry out the detailed mapping of visual attention necessary to distinguish among and generate new models of visual attention.

Change blindness can occur in many ways. Although variants of the phenomenon have been known for many years (e.g., Neisser & Becklen, 1975; Pashler, 1988; Sperling, 1960), there has been a recent revival of research on change blindness. Renewed interest followed the observation that changes in a scene that occur during a saccade are typically not noticed (McConkie & Currie, 1996). Others soon noted that changes that occur during a full-field blank or mask (Rensink et al., 1997) or partial field mask (O’Regan, Rensink, & Clark, 1999) are also difficult to detect. Others have shown that changes need not be masked by a blank or by a saccade at all. Scene changes that occur in plain view, but which are not attended (Becklen & Cervone, 1983; Haines, 1989; Mack & Rock, 1998; Rensink et al., 1997; Simons & Chabris, 1999) or which occur too slowly to be picked up by motion-energy detectors (Auvray & O’Regan, 2003; Simons, Franconeri, & Reimer, 2000) are also not typically noticed. Perhaps the simplest version of this phenomenon has been published as a game in magazines since the advent of photography. Two nearly identical photographs are placed side by side, and the viewer must localize the one thing that is different between them. This is not typically thought of as change blindness, but the difficulty of this task arises for the same reason that change localization is difficult in more modern paradigms. The location of the change is not detected by motion-energy detectors, which operate in parallel, so must be localized by serial comparisons made across successive scenes.

In the method described here there is a brief high-luminance full-field blank between two successive, overlapping images that differ (Rensink et al., 1997). An observer might not notice a prominent change in the image during the global blank even when searching for the change, because the global transient introduced by the blank undermines detection of the local transients used by the visual system to locate sudden changes. In particular, the first-order-motion (i.e., motion-energy) detectors that are usually used by the visual system to detect sudden motions, onsets, or changes in luminance are overwhelmed by the fact that a change has happened simultaneously at multiple locations. If many motion-energy detectors are activated by a global change, an isolated motion-energy detector’s signal cannot “pop out” on a salience map and capture attention. A global flash interferes with change detection because the visual system’s low-level change detectors have been incapacitated. When motion-energy detectors fail to provide a unique signal at a single location, differences between the pre- and postblank images are only detected at a given location when attention has been allocated to that location in both images, because only then can corresponding elements at that location be compared (Rensink, 2000b; Rensink et al., 1997; Simons, 1996; Simons & Levin, 1998). This comparison or monitoring of object states or locations over time is a defining characteristic of attention.

Attention permits serial comparisons because observers are able to select a subset of all the information in iconic memory for further operations, such as reporting (Sperling, 1960) or tracking (Pylyshyn & Storm, 1988). Without this selection of a location for an extended duration, information in the iconic store is simply lost as incoming sensory input continually replaces existing information in the iconic buffer. Although attention is generally thought to bind low-level features into spatially coherent objects (for reviews, see Treisman, 1998, 1999), attention can more accurately be thought to temporally bind successive spatially indexed representations into objects that can be tracked through space–time. Without this spatio-temporal binding, changes could not be noticed because without attention the contents of the visual iconic store would be disjoint (Horowitz & Wolfe, 1998) and could therefore not be compared with later contents of that buffer. Once selected and bound by attention, however, new information can be processed as a change in a monitored location or tracked figure (Kahneman, Treisman, & Gibbs, 1992). While an array of first-order motion (i.e., motion-energy) detectors can rapidly detect the location of a single change because detection occurs in parallel, attention-based change localization is not parallel, and therefore not rapid. Attention cannot be allocated everywhere to the same degree, and comparisons must be made with past stimuli stored in the iconic buffer.

Within this theoretical framework, attention can be defined as enhanced processing over a limited subset of sensory information that has been selected for monitoring or tracking over time. Unattended information is either not monitored or it may be actively suppressed (e.g., Tipper, 2001). Because the global blank used in the standard change blindness paradigm (Rensink et al., 1997) undermines the visual system’s (normally) automatic ability to detect changes on the basis of local transients, it can only detect changes by relying on attentional monitoring. The key assumption behind the present method is the view that the probability of change detection at a given location indicates the degree of attention at that location.

The basic experimental design, shown schematically in Fig. 1, exploits the hampered change detection that occurs in a flicker-induced change-blindness paradigm to create a map of how attention is redistributed in response to a peripherally flashed cue. Tse et al. (2003) used this method to answer the following question: How does the distribution of visual attention change in response to a task-irrelevant cue that flashes suddenly in the periphery?

Open in a separate window

Fig. 1

Experimental design. The trial sequence is illustrated in (b). On cued trials, a spot appeared in Frame 2, as shown, and Frame 3 was presented for one of three durations. On noncued trials, there was no spot in Frame 2, and Frame 3 always lasted 12 ms. In the final frame of the sequence (labeled 5 here) on both cued and noncued trials, a new square was shown in one of 149 positions of the array (a); these positions represented a subset of the overall grid that fit within an imaginary 25° circle centered at fixation. The new square always appeared in a random position that had been unoccupied on previous frames within a given trial. The final frame always contained one and only one new element. The observers’ task was to maintain fixation and report whether the new element was red or green. SOA = stimulus onset asynchrony.

2. Method

The method has been described in greater detail elsewhere (Tse et al., 2003), but is summarized here. Four participants (24–38 years of age) carried out the experiment. Three were paid observers who were naive regarding the purpose of the experiment, and one was the author. The naive observers completed approximately 100 h of training and testing over a 4-month period.

3. Apparatus

The visual stimulator was a dual-processor Pentium II workstation running Windows NT 4.0. The screen resolution was 1152 × 864 pixels, and the frame rate 85 Hz. Observers rested their chin in a chin rest. Fixation was ensured using an eyetracker (Sensomotoric Instruments, Berlin, Germany; Tse, Sheinberg, & Logothetis, 2002a, 2002b). Any time the observer’s monitored eye was outside a fixation window with a 1.5° radius, the trial would be automatically aborted, and a new trial would be chosen at random from those remaining. If three trials were aborted in a row, the state system automatically reverted control to the eyetracker’s calibration program. Once calibration was completed, the experiment resumed with a random trial.

4. Stimuli

The onset of a trial was indicated by the offset and immediate re-onset of the fixation point against the black background. The fixation point was a yellow circle 0.15° in diameter. The circular background, shown in Fig. 1b, was 30° in diameter and uniform black (<1 cd/m²). It spanned the height of the monitor at a viewing distance of 57 cm. The background outside this circular region was dark gray. A 23 × 23 array of square positions fit within a 30° × 30° square that circumscribed this circular “window.” On a given trial, half of these positions were occupied by red and green squares (0.69° × 0.69°); the shades of red and green used were equiluminant as measured by a photometer (Minolta CRT color analyzer CA-100). The probability that a square was red or green was 50%. Squares never overlapped. Their centers were at least 1.25° apart, and the orientation of each square was randomized on each trial. Squares at the edge of the window could be partly occluded.

In no-cue trials, the array was present for 565 ms, after which the screen turned entirely white for 47 ms. The array then reappeared with a new square in one of 149 positions (indicated by yellow circles in Fig. 1a); these positions represented a subset of the overall grid that fit within an imaginary 25° circle centered at fixation. The new square always appeared in a random position that had been unoccupied by a red or green square on previous frames within that trial. In cued trials, after 506 ms of the static array, 12.5 on the plus or minus x- or y-axis. The small outward apparent motion induced by this offset enhanced the salience of this cue. Cuing was followed by a return to the static array for 12, 82, 153, or 447 ms, and then a full-screen white blank that lasted 47 ms. After the blank, the array returned with a new square, as on no-cue trials.

The observers’ task was to maintain fixation and report whether the new element was red or green. Four temporal intervals between the onsets of the cue and new square were tested. Each trial could have either no cue or a cue at one of four positions and one of four temporal intervals, selected at random. This design resulted in a total of 25,330 test trials per observer: [1(no cue)+ 4(cue positions) × 4(temporal intervals)] × 149(test positions) × 10(trials per position). The training phase included half that number of trials. Each of 10 blocks of 2533 test trials was broken down into eight sessions of 316 or 321 trials. The intertrial interval was approximately 3 s in order to minimize possible effects of afterimages. A session typically lasted 35–45 min. Data were stored and later sorted and analyzed off-line.

5. Procedure

Observers were instructed to attend to the entire circular array of red and green squares and report the color of the new square in the final frame of each trial by pressing the appropriate button. There was no feedback on the correctness of responses. The instructions emphasized that the probability that this new square would appear at the cued location was the same as the probability that it would appear at any other tested location, and that therefore there was no advantage to attending to or ignoring the cues or their locations.

6. Practice phase

The blank (white) fourth frame (see Fig. 1) was 12 ms in duration during the 12,665 practice trials for the three naive observers. This was the shortest duration permitted by the refresh rate of the monitor, and was chosen to make learning the task as easy as possible. All observers were initially unable to see the new element if it appeared further than approximately 3° away from fixation. With practice, however, two of the observers learned to see the new square even when it was as far away as 12.5° from. The third observer did not learn to “ignore” the global blank, and continued to perform at chance beyond approximately 3° from fixation. He was disqualified because he could not perform the task over the whole area of the visual field tested, and because it became apparent that he was not paying full attention to the task, at times performing at chance even near fixation. Instead, the author served as the 3rd observer in the test phase of the experiment. He had received several thousand practice trials in the course of coding the experiment, but these practice data were not saved.

A blank of 47 ms was chosen for the test phase of the experiment because trained observers were neither perfect nor at chance at detecting the new square at this duration (observer 1: 72.8% of 25,330 trials correct; observer 2: 67.2% correct; observer 3: 68.4% correct).

7. Results

Fig. 2 shows the pooled data for the three observers from the test phase of the experiment. Change detection was significantly above chance for almost the entire 25° diameter circular area of the visual field tested, presumably because observers were instructed to attend to the whole array of squares. This corroborates the finding that outside the main focus of attention observers maintain a minimum degree of visual awareness with which they can detect and discriminate pop out classes of information at little or no cost to central attentional processing (Braun & Julesz, 1998). The attentional hot spot, corresponding to regions of near-perfect performance (yellow and white regions in Fig. 2), tended to change shape depending on the position of the cue, even though observers knew that the position of the cue was irrelevant to their task.

Open in a separate window

Fig. 2

Average attentional maps for three observers. Pooled percentage correct for no-cue trials and cued trials with each SOA and each cued position (white stars). Each of the 149 tested locations shown in Fig. 1a could have a maximum of 30 correct responses. Bilinear interpolation was used to obtain values for nontested positions, and results were smoothed using a rotationally symmetric 3 × 3 Gaussian low-pass filter with a standard deviation of 0.5 grid units. Values were then mapped to the 25° × 25° color maps shown. The probability that observed values were obtained by chance is indicated by the p values to the left of the bar. These values were obtained using the binomial test, where n = 30 and p = .5. The test region was approximately an imaginary circle that touched the borders of these square maps. Black regions outside these circles were not tested. The fixation point is indicated at the center of each map.

The no-cue case (Fig. 2, upper left) indicates the distribution of attention in the absence of a cue. This horizontal, elliptical hot spot can be thought of as the default shape of attention for this task. After the cue flashed, the hot spot elongated significantly (for statistics, see Tse et al., 2003) along the cue-fixation axis away from its default shape for most conditions. This is consistent with the finding that an abrupt onset causes an automatic allocation of attention to the location of that onset (Jonides & Yantis, 1988; Yantis & Egeth, 1999), particularly when attention has been set (Everling & Munoz, 2000; Folk, Remington, & Johnston, 1992) to detect changes, as here.

When the cue was to the left or right of fixation, the hot spot of attention tended to elongate along the horizontal axis, whereas when the cue was above or below fixation, the hot spot tended to elongate along the vertical axis. Also, for all observers, attention tended to pool not only at the cued location, but also at the corresponding location in the opposite hemifield. The hot spot spread out in the diametrically opposite direction from the flashed location along the cue-fixation axis, facilitating the detection of new elements up to 19° away from the cued location, relative to performance in the no-cue condition. The bilaterally elongated shape of the hot spot was most pronounced at 176 and 247 ms after cue onset, and diminished after that as the effects of cuing diminished.

8. Discussion of results

For most cue locations and stimulus onset asynchronies (SOAs), attention was enhanced (relative to the no-cue case) not only at the cued location, but also opposite the cued location. Opposite pooling of attention was stronger along the horizontal than the vertical axis, but this may be because the hot spot of attention was already biased to extend along the horizontal axis in the no-cue condition. Another asymmetry between the horizontal and vertical axes is that accuracy at the cued location was lower for cues on the vertical than on the horizontal axis. Why this should be is unclear, but it may be that the cue can interfere with the detection of changes at its location more strongly at upper vertical-axis locations, where attention has relatively low resolution (He, Cavanagh, & Intriligator, 1996; Intriligator & Cavanagh, 2001).

The data are consistent with previous results indicating that attention grows in an analog manner away from fixation toward a peripherally cued location (Shulman, Remington, & McLean, 1979; Tsal, 1983). However, the data do not rule out the possibility that there may also be non-analog shifts of attention under some conditions (Remington & Pierce, 1984; Sagi & Julesz, 1985; Sperling & Weichselgartner, 1995). Because the hot spot is aligned along the cue-fixation axis, it is possible that attention deforms in an analog manner along this axis, but need not traverse intermediate points between points that do not lie on this axis; this would be a compromise between temporally analog (Eriksen & St. James, 1986; Posner et al., 1980) and quantal (Remington & Pierce, 1984; Sagi & Julesz, 1985; Sperling & Weichselgartner, 1995) theories of attentional shifts.

It may be that the “unnatural” maintenance of fixation played a role in the tendency for attention to pool opposite the cued location. Future work should reveal whether opposite pooling occurs even when observers are permitted to saccade to the cue. These and other experiments will have to determine why attention pools opposite the cued location, and what role cuing and fixation play in triggering this pooling. Opposite pooling of attention should also be observable using other measures of attentional allocation and performance, such as detection thresholds.

9. General discussion of method

This method exploits the fact that change detection is hampered in a flicker-induced change-blindness paradigm to map how spatial attention changes in response to an irrelevant peripheral cue. The basis of this method is the assumption that the likelihood of detecting a change at a given location corresponds to the occurrence of attention at that location. On theoretical and experimental grounds (Rensink, 2000b; Rensink et al., 1997; Simons, 1996), it can be argued that the distribution of judgment accuracy correlates directly with the distribution of attention. In essence, change-detection accuracy affords an indirect measure of attentional occurrence because change-detection accuracy must increase and decrease with increases and decreases in the perceptual signal-to-noise ratio. If spatial attention is operationally defined as the mechanism that boosts this ratio (e.g., Lee, Itti, & Braun, 1999; Treue & Maunsell, 1999), then the present method measures a correlate of attentional occurrence at each location. Change detection accuracy can thus be added to other indirect measures of attention, such as reaction time (RT) and sensitivity.

References