Visual motion disambiguation by a subliminal sound

doi:10.1016/j.concog.2007.09.001

Consciousness and Cognition

Volume 17, Issue 3, September 2008, Pages 790-797

https://doi.org/10.1016/j.concog.2007.09.001 Get rights and content

Abstract

There is growing interest in the effect of sound on visual motion perception. One model involves the illusion created when two identical objects moving towards each other on a two-dimensional visual display can be seen to either bounce off or stream through each other. Previous studies show that the large bias normally seen toward the streaming percept can be modulated by the presentation of an auditory event at the moment of coincidence. However, no reports to date provide sufficient evidence to indicate whether the sound bounce-inducing effect is due to a perceptual binding process or merely to an explicit inference resulting from the transient auditory stimulus resembling a physical collision of two objects. In the present study, we used a novel experimental design in which a subliminal sound was presented either 150 ms before, at, or 150 ms after the moment of coincidence of two disks moving towards each other. The results showed that there was an increased perception of bouncing (rather than streaming) when the subliminal sound was presented at or 150 ms after the moment of coincidence compared to when no sound was presented. These findings provide the first empirical demonstration that activation of the human auditory system without reaching consciousness affects the perception of an ambiguous visual motion display.

Introduction

The ability to respond to external stimuli is enhanced by binding signals from multiple sensory modalities. Studies of perceptual illusions such as the ventriloquist and the McGurk effect in which conflicting multisensory information is erroneously perceived to be bound together suggest that cross-modal binding is a fast and pre-attentive process (Driver, 1996, McGurk and MacDonald, 1976, Sekuler et al., 1997). However, there has recently been much debate regarding cross-modal illusion perception, and the confounding influence of response bias and other decision factors (Bertelson & de Gelder, 2004).

The “bouncing–streaming” illusion involves two objects moving towards one another, reaching the same position, and then moving apart. This motion can be perceived as the objects moving in either a constant trajectory (i.e., streaming through one another) or a reverse trajectory (i.e., bouncing off one another as if following a collision). The possible subjective and interpretative effects of the sound influence in the bouncing–streaming visual illusion (Sekuler et al., 1997) have recently been addressed (Sanabria, Correa, Lupianez, & Spence, 2004). It has been shown (Sekuler et al., 1997) that the perception of bouncing can be increased by a sound at the moment of contact, suggesting that the sensory information perceived in one modality (audition) can modulate the perception of events occurring in another modality (i.e., ambiguous visual motion perception). However, this cross-modal effect may simply reflect a cognitive bias whereby the sound resembles the transient auditory stimulus produced by a physical collision of two objects, causing subjects to infer the reversal of motion direction from the presence of this factor generally associated with bouncing in the physical world.

Cognitive biases linked to subjective reports in the sound bounce-inducing effect have recently been ruled out by an elegant paradigm (Sanabria et al., 2004) in which the point of coincidence of two moving disks was hidden behind an occluder. When emerging from behind the occluder, the disks (one red, the other blue) could either follow the same trajectory (streaming) or else move in the opposite direction (bouncing). Participants made speeded discrimination responses regarding the side from which one of the disks emerged from behind the occluder. Participants responded more rapidly on streaming trials when no sound was presented compared to ‘streaming with sound’ trials, and also responded more rapidly on bouncing trials when sound was presented at the moment of coincidence compared to ‘bouncing without sound’ trials. Although this paradigm provides an implicit/objective behavioral measure of the sound bounce-inducing effect, it does not rule out interpretative response biases whereby subjects explicitly infer the reversal of motion direction from the presence of the sound even when the collision is not visible.

The present study used a novel method to overcome the issue of interpretative bias in the sound bounce-induced effect. The approach involved stimulating the auditory system without the subject being conscious of the stimulation. This was achieved by presenting a subliminal sound either 150 ms before, at, or 150 ms after the moment of coincidence of two disks. An increase in the proportion of “bounce” responses in the presence of a subliminal sound would be inconsistent with a cognitive bias regarding the bounce-inducing effect.

Section snippets

Subjects

The study involved 12 subjects (6 females and 6 males) who were paid volunteers and were unaware of the purpose of the experiment. Importantly, the subjects were not aware of the presence of a subliminal sound during the visual motion experiment. The experiment took approximately 20 min to complete and was performed in accordance with the ethical standards stated in the 1964 Declaration of Helsinki. Informed consent was obtained after the nature and possible consequences of the studies were

Subjects

Twelve subjects (7 females, 5 males) who had not participated in Experiment 1 were enrolled for Experiment 2.

Procedure

The same materials were used as in Experiment 1, except that sounds were played through 2 loud-speakers placed on each side of the computer monitor in order to maximize multisensory integration. Prior to the experiment, subjects were assessed for auditory detection thresholds for a brief sound using the same procedure as described in Experiment 1. The visual motion experiment used three

Experiment 3

Experiments 1 and 2 were designed to ensure that subjects could not be aware of the presence of a sound. While the results of Experiment 2 appeared to confirm this lack of awareness, it might be argued that the sounds were not truly subliminal in all trials even when subjects affirmed not having heard them. Such an objection might find its origin in the still-existing controversy over how to define conscious and unconscious perception and how to rule out alternative weak conscious perception

Discussions

The present study shows for the first time that visual perception can be modified by the activation of the auditory system without a conscious perception of the auditory stimulus. The effect of the subliminal auditory stimulus on the bouncing–streaming illusion perception observed in this study was comparable to that described in previous studies using supraliminal sounds (Bushara et al., 2003, Sanabria et al., 2004, Sekuler et al., 1997). However, unlike a previous study (Sekuler et al., 1997

References (17)

P.M. Merikle et al.
Parallels between perception without attention and perception without awareness
Consciousness and Cognition
(1997)
P. Bertelson et al.
The psychology of multimodal perception
K.O. Bushara et al.
Neural correlates of cross-modal binding
Nature Neuroscience
(2003)
J. Cheesman et al.
Priming with and without awareness
Perception & Psychophysics
(1984)
J. Cheesman et al.
Distinguishing conscious from unconscious perceptual processes
Canadian Journal of Psychology
(1986)
S. Dehaene et al.
Imaging unconscious semantic priming
Nature
(1998)
S.C. Draine et al.
Replicable unconscious semantic priming
Journal of Experimental Psychology: General
(1998)
J. Driver
Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading
Nature
(1996)

There are more references available in the full text version of this article.

Cited by (26)

Multiple phases of cross-sensory interactions associated with the audiovisual bounce-inducing effect
2020, Biological Psychology
Citation Excerpt :
As this effect of audition on visual motion perception was often referred to as “audiovisual bounce-inducing effect (ABE)” in recent studies (e.g., Grassi & Casco, 2009, 2010, 2012; Maniglia, Grassi, Casco, & Campana, 2012; Zhao, Wang, Xu, Feng, & Feng, 2018), the current study termed this phenomenon “ABE” for convenience. The ABE phenomenon has been consistently observed in numerous behavioral studies (Watanabe & Shimojo, 2001a, 2001b; Shimojo & Shams, 2001; Remijn, Ito, & Nakajima, 2004; Kawabe & Miura, 2006; Dufour, Touzalin, Moessinger, Brochard, & Després, 2008; Grassi & Casco, 2009, 2010, 2012; Grove & Sakurai, 2009; Grove, Ashton, Kawachi, & Sakurai, 2012; Grove, Robertson, & Harris, 2016; Roudaia, Sekuler, Bennett, & Sekuler, 2013; Zeljko & Grove, 2016; Parise & Ernst, 2017) and has been widely utilized as a case of cross-modal interaction for investigating many other scientific issues, such as recalibration of audiovisual simultaneity (Fujisaki, Shimojo, Kashino, & Nishida, 2004), oscillatory synchronization in cortical networks (Hipp, Engel, & Siegel, 2011), effect of mental imagery on multisensory perception (Berger & Ehrsson, 2013, 2017), and attentional modulation on temporal binding of audiovisual stimuli (Donohue, Green, & Woldorff, 2015). However, the neural mechanisms responsible for the ABE are only beginning to be understood.
Using event-related potential (ERP) recordings, the present study investigated the cross-modal neural activities underlying the audiovisual bounce-inducing effect (ABE) via a novel experimental design wherein the audiovisual bouncing trials were induced solely by the ABE. The within-subject (percept-based) analysis showed that early cross-modal interactions within 100–200 ms after sound onset over fronto-central and occipital regions were associated with the occurrence of the ABE, but the cross-modal interaction at a later latency (ND250, 220–280 ms) over fronto-central region did not differ between ABE trials and non-ABE trials. The between-subject analysis indicated that the cross-modal interaction revealed by ND250 was larger for subjects who perceived the ABE more frequently. These findings suggest that the ABE is generated as a consequence of the rapid interplay between the variations of early cross-modal interactions and the general multisensory binding predisposition at an individual level.
Early cross-modal interactions underlie the audiovisual bounce-inducing effect
2018, NeuroImage
Citation Excerpt :
This hypothesis suggested that the presentation of the sound at the moment of two visual disks' coincidence makes the observers really see the two disks bounce off each other, instead of explicit inference resulting from the transient sound resembling a physical collision of two objects. For instance, Dufour et al. (2008) found the ABE could also be induced even by a subliminal sound, which obviously could not be used to infer explicitly because subjects could not be aware of it, thereby providing strong evidence for the perceptual hypothesis of the generation of ABE. However, another hypothesis emerged in recent years proposed that the ABE derives merely from shifted response/cognitive bias by the sound at late decision-making stage.
Two identical visual disks moving towards one another on a two-dimensional display can be perceived as either “streaming through” or “bouncing off” each other after their coincidence/overlapping. A brief sound presented at the moment of the coincidence of the disks could strikingly bias the percept towards bouncing, which was termed the audiovisual bounce-inducing effect (ABE). Although the ABE has been studied intensively since its discovery, the debate about its origin is still unresolved so far. The present study used event-related potential (ERP) recordings to investigate whether or not early neural activities associated with cross-modal interactions play a role on the ABE. The results showed that the fronto-central P2 component ∼200 ms before the coincidence of the disks was predictive of subsequent streaming or bouncing percept in the unimodal visual display but not in auditory-visual display. More importantly, the cross-modal interactions revealed by the fronto-central positivity PD170 (125–175 ms after sound onset), as well as the occipital positivity PD190 (180–200 ms), were substantially enhanced on bouncing trials compared to streaming trials in the auditory-visual display. These findings provide direct electrophysiological evidence that early cross-modal interactions contribute to the origin of ABE phenomenon at the perceptual stage of processing.
Prevailing theories of consciousness are challenged by novel cross-modal associations acquired between subliminal stimuli
2018, Cognition
While theories of consciousness differ substantially, the ‘conscious access hypothesis’, which aligns consciousness with the global accessibility of information across cortical regions, is present in many of the prevailing frameworks. This account holds that consciousness is necessary to integrate information arising from independent functions such as the specialist processing required by different senses. We directly tested this account by evaluating the potential for associative learning between novel pairs of subliminal stimuli presented in different sensory modalities. First, pairs of subliminal stimuli were presented and then their association assessed by examining the ability of the first stimulus to prime classification of the second. In Experiments 1–4 the stimuli were word-pairs consisting of a male name preceding either a creative or uncreative profession. Participants were subliminally exposed to two name-profession pairs where one name was paired with a creative profession and the other an uncreative profession. A supraliminal task followed requiring the timed classification of one of those two professions. The target profession was preceded by either the name with which it had been subliminally paired (concordant) or the alternate name (discordant). Experiment 1 presented stimuli auditorily, Experiment 2 visually, and Experiment 3 presented names auditorily and professions visually. All three experiments revealed the same inverse priming effect with concordant test pairs associated with significantly slower classification judgements. Experiment 4 sought to establish if learning would be more efficient with supraliminal stimuli and found evidence that a different strategy is adopted when stimuli are consciously perceived. Finally, Experiment 5 replicated the unconscious cross-modal association achieved in Experiment 3 utilising non-linguistic stimuli. The results demonstrate the acquisition of novel cross-modal associations between stimuli which are not consciously perceived and thus challenge the global access hypothesis and those theories embracing it.
Auditory-induced bouncing is a perceptual (rather than a cognitive) phenomenon: Evidence from illusory crescents
2018, Cognition
Citation Excerpt :
Moreover, removing the ‘coincidence’ eliminates the effect: if the sound occurs as part of a temporal sequence of identical tones that extends both before and after the moment of overlap, then the temporally-aligned tone (now one among many identical tones) no longer leads to perceived bouncing, as it would in isolation (or with a tone that deviates from the other tones in the temporal sequence; Watanabe & Shimojo, 2001; see also Kawachi & Gyoba, 2006). Although we have been describing the audiovisual influence on bouncing/streaming as an effect on what we see (cf. Dufour, Touzalin, Moessinger, Brochard, & Després, 2008), others have suggested that it is an effect on (merely) our higher-level decisions about what must have happened in the display. In particular, a recent study employed signal detection theory to distinguish changes in sensitivity from changes in response biases in the context of bouncing vs. streaming—where sensitivity is associated with perceptual processes, and biases are associated with “decisional processes” (which are to be contrasted with perception.)
A central task for vision is to identify objects as the same persisting individuals over time and motion. The need for such processing is made especially clear in ambiguous situations such as the bouncing/streaming display: two discs move toward each other, superimpose, and then continue along their trajectories. Did the discs stream past each other, or bounce off each other? When people are likely to perceive streaming, playing a brief tone at the moment of overlap can readily cause them to see bouncing instead. Recent research has attributed this effect to decisional (rather than perceptual) processes by showing that auditory tones alter response biases but not the underlying sensitivity for detecting objective bounces. Here we explore the nature of this phenomenon using ‘illusory causal crescents’: when people perceive bouncing (or causal ‘launching’), they also perceive the second disc to begin moving before being fully overlapped with the first disc (i.e. leaving an uncovered crescent). Here we demonstrate that merely playing a sound coincident to the moment of overlap can also reliably induce the perception of such illusory crescents. Moreover, this effect is due to the coincidence of the tone, per se, since the effect disappears when the tone is embedded in a larger regular tone sequence. Because observers never have to explicitly categorize their percept (e.g. as streaming)—and because the effect involves a subtle quantitative influence on another clearly visual property (i.e. the crescent’s width)—we conclude that this audiovisual influence on the perception of identity over time reflects perceptual processing rather than higher-level decisions.
The origin of the audiovisual bounce inducing effect: A TMS study
2012, Neuropsychologia
The audiovisual bounce inducing effect (ABE) is a bouncing percept induced by the presence of a sound at the moment of two moving objects intercepting in a motion display otherwise perceived as streaming. The origin of the ABE is still debated: the effect could arise from the subtraction of attentional resources caused by the sound (needed to favor the perception of streaming), and/or from the cross-modal integration (binding) of visual and auditory information: indeed bouncing-like sounds are best in inducing the ABE. The neural mechanism responsible for the ABE is still unknown. Here, by using offline TMS, we investigated the role of the posterior parietal cortex (PPC), thought to be involved in both attentional and binding processes, in the generation of the ABE.
Results show that disrupting the functional integrity of the right (but not the left) PPC has the effect of weakening the binding of cross-modal information, which reduces the magnitude of the ABE. Indeed, if the effect of parietal stimulation was merely to disrupt attention, we would expect an increase (not a decrease) of bouncing percepts.
The present study not only shows the involvement of the right PPC in the ABE, but also support the notion that cross-modal binding (and not attention) is at the origin of the ABE.
Crossmodal influences on visual perception
2010, Physics of Life Reviews
Citation Excerpt :
The underlying mechanism for this change in percept is not clear, and it is possible that it is mediated by cognitive processes (i.e., by higher level knowledge that when objects collide they make a sound), rather than interactions at a perceptual level. The findings of a recent study showing that even a subliminal sound can induce the bias, however, suggest that the illusion does reflect interactions at a perceptual level of processing [21]. Other studies have shown that similar change from streaming to bouncing motion can be induced by other types of transient stimuli, including brief visual events at the time of the coincidence of the two moving objects [108,109].
Vision is generally considered the dominant sensory modality; self-contained and independent of other senses. In this article, we will present recent results that contradict this view, and show that visual perception can be strongly altered by sound and touch, and such alterations can occur even at early stages of processing, as early as primary visual cortex. We will first review the behavioral evidence demonstrating modulation of visual perception by other modalities. As extreme examples of such modulations, we will describe two visual illusions induced by sound, and a visual illusion induced by touch. Next, we will discuss studies demonstrating modulation of activity in visual areas by stimulation of other modalities, and discuss possible pathways that could underpin such interactions. This will be followed by a discussion of how crossmodal interactions can affect visual learning and adaptation. We will review several studies showing crossmodal effects on visual learning. We will conclude with a discussion of computational principles governing these crossmodal interactions, and review several recent studies that demonstrate that these interactions are statistically optimal.

View all citing articles on Scopus

View full text

Visual motion disambiguation by a subliminal sound

Abstract

Introduction

Section snippets

Subjects

Subjects

Procedure

Experiment 3

Discussions

Consciousness and Cognition

The psychology of multimodal perception

Neural correlates of cross-modal binding

Nature Neuroscience

Priming with and without awareness

Perception & Psychophysics

Distinguishing conscious from unconscious perceptual processes

Canadian Journal of Psychology

Imaging unconscious semantic priming

Nature

Replicable unconscious semantic priming

Journal of Experimental Psychology: General

Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading

Nature