This is an excerpt from a report on The Unity of Consciousness and Sensory Integration conference at Brown University in November of 2011, written by Kevin Connolly, Craig French, David M. Gray, and Adrienne Prettyman, and available at http://networksensoryresearch.utoronto.ca/Network_for_Sensory_Research.html 2. Are Some of the Basic Units of Consciousness Multimodal? In the McGurk effect, a subject views a video of a person saying one set of syllables (e.g. ga-ga), while the audio has been redubbed to a second set of syllables (e.g., ba-ba). The subject experiences yet a third set of syllables, distinct from the first two sets (e.g., da-da) (McGurk and MacDonald, 1976, p. 747). In his talk on how to model the unity of consciousness, Tim Bayne proposed two different interpretations of crossmodal cases such as the McGurk effect. On a strictly causal interpretation, seeing the person mouth ga-ga causes you to hear da-da instead of ba-ba. According to this interpretation, integration occurs between processing in the two different systems (the auditory system and the visual system), but the result of that processing can be fully decomposed into an audio component and a visual component. So, while the processing is multisensory, the result is not intrinsically multisensory. On a constitutive interpretation, on the other hand, the ga-ga visual input and ba-ba auditory input give you an experience that is constitutively both audio and visual (not just a conjunction of an audio and visual experience). According to this interpretation, the perceptual state that results from the processing cannot be fully decomposed into two unisensory token states, one auditory state and one visual. We can interpret other crossmodal cases constitutively or causally as well. In the motionbounce illusion, subjects look at a computer display of two disks moving steadily towards each other until they meet. If the subject hears a sound at or around the point of convergence, the disks typically appear to collide and bounce off one another. If the subject does not hear a sound, the disks appear to cross through one another (Sekuler et al., 1997). According to a strictly causal 2 interpretation, the motion-bounce illusion is a case where the sound simply causes you to have a certain visual experience (given the right visual input). According to a constitutive interpretation, on the other hand, it is a case where you have a constitutively audio-visual experience. In the conference's first panel discussion, Susanna Siegel argued that whether we take a constitutive or causal interpretation of crossmodal cases will determine whether we hold that some basic units of consciousness are fundamentally multimodal. If we hold a constitutive interpretation of the McGurk effect, for instance, then we hold that at least some of the basic units are audio-visual. A strictly causal interpretation, on the other hand, does not commit us to that. Siegel mentioned (and Fiona Macpherson further bolstered) one piece of evidence in support of holding a causal interpretation of the McGurk effect. When you see someone mouthing the syllables ga-ga, the auditory input is ba-ba, yet you hear da-da. But you could always hear the syllables da-da and you could always see someone mouthing the syllables ga-ga. So it seems like there is not some new type of unit created by the McGurk effect. Rather, it seems like the results of the McGurk effect are decomposable into an auditory unit and a visual unit. After all, you could experience each of those types on their own, outside of the McGurk scenario (for a similar argument, see Connolly, 2011, pp. 126-128). Siegel went on to suggest that whether we hold a causal or constitutive interpretation of cross-modal cases determines how we analyze the unity of consciousness. If we ask what conscious unity unifies, the answer to that question will depend on what the basic units of consciousness are, since they will be the most basic relata of the unity relations. Siegel pointed out that there are two options here: either all of the units of consciousness are unimodal, or else some of the units are not. If the basic units are strictly unimodal, then the building blocks unified 3 in conscious experiences will be unimodal. Most philosophers just take it for granted that the most basic units of consciousness are unimodal. However, if at least some basic units are fundamentally multimodal, then at least some of the building blocks of unified conscious experiences will be multimodal. References Connolly, Kevin (2011). The Role of Concepts in Perception. PhD Dissertation, University of Toronto. McGurk, H. and MacDonald, J. (1976). "Hearing Lips and Seeing Voices." Nature, 264, 746748 . Sekuler, R., Sekuler, A. B., and Lau, R. (1997). "Sound Alters Visual Motion Perception." Nature, 385, 308.