This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html 1. What Is Multisensory Integration? Imagine that you are sucking on a menthol sweet or candy.1 A menthol candy has a bitter taste, a minty aroma, and a cool sensation. Take any one of those three away, and it is not the flavor of menthol, since flavor experience requires taste, touch, and smell (see Smith, 2012). As menthol and other flavors show, experiences can be multisensory: single, unified perceptual experiences, which are the result of multisensory integration-an integration of taste, smell, tactile, and other sensations. Other examples of multisensory experiences include one raised by Mohan Matthen: experiences sometimes involve conflict between sense modalities. Some such experiences, as when someone gets spun around, and then asked to walk straight while her vision is moving, may even produce a feeling of sickness. In such cases, there is a sensation that indicates to you something is wrong, and that sensation is not just visual or just proprioceptive. Such an experience is a multisensory experience. Multisensory experiences are the product of a multisensory integration process. What is multisensory integration? According to Matthew Fulkerson, we should not think of it as a natural kind. That is to say, while multisensory integration exists in many different instances, we should not expect to find necessary and sufficient conditions for it. As theorists, we are interested in different groupings of the senses at different times-for eating a meal, we will be interested in a different group than in playing a basketball game. Given this, Fulkerson suggested that we should embrace sensory pluralism: the idea that there are lots of distinct, equally valid ways of dividing up the senses. On this view, there is no natural way of thinking of the senses as doing just one task. Instead, we ought to seek a better understanding of multisensory integration by outlining different ways of categorizing the senses. 1 Barry Smith outlined this example at the workshop. 2 Even if we cannot offer necessary and sufficient conditions for multisensory integration, there are still ways in which we can classify these interactions. In his talk, Casey O'Callaghan classified six different types of multisensory awareness. The first grade is minimally multisensory awareness, whereby at a given time, a subject has co-conscious perceptual awareness associated with more than one sensory modality. For instance, a subject might be aware of the fan whirring, while at the same time be aware of the light flickering. This is co-conscious awareness associated with audition and vision. Grade two is coordinated multisensory awareness, which is a type of multisensory awareness where stimulation in one modality influences experience in another. For example, in the common case of ventriloquism, seeing the movement of the ventriloquist dummy's mouth changes your experience of the auditory location of the vocals. Vision influences your experience of auditory location. The third grade is Intermodal feature binding awareness, which occurs when you consciously perceive multiple features from more than one sense modality jointly to belong to the same object or event. For example, if you are listening to live jazz and the drummer begins a solo, you might see the cymbal jolt and hear the clang, and be aware that the jolt and the clang are part of the same event. The fourth grade is awareness of novel feature instances, whereby one perceives feature instances that are accessible only multimodally. O'Callaghan used the example of baseball umpires, who determine whether the runner is out by watching the runner's foot strike the base while listening for the sound of the baseball hitting the fielder's glove. This is an example where they multisensorily perceive a temporal relation, order, or interval, which would be inaccessible unimodally. Grade five is multisensory awareness of novel feature types. For example, flavor is an emergent feature of a type that can't be experienced unimodally (as in the menthol example). The sixth and final grade of multisensory awareness is novel awareness in a sense modality. Experiences might be associated with only one modality, but at the same time, not be possible without an 3 experience in another modality. An example of this is cross-modal completion, a version of amodal completion that is multimodal. For example, you might hear an event that has visible features that you don't see. If these features affect your experience of its audible aspects, this would be a case of cross-modal completion. There is a question as to how O'Callaghan's account relates to Fulkerson's. Fulkerson's focus is on perceptual processing, while O'Callaghan's focus is on perceptual awareness (see question three of this report for a more detailed discussion of this issue). On its face, however, O'Callaghan's account of multisensory integration allows us accept Fulkerson's point that multisensory integration is not a natural kind, while still allowing us to have a substantial account of multisensory integration. By providing an account of different grades of multisensory awareness, we can have an informative account of multisensory integration in lieu of providing the necessary and sufficient conditions for multisensory integration. References: Smith, Barry C. (2013). "Taste, Philosophical Perspectives." In Pashler, Harold E. (Ed.) Encyclopedia of the mind. Thousand Oaks, Calif: SAGE Publications, Inc.