This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html 5. What Is the Purpose of Multisensory Integration? The two core sets of questions about multisensory integration that received the most focus at the workshop revolved around 1) how the perceptual processing involved in multisensory integration works, and 2) the content and character of multisensory experiences. However, a third important type of question focuses instead on the role that multisensory integration plays for an individual. What use does it have, and why might our perceptual systems have evolved so as to integrate information coming from multiple modalities? One way of approaching this question is to consider what the advantages of multisensory perception might be over unisensory perception. We certainly benefit from having more than one sensory modality, because it allows us to access more information about our environment-both quantitatively more overall, and more types of information- which can be useful for both navigation and survival. For example, if a predator has the ability to track its prey by sight and by scent, it will be more likely to be successful at hunting even if its sight is obscured by a forest, or if it is dark out. However, this sort of case does not yet illuminate the advantage that multisensory integration per se gives us over and above the advantage of access to input from multiple modalities, each of which may be processed and experienced in isolation from the others. In her talk, Jennifer Campos highlighted some such unique advantages in the domain of selfmotion perception, such as increased accuracy due to flexibility of input weightings for different contexts. In self-motion perception, inputs to the vestibular, proprioceptive, visual, and auditory systems are integrated to represent our own locations and trajectories in space. Ophelia Deroy noted in her commentary on Campos that the input sources that are combined 2 in self-motion perception are particularly interesting, because they involve both interoceptive (vestibular and proprioceptive) and exteroceptive (vision and audition) senses. Campos presented research on integration of such inputs for balance regulation and for estimating distance travelled. When there are multiple inputs available about an individual property, the brain must combine this information using a weighting algorithm, which dictates how much to rely on each particular source. Campos has conducted studies investigating how such weightings are influenced by variations in the input sources and their contents, and has found that subjects generally weight toward the sensory source that provides the most stable and reliable information. This means that often, we are actually responding more to proprioceptive cues than to visual ones, despite the fact that reflection on our own experience might lead us to believe that we are predominantly visual creatures. Deroy made the point that Campos's results indicate that even when vision may be necessary or sufficient for a given task, it may still not be the dominant modality in play, in terms of the weightings given to sensory cues. The dependence of the relative contribution of any given input source on its reliability indicates that our perceptual systems have evolved to make use of the information that is most likely to accurately represent the world. A core function of multisensory integration is to combine information sources to facilitate the production of such accurate representations across varying contexts. In a similar vein, Connolly noted that multisensory integration might also facilitate the production of accurate representations by increasing the efficiency of perception. If the perceptual system can generate representations with contents that incorporate properties derived from multiple sensory modalities fused together, this may eliminate the need for certain sequences of reasoning. For example, if upon hearing a clang and seeing a cymbal being hit, our perceptual system can on its own generate a single representation with multisensory contents representing the sound as emanating from the cymbal, this will 3 eliminate the need to consider the auditory and visual percepts independently, and judge that the properties represented in both attach to the same object. In general, perceptual processing proceeds more rapidly than deliberate inference, so multisensory integration may save us time in coping with the environment. Both of these accounts of the function of multisensory integration appeal to the idea that it allows us to better complete certain crucial tasks (for example, calculating distance estimates, and thereby generating the appropriate motor responses for a situation), due to the incorporation of multiple information sources that increase accuracy and efficiency. It does not, however, posit a distinct type of function that multisensory perception serves. On this view, the integration of multisensory inputs merely increases the likelihood that the function (or at least one of the major functions) of unisensory perception (accurately representing the world) will be fulfilled. A more radical answer to the question of the purpose of multisensory integration might say that it gives rise to truly novel sorts of information, which we could not even in principle access through unisensory processing, and that this novel information plays a crucial functional role. Casey O'Callaghan mentioned some multisensory experiences of this type in his talk, which he labeled as belonging to the "5th grade of multisensory awareness." Flavor might be an example of one such novel feature type that is constitutively dependent on inputs to olfaction, gustation, and tactition, and on their combination in a particular way. There might be certain cases in which detection of flavor properties (as opposed to detection of smell, taste, and/or touch properties) is distinctively useful in terms of making environmental discriminations that guide behavior, such as determining which foods are beneficial to an animal and would be worthwhile to pursue. It is also plausible that the particular hedonic response that such flavor experiences lead to are not achievable merely through the experience of their constituents of taste, touch, and smell, and these responses might also be useful for overall well-being. While the extent of the usefulness of flavor is an 4 empirical question in evolutionary perceptual psychology, it seems quite plausible that there are at least some cases in which perceptual awareness of novel feature types serves a particular function, related to the evolutionary success of an animal. Another way of approaching the question of the purpose of multisensory integration is to ask what it would we be like for us if we were not able to integrate multiple sources of information. We can glean some insight into this issue by looking at cases of selective impairments. Campos discussed studies that she had conducted on the interaction between vision, audition, and balance in subjects with cochlear implants. She found that deaf children who use cochlear implants had difficulty maintaining their balance when standing on one foot, but that with their implants in, their balance improved. This indicates that auditory cues, in addition to proprioceptive, vestibular, and visual cues, are used in balance, and so the ability to rely on and combine multiple types of perceptual cues is instrumental for navigating the environment. Campos's research in the area of the purpose of multisensory integration also has clear practical applications-for example, when strategizing ways to help older people who have trouble with balance, we should take into consideration potential deficits in all the senses involved in the integration processes, as well as errors in the way they are combined, as opposed to focusing exclusively on impairments to a single sensory modality.