1 Introduction Suppose you hear your colleague Magdalena speak with someone in the hallway while you are reading a paper on your computer in your office. In the envisaged scenario, you have an auditory experience of the sounds coming from Magdalena's mouth and a visual experience of the graphemes on your computer screen. These two experiences are constituent parts of the total sensory experience you currently have. They are not integrated in any substantial sense. They merely co-exist as constituents of your total experience. That is, the two states are not integrated in a way more substantial than the way any two co-conscious states are integrated into a total experience at a time. Suppose instead that you are having a conversation with Magdalena. In this case you see her lips move and you hear the sounds that come from them. We can literally say that you see Magdalena talk. The integration of your two experiences in this second scenario is different from the mere co-presence of your two experiences in the first. In the first scenario your experiences co-exist as part of your total experience. In the second scenario, your experiences are bound together. How do we account for the difference between the two cases?1 Casey O'Callaghan (2008, 2012, 2014, 2015; see also Dainton 2000; Nudds 2001; Bayne 2014; Deroy 2014; de Vignemont 2014a; Briscoe 2016 in press; Bourget 2017) argues that the difference between the first and the second scenario is that in the first scenario the phenomenology of the result of binding your separate experiences can be fully accounted for by appeal to the phenomenology of the individual sensory modalities but that this is not so in the second case. In the second case, he argues, the overall phenomenology reflects that the two experiences are bound together amodally in perceptual faculties that are neither auditory nor visual in nature-for instance, in higher non-sensory regions of the brain, such as the parietal cortex.2 In this chapter, we provide an argument for thinking that we can account for the difference in phenomenology between the two cases by appeal to the phenomenology of the individual sensory modalities. We argue that the phenomenology of one type of multisensory experience that goes beyond mere co-consciousness derives exclusively from the individual sensory modalities (for some empirical considerations in favor of a third type of multisensory experience, largely presented by speech perception, see Tuomainen et al. 2005). Call this type of experience "modal multisensory experience."3 We then argue that another kind of normal 24 MULTISENSORY CONSCIOUSNESS AND SYNESTHESIA Berit Brogaard and Elijah Chudnoff Berit Brogaard and Elijah Chudnoff Multisensory Consciousness and Synesthesia RHOC.indb 322 21-02-2018 20:05:32 Multisensory Consciousness and Synesthesia 323 Multisensory Consciousness and Synesthesia multisensory experience that goes beyond mere co-consciousness requires a different treatment. The second type of experience is one where the phenomenology is distinctively multisensory and perceptual, yet amodally integrated. Call this type of experience "amodal sensory experience." This appears to be the kind of multisensory experience O'Callaghan (2012) has in mind. When you perceptually attribute both of the features, having a coffee look and having a coffee smell, to the dark liquid in your mug, the phenomenology of your experience reflects this type of integration. In the final section of the chapter, we look at the case of synesthesia-a type of atypical integration in which different sensory streams are bound together in unusual ways, for instance, sounds may be bound together with color. We argue that some forms of synesthesia may be helpful in investigating the neural mechanism underlying amodal multisensory binding. The plan is as follows: In Section 2, we provide an account that draws a distinction among mere co-consciousness and modal and amodal multisensory experience. On this account, modal multisensory experience has a phenomenology that derives from the individual senses and hence lacks the amodal component of multisensory experiences such as that of seeing and holding a tomato. In Section 3, we give reasons for thinking that some cases of integration should be conceived of as instances of modal rather than amodal integration. In Section 4, we compare certain types of synesthesia to amodal multisensory perception and argue that these types of synesthesia may shed light on amodal integration. 2 Modal versus Amodal Integration The case in which you see Magdalena speak clearly differs from a case in which you have a unified experience visually representing graphemes on your computer screen and auditorily representing Magdalena's voice in the hallway.4 In the first case you seem to see the event that produces the sound, viz. Magdalena's moving lips. Experiences of this kind are quite different from experiences that attribute features perceived in different sensory modalities to an object, as in the case of visually attributing being coffee to the dark liquid in the mug and olfactorily attributing coffee smell to that same liquid; or perceiving the flavor of the Oxtail flatbread by gustatorily, olfactorily, somato-sensorily, thermally and nociceptually attributing features to the flatbread. Although you attribute the smell of coffee to the coffee in front of you, it's not as if you see the coffee smell in any way analogous to the way you hear someone speak. The case of seeing someone speak should thus be set apart from a case of multisensory experience that is merely about or directed at the same perceptible object or feature, such as a case in which you both hold and see a firm ripe tomato, or see and smell coffee.5 When you hold and see a tomato, a shape, viz. the common sensible roundness, is both seen and felt. When you see and smell coffee, the attributes being coffee and having coffee smell are both perceptually attributed to the coffee, one by the visual modality and the other by the olfactory modality. In both cases, you are perceptually attributing features to one and the same object but the integration of these attributions into a complete experience cannot be accounted for by appeal to individual sensory modalities. Rather: the integration appears to be amodal: it occurs independently of the mechanisms of the individual sensory modalities.6 This type of multisensory experience thus seems to have the characteristic that O'Callaghan (2012, 2014, 2015) thinks multisensory experience has. He calls this type of binding "amodal integration." Integration is also closely related to what Tim Bayne and David Chalmers (2003) call "objectual unity." We will look at the details of the arguments for thinking that the two types come apart below (e.g., seeing someone speak versus seeing and smelling coffee). Suffice it to say at this point that RHOC.indb 323 21-02-2018 20:05:32 Berit Brogaard and Elijah Chudnoff 324 if the cases come apart in that the phenomenology of the former (e.g., seeing someone speak) derives fully from the individual senses whereas the phenomenology of the latter (e.g., holding a tomato you also see) does not, then O'Callaghan's amodal view cannot be construed as a general view of multisensory perception. As noted in the previous section, although O'Callaghan does not argue that amodal unification is the only sort of integration that goes beyond mere co-consciousness, he does not distinguish between modal and amodal unification. The conditional claim made in the previous paragraph raises an interesting question. If the phenomenology in the first type of case (e.g., seeing someone speak) derives from the individual senses (viz., from vision and audition), how do we distinguish this type of case from the second type (e.g., seeing and feeling the roundness of the tomato)? The solution to this problem, we will now argue, is to reconceive of what is actually perceived by the individual senses in the first type of case when, say, we hear a source produce a sound. Our suggestion is that when we perceive, say, sound being produced by a source, the auditory experience attributes audible qualities to an object picked out by a perceptual demonstrative whose reference is anchored to an object, by virtue of that object being visible. For example, the auditory experience attributes sounding like such and such to a lip-moving event picked out by a perceptual demonstrative that refers to the event, by virtue of its presence in vision. A visual demonstrative is the perceptual equivalent of demonstrative terms that occur in ordinary language, such as "this" and "that." Demonstratives are referential terms that have a referent only when accompanied by a demonstration that successfully picks out an entity or a previously mentioned referent. A demonstration is, for example, a gesture, a glance or a nod in a particular direction or a speaker intention comprehensible by the hearer in the conversational context. When a demonstrative refers back to a previously mentioned referent, as in "John continually scratched his skull. This annoyed Anna," this is also known as "anaphora." In the example we just provided, the anaphoric pronoun "this" refers back to the event John's scratching of his skull. In anaphora, the referents of anaphoric pronouns (the anaphor) depend on the referents of the bit of language they are anaphoric on, i.e. the antecedent (or the postcedent in the case of anaphora, such as "It was her own fault that Jamie didn't get to go to the prom."). As we will see, some perceptual demonstratives function in a way analogous to anaphoric pronouns. Perceptual references to objects in different sensory modalities can thus be interdependent in the way that certain linguistic references to objects in different parts of speech are interdependent. In the case of seeing someone speak, the visual experience provides a visual demonstrative that picks out a speaking or lip-moving event, and the auditory experience attributes audible qualities to it by using the visual demonstrative. By using a visual demonstrative an auditory experience can become dependent on and not just co-conscious with the visual experience.7 It may be thought that seeing sound-events is the only example of multisensory experience in which perceptual unification takes place as a result of demonstrative reference being made by one sense and anchored by another. This, however, does not seem to be the case. Suppose you are lifting weights, holding one weight in your right hand. As you bend your arm, the tactile feel of the weight, together with the feeling of how heavy the weight is, attributes qualities to a demonstrative provided by the visual experience of the lifting event. That is, the feeling of exercising effort in lifting a weight consists in tactually and proprioceptively attributing qualities to a seen event, namely the lifting. Tactile experience itself may very well be multisensory in this sense (Brogaard 2012; de Vignemont and Massin 2015; Briscoe 2016; see Fulkerson 2014 for challenges to the RHOC.indb 324 21-02-2018 20:05:32 Multisensory Consciousness and Synesthesia 325 mainstream view that haptic touch is multisensory). Tactile experiences can reasonably be thought to involve not just representations of properties of objects but also properties of the body (Brogaard 2012; Briscoe 2016).8 Plausibly, you cannot have a tactile experience as of an object being hard without experiencing pressure to the part of your body that does the haptic touching. If you feel a rock press against the palm of your hand, we can take you to have an experience of the palm responding to the hardness of the rock, or alternatively we can take you to have an experience of the hardness of the rock producing a particular sensation in your hand. One aspect of touch thus anchors a tactile demonstrative reference to the rock. Another aspect of touch attributes causing certain bodily sensations in me. And that is what constitutes felt pressure. So, if as some research literature on touch suggests (for discussion see Loomis and Lederman 1986; Jones and Lederman 2006; Fulkerson 2011; Gallace and Spence 2014; Linden 2015), the two aspects of touch involve two different sensory modalities, then this is a case of modal multisensory integration. Further: on the assumption that emotions are multisensory experiences, it can be argued that they are also integrated by means of perceptual reference. Suppose you fear a particular tiger that bares her sharp teeth at you. Your bodily sensations (e.g., sensations of a quickened heartbeat, sweaty palms and shaky legs) are a response to the tiger's fearfulness (Brogaard 2012; Brogaard and Chudnoff 2016). Your being afraid of a seen tiger consists of attributing the property of causing bodily sensations indicating a threat to your well-being to the seen tiger. Vision allows you to refer to the tiger, and the sensations allow you to attribute properties such as causing sensations indicating threat to your well-being. The overall fearful response just is the act of attributing the properties introduced by the bodily sensation to the object introduced by vision. This type of integration can be cashed out as follows: your visual experience identifies a visual event, viz. the tiger baring her teeth, and the bodily sensation attributes certain qualities, such as causing various events felt in your body, to the visually identified event. For the case of visual-auditory binding, we can capture the distinctions among coconsciousness , modal integration, and amodal integration as follows: Co-Consciousness Your overall experience has the content: thatv is F and thath is G [where thatv is a visual demonstrative, F is a visible quality, that h is an auditory demonstrative reference, and G is an audible quality]. Modal Integration Your overall experience has the content: that v is F and that v is G [where that v is a visual demonstrative, F is a visible quality, G is an audible quality]. Amodal Integration Your overall experience has the content: that v is F and that h is G and that v = that h [where that v is a visual demonstrative, F is a visible quality, that h is an auditory demonstrative, G is an audible quality, and that v = that h is an amodally represented identification]. One might wonder how an experience can attribute an audible quality to the referent of a visual demonstrative, thinking, perhaps, that an experience can only attribute audible qualities to referents that are picked out in an auditory manner, say, by an auditory demonstrative or a description built out of audible qualities. However, as noted above in the discussion of anaphora, here we are simply extending a familiar form of representational dependence to representations in different sensory modalities. The familiar form of representational RHOC.indb 325 21-02-2018 20:05:32 Berit Brogaard and Elijah Chudnoff 326 dependence in which one act of reference depends on another act of reference. As noted above, this is a common phenomenon in linguistic representation, for instance, in anaphora and in communication across people. For example, you refer to something because your friend referred to it in her speech. The phenomenon is also common in mental representation. Say you see a chess piece in a certain position on a chessboard. Then you close your eyes and think about or imagine moving it to another position on the board. Your cognitive or imaginative reference to that particular piece depends on your visual reference to it. It is because you saw that piece that your thought or imaginings are about it and not something else. Before proceeding to our argument for the distinction between modal and amodal multisensory experience, let us consider some potential challenges to this account. One might argue that vision and audition have different manners of representation (Chalmers 2004): vision represents visually, whereas audition represents auditorily. But multisensory experience does not represent visually or auditorily. It represents amodally (Bourget 2017). So, the phenomenology of modal multisensory experience is not wholly derived from the phenomenology of the individual sensory modalities. Or so the argument goes. This argument can be resisted, however. Rather than saying that manners of representation change from visual to amodal when the sound is added, it is perfectly plausible to take manners of representation to be additive. When you hear someone speak, your experience represents in a visuo-auditory manner. A further challenge to the proposed account is that of explaining where in the brain binding takes place if indeed its phenomenology is fully derived from the phenomenology associated with the individual sensory modalities. This challenge can be met. We know from the McGurk effect that seeing lip movements can influence and alter what we hear. The McGurk effect arises when auditory speech cues are presented in synchrony with incongruent visual speech cues (McGurk and MacDonald 1976). For example, when the auditory syllable "ba" is presented in synchrony with a speaker mouthing "ga," subjects typically report hearing "da." We also know from the double-flash illusion that auditory input sometimes influences what we see. The double-flash illusion occurs when the presentation of two brief auditory beeps makes a single flash look like two flashes (Shams et al. 2000). So, just considering visuoauditory cases for now, the answer to the question of where in the brain this type of binding takes place is likely that it sometimes occurs in the auditory cortex and sometimes in the visual cortex. Whether the integration occurs in visual or auditory areas is likely to depend on what is taken to produce what. When seen lip movements are taken to produce sound in the McGurk illusion, it is likely that the binding takes place in the auditory cortex. When the beeps are taken to produce the flashes in the double-flash illusion, the binding likely takes place in the visual cortex. A third worry one might have about our proposed account is that it implies that multisensory integration is perceptual. But, it may be argued that multisensory integration is associative or inferential rather than perceptual. This has indeed been the traditional view of multisensory perception (see e.g. Bloom and Lazeron 1988). However, there are numerous empirical considerations in favor of the view that multisensory experience typically is genuinely perceptual and not e.g. associative (for an overview of empirical considerations, see e.g. Giard and Péronnet 1999; Molholm et al. 2002; Klemen and Chambers 2011; Talsma 2015). Here are two philosophical considerations in favor of the view that multisensory experience is perceptual rather than, say, associative. Ordinary visual experiences that result from stimulation of the individual senses, such as your visual experience of the line drawing of a rectangle in Figure 24.1, possess two interesting characteristics. RHOC.indb 326 21-02-2018 20:05:32 Multisensory Consciousness and Synesthesia 327 One is that your experience does not just represent something as being the case, but is also felt as putting you in touch with its subject matter (see Chudnoff 2014, 2016 for a discussion of this characteristic). One way to understand the idea of subject matter is in terms of truthmakers, where the truthmaker of an experience can be understood as the external mind-independent object in virtue of whose existence or non-existence the content of the experience is true or false (cf. Armstrong 1989: 88). So, here the truthmaker of your experience is the drawing of the rectangle. It is as if your experience makes you directly aware of the drawing of the rectangle. We'll call this characteristic "presentational phenomenology." In order for an experience to have presentational phenomenology, it is not necessary that we appear to see all aspects of what is presented to us. Consider the following case. You walk down the hallway and see a dog, partially occluded from your field of vision (Figure 24.2). In spite of the fact that only the non-occluded parts of the dog reflect light that reaches your retina, it appears to you as if there is a whole dog, not merely a part of a dog.9 So, your experience of the dog has presentational phenomenology. Another characteristic of ordinary visual experience is that it is evidence insensitive (understood as a feature of the phenomenology; see Brogaard 2016, in press a, for a discussion of this characteristic). Consider the Müller-Lyer illusion in Figure 24.3 (the figure on the left). Figure 24.1 Line Drawing of a Rectangle. Every part of your experience of the line drawing has presentational phenomenology Figure 24.2 Occluded Dog. Even though the occluded parts of the dog do not make an imprint on the retina, the visual system nonetheless generates a complete dog. This is also known as "amodal completion" RHOC.indb 327 21-02-2018 20:05:33 Berit Brogaard and Elijah Chudnoff 328 The two line segments on the left strongly appear to have different lengths. However, as the marking on the right illustrates, they have exactly the same length. Our knowledge of this fact (our possession of evidence), however, does not change the visual appearance of the line segments on the left. They continue to look as if they have different lengths. This evidence insensitivity is typical of the archetypes of visual experience. Just like experiences that result from amodal completion can have presentational phenomenology, they can also be evidence insensitive. Consider again the image of the occluded dog in Figure 24.2. Although an occluder obscures your line of sight, you naturally see this as a complete dog. Now, let's remove the occluder (Figure 24.4). The experience produced by the process of amodal completion in Figure 24.2 turned out to be illusory. The dog is lacking its middle part. However, even after it's revealed that there isn't a complete dog behind the occluder, what is presented in Figure 24.2 still appears equally complete. So, the amodally completed experience persists (i.e., the dog looks complete) even when we know that the world is not as it appears to be. Now, let's consider whether modal and amodal multisensory experience possess the two characteristics: presentational phenomenology and evidence insensitivity. We shall here focus on the modal integration cases, but nothing in what follows hinges on this. Consider once again a case of seeing a source produce a sound. Suppose we see a busboy lose his grip on a stack of plates he is carrying. They hit the tile floor in the restaurant right in front of our table. This results in the loud sound of plates breaking against the tile floor. We can literally hear the plates break. In the envisaged scenario, it would appear that we are in direct conscious touch with the event producing the sound. The multisensory experience of hearing the plates break has an integrated presentational phenomenology that consists partly in the phenomenology of the visual experience that produces the analog of a demonstrative and partly in the phenomenology of the auditory experience that attributes audible qualities to the seen event. The fact that the Figure 24.3 The Müller-Lyer Illusion. Even when you learn that the line segments on the left have the same length, they continue to appear as if they have different lengths Figure 24.4 Incomplete Drawing of a Dog. Even after seeing that there is nothing behind the occluder in this figure, the visual nonetheless still generates a visual experience of a dog when viewing the occluded figure in Figure 24.2 RHOC.indb 328 21-02-2018 20:05:33 Multisensory Consciousness and Synesthesia 329 multisensory experience has an integrated presentational phenomenology gives us some reason to think that the phenomenology is perceptual. Now consider the phenomenon of ventriloquism. We know that the ventriloquist produces the voice of the puppet in his hand. Even so, the voice perceptually appears to come from the puppet's mouth. The appearance that the puppet is speaking is so strong that it persists in spite of our knowledge that this is not so-which is to say, ventriloquism is evidence insensitive. In fact, most of us are taking advantage of the evidence-sensitivity of ventriloquism on a daily basis, when we watch television. Multisensory experience thus can have a presentational phenomenology and may be evidence insensitive. This indicates that the integration process is perceptual as opposed to inferential or loosely associative. 3 Modal versus Amodal Binding: An Argument Above, we distinguished a notion of modal integration and made a prima facie case for describing some cases of mutisensory integration in terms of it. The aim of this section is to argue that there are cases of integration that cannot be accounted for in terms of amodal integration but can be explained only given the notion of modal integration. Modal integration requires that qualities represented because they are perceived in one modality are attributed to an object or event represented because it is perceived in another modality. Amodal integration does not require this sort of dependence; modality 1 attributes qualities to an object because it is perceived in modality 1, and modality 2 attributes qualities to an object because it is perceived in modality 2; the integration, that is the identification of the object presented in modality 1 with the object presented in modality 2, is performed amodally. To see that the two notions of integration come apart, let us consider some phenomena of referential dependence that amodal integration by itself is unable to account for.10 Imagine being at a cocktail party whose acoustics disrupt one's ability to hear sounds as coming from specific directions and where everyone has the same voice and everyone is speaking the same words. Despite the unusual conditions you might have a sensory experience as of some specific person speaking. In order for this to be the case, you will need to have some sensory manner of picking out the person, let's stipulate a visual manner. The experience of seeing someone speak cannot be the result of visually referring to a person, aurally referring to a person, and amodally identifying the referents. Since you are in an environment where you cannot pick out individuals by their sounds alone there is no aural reference to any particular person. You can, however, pick out people by their looks, positions, and motions. So, you are able to identify who is saying what by visually referring to a person and aurally exploiting that reference in order to attribute the quality of saying something to him or her. This results in what we call a modally integrated experience as of some specific person speaking. It should be emphasized that the dependence relation in modal multisensory experience can go in both directions. Suppose you are out jogging one particularly foggy morning. You see a person wave to you from the other side of the street. As it turns out, it is your colleague Magdalena. But the visibility is not good enough for you to identify the speaking event as an event in which your colleague is speaking on the basis of the person's look, posture or gait. You can, however, identify the event as a speaking event by your colleague by the sound of her voice as she shouts "Hey! See you later at work!" In this case the low visibility prevents you from identifying the speaking event as being the event of your colleague speaking. You can, however, identify this event on the basis of your auditory perception of the sound event. Here visual qualities are attributed to a sound event identified by audition. RHOC.indb 329 21-02-2018 20:05:33 Berit Brogaard and Elijah Chudnoff 330 To summarize: In our first case vision enables you to pick out a person and their speaking motions. But your overall experience involves attributing audible qualities to that person and their speaking motions. Audition alone, however, is not sufficient for this attribution. Audition is dependent on vision in that it attributes the quality of making certain sounds to the seen person and their speaking motions. It does this by making use of a reference to that person and their speaking motions which is supplied by vision. In the third case the dependence goes in the other direction since vision depends on audition for its possession of the further content that the person speaking is one we recognize (e.g., Magdalena). Amodal integration differs from modal integration in that there is no referential dependence. If you are seeing and holding a tomato, the object you are seeing and holding can be picked out in virtue of how it appears within each sensory modality. You see the tomato as shiny, and your touch identifies the tomato as firm. You do not need vision to confirm that the tomato you see is firm, and you do not need your sense of touch to confirm that the tomato is shiny. What integration accomplishes in this case is the attribution of the two qualities shiny and firm to one and the same object. Of course, multisensory perception also attributes common sensibles to objects, for instance, roundness to the tomato. But you can confirm that the tomato is round by sight or touch alone. You don't need both sensory modalities to perceptually establish this. If you are unable to perceive the roundness in one sensory modality, this simply means that roundness is not a common sensible for you. Integration is needed for you to come to have a mutlisensory experience of tomato as round. This suggests that the unitary experiences in the case of amodal multisensory integration are prior temporally to the multisensory experience itself, which is consistent with the integration taking place in higher brain regions, such as the parietal cortex. So, amodal multisensory experience does not require that qualities perceived in one sensory modality are attributed to an event perceived in another sensory modality in order for the integration to occur. Hence, modal and amodal multisensory experience are distinct. 4 Synesthesia In the previous sections, we have been concerned primarily with ordinary multisensory experience. We should, however, briefly compare ordinary multisensory experience to one of the most common forms of atypical multisensory experience, viz. synesthesia (occurring in 4–7 percent of the population). Synesthesia is a peculiar way of experiencing the world in which internal or external input gives rise to atypical sensations or thoughts (Baron-Cohen et al. 1987; Cytowic 1989; Grossenbacher and Lovelace 2001; Ramachandran and Hubbard 2001b; Rich and Mattingley 2002; Ward 2013). For example, seeing the number 3 printed in black ink may lead to a sensation of copper green, hearing the word "abyss" may flood the mouth with the flavor of minestrone soup and hearing the key of C# minor may elicit a slowly contracting turquoise spiral. In grapheme-color synesthesia, one of the most common forms of synesthesia, perceiving or thinking about an achromatic grapheme (also known as the "inducer") triggers the sensation or thought (also known as the "concurrent") that the grapheme has a specific color with a highly specific hue, brightness and saturation (Simner et al. 2006). The concurrent images are either projected onto the external world (projector synesthesia) or perceived in the mind's eye (associator synesthesia) (Dixon et al. 2004). In projector synesthesia, the projected concurrent may be seen as instantiated like non-synesthetic colors, as floating above its inducer or as an "afterimage" that floats close to the subject's eyes. In associator synesthesia, the concurrent image is seen internally, much like a visual image retrieved from memory or produced by imagination. Two key characteristics of synesthesia are (i) automaticity and (ii) stability and consistency over time. Automaticity refers to the observation that synesthetes cannot suppress the RHOC.indb 330 21-02-2018 20:05:33 Multisensory Consciousness and Synesthesia 331 association between an inducer and its concurrent. Stability and consistency over time refer to the observation that inducer-concurrent associations are highly stable and consistent in more than 80 percent of cases (Mattingley et al. 2001). Automaticity is supported by research showing that synesthetes are susceptible to Stroop effects (Stroop 1935). The most common Stroop task demonstrates that it takes significantly longer for neurotypical individuals to name the color in which a color word is printed if the color referred to by the word is incongruent with the printed color (see Figure 24.5). Likewise, it takes significantly longer for synesthetes to name the printed color of a grapheme if the synesthetic color induced by the grapheme is incongruent with the printed color (Mattingley et al. 2001). Consistency and stability over time in grapheme-color and sound-color synesthesia is commonly tested using the synesthesia battery on separate occasions (Eagleman et al. 2007). In the test of grapheme-color synesthesia, a subject is presented with a randomly chosen grapheme, for which she must choose a specific hue, brightness and saturation from a color palette representing over 17.6 million distinct choices. After the subject has repeated the task three times for each grapheme (108 trials; graphemes A–Z and 0–9), the geometric distance among the subject's answers in red, green and blue color space is calculated. Synesthesia requires that the geometric distance falls below a normalized threshold. Projector synesthesia (where the concurrent is projected out onto the external visual scene) is not always a genuine form of multisensory or multisensory-stream experience. Evidence indicates that some grapheme-color synesthetes have an unusual structural connection between the color area in the brain and the neighboring form area (Ramachandran and Hubbard 2001a; Rouw and Scholte 2007; Jancke et al. 2009; Hanggi et al. 2011). Likewise, there is some evidence to suggest that some sound-color synesthetes have an unusual structural relation between auditory areas and the form area (Zamm et al. 2013). Because a structural relation directly combines two areas of the brain that are not normally directly combined, the synesthetic experiences in structurally induced synesthesia are not best characterized as multimodal but are better characterized as an augmented form of ordinary unimodal perception. The individual sensory pathways are simply mistakenly, or atypically, blended into a single pathway, thus forming an augmented sensory pathway that yields illusory or hallucinatory experiences (e.g., the experience of the musical note D as purple or the experience of a black letter as red). The more interesting cases of synesthesia for our purposes are cases of associator synesthesia (and perhaps functional cases of projector synesthesia) that are a result of unusual binding in higher areas of the brain, most likely the parietal cortex. On the so-called disinhibited integration model, synesthesia occurs owing to disinhibition of an area in the parietal cortex that is thought to bind information from different senses, causing information from one sensory modality to trigger the projection of information from another modality (Grossenbacher 1997; Armel and Ramachandran 1999; Grossenbacher and Lovelace 2001; Myles et al. 2003). Information from the two sensory sources is then integrated. For example, information about the identity of a grapheme may combine with abnormal color information, giving rise to an experience of an abnormally colored grapheme. One important piece of evidence cited in favor of this hypothesis comes from a case study in which a patient PH reported seeing visual movement in response to tactile stimuli following Figure 24.5 The Stroop Effect. The word "red" is displayed in the color black (left) and the color green (right-here displayed in gray). It takes longer for subjects to name the ink color of the word "red" when it is printed in green than when it is printed in black or red RHOC.indb 331 21-02-2018 20:05:33 Berit Brogaard and Elijah Chudnoff 332 acquired blindness (Armel and Ramachandran 1999). As PH was blind, he could not have received the information via standard visual pathways. It is plausible that the misperception was a result of disinhibited integration of tactile information and information from the visual motion areas. Another piece of evidence cited in favor of the disinhibited integration model is the observation that visual context and meaning can influence the phenomenal character of synesthetic experience (Myles et al. 2003; Dixon and Smilek 2005). To illustrate, consider the two words in Figure 24.6. Some grapheme-color synesthetes assign different colors to the shared letter depending on whether they interpret the string of letters as spelling the word "POT" or the word "JACK." For example, a grapheme-color synesthete might have an experience of the shared letter as yellow (O) when she reads the word "pot" but have an experience of the letter as pink (C) when she reads the word "Jack." One way to explain this phenomenon is that amodal completion of the shared grapheme takes place when the synesthete is reading the word "POT" but not when she is reading the word "JACK." This explanation is consistent with there being a direct structural connection between the form and the color area in the brain. However, the more widely accepted explanation is that it's not the shape actually presented in experience that triggers the color experience, but rather the higher-level property of being a particular grapheme (e.g., being the grapheme O or being the grapheme C ) (Cytowic and Eagleman 2009: 75). Synesthesia of this second type and ordinary multisensory experience that results from amodal integration both appear to involve higher-level perceptual brain regions (like the parietal cortex) in the integration process, and both types of integration involve attributing features to one and the same object (e.g., being the musical note D and being purple). Using the example of seeing and holding a firm tomato and hearing the musical note D as purple, we can illustrate the commonalities between the two phenomena as follows: Normal Amodal Integration Your overall experience has the content: thatv is a tomato and thatt is firm and thatv = that t [where that v is a visual demonstrative, and that t is a tactile demonstrative, and that v = that t is an amodally represented identification]. Figure 24.6 Jackpot Figure. Synesthetes interpret the middle letter as a C when it occurs in "Jack" and as an O when it occurs in "pot." The color of their synesthetic experience will depend on which word the grapheme is considered a part of RHOC.indb 332 21-02-2018 20:05:34 Multisensory Consciousness and Synesthesia 333 Amodal Integration in Sound-Color Synesthesia Your overall experience has the content: that h is the musical note D and that v is purple and that h = that v [where that h is an auditory demonstrative, and that v is a visual demonstrative, and that h = that v is an amodally represented identification]. Now, as noted above, synesthesia also occurs within a single sensory modality combining different sensory streams. In grapheme-color synesthesia, for example, a shape property (e.g., having the shape of the grapheme 3) and a color (e.g., being green) are visually attributed to a grapheme printed in black. Although this phenomenon is not genuinely multisensory, it nonetheless fits the model. Two visual properties that normally are not integrated are amodally attributed to one and the same object, after being computed separately in separate sensory streams. Using the example of seeing the grapheme 3 as green, we can illustrate this as follows: Amodal Integration in Grapheme-Color Synesthesia Your overall experience has the content: that v1 has the shape of the grapheme 3 and that v2 is purple and that v1 = that v2 [where that v1 is a form area demonstrative, and that v2 is a color area demonstrative, and that v1 = that v2 is an amodally represented identification]. Because of the similarities between synesthesia of the kind under consideration and amodal multisensory experience, research into synesthesia of this type will likely be able to shed light on the process underlying integration in ordinary amodal multisensory experience. 5 Conclusion We can divide multisensory experiences that go beyond mere co-consciousness (co-consciousness as in the experience of tasting the wine and hearing the siren from the street) into two broad categories. Multisensory experiences-such as feeling the roundness of a tomato through touch and seeing the roundness of the tomato, smelling the Indian curry and seeing it boil or perceiving the flavor of the Oxtail flatbread by gustatorily, olfactorily, somato-sensorily, thermaly and nociceptually attributing features to the flatbread-attribute one or more features to a single object. Experiences of this kind arguably have a phenomenology that reflect that they are integrated amodally. As we have seen, however, not all forms of multisensory perception are amodal in this sense. Some forms are distinctly perceptual and have a phenomenology that derives from the phenomenology of the individual sensory modalities. Seeing someone speak and feeling the rock press against the palm are experiences of this latter kind. Synesthesia is a form of atypical multisensory experience that in some instances involves integration of the first type. Research into this type of synesthesia might thus help shed light on the mechanism underlying amodal integration.11 Notes 1 It is widely agreed that there are temporal and spatial congruity constraints on multisensory integration (see e.g. O'Callaghan 2014). If, for example, the visual and audible properties are temporally incongruous you will fail to see a seen event as the one producing the sound. If, for instance, you see a drummer but then hear the drumming sounds only ten seconds later, you will fail to attribute the sound to the drumming. Likewise, if the visual and audible properties are blatantly spatially incongruous, you will fail to see a seen event as the one producing the sound. Suppose, for instance, that you see a person to the left of you move her lips and you also hear corresponding sounds in the distance-far too removed RHOC.indb 333 21-02-2018 20:05:34 Berit Brogaard and Elijah Chudnoff 334 from the person to be attributable to her. In that case, you will not perceive the person as producing the sounds. We are going to take that for granted in what follows. 2 The idea of the phenomenology deriving exclusively from the phenomenology of the individual sensory modalities is formulated as follows by O'Callaghan (2015: 555): "The phenomenal character of each perceptual episode is exhausted by that which is associated with each individual modality, along with whatever accrues thanks to mere co-consciousness." 3 While amodal experience may seem to be a kind of perception that is cognitively penetrated, most cognitive effects on the integration turn out primarily to be related to attention. Multisensory integration is thus largely accounted for by attentional mechanisms (see Talsma 2015). 4 For simplicity's sake, we shall here assume a representational account of experience according to which the phenomenology of experience (at least typically) reflects a representational content. This is also an assumption made by e.g. O'Callaghan (2012). See also Bourget (2017). Here we are not taking a stance on the question of whether strong representationalism about multisensory experience is feasible (for discussion see e.g. O'Dea 2006; Tye 2007; and Bourget 2017). 5 Bourget (2017) also distinguishes between these two types of multisensory experience (that go beyond mere co-consciousness). However, he argues for a view where the two have different generalized contents. Seeing something produce a sound has a content of the form ∃x,y(F(x) ∧ G(y) ∧ R(x,y), where x and y range over related entities to which different features are attributed. Seeing and feeling a tomato, by contrast, has a content of the form: ∃x(F(x) ∧ G(x)). Here different features are attributed to one and the same object. 6 The intermodal interaction can be direct or facilitated by cortico-thalamo-cortical pathways (see Talsma 2015). 7 We can still allow for the possibility that lip reading can produce an experience of meanings (cf. Brogaard 2016). In this case, however, the experience of meanings is not auditory but visual, much like the case of ordinary reading. 8 Bodily sensations (or bodily feelings-a sub-set of the set of interceptive experiences) have not traditionally been construed as sensory experiences. However, one might argue that the modality that produces bodily feelings just is a sensory modality closely related to proprioception, our sense of balance (the vestibular system) and nociception (pain and spice perception), which arguably are sensory modalities, unlike intuition and introspection (Macpherson 2011; Schwenkler 2013; Briscoe 2016). Not much hinges on how we settle this issue. 9 We shall set aside the issue of whether we can perceive high-level properties like that of being a dog. Let it be granted for argument's sake that we can perceive such properties. Nothing in what follows hinges on this assumption. 10 For other illustrative examples of cases where the information in one sensory modality cannot be decoded without the assistance of a second sensory modality, see e.g. Talsma (2015). One illuminating example is that of the Swedish chef in The Muppet Show. Upon your first encounter with the character, his speech sounds entirely garbled. After multiple other cues (primarily visual) have been presented to you, you realize that the character actually utters English sentences but with an extremely anomalous accent (analogous to sine-wave speech). 11 For comments on this chapter we are grateful to Anna Drozdzowicz, Rocco J. Gennaro, Anders Nes, Sebastian Watzl, the participants in a multisensory perception seminar in Oslo and an audience at a cognitive penetration workshop in Bergen. References Armel, K.C., and Ramachandran, V.S. (1999) "Acquired Synesthesia in Retinitis Pigmentosa," Neurocase 5: 293–296. Armstrong, D.M. (1989) Universals: An Opinionated Introduction, Boulder: Westview Press. Baron-Cohen, S., Wyke, M., and Binnie, C. (1987) "Hearing Words and Seeing Colors: An Experimental Investigation of Synesthesia," Perception 16: 761–767. Bayne, T. (2014) "The Multisensory Nature of Perceptual Consciousness," in D. Bennett and C. Hill (eds.) Sensory Integration and the Unity of Consciousness, Cambridge, MA: MIT Press. Bayne, T., and Chalmers, D. J., (2003) "What Is the Unity of Consciousness?" in A. Cleeremans (ed.) The Unity of Consciousness: Binding, Integration and Dissociation, Oxford: Oxford University Press. Bloom, F.E., and Lazeron, A. (1988) Brain, Mind, and Behavior, New York: W. H. Freeman and Company. RHOC.indb 334 21-02-2018 20:05:34 Multisensory Consciousness and Synesthesia 335 Bourget, D. (2017) "Representationalism and Sensory Modalities: An Argument for Intermodal Representationalism," American Philosophical Quarterly 54: 251–267. Briscoe, R. E. (2016) "Multisensory Processing and Perceptual Consciousness: Part I," Philosophy Compass 11 (2): 121–133. Briscoe, R. E. (In Press) "Multisensory Processing and Perceptual Consciousness: Part II," Philosophy Compass. Brogaard, B. (2012) "What Do We Say When We Say How or What We Feel?" Philosophers Imprint 12 (11), June 2012. Brogaard, B. (2016) "In Defense of Hearing Meanings," Synthese (2016). doi:10.1007/s11229-016-1178-x. Brogaard, B. (In Press a) Seeing and Saying, New York: Oxford University Press. Brogaard, B. (In Press b) "Knowledge-How and Perceptual Learning," in S. Heatherington and M. Valaris (eds.) Knowledge in Contemporary Philosophy, London: Bloomsbury. Brogaard, B., and Chudnoff, E. (2016) "Against Emotional Dogmatism," Philosophical Issues, a supplement to Nous 26 1: 59–77 Chalmers, D. J. (2004) "The Representational Character of Experience," in Brian Leiter (ed.) The Future for Philosophy, Oxford: Oxford University Press. Chudnoff, E. (2014) "Review of Tucker (eds.) Seemings and Justification," Notre Dame Philosophical Reviews. Chudnoff, E. (2016) "Moral Perception: High Level Perception or Low Level Intuition?" In T. Breyer and C. Gutland (eds.) Phenomenology of Thinking: Philosophical Investigations into the Character of Cognitive Experiences, New York: Routledge. Chudnoff, E. (In Press) "The Epistemic Significance of Perceptual Learning," Inquiry. Cytowic, R.E. (1989) Synesthesia: A Union of the Senses, New York: Springer Verlag. Cytowic, R.E., and Eagleman, D.M. (2009) Wednesday Is Indigo Blue, Cambridge, MA: MIT Press. Dainton, B. (2000) Stream of Consciousness: Unity and Continuity in Conscious Experience, New York: Routledge. Degenaar, M. and Lokhorst, G.J., "Molyneux's Problem," The Stanford Encyclopedia of Philosophy (Spring 2014 Edition), E.N. Zalta (ed.), URL = https://plato.stanford.edu/archives/spr2014/entries/moly neux-problem/. Deroy, O. (2014) "The Unity Assumption and the Many Unities of Consciousness," in D. Bennett and C. Hill (eds.) Sensory Integration and the Unity of Consciousness, Cambridge, MA: MIT Press. de Vignemont, F. (2014) "A Multimodal Conception of Bodily Awareness," Mind 123: 989–1020. de Vignemont, F., and Massin, O. (2015) "Touch." In M. Matthen (ed.) The Oxford Handbook of the Philosophy of Perception, Oxford: Oxford University Press. Dixon, M.J., Smilek, D., and Merikle, P.M. (2004) "Not All Synaesthetes are Created Equal: Projector versus Associator Synaesthetes," Cognitive, Affective, and Behavioral Neuroscience 4: 335–343. Dixon, M.J., and Smilek, D. (2005) "The Importance of Individual Differences in Grapheme-Color Synesthesia," Neuron 45: 821–823. Eagleman, D.M., Kagan, A.D., Nelson, S.S., Sagaram, D., and Sarma, A.K. (2007) "A Standardized Test Battery for the Study of Synesthesia," Journal of Neuroscience Methods 159: 139–145. Fulkerson, M. (2011) "The Unity of Haptic Touch," Philosophical Psychology 24: 493–516. Fulkerson, M. (2014) The First Sense: A Philosophical Study of Human Touch, Cambridge, MA: MIT Press. Gallace, A., and Spence, C. (2014) In Touch with the Future: The Sense of Touch from Cognitive Neuroscience to Virtual Reality, Oxford: Oxford University Press. Giard, M. H., and Péronnet, F. (1999) "Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study," Journal of Cognitive Neuroscience 11: 473–490. Grossenbacher, P.G. (1997) "Perception and Sensory Information in Synaesthetic Experience," in S. BaronCohen and J.E. Harrison (eds.) Synaesthesia: Classic and Contemporary Readings, Malden, MA: Blackwell Publishers. Grossenbacher, P. G., and Lovelace, C. T. (2001) "Mechanisms of Synesthesia: Cognitive and Physiological Constraints," Trends in Cognitive Science 5: 36–41. Hanggi, J., Wotruba, D., and Jäncke, L. (2011) "Globally Altered Structural Brain Network Topology in Grapheme-Color Synesthesia," Journal of Neuroscience 31: 5816–5828. Jancke, L., Beeli, G., Eulig, C., and Hanggi, J. (2009) "The Neuroanatomy of Grapheme-Color Synesthesia," European Journal of Neuroscience 29: 1287–1293. Jones, L.A., and Lederman, S.J. (2006) Human Hand Function, New York: Oxford University Press. Klemen, J., and Chambers, C. D. (2011) "Current Perspectives and Methods in Studying Neural Mechanisms of Multisensory Interactions," Neuroscience and Biobehavioral Reviews 36: 111–133. Linden, D. J. (2015) Touch: The Science of Hand, Heart, and Mind, New York: Penguin Publishing Group. RHOC.indb 335 21-02-2018 20:05:34 Berit Brogaard and Elijah Chudnoff 336 Loomis, J., and Lederman, S. (1986) "Tactual Perception," in K.R. Boff, L. Kaufman, and J.P. Thomas (eds.) Handbook of Perception and Human Performance, New York: Wiley and Sons. Macpherson, F. (ed.) (2011) The Senses: Classic and Contemporary Philosophical Perspectives, Oxford: Oxford University Press. McGurk, H., and MacDonald, J. (1976) "Hearing Lips and Seeing Voices," Nature 264: 746–748. Mattingley, J.B., Rich, A.N., Yelland, G., and Bradshaw, J.L. (2001) "Unconscious Priming Eliminates Automatic Binding of Colour and Alphanumeric Form in Synaesthesia," Nature 410: 580–582. Molholm, S., Ritter, W., Murray, M. M., Javitt, D. C., Schroeder, C. E., and Foxe, J. J. (2002) "Multisensory Auditory-Visual Interactions during Early Sensory Processing in Humans: A High-Density Electrical Mapping Study," Cognitive Brain Research 14: 115–128. Myles, K.M., Dixonm M.J., Smilek, D., and Merikle, P.M. (2003) "Seeing Double: The Role of Meaning in Alphanumeric-Colour Synaesthesia," Brain Cognition 53: 342–345. O'Callaghan, C. (2008) "Seeing What You Hear: Cross-Modal Illusions and Perception," Philosophical Issues 18: 316–338. O'Callaghan, C. (2012) "Perception and Multimodality," in E. Margolis, R. Samuels, and S. Stich (eds.) The Oxford Handbook of Philosophy of Cognitive Science, Oxford: Oxford University Press. O'Callaghan, C. (2014) "Not All Perceptual Experience Is Modality Specific," in D. Stokes, M. Matthen, and S. Biggs (eds.) Perception and Its Modalities, Oxford: Oxford University Press. O'Callaghan, C. (2015) "The Multisensory Character of Perception," Journal of Philosophy 112: 551–569. O'Dea, J. (2006) "Representationalism, Supervenience, and the Cross-Modal Problem," Philosophical Studies 130: 285–295. Ramachandran, V.S., and Hubbard, E.M. (2001a) "Psychophysical Investigations into the Neural Basis of Synaesthesia," Proceedings of the Royal Society B: Biological Sciences 268: 979–983. Ramachandran, V.S., (2001b) "Synaesthesia: A Window into Perception, Thought and Language," Journal of Consciousness Studies 8: 3–34. Rouw, R., and Scholte, H.S. (2007) "Increased Structural Connectivity in Grapheme-Color Synesthesia," Nature Neuroscience 10: 792–797. Rich, A.N., and Mattingly, J.B. (2002) "Anomalous Perception in Synaesthesia: A Cognitive Neuroscience Perspective," Nature Reviews Neuroscience 3: 43–52. Schwenkler, J. (2013) "The Objects of Bodily Awareness," Philosophical Studies 162: 465–472. Shams, L., Kamitani, Y., and Shimojo, S. (2000) "Illusions: What You See Is What You Hear," Nature 408 (6814): 788. Simner, J., Mulvenna, C., Sagiv, N., Tsakanikos, E., Witherby, S.A., Fraser, C., Scott, K., and Ward, J. (2006) "Synaesthesia: The Prevalence of Atypical Cross-modal Experiences," Perception 35: 1024–1033. Stroop, J.R. (1935) "Studies of Interference in Serial Verbal Reactions," Journal of Experimental Psychology 18: 643–662. Talsma, D. (2015) "Predictive Coding and Multisensory Integration: An Attentional Account of the Multisensory Mind," Frontiers in Integrative Neuroscience 9: 19. doi:10.3389/fnint.2015.00019. Tuomainen, J., Andersen, T. S., Tiippana, K., and Sams, M. (2005) "Audio-Visual Speech Perception Is Special," Cognition 96: B13–B22. Tye, M. (2007) "The Problem of Common Sensibles," Erkenntnis 66: 287–303. Ward, J. (2013) "Synesthesia," Annual Reviews of Psychology 64: 49–75. Zamm, A., Schlaug, G., Eagleman, D. M., and Loui, P. (2013) "Pathways to Seeing Music: Enhanced Structural Connectivity in Colored-Music Synesthesia," Neuroimage 74: 359–366. Related Topics The Unity of Consciousness Consciousness and Conceptualism Consciousness and Attention RHOC.indb 336 21-02-2018 20:05: