Rewired animals and sensory substitution: the cause is not cortical plasticity J. Kevin O'Regan, Laboratoire Psychologie de la Perception, Université Paris Descartes, Paris, France --‐--‐--‐--‐--‐ DRAFT : 14 September 2013 --‐--‐--‐--‐--‐ --‐--‐--‐--‐--‐ Abstract and Keywords added 3 Feb 2014 --‐--‐--‐--‐--‐ Abstract Cortical plasticity is often invoked to explain changes in the quality or location of experience observed in rewired animals, in sensory substitution, in extension of the body through tool use, and in the rubber hand illusion. However this appeal to cortical plasticity may be misleading, because it suggest that the cortical areas that are plastic are themselves the loci of generation of experience. This would be an error, I claim, since cortical areas do not generate experience. Cortical areas participate in enabling the interaction of an agent with its environment, and the quality of this interaction constitutes the quality of experience. Thus it is not plasticity in itself, but the change in modes of interaction which plasticity allows, which gives rise to the change of experience observed in these studies. Keywords: Plasticity, sensory substitution, sensorimotor theory, rewired animals. William James said that if we were able to re--‐direct nerves from the eye so that they connected to the brain centres concerned with hearing, and vice versa, then we would "hear the lightning and see the thunder" (James, 1890). Underlying this statement is the idea that what determines the quality of sensations is the particular brain area that processes the neural influx. Since James' time, an enormous literature has lent support to this idea, including several iconic results. Among these is Penfield's demonstration that tactile sensations in different body locations can be elicited by stimulation of corresponding areas of a somatosensory map or "Penfield homunculus", and that stimulation of other areas can actually generate visual or auditory experiences (Penfield, 1958). Since those days the idea of "localisation of function" in the brain has become a doctrine (cf. Phillips, Zeki, & Barlow, 1984), with a large portion of neuroscience research devoted to determining the particular functions of the different brain areas. Applied to perception, the localisation of function doctrine is often taken to suggest that there is something special about each kind of cortex that underlies the particular quality of sensations that that kind of cortex allows. Just as there is something special about Penfield's somatosensory cortex which allows it to generate tactile sensations, there is something special about visual, auditory, etc., cortices which allows them to generate visual, auditory, etc., sensations. Within this way of thinking, the studies of "rewired" animals and visual cortex in blind individuals have added an interesting twist, which reveals a subtle issue concerning the causal status of the notion of cortical plasticity. Rewired animals and visual cortex in blind persons The studies of rewired animals investigate what happens when animals are surgically rewired at birth in such a way that information from the eyes, usually transmitted to visual cortex, is re--‐ routed so that it projects into auditory cortex (Frost, Boire, Gingras, & Ptito, 2000; Pallas, Roe, & Sur, 1990; Pallas & Sur, 1993; Sharma, Angelucci, & Sur, 2000). The results show that when mature, the re--‐wired animals are able to perform visual behaviours almost as efficiently as normal animals using their visual cortex. Furthermore, the rewired auditory cortex reorganises and displays a two--‐dimensional orientation--‐sensitive structure similar to that observed in normal visual cortex. Of particular interest is one very ingenious study which was able to approach an answer to the question of whether the quality of the sensations experienced by such rewired animals via their auditory cortex was actually visual rather than auditory (von Melchner, Pallas, & Sur, 2000). The authors used ferrets that were only rewired in one hemisphere. They first trained these animals to distinguish between sounds and lights that were presented in such a way as to be processed only by the normal hemisphere. When completely trained, the animals were tested with lights that were processed only by the rewired hemisphere, that is, through auditory cortex. The animals invariably turned in a way suggesting that they perceived the lights visually, even though the perception occurred via their auditory cortex. When the auditory cortex was removed, the animals behaved as though they were blind. Note that the authors did not observe any lateral connections between the "converted" auditory cortex and existing visual cortex. Thus it truly appears that the visual sensations were related to the presence of the auditory cortex, and not to stimulation spreading to visual cortex. We thus seem to have a case in which, contrary to William James's prediction, it seems that the animals end up "seeing the lightning rather than hearing it". Another well--‐publicised result showing adaptation of sensation generated in cortical areas has emerged recently in studies on blind people. For such persons it has been shown that visual cortex can rewire itself so as to participate in other functions, including language processing, memory, motion detection and navigation. The results are particularly strong in the congenitally blind, where visual cortex has never been exposed to visual stimulation. Of interest as regards the question of the experienced quality of sensations are the results of Kupers et al. (2006) and Ptito, Schneider, Paulson, & Kupers (2008). They trained subjects to use a sensory substitution device called the "Tongue Display Unit", which converts visual stimuli captured by a video camera into tactile stimulation on the tongue. After learning to do orientation, form or motion discrimination tasks with the device, early blind subjects given Transcranial Magnetic Stimulation (TMS) directed onto their visual cortex felt tingling sensations on their tongue. Visual cortex thus seems now to underlie the generation of tactile sensations on the tongue. Cortical plasticity All these results show that whatever it is about a particular cortical area that gives rise to a particular quality of sensation (e.g. visual, auditory, tactile), it does not lie in the specific cortical area or type of neurons, but in something that can be modified as a function of adaptation to extrinsic stimulation. As noted by Hurley & Noë (2003), the cortex "defers" to the particular sensory input channel. Such a modification is generally referred to as being an instance of "cortical plasticity". Researchers will tend to say that cortical plasticity has allowed auditory cortex to change its function, going from generating auditory sensations to generating visual sensations. The implicit underlying assumption is that cortex generates sensations, and so to have different sensations, cortex must change its mode of functioning. But to say this is to fall into a "category mistake" (Ryle, 1949). For suppose it made sense to say that different cortical areas "generated" different types of sensations. Then we would be faced with the problem of explaining exactly how this came about. Exactly what is it about the different areas that allows them to generate the particular sensations they provide? This is the vexed problem of "qualia", for which a variety of esoteric hypotheses have been advanced, going from quantum gravity effects in microtubules, through synchronous oscillations in widespread cortical areas, to cortico--‐thalamic reverberations --‐--‐ see the list provided by Chalmers (1997). For any such hypothesis, as noted by Dennett (1991), an infinite regress of questions ("And then what happens?") can always be asked about what exactly it is about these mechanisms that produces feel rather than no feel? And why are the feels the way they are, rather than some other way? Any explanation for similarities and differences in feels that is made in terms of physico--‐chemical mechanisms requires making a link between the language of neural processes and the language of sensations: and there is no non--‐arbitrary way to make that link. For that reason the sensorimotor approach (O'Regan & Noë, 2001; O'Regan, 2011) proposes to reject the notion that sensations are the kinds of things that can be generated. The sensorimotor approach proposes instead that it is scientifically more fruitful to consider the qualities of sensations as laws of interaction with the world. For example the sensation of softness consists in the law that links the sensory inputs you receive from your fingers with the pressure you exert on the sponge. The sponge is soft if it squishes under your fingers. Taking this stance is not just a philosophical trick, it actually leads to scientific predictions and paradigms, like the work on change blindness, inattentional blindness, sensory substitution and color psychophysics. In conclusion then, if the sensorimotor approach is correct, then it is misleading to say that in rewired animals cortical plasticity has allowed "auditory cortex to change its function, going from generating auditory sensations to generating visual sensations." The real cause What then is the correct thing to say about what happens with re--‐wired animals? We must say that the animal has become able to use its auditory cortex instead of its visual cortex in overseeing the sensorimotor interactions it has with the world using its eyes. Initially it tries to react appropriately to input from the eyes, but because at first auditory cortex is not wired up to do the task efficiently, the animal does not process the input nor react quite as efficiently as if it were using visual cortex. But gradually there is learning, and auditory cortex reorganises itself to do the task more efficiently. When things stabilize however, it would be a mistake to say that auditory cortex was "generating" visual qualia. No: auditory cortex is now participating in visual--‐type interactions in a way that is almost as efficient as visual cortex usually would be. Such a situation is called "seeing". Of course we could call the change from inefficient to efficient use of the auditory cortex "cortical plasticity". But it is important to be aware that this "plasticity" is not in itself what is causing the transition to seeing with the auditory cortex. The plasticity is allowing the animal to interact in a visual fashion, but it is not effecting a magical transition from generating auditory to generating visual qualia. Indeed there is another argument supporting the view that cortical plasticity is not the cause of the radical changes in sensory quality observed in rewired animals. It comes from other phenomena where drastic changes in the locus, modality or nature of sensations can occur in a very short time, making cortical plasticity, which is presumably a fairly slow process, a highly unlikely cause. One such phenomenon is the feeling of "projecting" your tactile sensation onto the tip of a tool: for example, when you write, you feel the roughness of the paper at the tip of the pen: and this happens even though you may hold the pen in any number of ways, and even though the vibrations from the paper are transmitted to widely different areas of your hand. The effect is immediate on setting pen to paper, and is thus presumably not the product of cortical plasticity. A similar example is the feeling you get immediately after learning to drive, of the tires touching the road or the curb when you park your car. The rapidity of these adaptations argues against cortical plasticity as an explanation. On the other hand, cortical plasticity does of course occur. As has been shown by an extensive literature, cortical changes occur when you use a tool: receptive fields normally situated at the hand can extend to the end of the tool (cf. Maravita & Iriki, 2004). This cortical plasticity, which may take weeks to set up in a macaque monkey, participates in improving the monkey's ability to move objects with a rake. But it is not this cortical plasticity which is the original cause of a change in felt location of touch. On the contrary, the felt location of touch is the brain's way of best accounting for the currently occurring sensorimotor contingencies. Cortical plasticity may improve this estimation, but it is not itself the cause of the change in sensation. The rubber hand illusion is another example: in this illusion, which takes only minutes to establish, touch actually occurs on the observer's arm, but is perceived as occurring on the rubber arm (Botvinick & Cohen, 1998). This is not because of cortical plasticity, but because the sensorimotor contingencies established artificially by the experimental manipulation can best be interpreted by the brain in terms of touch on the rubber arm rather than on the observer's arm. Finally sensory substitution, though often "explained" in terms of cortical plasticity (Bach--‐y--‐Rita, Danilov, Tyler, & Grimm, 2005; Bach--‐y--‐Rita, 1972), is in fact also not essentially caused by cortical plasticity. People using the TVSS or the vOICe report after only a few hours, if not within minutes, that they are getting "something like" a visual sensation. Of course the sensation can only be visual to the extent that the apparatus, and the observer's motor, sensory and cognitive faculties allow the interaction to approximate visual--‐type interactions with the world. But the effect occurs almost immediately, and is thus presumably not caused by cortical plasticity. Again, it is of course the case that over time, when a person becomes experienced with sensory substitution devices, cortical changes will occur. Brain structures will adapt so as to more efficiently do the processing involved in entertaining interaction with the world corresponding to the perceived modality. The experienced sensation will then more accurately approximate the normal sensation in that modality. But again, the cause for the experienced sensations lies not in the plasticity itself, but in the fact that the plasticity has enabled a mode of interaction with the environment which better approximates the new sensory modality. Conclusion It seems that the notion of cortical plasticity that researchers often refer to in the context of rewiring and sensory substitution may be somewhat misleading. Whereas the dramatic changes in experienced sensations seem to warrant appeal to some equally significant cortical modifications, I claim that taking this view would be to fall into Ryle's category mistake of thinking that sensations are the kinds of things that can be generated in the brain. In fact, if we admit that the quality of sensory experiences are constituted by the modes of interaction with the environment that they involve, then we see that the real cause of modification of sensations in cases like rewired animals and sensory substitution will necessarily lie in changes in the modes of interaction which are involved. These changes in modes of interaction can be enabled, or favored by cortical plasticity, but cortical plasticity is not in itself what causes the changes. What causes the modification in sensory experience is just the fact that, through the use of special instrumentation or interfaces, a person or an animal interacts with the world in a way which approximates a sensory modality that involves cortical areas or sensory apparatus usually used for other modalities. Bibliography Bach--‐y--‐Rita, P. (1972). Brain mechanisms in sensory substitution. New York: Academic Press. Bach--‐y--‐Rita, P., Danilov, Y., Tyler, M. E., & Grimm, R. J. (2005). Late human brain plasticity: vestibular substitution with a tongue BrainPort human--‐machine interface. J Intellectica, 40, 115– 122. Botvinick, M., & Cohen, J. (1998). Rubber hands "feel" touch that eyes see. Nature, 391(6669), 756. doi:10.1038/35784 Chalmers, D. J. (1997). The conscious mind: in search of a fundamental theory. Oxford University Press US. Dennett, D. C. (1991). Consciousness Explained. Boston:: Little, Brown & Co. Frost, D. O., Boire, D., Gingras, G., & Ptito, M. (2000). Surgically created neural pathways mediate visual pattern discrimination. Proceedings of the National Academy of Sciences of the United States of America, 97(20), 11068–11073. doi:VL --‐ 97 Hurley, S., & Noë, A. (2003). Neural plasticity and consciousness. Biology and Philosophy, 18(1), 131–168. James, W. (1890). The principles of psychology. New York: Dover Publications. Kupers, R., Fumal, A., de Noordhout, A. M., Gjedde, A., Schoenen, J., & Ptito, M. (2006). Transcranial magnetic stimulation of the visual cortex induces somatotopically organized qualia in blind subjects. Proceedings of the National Academy of Sciences, 103(35), 13256–13260. Maravita, A., & Iriki, A. (2004). Tools for the body (schema). Trends in Cognitive Sciences, 8(2), 79–86. doi:10.1016/j.tics.2003.12.008 Von Melchner, L., Pallas, S. L., & Sur, M. (2000). Visual behaviour mediated by retinal projections directed to the auditory pathway. Nature, 404(6780), 871–876. doi:10.1038/35009102 O'Regan, J. K. (2011). Why red doesn't sound like a bell: Understanding the feel of consciousness. New York: Oxford University Press, USA. O'Regan, J. K., & Noë, A. (2001). A sensorimotor account of vision and visual consciousness. Behavioral and Brain Sciences, 24(5), 883–917. Pallas, S. L., Roe, A. W., & Sur, M. (1990). Visual projections induced into the auditory pathway of ferrets. I. Novel inputs to primary auditory cortex (AI) from the LP/pulvinar complex and the topography of the MGN--‐AI projection. The Journal of Comparative Neurology, 298(1), 50–68. doi:10.1002/cne.902980105 Pallas, S. L., & Sur, M. (1993). Visual projections induced into the auditory pathway of ferrets: II. Corticocortical connections of primary auditory cortex. The Journal of Comparative Neurology, 337(2), 317–333. doi:10.1002/cne.903370212 Penfield, W. (1958). Ssome mechanisms of consciousness discovered during electrical stimulation of the brain. Proceedings of the National Academy of Sciences of the United States of America, 44(2), 51–66. Phillips, C. G., Zeki, S., & Barlow, H. B. (1984). Localization of Function in the Cerebral Cortex: Past, Present and Future. Brain, 107(1), 328–361. doi:10.1093/brain/107.1.328 Ptito, M., Schneider, F. C. G., Paulson, O. B., & Kupers, R. (2008). Alterations of the visual pathways in congenital blindness. Experimental Brain Research, 187(1), 41–49. doi:10.1007/s00221--‐008--‐1273--‐4 Ryle, G. (1949). The concept of mind. University of Chicago Press. Sharma, J., Angelucci, A., & Sur, M. (2000). Induction of visual orientation modules in auditory cortex. Nature, 404(6780), 841–847.