Sensorimotor Signature, Skill, and Synaesthesia. Two Challenges for Enactive Theories of Perception JOERG FINGERHUT Introduction The condition of 'genuine perceptual synaesthesia' has been a focus of attention in research in psychology and neuroscience over the last decades.1 For subjects in this condition stimulation in one modality automatically and consistently over the subject's lifespan triggers a percept in another modality. In hearing→colour synaesthesia, for example, a specific sound experience evokes a perception of a specific colour. In this paper, I discuss questions and challenges that the phenomenon of synaesthetic experience raises for theories of perceptual experience in general, and for theories that see the content and modality of conscious experience as being constituted and determined by the active and skilful exploration of the environment in particular. The focus of my paper will be on the latter, 'enactive' view of perception and its theory of what determines the modality-specific 'feel' of a perceptual experience.2 In genuine synaesthesia a local element – e.g. another percept – reliably and sufficiently triggers a specific response experience: no involvement of the body that is related to the response percept (like eye saccades for visual experience) seems to be needed. This constitutes the first challenge to enactivism: in cases of hearing→colour synaesthesia colour experiences are elicited that are not related to the normal sensorimotor signature of vision. The strong correlation between skill and experience that is predicted by enactivism does not hold in these cases. A second challenge is constituted by the fact that the established correlations between trigger and response percept seem to be stable over the lifespan of the 1 It has been argued that this is due to the use of non-invasive neuroimaging techniques that enable scientists to verify synaesthesia beyond phenomenological reports as well as the new general scientific interest in consciousness. Cf. Cytowic/Eagleman: Wednesday is Indigo Blue, pp. 235-237; see also the introduction to this volume, pp. 10-13. 2 In its general form this view dates back to the embodied conception of the mind put forth by John Dewey in the 19th century and has found a contemporary expression in the sensorimotor contingency theory of visual awareness. A precise and extensive formulation of this theory is Alva Noë's and Kevin O'Regan's "A Sensorimotor Account of Vision and Visual Consciousness". See also Myin/O'Regan: "Perceptual Consciousness, Access to Modality and Skill Theories" for an enactive account with a specific focus on the modalities of experience. 102 Joerg Fingerhut synaesthete without the response modality ever being related to its normal sensorimotor signature. Yet, experiments with visually distorting goggles and sensory substitution systems have been cited by sensorimotor enactivists to show that the human perceptual system is in a strong sense adaptive. The question with regard to synaesthesia is: why doesn't the extra synaesthetic experience adapt away like it would do in normal perceptual cases? Why can a visual experience that is unrelated to vision not be unlearned? In the first part of the paper I introduce the phenomenon of 'genuine perceptual synaesthesia'. I then, in the second part, sketch a theory of active perception that I want to endorse: enactivism. This is done by focussing on two basic assumptions underlying enactivism, and by defining two claims that 'sensorimotor enactivism' derives from these assumptions. These claims are: (a) the modality of a perceptual experience is constituted by the sensorimotor signature (i.e. the specific dependencies relating movements to stimulations) and the larger bodyinvolving cycle underlying this modality; and: (b) distorting elements get integrated and become transparent for the perceptual systems over a learning time span. As indicated above, these claims are challenged by cases of genuine synaesthesia. In part three, I discuss possible replies of the more narrowly defined sensorimotor enactivism to the first challenge and show that those replies either fail or betray important enactive insights. I will argue in particular that a promising way to meet the challenge (i.e. to claim that synaesthetic colours lack the properties of 'bodiliness' and 'grabbiness' of normal perceptual experience) fails. In part four, I suggest that enactivism, because it is unable to explain the perception-like experiences in cases of genuine perceptual synaesthesia, has to focus, instead, on typical realizers of perceptual experiences and on a more general enactivism in order to meet the two challenges. I show that this goes hand in hand with the inclusion of other adaptive time spans (in the course of which the perceptual system of an organism is shaped) in the explanations of phenomena like synaesthesia. In my view enactivism has not made this recourse to longer time spans as opposed to the 'here-and-now' explicit enough, though it is inherent to enactivism, even in the narrower, sensorimotor version of the theory. In order to explain and integrate certain atypical expressions of a perceptual mechanism – as I will argue in the last part of the paper – it is necessary to also take into view the embodiment of cognitive solutions shaped over evolutionary time spans and to adopt a heuristics and engineering perspective on such phenomena. This, I conclude, allows us to meet the challenges and hold on to central tenets of an enactive theory of perception.3 3 For a take on enactivism and synaesthesia that also embraces a version of enactivism see Dan Hutto's talk: "Understanding Synaesthesia, Radically Enactively", presented at the conference "Habitus in Habitat III: Synaesthesia and Kinaesthetics", 22nd October 2010. Sensorimotor Signature, Skill, and Synaesthesia 103 1 What is 'genuine synaesthesia' and why should it be accounted for in theories of perception? The prevalence of synaesthesia is nowadays assumed to be 1 in 23 of the normal population. The variety of types of synaesthesia is fascinating – 61 variants reported to date – and the growing public awareness of synaesthesia has led subjects to come forth and report more and more different forms, adding to the already known varieties. The graphem→colour synaesthesia is by far the most common type, a condition most likely shared by over 60% of all synaesthetes, and there is a high chance that a subject with one type of synaesthesia will also present another. In general, colour is the most common response, with different triggers ranging from sounds to even perceived personality. The response characteristics differ widely from synaesthete to synaesthete. There are reported cases of complex three-dimensional forms projected into space, or smells and felt temperatures as synaesthetic responses.4 In what follows, I will focus on two specific features of the synaesthetic condition: automaticity and consistency over time, which cover two central aspects of genuine synaesthesia. In genuine synaesthesia there is an experienced element that automatically triggers another sensory experience. The triggered sense modality (or aspect) is different from the triggering modality (or aspect). In other words: given a certain perceived or imagined triggering element, the response percept occurs involuntarily and cannot be suppressed. Consider again the case of graphem→colour synaesthesia. Once a synaesthetic subject is aware of a certain number or letter, she can't help but see at the same time (and in all cases) a specific colour or hue. It is important to notice that the automatic trigger is perceived and has to some extend to be consciously present itself. The trigger is not just another element in the causal chain that might, under normal conditions, lead to a specific experience, like retinal input or neuronal excitations in the pathways involved, leading to the activation of specific brain areas. It is a percept itself that elicits the cooccuring synaesthetic experience.5 Besides the automatic and involuntary response to a specific experienced trigger, consistency over time has become the defining element of genuine 4 For the numbers concerning the general prevalence see Simner et al.: „Non-random associations of graphemes to colors in synaesthetic and normal populations". For a good introduction to the phenomenon and an overview on the variety of synaesthesia and the prevalence of the different types see chap. 2 of Cytowic/Eagleman: Wednesday is Indigo Blue, pp. 23-62. Cf. also Dixon et al.: "Not All Synaesthetes Are Created Equal". 5 Cf. Mattingley: "Attention, Automaticity, and Awareness in Synesthesia" for a good discussion of this topic. The latter point – that an experienced percept or concept has to be present in order to elicit the response – has been long disputed and is not part of the definition of genuine perceptual synaesthesia in classical papers like e.g. Ramachandran/Hubbard: "Synaesthesia", and "The Phenomenology of Synaesthesia". 104 Joerg Fingerhut synaesthesia. It has been used to separate genuine synaesthesia from other forms like acquired or drug-induced synaesthetic experiences, and as such it has become part and parcel of the standardized behavioural test to verify synaesthesia. This test has shown that pairings of triggers and responses, for example the pairings of specific numbers with specific colours in synaesthetes, are persistent over time to a much higher degree compared to control groups exposed to those pairings during extensive associative learning phases. Consistency in synaesthetes has been proven in large test-retest scenarios by using refined computerized colour matching paradigms, and has now become the 'gold standard' for determining genuine synaesthesia.6 Both aforementioned criteria are behaviourally determined and partly also make use of subjective descriptions as hetero-phenomenological data. Given the variety of cases, no single neuronal mechanism or condition has yet been identified which sufficiently characterizes synaesthesia, but some attempts have been made to replace the behavioural criterion with a neurobiologically based one.7 Independently of such attempts, neural correlates have been identified for the respective modality or aspect of trigger and response percept (e.g. the ventral temporal lobe, including colour-selective area V4/V8, for colour as a response).8 It is contested whether synaesthesia constitutes a case of hyperconnectivity established by extra-connections between different brain areas, or a lack of inhibition of crosstalk between the regions with the same amount of connections. I won't go into this debate here, but it is of great interest for the study of consciousness, since synaesthesia constitutes a case where two hetero-phenomenologically determined conscious percepts can be looked into at the same time, while (a) searching for their neural correlates and (b) looking at cross-influences of the necessary treshold levels of activation underlying these conscious experiences. One can also expect to learn a lot about cross-modal interactions and attention – to just name a few of the issues – by using insights from research on synaesthesia. By contrast, I want to treat synaesthesia as a kind of limiting case for theories of perception, comparable to cases of imagination and hallucination. These cases nevertheless deeply shape our understanding of what perception is. A theory of perception has to explain how the world appears to us in the way it does – and the commonalities of perceptual experiences of the world and those in cases of hallucination or synaesthesia have to be accounted for. In subjects experiencing synaesthesia, something works fascinatingly different. The way I treat synaesthesia with respect to theories of perception is in this sense more akin to 6 The standardized 'test of genuineness' (TOG) was developed in 1987, see Baron-Cohen et al.: "Hearing Words and Seeing Colours". For a more recent version and contemporary refinements see Asher et al.: "Diagnosing and Phenotyping Visual Synaesthesia". 7 See e.g. Simner: "Defining Synaesthesia". 8 For an overview on recent evidence from brain imaging studies see Mattingley: "Attention, Automaticity, and Awareness in Synesthesia", pp. 158-161. Sensorimotor Signature, Skill, and Synaesthesia 105 how cognitive science might look into the phenomenon: synaesthesia as a small but interesting 'breakdown' of a cognitive or perceptual mechanism. I will briefly introduce why I think this is worthwhile for philosophy. I consider it to be a philosophical endeavour to treat perception in terms of biological solutions. These solutions can be addressed in terms of the demands a specific 'Umwelt' poses to the organism, as well as in terms of the biologically realized ways the organism meets these demands. In this sense, perception does not have to be treated as enabling an ideal mirroring of the environment, but rather as based on a system that works pretty well overall but sometimes produces errors, and these errors tell us something about the system. Perceptions in this sense are heuristics for action. Errors produced by using a heuristic are not random, but systematically biased and allow us to learn something about the workings of perception by understanding the design of the system underlying it.9 The design in question includes the morphological structure of the whole organism, the specific sense organs and means of locomotion, as well as the neural mechanisms – something I will discuss in the final sections of this paper. These elements constitute an important part of a full-fledged theory of perception that in the end also has to explain what structures and interactions underlie our conscious percepts and the ways the world shows up for us. But what about the phenomenology of experience itself? Is it not the commonality between the perceptual and synaesthetic colour experience that is the starting point for the topics discussed in this paper? For the context of what I am doing here I will consider the phenomenology as more or less unproblematically given. This sounds like more of a concession than it actually is. It does not mean that one has to avoid the question of what determines the content and quality of a conscious percept. All it means is that reports by subjects are a sufficient reason (if some refining conditions are met) to assume the presence of a phenomenal state of a specific kind or modality.10 I thereby dispense with the 'hard problem of consciousness' and confine myself to discussions on intermodal, comparative gaps of the following form: why does (neuronal, bodily, worldly) activity give rise to visual experiences rather than auditory ones?11 Here, as I will show, synaesthesia has something to add to our development of a biological theory of perception. 9 Cf. for the relation of heuristics and design: Wimsatt: Re-Engineering Philosophy for Limited Beings. 10 For this account of heterophenomenology cf. Dennett: Consciousness Explained, pp. 66-98. In my paper I focus on the modalities and qualities and not, as Dennett does, on the contents of perception. 11 See Hurley/Noë: "Neural Plasticity and Consciousness" for these and related comparative gaps. In my paper I sometimes also discuss what should be called intramodal gaps, since the most common und thus best studied type of synaesthesia is graphem→colour synaesthesia (called weak synaesthesia) in which the transfer is within one modality: vision. These findings are, for the relevant aspects of this paper, transferable to intermodal gaps e.g. of the hearing→colour type (also called strong synaesthesia). See Chalmers' The Conscious Mind for question related to the 'hard problem' and for a thorough treatment of the question of how the 106 Joerg Fingerhut 2 Enactivism and synaesthesia: the first challenge I take two assumptions to underlie enactive theories, each of which has specific enactivist claims as consequences that will be scrutinized and challenged in light of the insights gained in synaesthesia research. The first of these assumptions captures the general idea that any theory of perception has to start with the whole human organism and its involvement with the world. As a consequence – and this is an important follow-up to the former, rather uncontested claim – other parts besides the brain (i.e. the body and to some extent the world itself) will participate in a nontrivial way in explanations of what determines our perception. More will have to be said regarding the nontrivial ways in which these extracranial elements are involved, but it is important to see that it goes further than the content of experience being determined by what is 'out there'. The very machinery that realizes a specific conscious experience extends beyond elements that can be located in the brain.12 In this sense John Dewey had already argued against what he saw as ill-conceived dualisms between periphery and centre, between action and cognition, between the physical act of moving and the psychical idea: "The sensory quale gives the value of the act, just as the movement furnishes its mechanism and control, but both sensation and movement lie inside, not outside the act."13 The second assumption is that every life form is situated and embedded in a structured as well as destabilizing environment. Consequently every organism has to cope with an ongoing change between stable and precarious conditions. In the case of human beings these interactions not only have shaped the body, sensory system, and brain in the course of phylogenetic adaptation, such that their structures were selectively developed. Moreover, these changes are ongoing throughout the course of an ontogenetic lifespan. The human brain is extremely plastic and can alter its structure in a way that was only recently fully acknowledged in the cognitive neurosciences.14 Studies in neuroplasticity have been used to show that subjects with a loss or absence of the 'normal machinery' of perception in one modality (e.g. sight in congenitally blind people) can nevertheless experience this modality to some degree. They can do so just in case they are qualitative feel of mental life can or cannot be integrated into our theory of nature and what 'absolute gap' the acknowledgment of conscious experience might force upon such a theory. 12 See e.g. Dewey: "The Reflex Arch Concept in Psychology", pp. 358 f.: "Upon analysis, we find that we begin not with a sensory stimulus, but with a sensori-motor coordination, the optical-ocular, and that in a certain sense it is the movement which is primary, and the sensation which is secondary, the movement of body, head and eye muscles determining the quality of what is experienced. In other words, the real beginning is with the act of seeing; it is looking, and not a sensation of light." (My italics). 13 Ibid., p. 359. 14 For an accessible but nevertheless comprehensive overview of research done in the field of plasticity cf. Doidge: The Brain that Changes Itself. Sensorimotor Signature, Skill, and Synaesthesia 107 enabled to engage in the respective modality-specific interaction with the world – when they perform the relevant sensorimotor signature, e.g. learn to interact with distal objects in their environment. Cases of sensory substitution are used to prove this point: congenitally blind people equipped with a head-mounted camera connected to a vibration array on their skin learn to see through these devices. After a short period of interaction with the environment (e.g. after learning to grasp or to avoid an approaching object), they report experiencing not a sense of touch on the skin but something like sight or, to be more neutral, a distal sense.15 Specific local machinery and specific sensory pathways are not necessary for the occurrence of a modality-specific experience like vision, but the mastery of a specific kind of interaction with the world is. Based on the second assumption, I claim that the reliance on cases of brain plasticity makes it indispensable for enactive theories to extend the time-frames under consideration beyond the here-and-now over the time period of learning or adaptation. As I would argue, elements that exert their influence over a learning time span should be granted explanatory priority over the here-and-now. From my point of view, this has not been made explicit enough in the enactive literature. Maybe because this methodological claim does not directly attack or contradict statements of other theories, it has not been regarded as constitutive and definitive for the enactive approach, unlike claims based on the first assumption. This extension to longer time spans beyond the unfolding in the here-and-now is nevertheless central and used in arguments to vindicate the sensorimotor theory of perception and consciousness.16 In the remainder of this paper, I will tackle the two challenges that synaesthesia represents for an enactive theory of perception. Let's start with the first and postpone the second (lack of perceptual adaptivity) for now. In genuine synaesthesia, local triggers (percepts) elicit a visual experience without any world-engaging visual activity at its basis. For example, in cases of hearing→colour synaesthesia, the experience of a heard sound, or of the syllables of a spoken word, triggers a visual colour experience. In this case the subject does not engage in any sensorimotor contingencies related to vision. In order to understand why this might be considered a challenge, it is important to call to mind again one of sensorimotor enactivism's central tenets: intrinsic character of neural events does not provide the appropriate means to explain experiential quality; local properties of neuronal assemblies are not sufficient to explain the character of our experiences. They do play an important role, though, but one they can only play because they are recruited into world-engaging loops of the organism. Seeing is an activity, a process of active, exploratory engagement mediated by the possession and exercise of a certain body of knowledge concerning sensorimotor 15 Cf. the studies on visual sensory substitution with TVSS-devices conducted by Bach-y-Rita. See also, for the specific use of these studies that is relevant to the present paper, the treatment in O'Regan/Noë: "A Sensorimotor Account of Vision", pp. 957f. 16 See especially Hurley/Noë: "Neural Plasticity and Consciousness". 108 Joerg Fingerhut dependencies, dependencies that link potential actions (eye saccades, head movements) and their sensory consequences. In short: seeing is a skill and the mastery of this skill is necessary and sufficient for one or the other modality of conscious experience to occur. It is the exercised sensorimotor skill that determines the character and modality of what is experienced. We can now make the challenge explicit by showing that the following two claims cannot hold at the same time: (1) For any strong sensorimotor difference in a domain of interaction with the environment associated with a conscious experience, there must be a discriminable difference in the experience (intra/intermodal) (2) In genuine hearing→colour synaesthesia, subjects with no skilful interaction with the environment related to the visual response modality have a visual colour experience.17 Genuine synaesthesia is a condition in which, despite a strong sensorimotor difference and very different sensorimotor expectancies, a conscious visual experience is generated that is similar to a normal, veridical visual experience. The second claim contradicts the first because the same modality seems to be associated with two different functions: "Given [the sensorimotor] conceptual framework, there should be no case in which one quale is associated with two disparate functions. Yet this is exactly what happens in coloured-hearing synaesthesia. Colour qualia, affirmed by the synaesthete herself to be closely alike, occur in response to both heard words and seen coloured surfaces."18 Let us look at the second claim first and raise the question: is the visual experience of colour really the same in perceptual and synaesthetic experiences, or at least similar enough to call it a 'visual colour experience'? If not, the challenge would not hold. There are reliable hetero-phenomenological reports of colour identification and comparison to 'normal' colours in synaesthetes, i.e. they consistently identify colours of the synaesthetic and veridical variety. So the 'identification criterion' gives prima facie plausibility to the sameness claim. But do synaesthetic experiences have the same 'feel' as veridical ones? This question has been tackled and answered in two recent replies to a comparable challenge with verdicts against sufficient similarity, since, it has been argued, synaesthetic subjects would never confuse synaesthetic and veridical colour.19 On the other hand, synaesthetes do report that synaesthetic colours 'look' like those in normal perception. One might also suspect that synaesthetes can distinguish veridical from synaesthetic colours, due to the fact that they have learnt some additional 17 This is a version of an argument put forth by Gray et al.: "Evidence Against Functionalism From Neuroimaging of the Alien Colour Effect in Synaesthesia", Gray et al.: "Implications of Synaesthesia for Functionalism" and see esp. Gray: "How Are Qualia Coupled to Functions?" for a discussion of sensorimotor accounts. 18 Gray: "How Are Qualia Coupled to Functions?", p. 194. 19 See Hurley/Noë: "Can Hunter-Gatherers Hear Color?" pp. 70f. See also the criteria and negative verdict in Macpherson: "Synaesthesia, Functionalism and Phenomenology", pp. 73-77. Sensorimotor Signature, Skill, and Synaesthesia 109 facts about their condition. It has been reported that younger synaesthetes often are astonished to learn that not everybody is sharing their experiences because their perception of, say, every number '2' as red feels completely natural to them. And even after learning to separate more clearly what might be called the synaesthetic and the perceptual colour synaesthetes still experience disconcerting effects when, for example, they see a letter written in a colour conflicting with what is synaesthetically associated with this specific letter.20 The latter, in my view, suggests something like perceptual conflict in these cases and favours the claim of sufficient similarity between perceptual and synaesthetic colour experience.21 If this is so the first challenge to sensorimotor enactivism still holds: synaesthesia constitutes a case where sufficiently similar experiences occur despite strong diffences in sensorimotor signatures related to these experiences. 3 Can bodiliness and grabbiness of perception help to meet the first challenge? Two elements have not yet been emphasized in the debate, but might help to separate the experience of extra-colour in synaesthesia from normal experiences of colour in perception. These elements have the advantage of defining a rather general perceptual feature and hence not putting too much weight on otherwise marginal elements. As O'Regan and Noë make clear, there are additional phenomenological components included in their sensorimotor theory relating to the specific presence of perceptual experiences: 'bodiliness' and 'grabbiness'.22 'Bodiliness' emphasises the tight link to body motions: movement produces immediate changes in input, which are accounted for in perceptual experience. This can be seen as a kind of control condition; it reassures the organism that something is a 'normal' object. When we move the object's perceivable properties change and, for example, elements of the objects surface become vivid and present. The other element is 'grabbiness', which is based in an alerting capacity that allows transient elements in the environment to grab the subject's attention. Our visual system is contrived in such a way that sudden changes or movements in the environment are easily detected and the organism can become aware of them. 20 See Cytowic/Eagleman: Wednesday is Indigo Blue, p. 63. 21 This goes beyond Stroop-like inferences which can also occur when there is a conflict between the meaning of a written word and its colouring, e.g. a 'red' written in green ink. Here the conflict does not become conscious. See MacLeod/Dunbar: "Training and Stroop-Like Interference". 22 In response to the commentaries to their 2001 paper, they mark these two elements as their "most important clarification"; see Noë/O'Regan: "A Sensorimotor Account of Vision and Visual Consciousness", author's response, p. 1011. 110 Joerg Fingerhut Before I get to the comparison of those 'normal' features of perceptual experience with synaesthetic experiences, I should note a possible problem. Introducing those additional elements as part of the phenomenology of colour seems to be a circular move by the enactivist. It presupposes what it seeks to show. In our case, the enactivist seems to include bodily movements and behavioural criteria in the description of the phenomenology in order to prove that the nonbody-involving synaesthetic colour experience misses out in terms of this description. The circularity can be avoided if it can be shown that it is in fact a characteristic of all perceptual colour experiences of non-synaesthetes to be accompanied by such experiences of bodiliness and grabbiness, for example by collecting sufficient hetero-phenomenological data to support this claim. As it turns out, however, one can grant this move (to include bodiliness in the phenomenology of perception) to the enactivist, because it does not threaten the similarity claim after all: synaesthetic colours seem to be sufficiently similar to normally perceived colours, even with regard to bodiliness. In a review article, David Eagleman and Melvyn Goodale have collected data showing that other object-like features (including surface properties like texture) are also experienced in synaesthetic colour.23 These extra features have been used to account for the reported vividness of the synaesthetic colour experience. By attending thoroughly to subjective descriptions of colour-response synaesthetes, they have found references to texture even when this was not specifically asked for in the experimental paradigm. In conjunction with that, they report that larger regions of the medial ventral stream (that have been associated with the processing of texture properties)24 are activated in colour-synaesthesia. One can expect to get more results supporting this in the near future, for the simple reason that these areas have not yet been the main areas of interest for neuroimaging studies of colour-synaesthesia. One preliminary result is that even if one allows the enactivist to include bodiliness as part of the phenomenology of normal perception, she cannot rule out synaesthetic experiences on this basis. This is why synaesthesia is such a problematic case: one seems to get the feeling of bodiliness without interactions, or without any bodily activity involved. You get it for free, and this goes against the heart of the enactivist account. It seems doubtful that similar things can be shown for grabbiness. In this case it would be necessary for the defender of the challenge against enactivism to fall back on the first strategy of showing a circularity in the enactivist response to the challenge, namely that of stipulating a behavioural feature as part of the phenomenology. One would then also have to show that grabbiness does not accompany all visual experiences, and I do think that this is a possible way to go. Yet, again, one might not need to do so since some features of synaesthetic experience have also been associated with grabbiness. These features are pop-out phenom- 23 Eagleman/Goodale: "Why Color Synesthesia Involves More Than Color". 24 That is besides the traditional colour area V4, which recently has to be known to process texture information as well. Cf. Arcizet et al.: "Natural Textures Classification in Area V4". Sensorimotor Signature, Skill, and Synaesthesia 111 ena reported to occur in synaesthetes.25 Pop-out is an effect that enables subjects to easily pick out a target from an array of distractors when the target is constituted by different elements than the distractors. Synaesthetes are better than nonsynaesthetes at identifying a target (a triangle of 2s) among distractors (randomly arranged 5s) of the same colour [fig. 1a] when the target induced a different synaesthetic colour than the distractors [fig. 1a synaesthetically experienced as fig. 1b]. Here a salient feature of the environment seems to attract or'grab' the fig. 1a, 1b: the figure on the left shows how non-synaesthetes would experience the field, the figure on the right how a number→colour synaesthete might experience the same field. attention of the synaesthete. But these results are more than disputed: "As it turns out, despite initial excitement, synesthetes are generally not able to spot the hidden shape any better than nonsynesthetes. This result is important, because it demonstrates the stage at which synesthetic colours are perceived: the number must be attended to for it to be synesthetically coloured – it does not evoke colour before the viewer is conscious of the number's identity."26 It has been convincingly shown that number→colour synaesthetes only sometimes perform better in tasks of identifying targets [such as the triangle hidden in fig. 1a] because they 'anchor their search-pattern in the first oddball '2' they find (which appears red to them among the green 5s) as they go on searching for the other oddballs. In these cases the synaesthetic colour does not seem to be attention-grabbing in the first place. But even if some elements of the perceptual feature of grabbiness are missing, the case of bodiliness shows that compared to cases of mental imagery or cases of remembering synaesthetic experience has an enhanced vividness and forcible presence that is comparable to real perception. So even if the results are still tentative, the vividness that the enactivist account links to skill and active sensorimotor encounters with the environment is to some extent present in both perceived and synaesthetic colours. Yet, in synaesthesia no visual-like interaction with an object, no sensorimotor testing of conditions of the environment seems to be necessary to experience an enhanced visual vividness. In short: the narrow version of enactivism that I just sketched sisunable to explain why the vividness is present in both cases. 25 Cf. Ramachandran/Hubbard: "Synaesthesia". 26 Cf. Cytowic/Eagleman: Wednesday is Indigo Blue, p. 49; they cite Edquist et al.: "Do Synaesthetic Colours Act As Unique Features in Visual Search?". 112 Joerg Fingerhut 4 Normative enactivism and the second challenge from synaesthesia So, if one grants sufficient similarity in the experience, the phenomenon of hearing→colour synaesthesia contradicts the claim that characterizes enactivism and the problem persists: there is a strong sensorimotor difference but nevertheless the same look. But should enactivism hold such strong claims in the first place? Couldn't it give up the strong reading of claim (1) and allow for two signatures triggering one kind of experience? In this case there might not be one common but rather two different world-exploring roles realizing visual colour experience. This would be tantamount to giving synaesthetes their own 'extra' enactive signature (or a disjunctive signature underlying veridical and synaesthetic colour experiences). The first immediate problem one would have to face is that colourexperiences in hearing→colour synaesthetes do not seem to have environmentexploring functional descriptions at all: there seems to be no sensorimotor feedback-loop at work. Instead the experience is triggered automatically by a local percept or concept. So this move to weaken claim (1) is excluded for enactivists if they want to keep the actionist vein of the theory.27 This also holds for the suggestion to combine the visual experience of the hearing→colour synaesthetes with the auditory sensorimotor pattern and to say, that the disjunctive base for all visual colour experience is, besides the visual skill, an auditory skill as well. The option seems valid but is not very satisfying as it undermines the claim that modalities are determined in terms of mastery of a specific skilful interaction, that is to say by the specific sensorimotor signature underlying the modality. Consequently, this suggestion reinforces the local (neuronal) supervenience base theory against enactivism because this theory offers a much simpler explanation of why in both cases a visual experience arises: the local neuronal activation in V4. The more promising strategy for enactivism is to strengthen a normative element. This is to say that there is a typical realizer of a perceptual state, which is the larger sensorimotor cycle. Once done so, one can allow for exceptions when the normal conditions that are at work in typical occurrences are disturbed in one or another way. Like in hallucinations and misperceptions, something can go wrong. In cases of synaesthesia, this means that something as vivid and perceptual as in the veridical situation can be experienced. This is also a substantial weakening of claim (1) because now a strong difference in interaction (even the lack of such interactions) is allowed to generate a similar look in some cases. To make this weakening of the claim bearable, enactivism must now come up with 27 The same holds for focusing on the one core functional role played in both cases. In this case the underlying activation loop would for example include only V4 and adjacent brain area activity as a common core; see Macpherson: "Synaesthesia, Functionalism and Phenomenology", p. 73. But this works against the wide enactivism including action and body and would leave us with the local constitutionism enactivism wants to avoid. Sensorimotor Signature, Skill, and Synaesthesia 113 an explanation of what makes these cases less interesting or less binding for explanations of perceptual experience, and it has to give alternative reasons that speak in favour of the typical roles played in the cases of perception. Ultimately, or so I will argue, those reasons lie in the enactivist endorsement of a biological theory of the human organism as expressed in the two assumptions I introduced in part two. I'll come to those in a moment, but for the first challenge there is an argument for a typical role that does not have to refer to these rather general considerations. One possible way to go is to point out the necessity of engaging in the relevant sensorimotor interactions – of learning them – at least once during an ontogenetic lifespan. Explanations of how someone comes to experience a specific modality in the first place, as e.g. shown in the cases of sensory substitution systems, always involve the element of engaging in the relevant sensorimotor skill. This gets the specific experience running in these subjects, and without it the experience would not occur. This is also true for colour-synaesthetes: in early childhood they have at least to learn to engage visually. One explanation of what happens in cases of synaesthesia is that during childhood development, the intensive crosstalk of early infancy between different brain regions (for example between those that process sound and those that process visual information) stays vivid and intense, though it is meant to decrease during the development with the pruning of the connections between the regions.28 As a result colour experiences are elicited e.g. by activation of brain regions whose activity is related to sound or touch. Hence a normative element in enactivism can be maintained: in normal cases an experience occurs when a skill, a sensorimotor signature is learned. Skill explains even in these cases the onset of a modality of experience, something the biological organism is prepared for – it has the local elements ready to run and awaiting the skilful exposure – but which, lacking the relevant skilful engagement, might as well just not occur. In standard cases, this also explains why a certain experience persists or not. Enactivism would predict that the synaesthetic colour should become transparent or fade away – to use an admittedly odd metaphor – like the way glasses become transparent after a short time of wearing them, because we do not interact visually with them and their properties do not show up in the contents and characteristics of our experience. If sounds (like in hearing→colour synaesthesia) are sometimes bound to visual experiences, this is no problem at first since the theory just would predict that the visual experience would disappear as we do not interact visually with these sounds. But what defines synaesthesia is that these correlations do persist over the ontogenetic lifespan. This is the second challenge from synaesthesia: its persistence over time. This point has been raised in a paper against Hurley & Noë's view of enactive perception and their prediction that in the course of development and within 28 Cf. Maurer/Mondloch: "Neonatal Synesthesia". 114 Joerg Fingerhut a learning time scale, such activity should become integrated into normal activity and the perceptual colour effect should vanish. This should especially hold since the 'extra colours' in synaesthesia sometimes have confusing and cognitively interfering effects.29 In classical examples of perceptual adaptation and visually distorting prisms in non-synaesthetes, such elements do adapt away.30 So the question is, how can the theory explain both the 'adapting away' in these cases and the life-long persistence in cases of genuine synaesthesia? Hurley and Noë, in a response, strike sail and declare that one would have to "go beyond" the pure interactionist account and suggest not to expect that the character of experience will be explained "either just in terms of what happens in the brain, or just in terms of the active subject's relations to the world;" and conclude: "[t]he sensorimotor dynamics that govern experience are in principle distributed across brain, body, behaviour, and environment (though they can be so distributed to different degrees)."31 Here, again, it seems to me that it is necessary to point out a normative element, but one of a second-order: adaptivity. Many more cases and studies would have to (and can) be cited in favour of the profound adaptivity of the human perceptual systems to vindicate this point. But for the time being the examples of cognitive integration and the already mentioned sensory substitution cases have to suffice as support of the claim that adaptivity is the standard, and persistence without a world-engaging role the derivation. So adaptivity on this level is in itself a cognitive 'mechanism' that has a normal outcome but can be disturbed as well. Two short, but interesting notes on the side: some adaptivity, albeit to a much smaller degree, seems to be at work in synaesthetes as well: despite the strong consistency of the pairings in synaesthesia they do nevertheless change, although much more slowly.32 Furthermore it has been shown that under specific experimental conditions, something like synaesthetic consistency over time can also be induced in non-synaesthete subjects: in posthypnotic situations, subjects consciously experience black ink letters in a colour after having learned to pair 29 See Gray: "How Are Qualia Coupled to Functions?" He quotes studies of behavioural disadvantages of graphem-colour-synaesthetes because of Stroop-like inferences due to an 'alien colour effect'(ACE), in which synaesthetes e.g. see the written word 'green' as synaesthetically red and their response-speed for certain tasks is reduced. 30 Most famously cited are experiments by researchers like Stratton, Taylor, and Kohler with distorting goggles, where the subjects after short adaptation time learn to see 'normal' again. See the discussion in Hurley: Consciousness in Action, pp. 346-351. 31 Hurley/Noë: "Can Hunter-Gatherers Hear Color?" p. 80. One promising way to go beyond their account of sensorimotor enactivism, which I will not discuss in this paper, is to extend it towards a theory of the organism and the dynamics of the autopoietic systems it constitutes in order to explain the occurrence of certain kinds of experienes. This variant of enactivism follows Varela/Thompson/Rosch's The Embodied Mind. 32 See the case of Erica F. described in Cytowic/Eagleman: Wednesday is Indigo Blue, p. 237. Sensorimotor Signature, Skill, and Synaesthesia 115 them with the respective colour during hypnosis.33 These experiences are reported to persist and, if at all, only seem to slowly vanish. This might suggest that the pressure to adapt away is just not strong enough in colour-response synaesthesia (otherwise it would go away in the non-genuine hypnosis cases) and that the adaptation system might be just slowed down by a lot in these genuine cases. But even this in itself would not suffice to meet the challenge: the consistency and persistence in the genuine synaesthesia cases would be still so strong that we could not say enactivism accounts for it – especially, and this is decisive, since only the pattern of the one-to-one pairings (specific sound to specific colour) might change to some extent but the general intermodal pairing, e.g. of sound percepts with vision percepts, stays fully intact. 5 A heuristics and engineering perspective on perception What I have shown so far is that synaesthesia indeed poses a twofold challenge that cannot be easily discarded. Strengthening the normative claims in enactivism in response and allowing synaesthesia to be a derivation offers a way to meet the challenges – all the more so if one can show how enactivism could sustain the assumptions introduced earlier (the first being that perception is an activity of the whole organism, the second highlighting the adaptivity of organisms over different time spans) and nevertheless integrate synaesthesia into such a view. In concluding this paper I will sketch a way in which this could be done. Enactivists would never deny that neuronal elements are to a large degree involved in the processes that determine the character of an experience. But they claim that this is completely different from assuming that those internal features in principle are better suited to provide an explanation of why certain phenomenal features arise and persist, and thus why an experience is of this modality or another. In this respect references to bodyand world-involving dynamics provide more satisfying and richer explanations. Yet, questions concerning the metaphysics of consciousness have traditionally focused on local supervenience bases of mental states (as opposed to wider realizations that include the body and the environment of the organism) and they have focused on the here-and-now time scale – almost exclusively dealing with what realizes a mental state at those levels. Sensorimotor enactivism already takes a step in the right direction by taking into consideration the time scales of ontogenetic integration, and by claiming that the ability to interact with an environment is a necessary element in explaining why certain conscious experiences arise. It should thereby focus on a 33 Cf. the preliminary results in Cohen et al.: "Induced Cross-Modal Synaesthetic Experience Without Abnormal Neuronal Connections". 116 Joerg Fingerhut theory of constitution that includes as integral theoretical elements the interaction-based changes that are realized over such time spans. But Hurley and Noë have already suggested that in order to deal with the obstacles raised by phenomena such as synaesthesia, one might have to go beyond this kind of sensorimotor enactivism "to bring brain activity and the extended dynamics in which it is embedded within a unified explanatory framework."34 Although they did not specify how to do so, they did say that the aim would be to hold on to the insights of their dynamicism and at the same time to seriously consider the constraints of the brain and nervous system. The envisioned explanatory framework should be extended, I would argue, especially in terms of the time scales of the dynamics that shape these interactions and the underlying biological structures. As we have seen, sensorimotor enactivism highlights learning and ontogenetic adaptivity. But the way I introduced the general assumptions of enactivism, it also takes into account the organism-environment couplings that shape the organism (and the environmental niche) over larger time scales. These elements and the powers they exert on cognitive systems have been specifically acknowledged by theories of embodied cognition that include our 'contingent embodiment', and the 'tinkering' of nature at the level of evolutionary design, something that enactivism does not specifically account for. One then can highlight the different levels of adaptivity of organisms and the specific 'design' underlying the cognitive and perceptual solutions. Such accounts also go hand in hand with what I would call the view of perception as heuristics, because they acknowledge that heuristics are especially telling with regard to engineered solutions in physical and biological systems. Extended time scales for organismic perceptual designs in this perspective might include at least four different levels with specific properties and ways of exerting influence on cognitive solutions embodied in organisms:35 the evolutionary time scale (t3), the developmental time scale (t2), the learning time scale (t1), and the time scale of the here-and-now (t0). Adaptivity and design solutions of organisms on the first three time scales explain the behaviour and experience of subjects in the here-and-now. Back to perceptual systems in the human case: We saw that what allows the perceptual unfolding in the present is mediated by structures acquired on (t2) and (t1) especially, which are in the narrower focus of sensorimotor enactivism. What makes genuine synaesthesia an exceptional and challenging case relates in particular to the childhood development time scale, and the learning time scale. This is to say that in a 'critical period' in childhood, pruning and inhibition do not get underway, and since then perceptual adaptivity is impaired and normal perceptual learning mechanism are disturbed. Under normal circumstances, enactive 34 Hurley/Noë: "Can Hunter-Gatherers Hear Color?", p. 80. 35 Time scales for design and their specific properties have entered theories of embodied cognition esp. via the research and engineering done in robotics, cf. Pfeifer/Bongard: How the Body Shapes the Way We Think. Sensorimotor Signature, Skill, and Synaesthesia 117 explanations of underlying sensorimotor skills for perceptual experience should focus on these ontogenetic time scales. But in this special case and if we want to posit further normative elements as well as to include the specific implementations or embodiments of cognitive solutions, it might be necessary to defer to elements that are shaped in course of (t3). These theoretical elements taken together might offer a way to deal with both 'breakdowns' constituted by genuine perceptual synaesthesia: (a) the disturbance of the 'normal' relation of sensorimotor signature and modality of experience, and (b) the lack of 'normal' adaptivity. If we do understand perception in general as a heuristic for actions, and take up the engineering or 'contingent embodiment' perspective, it is possible to explain both of these as instances of breakdowns at two different levels of analysis and also explain what happens in synaesthesia without giving up the central claims of enactivism. Here it is important to see that this embodiment thesis includes body, environment, and, last but not least, the neuronal structures and mechanisms, all of which put constraints on the unfolding of a cognitive solution. Heuristics of cognitive solutions can go wrong, and heuristics are also realized in systems that can break down. Focusing on the way implementations work, but also on how they can go wrong, can bring brain activity (that leads to untypical experiential effects) into a larger framework. In order to give a hint at how the engineering and heuristics perspective might help to deal with a phenomenon like synaesthesia, I want to conclude with a genuine insight stemming from this perspective: One has to be careful not to confuse the part that causes the breakdown with the elements and dynamics that constitute the cognitive or perceptual solution. Neither (a) non-world-involving local activation nor (b) non-adaptive persistence over time figure prominently in the explanation of what constitutes a perceptual experience. Nor do they explain what determines the modality of such an experience. Both highlight elements that are causally responsible for the specific 'breakdown' of a perceptual solution. But the part responsible for this atypical expression, despite having a big effect, might not show up among the key features that explain the mental phenomenon in the first place. Conclusion In the light of the heuristics and engineering perspective, the insights of enactivism can be sustained even in the face of the challenges constituted by synaesthesia. Visual perceptual awareness or consciousness, although dependent on the brain, is not just a property of an event in the brain, and atypical functioning on this level does not contradict the underlying assumptions of enactivism and its basic claims concerning perceptual experience. But to be able to integrate these 118 Joerg Fingerhut atypical effects one has to take the level of implementation seriously and thereby also the role of neuronal expressions of perceptual solutions that are embedded in a larger supervenience base. Otherwise one might end up exaggerating the theoretical role of those phenomena. These elements do not by themselves provide the resources for a comprehensive theory of perceptual experience, though they should be given a place within the theory. References Arcizet, Fabrice/Christophe Jouffrais/Pascal Girard: "Natural Textures Classification in Area V4 of the Macaque Monkey", in: Experimental Brain Research, 189 (2008), pp. 109-120. Asher, Julian E./Michael R.F. Aitken/Nasr Farooqi et al.: "Diagnosing and Phenotyping Visual Synaesthesia: A Preliminary Evaluation of the Revised Test of Genuineness (TOG-R)", in: Cortex, 42 (2006) 2, pp. 137-46. Baron-Cohen, Simon/Maria A. Wyke/Colin Binnie: "Hearing Words and Seeing Colours: An Experimental Investigation of a Case of Synaesthesia", in: Perception, 16 (1987) 6, pp. 761-767. Chalmers, David J.: The Conscious Mind. In Search of a Fundamental Theory, New York (OUP) 1996. Cohen Kadosh, Roi/Avishai Henik et al.: "Induced Cross-Modal Synaesthetic Experience Without Abnormal Neuronal Connections", in: Psychological Science, 20 (2009) 2, pp. 258-265. Cytowic, Richard E./David Eagleman: Wednesday Is Indigo Blue: Discovering the Brain of Synesthesia, Cambridge, Mass. (MIT Press) 2009. Dennett, Daniel C.: Consciousness Explained, Boston (Little, Brown & Co) 1991. Dewey, John: "The Reflex Arc Concept in Psychology", in: Psychological Review, 3 (1896) 4, pp. 357-370. Dixon, Mike J./Daniel Smilek/Philip M. Merikle: "Not All Synaesthetes Are Created Equal. Projector Versus Associator Synaesthetes", in: Cognitive, Affective, & Behavioral Neuroscience, 4 (2004) 5, pp. 335-343. Doidge, Norman: The Brain That Changes Itself. Stories of Personal Triumph From the Frontiers of Brain Science, New York (Viking) 2007. Eagleman, David M./Melvyn A. Goodale: "Why Color Synesthesia Involves More Than Color", in: Trends in Cognitive Sciences, 13 (2009) 7, pp. 288292. Edquist, Jessica/Anina N. Rich/Cobie Brinkman et al.: "Do Synaesthetic Colours Act As Unique Features in Visual Search?", in: Cortex, 42 (2006) 2, pp. 222-231. Sensorimotor Signature, Skill, and Synaesthesia 119 Gray, Jeffrey A./David M. Parslow/Michael J. Brammer et al.: "Evidence Against Functionalism From Neuroimaging of the Alien Colour Effect in Synaesthesia", in: Cortex, 42 (2006) 2, pp. 309-318. Gray, Jeffrey A./Susan Chopping/Julia Nunn et al.: "Implications of Synaesthesia for Functionalism: Theory and Experiments", in: Journal of Consciousness Studies, 9 (2002) 12, pp. 5-31. Gray, Jeffrey A.: "How Are Qualia Coupled to Functions?", in: Trends in Cognitive Sciences, 7 (2003) 5, pp. 192-194. Hurley, Susan/Alva Noë: "Can Hunter-Gatherers Hear Color?", in: Brennan, Geoffrey/Robert Goodin/Frank Jackson et al. (eds.): Common Minds: Themes From the Philosophy of Philip Pettit, New York (OUP) 2007, pp. 55-83. Hurley, Susan/Alva Noë: "Neural Plasticity and Consciousness", in: Biology and Philosophy, 18 (2003) 1, pp. 131-168. Hurley, Susan: Consciousness in Action, Cambridge (HUP) 1998. Hutto, Daniel D.: "Understanding Synaesthesia, Radically Enactively". Talk at the conference "Habitus in Habitat III: Synaesthesia and Kinaesthetics", Berlin, 22nd October 2010 (http://itunes.apple.com/gb/podcast/philosophicalpsychology-2/id386952605?i=87011854). MacLeod, Colin M./Kevin Dunbar: "Training and Stroop-Like Interference: Evidence for a Continuum of Automaticity", in: Journal of Experimental Psychology: Learning, Memory, and Cognition, 14 (1988) 1, pp. 126-135. Macpherson, Fiona: "Synaesthesia, Functionalism and Phenomenology", in: Marraffa, Massimo/Mario De Caro/Francesco Ferrett (eds.): Cartographies of the Mind. Philosophy and Psychology in Intersection, Dordrecht (Springer) 2007, pp. 65-80. Mattingley, Jason B.: "Attention, Automaticity, and Awareness in Synesthesia", in: Annals of the New York Academy of Sciences, 1156 (2009), pp. 141-167. Maurer, Daphne/Catherine Mondloch: "Neonatal Synesthesia: A Re-Evaluation", in: Lynn C. Robertson/Noam Sagiv (eds.): Synesthesia: Perspectives From Cognitive Neuroscience, Oxford/New York (OUP) 2005, pp. 193-213. Myin, Eric/J. Kevin O'Regan: "Perceptual Consciousness, Access to Modality and Skill Theories. A Way to Naturalize Phenomenology?", in: Journal of Consciousness Studies, 9 (2002) 1, pp. 27-46. Noë, Alva: Action in Perception, Cambridge, Mass. (MIT Press) 2004. Noë, Alva/Susan Hurley: "The Deferential Brain in Action: Response to Jeffrey Gray", in: Trends in Cognitive Sciences, 7 (2003) 5, pp. 195-196. O'Regan, J. Kevin/Alva Noë: "A Sensorimotor Account of Vision and Visual Consciousness", in: The Behavioral and Brain Sciences, 24 (2001) 5, pp. 939-73; discussion pp. 973-1031. Pfeifer, Rolf/Josh C. Bongard: How the Body Shapes the Way We Think. A New View of Intelligence, Cambridge, Mass. (MIT Press) 2007. 120 Joerg Fingerhut Ramachandran, Vilayanur S./Edward M. Hubbard: "Synaesthesia A Window Into Perception, Thought and Language", in: Journal of Consciousness Studies, 8 (2001) 12, pp. 3-34. Ramachandran, Vilayanur S./Edward M. Hubbard: "The Phenomenology of Synaesthesia", in: Journal of Consc. Studies, 10 (2003) 8, pp. 49-57. Simner, Julia: "Defining Synaesthesia", in: British Journal of Psychology, (2011, forthcoming). Simner, Julia/Jamie Ward/Monika Lanz et al.: „Non-random associations of graphemes to colors in synaesthetic and normal populations." in: Cognitive Neuropsychology, 22 (2005) 8, pp. 1069–1085. Varela, Francisco J./Evan Thompson/Eleanor Rosch: The Embodied Mind: Cognitive Science and Human Experience, Cambridge, Mass. (MIT Press) 1991. Wimsatt, William C.: Re-Engineering Philosophy for Limited Beings. Piecewise Approximations to Reality, Cambridge, Mass. (HUP) 2007.