This is an excerpt from a report on the Perceptual Learning and Perceptual Recognition II Workshop at the University of Toronto, Mississauga in May of 2012, written by Kevin Connolly, John Donaldson, David M. Gray, Emily McWilliams, Sofia Ortiz-Hinojosa, and David Suarez, and available at http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html 3. Does Our Representation of Time Provide an Amodal Framework for Multi-Sensory Integration? A previous workshop discussed the idea that amodal representations of space and time could provide a framework for integrating information from different sensory modalities (see http://networksensoryresearch.utoronto.ca/Brown-Q4.html). The current workshop discussed evidence from experimental psychology suggesting that temporal features are used to coordinate information from proprioception and vision. This evidence lends some support to a temporal framework hypothesis, which claims that our representation of time serves as an amodal framework for multi-sensory integration. There are some obstacles to drawing such a conclusion, however. Even if temporal features are used for multi-sensory integration, these features might not be represented amodally, since each modality might encode temporal features differently. In her talk, Cecilia Heyes discussed how we are better at visually recognizing our own bodily movements than at recognizing the movements of friends, even when those movements are reduced to point-lights and shown from a third-person point of view (Prasad and Shiffrar, 2009). Visual experience seems to be insufficient to explain this advantage because we typically have far more experience viewing others than we do viewing ourselves. What then explains the visual self-recognition advantage? In a series of experiments testing visual recognition of recorded facial movements (displayed by means of anonymized computer avatars), Cook, Johnston, and Heyes found that although manipulating the spatial orientation of facial movements impeded the recognition of friends' movements, it had little effect on selfrecognition (2012). By contrast, manipulating the temporal features of recorded movements 2 impeded self-recognition far more than manipulation of spatial features. For instance, changes in timing and rhythm affected visual self-recognition far more than changes in orientation and topographical configuration. Since by itself visual experience seems insufficient to account for the visual self-recognition advantage, it is plausible to suppose that the advantage depends instead on a transfer of information across different sensory modalities. One reasonable hypothesis is that information derived from first-person proprioception is deployed in the visual self-recognition of bodily movements. Cook, Johnston, and Heyes' results suggest that it may be temporal information in particular which enables the requisite coordination of representations across different modalities. While it would be overly hasty to conclude that these results confirm the temporal framework hypothesis, nevertheless, if temporally-based integration turned out to be very prevalent, the hypothesis might be more plausible since the existence of such a framework would help to explain a great many cases of sensory integration. Most troubling for supporters of the temporal framework hypothesis, however, is the possibility that there is a 'Molyneux problem' for temporal features. That is, it might be the case that each modality encodes temporal features differently (as suggested by Barry Smith), and that the correspondence of temporal features across different modalities must be learned through the association of modality-specific representations. If the functioning of unimodal perceptual mechanisms turned out to underlie temporally-based sensory integration, this would make the temporal framework hypothesis less plausible, since it would reduce the need for an amodal representational framework. References: Cook, R., Johnston, A., and Heyes, C. (2012). "Self-Recognition of Avatar Motion: How do I know it's me?" Proceedings of the Royal Society B: Biological Sciences 279 (1729) (February 22): 669–674. doi:10.1098/rspb.2011.1264. 3 Prasad, S., and Shiffrar, M. (2009). "Viewpoint and the recognition of people from their movements." Journal of Experimental Psychology: Human Perception and Performance 35 (1) (February 1): 39–49.