Abstract
Functional neuroimaging is sometimes criticized as showing only where in the brain things happen, not how they happen, and thus being unable to inform us about questions of mental and neural representation. Novel analytical methods increasingly make clear that imaging can give us access to constructs of interest to psychology. In this paper I argue that neuroimaging can give us an important, if limited, window into the large-scale structure of neural representation. I describe Representational Similarity Analysis, increasingly used in neuroimaging studies, and lay out desiderata for representations in general. In that context I discuss what RSA can and cannot tell us about neural representation. I compare RSA with fMRI to a different experimental paradigm which has been embraced as being indicative of representation in psychology, and argue that it compares favorably.
This is a preview of subscription content, log in via an institution to check access.
Source: This figure is reprinted with permission from Kriegeskorte and Kievit (2013)
Source: This figure originally appeared in Kriegeskorte et al. (2008a)
Similar content being viewed by others
Notes
MVPA, or multi-voxel pattern analysis, is an umbrella term used for all sorts of methods which involve analyzing fine-grained fMRI data. We use MVPC, or multi-voxel pattern classification, to indicate MVPA methods using classifiers (see Haxby et al. 2014).
Additional desiderata have been suggested for vehicles, at least for the vehicles of representations in a digital system: Clark notes that vehicles need to be “portable”, in that the same vehicle can play a role in different computations, and that they be “type-able”, or able to be classified into types. These desiderata are more contestable, and seem to be suited to specifically digital representations, as properties of representations that operate in a digital system. It seems to remain a possibility that analog representations, like pictures, be neither typeable or portable in the way they must be in a digital system, yet that they are legitimate representations, and perhaps even paradigmatic representations.
RSA yields results which are informative without being determinative: We can assess representational structure relative to a hypothetical model or relative to competing hypotheses, but cannot rule out other interpretations that share representational geometries.
Although the vehicle and content questions are logically separable, in practice they are interdependent. You cannot investigate content without identifying the vehicles that carry the content, but a way of identifying the vehicles in natural systems is to look for structures embodying content-relevant relationships in causal pathways that are candidates for representing.
Here we have evidence from anatomy that there are stages of processing hierarchically organized, and thus can infer that representations at one stage causally affect the next. But fMRI does not provide direct evidence of causal connection.
This presupposes that there is a readout that respects the representational geometry.
Egan, personal communication.
Interestingly, it was Shepard who was one of the first to apply representational geometry methods in psychology, using similarity as a way to characterize content (see Shepard 1987).
The Shepard data has often been interpreted as indicating that the relevant mental representations are image-like. But in the multidimensional framework discussed here, the RT data could be interpreted as being consistent with the dynamics of a movement of an activation vector through a space homologous with 3-D rotational space. In such a framework the clear distinction between image-like and digital or discursive tends to fall apart.
References
Buxton, R. B. (2009). Introduction to functional magnetic resonance imaging: principles and techniques (2nd ed.). Cambridge: Cambridge University Press.
Carpenter, P. A., & Just, M. A. (1978). Eye fixations during mental rotation. In J. W. Senders, D. F. Fisher, & R. A. Monty (Eds.), Eye Movements and the higher psychological functions (pp. 115–133). New Jersey: Erlbaum.
Charest, I., Kievit, R. A., Schmitz, T. W., Deca, D., & Kriegeskorte, N. (2014). Unique semantic space in the brain of each beholder predicts perceived similarity. Proceedings of the National Academy of Sciences, 111(40), 14565–14570. https://doi.org/10.1073/pnas.1402594111.
Clark, A. (1996). Being There. Cambridge: MIT Press.
Coltheart, M. (2006a). What has functional neuroimaging told us about the mind (so far)? Cortex, 42, 323–331.
Coltheart, M. (2006b). Perhaps functional neuroimaging has not told us anything about the mind (so far). Cortex, 42, 422–427.
Connolly, A. C., Guntupalli, J. S., Gors, J., Hanke, M., Halchenko, Y. O., Wu, Y.-C., et al. (2012). The representation of biological classes in the human brain. Journal of Neuroscience, 32(8), 2608–2618. https://doi.org/10.1523/JNEUROSCI.5547-11.2012.
Diedrichsen, J., & Kriegeskorte, N. (2017). Representational models: A common framework for understanding encoding, pattern-component, and representational-similarity analysis. PLOS Computational Biology, 13(4), e1005508. https://doi.org/10.1371/journal.pcbi.1005508.
Egan, F. (2010). Computational models: A modest role for content. Studies in History and Philosophy of Science Part A, 41(3), 253–259. https://doi.org/10.1016/j.shpsa.2010.07.009.
Egan, F. (2018). The nature and function of content in computational models. In M. Sprevak & M. Colombo (Eds.), The routledge handbook of the computational mind. Abingdon: Routledge.
Egan, F. (2020). A Deflationary Account of Mental Representation. In J. Smortchkova, K. Dołęga, & T. chlicht, T. (Eds.), What are Mental Representations? Oxford University Press.
Fodor, J. (1999). “Diary” London Review of Books, September 30. https://www.lrb.co.uk/the-paper/v21/n19/jerry-fodor/diary.
Godfrey-Smith, P. (2006). Mental representation, naturalism, and teleosemantics. In Teleosemantics: new philosophical essays (pp. 42–68). Oxford: Clarendon Press.
Guntupalli, J. S., Wheeler, K. G., & Gobbini, M. I. (2017). Disentangling the representation of identity from head view along the human face processing pathway. Cerebral Cortex (New York, N.Y.: 1991), 27(1), 46–53. https://doi.org/10.1093/cercor/bhw344.
Haxby, J. V., Connolly, A. C., & Guntupalli, J. S. (2014). Decoding neural representational spaces using multivariate pattern analysis. Annual Review of Neuroscience, 37(1), 435–456. https://doi.org/10.1146/annurev-neuro-062012-170325.
Haxby, J. V., Gobbini, M. I., Furey, M. L., Ishai, A., Schouten, J. L., & Pietrini, P. (2001). Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, 293(5539), 2425–2430. https://doi.org/10.1126/science.1063736.
Hubel, D. H., & Wiesel, T. N. (1959). Receptive fields of single neurones in the cat’s striate cortex. The Journal of Physiology, 148, 574–591. https://doi.org/10.1113/jphysiol.1959.sp006308.
Hubel, D. H., & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of Physiology, 160(1), 106–154. https://doi.org/10.1113/jphysiol.1962.sp006837.
Hubel, D. H., & Wiesel, T. N. (1998). Early exploration of the visual cortex. Neuron, 20(3), 401–412. https://doi.org/10.1016/S0896-6273(00)80984-8.
Just, M. A., & Carpenter, P. A. (1976). Eye fixations and cognitive processes. Cognitive Psychology, 8, 441–480.
Kamitani, Y., & Tong, F. (2005). Decoding the visual and subjective contents of the human brain. Nature Neuroscience, 8(5), 679–685. https://doi.org/10.1038/nn1444.
Khaligh-Razavi, S.-M., Henriksson, L., Kay, K., & Kriegeskorte, N. (2017). Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models. Journal of Mathematical Psychology, 76(Pt B), 184–197. https://doi.org/10.1016/j.jmp.2016.10.007.
Kiani, R., Esteky, H., Mirpour, K., & Tanaka, K. (2007). Object category structure in response patterns of neuronal population in monkey inferior temporal cortex. Journal of Neurophysiology, 97(6), 4296–4309. https://doi.org/10.1152/jn.00024.2007.
Kriegeskorte, N., & Bandettini, P. (2007). Combining the tools: Activation- and information-based fMRI analysis. NeuroImage, 38(4), 666–668. https://doi.org/10.1016/j.neuroimage.2007.06.030.
Kriegeskorte, N., & Diedrichsen, J. (2019). Peeling the onion of brain representations. Annual Review of Neuroscience, 42(1), 407–432. https://doi.org/10.1146/annurev-neuro-080317-061906.
Kriegeskorte, N., Goebel, R., & Bandettini, P. (2006). Information-based functional brain mapping. Proceedings of the National Academy of Sciences, 103(10), 3863–3868. https://doi.org/10.1073/pnas.0600244103.
Kriegeskorte, N., & Kievit, R. A. (2013). Representational geometry: Integrating cognition, computation, and the brain. Trends in Cognitive Sciences, 17(8), 401–412. https://doi.org/10.1016/j.tics.2013.06.007.
Kriegeskorte, N., Mur, M., & Bandettini, P. (2008a). Representational similarity analysis—connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2,. https://doi.org/10.3389/neuro.06.004.2008.
Kriegeskorte, N., Mur, M., Ruff, D. A., Kiani, R., Bodurka, J., Esteky, H., et al. (2008b). Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron, 60(6), 1126–1141. https://doi.org/10.1016/j.neuron.2008.10.043.
Naselaris, T., & Kay, K. N. (2015). Resolving ambiguities of MVPA using explicit models of representation. Trends in Cognitive Sciences, 19(10), 551–554. https://doi.org/10.1016/j.tics.2015.07.005.
Parvizi, J., Jacques, C., Foster, B. L., Withoft, N., Rangarajan, V., Weiner, K. S., & Grill-Spector, K. (2012). Electrical stimulation of human fusiform face-selective regions distorts face perception. Journal of Neuroscience, 32(43), 14915–14920. https://doi.org/10.1523/JNEUROSCI.2609-12.2012.
Pylyshyn, Z. W. (1973). What the Mind’s Eye Tells the Mind’s Brain: A Critique of Mental Imagery. Psychological Bulletin, 80, 1–25.
Ramsey, W. M. (2007). Representation reconsidered. Cambridge: Cambridge University Press.
Ritchie, J. B., Kaplan, D. M., & Klein, C. (2019). Decoding the brain: neural representation and the limits of multivariate pattern analysis in cognitive neuroscience. The British Journal for the Philosophy of Science, 70(2), 581–607. https://doi.org/10.1093/bjps/axx023.
Roskies, A. L. (2008). Neuroimaging and inferential distance. Neuroethics, 1(1), 19–30. https://doi.org/10.1007/s12152-007-9003-3.
Roskies, A. L. (2009). Brain-mind and structure-function relationships: A methodological response to Coltheart. Philosophy of Science, 76(5), 927–939.
Schalk, G., Kapeller, C., Guger, C., Ogawa, H., Hiroshima, S., Lafer-Sousa, R., et al. (2017). Facephenes and rainbows: Causal evidence for functional and anatomical specificity of face and color processing in the human brain. Proceedings of the National Academy of Sciences, 114(46), 12285–12290. https://doi.org/10.1073/pnas.1713447114.
Shea, N. (2013). Naturalising representational content. Philosophy Compass, 8(5), 496–509. https://doi.org/10.1111/phc3.12033.
Shea, N. (2018). Representation in cognitive science. Oxford: Oxford University Press.
Shea, N., Godfrey-Smith, P., & Cao, R. (2018). Content in simple signalling systems. The British Journal for the Philosophy of Science, 69(4), 1009–1035. https://doi.org/10.1093/bjps/axw036.
Shepard, R. N. (1987). Toward a universal law of generalization for psychological science. Science, 237(4820), 1317–1323. https://doi.org/10.1126/science.3629243.
Shepard, R. N., & Cooper, L. A. (1982). Mental images and their transformations. Cambridge: MIT Press.
Shepard, R. N., & Metzler, J. (1971). Mental rotation of three-dimensional objects. Science, 171(3972), 701–703. https://doi.org/10.1126/science.171.3972.701.
Tong, F., Harrison, S. A., Dewey, J. A., & Kamitani, Y. (2012). Relationship between BOLD amplitude and pattern classification of orientation-selective activity in the human visual cortex. NeuroImage, 63(3), 1212–1222. https://doi.org/10.1016/j.neuroimage.2012.08.005.
Yamins, D. L. K., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014). Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences of the United States of America, 111(23), 8619–8624. https://doi.org/10.1073/pnas.1403112111.
Yildirim, I., Wu, J., Kanwisher, N., & Tenenbaum, J. (2019). An integrative computational architecture for object-driven cortex. Current Opinion in Neurobiology, 55, 73–81. https://doi.org/10.1016/j.conb.2019.01.010.
Acknowledgements
This paper has benefitted from feedback from the fellows at the University of Pittsburgh Center for Philosophy of Science, from Jim Haxby, and from the participants of Neural Mechanisms Online, especially Charles Rathkopf, Dan Weiskopf, Matteo Grasso, and Michael Anderson. The work was supported in part by a fellowship from University of Pittsburgh Center for Philosophy of Science.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Roskies, A.L. Representational similarity analysis in neuroimaging: proxy vehicles and provisional representations. Synthese 199, 5917–5935 (2021). https://doi.org/10.1007/s11229-021-03052-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11229-021-03052-4