Abstract
According to Thagard and Stewart (Cogn Sci 35(1):1–33, 2011), creativity results from the combination of neural representations (an idea which Thagard calls ‘the combinatorial conjecture’), and combination results from convolution, an operation on vectors defined in the holographic reduced representation (HRR) framework (Plate, Holographic reduced representation: distributed representation for cognitive structures, 2003). They use these ideas to understand creativity as it occurs in many domains, and in particular in science. We argue that, because of its algebraic properties, convolution alone is ill-suited to the role proposed by Thagard and Stewart. The semantic pointer concept (Eliasmith, How to build a brain, 2013) allows us to see how we can apply the full range of HRR operations while keeping the modal representations so central to Thagard and Stewart’s theory. By adding another combination operation and using semantic pointers as the combinatorial basis, this modified version overcomes the limitations of the original theory and perhaps helps us explain aspects of creativity not covered by the original theory. While a priori reasons cast doubts on the use of HRR operations with modal representations (Fisher et al., Appl Opt 26(23):5039–5054, 1987) such as semantic pointers, recent models point in the other direction, allowing us to be optimistic about the success of the revised version.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The combinatorial conjecture would not be trivial if it claimed that creativity in science and technology involves the creation of new concepts generated by combinations of old ones, but the conjecture, in all its generality, applies only to propositions. Since it is obvious that new sentences are associated with scientific discoveries and technological innovations, and since (as long as one is representationalist) mental representation systematically related to these sentences (i.e., propositions) must be involved, the charge of triviality is serious.
Another reading in which the thesis would not be trivial is the idea that only combinations can be creative (discussed in Thagard 2012, p. 153), while abstraction would not (or would be simply a form of combinations). However, Thagard’s focus on propositions instead of concepts again seems to block this reading: New concepts could be formed by another process, but the conjecture states that creativity is the result of combining concepts (and percepts, etc.) into new propositions so this would not be a counterexample (but we will come back on this topic at the end of the paper).
They write: “We are primarily interested in creativity as a mental process, but we cannot neglect the fact that it also often involves interaction with the world (...) External representations such as diagrams and equations on paper can (...) be useful in creative activities, as long as they interface with the internal mental representations that enables people to interact with the world” (Thagard 2012, p. 112) Such a recognition is typical of Thagard’s general treatment of extended mind situations: the strategy is to view representations and computations as primarily neural, and to treat the remaining extended mind computations as a second order term in the approximation, to be handled eventually. As the mechanism for combination is only defined for mental representations, it is clear that the success of this strategy will depend on the extent to which the combination of representations can occur outside the brain via the manipulation of external representations – and if, as Thagard himself writes, “[s]cientific cognition is increasingly distributed not only among different researchers but also among researchers and computers with which they interact” (Thagard 2012, p. 190), this strategy may be too restrictive. We won’t pursue this question here though.
This analogy was noted by Plate (2003) and Eliasmith (2013). If a more rigorous algebraic characterization is wanted, Plate offers this one: “A finite-dimensional vector space over the real numbers, with circular convolution as multiplication and the usual definitions of scalar multiplication and vector addition, forms a commutative linear algebraic system” (Plate 2003, p. 114).
Rate coding is a complex issue in the NEF, since it is used as a means to understand and create population coding; see the discussion regarding time versus rate coding on p. 89–91 of Eliasmith and Anderson (2002), and more generally Chap. 2, on population coding and Chap. 5, on population-temporal coding.
Whether this account applies well to more complex cases, for instance to representations involved in imagining, expecting or remembering, to abstract properties (e.g. justice), to desires or to motor commands, that are not responses to physical stimuli that are currently present in the environment (or even have been or will be), is in part an open question. For motor commands, see the discussion in Eliasmith (2013, Sects. 3.6 and 3.7). Ideas that might help understand how this notion of representation could be used in accounting intentions can be found in Schröder et al. (2013). Eliasmith and Anderson (2002, p. 7) speculate that more abstract properties could be explained in terms of transformations of physical properties.
The set of representations recognized by the clean-up memory is called ‘the lexicon’.
This problem was noted by Plate (2003, p. 108). We will examine Plate’s remarks in more detail when we consider possible solutions to the problem.
One might think that taking concept-role pairs as basic would solve the present problem with convolution. However, this problem is a close parent of a problem with associationism with which Fodor and Pylyshyn deal in their classic (1988) paper on structured representations, and they demonstrated that taking concept-role pairs as basic would only lead to a combinatorial explosion in basic representations.
It is not even clear that convolution solves the first binding problem, since, if the only combination mechanism we have is convolution, there doesn’t seem to be a way two distinguish ‘there is a red square’ from ‘there is something red and there is something square.’ Without also bringing superposition into the toolkit (red+square vs. red*square), there isn’t any obvious manner of expressing these two different thoughts in a way that allows them to be distinguished.
This is why the short discussion of “selective convolution” (p. 133) which would operate on “slots and values” like combination in Thagard’s (1988) symbolic model may not suffice as a solution to the problem. This new form of convolution (left for “future research”) would have to accommodate modal representation, and it is not obvious how this could be done at the moment. To be fair, the solution we propose suffers from the same problem (see Note 22 for a brief discussion).
A few examples. (1) “Then creative thinking is a matter of combining neural patterns into ones that are both novel and useful. We advocate the hypothesis that such combinations arise from mechanisms that bind together neural patterns by a process of convolution rather than synchronization [...]” (Thagard 2012, p. 107). (2) Thesis 4 is “4. Neural representations are combined by convolution, a kind of twisting together of existing representations” (p. 108). (3) Also, in Chap. 9 of his (2012 book, Thagard writes: “Chap. 8 used convolution in neural networks to provide a theoretical perspective on creative conceptual combination by means of a new neurocomputational mechanistic explanation of how spiking neural representations can be combined. [...] the combinatorial conjecture can be fleshed out as follows: All creativity results from convolution-based combination of mental representations consisting of patterns of firing in neural populations” (pp. 141–142). To these various quotes we must add the fact that their model of the Aha! Experience only uses convolution and the fact that the convolution mechanism serves to immunize the conjecture from triviality. One cannot be faulted for reading in all of this the proposal that convolution is the sole mechanism for combination.
Plate also adds: “The ambiguity is greatly alleviated by using frame-specific role vector rather than generic role vectors (e.g., a generic agent vector.) A situation where ambiguity can still arise is when two instantiations of the same frame are nested in another instantiation of that same frame; e.g.: ’(A causes B) causes (C causes D)’ [...] Thus the HRR encoding for ’(A causes B) causes (C causes D)’ is identical to the HRR encoding of ‘(A causes C) causes (B causes D)’. Whether this would cause problems with the type of knowledge structures encountered in real-world information processing tasks (by humans or machines) is unclear—it is difficult to conceive of realistic examples that lead to complete ambiguity as in the above example” (Plate 2003, p. 108) This shows that even a role/filler structure, as examined below, is not immune to the problem generated by commutativity, but since they are less severe, we ignore them in this context.
While performance on fluid intelligence tests seems at first glance to make use of creativity, the overall correlation between creativity and IQ is quite low (Kim 2005) though for higher IQ levels the correlation is stronger (Sligh et al. 2005). The test is a version of Raven Progressive Matrices. We will come back to this test and its possible link to creativity later in the paper.
For Eliasmith’s own characterization, see Eliasmith (2013, Chap. 3), in particular the section “The semantic pointer hypothesis” (3.1) and “What is a semantic pointer” (3.2). Deep and shallow semantics are also explained in fuller details later in this chapter.
Some limitations must be mentioned. See Note 16 for one such limitation. Plate mentioned another: “It turns out that the most difficult thing to do is to leave something untransformed. [...] The only way to solve the problem is to do transformations separately, and clean up intermediate results.” (Plate 2003, pp. 223–224). Consider for instance the problem of transforming a*term1 + causes*relation + b*term2 into b*term1 + causes*relation + a*term2. Such a transformation must leave causes*relation untransformed, and is thus subject to the problem mentioned by Plate.
We do not want to leave the impression that we claim that amodal pointers are somewhat illegitimate. As Mahon and Caramazza (2008) argue, there is still a long way to go before proving that modality is a feature of all concepts.
The problem of the origin of role pointers resurfaces here. “Embodied” models of abstract concepts, or systems such as BEAGLE where roles emerge through order and meaning information and a placeholder scheme could be good places to search for solutions. Alternatively, role pointers could be amodal, perhaps even innate. But we won’t try to argue for or against the plausibility of such “role pointers” here, and continue as if the problem can be solved, since the semantic pointer architecture solves most of the other problems (detailed in Sect. 5) that Thagard and Stewart’s proposition faces.
Such HRRs are sometimes called “random”, and we will follow this usage here, but it must be noted that the term only qualifies the process by which they are generated. Once they are generated, these “random” HRRs remain the same throughout the simulation: They are constant vectors, not random ones.
They write: “The information encoding subsystem maps semantic pointers generated from images to conceptual semantic pointers encoding progression relations. This information encoding is used for most tasks except for the copy drawing task, where it is the visual features of the input, and not its conceptual properties, that matter for successful completion of the task” (Eliasmith et al. 2012, Supplementary Material, p. 14).
The dot product between two (unit) vectors is a measure of their similarity. If we picture the unit vectors with their origins at the center of a unit n-sphere and their end points on its surface, the dot product of two vectors will be greater the closer their end points are on the sphere, because the dot product of two unit vectors is just their cosine. See Eliasmith (2013, pp. 79–80).
Since incompressibility and randomness are closely related concepts in computer science, it is to be expected that any compression method will somewhat alleviate the problems of dependence between components.
We use the word “pointers”, plural, because different (but similar) compressed representations correspond to different tokens of the same digit, since these are written digits and they are written in different ways. Whether these compressed representations really qualify as different pointers would depend on their role in the model; if they were used for conceptual processing, they would probably correspond to a single item in the lexicon so any difference between them would be expected to be cleaned up during processing, but in Spaun, these pointers are used in the drawing task and these differences matter.
Sadly, in discussions of modal and amodal representations, it is not always as clear as it should be what the modal/amodal distinction amounts to. Obviously, one could define amodality in a way that makes the absence of amodal representations trivially true. As Markman and Stilwell (2004) observe, the organization of the brain makes a distinction between amodality and multimodality difficult to sustain, contrary to what neo-empiricists such as Prinz (2002) imply; no neuron in the brain is totally without causal links from sensory inputs or to motor output, so defining amodality as isolation from the periphery would make the absence of amodal representations true, but trivially so, and defining it in any other way seems to lead to the conclusion that there is no principled difference between amodality and multimodality.
An influential meta-analysis of semantic processing in the brain presents the distinction in the following way: “A useful qualitative distinction can still be drawn [...] between ‘modal’ cortex, where processing reflects a dominant sensory or motor modality, and ‘amodal’ cortex, where input from multiple modalities is more nearly balanced and highly convergent” (Binder et al. 2009, p. 2774). It is difficult to say whether the random pointers used in Spaun should be considered modal on this account, since one would have to ascertain the relative strengths of the causal links between these random pointers and, on one side, perceptual stimuli and, on the other, motor outputs. It is our opinion that random pointers would reflect an equilibrium between the two, but only a careful analysis of the simulation program would tell.
Alternatively, one could argue that the internal structure of modal and multimodal representations should mirror in some way the structure of the sensory input, and that the relation of a modal pointer to what it points to should be non-arbitrary. This is not so with symbols, and this is also not the case with random pointers (e.g., if the random pointer p represents the set of things P and the random pointer q represents the set of things Q, and elements of P and Q are perceptually more similar to each other than to elements of a set R represented by a random pointer r, then p and q will not in general be more similar to each other than to r). This kind of structural distinction between modal and amodal pointers may be more adequate than a purely causal one, since pointers are “more mobile” (Eliasmith 2013, p. 358) than representations in traditional neural network models. They can move around the system and are not tied to a particular neural population, making it more difficult to ascertain whether there is an equilibrium of influences from different modalities. But defining more precisely (to say nothing of defining quantitatively) what this structural relation amounts to is tricky.
References
Anonymous (2014). Binding Problem. Wikipedia.org. Online. Consulted on January 31, 2014. http://en.wikipedia.org/wiki/Binding_problem
Binder, J. R., & Desai, R. H. (2011). The neurobiology of semantic memory. Trends in Cognitive Sciences, 15(11), 527–536.
Binder, J. R., Desai, R. H., Graves, W. W., & Conant, L. L. (2009). Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. Cerebral Cortex, 19(12), 2767–2796.
Churchland, P. (1992). A neurocomputational perspective. Cambridge, MA: MIT Press.
De Pasquale, J.-F., & Poirier, P. (2014). Can holographic reduced representations operations be used with modal representations? An experiment. UQAM, Laboratoire d’analyse cognitive de l’information, Montréal, Cahiers du Lanci, 17 p.
Eliasmith, C. (2004). Learning context sensitive logical inference in a neurobiological simulation. In S. Levy & R. Gayler (Eds.), Compositional connectionism in cognitive science (pp. 17–20). AAAI Fall Symposium. Menlo Park: AAAI Press.
Eliasmith, C. (2005). Cognition with neurons: A large-scale, biologically realistic model of the Wason task. In G. Bara, L. Barsalou, & M. Bucciarelli (Eds.), Proceedings of the 27th Annual Meeting of the Cognitive Science Society. Italy: Stresa.
Eliasmith, C. (2013). How to build a brain. Oxford: Oxford University Press.
Eliasmith, C. E., & Anderson, C. H. (2002). Neural engineering. Computation, representation, and dynamics in neurobiological systems. Cambridge, MA: MIT Press.
Eliasmith, C., Stewart, T. C., Choo, X., Bekolay, T., DeWolf, T., Tang, Y., et al. (2012). A large-scale model of the functioning brain. Science, 338(6111), 1202–1205.
Fisher, A. D., Lippincott, W. L., & Lee, J. N. (1987). Optical implementations of associative networks with versatile adaptive learning capabilities. Applied Optics, 26(23), 5039–5054.
Fodor, J. A., & Pylyshyn, Z. W. (1988). Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1), 3–71.
Gayler, R. W. (2003). Vector Symbolic Architectures answer Jackendoff’s challenges for cognitive neuroscience. In P. Slezak (Ed.), ICCS /ASCS International Conference on Cognitive Science (pp. 133–138). Sydney, Australia: University of New South Wales.
Giere, R. (2002). Scientific cognition as distributed cognition. In P. Carruthers, S. Stich, & M. Siegal (Eds.), The cognitive basis of science. Cambridge: Cambridge University Press.
Gopnik, A. (2000). Explanation as orgasm, and the drive for causal understanding: The evolution, function and phenomenology of the theory-formation system. In F. Keil & R. Wilson (Eds.), Cognition and explanation (pp. 299–323). Cambridge, MA: MIT Press.
Gross, J. J., & Barrett, L. F. (2011). Emotion generation and emotion regulation: One or two depends on your point of view. Emotion Review, 3, 8–16.
Halliday, D., Resnick, R., & Walker, J. (2004). Fundamentals of physics. New York: Wiley.
Hickok, G. (2014). The myth of mirror neurons: The real neuroscience of communication and cognition. New York: WW Norton & Company.
Hinton, G. (1990). Mapping part-whole hierarchies into connectionist networks. Artificial Intelligence, 46, 47–75.
Hinton, G. E. (2009). Deep belief networks. Scholarpedia, 4(5), 5947.
Hinton, G. E., Osindero, S., & Teh, Y. W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18(7), 1527–1554.
Hunsberger, E., Blouw, P., Bergstra, J., & Eliasmith, C. (2013). A neural model of human image categorization. In 35th Annual Conference of the Cognitive Science Society, pp. 633-638.
Jones, M. N., & Mewhort, D. J. (2007). Representing word meaning and order information in a composite holographic lexicon. Psychological Review, 114(1), 1–37.
Kelly, M. A. (2010). Advancing the theory and utility of holographic reduced representations. Master Thesis, Kingston, ON: Queen’s University.
Kelly, M. A., Blostein, D., & Mewhort, D. J. K. (2013). Encoding structure in holographic reduced representations. Canadian Journal of Experimental Psychology/Revue Canadienne de Psychologie Expérimentale, 67(2), 79.
Kim, K. H. (2005). Can only intelligent people be creative? A meta-analysis. Prufrock Journal, 16(2–3), 57–66.
Levy, S. D. (2007). Changing semantic role representations with holographic memory. In Computational approaches to representation change during learning and development: Papers from the 2007 AAAI Symposium (pp. 40–45). Washington, DC: AAAI Press.
Levy, S. D., & Gayler, R. W. (2008). Vector symbolic architectures: A new building material for artificial general intelligence. Proceedings of the First Conference on Artificial General Intelligence (AGI-08). IOS Press, pp. 414–418.
Mahon, B. Z., & Caramazza, A. (2008). A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content. Journal of Physiology, Paris, 102(2008), 59–70.
Markman, A., & Stilwell, H. (2004). Concepts á la modal: An extended review of Prinz’s Furnishing the mind. Philosophical Psychology, 17(3), 391–401.
Milner, P. M. (1974). A model for visual shape recognition. Psychological Review, 81(6), 521–535.
Neumann, J. (2001). PhD Thesis. Holistic Processing of Hierarchical Structures in Connectionist Networks. Edinburgh, UK: University of Edinburgh.
Plate, T. A. (2003). Holographic reduced representation: Distributed representation for cognitive structures. Stanford: CSLI Publications.
Prinz, J. J. (2002). Furnishing the mind. Cambridge, MA: MIT press.
Rasmussen, D., & Eliasmith, C. (2011). A neural model of rule generation in inductive reasoning. Topics in Cognitive Science, 3(1), 140–153.
Revonsuo, A. (1999). Binding and the phenomenal unity of consciousness. Consciousness and Cognition, 8(2), 173–185.
Russell, J. A. (2009). Emotion, core affect, and psychological construction. Cognition and Emotion, 23, 1259–1283.
Rutledge-Taylor, M. F., Kelly, M. A., West, R. L., & Pyke, A. A. (2014). Dynamically structured holographic memory. Biologically Inspired Cognitive Architectures, 9, 9–32.
Sander, D., Grandjean, D., & Scherer, K. R. (2005). A systems approach to appraisal mechanisms in emotion. Neural Networks, 18(4), 317–352.
Schröder, T., Stewart, T. C., & Thagard, P. (2013). Intention, emotion, and action: A neural theory based on semantic pointers. Cognitive Science (published online November 13, 2013).
Schröder, T., & Thagard, P. (2013). The affective meanings of automatic social behaviors: Three mechanisms that explain priming. Psychological Review, 120(1), 255–280.
Shadlen, M. N., & Movshon, J. A. (1999). Synchrony unbound: A critical evaluation of the temporal binding hypothesis. Neuron, 24, 67–77.
Sligh, A. C., Conners, F. A., & Roskos-Ewoldsen, Beverly. (2005). Relation of creativity to fluid and crystallized intelligence. The Journal of Creative Behavior, 39(2), 123–136.
Smolensky, P. (1990). Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial Intelligence, 46, 159–216.
Smythies, J. R. (1994). The walls of Plato’s cave. Aldershot: Avebury.
Stewart, T., & Eliasmith, C. (2012). Compositionality and biologically plausible models. In W. Hinzen, E. Machery, & M. Werning (Eds.), Oxford handbook of compositionality. Oxford: Oxford University Press.
Stewart, T. C., Tang, Y., & Eliasmith, C. (2011). A biologically realistic cleanup memory: Autoassociation in spiking neurons. Cognitive Systems Research, 12(2), 84–92.
Tang, Y., & Eliasmith, C. (2010). Deep networks for robust visual recognition. In Proceedings of the 27th International Conference on Machine Learning (ICML-10) (pp. 1055–1062).
Thagard, P. (1988). Computational Philosophy of Science. Cambridge, MA: MIT Press.
Thagard, P. (2012). The cognitive science of science: Explanation, discovery, and conceptual change. Cambridge, MA: MIT Press.
Thagard, P., & Schröder, T. (2013). Emotions as semantic pointers: Constructive neural mechanism. In L. F. Barrett & J. A. Russell (Eds.), The physiological construction of emotions. New York: Guilford.
Thagard, P., & Stewart, T. C. (2011). The Aha! experience: Creativity through emergent binding in neural networks. Cognitive Science, 35(1), 1–33; reprinted in Thagard, P. (2012), The cognitive science of science: Explanation, discovery, and conceptual change. Cambridge, MA: MIT Press, Chap. 8.
Tripp, B. P., & Eliasmith, C. (2010). Population models of temporal differentiation. Neural Computation, 22(3), 621–659.
von der Malsburg, C. (1981). The correlation theory of brain function. MPI Biophysical Chemistry, Internal Report 81-2. Reprinted in E. Domany, J.L. van Hemmen, and K. Schulten (Eds.), (1994), Models of neural networks II. Berlin: Springer.
Acknowledgments
We thank Paul Thagard for sending us the data on the combinatorial hypothesis, as well as Chris Eliasmith and Terry Stewart for patiently answering the questions of one of us (de Pasquale) about Spaun. We however take full responsibility for any error or misinterpretation in the present article.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
de Pasquale, JF., Poirier, P. Convolution and modal representations in Thagard and Stewart’s neural theory of creativity: a critical analysis. Synthese 193, 1535–1560 (2016). https://doi.org/10.1007/s11229-015-0934-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11229-015-0934-7