Representation in Connectionism

Fundamental assumptions behind qualitative modelling are critically considered and some inherent problems in that modelling approach are outlined. The problems outlined are due to the assumption that a sufficient set of symbols representing the fundamental features of the physical world exists. That assumption causes serious problems when modelling continuous systems. An alternative for intelligent system building for cases not suitable for qualitative modelling is proposed. The proposed alternative combines neural networks and quantitative modelling.

Localizing content in neural networks provides a bridge to understanding the way in which the brain stores and processes information. In this paper, I propose the existence of polytopes in the state space of the hidden layer of feedforward neural networks as vehicles of content. I analyze these geometrical structures from an information-theoretic point of view, invoking mutual information to help define the content stored within them. I establish how this proposal addresses the problem of misclassification and provide a novel (...) solution to the disjunction problem, which hinges on the precise nature of the causal-informational framework for content advocated herein. (shrink)

Connectionist attention to variables has been too restricted in two ways. First, it has not exploited certain ways of doing without variables in the symbolic arena. One variable-avoidance method, that of logical combinators, is particularly well established there. Secondly, the attention has been largely restricted to variables in long-term rules embodied in connection weight patterns. However, short-lived bodies of information, such as sentence interpretations or inference products, may involve quantification. Therefore short-lived activation patterns may need to achieve the effect of (...) variables. The paper is mainly a theoretical analysis of some benefits and drawbacks of using logical combinators to avoid variables in short-lived connectionist encodings without loss of expressive power. The paper also includes a brief survey of some possible methods for avoiding variables other than by using combinators. (shrink)

This paper reviews four significant advances on the feedforward architecture that has dominated discussions of connectionism. The first involves introducing modularity into networks by employing procedures whereby different networks learn to perform different components of a task, and a Gating Network determines which network is best equiped to respond to a given input. The second consists in the use of recurrent inputs whereby information from a previous cycle of processing is made available on later cycles. The third development involves developing (...) compressed representations of strings in which there is no longer an explicit encoding of the components but where information about the structure of the original string can be recovered and so is present functionally. The final advance entails using connectionist learning procedures not just to change weights in networks but to change the patterns used as inputs to the network. These advances significantly increase the usefulness of connectionist networks for modeling human cognitive performance by, among other things, providing tools for explaining the productivity and systematicity of some mental activities, and developing representations that are sensitive to the content they are to represent. (shrink)

The relation between logic and thought has long been controversial, but has recently influenced theorizing about the nature of mental processes in cognitive science. One prominent tradition argues that to explain the systematicity of thought we must posit syntactically structured representations inside the cognitive system which can be operated upon by structure sensitive rules similar to those employed in systems of natural deduction. I have argued elsewhere that the systematicity of human thought might better be explained as resulting from the (...) fact that we have learned natural languages which are themselves syntactically structured. According to this view, symbols of natural language are external to the cognitive processing system and what the cognitive system must learn to do is produce and comprehend such symbols. In this paper I pursue that idea by arguing that ability in natural deduction itself may rely on pattern recognition abilities that enable us to operate on external symbols rather than encodings of rules that might be applied to internal representations. To support this suggestion, I present a series of experiments with connectionist networks that have been trained to construct simple natural deductions in sentential logic. These networks not only succeed in reconstructing the derivations on which they have been trained, but in constructing new derivations that are only similar to the ones on which they have been trained. (shrink)

The notion of a ‘symbol’ plays an important role in the disciplines of Philosophy, Psychology, Computer Science, and Cognitive Science. However, there is comparatively little agreement on how this notion is to be understood, either between disciplines, or even within particular disciplines. This paper does not attempt to defend some putatively ‘correct’ version of the concept of a ‘symbol.’ Rather, some terminological conventions are suggested, some constraints are proposed and a taxonomy of the kinds of issue that give rise to (...) disagreement is articulated. The goal here is to provide something like a ‘geography’ of the various notions of ‘symbol’ that have appeared in the various literatures, so as to highlight the key issues and to permit the focusing of attention upon the important dimensions. In particular, the relationship between ‘tokens’ and ‘symbols’ is addressed. The issue of designation is discussed in some detail. The distinction between simple and complex symbols is clarified and an apparently necessary condition for a system to be potentially symbol, or token bearing, is introduced. (shrink)

“There’s Plenty of Room at the Bottom”, said the title of Richard Feynman’s 1959 seminal conference at the California Institute of Technology. Fifty years on, nanotechnologies have led computer scientists to pay close attention to the links between physical reality and information processing. Not all the physical requirements of optimal computation are captured by traditional models—one still largely missing is reversibility. The dynamic laws of physics are reversible at microphysical level, distinct initial states of a system leading to distinct final (...) states. On the other hand, as von Neumann already conjectured, irreversible information processing is expensive: to erase a single bit of information costs ~3 × 10−21 joules at room temperature. Information entropy is a thermodynamic cost, to be paid in non-computational energy dissipation. This paper addresses the problem drawing on Edward Fredkin’s Finite Nature hypothesis: the ultimate nature of the universe is discrete and finite, satisfying the axioms of classical, atomistic mereology. The chosen model is a cellular automaton with reversible dynamics, capable of retaining memory of the information present at the beginning of the universe. Such a CA can implement the Boolean logical operations and the other building bricks of computation: it can develop and host all-purpose computers. The model is a candidate for the realization of computational systems, capable of exploiting the resources of the physical world in an efficient way, for they can host logical circuits with negligible internal energy dissipation. (shrink)

There is a gap between two different modes of computation: the symbolic mode and the subsymbolic (neuron-like) mode. The aim of this paper is to overcome this gap by viewing symbolism as a high-level description of the properties of (a class of) neural networks. Combining methods of algebraic semantics and non-monotonic logic, the possibility of integrating both modes of viewing cognition is demonstrated. The main results are (a) that certain activities of connectionist networks can be interpreted as non-monotonic inferences, and (...) (b) that there is a strict correspondence between the coding of knowledge in Hopfield networks and the knowledge representation in weight-annotated Poole systems. These results show the usefulness of non-monotonic logic as a descriptive and analytic tool for analyzing emerging properties of connectionist networks. Assuming an exponential development of the weight function, the present account relates to optimality theory – a general framework that aims to integrate insights from symbolism and connectionism. The paper concludes with some speculations about extending the present ideas. (shrink)

Bayesian models can be related to cognitive processes in a variety of ways that can be usefully understood in terms of Marr's distinction among three levels of explanation: computational, algorithmic and implementation. In this note, we discuss how an integrated probabilistic account of the different levels of explanation in cognitive science is resulting, at least for the current research practice, in a sort of unpredicted epistemological shift with respect to Marr's original proposal.

We argue that neural networks for semantic cognition, as proposed by Rogers & McClelland (R&M), do not acquire semantics and therefore cannot be the basis for a theory of semantic cognition. The reason is that the neural networks simply perform statistical categorization procedures, and these do not require any semantics for their successful operation. We conclude that this has severe consequences for the semantic cognition views of R&M.

Two additional sources of evidence are provided in support of localist coding within connectionist networks. First, only models with localist codes can currently represent multiple pieces of information simultaneously or represent order among a set of items on-line. Second, recent priming data appear problematic for theories that rely on distributed representations. However, a faulty argument advanced by Page is also pointed out.

Connectionism provides hope for unifying work in neuroscience, computer science, and cognitive psychology. This promise has met with some resistance from Classical Computionalists, which may have inspired Connectionists to retaliate with bold, inflationary claims on behalf of Connectionist models. This paper demonstrates, by examining three intimately connected issues, that these inflationary claims made on behalf of Connectionism are wrong. This should not be construed as an attack on Connectionism, however, since the inflated claims made on its behalf have the look (...) of cures for which there are no ailments. There is nothing wrong with Connectionism for its failure to solve illusory problems. (shrink)

Jerry Fodor and Ernest Lepore [(1992) Holism: a shopper's guide, Oxford: Blackwell; (1996) in R. McCauley (Ed.) The Churchlands and their critics , Cambridge: Blackwell] have launched a powerful attack against Paul Churchland's connectionist theory of semantics--also known as state space semantics. In one part of their attack, Fodor and Lepore argue that the architectural and functional idiosyncrasies of connectionist networks preclude us from articulating a notion of conceptual similarity applicable to state space semantics. Aarre Laakso and Gary Cottrell [(1998) (...) in M. A. Gernsbacher & S. Derry (Eds) Proceedings of the 20th Annual Conference of the Cognitive Science Society, Mahway, NJ: Erlbaum; Philosophical Psychology ] 13, 47-76 have recently run a number of simulations on simple feedforward networks and applied a mathematical technique for measuring conceptual similarity in the representational spaces of those networks. Laakso and Cottrell contend that their results decisively refute Fodor and Lepore's criticisms. Paul Churchland [(1998) Journal of Philosophy, 95, 5-32 ] goes further. He uses Laakso and Cottrell's neurosimulations to argue that connectionism does furnish us with all we need to construct a robust theory of semantics and a robust theory of translation. In this paper I shall argue that whereas Laakso and Cottrell's neurocomputational results may provide us with a rebuttal of Fodor and Lepore's argument, Churchland's conclusion is far too optimistic. In particular, I shall try to show that connectionist modelling does not provide any objective criterion for achieving a one-to-one accurate translational mapping across networks. (shrink)

As Ruben notes, the macrostrategy can allow that the distinction may also be drawn at some micro level, but it insists that descent to the micro level is ...

It is widely appreciated that the difficulty of a particluar computation varies according to how the input data are presented. What is less understood is the effect of this computation/representation tradeoff within familiar learning paradigms. We argue that existing learning algoritms are often poorly equipped to solve problems involving a certain type of important and widespread regularity, which we call 'type-2' regularity. The solution in these cases is to trade achieved representation against computational search. We investigate several ways in which (...) such a trade-off may be pursued including simple incremental learning, modular connectionism, and the developmental hypothesis of 'representational redescription'. In addition, the most distinctive features of human cognition- language and culture- may themselves be viewed as adaptions enabling this representation/computation trade-off to be pursued on an even grander scale. (shrink)

Some regularities enjoy only an attenuated existence in a body of training data. These are regularities whose statistical visibility depends on some systematic recoding of the data. The space of possible recodings is, however, infinitely large type-2 problems. they are standardly solved! This presents a puzzle. How, given the statistical intractability of these type-2 cases, does nature turn the trick? One answer, which we do not pursue, is to suppose that evolution gifts us with exactly the right set of recoding (...) biases so as to reduce specific type-2 problems to (tractable) type-1 mappings. Such a heavy-duty nativism is no doubt sometimes plausible. But we believe there are other, more general mechanisms also at work. Such mechanisms provide general (not task-specific) strategies for managing problems of type-2 complexity. Several such mechanisms are investigated. At the heart of each is a fundamental ploy representational redescription language and culture – may themselves be viewed as adaptations enabling this representation/computation trade-off to be pursued on an even grander scale. (shrink)

The character of computational modelling of cognition depends on an underlying theory of representation. Classical cognitive science has exploited the syntax/semantics theory of representation that derives from logic. But this has had the consequence that the kind of psychological explanation supported by classical cognitive science is " _conceptualist_: " psychological phenomena are modelled in terms of relations that hold between concepts, and between the sensors/effectors and concepts. This kind of explanation is inappropriate for the Proper Treatment of Connectionism.

We describe and try to motivate our project to build systems using both a knowledge based and a neural network approach. These two approaches are used at different stages in the solution of a problem, instead of using knowledge bases exclusively on some problems, and neural nets exclusively on others. The knowledge base (KB) is defined first in a declarative, symbolic language that is easy to use. It is then compiled into an efficient neural network (NN) representation, run, and the (...) results from run time and (eventually) from learning are decompiled to a symbolic description of the knowledge contained in the network. After inspecting this recovered knowledge, a designer would be able to modify the KB and go through the whole cycle of compiling, running, and decompiling again. The central question with which this project is concerned is, therefore, How do we go from a KB to an NN, and back again? We are investigating this question by building tools consisting of a repertoire of language/translation/network types, and trying them on problems in a variety of domains. (shrink)

There has been a long-standing debate between symbolicists and connectionists concerning the nature of representation used by human cognizers. In general, symbolicist commitments have allowed them to provide superior models of high-level cognitive function. In contrast, connectionist distributed representations are preferred for providing a description of low-level cognition. The development of Holographic Reduced Representations (HRRs) has opened the possibility of one representational medium unifying both low-level and high-level descriptions of cognition. This paper describes the relative strengths and weaknesses of symbolic (...) and distributed representations. HRRs are shown to capture the important strengths of both types of representation. These properties of HRRs allow a rebuttal of Fodor and McLaughlin's (1990) criticism that distributed representations are not adequately structure sensitive to provide a full account of human cognition. (shrink)

A Bayesian network (BN) is a graphical model of uncertainty that is especially well suited to legal arguments. It enables us to visualize and model dependencies between different hypotheses and pieces of evidence and to calculate the revised probability beliefs about all uncertain factors when any piece of new evidence is presented. Although BNs have been widely discussed and recently used in the context of legal arguments, there is no systematic, repeatable method for modeling legal arguments as BNs. Hence, where (...) BNs have been used in the legal context, they are presented as completed pieces of work, with no insights into the reasoning and working that must have gone into their construction. This means the process of building BNs for legal arguments is ad hoc, with little possibility for learning and process improvement. This article directly addresses this problem by describing a method for building useful legal arguments in a consistent and repeatable way. The method complements and extends recent work by Hepler, Dawid, and Leucari (2007) on object-oriented BNs for complex legal arguments and is based on the recognition that such arguments can be built up from a small number of basic causal structures (referred to as idioms). We present a number of examples that demonstrate the practicality and usefulness of the method. (shrink)

A similarity space is a hyperspace in which the dimensions represent various dimensions on which objects may differ. The similarity space theory of concepts is the thesis that concepts are regions of similarity spaces that are somehow realized in the brain. Proponents of such a theory of concepts include Paul Churchland and Peter Gärdenfors. This paper argues that the similarity space theory of concepts is mistaken because regions of similarity spaces cannot serve as the components of judgments. It emerges that (...) although similarity spaces cannot model concepts, they may model a kind of nonconceptual representation. (shrink)

?Representation? is a concept which occurs both in cognitive science and philosophy. It has common features in both settings in that it concerns the explanation of behaviour in terms of the way the subject categorizes and systematizes responses to its environment. The prevailing model sees representations as causally structured entities correlated on the one hand with elements in a natural language and on the other with clearly identifiable items in the world. This leads to an analysis of representation and cognition (...) in terms of formal symbols and their relations. But human perception and cognition use multiple informational constraints and deal with unsystematic and messy input in a way best explained by Parallel Distributed Processing models. This undermines the claim that a formal representational theory of mind is ?the only game in town?. In particular it suggests a radically different model of brain function and its relation to epistemology from that found in current representational theories. (shrink)

The past decade has witnessed the emergence of a novel stance on semantic representation, and its relationship to context sensitivity. Connectionist-minded philosophers, including Clark and van Gelder, have espoused the merits of viewing hidden-layer, context-sensitive representations as possessing semantic content, where this content is partially revealed via the representations'' position in vector space. In recent work, Bodén and Niklasson have incorporated a variant of this view of semantics within their conception of semantic systematicity. Moreover, Bodén and Niklasson contend that they (...) have produced experimental results which not only satisfy a kind of context-based, semantic systematicity, but which, to the degree that reality permits, effectively deals with challenges posed by Fodor and Pylyshyn (1988), and Hadley (1994a). The latter challenge involved well-defined criteria for strong semantic systematicity. This paper examines the relevant claims and experiments of Bodén and Niklasson. It is argued that their case fatally involves two fallacies of equivocation; one concerning ''semantic content'' and the other concerning ''novel test sentences''. In addition, it is argued that their ultimate construal of context sensitive semantics contains serious confusions. These confusions are also found in certain publications dealing with "latent semantic analysis". Thus, criticisms presented here have relevance beyond the work of Bodén and Niklasson. (shrink)

Harnad's main argument can be roughly summarised as follows: due to Searle's Chinese Room argument, symbol systems by themselves are insufficient to exhibit cognition, because the symbols are not grounded in the real world, hence without meaning. However, a symbol system that is connected to the real world through transducers receiving sensory data, with neural nets translating these data into sensory categories, would not be subject to the Chinese Room argument. Harnad's article is not only the starting point for the (...) present debate, but is also a contribution to a longlasting discussion about such questions as: Can a computer think? If yes, would this be solely by virtue of its program? Is the Turing Test appropriate for deciding whether a computer thinks? (shrink)

After people learn to sort objects into categories they see them differently. Members of the same category look more alike and members of different categories look more different. This phenomenon of within-category compression and between-category separation in similarity space is called categorical perception (CP). It is exhibited by human subjects, animals and neural net models. In backpropagation nets trained first to auto-associate 12 stimuli varying along a onedimensional continuum and then to sort them into 3 categories, CP arises as a (...) natural side-effect because of four factors: (1) Maximal interstimulus separation in hidden-unit space during autoassociation learning, (2) movement toward linear separability during categorization learning, (3) inverse-distance repulsive force exerted by the between-category boundary, and (4) the modulating effects of input iconicity, especially in interpolating CP to untrained regions of the continuum. Once similarity space has been "warped" in this way, the compressed and separated "chunks" have symbolic labels which could then be combined into symbol strings that constitute propositions about objects. The meanings of such symbolic representations would be "grounded" in the system's capacity to pick out from their sensory projections the object categories that the propositions were about. (shrink)

There is disagreement over the notion of representation in cognitive science. Many investigators equate representations with symbols, that is, with syntactically defined elements in an internal symbol system. In recent years there have been two challenges to this orthodoxy. First, a number of philosophers, including many outside the symbolist orthodoxy, have argued that "representation" should be understood in its classical sense, as denoting a "stands for" relation between representation and represented. Second, there has been a growing challenge to orthodoxy under (...) the banner of connectionism. Although this connectionist challenge has evoked emotionally-charged rebukes from the symbolist camp, connectionists as a group have not articulated a conception of representation to replace the symbolist view. Nonetheless, most agree on the need for a nonsymbolic notion of representation. This paper advances a "stands for" sense of representation as primary by incorporating it into a general approach to cognitive science consonant with the connectionism. The idea is to marry connectionism to a particular version of functionalism, viz., one that builds its notion of "function" on the similarity between functional analysis in biology and in psychology. It builds on my earlier work adapting and revising Marr's tri-level approach to cognition. My proposal frees Marr's analysis from its moorings in the orthodox symbolic view of representation and allies it with a notion of functional analysis akin to that proposed by Cummins, developed further by Haugeland, and invoked by Millikan and Dretske. Unlike these authors, however, I do not wed representation to a general belief-desire analysis of behavior. Rather, I follow what I take to be the lesson of psychological practice, according to which the investigator does not seek to explain behavior in general but seeks to analyze the cognitive capacities that underlie behavior, such as vision, memory, learning, and linguistic capacities. Accordingly, ascriptions of representational content are made not by working back from belief-desire ascriptions, but in the context of forming psychological models to account for specific cognitive capacities. Because conceptions of representation in cognitive science typically are embedded in a general approach to the study of cognition, arguments for the comparative plausibility of a particular conception of representation must scout these larger frameworks. I therefore begin by characterizing the interlocking set of assumptions that gave life to the orthodox symbolist approach, paying special attention to the complementarity between representation and process that naturally arises from those assumptions. I then consider two versions of an alternative approach to representation: Dretske's and my own. Finally, I urge the merits of the "cognitive capacities" over the "belief-desire" approach to the subject-matter of cognitive science. (shrink)

In this paper I defend the propriety of explaining the behavior of distributed connectionist networks by appeal to selected data stored therein. In particular, I argue that if there is a problem with such explanations, it is a consequence of the fact that information storage in networks is superpositional, and not because it is distributed. I then develop a ``proto-account'''' of causation for networks, based on an account of Andy Clark''s, that shows even superpositionality does not undermine information-based explanation. Finally, (...) I argue that the resulting explanations are genuinely informative and not vacuous. (shrink)

Analogy making from examples is a central task in intelligent system behavior. A lot of real world problems involve analogy making and generalization. Research investigates these questions by building computer models of human thinking concepts. These concepts can be divided into high level approaches as used in cognitive science and low level models as used in neural networks. Applications range over the spectrum of recognition, categorization and analogy reasoning. A major part of legal reasoning could be formally interpreted as an (...) analogy making process. Because it is not the same as reasoning in mathematics or the physical sciences, it is necessary to use a method, which incorporates first the ability to specify likelihood and second the opportunity of including known court decisions. We use for modelling the analogy making process in legal reasoning neural networks and fuzzy systems. In the first part of the paper a neural network is described to identify precedents of immaterial damages. The second application presents a fuzzy system for determining the required waiting period after traffic accidents. Both examples demonstrate how to model reasoning in legal applications analogous to recent decisions: first, by learning a system with court decisions, and second, by analyzing, modelling and testing the decision making with a fuzzy system. (shrink)

Page argues convincingly for several important properties of localist representations in connectionist models of cognition. I argue that another important property of localist representations is that they serve as the starting point for connectionist representations of symbolic (relational) structures because they express meaningful properties independent of one another and their relations.

This paper examines the use of connectionism (neural networks) in modelling legal reasoning. I discuss how the implementations of neural networks have failed to account for legal theoretical perspectives on adjudication. I criticise the use of neural networks in law, not because connectionism is inherently unsuitable in law, but rather because it has been done so poorly to date. The paper reviews a number of legal theories which provide a grounding for the use of neural networks in law. It then (...) examines some implementations undertaken in law and criticises their legal theoretical naïvete. It then presents a lessons from the implementations which researchers must bear in mind if they wish to build neural networks which are justified by legal theories. (shrink)

This collection offers an introduction to Churchland's work, as well as a critique of some of his most famous philosophical positions.

The concept of representation has become almost inextricably bound to the concept of symbol systems. the concepts is nowhere more prevalent than in descriptions of "internal representations." These representations are thought to occur in an internal symbol system that allows the brain to store and use information. In this paper we explore a different approach to understanding psychological processes, one that retains a commitment to representations and computations but that is not based on the idea that information must be stored (...) and manipulated in symbol systems. In particular, we suggest that the notion of a symbol system as currently understood construes psychological processes in terms of a specific type of computational system, in which a control function "reads," "interprets," and manipulates discrete entities called "symbols." We argue that other types of computational systems may provide a more appropriate characterization of psychological processes. One implication of our argument is the need to consider the constraints placed on computational theories in psychology by the nature of the computing device itself, the human brain. Perhaps surprisingly, this implication leads us to the conclusion that a "functionalist" conception of psychological processes (discussed below) does not entail that physiology is irrelevant to psychology, as has been maintained by prominent adherents of the symbol-systems approach. (shrink)

The parallel distributed processing (PDP) perspective brings forward the important point that all semantic phenomena are based on analog underlying mechanisms, involving the weighted summation of multiple inputs by individual neurons. It falls short of indicating, however, how the essentially discrete nature of semantic processing may emerge at the cognitive level. Bridging this gap probably requires attractor networks.

If connectionism is to be an adequate theory of mind, we must have a theory of representation for neural networks that allows for individual differences in weighting and architecture while preserving sameness, or at least similarity, of content. In this paper we propose a procedure for measuring sameness of content of neural representations. We argue that the correct way to compare neural representations is through analysis of the distances between neural activations, and we present a method for doing so. We (...) then use the technique to demonstrate empirically that different artificial neural networks trained by backpropagation on the same categorization task, even with different representational encodings of the input patterns and different numbers of hidden units, reach states in which representations at the hidden units are similar. We discuss how this work provides a rebuttal to Fodor and Lepore's critique of Paul Churchland's state space semantics. (shrink)