Measuring category intuitiveness in unconstrained categorization tasks

Abstract

What makes a category seem natural or intuitive? In this paper, an unsupervised categorization task was employed to examine observer agreement concerning the categorization of nine different stimulus sets. The stimulus sets were designed to capture different intuitions about classification structure. The main empirical index of category intuitiveness was the frequency of the preferred classification, for different stimulus sets. With 169 participants, and a within participants design, with some stimulus sets the most frequent classification was produced over 50 times and with others not more than two or three times. The main empirical finding was that cluster tightness was more important in determining category intuitiveness, than cluster separation. The results were considered in relation to the following models of unsupervised categorization: DIVA, the rational model, the simplicity model, SUSTAIN, an Unsupervised version of the Generalized Context Model (UGCM), and a simple geometric model based on similarity. DIVA, the geometric approach, SUSTAIN, and the UGCM provided good, though not perfect, fits. Overall, the present work highlights several theoretical and practical issues regarding unsupervised categorization and reveals weaknesses in some of the corresponding formal models.

Introduction

Without concepts, human thought would be impossible as we know it. Concepts help us organize briefly and efficiently the information around us, but they are also at the heart of many abilities which we consider uniquely human, such as reasoning on the basis of abstract ideas (Murphy, 2004, Pothos and Wills, 2011). The question of how concepts arise is one of fundamental importance for our understanding of human behavior. Many concepts are taught, through language, social convention, or education. This tradition of supervised categorization inspired highly influential formalisms, such as prototype and exemplar theory (e.g., Hampton, 2000, Minda and Smith, 2000, Nosofsky, 1984, Vanpaemel and Storms, 2008). Equally, it seems that in many situations groupings can be constructed in an unsupervised manner, that is, without being guided by an external teacher signal. For some time now, researchers have been recognizing the importance of unsupervised categorization processes in the understanding of human concepts.

The focus of the present study is unsupervised categorization in the context of free sorting tasks such as the following: participants receive a set of schematic stimuli, presented individually on printed cards; they are asked to divide the stimuli in any way they like, with no constraints on the number of groups or the number of elements per group. That such a process is unsupervised is evident in that there are no external constraints to guide categorization; a participant can create any kind of groups he/she wants. Several researchers have employed free sorting tasks, mostly to examine the impact of various methodological variations on participant performance (Ashby et al., 1999, Handel and Imai, 1972, Handel and Preusser, 1969, Handel and Rhodes, 1980, Medin et al., 1987, Milton and Wills, 2004, Regehr & Brooks, 1995). For example, does it make a difference whether participants see all the stimuli at once instead of sequentially? Are there circumstances that encourage participants to create classifications on the basis of a single stimulus dimension? This research has produced many important insights, even though the range of stimulus structures employed has been typically limited. One of the objectives of the present research is to motivate and test a wide range of stimulus structures.

We seem to have a natural tendency to organize information in the world. When exposed to a new domain, we implicitly or instinctively look to identify the basic ‘kinds’ that go together. This is a paradigmatic case of unsupervised categorization, though in adult thought it is often hard to separate out such unsupervised categorization processes from influences based on linguistic labels and existing categories. This is not to say that there are not everyday life situations when we engage in purely unsupervised categorization processes similar to those in the lab-based free sorting tasks: for example, arranging books in a bookcase, organizing administrative paperwork, archiving literature search articles, or arranging household items in a garage or garden shed. In all cases, the stimuli can be described with a set of dimensions (not all perceptual), so that there is a similarity structure for the stimuli. Also, in all cases there is a sorting problem, that of deciding which items go together. Such examples show the relevance of unsupervised categorization in limited problem-solving situations, but we contend that the impact of unsupervised categorization in human thought is both more profound and more pervasive.

A controversial issue in development psychology concerns the relation between linguistic and conceptual development. One view is that linguistic development guides conceptual development, as linguistic labels are employed to facilitate the acquisition of concepts. An alternative view is that children first develop concepts, so that at a later stage labels are matched to appropriate concepts (e.g., Nelson, 1974, Quinn and Eimas, 1986, Schyns, 1991). Such a view is supported by evidence that parent–child interaction may involve limited or no corrective feedback, when it comes to children’s inappropriate use of linguistic labels (e.g., Chapman et al., 1986, Nelson et al., 1993; see also, Brown and Hanlon, 1970, Demetras et al., 1986, Johnson and Riezler, 2002). The process which allows conceptual development in children is in some ways analogous to a free sorting task, in that in both cases it is recognized that some items go with others. Indeed, developmental psychologists have shown that children can perform free sorting tasks like the one described above (e.g., Gopnik & Meltzoff, 1997).

The sense that certain items go together, which is thought to drive behavior in free sorting experiments, appears relevant in perceptual organization as well. When we interpret a novel visual scene there is often a very strong intuition that certain elements form groups. For example, in the experiments of Compton and Logan, 1993, Compton and Logan, 1999), participants were presented with arrangements of dots in two-dimensional spaces. In some cases, there was a very strong intuition about the presence of groups and most participants agreed in how the dots should be classified. This idea, that when a set of items can be classified in an intuitive way there should be more consistency in participants’ classifications, is a key element of the present research as well. More generally, the link between perceptual organization and unsupervised categorization has been taken up by some researchers, who proposed models of unsupervised categorization based on perceptual principles (Compton and Logan, 1993, Compton and Logan, 1999, Pothos and Chater, 2002).

We can extrapolate this intuition of ‘things going together’ with adult concepts as well. The relative contribution of supervised (through language, social interaction, etc.) and unsupervised processes in adult concepts is difficult to quantify (cf. Malt and Sloman, 2007, Malt et al., 1999; Malt & Sloman, 2007, for assumptions about categories induced by linguistic labels and the impact of linguistic labels on categorization). But, we can observe that many of our categories involve coherent collections of objects, that is objects which are similar to each other or, at the very least, make sense together (Murphy & Medin, 1985). What is the glue which binds together the members of a category? For example, why do we consider a category like ‘chairs’ as intuitive (coherent), a category like ‘games’ as less intuitive (in the sense that people disagree more about the membership of this category), and a category composed of ‘babies, the moon, and rulers’ completely nonsensical? We can call this the problem of what determines category intuitiveness and it is clearly a fundamental one for cognitive psychology. We would like to suggest that, at least part of the solution, relates to understanding performance in free sorting tasks. This is because there seems to be a fundamental equivalence between many of our concepts and the groups created in free sorting tasks, in that in both cases people recognize that certain items should be grouped together.

Research in supervised categorization has been much more extensive than research in unsupervised categorization. If one considers entirely unconstrained classification tasks with an aim related to category intuitiveness, there are few studies, which are methodologically limited (e.g., Compton and Logan, 1999, Pothos and Chater, 2002). Note that unsupervised categorization is not the same as unsupervised learning, though it is possible of course that the two processes are based on similar computational principles (after all, they are both instances of inductive inference). The former concerns the specific (empirical) objective of spontaneously grouping some stimuli together. The latter is more general and concerns all situations of generalizing from some initial stimuli without feedback (e.g., Billman and Knutson, 1996, Fiser and Aslin, 2005, Reber, 1967). Also, while there has been some work on unsupervised categorization, much of it is not appropriate for the study of category intuitiveness. For example, researchers have employed sequential or concurrent presentation procedures for the stimuli. However, there seems to be a strong sense of category intuitiveness only in the case of concurrently presented stimuli. Equally, some researchers have asked participants to spontaneously group a set of stimuli into a specific number of categories (e.g., Medin et al., 1987, Milton et al., 2008). Such a procedure is less appropriate when studying category intuitiveness, since it can restrict participant performance. We employed an unsupervised classification task, with no constraints in the number of groups which could be created.

Part of the success of research in supervised categorization can be attributed to the existence of standard datasets (e.g., Medin and Schaffer, 1978, Shepard et al., 1961), specific dependent variables (e.g., classification probability of novel instances or speed of learning), and detailed computational comparisons between competing formal models (Minda and Smith, 2000, Nosofsky, 1990). With the present work, we try to make progress in unsupervised categorization in all these respects. First, we motivate a dependent variable appropriate for the study of category intuitiveness. Second, we specify a range of stimulus sets, created so as to contrast various factors possibly relevant in free sorting performance, and collect data from a large population sample. Third, we concurrently apply several computational models of unsupervised categorization. Medin et al. (1987) provide an eloquent statement motivating a modeling effort in unsupervised categorization (p. 43): “The categories which people normally create and use represent a tiny subset of the many possible ways in which entities and experiences could be partitioned. Therefore, a central question is what basic principles underlie category construction.” The objective of the unsupervised categorization models is exactly this, to provide hypotheses about the computational principles (and mechanisms) which underlie category construction.