A probabilistic model of theory formation

Abstract

Concept learning is challenging in part because the meanings of many concepts depend on their relationships to other concepts. Learning these concepts in isolation can be difficult, but we present a model that discovers entire systems of related concepts. These systems can be viewed as simple theories that specify the concepts that exist in a domain, and the laws or principles that relate these concepts. We apply our model to several real-world problems, including learning the structure of kinship systems and learning ontologies. We also compare its predictions to data collected in two behavioral experiments. Experiment 1 shows that our model helps to explain how simple theories are acquired and used for inductive inference. Experiment 2 suggests that our model provides a better account of theory discovery than a more traditional alternative that focuses on features rather than relations.

Introduction

Parent: A person who has begotten or borne a child.

Child: The offspring, male or female, of human parents.

The Oxford English Dictionary, 2nd edition, 1989.

Samuel Johnson acknowledges that his dictionary of 1755 is far from perfect, but suggests that “many seeming faults are to be imputed rather to the nature of the undertaking, than the negligence of the performer.” He argues, for instance, that “some explanations are unavoidably reciprocal or circular, as hind, the female of the stag; stag, the male of the hind.” Analogies between dictionary definitions and mental representations can only extend so far, but Johnson appears to have uncovered a general truth about the structure of human knowledge. Scholars from many disciplines have argued that concepts are organized into systems of relations, and that the meaning of a concept depends in part on its relationships to other concepts (Block, 1986, Carey, 1985, Field, 1977, Goldstone and Rogosky, 2002, Quillian, 1968, Quine and Ullian, 1978). To appreciate the basic idea, consider pairs of concepts like parent and child, disease and symptom, or life and death. In each case it is difficult to imagine how a learner could fully understand one member of the pair without also understanding the other. Systems of concepts, however, are often much more complex than mutually dependent pairs. Concepts like life and death, for instance, are embedded in a system that also includes concepts like growth, eating, energy and reproduction (Carey, 1985).

Systems of concepts capture some important aspects of human knowledge but also raise some challenging puzzles (Fodor & Lepore, 1992). Here we mention just two. First, it is natural to think that many concepts (including dog, tree and electron) are shared by many members of our society, but if the meaning of any concept depends on its role within an entire conceptual system, it is hard to understand how two individuals with different beliefs (and therefore different conceptual systems) could have any concepts in common (Fodor & Lepore, 1992). Second, a holistic approach to concept meaning raises a difficult acquisition problem. If the meaning of each concept depends on its role within a system of concepts, it is difficult to see how a learner might break into the system and acquire the concepts that it contains (Hempel, 1985, Woodfield, 1987). Goldstone and Rogosky (2002) recently presented a formal model that helps to address the first puzzle, and here we present a computational approach that helps to address the second puzzle.

Following prior usage in psychology (Carey, 1985) and artificial intelligence (Davis, 1990) we use the term theory to refer to a system that specifies a set of concepts and relationships between these concepts. Scientific theories are paradigmatic examples of the systems we will consider, but psychologists have argued that everyday knowledge is organized into intuitive theories that are similar to scientific theories in many respects (Carey, 1985, Murphy and Medin, 1985, Wellman and Gelman, 1992). Both kinds of theories are believed to play several important roles. As we have already seen, theories help to individuate concepts, and many kinds of concepts derive their meaning from the roles they play in theories. Theories allow learners to explain existing observations, and to make predictions about new observations. Finally, theories guide inductive inferences by restricting a learner’s attention to features and hypotheses that are relevant to the task at hand.

Theories may take many different forms, and the examples we focus on are related to the “framework theories” described by Wellman and Gelman (1992). Framework theories specify the fundamental concepts that exist in a domain and the possible relationships between these concepts. A framework theory of medicine, for example, might indicate that two of the fundamental concepts are chemicals and diseases, and that chemicals can cause diseases (Fig. 1). A “specific theory” is a more detailed account of the phenomena in some domain, and is typically constructed from concrete instances of the abstract categories provided by the framework theory. Extending our medical example, a specific theory might indicate that asbestos can cause lung cancer, where asbestos is a chemical and lung cancer is a disease. The framework theory therefore suggests that any specific correlation between asbestos exposure and lung cancer is better explained by a causal link from asbestos to lung cancer than a link in the opposite direction. Although researchers should eventually aim for models that can handle both framework theories and specific theories, working with framework theories is a useful first step. Framework theories are important since they capture some of our most fundamental knowledge, and in some cases they appear simple enough that we can begin to think about them computationally.

Three fundamental questions can be asked about theories: what are they, how are they used to make inductive inferences, and how are they acquired? Philosophers and psychologists have addressed all three questions (Carey, 1985, Hempel, 1972, Kuhn, 1970, Popper, 1935, Wellman and Gelman, 1998), but there have been few attempts to provide computational answers to these questions. Our work takes an initial step in this direction: we consider only relatively simple theories, but we specify these theories formally, we use these theories to make predictions about unobserved relationships between entities, and we show how these theories can be learned from raw relational data.

The first of our three fundamental questions requires us to formalize the notion of a theory. We explore the idea that framework theories can be represented as a probabilistic model which includes a set of categories and a matrix of parameters specifying relationships between those categories. Representations this simple will only be able to capture some aspects of framework theories, but working with simple representations allows us to develop tractable answers to our remaining two questions.

The second question asks how theories can be used for inductive inference. Each of our theories specifies the relationships between categories that are possible or likely, and predictions about unobserved relationships between entities are guided by inductive inferences about their category assignments. Since we represent theories as probabilistic models, Bayesian inference provides a principled framework for inferences about category assignments, relationships between categories and relationships between entities.

The final question—how are theories acquired?—is probably the most challenging of the three. Some philosophers suggest that this question will never be answered, and that there can be “no systematic, useful study of theory construction or discovery” (Newton-Smith, 1981, p. 125). To appreciate why theory acquisition is challenging, consider a case where the concepts belonging to a theory are not known in advance. Imagine a child who stumbles across a set of identical-looking metal objects. She starts to play with these objects and notices that some pairs seem to exert mysterious forces on each other when they come into close proximity. Eventually she discovers that there are three kinds of objects—call them magnets, magnetic objects and non-magnetic objects. She also discovers causal laws that capture the relationships between these concepts: magnets interact with magnets and magnetic objects, magnetic objects interact only with magnets, and non-magnetic objects do not interact with any other objects. Notice that the three hidden concepts and the causal laws are tightly coupled. The causal laws are only defined in terms of the concepts, and the concepts are only defined in terms of the causal relationships between them. This coupling raises a challenging learning problem. If the child already knew about the three concepts—suppose, for instance, that different kinds of objects were painted different colors—then discovering the relationships between the concepts would be simple. Similarly, a child who already knew the causal laws should find it easy to group the objects into categories. We consider the case where neither the concepts nor the causal laws are known. In general, a learner may not even know when there are new concepts to be discovered in a particular domain, let alone how many concepts there are or how they relate to one another. The approach we describe attempts to solve all of these acquisition problems simultaneously.

We suggested already that Bayesian inference can explain how theories are used for induction, and our approach to theory acquisition is founded on exactly the same principle. Given a formal characterization of a theory, we can set up a space of possible theories and define a prior distribution over this space. Bayesian inference then provides a normative strategy for selecting the theory in this space that is best supported by the available data. Many Bayesian accounts of human learning work with relatively simple representations, including regions in multidimensional space and sets of clusters (Anderson, 1991, Shepard, 1987, Tenenbaum and Griffiths, 2001). Our model demonstrates that the Bayesian approach to knowledge acquisition can be carried out even when the representations to be learned are richly structured, and are best described as relational theories.

Our approach draws on previous proposals about relational concepts and on existing models of categorization. Several researchers (Gentner and Kurtz, 2005, Markman and Stilwell, 2001) have emphasized that many concepts derive their content from their relationships to other concepts, but there have been few formal models that explain how these concepts might be learned (Markman & Stilwell, 2001). Most models of categorization take features as their input, and are able only to discover categories defined by characteristic patterns of features (Anderson, 1991, Medin and Schaffer, 1978, Nosofsky, 1986). Our approach brings these two research programs together. We build on formal techniques used by previous models—in particular, our approach extends Anderson’s rational model of categorization—but we go beyond existing categorization models by working with rich systems of relational data.

The next two sections introduce the simple kinds of theories that we consider in this paper. We then describe our formal approach and evaluate it in two ways. First we demonstrate that our model learns large-scale theories given real-world data that roughly approximate the kind of information available to human learners. In particular, we show that our model discovers theories related to folk biology and folk sociology, and a medical theory that captures relationships between ontological concepts. We then turn to empirical studies and describe two behavioral experiments where participants learn theories analogous to the simple theory of magnetism already described. Our model helps to explain how these simple theories are learned and used to support inductive inferences, and we show that our relational approach explains our data better than a feature-based model of categorization.