In Plato’s famous dialogue Phaedo, Simmias is asked to determine who, amongst all sorts of men, is able to attain true knowledge. Is it not he, Socrates asks, who

pursues the truth by applying his pure and unadulterated thought to the pure and unadulterated object, cutting himself off as much as possible from his eyes and ears and virtually all the rest of his body, as an impediment which by its presence prevents the soul from attaining to truth and clear thinking? (Plato, 1961, 65e–66a, p. 48)

He then continues: “we are in fact convinced that if we are ever to have pure knowledge of anything, we must get rid of the body and contemplate things by themselves with the soul by itself” (Plato, 1961, 66b–67b, p. 49). In tune with Plato’s ideas, Descartes maintained that the objects of the external world are known “by the intellect alone,” for things “are not perceived because they are seen and touched, but only because they are rightly comprehended by the mind” (Descartes, 1651, II.16).

The belief in a dualism that separates the mind from the body has had a strong influence on both mathematics and mathematics education for some time. The purpose of this Special Issue is not only to bring the body fully into our attempts to understand mathematical thinking but also to explore the range of specific ways that embodiment is enacted in mathematical situations. We will examine the construction of mathematical meaning from the perspective of multimodality, that is, taking into account the range of cognitive, physical, and perceptual resources that people utilize when working with mathematical ideas. These resources or modalities include both oral and written symbolic communication as well as drawing, gesture, the manipulation of physical and electronic artifacts, and various kinds of bodily motion.

The inclusion of the body in the act of knowing is a recent epistemological trend that can be traced back to the work of social theorists, epistemologists and phenomenologists such as Husserl (1931), Gelhen (1988), and Merleau-Ponty (1945). Although their respective theoretical perspectives differed in important aspects, they agreed on one point: knowledge is much more than the result of formal abstract deductive mechanisms. Crucial to the production of knowledge is the individual’s experience in the act of knowing and the fact that this experience is mediated by one’s own body. However, this return of the body to epistemology and cognition does not amount to a disguised form of empiricism. Conceptual ideas are not merely the impression that material things make on us, as Hume (1991) and other 18th century empiricists once claimed. The return of the body is rather the awareness that, in our acts of knowing, different sensorial modalities—tactile, perceptual, kinesthetic, etc.—become integral parts of our cognitive processes. This is what is termed here the multimodal nature of cognition.

From this perspective, mathematics is a powerful and stable product of human imagination, with its ultimate origins in physical experience. Nemirovsky and Borba (2003) have emphasized the role of perceptuo-motor action in the processes of knowing:

While modulated by shifts of attention, awareness, and emotional states, understanding and thinking are perceptuo-motor activities; furthermore, these activities are bodily distributed across different areas of perception and motor action based on how we have learned and used the subject itself. [As a consequence,] the understanding of a mathematical concept, rather than having a definitional essence, spans diverse perceptuo-motor activities, which become more or less active depending on the context. (p. 108)

Yet the theoretical description of the cognitive and epistemic nature of multimodality is still a work in progress. As current literature intimates, there are different ways of theorizing and emphasizing embodiment and multimodality (see, e.g., Lakoff & Núñez, 2000; Gallese, Keysers & Rizzolatti, 2004; Gallese & Lakoff, 2005; Seitz, 2000; Varela, Thompson & Rosch, 1991). Embodiment can be examined on at least three levels:

  • Microgenetically, through the analysis of gesture, speech, gaze, writing, drawing, and bodily motion, as these unfold in a situation of interaction, for instance, a classroom,

  • Developmentally, through the analysis of the physical experiences of human beings throughout their lifetimes, for example, the use of “hands-on” manipulatives to teach mathematics to children; and

  • Biologically, through the analysis of the constraints and capabilities developed by the human species through evolutionary time, as a result of interacting with its physical and social environment.

This new attention to gesture and the body does not negate the fact that mathematics and other forms of human knowledge are “inseparable from symbolic tools” and that cognition is a “culturally shaped phenomenon” (Sfard & McClain, 2002, p. 156). Different cultural artifacts are examined in this issue; however, far from being seen as “conveying” information to the learner, these artifacts are seen as part of the set of resources available within the context of multiple semiotic modalities.

It seems pertinent to relate the roots of the current interest in multimodality in mathematics education to the attention that classroom discourse acquired in the 1990s (see, e.g., Steinbring, Bartolini Bussi, & Sierpinska, 1998). The focus on discourse led to a close scrutiny of the kind of words the students and teachers were using in social interaction. Inspired by the seminal work of Lakoff and Núñez (2000) and theories of discourse developed by Michail Bakhtin (1981, 1986), among others, the social and embodied nature of words, metaphors, and other products of conceptual integration in students’ thinking quickly became salient (Fauconnier & Turner, 2002; Núñez, 2000). For example, important traces of linguistic and perceptual embodied elements were found to be at the basis of the students’ first symbolic algebraic formulas (Radford, 2000, 2003). However, the possibilities afforded by new technologies, in particular the technologies of video recording that displaced the traditional equipment of audio recordings, made clear that the embodied nature of language was only the tip of the iceberg: these tools captured a previously “invisible” aspect of communication, physical gesture. Gestures, in conjunction with a wide range of other modalities, have come to be recognized as key elements in communication and conceptualization within science and mathematics (Roth, 2001).

The phenomenon of gesture has also been explored in ordinary communicational contexts by linguists and psychologists (e.g., Alibali, Bassok, Solomon, Syc, & Goldin-Meadow, 1999; Goldin-Meadow, 2003; Kita, 2000; McNeill, 1985, 1992, 2000, 2005). McNeill, a seminal figure in gesture studies, noted that, “Speech and gesture are elements of a single integrated process of utterance formation in which there is a synthesis of opposite modes of thought—global-synthetic and instantaneous imagery with linear-segmented temporally extended verbalization” (McNeill, 1992, p. 35). By this, McNeill means that speech is composed of segments that are produced in a linear fashion through time, and which are structured in an analytic, decomposable hierarchy (sentences made up of words, words of phonemes, etc.). In contrast, the meaning of a gesture derives from the whole, and is not decomposable into parts with separate, fixed meanings. Whether talking of graphing, geometry, or the visualization of more abstract entities, there are many ideas within mathematics that require both imagery as well as analytic language for full understanding. This Special Issue offers examples where gesture and other modalities provide a bridge between personal, internal imagery and shared, external speech and inscriptions about mathematics (Roth, 2002).

McNeill (2005) has classified gestures according to several non-exclusive dimensions: deixis (pointing to existing or virtual objects); metaphoricity (referencing an abstraction); iconicity (where the form is directly related to the semantic content of speech); temporal highlighting (simple repeated gestures used for emphasis) and gestures that modulate social interactivity. All of these dimensions play essential roles in communicating and thinking about mathematics. Research in both mathematical and non-mathematical domains has examined how gesture acts in the generation as well as the communication of meaning. Alibali, Kita and Young (2000), for example, have developed a theory of the function of gesture in conceptualization in their Information Packaging Hypothesis:

Gesture is involved in the conceptual planning of the messages, helps speakers to “package” spatial information into verbalisable units, by exploring alternative ways of encoding and organising spatial and perceptual information…gesture plays a role in speech production because it plays a role in the process of conceptualisation. (pp. 594–595)

Each of the papers in this Special Issue considers the important role of gesture, interacting with other modalities, in the construction of mathematics. First, Arzarello, Paola, Robutti and Sabena present the notion of the semiotic bundle as a model for examining the various resources utilized within the classroom, for example, in sketching the form of the graph of a function. Radford also considers graphing, in the context of an electronic artifact that can capture and display motion, in presenting a theory of knowledge objectification that brings together both the sensuous and the cultural. Edwards utilizes the framework of conceptual integration to analyze speech and gesture in the context of recollection, as adult students describe how they first learned about fractions. The role of semiotic mediation in students’ interactions with physical artifacts is explored by Maschietto and Bartolini Bussi, while Nemirovsky and Ferrara examine the mathematical imagination and how it is expressed through the utterances of a student in a high school algebra class. Finally, Roth and Thom reconceptualize mathematical conceptions from a phenomenological point of view, illustrating their framework by drawing on an episode in which a second-grader begins to make sense of a three-dimensional shape.

Most of the papers presented in this Special Issue are revised and expanded versions of presentations at a Research Forum on “Gesture and the Construction of Mathematical Meaning” at the 29th Annual Meeting of the International Group for the Psychology of Mathematics Education in Melbourne (Arzarello & Edwards, 2005). Julian Williams was a reactor at that forum, and graciously agreed to write a new and expanded reaction to the updated set of papers. Anna Sfard also generously offered her reaction to the reports presented in the Special Issue. Our late colleague and friend Jim Kaput was also a reactor in 2005, and many of us will remember him delivering his vivid and interesting reaction from Massachusetts via videoconference. We want to thank Susan Kaput for allowing us to include his reaction here.

Naturally, multimodality, and the study of gesture in particular, are still young research fields within the ampler field of mathematics education, and many theoretical and methodological questions remain open. These phenomena, for one thing, call on us to revisit the difficult question of what is understood by thinking and cognition. We certainly need to forge more detailed concepts and finer-grained methodologies. We hope nonetheless that these papers, taken together, offer an overview of current issues and problems in understanding multimodal cognition in the teaching and learning of mathematics.