Skip to main content
SpringerLink
Log in

Analyzing children’s computational thinking through embodied interaction with technology: a multimodal perspective

  • Published:
Educational Technology Research and Development Aims and scope Submit manuscript

Abstract

The purpose of this paper is twofold. We first present a methodological framework for the analysis of embodied interaction with technology captured through video recording. The framework brings together a social semiotic approach to multimodality with the philosophical and theoretical roots of embodied cognition. We then demonstrate the application of the framework by exploring how the computational thinking of two fifth grade learners emerged as an embodied phenomenon during an educational robotics activity. The findings suggest that, for young children, computational thinking was extended to include the structures in the environment and guided by their embodiment of mathematical concepts. Specifically, the participants repeatedly used their bodies to simulate different possibilities for action while incorporating both perceptual and formal multiplicative reasoning strategies to conceptualize the robot’s movements. Implications for the design of embodied educational robotics activities and future application of the methodological framework are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Alibali, M.W., Boncoddo, R., Hostetter, A.B. (2014). Gesture in reasoning: an embodied perspective. In The Routledge handbook of embodied cognition (pp. 168-177). Routledge.

  • Abrahamson, D. (2017). Embodiment and mathematical learning. The SAGE encyclopedia of out-of-school learning. New York: SAGE.

    Google Scholar 

  • Abrahamson, D., & Lindgren, R. (2014). Embodiment and embodied design. In R.K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (2nd ed.). Cambridge, MA: Cambridge University Press.

  • Abrahamson, D., & Trninic, D. (2015). Bringing forth mathematical concepts: Signifying sensorimotor enactment in fields of promoted action. ZDM, 47(2), 295–306. https://doi.org/10.1007/s11858-014-0620-0.

    Article  Google Scholar 

  • Alibali, M. W., Nathan, M. J., Wolfgram, M. S., Church, R. B., Jacobs, S. A., Johnson Martinez, C., et al. (2014). How teachers link ideas in mathematics instruction using speech and gesture: A corpus analysis. Cognition and Instruction, 32(1), 65–100. https://doi.org/10.1080/10508406.2011.611446.

    Article  Google Scholar 

  • Alibali, M. W., & Nathan, M. J. (2012). Embodiment in mathematics teaching and learning: Evidence from learners' and teachers' gestures. Journal of the Learning Sciences, 21(2), 247–286. https://doi.org/10.1080/10508406.2011.611446.

    Article  Google Scholar 

  • Atit, K., Weisberg, S. M., Newcombe, N. S., & Shipley, T. F. (2016). Learning to interpret topographic maps: Understanding layered spatial information. Cognitive Research: Principles and Implications, 1(2), 1–18. https://doi.org/10.1186/s41235-016-0002-y.

    Article  Google Scholar 

  • Atmatzidou, S., & Demetriadis, S. (2016). Advancing students’ computational thinking skills through educational robotics: A study on age and gender relevant differences. Robotics and Autonomous Systems, 75, 661–670. https://doi.org/10.1016/j.robot.2015.10.008.

    Article  Google Scholar 

  • Baker, W. D., Green, J. L., & Skukauskaite, A. (2008). Video-enabled ethnographic research: A microethnographic perspective. How to do educational ethnography, 76-114.

  • Barth, H., Baron, A., Spelke, E., & Carey, S. (2009). Children’s multiplicative transformations of discrete and continuous quantities. Journal of Experimental Child Psychology, 103(4), 441–454. https://doi.org/10.1016/j.jecp.2009.01.014.

    Article  Google Scholar 

  • Baxter, P., & Jack, S. (2008). Qualitative case study methodology: Study design and implementation for novice researchers. The qualitative report, 13(4), 544-559. https://www.nova.edu/ssss/QR/QR13-4/baxter.pdf

  • Berland, M., & Wilensky, U. (2015). Comparing virtual and physical robotics environments for supporting complex systems and computational thinking. Journal of Science Education and Technology, 24(5), 628–647. https://doi.org/10.1007/s10956-015-9552-x.

    Article  Google Scholar 

  • Bers, M. U., Flannery, L., Kazakoff, E. R., & Sullivan, A. (2014). Computational thinking and tinkering: Exploration of an early childhood robotics curriculum. Computers & Education, 72, 145–157. https://doi.org/10.1016/j.compedu.2013.10.020.

    Article  Google Scholar 

  • Bezemer, J. (2014). Multimodal transcription: A case study. Interactions, images and texts: A reader in multimodality, 11, 155–170.

    Article  Google Scholar 

  • Bezemer, J., & Mavers, D. (2011). Multimodal transcription as academic practice: A social semiotic perspective. International Journal of Social Research Methodology, 14(3), 191–206. https://doi.org/10.1080/13645579.2011.563616.

    Article  Google Scholar 

  • Blikstad-Balas, M. (2017). Key challenges of using video when investigating social practices in education: Contextualization, magnification, and representation. International Journal of Research & Method in Education, 40(5), 511–523. https://doi.org/10.1080/1743727X.2016.1181162.

    Article  Google Scholar 

  • Boyer, T. W., & Levine, S. C. (2015). Prompting children to reason proportionally: Processing discrete units as continuous amounts. Developmental Psychology, 51(5), 615. https://doi.org/10.1037/a0039010.

    Article  Google Scholar 

  • Brennan, K., & Resnick, M. (2012). New frameworks for studying and assessing the development of computational thinking. Paper presented in the proceedings of the 2012 annual meeting of the American Educational Research Association, Canada, Vancouver.

  • Calvo, P., & Gomila, T. (Eds.). (2008). Handbook of cognitive science: An embodied approach. Amsterdam: Elsevier.

    Google Scholar 

  • Chemero, A. (2009). Radical embodied cognition. Cambridge, MA: MIT Press.

    Book  Google Scholar 

  • Chen, G., Shen, J., Barth-Cohen, L., Jiang, S., Huang, X., & Eltoukhy, M. (2017). Assessing elementary students’ computational thinking in everyday reasoning and robotics programming. Computers & Education, 109, 162–175. https://doi.org/10.1016/j.compedu.2017.03.001.

    Article  Google Scholar 

  • Dreyfus, H. L. (1996). The current relevance of Merleau-Ponty’s phenomenology of embodiment. The Electronic Journal of Analytic Philosophy, 4(4), 1–16.

    Google Scholar 

  • Duijzer, C. A., Shayan, S., Bakker, A., Van der Schaaf, M. F., & Abrahamson, D. (2017). Touchscreen tablets: Coordinating action and perception for mathematical cognition. Frontiers in Psychology, 8, 144. https://doi.org/10.3389/fpsyg.2017.00144.

    Article  Google Scholar 

  • Favela, L. H., & Chemero, A. (2016). The animal-environment system. In Y. Coelllo & M. H. Fischer (Eds.), Foundations of embodied cognition: Perceptual and emotional embodiment (Vol. 1, pp. 59–74). New York: Routledge.

    Google Scholar 

  • Ferrara, F. (2014). How multimodality works in mathematical activity: Young children graphing motion. International Journal of Science and Mathematics Education, 12(4), 917–939. https://doi.org/10.1007/s10763-013-9438-4.

    Article  Google Scholar 

  • Flood, V.J., & Abrahamson, D. (2015). Refining mathematical meanings through multimodal revoicing interactions: The case of ‘‘faster’’. In Annual Meeting of the American Educational Research Association, Chicago, April (pp. 16-20).

  • Flood, V. J., Harrer, B. W., & Abrahamson, D. (2016). The interactional work of configuring a mathematical object in a technology-enabled embodied learning environment. Singapore: International Society of the Learning Sciences.

    Google Scholar 

  • Gallese, V., & Lakoff, G. (2005). The brain’s concepts: The role of the sensory-motor system in conceptual knowledge. Cognitive Neuropsychology, 22(3–4), 455–479. https://doi.org/10.1080/02643290442000310.

    Article  Google Scholar 

  • Garcez, P. M. (1997). Microethnography. In Encyclopedia of language and education (pp. 187-196). Springer, Dordrecht.

  • Gibson, J.J. (1977). The theory of affordances. In R. E. Shaw & J. Bransford (Eds.), Perceiving, Acting, and Knowing. Hillsdale, NJ: Lawrence Erlbaum Associates.

  • Gibson, J.J. (1986). The ecological approach to visual perception. New York, NY: Taylor & Francis Group. (Original work published in 1979).

  • Glenberg, A. M., & Kaschak, M. P. (2002). Grounding language in action. Psychonomic Bulletin & Review, 9(3), 558–565. https://doi.org/10.3758/BF03196313.

    Article  Google Scholar 

  • Goffman, E. (1964). The neglected situation. American Anthropologist, 66(6), 133–136.

    Article  Google Scholar 

  • Goodwin, M. H. (2007). Participation and embodied action in preadolescent girls' assessment activity. Research on Language and Social Interaction, 40(4), 353–375. https://doi.org/10.1080/08351810701471344.

    Article  Google Scholar 

  • Gordon, C. L., Shea, T. M., Noelle, D. C., & Balasubramaniam, R. (2019). Affordance compatibility effect for word learning in virtual reality. Cognitive Science, 43(6), 1–17.

    Article  Google Scholar 

  • Grover, S., & Pea, R. (2013). Computational thinking in K–12: A review of the state of the field. Educational Researcher, 42(1), 38–43. https://doi.org/10.3102/0013189X12463051.

    Article  Google Scholar 

  • Hall, R., & Nemirovsky, R. (2012). Introduction to the special issue: Modalities of body engagement in mathematical activity and learning. Journal of the Learning Sciences, 21(2), 207–215. https://doi.org/10.1080/10508406.2011.611447.

    Article  Google Scholar 

  • Harlow, D. B., Dwyer, H. A., Hansen, A. K., Iveland, A. O., & Franklin, D. M. (2018). Ecological design-based research for computer science education: Affordances and effectivities for elementary school students. Cognition and Instruction, 36(3), 224–246. https://doi.org/10.1080/07370008.2018.1475390.

    Article  Google Scholar 

  • Hirose, N. (2002). An ecological approach to embodiment and cognition. Cognitive Systems Research, 3(3), 289–299. https://doi.org/10.1016/S1389-0417(02)00044-X.

    Article  Google Scholar 

  • Hostetter, A. B., & Alibali, M. W. (2008). Visible embodiment: Gestures as simulated action. Psychonomic Bulletin & Review, 15(3), 495–514. https://doi.org/10.3758/PBR.15.3.495.

    Article  Google Scholar 

  • Howison, M., Trninic, D., Reinholz, D., Abrahamson, D. (2011). The Mathematical Imagery Trainer: From embodied interaction to conceptual learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1989-1998).

  • Hutchins, E. (1995). Cognition in the Wild. MIT press

  • Hutchins, E., & Nomura, S. (2011). Collaborative construction of multimodal utterances. In J. Streeck, C. Goodwin, C. LeBaron (Eds.) Embodied interaction: Language and body in the material world, p. 29-43. Cambridge Press.

  • Jaspers, J. (2013). Interactional sociolinguistics and discourse analysis. In The Routledge handbook of discourse analysis(pp. 161-172). Routledge.

  • Jeong, Y., Levine, S. C., & Huttenlocher, J. (2007). The development of proportional reasoning: Effect of continuous versus discrete quantities. Journal of Cognition and Development, 8(2), 237–256. https://doi.org/10.1080/15248370701202471.

    Article  Google Scholar 

  • Jewitt, C., Bezemer, J., & O'Halloran, K. (2016). Introducing multimodality. Routledge.

  • Johnson-Glenberg, M. C., & Megowan-Romanowicz, C. (2017). Embodied science and mixed reality: How gesture and motion capture affect physics education. Cognitive Research: Principles and Implications, 2(1), 24. https://doi.org/10.1186/s41235-017-0060-9.

    Article  Google Scholar 

  • Kong, S.C., Abelson, H., Lai, M. (2019). Introduction to Computational Thinking Education. In Computational Thinking Education (pp. 1-10). Springer, Singapore.

  • Kopcha, T. J., McGregor, J., Shin, S., Qian, Y., Choi, J., Hill, R., et al. (2017). Developing an integrative STEM curriculum for robotics education through Educational Design Research. Journal of Formative Design in Learning, 1(1), 31–44. https://doi.org/10.1007/s41686-017-0005-1.

    Article  Google Scholar 

  • Kopcha, T., & Ocak, C. (2019). Embodiment of computational thinking during collaborative robotics activity. In Proceedings of Computer Supported Collaborative Learning, International Society of the Learning Sciences. https://repository.isls.org//handle/1/1604

  • Lakoff, G., & Johnson, M. (2008). Metaphors We Live By. University of Chicago press.

  • Lakoff, G., & Núñez, R. E. (2000). Where mathematics comes from: How the embodied mind brings mathematics into being. New York, NY: Basic Books.

    Google Scholar 

  • Lan, Y. J., Fang, W. C., Hsiao, I. Y., & Chen, N. S. (2018). Real body versus 3D avatar: The effects of different embodied learning types on EFL listening comprehension. Educational Technology Research and Development, 66(3), 709–731.

    Article  Google Scholar 

  • Lannin, J. K., Chval, K. B., & Jones, D. (2013). Putting essential understanding of multiplication and division into practice in grades 3–5. Incorp: National Council of Teachers of Mathematics.

    Google Scholar 

  • Liben, L. S. (2012). Embodiment and children’s understanding of the real and represented world. In W. F. Overton, U. Müller, & J. L. Newman (Eds.), Developmental perspectives on embodiment and consciousness (pp. 207–240). New York: Lawrence Erlbaum Press.

    Google Scholar 

  • Malafouris, L. (2013). How things shape the mind. MIT Press.

  • Malinverni, L., Schaper, M. M., & Pares, N. (2019). Multimodal methodological approach for participatory design of full-body interaction learning environments. Qualitative Research, 19(1), 71–89. https://doi.org/10.1177/1468794118773299.

    Article  Google Scholar 

  • McNeill, D. (1992). Hand and mind: What gestures reveal about thought. University of Chicago press.

  • Merleau-Ponty, M. (1962). Phenomenology of Perception. Trans. by C. Smith. London: Routledge & Kegan Paul.

  • Milkowski, M. (2013). Explaining the computational mind. Cambridge, MA: MIT Press.

    Book  Google Scholar 

  • Möhring, W., Frick, A., & Newcombe, N. S. (2018). Spatial scaling, proportional thinking, and numerical understanding in 5-to 7-year-old children. Cognitive Development, 45, 57–67. https://doi.org/10.1016/j.cogdev.2017.12.001.

    Article  Google Scholar 

  • Mondada, L. (2018). Multiple temporalities of language and body in interaction: challenges for transcribing multimodality. Research on Language and Social Interaction, 51(1), 85–106. https://doi.org/10.1080/08351813.2018.1413878.

    Article  Google Scholar 

  • Nathan, M. J. (2008). An embodied cognition perspective on symbols, grounding, and instructional gesture. In M. DeVega, A. M. Glenberg, & A. C. Graesser (Eds.), Symbols and embodiment: Debates on meaning and cognition (pp. 375–396). Oxford, England: Oxford University Press.

    Chapter  Google Scholar 

  • Nathan, M. J., & Walkington, C. (2017). Grounded and embodied mathematical cognition: Promoting mathematical insight and proof using action and language. Cognitive Research: Principles and Implications, 2(1), 9. https://doi.org/10.1186/s41235-016-0040-5.

    Article  Google Scholar 

  • National Research Council. (2010). Report of a workshop on the scope and nature of computational thinking. National Academies Press.

  • National Science & Technology Council (2018). Charting a course for success: America’s strategy for STEM education.https://www.whitehouse.gov/wp-content/uploads/2018/12/STEM-Education-Strategic-Plan-2018.pdf

  • Nemirovsky, R., & Ferrara, F. (2009). Mathematical imagination and embodied cognition. Educational Studies in Mathematics, 70, 159–174. https://doi.org/10.1007/s10649-008-9150-4.

    Article  Google Scholar 

  • Newell, A., & Simon, H. A. (1972). Human problem solving. Englewood Cliffs, NJ: Prentice-Hall.

    Google Scholar 

  • Paivio, A. (1986). Mental Representations: A Dual Coding Approach (p. 1986). New York: Oxford University Press.

    Google Scholar 

  • Papert, S. (1980). Mindstorms (p. 607). New York: Basic Rooks.

    Google Scholar 

  • Reinholz, D., Trninic, D., Howison, M., & Abrahamson, D. (2010). It's not easy being green: embodied artifacts and the guided emergence of mathematical meaning. In Proceedings of the thirty-second annual meeting of the North-American chapter of the international group for the psychology of mathematics education (PME-NA 32) (Vol. 6, pp. 1488-1496). Columbus, OH: PME-NA.

  • Richardson, M. J., & Chemero, A. (2014). Complex dynamical systems and embodiment. In The Routledge handbook of embodied cognition (pp. 57-68). Routledge.

  • Rietveld, E., & Kiverstein, J. (2014). A rich landscape of affordances. Ecological Psychology, 26(4), 325–352. https://doi.org/10.1080/10407413.2014.958035.

    Article  Google Scholar 

  • Richey, R.C., Klein, J.D., Tracey, M.W. (2010). The instructional design knowledge base: Theory, research, and practice. Routledge.

  • Román-González, M., Pérez-González, J. C., Moreno-León, J., & Robles, G. (2018). Can computational talent be detected? Predictive validity of the Computational Thinking Test. International Journal of Child-Computer Interaction, 18, 47–58. https://doi.org/10.1016/j.ijcci.2018.06.004.

    Article  Google Scholar 

  • Romdenh-Romluc, K. (2010). Routledge philosophy guidebook to Merleau-Ponty and phenomenology of perception. Routledge.

  • Rowlands, M. (2010). The new science of the mind: From extended mind to embodied phenomenology. MIT Press.

  • Shapiro, L. (2019). Embodied cognition. Routledge.

  • Streeck, J., Goodwin, C., LeBaron, C. (2011). Embodied interaction in the material world: An introduction. In Streek, Goodwin, and LeBaron (Eds.) Embodied interaction: Language and body in the material world, 1-26.

  • Sung, W., Ahn, J.H., Kai, S.M., Black, J. (2017, March). Effective planning strategy in robotics education: An embodied approach. In Society for Information Technology & Teacher Education International Conference (pp. 1065-1071). Association for the Advancement of Computing in Education (AACE). https://www.learntechlib.org/primary/p/177387/

  • Sutton-Smith, B. (2009). The ambiguity of play. Harvard University Press.

  • Thelen, E., & Smith, L. B. (1994). A dynamic systems approach to the development of perception and action. Cambridge, MA: MIT Press.

    Google Scholar 

  • Weintrop, D., Beheshti, E., Horn, M., Orton, K., Jona, K., Trouille, L., et al. (2016). Defining computational thinking for mathematics and science classrooms. Journal of Science Education and Technology, 25(1), 127–147. https://doi.org/10.1007/s10956-015-9581-5.

    Article  Google Scholar 

  • Weisberg, S.M., & Newcombe, N.S. (2017). Embodied cognition and STEM learning: overview of a topical collection in CR:PI. Cognitive Research Principles and Implications, 2(38). https://doi.org/10.1186/s41235-017-0071-6

  • Williams, R. F. (2012). Image schemas in clock-reading: Latent errors and emerging expertise. Journal of the Learning Sciences, 21(2), 216–246. https://doi.org/10.1080/10508406.2011.553259.

    Article  Google Scholar 

  • Wilson, M. (2002). Six views of embodied cognition. Psychonomic Bulletin & Review, 9(4), 625–636. https://doi.org/10.3758/BF03196322.

    Article  Google Scholar 

  • Wing, J. (2006). Computational thinking. Communications of the ACM, 49(3), 33–35.

    Article  Google Scholar 

  • Wing, J. (2011). Research notebook: Computational thinking—What and why? The Link Magazine, Spring. Carnegie Mellon University, Pittsburgh. https://link.cs.cmu.edu/article.php?a=600

  • Yasar, O. (2018). Computational thinking, redefined. In Society for Information Technology & Teacher Education International Conference (pp. 72-80). Association for the Advancement of Computing in Education (AACE). https://www.learntechlib.org/primary/p/182505/

  • Yuen, T.T., Stone, J., Davis, D., Gomez, A., Guillen, A., Price Tiger, E., Boecking, M. (2015). A model of how children construct knowledge and understanding of engineering design within robotics focused contexts. International Journal of Research Studies in Educational Technology, 5(1). https://www.learntechlib.org/p/152295/

Download references

Funding

This study was not conducted with any funding sources.

Author information

Authors and Affiliations

  1. The University of Georgia, 205 River’s Crossing, 850 College Station Road, Athens, GA, 30602, Greece

    Theodore J. Kopcha & Ceren Ocak

  2. Learning and Cognition Lab, Arizona State University, 1000 S Forest Mall, Tempe, AZ, 85281, USA

    Yingxiao Qian

Authors
  1. Theodore J. Kopcha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ceren Ocak
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yingxiao Qian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Theodore J. Kopcha.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This study was approved by our institution’s Internal Review Board (IRB).

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kopcha, T.J., Ocak, C. & Qian, Y. Analyzing children’s computational thinking through embodied interaction with technology: a multimodal perspective. Education Tech Research Dev 69, 1987–2012 (2021). https://doi.org/10.1007/s11423-020-09832-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11423-020-09832-y

Keywords

Search

Navigation