Skip to main content
SpringerLink
Log in

Assessing metacognitive activities: the in-depth comparison of a task-specific questionnaire with think-aloud protocols

  • Published:
European Journal of Psychology of Education Aims and scope Submit manuscript

Abstract

Teaching and assessing metacognitive activities are important educational objectives, and teachers are calling for efficient instruments. The advantages of questionnaires in measuring metacognitive activities are obvious, but serious validity issues appear. For example, correlations of questionnaire data with think-aloud measures are generally moderate to low. An explanation may be that these questionnaires are not constructed in line with the metacognitive activities measured by the think-aloud method. In the present study, a questionnaire is constructed based directly on a taxonomy for coding think-aloud protocols. Twenty ninth-graders studied a text while thinking aloud, after which they immediately received the questionnaire. The overall correlation between the questionnaire and the think-aloud protocols (r = 0.63) was promising. However, scale and item analyses clearly demonstrate some new validity issues. Comparing the questionnaire and the think-aloud results, the students seem to report overt metacognitive activities corresponding more with their behavior reported in the protocols than covert ones. In-depth explorations are presented.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. We can only estimate this theoretical maximum because questionnaire’s reliabilities were not presented in the cited studies.

  2. The formula of the Pearson’s contingency coefficient is \( C = \sqrt {{\frac{{{\chi^2}}}{{N + {\chi^2}}}}} \)

  3. Because of the small group of participants (n = 20) combined with the relatively large number of items (n = 56), we do not report in-depth the results of the performed factor analysis (principal component analysis; Varimax rotation; extraction = four factors). Because most of the distributions of frequencies of metacognitive activity were positively skewed, a square-root transformation was carried out on the think-aloud data (cf. Meijer et al. 2011). Both the factor structure of the think-aloud data (Explained Variance = 49.75 %) and the structure of the questionnaire’s data (Explained Variance = 48.84 %) differed from the anticipated four-factor solution (Orientation and Planning; Executing; Monitoring; and Elaboration and Evaluation).

References

  • Afflerbach, P. (2000). Verbal reports and protocol analysis. In M. L. Kamil, P. B. Mosenthal, P. D. Pearson, & R. Barr (Eds.), Handbook of reading research (volume 3) (pp. 163–181). New York: Longman.

    Google Scholar 

  • Ainley, M., & Patrick, L. (2006). Measuring self-regulated learning processes through tracking patterns of student interaction with achievement activities. Educational Psychology Review, 18, 267–286.

    Article  Google Scholar 

  • Azevedo, R. (2005). Using hypermedia as a metacognitive tool for enhancing student learning? The role of self-regulated learning. Educational Psychologist, 40, 199–209.

    Article  Google Scholar 

  • Baddeley, A. D. (2003). Working memory: Looking back and looking forward. Nature Reviews Neuroscience, 4, 829–839.

    Article  Google Scholar 

  • Bannert, M., & Mengelkamp, C. (2008). Assessment of metacognitive skills by means of instruction to think aloud and reflect when prompted. Does the verbalization method affect learning? Metacognition and Learning, 3, 39–58.

    Article  Google Scholar 

  • Blom, S., & Severiens, S. E. (2008). Engagement in self-regulated deep learning of successful immigrant and non-immigrant students in inner city schools. European Journal of Psychology of Education, 23, 41–58.

    Google Scholar 

  • Bråten, I., & Samuelstuen, M. S. (2007). Measuring strategic processing: Comparing task-specific self-reports to traces. Metacognition and Learning, 2, 1–20.

    Article  Google Scholar 

  • Breuker, J. A., Elshout, J. J., Van Someren, M. W., & Wielinga, B. J. (1986). Hardopdenken en protokolanalyse. [Thinking-aloud and protocol-analysis]. Tijdschrift voor Onderwijsresearch, 11, 241–254.

    Google Scholar 

  • Broekkamp, H., & Van Hout-Wolters, B. H. A. M. (2007). Students’ adaptation of study strategies when preparing for classroom tests. Educational Psychology Review, 19, 401–428.

    Article  Google Scholar 

  • Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitraitmultimethod matrix. Psychological Bulletin, 56, 81–105.

    Article  Google Scholar 

  • Coté, N., Goldman, S. R., & Saul, E. U. (1998). Students making sense of informational text: Relations between processing and representation. Discourse Processes, 25, 1–53.

    Article  Google Scholar 

  • Cromley, J. G., & Azevedo, R. (2006). Self-report of reading comprehension strategies: What are we measuring? Metacognition and Learning, 1, 229–247.

    Article  Google Scholar 

  • Dignath, C., & Büttner, G. (2008). Components of fostering self-regulated learning among students. A meta-analysis on intervention studies at primary and secondary school level. Metacognition and Learning, 3, 231–264.

    Article  Google Scholar 

  • Dinsmore, D. L., Alexander, P. A., & Loughlin, S. M. (2008). Focusing the conceptual lens on metacognition, self-regulation, and self-regulated learning. Educational Psychology Review, 20, 391–409.

    Article  Google Scholar 

  • Ericsson, K. A. (1988). Concurrent verbal reports on text comprehension: A review. Text, 8, 295–235.

    Google Scholar 

  • Ericsson, K. A., & Simon, H. A. (1984). Protocol analysis. Verbal reports as data. Cambridge: Institute of Technology Press.

    Google Scholar 

  • Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge: MIT Press.

    Google Scholar 

  • Ericsson, K. A., & Simon, H. A. (1994). Verbal reports as data. Psychological Review, 87, 215–251.

    Article  Google Scholar 

  • Field, A. (2005). Discovering statistics using SPSS (2nd ed.). London: SAGE Publications.

    Google Scholar 

  • Flavell, J. H. (1979). Metacognition and cognitive monitoring. American Psychologist, 34, 906–911.

    Article  Google Scholar 

  • Fox, E. (2009). The role of reader characteristics in processing and learning from informational text. Review of Educational Research, 79, 197–261.

    Article  Google Scholar 

  • Fox, M. C., Ericsson, K. A., & Best, R. (2011). Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. Psychological Bulletin, 137, 316–344.

    Article  Google Scholar 

  • Garson, D. (2004). Nominal Association: Phi, Contingency Coefficient, Tschuprow's T, Cramer's V, Lambda, Uncertainty Coefficient, from http://www2.chass.ncsu.edu/garson/pa765/assocnominal.htm.

  • Gilpin, A. R. (1993). Table for conversion of Kendall’s tau to Spearman's rho within the context of measures of magnitude of effect for meta-analysis. Educational and Psychological Measurement, 53, 87–92.

    Article  Google Scholar 

  • Hadwin, A. L., Winne, P. H., Stockley, D. B., Nesbit, J. C., & Woszczyna, C. (2001). Context moderates students’ self-reports about how they study. Journal of Educational Psychology, 93, 477–487.

    Article  Google Scholar 

  • Hattie, J. A. C. (2009). Visible learning. A synthesis of over 800 meta-analyses relating to achievement. London: Routledge.

    Google Scholar 

  • Hill, J. R., & Hannafin, M. J. (1997). Cognitive strategies and learning from the World Wide Web. Educational Technology Research and Development, 45(4), 37–64. http://www.education.auckland.ac.nz/staff/j.hattie/presentations.cfm.

    Article  Google Scholar 

  • Hofer, B. K. (2004). Epistemological understanding as a metacognitive process: Thinking aloud during online searching. Educational Psychologist, 39, 43–55.

    Article  Google Scholar 

  • Magliano, J. P., & Graesser, A. C. (1991). A three-pronged method for studying inference generation in literary text. Poetics, 20, 193–232.

    Article  Google Scholar 

  • Magliano, J. P., Millis, K. K., The, R. S. A. T., Team, D., Levinstein, I., & Boonthum, C. (2011). Assessing comprehension during reading with the Reading Strategy Assessment Tool (RSAT). Metacogntion and Learning, 6, 131–154.

    Article  Google Scholar 

  • McNamara, D. S., & Magliano, J. P. (2009). Self-explanation and metacognition: The dynamics of reading. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Handbook of metacognition in education (pp. 60–81). Mahwah: Erlbaum.

    Google Scholar 

  • Meijer, J., Veenman, M. V. J., & van Hout-Wolters, B. H. A. M. (2006). Metacognitive activities in text studying and problem solving: Development of a taxonomy. Educational Research and Evaluation, 12, 209–237.

    Article  Google Scholar 

  • Meijer, J., Veenman, M. V. J., & van Hout-Wolters, B. H. A. M. (2011). Multi-domain, multi-method measures of metacognitive activity: What is all the fuss about metacognition…indeed? Research Papers in Education. doi:10.1080/02671522.2010.550011.

  • Mokhtari, K., & Reichard, C. (2002). Assessing students’ metacognitive awareness of reading strategies. Journal of Educational Psychology, 94(2), 249–259.

    Article  Google Scholar 

  • Muis, K. R., Winne, P. H., & Jamieson-Noel, D. (2007). Using a multitrait-multimethod analysis to examine conceptual similarities of three self-regulated learning inventories. British Journal of Educational Psychology, 77, 177–195.

    Article  Google Scholar 

  • Nelson, T. O. (1996). Consciousness and metacognition. American Psychologist, 51, 102–116.

    Article  Google Scholar 

  • Pressley, M., & Afflerbach, P. (1995). Verbal protocols of reading: The nature of constructively responsive reading. Hillsdale: Erlbaum.

    Google Scholar 

  • Richardson, J. T. E. (2004). Methodological issues in questionnaire-based research on student learning in higher education. Educational Psychology Review, 16, 347–358.

    Article  Google Scholar 

  • Samuelstuen, M. S., & Bråten, I. (2007). Examining the validity of self-reports on scales measuring students’ strategic processing. British Journal of Educational Psychology, 77, 351–378.

    Article  Google Scholar 

  • Schellings, G. L. M. (2011). Applying learning strategies questionnaires: Problems and possibilities. Metacognition and Learning, 6, 91–109.

    Article  Google Scholar 

  • Schellings, G. L. M., & Broekkamp, H. (2011). Signaling task awareness in think-aloud protocols from students selecting relevant information from text. Metacognition and Learning, 6, 65–82.

    Article  Google Scholar 

  • Schellings, G. L. M., & Van Hout-Wolters, B. H. A. M. (2006). Leertaken in de mens-en maatschappijvakken in de tweede fase van het voortgezet onderwijs: Nieuwe onderwijsmethodes in de praktijk. [Learning tasks in the human and society courses in the secondary phase in secondary education: Instructional methods in practice. VELON. Tijdschrift voor Lerarenopleiders, 27, 23–34.

    Google Scholar 

  • Schellings, G. L. M., & Van Hout-Wolters, B. H. A. M. (2011). Measuring strategy use with self-report instruments: Theoretical and empirical considerations. Metacognition and Learning, 6, 83–90.

    Article  Google Scholar 

  • Schellings, G.L.M., Van Hout-Wolters, B.H.A.M., Veenman, M.V.J., & Meijer, J. (2007). Assessing metacognitive activities: Matching a questionnaire with thinking aloud. Paper presented on the 12th Biennial EARLI-Conference, Budapest, Hungary, August 28th –September 1st, 2007.

  • Schraw, G., & Moshman, D. (1995). Metacognitive theories. Educational Psychology Review, 7, 351–371.

    Article  Google Scholar 

  • Schwarz, N. (1999). Self-reports. How questions shape the answers. American Psychologist, 54, 93–105.

    Article  Google Scholar 

  • Van Hout-Wolters, B. H. A. M. (2000). Assessing self-directed learning. In P. R. J. Simons, J. van der Linden, & T. Duffy (Eds.), New learning (pp. 83–101). Dordrecht: Kluwer.

    Google Scholar 

  • Van Hout-Wolters, B. H. A. M. (2009). Leerstrategieën meten. Soorten meetmethoden en hun bruikbaarheid in onderwijs en onderzoek. [Measuring Learning strategies. Different kinds of assessment methods and their usefulness in education and research]. Pedagogische Studiën, 86, 110–103.

    Google Scholar 

  • Van Someren, M. W., Bernard, Y., & Sandberg, J. (1993). The think-aloud method. Amsterdam: University of Amsterdam.

    Google Scholar 

  • Veenman, M. V. J. (2005). The assessment of metacognitive skills: What can be learned from multi-method designs? In C. Artelt & B. Moschner (Eds.), Lernstrategien und Metakognition: Implikationen für Forschung und Praxis (pp. 77–99). Münster: Waxmann.

    Google Scholar 

  • Veenman, M. V. J., & Alexander, P. (2011). Learning to self-monitor and self-regulate. In R. Mayer & P. Alexander (Eds.), Handbook of research on learning and instruction (pp. 197–218). New York: Routledge.

    Google Scholar 

  • Veenman, M. V. J., & Beishuizen, J. J. (2004). Intellectual and metacognitive skills of novices while studying texts under conditions of text difficulty and time constraint. Learning and Instruction, 14, 619–638.

    Google Scholar 

  • Veenman, M. V. J., & Van Cleef, D. (2007). Validity of assessing metacognitive skills for mathematic problem solving. In A. Efklides & M. H. Kosmidis (Eds.), 9th European Conference on Psychological Assessment. Program and abstracts (pp. 87–88). Thessaloniki: Aristotle University of Thessaloniki.

    Google Scholar 

  • Veenman, M. V. J., Elshout, J. J., & Groen, M. G. M. (1993). Thinking aloud: Does it affect regulatory processes in learning? Tijdschrift voor Onderwijsresearch, 18, 322–330.

    Google Scholar 

  • Veenman, M. V. J., Prins, F. J., & Verheij, J. (2003). Learning styles: Self-reports versus thinking-aloud measures. British Journal of Educational Psychology, 73, 357–372.

    Article  Google Scholar 

  • Veenman, M. V. J., Van Hout-Wolters, B. H. A. M., & Afflerbach, P. (2006). Metacognition and learning: Conceptual and methodological considerations. Metacognition and Learning, 1, 3–14.

    Article  Google Scholar 

  • Verheul, I., & Yang, J. (1986). Het traceren van leerprocessen via de hardop-leermethode en een procesvragenlijst. [Tracing learning processes through the learn-aloud method and a process questionnaire.]. In W. J. Van der Linden & J. M. Wijnstra (Eds.), Ontwikkelingen in de methodologie van onderwijsonderzoek. Lisse: Swets & Zeitlinger.

    Google Scholar 

  • Wernke, S., Wagener, U., Anschuetz, A., & Moschner, B. (2011). Assessing cognitive and metacognitive learning strategies in school children: Construct validity and arising questions. The International Journal of Research and Review, 6, 19–37.

    Google Scholar 

  • Winne, P. H., & Perry, N. E. (2000). Measuring self-regulated learning. In P. R. Pintrich, M. Boekaerts, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 531–567). Orlando: Academic Press.

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Research Institute of Child Development and Education, University of Amsterdam, Nieuwe Prinsengracht 130, 1018 VZ, Amsterdam, The Netherlands

    Gonny L. M. Schellings & Bernadette H. A. M. van Hout-Wolters

  2. Institute for Psychological Research, Department of Developmental and Educational Psychology, Leiden University, Wassenaarseweg 52, 2333 AK, Leiden, The Netherlands

    Marcel V. J. Veenman

  3. SCO-Kohnstamm Institution of the Faculty of Social and Behavioural Sciences, University of Amsterdam, Plantage Muidergracht 24, 1018 TV, Amsterdam, The Netherlands

    Joost Meijer

Authors
  1. Gonny L. M. Schellings
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bernadette H. A. M. van Hout-Wolters
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marcel V. J. Veenman
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Joost Meijer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gonny L. M. Schellings.

Additional information

Gonny L.M. Schellings. Research Institute of Child Development and Education, University of Amsterdam Spinozastraat 55, 1018 HJ Amsterdam, The Netherlands. Phone: +20-525 1599, E-mail: g.l.m.schellings@uva.nl

Current themes of research: Metacognition. Text studying. Learning activities.

Most relevant publications in the field of Psychology of Education:

Schellings, G.L.M. (2011). Applying learning strategies questionnaires: Problems and possibilities. Metacognition and Learning, 6, 91-109.

Schellings, G.L.M. & Broekkamp, H. (2011). Signaling task awareness in think-aloud protocols from students selecting relevant information from text. Metacognition and Learning, 6, 65-82.

Schellings, G.L.M. & Van Hout-Wolters, B.H.A.M. (2011). Measuring strategy use with self-report instruments: theoretical and empirical considerations. Metacognition and Learning, 6, 83-90.

Bernadette H.A.M. van Hout-Wolters. Research Institute of Child Development and Education, University of Amsterdam, Spinozastraat 55, 1018 HJ Amsterdam, The Netherlands. E-mail: b.h.a.m.vanhout-wolters@uva.nl

Current themes of research: Metacognition. Text studying. Learning activities.

Most relevant publications in the field of Psychology of Education:

Van Hout-Wolters, B.H.A.M. (2000). Assessing self-directed learning. In P.R.J. Simons, J. van der Linden & T. Duffy (Eds.), New Learning. (pp. 83-101). Dordrecht: Kluwer.

Van Hout-Wolters, B.H.A.M. (2009). Leerstrategieën meten. Soorten meetmethoden en hun bruikbaarheid in onderwijs en onderzoek. [Measuring Learning strategies. Different kinds of assessment methods and their usefulness in education and research]. Pedagogische Studiën, 86, 110-103.

Marcel V.J. Veenman. Institute for Psychological Research, Department of Developmental and Educational Psychology, Leiden University, Wassenaarseweg 52, 2333 AK Leiden, The Netherlands. E-mail: Veenman@fsw.leidenuniv.nl

Current themes of research: Metacognition.

Most relevant publications in the field of Psychology of Education:

Veenman, M.V.J. (2005). The assessment of metacognitive skills: What can be learned from multi-method designs? In C. Artelt & B. Moschner (Eds), Lernstrategien und Metakognition: Implikationen für Forschung und Praxis (pp. 77-99). Münster: Waxmann.

Veenman, M.V.J. (2011). Learning to self-monitor and self-regulate. In R. Mayer & P.Alexander (Eds), Handbook of research on learning and instruction (pp.197-218). New York: Routledge.

Joost Meijer. SCO-Kohnstamm Institution of the Faculty of Social and Behavioural Sciences, University of Amsterdam, Plantage Muidergracht 24, 1018 TV Amsterdam. E-mail: j.meijer@uva.nl

Current themes of research: Metacognition.

Most relevant publications in the field of Psychology of Education:

Meijer, J., Veenman, M.V.J., & van Hout-Wolters, B.H.A.M. (2006). Metacognitive activities in text studying and problem solving: Development of a taxonomy. Educational Research and Evaluation, 12, 209-237.

Meijer, J., Veenman, M.V.J., & van Hout-Wolters, B.H.A.M. (2011). Multi-domain, multi-method measures of metacognitive activity: What is all the fuss about metacognition…indeed? Research Papers in Education, DOI: 10.1080/02671522.2010.550011 1WR.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Schellings, G.L.M., van Hout-Wolters, B.H.A.M., Veenman, M.V.J. et al. Assessing metacognitive activities: the in-depth comparison of a task-specific questionnaire with think-aloud protocols. Eur J Psychol Educ 28, 963–990 (2013). https://doi.org/10.1007/s10212-012-0149-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10212-012-0149-y

Keywords

Search

Navigation