Elsevier

Learning and Instruction

Volume 24, April 2013, Pages 62-66
Learning and Instruction

Calibration and confidence: Where to next?

https://doi.org/10.1016/j.learninstruc.2012.05.009Get rights and content

Abstract

One of the key feedback questions is “where to next?” and this article provides some directions as to where to next for research based on a review of the five articles in this special issue. The directions relate to the critical importance of calibration, the multidimensionality of calibration, the relation of calibration to self-regulation strategies, whether calibration is specific to the task or more general within the student, how to measure calibration, how much confidence should be given to partial knowledge when calibrating, the role of overconfidence and knowing “when one does not know”, and how to improve the accuracy of judgments.

Highlights

► Calibration is not a unidimensional notion. ► The importance of calibration to self-regulation. ► How to measure calibration. ► The role of overconfidence and knowing when one does not know. ► How to improve the accuracy of judgments.

Introduction

A major purpose of a special issue is to generate a solid basis for a topic, to generate excitement for further research on the topic, but most of all to bring a new perspective to a worthwhile problem. Calibration is often sidelined with questions raised as to why teachers should care so much about how students calibrate their learning. The five articles discussed in this commentary turn this way of thinking upside down. These articles show how students' calibrations of their own confidence and accuracy can be important enablers or barriers to learning; they show how critical it can be for teachers to attend to these calibrations, especially for struggling learners, and they provide many arguments in favor of the backward design approach to teaching. In terms of this backward design, teachers should:

  • explain to students exactly what they will be learning before beginning the teaching process;

  • provide clear success criteria for them;

  • adjudge what students already know and believe relative to these goals;

  • ensure teaching is directed to reducing the gap between what students believe they know and understand, and what teachers want them to know and understand;

  • ensure feedback is provided and received to reduce these gaps (Sadler, 1989).

The basic messages are: to attend to what students feel confident about, the accuracy they have of their prior learning, and their beliefs about the effectiveness of the learning methods they typically use—when they enter the classroom.

One of the strengths of these articles is to ‘bring back’ into the equation the (somewhat novel) notion of student calibration for effective teaching and learning. To bring back is important, as these notions of the power of students' beliefs and confidence have been a key part in many of the learning theories from Piaget (1962), Ausubel (1968), and Bandura (1986). So often the accuracy of these levels of confidence and beliefs is not sufficiently taken into account, with claims made about how students learn (best), how to help students construct knowledge, and how to ensure a greater number of students know more. A major message in these articles is that by ignoring students' beliefs concerning their confidence and accuracy, we are ignoring a major precursor to their learning.

From these articles, I can see at least nine major messages that should aim to spur further research on calibration. These messages relate to the importance of calibration, the polymorphous nature of calibration, the importance of self-regulation, whether the task is specific or more general, the measurement issues, the way students consider the role of partial knowledge, the place of knowing when one does not know, and the improvement of the accuracy of judgments.

Section snippets

The importance of calibration

In my review of more than 900 meta-analyses on the factors that most affect student learning, student expectations and self-reported grades came out on top (Hattie, 2009, 2012). This means that if a teacher were to say to a group of students “it is now test time, however, before you complete the test, I would like you to estimate the score or grade you think you would get,” the students would be very adept at such a task; they can calibrate their expected performance across a series of items

Calibration is not a unidimensional notion

These articles show that our ability to accurately judge our performance, confidence, strategies, or problem-solving is not only learned (and thus can be taught), but can also be an enabler or barrier when engaging in more challenging learning tasks. Similarly, the accuracy of judgments by teachers about what students can or cannot do is essential to engaging students in learning and successful teaching. Calibration of the gap between a student's current desired performance should inform the

The importance of calibration to self-regulation

It seems a tautology to claim that learners' calibration of their confidence and accuracy are key parts of metacognition and self-regulation. There can be little, if any, self-regulation unless students have some knowledge or beliefs about their current and desired learning state. This involves not only a sense of accuracy about both states, but also a sense of confidence that the student has the required learning or study strategies to reduce the gap – without this confidence, the student

Is calibration specific to the task or more general to the student?

Dinsmore and Parkinson (2013) were interested in students' explanations for their confidence ratings. They use Bandura's (1986) model of reciprocal determinism to show the personal, behavioral, and environmental factors that influence the forming of such ratings for students. They showed that students were able to take multiple factors into account when making their confidence judgments, including text characteristics, item characteristics, prior knowledge, and guessing (in that order),

How to measure calibrations

The core of the notion of calibration is a discrepancy between our judgment and the accuracy of the situation. Discrepancy scores, also known as change or gain scores, have a long and often, notorious history, which many of the arguments in these papers rehearse in the context of calibration. It has long been known that discrepancy scores tend to have low reliability under certain circumstances, leading to Lord (1956) noting that “differences between scores tend to be much more unreliable than

How much confidence should be given to partial knowledge?

In the 1970–1980's there was much work on confidence or the differential weighting of multiple-choice items with the objective of allowing for partial information in the distracters, rather than all information residing in the correct response. Wang and Stanley (1970), however, showed that any extra information gained from asking students to declare probabilities or weightings on items to reflect their confidence in choosing the correct answer, led to minimal changes in the estimates of

The role of overconfidence and knowing when one does not know

Over the past ten years I have overseen the building and implementation of a national assessment reporting system for elementary and high schools (Hattie, Brown, & Keegan, 2005). In one of the research projects we asked teachers to estimate the difficulty of items that had known difficulty properties. In general, elementary teachers underestimated the average difficulty of the items by about a year and high school teachers overestimated by about a year. In another part of the application,

How to improve the accuracy of judgments

These articles hint at five major directions for teachers to improve the accuracy of judgments. First, van Loon et al. (2013) demonstrated that the way to reverse the inaccuracy of judgments is to focus learners' attention on valid cues when judging learning, particularly when these are delayed, rather than immediate judgments. There may need to be some time between reflecting on the performance of a task and the accuracy of this performance. This relates to the comments above about the

References (27)

  • J.A.C. Hattie

    Visible learning: A synthesis of 800+ meta-analyses on achievement

    (2009)
  • J.A.C. Hattie

    Visible learning for teachers. Maximizing impact on achievement

    (2012)
  • J.A.C. Hattie et al.

    A national teacher-managed, curriculum-based assessment system: Assessment Tools for Teaching & Learning (asTTle)

    International Journal of Learning

    (2005)
  • Cited by (75)

    • Using automated feedback to develop writing proficiency

      2021, Computers and Composition
      Citation Excerpt :

      There is limited evidence to suggest that use of AWE leads to transfer (Stevenson & Phakiti, 2014)—transfer is the ability to acquire skills in one context and effectively apply them in another (National Research Council, 2000). Theoretically, by (a) facilitating a rapid practice-feedback loop (Kellogg et al., 2010), (b) helping students learn the criteria for high quality writing (see MacArthur 2016), and (c) aiding students in better monitoring and calibrating their performance against these criteria (see Hattie 2013), ongoing use of AWE may help students gradually learn to be better writers when composing independently, without the use of AWE. If use of AWE was associated with gains in independent composing skills, this would be evidence of unaided transfer.

    View all citing articles on Scopus
    View full text