Construct and consequential validity for learning analytics based on trace data

doi:10.1016/j.chb.2020.106457

Computers in Human Behavior

Volume 112, November 2020, 106457

https://doi.org/10.1016/j.chb.2020.106457 Get rights and content

Highlights

•
Trace data are increasingly useful in developing learning analytics.
•
“Raw” data are biased by the theory that recommends observing those data.
•
Self-regulating learners acting as agents complicate reliability of trace data.
•
Reliability of trace data concerns dynamic events, not static aspects of a measure.
•
Generalizability over facets of data sets limits on reliability and validity.

Abstract

This article analyzes the concept of validity to set out key factors bearing on claims about validity in general and particularly regarding learning analytics. Because uses of trace data in learning analytics are increasing rapidly, specific consideration is given to reliability of trace data and their role in claiming validity for interpretations grounded on trace data. This analysis reveals the essential and inescapable role of theory in deciding what trace data should be gathered and how trace data can contribute to recommendations for improving learning, one main goal for generating and using learning analytics.

Section snippets

Construct and consequential validity for learning analytics based on trace data

Today's learners regularly use modern software tools to search for information, annotate information they find online, study for exams, collaborate on projects, analyze data gathered in local or distantly run experiments, and compose essays. With learners' consent, some software they use can be engineered to record detailed and extensive trace data (Winne, 1982, 2010), a kind of data increasingly sought to develop learning analytics. A trace datum is “an objective (i.e., readily agreed to)

One goal for learning analytics

To learn new information of kinds educators and the larger society deem important, I posit learners make choices about which cognitive operations they will apply to which particular information. That is, learners are intrinsically self-regulating (Winne, 1995). Whatever interventions instructional designers and whichever tools software developers might design, each learner's choices about how and what to learn are the proximal variables that shape whether learning happens and what is learned (

Reliability as a faceted event

A first step toward examining the concept of reliability as it applies to events is describing what an event is. Systems in which events happen have, at any particular point in time, a state. A state is described by particular values for one or several attributes. For example, suppose a learner is viewing text and has access to a tool to mark the text, e.g., by dragging a cursor across it to shade white space surrounding letters that create a yellow highlight. Knowing the values of attributes

Reliability of data in learning analytics

Like other sciences, learning analytics seeks, first, to identify facets that are sources of variance in learning events; second, to quantify the degree of each source's variability; and, third, to recommend changes in learning practices that are valid in the sense that the analytic leads to the result that is intended and that has value, so learners (or instructors) will act on information the analytic supplies. The first two steps establish that changing a facet's value or quality produces

Validity in the context of learning analytics

Messick defined validity as the “degree to which empirical evidence and theoretical rationale supports the adequacy and appropriateness of inferences and actions based on test scores” (Messick, 1989p. 5; emphasis is mine). Later, he described a particular view of validity regarding actions, called consequential validity, as the appropriateness of shaping instructional processes guided, in part, by assessment data (Messick, 1995).

Adopting Messick's widely acknowledged descriptions, it becomes

Validity as a network of issues

The concepts of validity and reliability are complex. The validity of claims and of the match between intended and actual consequences are conditional on the generalizability of data needed to assert validity. As is likely obvious, the definition of a trace datum quoted at the beginning of this article explicitly binds issues of reliability to what a datum represents, i.e., to some theoretical foundation. This itself begs questions about validity.

Most reports in the educational and

CRediT authorship contribution statement

Philip H. Winne: Conceptualization, Writing - original draft, Writing - review & editing.

References (13)

L.F. Barrett
Zombie ideas
APS Observer
(2019)
L.J. Cronbach et al.
Theory of generalizability: A liberalization of reliability theory
British Journal of Statistical Psychology
(1963)
S. Messick
Meaning and values in test validation: The science and ethics of assessment
Educational Researcher
(1989)
S. Messick
Standards of validity and the validity of standards in performance assessment
Educational Measurement: Issues and Practice
(1995)
G.F. Striedter
Principles of brain evolution
(2005)

There are more references available in the full text version of this article.

Cited by (60)

Intense, turbulent, or wallowing in the mire: A longitudinal study of cross-course online tactics, strategies, and trajectories
2023, Internet and Higher Education
Research has repeatedly demonstrated that students with effective learning strategies are more likely to have better academic achievement. Existing research has mostly focused on a single course or two, while longitudinal studies remain scarce. The present study examines the longitudinal sequence of students' strategies, their succession, consistency, temporal unfolding, and whether students tend to retain or adapt strategies between courses. We use a large dataset of online traces from 135 students who completed 10 successive courses (i.e., 1350 course enrollments) in a higher education program. The methods used in this study have shown the feasibility of using trace data recorded by learning management systems to unobtrusively trace and model the longitudinal learning strategies across a program. We identified three program-level strategy trajectories: a stable and intense trajectory related to deep learning where students used diverse strategies and scored the highest grades; a fluctuating interactive trajectory, where students focused on course requirements, scored average grades, and were relatively fluctuating; and a light trajectory related to surface learning where students invested the least effort, scored the lowest grades, and had a relatively stable pathway. Students who were intensely active were more likely to transfer the intense strategies and therefore, they were expected to require less support or guidance. Students focusing on course requirements were not as effective self-regulators as they seemed and possibly required early guidance and support from teachers. Students with consistent light strategies or low effort needed proactive guidance and support.
Reinforcement learning for automatic detection of effective strategies for self-regulated learning
2023, Computers and Education: Artificial Intelligence
Self-regulated learning (SRL) is an essential skill for achieving one's learning goals, particularly in Digital Learning Environments (DLEs) where system support is often limited compared to traditional classroom settings. However, research has found that learners often struggle to adapt their behaviour to the self-regulatory demands of DLEs. Furthermore, existing SRL analysis tools have limited utility for real-time or individualized prescriptive support of a learner's SRL strategy during a study session.
In response to these challenges, we propose a novel approach using reinforcement learning as a framework to optimize the sequence of SRL processes for a learning task. This framework allows us to model and optimize the SRL strategy as a sequential decision-making problem, where each decision corresponds to an SRL process. The goal is to find an optimal sequence of decisions that maximizes performance in learning outcomes such as assessment score or learning gains.
We compare the performance of our reinforcement learning framework with other sequential machine learning tools, such as Long Short-Term Memory (LSTM) neural networks and Genetic Algorithms (GA). The results of our study show that our reinforcement learning model outperforms GA and LSTM models in optimizing SRL strategy.
The contributions of this work can facilitate the development of a tool which can detect sub-optimal SRL strategy in real-time and enable individualized SRL focused scaffolding.
A Case Study on University Student Online Learning Patterns Across Multidisciplinary Subjects
2024, ACM International Conference Proceeding Series
Measurement of Self-regulated Learning: Strategies for mapping trace data to learning processes and downstream analysis implications
2024, ACM International Conference Proceeding Series
Using learning analytics to measure self-regulated learning: A systematic review of empirical studies in higher education
2024, Journal of Computer Assisted Learning
Investigating the reliability of aggregate measurements of learning process data: From theory to practice
2024, Journal of Computer Assisted Learning

View all citing articles on Scopus

^☆: Faculty of Education, EDB 8515, Simon Fraser University, 8888 University Drive, Burnaby, BC V5A 1S6, Canada.

^☆☆: This work was supported by grants to Philip H. Winne from the Social Sciences and Humanities Research Council of Canada #435-2016-0379 and Simon Fraser University.

View full text

Full length articleConstruct and consequential validity for learning analytics based on trace data☆,☆☆

Highlights

Abstract

Section snippets

Construct and consequential validity for learning analytics based on trace data

One goal for learning analytics

Reliability as a faceted event

Reliability of data in learning analytics

Validity in the context of learning analytics

Validity as a network of issues

CRediT authorship contribution statement

Zombie ideas

APS Observer

Theory of generalizability: A liberalization of reliability theory

British Journal of Statistical Psychology

Meaning and values in test validation: The science and ethics of assessment

Educational Researcher

Standards of validity and the validity of standards in performance assessment

Educational Measurement: Issues and Practice

Principles of brain evolution

Full length article
Construct and consequential validity for learning analytics based on trace data☆,☆☆