Skip to main content
SpringerLink
Log in

Exploring the potential of LMS log data as a proxy measure of student engagement

  • Published:
Journal of Computing in Higher Education Aims and scope Submit manuscript

Abstract

This study examines the relationship between log data of student activity in learning management systems and self-reported student engagement survey scores. Log data has the potential to serve as a meaningful proxy for survey scores. Should this be the case, log data could be used as a minimally disruptive and scalable approach to quickly identify who needs help, evaluate design, and personalize instruction. We correlated LMS log data variables to student engagement survey scores to study the relationship between these two sources of data. Overall, log data was not a statistically significant proxy measure of students’ self-reported cognitive and emotional engagement. Our results underscore the complexity of learning and the relationship between observed and reported cognitive and emotional states. Future educational research using log data will need to account for other factors that help explain trends in student engagement. Exploring the Potential of LMS Log Data as a Proxy Measure of Student Engagement.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Appleton, J. J., Christenson, S. L., Kim, D., & Reschly, A. L. (2006). Measuring cognitive and psychological engagement: Validation of the student engagement instrument. Journal of School Psychology, 44, 427–445. doi:10.1016/j.jsp.2006.04.002.

    Article  Google Scholar 

  • Baker, R. S. J., Gowda, S. M., Wixon, M., Kalka, J., Wagner, A. Z., Aleven, V., Rossi, L. et al. (2012). Towards sensor-free affect detection in cognitive tutor algebra. In K. Yacef, O. Zaïane, H. Hershkovitz, M. Yudelson, & J. Stamper (Eds.), Proceedings of the 5th international conference on educational data mining (pp. 126–133), Chania. Retrieved April 27, 2015 from http://educationaldatamining.org/EDM2012/index.php?page=proceedings.

  • Baumeister, R. F., Vohs, K. D., DeWall, C. N., & Zhang, L. (2007). How emotion shapes behavior: Feedback, anticipation, and reflection, rather than direct causation. Personality and Social Psychology Review, 11(2), 167–203. doi:10.1177/1088868307301033.

    Article  Google Scholar 

  • Beer, C., Clark, K., & Jones, D. (2010). Indicators of engagement. In Proceedings of ASCILITE 2010 (pp. 75–86), Sydney. Retrieved November 15, 2015 from http://ascilite.org.au/conferences/sydney10/procs/Beer-full.pdf.

  • Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1), 289–300. Retrieved from http://www.jstor.org.byui.idm.oclc.org/stable/2346101.

  • Bienkowski, M., Feng, M., & Means, B. (2012). Enhancing teaching and learning through educational data mining and learning analytics: An issue brief. Washington, DC: SRI International. Retrieved from http://www.ed.gov/edblogs/technology/files/2012/03/edm-la-brief.pdf.

  • Bodily, R. G., Graham, C. R., & Bush, M. D. (2017). Online learner engagement: Opportunities and challenges with using data analytics. Educational Technology, 57(1), 10–18.

    Google Scholar 

  • Borup, J., Graham, C., & Davies, R. (2013). The nature of parental interactions in an online charter school. American Journal of Distance Education, 27, 40–55. doi:10.1080/08923647.2013.754271.

    Article  Google Scholar 

  • Cocea, M., & Weibelzahl, S. (2011). Disengagement detection in online learning: Validation studies and perspectives. IEEE Transactions on Learning Technologies, 4, 114–124. doi:10.1109/TLT.2010.14.

    Article  Google Scholar 

  • Cooley, R., Mobasher, B., & Srivastava, J. (2013). Data preparation for mining world wide web browsing patterns. Knowledge and Information Systems, 1(1), 5–32. doi:10.1007/BF03325089.

    Article  Google Scholar 

  • D’Mello, S. K., & Graesser, A. (2011). The half-life of cognitive-affective states during complex learning. Cognition and Emotion, 25, 1299–1308. doi:10.1080/02699931.2011.613668.

    Article  Google Scholar 

  • D’Mello, S., & Graesser, A. (2012). AutoTutor and affective AutoTutor. ACM Transactions on Interactive Intelligent Systems, 2(4), 1–39. doi:10.1145/2395123.2395128.

    Article  Google Scholar 

  • Dixson, M. D. (2010). Creating effective student engagement in online courses: What do students find engaging? Journal of the Scholarship of Teaching and Learning, 10(2), 1–13. Retrieved November 15, 2015 from http://josotl.indiana.edu/.

  • Drutsa, A., & Serdyukov, P. (2015). Future user engagement prediction and its application to improve the sensitivity of online experiments. In WWW 2015 (pp. 256–266), Florence. doi:10.1145/2736277.2741116.

  • Elliot, J. (2004). Multimethod approaches in educational research. International Journal of Disability, Development and Education, 51, 135–149. doi:10.1080/10349120410001687364.

    Article  Google Scholar 

  • Ferguson, R. (2012). Learning analytics: Drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5/6), 304–317. doi:10.1504/IJTEL.2012.051816.

    Article  Google Scholar 

  • Finn, J. D., & Owings, J. (2006). The adult lives of at-risk students: The roles of attainment and engagement in high school. Statistical Analysis Report (NCES 2006-328). Washington, DC: U.S. Department of Education, National Center for Education Statistics. Retrieved November 15, 2015 from http://eric.ed.gov/?id=ED491285.

  • Finn, J. D., & Zimmer, K. S. (2012). Student engagement: What is it? Why does it matter? In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 97–131). Boston, MA: Springer. doi:10.1007/978-1-4614-2018-7_5.

    Chapter  Google Scholar 

  • Fredricks, J. A., Blumenfeld, P., Friedel, J., & Paris, A. (2005). School engagement. In K. A. Moore & L. Lippman (Eds.), What do children need to flourish? Conceptualizing and measuring indicators of positive development (pp. 305–321). New York, NY: Kluwer Academic/Plenum Press. doi:10.1007/0-387-23823-9_19.

    Chapter  Google Scholar 

  • Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74, 59–109. doi:10.3102/00346543074001059.

    Article  Google Scholar 

  • Fredricks, J. A., & McColskey, W. (2012). The measurement of student engagement: A comparative analysis of various methods and student self-report instruments. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 763–782). Boston, MA: Springer. doi:10.1007/978-1-4614-2018-7_37.

    Chapter  Google Scholar 

  • Fredricks, J., McColskey, W., Meli, J., Mordica, J., Montrosse, B., & Mooney, K. (2011). Measuring student engagement in upper elementary through high school: A description of 21 instruments. (Issues & Answers Report, REL 2011–No. 098). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southeast. Retrieved April 27, 2015 from http://ies.ed.gov/ncee/edlabs/regions/southeast/pdf/REL_2011098.pdf.

  • Fulmer, S. M., & Frijters, J. C. (2009). A review of self-report and alternative approaches in the measurement of student motivation. Educational Psychology Review, 21, 219–246. doi:10.1007/s10648-009-9107-x.

    Article  Google Scholar 

  • Gobert, J. D., Baker, R. S., & Wixon, M. B. (2015). Operationalizing and detecting disengagement within online science microworlds. Educational Psychologist, 50, 43–57. doi:10.1080/00461520.2014.999919.

    Article  Google Scholar 

  • Halverson, L.R., & Graham, C. R. (2015). Learner engagement in blended learning environments (Submitted for publication).

  • Halverson, L., Spring, K., Huyett, S., Henrie, C., & Graham, C. (2016). Blended learning research in higher education and K-12 settings. In J. M. Spector, B. B. Lockee, & M. D. Childress (Eds.), Learning, design, and technology: An international compendium of theory, research, practice and policy. Berlin: Springer. doi:10.1007/978-3-319-17727-4_31-1.

    Google Scholar 

  • Hawkins, A., Graham, C. R., Sudweeks, R. R., & Barbour, M. K. (2013). Academic performance, course completion rates, and student perception of the quality and frequency of interaction in a virtual high school. Distance Education, 34, 64–83. doi:10.1080/01587919.2013.770430.

    Article  Google Scholar 

  • Heinemann, M. H. (2005). Teacher-student interaction and learning in online theological education. Part II: Additional theoretical frameworks. Christian Higher Education, 4(4), 277–297. doi:10.1080/15363750500182794.

    Article  Google Scholar 

  • Henrie, C., Bodily, R., Manwaring, K., & Graham, C. (2015). Exploring intensive longitudinal measures of student engagement in blended learning. The International Review of Research in Open and Distributed Learning, 16(3), 131–155. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/2015.

  • Henrie, C. R., Halverson, L. R., & Graham, C. R. (2015b). Measuring student engagement in technology-mediated learning: A review. Computers & Education, 90, 36–53. doi:10.1016/j.compedu.2015.09.005.

    Article  Google Scholar 

  • Henrie, C. R., Larsen, R., Manwaring, K., Halverson, L. R., & Graham, C. R. (2016). Validation of a longitudinal activity-level measure of student engagement (Submitted for publication).

  • Hollands, F., & Bakir, I. (2015). Efficiency of automated detectors of learner engagement and affect compared with traditional observation methods. New York, NY: Center for Benefit-Cost Studies of Education, Teachers College, Columbia University. Retrieved November 15, 2015 from http://cbcse.org/

  • Janosz, M. (2012). Part IV commentary: Outcomes of engagement and engagement as an outcome: Some consensus, divergences, and unanswered questions. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 695–703). Boston, MA: Springer. doi:10.1007/978-1-4614-2018-7_33.

    Chapter  Google Scholar 

  • Jordan, K. (2014). Initial trends in enrollment and completion of massive open online courses. The International Review of Research in Open and Distance Learning, 15(1), 133–159. Retrieved April 27, 2015 from http://www.irrodl.org/index.php/irrodl/article/view/1651.

  • Kuh, G. D. (2001). Assessing what really matters to student learning: Inside the national survey of student engagement. Change: The Magazine of Higher Learning, 33(3), 10–17. Retrieved April 27, 2015 from http://www.jstor.org/stable/40165768.

  • Kuh, G. D., Kinzie, J., Buckley, J. A., Bridges, B. K., & Hayek, J. C. (2007). Piecing together the student success puzzle: Research, propositions, and recommendations. ASHE Higher Education Report, 32(5), 1–182. doi:10.1002/aehe.3205.

    Article  Google Scholar 

  • Lawson, M. A., & Lawson, H. A. (2013). New conceptual frameworks for student engagement research, policy, and practice. Review of Educational Research, 83, 432–479. doi:10.3102/0034654313480891.

    Article  Google Scholar 

  • Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. Computers & Education, 54, 588–599. doi:10.1016/j.compedu.2009.09.008.

    Article  Google Scholar 

  • Morris, L. V., Finnegan, C., & Wu, S.-S. (2005). Tracking student behavior, persistence, and achievement in online courses. The Internet and Higher Education, 8, 221–231. doi:10.1016/j.iheduc.2005.06.009.

    Article  Google Scholar 

  • Oncu, S., & Cakir, H. (2011). Research in online learning environments: Priorities and methodologies. Computers & Education, 57, 1098–1108. doi:10.1016/j.compedu.2010.12.009.

    Article  Google Scholar 

  • Park, S. (2015). Examining learning experience in two online courses using web logs and experience sampling method (ESM). In B. Hokanson, G. Clinton, & M. W. Tracey (Eds.), The design of learning experience (pp. 269–287). Boston, MA: Springer. doi:10.1007/978-3-319-16504-2_18.

    Chapter  Google Scholar 

  • Prince, S. A., Adamo, K. B., Hamel, M. E., Hardt, J., Gorber, S. C., & Tremblay, M. (2008). A comparison of direct versus self-report measures for physical activity in adults: A systematic review. International Journal of Behavioral Nutrition and Physical Activity. doi:10.1186/1479-5868-5-56.

    Google Scholar 

  • Reeve, J., & Tseng, C.-M. (2011). Agency as a fourth aspect of students’ engagement during learning activities. Contemporary Educational Psychology, 36, 257–267. doi:10.1016/j.cedpsych.2011.05.002.

    Article  Google Scholar 

  • Reschly, A. L., & Christenson, S. L. (2012). Jingle, jangle, and conceptual haziness: Evolution and future directions of the engagement construct. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 3–20). New York, NY: Springer. doi:10.1007/978-1-4614-2018-7_1.

    Chapter  Google Scholar 

  • Russell, J., Ainley, M., & Frydenberg, E. (2005). Schooling issues digest: Student motivation and engagement (pp. 1–16), Canberra, AU. Retrieved April 27, 2015 from http://docs.education.gov.au

  • Siemens, G. (2013). Learning analytics: The emergence of a discipline. American Behavioral Scientist, 57, 1380–1400. doi:10.1177/0002764213498851.

    Article  Google Scholar 

  • Sinatra, G. M., Heddy, B. C., & Lombardi, D. (2015). The challenges of defining and measuring student engagement. Educational Psychologist, 55, 1–13. doi:10.1080/00461520.2014.1002924.

    Article  Google Scholar 

  • Skinner, E. A., Kindermann, T. A., & Furrer, C. J. (2009). A motivational perspective on engagement and disaffection: Conceptualization and assessment of children’s behavioral and emotional participation in academic activities in the classroom. Educational and Psychological Measurement, 69, 493–525. doi:10.1177/0013164408323233.

    Article  Google Scholar 

  • Skinner, E. A., & Pitzer, J. R. (2012). Developmental dynamics of student engagement, coping, and everyday resilience. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 21–44). Boston, MA: Springer. doi:10.1007/978-1-4614-2018-7_2.

    Chapter  Google Scholar 

  • Wang, Z., Bergin, C., & Bergin, D. A. (2014). Measuring engagement in fourth to twelfth grade classrooms: The classroom engagement inventory. School Psychology Quarterly, 29, 517–535. doi:10.1037/spq0000050.

    Article  Google Scholar 

  • Watson, J., Pape, L., Murin, A., Gemin, B., & Vashaw, L. (2014). Keeping pace with K-12 digital learning. Retrieved April 27, 2015 from http://www.kpk12.com/reports/.

Download references

Author information

Authors and Affiliations

  1. Brigham Young University-Idaho, Rexburg, ID, 83440, USA

    Curtis R. Henrie

  2. Brigham Young University, Provo, UT, 84602, USA

    Robert Bodily, Ross Larsen & Charles R. Graham

Authors
  1. Curtis R. Henrie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robert Bodily
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ross Larsen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Charles R. Graham
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Curtis R. Henrie.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Appendix

Appendix

Emotional engagement scale.

  1. 1.

    Did you enjoy this activity? Not at all 1 2 3 4 5 very much.

  2. 2.

    Was this activity interesting? Not at all 1 2 3 4 5 very much.

  3. 3.

    Did you wish you had been doing something else? Not at all 1 2 3 4 5 very much.

  4. 4.

    Describe your mood during this activity: Excited 1 2 3 (Neither 4) 5 6 7 bored.

Cognitive engagement scale.

  1. 5.

    How well were you concentrating? Not at all 1 2 3 4 5 very much.

  2. 6.

    Describe your mood during this activity: Passive 1 2 3 (Neither 4) 5 6 7 active.

  3. 7.

    Describe your mood during this activity: Focused 1 2 3 (Neither 4) 5 6 7 distracted.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Henrie, C.R., Bodily, R., Larsen, R. et al. Exploring the potential of LMS log data as a proxy measure of student engagement. J Comput High Educ 30, 344–362 (2018). https://doi.org/10.1007/s12528-017-9161-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12528-017-9161-1

Keywords

Search

Navigation