Skip to main content
SpringerLink
Log in

Using the “Indicators of Engaged Learning Online” Framework to Evaluate Online Course Quality

  • Original Paper
  • Published:
TechTrends Aims and scope Submit manuscript

Abstract

This article is a case study of the use of the Indicators of Engaged Learning Online (IELO) framework (See Appendix 1) as a guide to evaluate the quality of online courses. The framework lends itself well to measures of engagement, particularly, in terms of online course design because of its comprehensiveness. Six online courses were evaluated for quality in terms of engaged learning based on thirty indicators contained within the framework. Results ranged from a score of 21 to 71 out of a potential total score of 90. This 0–90 scale represented a continuum of passive to engaged learning. The purpose of the pilot study was to explore how the Indicators of Engaged Learning Online (IELO) framework could be used as a tool for evaluating the quality of online courses by instructional designers (IDs) and instructors. Insights into the practical use of the IELO framework and the need for improved guidelines for IDs and instructors as they assess the amount of student engagement designed in a course are provided. Recommendations for practice have implications for both aspects of engagement: how a course is designed and how it is delivered.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Allen, I., & Seaman, J. (2015). Grade level: Tracking online education in the United States. Babson Survey Research Group.

  • ASHE Higher Education Report (2014). Effects from student engagement online, 40(6), 67–73. https://doi.org/10.1002/aehe.20018.

  • Association of American Colleges and Universities (AAC&U). (2007). College learning for the new global century: A report from the national leadership council for liberal education & America's promise. Washington, DC: Author.

    Google Scholar 

  • Bigatel, P., & Williams, V. (2015). Measuring student engagement in an online program. Online journal of distance learning and administration. Retrieved from http://www.westga.edu/~distance/ojdla/summer182/bigatel_williams182.html.

  • Blackmon, S. J., & Major, C. (2012). Student experiences in online courses: A qualitative research synthesis. The Quarterly Review of Distance Education, 13(2), 77–85.

    Google Scholar 

  • Boston, W. E., & Ice, P. (2011). Assessing retention in online learning: An administrative perspective. Online Journal of Distance Learning Administration. Retrieved from, http://www.westga.edu/~distance/ojdla/summer142/boston_ice142.html.

  • Boston, W., Diaz, S. R., Gibson, A. M., Ice, P., Richardson, J., & Swan, K. (2010). An exploration of the relationship between indicators of the Community of Inquiry framework and retention in online programs. Journal of Asynchronous Learning Networks, 14(1), 3–19.

    Google Scholar 

  • Chickering, A. W., & Gamson, Z. F. (1987). Seven principles for good practice in undergraduate education. AAHE Bulletin, 3, 7.

    Google Scholar 

  • Czerkawski, B. C., & Lyman III, E. W. (2016). An instructional design framework for fostering student engagement in online learning environments. TechTrends, 60, 532–539. https://doi.org/10.1007/s11528-016-0110-z.

    Article  Google Scholar 

  • Dayton, D., & Vaughn, M. M. (2007). Developing a quality assurance process to guide the design and assessment of online courses. Technical Communication, 54(4), 475–489.

    Google Scholar 

  • Edel-Malizia, S., & Brautigam, K. (2014). Gauging the quality of online learning by measuring 21st century engagement, in Proceedings of the 13th European Conference on E-Learning, Copenhagen (pp. 700–703). Copenhagen: Aalborg University.

    Google Scholar 

  • Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59–109.

    Article  Google Scholar 

  • Garrison, D. R. (2007). Online community of inquiry review: Social, cognitive, and teaching presence issues. Journal of Asynchronous Learning Networks, 7(1), 61–72.

    Google Scholar 

  • Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. American Journal of Distance Education, 15(1), 7–23.

    Article  Google Scholar 

  • Gray, J. A., & DiLoreto, M. (2016). The effects of student engagement, student satisfaction, and perceived learning in online learning environments. International Journal of Educational Leadership Preparation, 11(1), n1.

    Google Scholar 

  • Jones, B. F., Valdez, G., Nowakowski, J., & Rasmussen, C. (1995). Plugging in: Choosing and using educational technology. Washington, DC: North central regional educational lab., oak brook, IL; Council for Educational Development and Research.

    Google Scholar 

  • Jung, I. (2011). The dimensions of e-learning quality: From the learner’s perspective. Educational Technology Research and Development, 59(4), 445–464.

    Article  Google Scholar 

  • Kahn, P. E. (2014). Theorising student engagement in higher education. British Educational Research Journal, 40(6), 1005–1018.

    Article  Google Scholar 

  • Kuh, G. D. (2001). Assessing what really matters to student learning inside the national survey of student engagement. Change: The Magazine of Higher Learning, 33(3), 10–17.

    Article  Google Scholar 

  • Kuh, G. D. (2003). What we’re learning about student engagement from NSSE: Benchmarks for effective educational practices. Change, 35(2), 24–32.

    Article  Google Scholar 

  • Kuh, G. D. (2008). Excerpt from high-impact educational practices: What they are, who has access to them, and why they matter. Association of American Colleges and Universities. Retrieved from http://ueeval.ucr.edu/teaching_practices_inventory/Kuh_2008.pdf.

  • Lehman, R. M., & Conceição, S. C. O. (2014). Motivating and retaining online students: Research-based strategies that work. San Francisco: Jossey-Bass/Wiley.

    Google Scholar 

  • Means, B. (1993). Using technology to support education reform (pp. 20402–29328). US Government Printing Office, Superintendent of Documents, Mail Stop: SSOP, Washington, DC.

    Google Scholar 

  • Menchaca, M. P., & Bekele, T. A. (2008). Learner and instructor identified success factors in distance education. Distance Education, 29(3), 231–252.

    Article  Google Scholar 

  • Morris, L. V., & Finnegan, C. L. (2009). Best practices in predicting and encouraging student persistence and achievement online. Journal of College Student Retention, 10(1), 55–64.

    Article  Google Scholar 

  • National Survey of Student Engagement. (2000). Improving the college experience: National benchmarks of effective educational practice. Bloomington: Indiana University Center for Postsecondary Research and Planning.

    Google Scholar 

  • North Central Regional Educational Laboratory & the Metiri Group. (2003). enGuage 21st century skills: Literacy in the digital age. Retrieved September 30, 2016, from http://pict.sdsu.edu/engauge21st.pdf.

  • Phipps, R., & Merisotis, J. (2000). Quality on the line: Benchmarks for success in internet-based distance education. Retrieved from http://www.ihep.org/research/publications/quality-line-benchmarks-success-internet-based-distance-education.

  • Quality Matters (QM). (2011). Helping you deliver on your online promise. Retrieved from https://www.qualitymatters.org/.

  • Revere, L., & Kovach, J. V. (2011). Online technologies for engaged learning: A meaningful synthesis for educators. The Quarterly Review of Distance Education, 12(2), 113–124.

    Google Scholar 

  • Robinson, C. C., & Hullinger, H. (2008). New benchmarks in higher education: Student engagement in online learning. Journal of Education for Business, 84(2), 101–109.

    Article  Google Scholar 

  • Rovai, A. P., & Wighting, M. J. (2005). Feelings of alienation and community among higher education students in a virtual classroom. The Internet and Higher Education, 8, 97–110.

    Article  Google Scholar 

  • Ryan, R. M., & Deci, E. L. (2000). Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary Educational Psychology, 25, 54–67.

    Article  Google Scholar 

  • Southard, S., & Mooney, M. (2015). A comparative analysis of distance education quality assurance standards. The Quarterly Review of Distance Education, 16(1), 55–68.

    Google Scholar 

  • Street, H. D. (2010). Factors influencing a learner’s decision to drop-out or persist in higher education distance learning. Online Journal of Distance Learning Administration, 13(4). Retrieved from http://wp.westga.edu/wp-content/uploads/sites/40/ojdla/winter134/street134.pdf.

  • Swaner, L., & Brownell, J. (2009). Outcomes of high impact practices for undeserved students. A review of the literature. Prepared for the Association of American Colleges and Universities.

  • Wyatt, L. G. (2011). Nontraditional student engagement: Increasing adult student success and retention. The Journal of Continuing Higher Education, 59(1), 10–20.

    Article  Google Scholar 

  • Yin, K. (2009). Case study research: Design and method (4th ed.). Thousand Oaks: Sage.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. World Campus, Faculty Development, Penn State University, 223 Outreach Bldg., University Park, PA, 16802, USA

    Paula M. Bigatel

  2. Education Technology Services, Penn State University, 003A Shields Bldg., University Park, PA, 16802, USA

    Stephanie Edel-Malizia

Authors
  1. Paula M. Bigatel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stephanie Edel-Malizia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Paula M. Bigatel.

Ethics declarations

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Conflict of Interest

Paula M Bigatel declares that she has no conflict of interest. Stephanie Edel-Malizia declares that she has no conflict of interest.

Appendices

Appendix 1

figure a
figure b

Appendix 2

Faculty Engagement Checklist

Table 3 Take a couple of minutes to think about the events in this class. Did you engage your students? Let’s look at what makes learning engaging
Full size table

Student Engagement Checklist

Table 4 Take a couple of minutes to think about the events in this class. Did you feel engaged? Let’s look at what makes learning engaging
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bigatel, P.M., Edel-Malizia, S. Using the “Indicators of Engaged Learning Online” Framework to Evaluate Online Course Quality. TechTrends 62, 58–70 (2018). https://doi.org/10.1007/s11528-017-0239-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11528-017-0239-4

Keywords

Search

Navigation