Abstract
To better understand the sustainable development of open educational resources (OER), this paper aimed to break through the original ‘fixed’, highly structured evaluation system and develop an open and flexible “1 + X” evaluation framework of OER by using the modified Delphi method. “1” refers to the mandatory basic criteria that the repository must reflect essential attributes and core functions, which are consist of four components: information content, learning process, system technology, and user experience. “X” refers to optional distinctive criteria that the repository shall focus on one or more developmental orientations, which may include: resource evolution, international collaboration, inclusiveness and equality, open educational practices, and value-added services. Optional “X” criteria, in line with the criteria’s diversity aims, can lead to the sustainable development of OER and globalized lifelong learning. The framework of this study provides a basis for the evaluation and optimization of existing OER repositories, and also serves as functional scaffolding for the construction of new repositories.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Achieve.org (2011), Rubrics for evaluating open education resource (OER) objects. Washington, D.C. https://www.achieve.org/files/AchieveOERRubrics.pdf (accessed January 6, 2021).
Aguilar, J, V, B. (2011). Rubrics to Evaluate Open Educational Resources. https://dev1.oerafrica.org/resource/rubrics-evalute-open-educational-resources (accessed January 14, 2021).
Annand, D. (2015). Developing a sustainable financial model in higher education for open educational resources. The International Review of Research in Open and Distributed Learning, 16(5), 1–15. https://doi.org/10.19173/irrodl.v16i5.2133 (accessed January 14, 2021).
Atenas, J., & Havemann, L. (2013). Quality assurance in the open: An evaluation of OER repositories. The International Journal for Innovation and Quality in Learning, 1(2), 22–34 http://oro.open.ac.uk/56347/1/30-288-1-PB%5B1%5D.pdf (accessed January 14, 2021).
Atenas, J., & Havemann, L. (2014). Questions of quality in repositories of open educational resources: a literature review. Research in Learning Technology, 22. https://doi.org/10.3402/rlt.v22.20889 (accessed January 14, 2021).
Atenas, J., Havemann, L., & Priego, E. (2014). Opening teaching landscapes: The importance of quality assurance in the delivery of open educational resources. Open Praxis, 6(1), 29–43. https://doi.org/10.5944/openpraxis.6.1.81 (accessed January 14, 2021).
Atkins, D, E., Brown, J, S., & Hammond, A, L. (2007). A review of the open educational resources (OER) movement: Achievements, challenges, and new opportunities. Mountain View: Creative common. https://core.ac.uk/download/pdf/71348489.pdf (accessed January 14, 2021).
Butcher, N. (2011). A basic guide to open education resources (OER). Vancouver: Commonwealth of Learning. http://oasis.col.org/bitstream/handle/11599/36/2011_UNESCO_COL_A-Basic-Guide-to-OER.pdf?sequence=5&isAllowed=y (accessed January 14, 2021).
Camilleri, A, F., Ehlers, U., & Pawlowski, J. (2014). State of the art review of quality issues related to open educational resources (OER). Publications Office of the European Union. https://doi.org/10.2791/80171 (accessed January 14, 2021).
Caswell, T., Henson, S., Jensen, M., & Wiley, D. (2008). Open content and open educational resources: Enabling universal education. The International Review of Research in Open and Distributed Learning, 9(1). https://doi.org/10.19173/irrodl.v9i1.469 (accessed January 6, 2021).
Clements, K., Pawlowski, J., & Manouselis, N. (2015). Open educational resources repositories literature review-towards a comprehensive quality approaches framework. Computers in Human Behavior, 51, 1098–1106. https://doi.org/10.1016/j.chb.2015.03.026.
Coursera.org (2021). Meet our partners. https://www.coursera.org/about/partners (accessed January 14, 2021).
Davis, H. C., Carr, L. A., Hey, J. M., Howard, Y., Millard, D., Morris, D., & White, S. (2009). Bootstrapping a culture of sharing to facilitate open educational resources. IEEE Transactions on Learning Technologies, 3(2), 96–109. https://doi.org/10.1109/TLT.2009.34 (accessed January 14, 2021).
Davis, E., Cochran, D., Fagerheim, B., & Thoms, B. (2016). Enhancing teaching and learning: Libraries and open educational resources in the classroom. Public Services Quarterly, 12(1), 22–35. https://doi.org/10.1080/15228959.2015.1108893.
De los Arcos, B., Farrow, R., Perryman, L, A., Pitt, R. & Weller, M. (2014). OER evidence report 2013–2014. OER Research Hub. https://oerresearchhub.files.wordpress.com/2014/11/oerrh-evidence-report-2014.pdf (accessed January 6, 2021).
Downes, S. (2007). Models for sustainable open educational resources. Interdisciplinary Journal of E-Learning and Learning Objects, 3(1), 29–44 http://www.ijello.org/Volume3/IJKLOv3p029-044Downes.pdf (accessed January 6, 2021).
edX.org (2021). Schools and partners. https://www.edx.org/schools-partners (accessed January 14, 2021).
Ehlers, U. D. (2011). Extending the territory: From open educational resources to open educational practices. Journal of Open, Flexible, and Distance Learning, 15(2), 1–10.
Geser, G. (2007). Open educational practices and resources. OLCOS Roadmap, 2012. Open e-Learning Content Observatory Services. https://doi.org/10.7238/rusc.v4i1.295 (accessed January 14, 2021), 4
Habibi, A., Sarafrazi, A., & Izadyar, S. (2014). Delphi technique theoretical framework in qualitative research. The International Journal of Engineering and Science, 3(4), 8–13 https://parsmodir.com/wp-content/uploads/2018/11/Delphi2014-En.pdf (accessed January 14, 2021).
Hylén, J. (2006). Open educational resources: Opportunities and challenges. Proceedings of open education. https://www.knowledgeall.net/files/Additional_Readings-Consolidated.pdf (accessed January 14, 2021).
Jiang, X.-P., Yan, L., Zheng, X.-L., Liu, X., & Wei, X.-Q. (2016). Development and evaluation of a new curriculum based on the Delphi method for master of nursing programs in China. Chinese Nursing Research, 3(4), 162–167. https://doi.org/10.1016/j.cnre.2016.11.005 (accessed January 14, 2021).
Johnstone, S. (2005). Forum on the impact of open courseware for higher education in developing countries. Education Quarterly, 3, 15–18.
Kanwar, A., Kodhandaraman, B., & Umar, A. (2010). Toward sustainable open education resources: A perspective from the global south. American Journal of Distance Education, 24(2), 65–80. https://doi.org/10.1080/08923641003696588.
Kawachi, P. (2015). Quality assurance for OER: Current state of the art and the TIPS framework. eLearning Papers, 40, 3–13.
Landeta, J. (2006). Current validity of the Delphi method in social sciences. Technological Forecasting and Social Change, 73(5), 467–482. https://doi.org/10.1016/j.techfore.2005.09.002.
Lilja, K. K., Laakso, K., & Palomki, J. (2011). Using the Delphi method. Portland: Technology Management in the Energy Smart World.
McAndrew, P., Farrow, R., Elliott-Cirigottis, G., & Law, P. (2012). Learning the lessons of openness. Journal of Interactive Media in Education, 2. https://doi.org/10.5334/2012-10 (accessed January 14, 2021)
McGreal, R. (2017). Special report on the role of open educational resources in supporting the sustainable development goal 4: Quality education challenges and opportunities. The International Review of Research in Open and Distributed Learning, 19(7), 293–305. https://doi.org/10.19173/irrodl.v18i7.3541 (accessed January 14, 2021).
Oercommons.org (2021). Professional learning & teacher training. https://www.oercommons.org/training (accessed January 14, 2021).
Okoli, C., & Pawlowski, S. D. (2004). The Delphi method as a research tool: An example, design considerations and applications. Information & Management, 42(1), 15–29. https://doi.org/10.1016/j.im.2003.11.002.
Oxford Advanced American Dictionary, (2021). X factor term https://www.oxfordlearnersdictionaries.com/definition/english/x-factor?q=X+factor (accessed January 14, 2021).
Pérez, L, I, G., Montoya, M, S, R., & García-Peñalvo, F, J. (2016). Open access to educational resources in energy and sustainability: Usability evaluation prototype for repositories. Proceedings of the Fourth International Conference on Technological Ecosystems for Enhancing Multiculturality-TEEM (pp.1103-1108). https://doi.org/10.1145/3012430.3012654.
Quality Matter.org (n.d.). Helping you deliver on your online promise. https://www.qualitymatters.org/ (accessed January 14, 2021).
Ramírez-Montoya, M. S., García-Peñalvo, F. J., & McGreal, R. (2018). Shared science and knowledge. Comunicar, 26(54), 1–5. https://doi.org/10.1080/19345747.2010.487927.
Reich, J., & Ruipérez-Valiente, J. A. (2019). The MOOC pivot. Science, 363(6423), 130–131. https://doi.org/10.1126/science.aav7958.
Ruiz-Iniesta, A., Jimenez-Diaz, G., & Gomez-Albarran, M. (2014). A semantically enriched context-aware OER recommendation strategy and its application to a computer science OER repository. IEEE Transactions on Education, 57(4), 255–260. https://doi.org/10.1109/te.2014.2309554.
Santos-Hermosa, G., Ferran-Ferrer, N., & Abadal, E. (2017). Repositories of open educational resources: An assessment of reuse and educational aspects. The International Review of Research in Open and Distributed Learning, 18(5), 84–120. https://doi.org/10.19173/irrodl.v18i5.3063 (accessed January 14, 2021).
Shah, H. A., & Kalaian, S. A. (2009). Which is the best parametric statistical method for analyzing Delphi data? Journal of Modern Applied Statistical Methods, 8(1), 226–232. https://doi.org/10.22237/jmasm/1241137140 (accessed January 14, 2021).
Smith, M. S., & Casserly, C. M. (2006). The promise of open educational resources. Change: The Magazine of higher learning, 38(5), 8–17. https://doi.org/10.3200/CHNG.38.5.8-17.
Tuomi, I. (2013). Open educational resources and the transformation of education. European Journal of Education, 48(1), 58–78. https://doi.org/10.1111/ejed.12019.
UNESCO (2018). The Paris OER Declaration 2012. https://en.unesco.org/oer/paris-declaration (accessed January 14, 2021).
UNESCO (2020a). UNESCO recommendation on open educational resources (OER). https://en.unesco.org/news/unesco-recommendation-open-educational-resources-oer (accessed January 14, 2021).
UNESCO (2020b). OER dynamic coalition. https://en.unesco.org/themes/building-knowledge-societies/oer/dynamic-coalition/ (accessed January 14, 2021).
Vlădoiu, M. (2012). Towards a quality model for open courseware and open educational resources. In international conference on web-based learning (pp. 213–220). Berlin, Heidelberg: Springer.
Wang, X. C., Liu, M. R., Sun, Y. X., Li, Q. H., & Gao, Y. (2017). Systematic analysis of international OER platforms and implications of educational resource construction of the B&R initiative. E-education Research, 38(12), 106–113.
Wiley, D., Bliss, T, J., & McEwen, M. (2014). Open educational resources: A review of the literature. In Handbook of research on educational communications and technology (pp. 781–789). Berlin, Heidelberg: Springer.
Willems, J., & Bossu, C. (2012). Equity considerations for open educational resources in the glocalization of education. Distance Education, 33(2), 185–199. https://doi.org/10.1080/01587919.2012.692051.
Zeedick, D, M. (2010). The modified Delphi method to analyze the application of instructional design theory to online graduate education [unpublished doctoral dissertation]. The George Washington University. https://search.proquest.com/openview/26828804f13b2945857549df75dd9c5c/1?pq-origsite=gscholar&cbl=18750&diss=y (accessed January 14, 2021).
Funding
The study is partially supported by the National Social Science Fund of China (Grant No. BCA170073).
Author information
Authors and Affiliations
Contributions
Xiaochen Wang analyzed the data and prepared the initial manuscript.
Tingting Chen collected the data.
Yihan Zhang collected the data.
Harrison Hao Yang provided the conceptual input and revised the manuscript with Xiaochen Wang.
Corresponding author
Ethics declarations
Conflicts of interest/competing interests
The authors declare that they have no conflict of interest/competing interests.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix 1 Evaluation Framework of OER (Delphi Round 1)
To better understand the sustainable development of open educational resources (OER), this study aimed to break through the original ‘fixed’, highly structured evaluation system and develop an open and flexible “1 + X” evaluation framework of OER. “1” refers to the mandatory basic criteria that the repository must reflect essential attributes and core functions. “X” refers to optional distinctive criteria that the repository focuses on distinctive and diversified developmental orientations.
-
1.
What do you think of the above “1 + X” evaluation framework of OER.
-
2.
The preliminary proposal of the criteria “1” is showed in the following table. Please rate the indicators based on your experience. (from “not at all important” to “extremely important”, value 1 to 5)
Tier 1 | Tier 2 | Source | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|---|---|
Information Content | Open Protocol | OER Rubric, TIPS, LORQAF | |||||
Accessibility | OER Rubric, Quality Matter, LORQAF | ||||||
Quality Audit | LORQAF | ||||||
Learning Process | Learning Activity | OER Rubric | |||||
Learning Community | New Factor | ||||||
Coaching | New Factor | ||||||
Mentoring and Feedback | TIPS | ||||||
System Technology | Reliability | Quality Matter, OER rubric, TIPS | |||||
Open Source | New Factor | ||||||
Technical Support | New Factor | ||||||
Visual of Digital Environment | Interface Design | TIPS | |||||
Interaction Design | OER Rubric, TIPS, LORQAF | ||||||
Navigation Design | TIPS |
-
3.
Please list at least 7 “X” indicators based on your experience.
Appendix 2 Evaluation Framework of OER (Delphi Round 2)
-
1.
The revised preliminary proposal of the criteria “1” is showed in table according the result of Delphi Round 1. The main optimizations were: (i) excluding or further integrating the indicators of ‘Coaching’ and ‘Navigation Design’ with a mean value below 3.5 and CV ≥ 0.25, (ii) integrating ‘Instructional Design’, which is of general concern to experts in the interviews, into the ‘Learning Process’ dimension of the basic indicators, and merging the existing secondary criteria of ‘Learning Activities’, and (iii) adding ‘User Guide’ and ‘Ease of Use’ criteria. Please rate the indicators based on your experience. (from “not at all important” to “extremely important”, value 1 to 5)
Tier 1 | Tier 2 | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|---|
Information Content | Open Protocol | |||||
Accessibility | ||||||
Quality Audit | ||||||
Learning Process | Instruction Design | |||||
Learning Community | ||||||
Mentoring and Feedback | ||||||
System Technology | Reliability | |||||
Open-Source | ||||||
Technical Support | ||||||
Visual of Digital Environment | User Guide | |||||
Interface Design | ||||||
Interaction Support | ||||||
Ease of Use |
-
2.
We aggregated similar terminology expressions of expert feedback with the criteria “X” in Delphi Round 1. Please rate the indicators based on your experience. (from “not at all important” to “extremely important”, value 1 to 5)
X Criteria | Frequency | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|---|
International Collaborative | 11 | |||||
Multilingual | 10 | |||||
Customized Service | 10 | |||||
Certification | 9 | |||||
Open Educational Practices | 9 | |||||
Resource Evolution | 9 | |||||
Diversified Access | 8 | |||||
Online Authoring Tools | 8 | |||||
Accessibility | 8 | |||||
General Standard | 7 | |||||
Diversified Cost-Sharing Mechanisms | 4 |
-
3.
Do you have any other suggestions?
Appendix 3 Evaluation Framework of OER (Delphi Round 3)
-
1.
The revised preliminary proposal of the criteria “1” is showed in table according the result of Delphi Round 2. The main optimizations were changing ‘Visual of Digital Environment’ to ‘User Experience’. Please rate the indicators based on your experience. (from “not at all important” to “extremely important”, value 1 to 5)
Tier 1 | Tier 2 | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|---|
Information Content | Open Protocol | |||||
Accessibility | ||||||
Quality Audit | ||||||
Learning Process | Instruction Design | |||||
Learning Community | ||||||
Mentoring and Feedback | ||||||
System Technology | Reliability | |||||
Open-Source | ||||||
Technical Support | ||||||
User Experience | User Guide | |||||
Interface Design | ||||||
Interaction Support | ||||||
Ease of Use |
-
2.
The revised preliminary proposal of the criteria “X” is showed in table according the result of Delphi Round 2. The main optimizations were: a) delete “Diversified Cost-Sharing Mechanism” according to the consensus threshold; b) add “niche resource” and “personal learning spaces”; 3) proposed five emerging directions for development. Please rate the indicators based on your experience. (from “not at all important” to “extremely important”, value 1 to 5)
No. | Featured Criteria | Development Orientation | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|---|---|
1 | Multilingual | International Collaborative | |||||
2 | International Cooperation | ||||||
3 | General Standard | ||||||
4 | Customized Service | value-added service | |||||
5 | Certification | ||||||
6 | Resource Self-Growth | Resource Evolution | |||||
7 | Online Authoring Tools | ||||||
8 | Diversified Access | Inclusive and Fair | |||||
9 | Accessibility | ||||||
10 | Building Niche Resources | ||||||
11 | Open Educational Practices | Open Educational Practices | |||||
12 | Personal Learning Space |
-
3.
Do you have any other suggestions?
Rights and permissions
About this article
Cite this article
Wang, X., Chen, T., Zhang, Y. et al. Implications of the Delphi method in the evaluation of sustainability open education resource repositories. Educ Inf Technol 26, 3825–3844 (2021). https://doi.org/10.1007/s10639-021-10452-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10639-021-10452-z