Assessment of Learning in Digital Interactive Social Networks: A Learning Analytics Approach

Mark Wilson; Kathleen Scalise; Perman Gochyyev

doi:10.24059/olj.v20i2.799

Authors

Mark Wilson University of California, Berkeley
Kathleen Scalise University of Oregon
Perman Gochyyev University of California, Berkeley

DOI:

https://doi.org/10.24059/olj.v20i2.799

Keywords:

Data analytics, learning analytics, assessment, ICT literacy, digital literacy, K-12, science, mathematics, language arts, second language acquisition, ATC21S

Abstract

This paper summarizes initial field-test results from data analytics used in the work of the Assessment and Teaching of 21st Century Skills (ATC21S) project, on the “ICT Literacy – Learning in digital networks” learning progression. This project, sponsored by Cisco, Intel and Microsoft, aims to help educators around the world enable students with the skills to succeed in future career and college goals. The paper begins with describing some expansions to a common definition of learning analytics, then includes a review of the literature on ICT literacy, including the specific development that led to the ATC21S effort. This is followed by a description of the development of a “learning progression” for this project, as well as the logic behind the instrument construction and data analytics, along with examples of each. Data were collected in a demonstration digital environment in four countries: Australia, Finland, Singapore and the U.S. The results indicate that the new constructs developed by the project, and the novel item forms and analytics that were employed, are indeed capable of being employed in a large-scale digital environment. The paper concludes with a discussion of the next steps for this effort.

References

ACARA. (2012). Australian Curriculum, Assessment and Reporting Authority, National Assessment Program, ICT Literacy, Years 6 and 10 Report 2011.

Adams R. J.,& Khoo, S-T. (1996). Quest [computer program]. Melbourne, Australia: ACER Press.

Adams, R.J., & Wilson, M. (1996). Formulating the Rasch model as a mixed coefficients multinomial logit. In, G. Engelhard & M. Wilson (Eds.), Objective measurement III: Theory into practice. Norwood, NJ: Ablex.

Adams, R.J., Wu, M., & Wilson, M. (2012). ConQuest 3.0 [computer program]. Hawthorn, Australia: ACER.

Ainley, J., Fraillon, J., Schulz, W., & Gebhardt, E. (2014). Measuring Changes in ICT Literacy over Time. Paper presented at the National Council on Measurement in Education, Philadelphia, PA.

Binkley, M., Erstad, O, Herman, J., Raizen, S., Ripley, M., Miller-Ricci, M., & Rumble, M. (2012). Defining Twenty-First century skills. In Griffin, P, McGaw B. & Care, E. (Eds.), Assessment and Teaching of 21st Century Skills. Dordrecht, Germany: Springer.

Briggs, D. C., & Wilson, M. (2003). An introduction to multidimensional measurement using Rasch models. Journal of Applied Measurement, 4(1), 87â€“100.

Chi, E. H., Pirolli, P., Suh, B., Kittur, A., Pendleton, B., & Mytkowicz, T. (2008). Augmented Social Cognition. Palo Alto, CA: Palo Alto Research Center.

Cisco, Intel, Microsoft (CIM). (2008). Transforming Education: Assessing and Teaching 21st Century Skills. Authors. (Downloaded from: http://atc21s.org/wp-content/uploads/2011/04/Cisco-Intel-Microsoft-Assessment-Call-to-Action.pdf)

Cooke, N. J., Gorman, J. C., & Winner, J. L. (2007). Team Cognition. In F. T. Durso (Ed.), Handbook of Applied Cognition (pp. 239-268). New York: John Wiley & Sons, Ltd.

Dhar, V., & Olson, M. H. (1989). Assumptions Underlying Systems that Support Work Group Collaboration. In M. H. Olson (Ed.), Technology Support for Work Group Collaboration. Hillsdate, NJ: Lawrence Erlbaum Associates.

Ferrari, A. (2012). Digital competence in practice: An analysis of frameworks. Seville, Spain: Institute for Prospective Technological Studies, European Commission.

Fraillon, J. (2014). Assessing Computer and Information Literacy across 20 Countries: The International Computer and Information Literacy (ICILS). Paper presented at the National Council on Measurement in Education (NCME), Philadelphia, PA.

Friedman, T. L. (2007). The World is Flat 3.0: A Brief History of the Twenty-first Century. New York: Picador.

Griffin, P, McGaw B. & Care, E. (Eds.) (2012). Assessment and Teaching of 21st Century Skills. Dordrecht, Germany: Springer.

Griffiths, T. L., & Steyvers, M. (2007). Topics in Semantic Representation. Psychological Review, 114(2), 211-244.

Kozma, R. (Ed.). (2003). Technology, innovation, and educational change. A global perspective. A report of the Second Information Technology in Education Study, Module 2. Amsterdam: International Association for the Evaluation of Educational Achievement.

Lampe, C., Resnick, P., Forte, A., Yardi, S., Rotman, D., Marshall, T. & Lutters, W. (2010). Educational priorities for technology-mediated social participation. IEEE Computer, 43(11), 60-67.

Lehrer, R., Kim, M-J., Ayers, E., & Wilson, M. (2014). Toward establishing a learning progression to support the development of statistical reasoning. In J. Confrey and A. Maloney (Eds.), Learning over Time: Learning Trajectories in Mathematics Education. Charlotte, NC: Information Age Publishers.

Masters, G. (1982). A Rasch model for partial credit scoring. Psychometrika, 47(2), 149-174.

National Research Council (NRC). (2001). Knowing what students know: The science and design of educational assessment. Committee on the Foundations of Assessment. J. Pellegrino, N. Chudowsky, & R. Glaser (Eds). Washington, DC: National Academy Press.

Park, J. (2011). ICT in Education in Asia-Pacific. Asia-Pacific Regional Forum on ICT Applications. Retrieved from http://www.itu.int/ITU-D/asp/CMS/Events/2011/ict-apps/s5_JPark.pdf.

Pedersen, S. G., Malmberg, P., Christensen, A. J., Pedersen, M., Nipper, S., GrÃ¦m, C. D., & NorrgÃ¥rd, J. (Eds.). (2006). E-learning Nordic 2006: The impact of ICT on educatio. Copenhagen, Denmark: Ramboll Management.

Pirolli, P. (2007). Cognitive Models of Human-Information Interaction. In F. T. Durso (Ed.), Handbook of Applied Cognition (pp. 443-470). New York: John Wiley & Sons, Ltd.

Pirolli, P. (2009). An elementary social information foraging model. CHI 2009, ACM Conference on Human Factors in Computing Systems, (pp. 605-614). Boston: Association for Computing Machinery.

Pirolli, P., Preece, J., & Shneiderman, B. (2010). Cyberinfrastructure for social action on national priorities. IEEE Computer, 43(11), 20-21.

Pirolli, P., Preece, J. & Shneiderman, B. (2010). Cyberinfrastructure for social action on national priorities. IEEE Computer, 43(11), 20-21.

Pirolli, P., & Wilson, M. (1998). A Theory of the Measurement of Knowledge Content, Access, and Learning. Psychological Review, 105(1), 58-82.

Qualifications and Curriculum Authority. (2007). About information and communication technology: Assessment guidance Retrieved from http://web.archive.org/web/20040815054520/http://www.qca.org.uk/7889.html

Rogers, W. A., Pak, R., & Fisk, A. D. (2007). Applied Cognitive Psychology in the Context of Everyday Living. In F. T. Durso (Ed.), Handbook of Applied Cognition (pp. 3-27). New York: John Wiley & Sons, Ltd.

U.S. Department of Education. (2010). Transforming American Education, Learning Powered by Technology National Education Technology Plan 2010. Retrieved from http://www.ed.gov/sites/default/files/netp2010.pdf

Wilson, M. (2005). Constructing Measures: An Item Response Modeling Approach. Mahwah, NJ: Erlbaum.

Wilson, (2009a, December). Assessment for Learning and for Accountability. Paper presented at the Exploratory Seminar: Measurement Challenges Within the Race to the Top Agenda, at ETS, Princeton, NJ. . (Download from:

Wilson, (2009b, December). Assessment for Learning and for Accountability. Policy Brief from the Exploratory Seminar: Measurement Challenges Within the Race to the Top Agenda, at ETS, Princeton, NJ. . (Download from:

Wilson, M. (2012). Responding to a challenge that learning progressions pose to measurement practice: hypothesized links between dimensions of the outcome progression. In A .C. Alonzo & A. W. Gotwals, (Eds.), Learning Progressions in Science. Rotterdam, The Netherlands: Sense Publishers.

Wilson, M., Bejar, I., Scalise, K., Templin, J., Wiliam, D., & Torres-Irribarra, D. (2012). Perspectives on methodological issues. In Griffin, P, McGaw B. & Care, E. (Eds.), Assessment and Teaching of 21st Century Skills. Dordrecht, Germany: Springer.

Wilson, M. & Carstensen, C. (2007). Assessment to improve learning in mathematics: The BEAR Assessment system. In A. Schoenfeld (Ed.), Assessing mathematical proficiency (pp. 311-332). London: Cambridge University Press.

Wilson, M., & Hoskens, M. (2005). Multidimensional item responses: Multimethod/multitrait perspectives. In S. Alagumalai, D. D. Curtis, & N. Hungi (Eds.) Applied Rasch measurement: A Book of Exemplars (Papers in honour of John Keeves) (pp. 287-308). Dordrecht , The Netherlands: Springer-Kluwer Academic Publishers.

Wilson, M., & Scalise, K. (2015). Assessment of Learning in Digital Networks. In, Griffin, P., & Care, E. (Eds.), Assessment and Teaching of 21st Century Skills Volume 2 - Methods & Approaches. Dordrecht: Springer.

Wilson, M., Scalise, K., & Gochyyev, P. (in press). Rethinking ICT literacy: â€¨From computer skills to social network settings. Thinking Skills & Creativity

Wilson, M. & Sloane, K. (2000). From principles to practice: An embedded assessment system. Applied Measurement in Education, 13(2), 181-208.

(Download from: http://www.informaworld.com/smpp/content~content=a783685281~db=all)