Skip to main content
SpringerLink
Log in

How Difficult is the Task for you? Modelling and Analysis of Students' Task Difficulty Sequences in a Simulation-Based POE Environment

  • Article
  • Published:
International Journal of Artificial Intelligence in Education Aims and scope Submit manuscript

Abstract

Task difficulty (TD) reflects students' subjective judgement on the complexity of a task. We examine the TDs data of 236 undergraduate students in a simulation-based Predict-Observe-Explain (POE) environment using three different labels easy, medium and hard. Generally, the students who perceive the tasks to be easy or hard perform poorly at the transfer task than the students who perceive the tasks to be medium or moderately difficult. Sequences of students' TDs are analysed which consist of a set of several judgements, collected once for each task in a POE sequence. The analysis suggests that given a sequence of TDs, difficulty level hard followed by a hard may lead to poorer learning outcomes at the transfer task. By contrast, difficulty level medium followed by a medium may lead to better learning outcomes at the transfer task. In terms of the TD models, we identify student behaviours that can be reflective of their perceived difficulties. Generally, the students who report that the tasks are easy, adopt a trial-and-error behaviour where they spend lesser time and make more attempts on tasks. By comparison, the students who complete the tasks in a longer time by making more attempts are likely to report that the following task is hard. For the students who report medium TDs, mostly these students seem to reflect on tasks where they spend a long time and require fewer attempts for task completions. Additionally, these students provide longer texts for explaining their hypothesis reasoning. Understanding how student behaviours and TDs manifest over time and how they impact students' learning outcomes is useful, especially when designing for real-time educational interventions, where the difficulty of the tasks could be optimised for students. It can also help in designing and sequencing the tasks for the development of effective teaching strategies that can maximise students' learning.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. https://www.smartsparrow.com/research/

References

  • Arroyo, I., Woolf, B. P., Cooper, D. G., et al. (2011) The impact of animated pedagogical agents on girls’ and boys’ emotions, attitudes, behaviors and learning. In: International Conference on Advanced Learning Technologies (ICALT) (pp. 506–510). IEEE

  • Baker, R., Corbett, A. T., Koedinger, K. R., et al. (2004) Off-task behavior in the cognitive tutor classroom: when students "game The system". In: SIGCHI Conference on Human Factors in Computing Systems (pp. 383–390). ACM

  • Baker, R., D'mello, S., Rodrigo, M. M. T., et al. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68, 223–241

    Article  Google Scholar 

  • Baker, R. S., Ogan, A. E., Madaio, M., et al. (2019). Culture in Computer-Based Learning Systems: Challenges and Opportunities. Computer-Based Learning in Context, 1, 1–13

    Google Scholar 

  • Bandura, A. (1997). Self-efficacy: The exercise of control. Freeman

  • Bandura, A. (1977). Social Learning Theory. Prentice Hall

  • Bell, A. W., Brekke, G., & Swan, M. (1987) Misconceptions, conflict and discussion in the teaching of graphical interpretation. In: Novak J (ed) Proceedings of the Second International Seminar: Misconceptions and Educational Strategies in Science and Mathematics (p 46–48). Comell University

  • Belmont, J. M., & Mitchell, D. W. (1987). The general strategy hypothesis as applied to cognitive theory in mental retardation. Intelligence, 11, 91–105

    Article  Google Scholar 

  • Berkson, J. (1944). Application of the logistic function to Bio-Assay. Journal of the American Statistical Association, 39, 357–365

    Google Scholar 

  • Bjork, R. A. (2013). Desirable difficulties perspective on learning. Encyclopedia of the Mind, 4, 134–146

    Google Scholar 

  • Bradley, A. P. (1997). The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30, 1145–1159

    Article  Google Scholar 

  • Bunderson, V. C., Inouye, D. K., & Olsen, J. B. (1989). The four generations of computerized educational measurement. In R. L. Linn (Ed.), Educational measurement. Macmillan

  • Campbell, D. J. (1988). Task complexity: A review and analysis. Academy of Management Review, 13, 40–52

    Article  Google Scholar 

  • Campbell, D. J. (2008). Subtraction by addition. Memory & Cognition, 36, 1094–1102

    Article  Google Scholar 

  • Carpentar, V. L., Friar, S., & Lipe, M. G. (1993). Evidence on the performance of accounting students: Race, gender and expectations. Issues in Accounting Education, 8, 1–17

    Google Scholar 

  • Chen, Y., Yu, B., Zhang, X., et al. (2016). Topic modeling for evaluating students' reflective writing: a case study of pre-service teachers' journals. In D. Gašević, G. Lynch, S. Dawson, H. Drachsler, C. P. Rosé (Eds.), Learning Analytics and Knowledge LAK (pp. 1–5). Association for Computing Machinery (ACM)

  • Christensen, T. E., Fogarty, T. J., & Wallace, W. A. (2002). The association between the directional accuracy of self-efficacy and accounting course performance. Issues in Accounting Education, 17, 1–26

    Article  Google Scholar 

  • Colthorpe, K., Zimbardi, K., Ainscough, L., et al. (2015). Know Thy Student! Combining Learning Analytics and Critical Reflections to Increase Understanding of Students' Self-Regulated Learning in an Authentic Setting. Journal of Learning Analytics, 2, 134–155

    Article  Google Scholar 

  • Coştu, B., Ayas, A., & Niaz, M. (2012). Investigating the effectiveness of a POE-based teaching activity on students’ understanding of condensation. Instructional Science, 40, 47–67

    Article  Google Scholar 

  • Craig, S., Graesser, A., Sullins, J., et al. (2004). Affect and learning: an exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media, 29, 241–250

    Article  Google Scholar 

  • Csikszentmihalyi, M. (1979) The flow experience. Consciousness: Brain and states of awareness and mysticism (pp. 63–67)

  • Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. Harper Perennial

  • Csikszentmihalyi, M. (1997) Finding Flow: The Psychology of Engagement with Everyday Life" in University of Chicago: Harper Collins.

  • Csikszentmihalyi, M. (2000) Beyond boredom and anxiety. Jossey-Bass

  • Dalziel, J. (2010). Practical eTeaching strategies for predict – observe – explain, problem-based learning and role plays. LAMS International

  • Deloache, J. S., Cassidy, D. J., & Brown, A. L. (1985). Precursors of mnemonic strategies in very young children's memory. Child Development, 56, 125–137

    Article  Google Scholar 

  • D'Mello, S., & Graesser, A. (2010). Modeling Cognitive-Affective Dynamics with Hidden Markov Models. Proceedings of the Annual Meeting of the Cognitive Science Society, 32. Retrieved from https://escholarship.org/uc/item/06b952g5

  • D’mello, S., & Graesser, A. (2012). Dynamics of affective states during complex learning. Learning and Instruction, 22, 145–157

    Article  Google Scholar 

  • D'mello, S., & Graesser, A. (2014). Confusion and its dynamics during device comprehension with breakdown scenarios. Acta psychologica, 151, 106–116

    Article  Google Scholar 

  • D’Mello, S. K., Person, N., & Lehman, B. A. (2009). Antecedent-Consequent Relationships and Cyclical Patterns between Affective States and Problem Solving Outcomes. In V. Dimitrova, R. Mizoguchi, B. du Boulay, & A. Graesser (Eds.), Proceedings of 14th International Conference on Artificial Intelligence In Education.(pp. 57-64). Amsterdam: IOS Press.

  • D’Mello, S. K., Taylor, R., & Graesser, A. C. (2007). Monitoring Affective Trajectories during Complex Learning. In D. S. McNamara & J. G. Trafton (Eds.), Proceedings of the 29th Annual Cognitive Science Society (pp. 203-208). Austin, TX: Cognitive Science Society.

  • Dowell, N. M. M., & Graesser, A. (2014). Modeling learners’ cognitive, affective, and social processes through language and discourse. Journal of Learning Analytics, 1, 183–186

    Article  Google Scholar 

  • Driver, R. (1983). The pupil as scientist? Open University Press

  • Eccles, J. S., Adler, T. F., Futterman, R., et al. (1983). Expectancies, values and academic behaviors. In J. T. Spence, et al. (Eds.), Achievement and achievement motives (pp. 75–146). W.H. Freeman

  • Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53, 109–132

    Article  Google Scholar 

  • Ellis, R. A., & Han, F. (2017). Improving learning analytics – combining observational and self-report data on student learning. Educational Technology & Society, 20, 158–169

    Google Scholar 

  • Field, A., Miles, J., & Field, Z. (2012) Discovering statistics using R. Sage Publications

  • Gašević, D., Jovanovic, J., Pardo, A., et al. (2017). Detecting learning strategies with analytics: Links with self-reported measures and academic performance. Journal of Learning Analytics, 4, 113–128

    Article  Google Scholar 

  • Gobert, J. D., Baker, R., & Wixon, M. B. (2015). Operationalizing and detecting disengagement within online science microworlds. Educational Psychologist, 50, 43–57

    Article  Google Scholar 

  • Gorsky, P., & Finegold, M. (1994). The role of anomaly and of cognitive dissonance in restructuring students' concepts of force. Instructional Science, 22, 75–90

    Article  Google Scholar 

  • Graesser, A., Lu, S., Olde, B. A., et al. (2005). Question asking and eye tracking during cognitive disequilibrium: Comprehending illustrated texts on devices when the devices break down. Memory & Cognition, 33, 1235–1247

    Article  Google Scholar 

  • Greiff, S., Wüstenberg, S., Csapó, B., et al. (2014). Domain-general problem solving skills and education in the 21st century. Educational Research Review, 13, 74–83

    Article  Google Scholar 

  • Guadagnoli, M. A., & Lee, T. D. (2004). Challenge point: A framework for conceptualizing the effects of various practice conditions in motor learning. Journal of Motor Behavior, 36, 212–224

    Article  Google Scholar 

  • Gunstone, R., & White, R. (1980). A matter of gravity. Research in Science Education, 10, 35–44

    Article  Google Scholar 

  • Hom, H. L., & Maxwell, F. R. (1983). The impact of task difficulty expectations on intrinsic motivation. Motivation and Emotion, 7, 19–24

    Article  Google Scholar 

  • Horodyskyj, L. B., Mead, C., Belinson, Z., et al. (2018). Habitable Worlds: Delivering on the Promises of Online Education. Astrobiology, 18, 86–99

    Article  Google Scholar 

  • Kapur, M., & Bielaczyc, K. (2012). Designing for productive failure. The Journal of the Learning Sciences, 21, 45–83

    Article  Google Scholar 

  • Kapur, M., & Rummel, N. (2012). Productive failure in learning and problem solving. Instructional Science, 40, 645–650

    Article  Google Scholar 

  • Karumbaiah, S., Andres, J. M. A. L., Botelho, A. F., Baker, R.S., & Ocumpaugh, J. (2018). The implications of a subtle difference in the calculation of affect dynamics. In: J.C. Yang, M. Chang, L.H. Wong, M.M.T. Rodrigo (eds.), 26th International Conference on Computers in Education, Philippines (p. 29-38) . Asia-Pacific Society for Computers in Education

  • Karumbaiah S., Baker R.S., & Ocumpaugh J. (2019). The Case of Self-transitions in Affective Dynamics. In: Isotani S., Millán E., Ogan A., Hastings P., McLaren B., Luckin R. (eds) Artificial Intelligence in Education. AIED 2019 (p. 172-181). Lecture Notes in Computer Science, vol 11625. Springer, Cham. https://doi.org/10.1007/978-3-030-23204-7_15

  • Karumbaiah, S., Ocumpaugh, J., & Baker, R.S. (2019). The influence of school demographics on the relationship between students' help-seeking behavior and performance and motivational measures. In: C.F. Lynch, A. Merceron, M. Desmarais, R. Nkambou (eds.), 12th International Conference on Educational Data Mining (EDM) (p. 99–108). International Educational Data Mining Society

  • Kennedy, G. & Lodge, J. (2016). All roads lead to Rome: Tracking students’ affect as they overcome misconceptions. Barker, S., Dawson, S., Pardo, A., Colvin, C. (eds). ASCILITE 2016 - Conference Proceedings - 33rd International Conference of Innovation, Practice and Research in the Use of Educational Technologies in Tertiary Education: Show Me the Learning, pp.318-328. ASCILITE.

  • Kibirige, I., Osodo, J., & Tlala, K. M. (2014). The effect of Predict-Observe-Explain strategy on learners’ misconceptions about dissolved salts. Mediterranean Journal of Social Sciences 5(4), (p. 300-310), https://doi.org/10.5901/mjss.2014.v5n4p300

  • Krosnick, J. A. (1999). Survey research. Annual Review of Psychology, 50, 537–567

    Article  Google Scholar 

  • Kulhavy, R. W. (1977). Feedback in written instruction. Review of educational research, 47, 211–232

    Article  Google Scholar 

  • Kulhavy, R. W., Yekovich, F. R., & Dyer, J. W. (1976). Feedback and response confidence. Journal of Educational Psychology, 68, 522–528

    Article  Google Scholar 

  • Lazarus, R. S. (1991). Cognition and motivation in emotion. American Psychologist, 46, 352–367

    Article  Google Scholar 

  • Li, W., Lee, A., & Solmon, M. (2007). The role of perceptions of task difficulty in relation to self-perceptions of ability, intrinsic value, attainment value, and performance. European Physical Education Review, 13, 301–318

    Article  Google Scholar 

  • Long, M. H., & Crookes, G. (1992). Three approaches to task-based syllabus design. TESOL quarterly, 26, 27–56

    Article  Google Scholar 

  • Mangos, P. M., & Steele-Johnson, D. (2001). The role of subjective task complexity in goal orientation, self-efficacy, and performance relations. Human Performance, 14, 169–185

    Article  Google Scholar 

  • Mayfield, E., Madaio, M., Prabhumoye, S., et al. (2019) Equity beyond bias in language technologies for education. In: In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics (p. 444–460)

  • Maynard, D. C., & Hakel, M. D. (1997). Effects of Objective and Subjective Task Complexity on Performance. Human Performance, 10, 303–330

    Article  Google Scholar 

  • Mooi, T. L. (2006). Self-efficacy and student performance in an accounting course. Journal of Financial Reporting and Accounting, 4, 129–146

    Article  Google Scholar 

  • Nawaz, S., Kennedy, G., Bailey, J., et al. (2018) Struggle town? Developing profiles of student confusion in simulation-based learning environments. In: M. Campbell, J. Willems, C. Adachi, D. Blake, I. Doherty, S. Krishnan, S. Macfarlane, L. Ngo, M. O’Donnell, S. Palmer, L. Riddell, I. Story, Suri H, Tai J (eds) 35th International Conference on Innovation, Practice and Research in the Use of Educational Technologies in Tertiary Education, ASCILITE 2018 (p. 224–233). Deakin University

  • Nawaz, S., Kennedy, G., Bailey, J., & Mead, C., (2020). Moments of confusion in simulation-based learning environments. Journal of Learning Analytics, 7(3), 118–137. https://doi.org/10.18608/jla.2020.73.9.

  • Ocumpaugh, J., Baker, R., Gowda, S., et al. (2014). Population validity for Educational Data Mining models: A case study in affect detection. British Journal of Educational Technology, 45, 487–501

    Article  Google Scholar 

  • Paquette, L., Ocumpaugh, J., Li, Z., et al. (2020). Who’s learning? Using demographics in EDM research. Journal of Educational Data Mining, 12, 1–30

    Google Scholar 

  • Paulhaus, D. L. (1991). Measurement and control of response bias. In J. P. Robinson, P. R. Shaver & L. S. Wrightsman (Eds.), Measures of personality and social psychological attitudes (pp. 17–59). Academic

  • Piaget, J. (1952). The origins of intelligence in children. International University Press

  • Piaget, J. (1985) The equilibration of cognitive structures: The central problem of intellectual development. University of Chicago Press

  • Pintrich, P. R., & Schunk, D. H. (2002) Motivation in education: Theory, research, and applications. Prentice Hall

  • Posada, D., & Buckley, T. R. (2004). Model selection and model averaging in Phylogenetics: Advantages of Akaike Information Criterion and Bayesian approaches over likelihood ratio tests. Systematic Biology, 53, 793–808

    Article  Google Scholar 

  • Robinson, P. (2001a). Task complexity, cognitive resources and syllabus design: A triadic framework for examining task influences on SLA. In P. Robinson (Ed.), Cognition and second language instruction (pp. 185–316). Cambridge University Press

  • Robinson, P. (2001b). Task complexity, task difficulty, and task production: Exploring interactions in a componential framework. Applied Linguistics, 22, 27–57

    Article  Google Scholar 

  • Robinson, P. (2005). Cognitive complexity and task sequencing: A review of studies in a Componential Framework for second language task design. International Review of Applied Linguistics in Language Teaching, 43, 1–33

  • Rodrigo, M. M. T., Baker, R.S., Agapito, J., Nabos, J., Repa;am, M.C., Reyes Jr., S.S.T., & Pedro, C.S., (2012) The effects of an interactive software agent on student affective dynamics while using an intelligent tutoring system. In: IEEE Transactions on Affective Computing, Vol.3, no. 2, (p 224–236), https://doi.org/10.1109/T-AFFC.2011.41.

  • Rodriguez, F., Yu, R., Park, J., Rivas, M.J., Warschauer, M., & Sato, B.K., (2019) Utilizing learning analytics to map students' self-reported study strategies to click behaviors in STEM courses. In: International conference on learning analytics & knowledge (LAK) (p 456–460). Tempe, AZ, USA, https://doi.org/10.1145/3303772.3303841.

  • Schneider, D. W., & Anderson, J. R. (2010). Asymmetric switch costs as sequential difficulty effects. The Quarterly Journal of Experimental Psychology, 63, 1873–1894

    Article  Google Scholar 

  • Shute, V. J., D'mello, S., Baker, R.S., Cho, K., Bosch, N., Ocumpaugh, J., Ventura, M., Almeda, V., (2015). Modeling how incoming knowledge, persistence, affective states, and in-game progress influence student learning from an educational game. Computers & Education, 86, 224–235, https://doi.org/10.1016/j.compedu.2015.08.001

  • Skehan, P. (1998). A cognitive approach to language learning. Oxford University Press

  • Sreerekha, S., Arun, R. R., & Sankar, S. (2016) Effect of Predict-Observe-Explain strategy on achievement in chemistry of secondary school students. International Journal of Education & Teaching Analytics, 1(1), 1-5.

  • Stephanou, G., Kariotoglou, P., & Dinas, K. D. (2011). University students' emotions in lectures: The effect of competence beliefs, value beliefs and perceived task-difficulty, and the impact on academic performance. International Journal of Learning, 18, 45–72

    Google Scholar 

  • Tao, P. K., & Gunstone, R. F. (1999). The process of coneptual change in force and motion during computer-supported physics instruction. Journal of Research in Science Teaching, 36, 859–882

    Article  Google Scholar 

  • Vosniadou, S. (1994). Capturing and modeling the process of conceptual change. Learning and Instruction, 4, 45–69

    Article  Google Scholar 

  • Vygotsky, L. S. (1978). Mind and society: The development of higher mental processes. Harvard University Press

  • White, R., & Gunstone, R. (1992) Probing understanding. Routledge

  • Yancey, K. B. (1998) Reflection in the Writing Classroom. Utah State University

  • Zou, X., Ma, W., Ma, Z., et al. (2019) Towards helping teachers select optimal content for students. In: Isotani S, Millán E, Ogan A, Hastings P, McLaren B, Luckin R (eds) International Conference on Artificial Intelligence in Education (AIED) (p. 413–417). Springer

Download references

Acknowledgements

We wish to thank Prof. Arial Anbar, Dr Lev Horodyskyj and Dr Chris Mead for providing us with the Habitable Worlds data for this research. We thank Dr Linda Corrin and Donia Malekian for the useful discussion on this work. We are also thankful to the anonymous reviewers for their time and valuable feedback on this paper. The quality of this manuscript has improved because of their insightful suggestions. This research is supported by the Research Training Program (RTP) Scholarship, Melbourne Research Scholarship and the Science of Learning Research Center (SLRC) top-up scholarship.

Author information

Authors and Affiliations

  1. The University of Melbourne, Parkville, VIC, 3010, Australia

    Sadia Nawaz, Namrata Srivastava, Anam Ahmad Khan, Gregor Kennedy & James Bailey

  2. The University of Michigan, Ann Arbor, MI, 48109, USA

    Ji Hyun Yu

  3. University of Pennsylvania, Philadelphia, PA, 19104, USA

    Ryan Shaun Baker

Authors
  1. Sadia Nawaz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Namrata Srivastava
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ji Hyun Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Anam Ahmad Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gregor Kennedy
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. James Bailey
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Ryan Shaun Baker
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sadia Nawaz.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Note from authors

This paper is an expanded version of an earlier conference paper:

Nawaz, S., Srivastava, N., Yu, J. H., Baker, R. S., Kennedy, G., & Bailey, J. (2020). Analysis of Task Difficulty Sequences in a Simulation-Based POE Environment. Artificial Intelligence in Education: 21st International Conference, AIED 2020, Ifrane, Morocco, July 6–10, 2020, Proceedings, Part I, 12163, 423–436. https://doi.org/10.1007/978-3-030-52237-7_34.

Appendix

Appendix

Table 7 Definition of various features for task difficulty detectors
Full size table
Table 8 Probability of TD transitions between consecutive POE tasks.
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nawaz, S., Srivastava, N., Yu, J.H. et al. How Difficult is the Task for you? Modelling and Analysis of Students' Task Difficulty Sequences in a Simulation-Based POE Environment. Int J Artif Intell Educ 32, 233–262 (2022). https://doi.org/10.1007/s40593-021-00242-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40593-021-00242-6

Keywords

Search

Navigation