Abstract
The use of programming online judges (POJ) to support students acquiring programming skills is common nowadays because this type of software contains a large collection of programming exercises to be solved by students. A POJ not only provides exercises but also automates the code compilation and its evaluation process. A common problem that students face when using POJ is information overload, as choosing the right problem to solve can be quite frustrating due to the large number of problems offered. The integration of current POJs into e-learning systems such as Intelligent Tutoring Systems (ITSs) is hard because of the lack of necessary information in ITSs. Hence, the aim of this paper is to support students with the information overload problem by using a collaborative filtering recommendation approach that filters out programming problems suitable for students’ programming skills. It uses an enriched user-problem matrix that implies a better student role representation, facilitating the computation of closer neighborhoods and hence a more accurate recommendation. Additionally a novel data preprocessing step that manages anomalous users’ behaviors that could affect the recommendation generation is also integrated in the recommendation process. A case study is carried out on a POJ real dataset showing that the proposal outperforms other previous approaches.
Similar content being viewed by others
References
Adomavicius G, Tuzhilin AT (2005) Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Trans Knowl Data Eng 17(6):734–749
Ala-Mutka KM (2005) A survey of automated assessment approaches for programming assignments. Comput Sci Educ 15(2):83–102
Amatriain X, Pujol JM (2015) Data mining methods for recommender systems. In: Recommender Systems Handbook. Springer, pp 227–262
Amatriain X, Pujol JM, Oliver N (2009a) I like it... i like it not: Evaluating user ratings noise in recommender systems. In: User modeling, adaptation, and personalization. Springer, pp 247–258
Amatriain X, Pujol JM, Tintarev N (2009b) Rate it again: increasing recommendation accuracy by user re-rating. In: Proceedings of the third ACM conference on Recommender systems. ACM, pp 173–180
Arefin AS (2006) Art of Programming Contest. Gyankosh Prokashonia
Caiza J, Del Amo J (2013) Programming assignments automatic grading: Review of tools and implementations. In: Proceedings of INTED, vol 2013, pp 5691–5700
Campos PG, Díez F, Cantador I (2014) Time-aware recommender systems: a comprehensive survey and analysis of existing evaluation protocols. User Model User-Adap Inter 24(1-2):67–119
Castro J, Yera Toledo R, Martínez L (2016) An empirical study of natural noise management in group recommendation systems. Decision Support Systems. doi:10.1016/j.dss.2016.09.020
Chen LC, Kuo PJ, Liao IE (2015) Ontology-based library recommender system using mapreduce. Clust Comput 18(1):113–121
Christensen I, Schiaffino S (2014) Social influence in group recommender systems. Online Inf Rev 38 (4):524–542
Dascalu MI, Bodea CN, Mihailescu MN, Tanase EA, Ordoñez de Pablos P (2016) Educational recommender systems and their application in lifelong learning. Behav Inform Technol 35(4):290–297
De Maio C, Fenza G, Gaeta M, Loia V, Orciuoli F, Senatore S (2012) Rss-based e-learning recommendations exploiting fuzzy fca for knowledge modeling. Appl Soft Comput 12(1):113–124
De Oliveira MG, Ciarelli PM, Oliveira E (2013) Recommendation of programming activities by multi-label classification for a formative assessment of students. Expert Syst Appl 40(16):6641–6651
Derrac J, García S, Molina D, Herrera F (2011) A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. Swarm Evol Comput 1(1):3–18
Drachsler H, Verbert K, Santos OC, Manouselis N (2015) Panorama of recommender systems to support learning. In: Recommender systems handbook. Springer, pp 421–451
Fenza G, Orciuoli F (2016) Building pedagogical models by formal concept analysis. In: International Conference on Intelligent Tutoring Systems. Springer, pp 144–153
Gaeta M, Orciuoli F, Paolozzi S, Salerno S (2011) Ontology extraction for knowledge reuse: The e-learning perspective. IEEE Trans Syst Man Cybern Part A Syst Humans 41(4):798–809
Gruber TR (1995) Toward principles for the design of ontologies used for knowledge sharing? Int J Hum Comput Stud 43(5):907–928
Guarino N, Giaretta P (1995) Ontologies and knowledge bases towards a terminological clarification. In: Towards very large knowledge bases: knowledge building & knowledge sharing. IOS Press, pp 25–32
Gunawardana A, Shani G (2009) A Survey of Accuracy Evaluation Metrics of Recommendation Tasks. J Mach Learn Res 10:2935–2962
Hsiao IH, Sosnovsky S, Brusilovsky P (2010) Guiding students to the right questions: adaptive navigation support in an e-learning system for java programming. J Comput Assist Learn 26(4):270–283
Huang Z, Zeng D, Chen H (2007) A comparison of collaborative-filtering recommendation algorithms for e-commerce. IEEE Intell Syst 5:68–78
Klaṡnja-Milićević A, Vesin B, Ivanović M, Budimac Z (2011) E-learning personalization based on hybrid recommendation strategy and learning style identification. Comput Educ 56(3):885–899
Klaṡnja-Milićević A, Ivanović M, Nanopoulos A (2015) Recommender systems in e-learning environments: a survey of the state-of-the-art and possible extensions. Artif Intell Rev 44(4):571–604
Krishnan S, Patel J, Franklin MJ, Goldberg K (2014) A methodology for learning, analyzing, and mitigating social influence bias in recommender systems. In: Proceedings of the 8th ACM Conference on Recommender systems. ACM, pp 137–144
Kurnia A, Lim A, Cheang B (2001) Online judge. Comput Educ 36(4):299–315
Leal JP, Silva F (2003) Mooshak: a web-based multi-site programming contest system. Software: Practice and Experience 33(6):567–581
Li B, Chen L, Zhu X, Zhang C (2013) Noisy but non-Malicious user detection in social recommender systems. World Wide Web 16(5-6):677–699
Llana L, Martin-Martin E, Pareja-Flores C, Velázquez-Iturbide JA ́ (2014) Flop: a user-friendly system for automated program assessment. Journal of Universal Computer Science 20(9):1304–1326
Lops P, Gemmis M, Semeraro G (2011) Content-based recommender systems: State of the art and trends. In: Recommender systems handbook springer US, chap, vol 3, pp 73–105
Lu J, Wu D, Mao M, Wang W, Zhang G (2015) Recommender system application developments: a survey. Decis Support Syst 74:12–32
Martínez L, Pérez L G, Barranco M (2007) A multigranular linguistic content-based recommendation model. Int J Intell Syst 22(5):419–434
Martínez L, Barranco MJ, Pérez LG, Espinilla M (2008) A knowledge based recommender system with multigranular linguistic information. International Journal of Computational Intelligence Systems 1(3):225–236
Miranda S, Orciuoli F, Sampson DG (2016) A skos-based framework for subject ontologies to improve learning experiences. Comput Hum Behav 61:609–621
Murray T (1999) Authoring intelligent tutoring systems: an analysis of the state of the art. Int J Artif Intell Educ 10:98–129
Nadolski RJ, Van den Berg B, Berlanga AJ, Drachsler H, Hummel HG, Koper R, Sloep PB (2009) Simulating light-weight personalised recommender systems in learning networks: a case for pedagogy-oriented and rating-based hybrid recommendation strategies. Journal of Artificial Societies and Social Simulation 12(1):4
Ning X, Desrosiers C, Karypis G (2015) A comprehensive survey of neighborhood-based recommendation methods. In: Recommender Systems Handbook. Springer, pp 37–76
Noguera J, Barranco M, Segura R, Martínez L (2012) A mobile 3d-gis hybrid recommender system for tourism. Inf Sci 215:37– 52
O’Mahony MP, Hurley NJ, Silvestre G (2006) Detecting noise in recommender system databases. In: Proceedings of the 11th international conference on Intelligent user interfaces. ACM, pp 109–115
Petit J, Giménez O, Roura S (2012) Jutge. org: an educational programming judge. In: Proceedings of the 43rd ACM technical symposium on Computer Science Education. ACM, pp 445– 450
Pham HX, Jung JJ (2013) Preference-based user rating correction process for interactive recommendation systems. Multimedia tools and applications 65(1):119–132
Piramuthu S, Kapoor G, Zhou W, Mauw S (2012) Input online review data and related bias in recommender systems. Decis Support Syst 53(3):418–424
Polson MC, Richardson JJ (2013) Foundations of intelligent tutoring systems. Psychology Press
Regueras LM, Verdú E, Muňoz MF, Pérez MA, De Castro JP, Verdú MJ (2009) Effects of competitive e-learning tools on higher education students: a case study. IEEE Trans Educ 52(2):279–285
Ricci F (2015) Recommender systems handbook. Springer, Shapira B
Ruiz-Iniesta A, Jimenez-Diaz G, Gomez-Albarran M (2014) A semantically enriched context-aware oer recommendation strategy and its application to a computer science oer repository. IEEE Trans Educ 57(4):255–260
Said A, Jain BJ, Narr S, Plumbaum T (2012) Users and noise: The magic barrier of recommender systems. In: User modeling, Adaptation, and Personalization. Springer, pp 237–248
Skiena SS (2006) Revilla MA. The programming contest training manual. Springer Science & Business Media, Programming challenges
Sleeman D, Brown JS (1982) Intelligent tutoring systems. Academic Press, London
Thai-Nghe N, Drumond L, Horváth T, Nanopoulos A, Schmidt-Thieme L (2011) Matrix and tensor factorization for predicting student performance. In: Proceedings of the 3rd International Conference on Computer Supported Education (CSEDU), pp 69–78
Verdú E, Regueras LM, Verdú MJ, Leal JP, de Castro JP, Queirós R (2012) A distributed system for learning programming on-line. Comput Educ 58(1):1–10
Vesin B, Klaṡnja-Milićević A, Ivanović M, Budimac Z (2013) Applying recommender systems and adaptive hypermedia for e-learning personalization. Computing and Informatics 32(3):629–659
Wang GP, Chen SY, Yang X, Feng R (2016) Ojpot: online judge & practice oriented teaching idea in programming courses. Eur J Eng Educ 41(3):304–319
Wang PY, Yang HC (2012) Using collaborative filtering to support college students’ use of online forum for english learning. Comput Educ 59(2):628–637
Winoto P, Tang TY, McCalla GI (2012) Contexts in a paper recommendation system with collaborative filtering. The International Review of Research in Open and Distributed Learning 13(5):56–75
Yera Toledo R, Caballero Mota Y (2014) An e-learning collaborative filtering approach to suggest problems to solve in programming online judges. International Journal of Distance Education Technologies 12(2):51–65
Yera Toledo R, Caballero Mota Y, Garcia Borroto M (2013) A regularity-based preprocessing method for collaborative recommender systems. J Inf Process Syst 9(3):435– 460
Yera Toledo R, Caballero Mota Y, Martínez L (2015) Correcting noisy ratings in collaborative recommender systems. Knowl-Based Syst 76:96–108
Yera Toledo R, Castro J, Martínez L (2016) A fuzzy model for managing natural noise in recommender systems. Appl Soft Comput 40:187–198
Yu R, Cai Z, Du X, He M, Wang Z, Yang B, Chang P (2015) The research of the recommendation algorithm in online learning. International Journal of Multimedia and Ubiquitous Engineering 10(4):71–80
Acknowledgments
This research work was partially supported by the Research Project TIN2015-66524-P.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Yera, R., Martínez, L. A recommendation approach for programming online judges supported by data preprocessing techniques. Appl Intell 47, 277–290 (2017). https://doi.org/10.1007/s10489-016-0892-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-016-0892-x