Introduction

Modern societies require a college and/or university degree as a pillar for economic progress and responsible citizenship (Kuh et al. 2008). Nevertheless, students face certain problems during their university studies, so they drop out or take more time in obtaining their degrees (Berkner et al. 2002).

The first year in higher education is probably one of the most important changes in a student’s academic trajectory (Chemers et al. 2001). First-year students are at the greatest risk of dropping out or not achieving acceptable grades (Horstmanshof and Zimitat 2007; Kovacic 2010; Strayhorn 2009). This transition demands high levels of self-regulation, adequate coping strategies to new academic problems and situations, efficient use of the student’s cognitive skills, and the presence of other favorable circumstances in the student’s life in order to make academic success possible (Bryde and Milburn 1990; Kuh, Kinzie, Schuh, Whitt 2005; Strayhorn 2009).

A considerable number of studies have reported the predictive role of GPA (as a proxy for overall academic performance) for the expected academic performance in other levels of education (Kuncel & Hezlett, 2004; Kuncel, Crede, & Thomas, 2005), job performance (Roth, BeVier, Switzer & Schippmann, 1996), and salary (Roth & Clarke, 1998). In addition, the literature suggests other contextual factors impacting on academic retention (Jun 2005; Kovacic 2010). Moreover, previous studies indicate that successful completion of a university degree is a complex phenomenon explained by a large and diverse set of factors: students’ individual differences, institutional characteristics, and environmental contingencies (Astin & Oseguera, 2012).

Applications of data science in the prediction of certain and complex educational outcomes, combined with machine learning approaches, have increased during the last decade (Abu Naser 2012; Kanakana and Olanrewaju 2011; Musso et al. 2012; Ramaswami and Bhaskaran 2010; Zambrano Matamala et al. 2011). However, it would be advantageous to extend these applications to a wide range of problems during all of the students’ university trajectory, and to different socio-economic contexts such as those in developing countries. One of the crucial problems in these contexts refers to the very low achievement levels in math, reading, and sciences in Latin American countries when compared with high-income countries (OECD, 2019). Students and professors have to deal with this gap between what students bring and the demands of university curricula. Another problem refers to the low graduation rates observed in these countries. Although higher education enrollment has doubled in the last decades, there are several quality concerns regarding the low internal efficiency of the tertiary institutions (Holm-Nielsen et al. 2005). For example, in Argentina, there is a drop-out rate of more than 40% among first-year students (Marquis, 2003; Theiler 2005). In addition, students in Argentina spend more time than it is expected to complete their studies (MECYT 2000), and all these problems contribute to very low completion rates (MECYT 2000).

In this research, we have developed machine learning models in order to accurately predict significant educational outcomes during the students’ university trajectories and, at the same time, to identify the contribution of each variable to each of these various outcomes. This study is focused on the early prediction of three of these key outcomes in the university students’ academic life: (1) GPA at the end of the first academic year; (2) drop-outs at the end of the second academic year; and (3) successful degree completion within a 5-year period. Moreover, this research adds the identification of specific variables, and their relative importance, that participate in the accurate prediction of these educational outcomes. This is a very substantial contribution for the Latin American university context, specifically in Argentina, where educational decisions and the research that informs them are, at best, based on the application of classical statistical analyses, and where there has been very little development of educational data mining and new statistical technics in education.

Machine learning approaches as artificial neural networks (ANN) allow the use of large volumes of data and non-linear relationships between predictors, and they have been shown to be very effective to classify various educational outcomes (e.g., Abu Naser 2012; Ahmad & Shahzadi 2018; Kanakana and Olanrewaju 2011; Musso et al. 2012, 2013; Lau, Sun & Yang 2019). Other advantages of ANN are that they do not require the fulfillment of assumptions of normality, linearity, and completeness (Kent 2009, Garson 1998). In addition, ANN are robust predictive systems with multiple non-parametric applications, even when a small number of data points are available for the analysis (Garson 1998). Furthermore, with the use of new modeling techniques, it is possible to identify the level of participation of each variable involved in the modeling of the problem, while achieving great accuracy in the predictive classifications, at levels of precision not usually achieved by traditional approaches. Although there are also non-linear regression methods available, two limitations about their use should be acknowledged. First, the number of ways to combine the parameters in the regression equation is very large, which might convert the fitting of the non-linear model in a trial-and-error task. Second, it is rather common to estimate the variance accounted by a non-linear model through the coefficient of determination (R2). However, in a non-linear model, R2 is not the correct choice given that the total variance of the model is not equal to the explained variance plus the error variance (Spiess and Neumeyer 2010).

The early identification of vulnerable and successful students is a very relevant issue, with a high degree of impact for the students themselves and for academic and administrative staff at the university. Obtaining early-warning information on students at risk can help staff plan and implement support, retention strategies, and other pro-active measures to facilitate the attainment of positive educational outcomes of the at-risk student population. Such a program would also benefit all the student population, with more successful allocation of students to advanced or more challenging courses (Kovacic 2010; Musso et al. 2012). Thus, the early prediction of key outcomes in a student’s academic trajectory such as grade point average (GPA), academic retention, and degree completion would allow targeted intervention programs in higher education.

Theoretical framework

In order to understand the patterns of variables predicting educational outcomes in higher education, we used an approach based on the concept of structured neural network (Lee, Rey, Mentele, & Garver, 2005). This approach implicates the consideration of robust theoretical models to design the NN structure guiding the selection of the predictors. In this way, the relationships between inputs and the output variables become more transparent when we want to interpret them. Therefore, the NN structure used in this study is based on the most important theoretical constructs found in the self-regulated learning field, cognitive and educational sciences.

Several integrative models explaining academic performance have emerged in the last twenty five years in the self-regulated learning literature from different theoretical and methodological approaches (e.g., Boekaerts 1997; Pintrich, 2000; Zimmerman & Schunk, 2011). They have outlined some cognitive and non-cognitive variables impacting academic performance in general and in some specific domains (e.g., Boekaerts, Pintrich, & Zeidner, 2000; De Corte, Mason, Depaepe, & Verschaffel, 2011; Zimmerman & Schunk, 2011). Cognitive variables include, but are not limited to, working memory capacity (WMC) (Engle 2002; Musso et al. 2019), attention (Kyndt, Cascallar & Dochy, 2012; Riccio, Lee, Romine, Cash, & Davis, 2002), and learning strategies (Weinstein et al. 1987). Among non-cognitive variables, research suggests sociodemographic background information variables (Jun 2005; Kovacic 2010), motivational/coping strategies (Boekaerts and Niemivirta 2000; Boekaerts 1997), and social support (Scott, Spielmans, & Julka, 2004).

Previous research has shown that academic achievement in the first academic year and retention have different causes (e.g., Tross, Harper, Osher, & Kneidinger, 2000). For example, Scott et al. (2004) have found that high school GPA was a very important predictor of academic achievement but not for retention. Moreover, they suggest the development of more sophisticated models to predict a complex construct such as academic retention (Scott, Spielmans, & Julka, 2004). In addition, the literature on drop-out in higher education has shown the importance of subjective reasons to leave university (Ulriksen, Madsen, & Holmegaard, 2010; Stadler, Becker, Greiff, & Spinath, 2015). Although academic performance, retention, and degree attainment are not the same phenomenon, they share some common variables that are described as factors impacting on general academic success.

WMC is a limited control system that enables the active maintenance and processing of information (Conway et al., 2005). Individual differences in both storage and processing components of WMC predict differences in reading, reasoning, math performance, and complex problem solving (Colom et al. 2007; Conway et al. 2002; Engle and Kane 2004; Unsworth, Redick, Heitz, Broadway, & Engle, 2009). Students with high WMC can maintain more activated information and apply effectively learning strategies according to a specific learning goal (Dunlosky and Kane 2007; Dunlosky and Thiede 2004; Dunning and Holmes 2014). Learning strategies have been defined as procedures (behaviors or thoughts) used for acquiring and integrating new information with previous knowledge (Weinstein, Palmer, & Schulte, 1987).

This research also studied another cognitive system as a predictor of academic performance: attentional networks, which include orienting, alerting, and executive control processes (Fan et al. 2002; Posner and Petersen 1989). The orienting network refers to the process that participates in the selection of information from stimuli entering into the system. The alerting network allows the cognitive system to achieve and sustain an alert state. Executive attention involves a control of interference solving conflicts between all possible responses (Fan et al. 2002) while maintaining the focus on the task (Checa & Rueda, 2011, Kane, Conway, Hambrick, & Engle, 2008; Posner, Rothbart, Sheese, & Voelker, 2014).

Motivational and coping strategies also play a crucial role when students deal with an academic task, impacting on educational outcomes (Boekaerts and Niemivirta 2000; Boekaerts 1997; Lazarus and Folkman 1986). When students self-regulate their learning process, they are seeking to first identify and improve their skills and, second, to maintain their personal well-being (Boekaerts and Niemivirta 2000). Positive appraisals of the learning situation (e.g., self-concept) activate a mastery mode where learning strategies and other resources are destined to increase in competence. Negative appraisals activate a coping mode in order to prevent the loss of resources (Boekaerts and Niemivirta 2000). In addition, coping strategies involve those cognitive, emotional, and behavioral efforts that are developed to handle stressors (Lazarus and Folkman 1986), specifically academic situations perceived as stressful by the student. Previous studies have also demonstrated the importance of social support for students to cope with academic stress during their transition to college (Fisher and Hood 1987), predicting academic achievement (Scott, Spielmans, & Julka, 2004).

Literature reviews have also identified several non-cognitive variables predicting drop-out: individual sociodemographic background, academic and social integration, technological support, and motivation (Jun 2005; Kovacic 2010). Background factors involve socio-demographic and environmental variables (gender, age, occupation, among others). The evidence regarding the contribution of background characteristics is controversial in the literature because it depends on the academic level, the actual variables involved, and the methodological approach used (Kovacic 2010). Although background factors were significant for the prediction of academic performance, the overall predictive accuracy was relatively low, around 60% (Kovacic 2010). According to Kovacic (2010), the background information collected in the enrollment process does not contain sufficient information to predict accurately which students will be successful or not in their studies.

Materials and methods

Participants

The study was carried out in the Department of Psychology at a large private university in Buenos Aires, Argentina. Students enrolled in social sciences have to take General Psychology as part of their program, during their first academic year. Therefore, the participants were 655 undergraduate students (female 52.3%; age: M = 19.90, SD = 3.43; cohorts 2009, 2010, and 2011), from introductory psychology classes in the social science disciplines at this university (Psychology, Communication, Business, and Marketing); most of the students were from medium-high socioeconomic status. Students participated voluntarily in this research, and they did not receive economic incentives for participation. Instead, they satisfied a brief-report requirement for their classes describing their cognitive performance, from the feedback information given to them.

Measures

Attention Network Test (ANT) (Fan et al. 2002)

This is a computerized test to assess three attentional networks: alerting, orienting, and executive attention in terms of reaction times expressed in milliseconds (for a more detailed description of the test, see Fan et al. 2002). Reliability studies have indicated a high reliability for total reaction time (.87) and acceptable test-retest consistency for each network (.77 for executive attention, .61 for orienting, and .52 for alerting) (Fan et al. 2002).

Automated Ospan (AOSPAN)

This is a computerized test that measures WMC given by the information processing capacity of the cognitive system while an interference task is presented (see Unsworth et al. 2005 for a task description). Reaction times also are included. Test-retest reliabilities for the absolute score AOSPAN is .77. The reliability studies indicated relatively small practice effects on the AOSPAN, and the rank-ordering of individuals was stable across test sessions (Redick et al. 2012).

Learning Strategies Questionnaire (LASSI, Weinstein and Palmer 2002)

A validated Spanish version of the LASSI was administered (Meza and Lazarte 1998). It is a 77-item questionnaire grouped in 10 subscales that assesses “the students’ awareness about, and use of, learning and study strategies related to skill, will, and self-regulation components of strategic learning” (Weinstein and Palmer 2002; p. 4): Time Management Scale (α = .85), Concentration Scale (α = .86), Information Processing Scale (α = .84), Selecting Main Ideas Scale (α = .89), Study Aids Scale (α = .73), Self-evaluation Scale (α = .84), Test Strategies Scale (α = .80), Attitude Scale (α = .77), Motivation Scale (α = .84), and Anxiety Scale (α = .87).

Adolescent Coping Scale, Spanish version (Richaud de Minzi 2003)

This is a 46-item self-report measure to assess 11 coping strategies of academic situations perceived as stressful: “Cognitive Redefinition, Self-Blame, Fatalism, Evasion through Amusement, Problem-Focused Coping (which includes requests for information and action), Evasion Through Physical Activity, Emotional Support, Emotional Discharge and Somatization, Anxiety, Isolation, and No Action” (Richaud de Minzi 2003; p. 6). The reliability coefficients were found to be satisfactory. Reliability, as internal consistency, was adequate (α range = .56 to .70).

Perceived Social Support Scale, Spanish version of Multidimensional Scale of Perceived Social Support (Zimet, Dahlem, Zimet, & Farley 1988, Arechabala & Miranda, 2002)

This is a 12-item self-report Likert scale collecting information on social support perceived by students in three areas: family, friends, and significant others. Exploratory and confirmatory factor analyses found three factors (family, friends, and significant other), and the reliability was moderate to high (α = .839; α = .907, and α = .846, respectively).

SMU Health Questionnaire (SMU-HQ) (Watson and Pennebaker 1989)

This questionnaire was applied to measure a broad range of health problems. It consists of 63 items regarding symptoms and complaints, minor illnesses, and more serious and chronic health problems experienced during the last year (α = .71).

Remoralization Scale (Spanish version) (Musso et al. 2017)

This is a brief 12-item questionnaire with a 4-point Likert scale to assess self-satisfaction (α = .813) and self-concept (α = .794).

Socio-demographic questionnaire

A broad set of background information was collected: age, gender, environmental conditions, type of high school, socio-economic data, and level of education and occupation of parents.

Grade point average

It was collected from official records at the beginning of the second academic year. A total of 341 grades were collected for the study with this data set (rating scale from 0 to 10). It must be noted that collecting final GPA information required a series of approvals (the student’s consent and the examination office) and proved to be very difficult. Following up on the group not represented in the GPA data, no significant differences were found in any of the variables with the group whose GPA was collected, and they had quite similar drop-out rates (11.1% within the missing GPA group; 9.2% within the GPA group).

Academic retention

It was collected at the end of the registration period at the beginning of year three. It measured whether the student registered (or not) the following academic year (between years two and three) in the same discipline at the time data were collected (active students in year three = 589; non-active (drop-outs) = 66). This drop-out group did not transfer to any other discipline at the university, and it represents the average drop-out rate for these disciplines at the institution.

Degree completion (“degree completion” n = 231; “no-degree” within 5 years = 185; “non-active” group = 173)

It was measured by establishing if the student completed (or not) the degree (in which the student was registered at the time data were collected) within a period of 5 years. The “no-degree” group was still enrolled in the discipline; the “non-active” group abandoned their studies between their third and fifth years. The ANN was designed to classify only between “degree completion” and “no-degree” groups.

Procedure and analyses

Institutional permission for carrying out this research with human subjects was obtained, and the study protocol was reviewed and approved by the ethics committee of the university. Students participated voluntarily, and they signed an informed consent following the APA Code of Conduct guidelines (American Psychological Association, 2002). They were informed about the purpose of the research, the session duration, their right to decline to participate without any penalty, the information that would be collected, and their right to withdraw at any point from the research.

All the cognitive tests and self-report scales were individually administered in the same order during a 2-h session in a computer-based classroom, at the beginning of the second semester of the first academic year (cohorts 2009, 2010, and 2011). The GPA of the participating students was collected from official records at the beginning of the second academic year. In addition, information on the students who continued or not registered at university at the start of their third year was collected at the end of the registration period for this university. Finally, the data about degree attainment (the same discipline in which students were enrolled when testing was applied) were collected from official records, after five years from the time they began their studies.

Several multilayer perceptrons (MLP) were built with the purpose of identifying the ANN which best classified low 33% and high 33% levels of GPA, academic retention, and degree completion. Some predictors were introduced in the ANN as continuous variables, and other ones were categorized. The categories of “high” and “low” for working memory, executive control, and other non-cognitive variables (social support, friends, self-concept, etc.) were defined as those in the top and lower third (percentiles 33 and 66), respectively.

The sample was split into two random sets: a training and a validation set (see percentages in Table 1). Table 1 shows the ANN architecture for each model predicting levels of GPA, academic retention, and degree completion. A model of the parameter weights is developed in the training phase using the vector matrix involving all predictor variables for each student. The ANN gives preliminary weights to each predictor and its interactions and changes these weights as the learning progresses. The backpropagation algorithm uses the error term to adjust the weights in the hidden layer in order to minimize the error, and gradually improves the classification outcome through an iterative learning process. Therefore, the correct classification for each record is known to the network, so that the output node can be assigned a “correct” or “incorrect” classification based on the probability of the case belonging to one or the other of the classification classes. Different parameters of the MLPs (learning rate, momentum, number of hidden layers, and transfer functions) were modified in order to maximize both predictive classification of each class and total accuracy. Once the NN model has reached predetermined stopping criteria (e.g., desired level of accuracy, number of steps without change), the validation phase starts. This phase runs the same model optimized during the training, but this time on the randomly selected sample of cases that were not included in the training phase. In this validation set, the correct classification for each case vector is not given to the network. In order to evaluate the network and to observe any evidence of overfitting, the accuracy of the classification of these new cases is observed. To evaluate the quality of the obtained models, confusion matrices and ROC analyses were used. Overall results show both the predictive classifications and the predictive weight of each input variable.

Table 1 Architecture of ANN for GPA, academic retention, and degree completion
Full size table

The top predictors were analyzed considering the largest incremental change in predictive weight as cut point. In addition, different patterns of variables were analyzed comparing the importance of their contribution to the prediction of the various key moments in the academic trajectories.

Results

ANN predicting GPA, academic retention, and degree completion

Results for each ANN for GPA, academic retention, and degree completion are presented in Table 2. These measures allow the evaluation of the quality of the solutions offered by the neural network models designed.

Table 2 Measures for ANN in the prediction of GPA, academic retention (AR), and degree completion (DC) in the validation phase
Full size table

The ANN models achieved very high accuracy for each of the outputs. Both measures, “accuracy for the target group” and “accuracy for the rest group,” are the percentages of the correct classifications in each group (low and high 33% in the GPA model; drop-out and retention for the academic retention model; and degree completion and no degree completion for the third model). As Table 2 shows, all ANN models achieved solutions with excellent “recall” (or sensitivity) that refers to the proportion of correctly identified targets, out of all targets actually presented in the set. In addition, the “precision” and “specificity” measures were very good. Precision represents the proportion of correctly identified targets, out of all true targets presented to the system. Specificity is the proportion of correctly identified non-targets, out of all true-non-targets presented in the set. The F1 score is the harmonic mean of precision and recall, taking both false positives and false negatives into account. Therefore, it is a more comparable measure across studies with different proportions of cases in each class. The area under the ROC curve represents the true-positive rate (sensitivity) plotted as a function of the false-positive rate (specificity) for different cut-off points, and it can be viewed as a measure of the overall model performance across all possible thresholds, that is, how well it distinguishes between two groups.

Predictive contribution of variables for GPA, academic retention, and degree completion

The most important predictors for the classification of students belonging to the low or high 33% groups of GPA were motivation (as learning strategy), isolation (as coping strategy), processing of information (learning strategy), and the total reaction time of attentional mechanism (cognitive measure from the ANT test). The Appendix shows the predictors sorted by predictive weights and the significant differences found between low- and high-GPA groups. The low 33% GPA group has low scores in learning strategies: motivation (t test = 3.96 (189); p < .001), (t test = 2.06 (184); p < .05); test strategies (t test = − 5.58; p < .001); management of anxiety (t test = − 2.07; p < .05) and time (t test = − 3.32; p < .001); concentration (t test = − 2.83; p < .01); main ideas techniques (t test = − 2.71; p < .01); and low attitude (t test = − 2.90; p < .01). Students in the low 33% GPA group also have more isolation as coping strategy (t test = 2.06; p < .05) and slow RT (t test = 2.16; p < .05; t test = 2.68; p < .01). In addition, gender was associated with levels of GPA (X2 = 23.366; p < .001). Only 37.6% of female students were in the low GPA level, but 74.6% of male students were in this low-level group. Levels of WMC were related to GPA (X2 = 6.287, p < .05): 40% of high-WMC students were in the low-GPA group and 60% in the high-GPA group; 56.2% of low WMC were in the low-GPA group and 43.8% in the high-GPA group; 58.8% of moderate WMC were in the low-GPA group and 41.2% in the high group.

The predictors were grouped according to categories developed by expert judges. Figure 1 summarizes the predictive weights of each category. Learning and coping strategies are the most important categories predicting low and high levels of GPA (see Fig. 1).

Fig. 1
figure 1

Comparative predictive weight contribution for GPA by each of the categories of predictor variables

Full size image

Results from the ANN for those students expected to be in the drop-out group or retention group show that the top predictors with the most significant participation in the discrimination between the two categories were total courses completed, and emotional discharge as coping strategy of academic situations. Variables and their predictive weights for this ANN are presented in the Appendix. It also shows the significant differences between drop-out vs retention groups. Students that drop out had a significantly lower GPA than students that continue their studies (t test = − 5.274; p < .001), lower number of courses completed (t test = − 6.207; p < .001), lower size living group (t test = − 2.88; p < .01) and lower number of friends at university (t test = − 2.11; p < .05), and lower-frequency use of the internet at university (t test = − 2.089; p < .05) and at the library (t test = − 2.167; p < .05); they worked more hours per week (t test = 4.443; p < .001), and they perceived more social support from other special person (t test = 2.236; p < .05). However, 29.5% of students in the drop-out group had no friends in class, compared with only 14.2% of the retained group (X2 = 7.198, p < .05). In addition, 52.3% of the drop-out group worked while only 33.5% of the retained group did (X2 = 6.192; p < .05). Most of the group (72.7%) that dropped out did not have extracurricular activities, compared with 51.8% of the retained group (X2 = 7.083; p < .01). The predictors were grouped according to the same categories as for GPA. Figure 2 summarizes the predictive weights of each category. Background variables (size of the group living with, number of books, parent education, etc.), coping strategies, and learning strategies are the most important categories contributing to the prediction of academic retention (see Fig. 2).

Fig. 2
figure 2

Comparative predictive weight contribution for academic retention by each of the categories of predictor variables

Full size image

The most important variables, according to their predictive weights, classifying students that finished their degree or not within the 5-year period were two learning strategies (selecting main ideas, information processing, and management of anxiety), coping strategies (anxiety, emotional discharge, etc.) (see Appendix). Students who achieved their degree reported more use/level of learning strategies: study aids (t test = 2.511; p < .05) and test strategies (t test = 4.020; p < .001). In addition, the degree group used more the social networks. They also are younger students and they had higher GPA (t test = 5.893; p < .001) and faster RT (t test = − 2.143; p < .05). Within the degree group, there was a higher percentage of students with high “social support from other” (56.5%), higher percentage of females (75.6%), less students working (28.9%), and higher percentage of students with friends in the classes (91.8%) compared with the no-degree group (82.4%). The predictors were grouped according to the same categories as for the other ANN. Figure 3 summarizes the predictive weight of each category. Coping and learning strategies are the most important categories contributing to the prediction of degree completion (see Fig. 3).

Fig. 3
figure 3

Comparative predictive weight contribution for degree completion by each of the categories of predictor variables

Full size image

Discussion

Robustness of the method approach

An important challenge in developing an early warning system of possible future negative or positive educational outcomes is the determination of the most reliable predictors of academic success (Beck and Davidson 2001). A large body of research highlights cognitive and non-cognitive factors determining different key indicators of academic success (Beck and Davidson 2001). The present research approached this challenge through the development of several neural network models using multiple variables, already identified in the literature as related to the outcomes of interest, in order to predict—early and with very high accuracy—three important educational outcomes at different points in the students’ trajectory.

Consistent with previous applications of machine-learning approaches developing predictive systems in the education and health fields (Abu Naser 2012; Herzog 2006; Zambrano Matamala et al. 2011), the classification accuracy of the three neural network models was very high for (1) those students that would have either a low or high GPA at the end of their first academic year; (2) students that would drop out or be retained at the beginning of their third academic year; and (3) students that would finish their degree or not within a 5-year period.

The results of the present study have demonstrated the predictive power of ANN compared with some other measures currently used. For example, the combined SAT test score from The College Board explains approximately 28% of the variance of first-year college GPA (Shaw et al. 2016). Even when high-school GPA is considered together with the combined SAT score, only 34% of first-year GPA variance is explained (Shaw et al. 2016), and the SAT does not seem to predict success in university beyond what prior academic achievement already indicates (Kirkup et al. 2010). In addition, although classical statistical analyses do not detect significant differences between groups in some predictors, the ANN can use information of the multiple interactions between all predictors to estimate the classification of each student, thus increasing the information available to the network for the estimation of the output. This robustness of ANN was also found in previous studies, even when they are faced with a small number of data points (Garson 1998).

The level of accuracy observed in this study depends on two important conditions: (1) the amount of information provided by all the predictor variables in the model and (2) the level of precision (or “granularity”) desired on the dependent variable criteria for classification. That is, if instead of a 33% interval for the classification, a more rigorous 10% of either high or low cases was desired to be achieved through the classification, a much lower accuracy (both in terms of true-positive and true-negative correct classifications) would likely be achieved. In fact, in our study, although the institution behind this research had requested a 33% classification for high- and low-GPA students, we developed models for the top 25% GPA candidates and for the 25% low-GPA candidates. In both instances, the accuracy was reduced to between 85% and 95% for the various models. In the case of the retention and the degree-completion data, the model simply classified between those that did or did not belong to either category in both research questions, resulting in a binary and relatively easy classification for the optimal models found, given the amount of information provided by the variables in the study.

Relative importance of the predictors

Firstly, we need to take into account the relatively small contribution of each predictor variable (predictive weights, as contribution to the prediction, between 7.4% and .2%). This finding demonstrates that what is important for the accuracy of the predictive classification is the combined effect of the full vector of variables and their interactions for each case, rather than just the value of each individual variable. Most of the information that the model works with is derived from the complex pattern of interactions resulting in unique vectors for each case. It is the “learning” of these various complex vectors in the data set that provides the information for the ANN to carry out its predictive classification, by looking for the overall minima of the corresponding functions. In addition, the relative importance analyses of the ANN allow the ranking of the variables contributing to the early prediction of crucial educational outcomes (Musso et al. 2012). This information provides guidance to prioritize certain objectives in more targeted and focused interventions on those variables over which the institution has certain control.

A wide range of previously existing individual competences to face new academic challenges provides relevant information for the prediction of GPA, retention, and degree completion. Our results are coherent with the Model of Adaptable Learning (Boekaerts and Niemivirta 2000) which states that when students self-regulate their learning, they not only want to improve certain content or skills but also wish to maintain their personal well-being and personal values. In addition, the association between socio-emotional competences and academic success has been analyzed especially in the transition from high school to university (Brooks and DuBois 1995; Pancer et al. 2000; Perry et al. 2001; Pratt et al. 2000). The present results show that the way in which students confront new challenges in their first academic year (to learn study habits and specific learning strategies, to make new friends, to self-regulate own time and manage anxiety) determines not only the GPA but also subsequent achievements in their academic trajectory. According to previous studies, students perceive this transition as a stressful situation (Cantor et al. 1987; Parker et al. 2004). Parker et al. (2004) have found that adaptability factors, intrapersonal dimension, and management of stress were more relevant predictors of GPA than high school GPA. Adaptability factors are related to the use of flexible and realistic coping strategies. Other studies have found that coping mediates the relationship between emotion management and academic achievement via problem-focused coping as the main mechanism: students with higher emotion management capability are able to focus on the underlying problem across several academic stressors, resulting in higher levels of achievement (MacCann et al. 2011). In our study, non-adaptive coping strategies such as isolation, self-blame, and no action appear within the first 10 predictors for the classification between low- and high-GPA students. Scott et al. (2004) have also found that a self-blame coping style predicted poor academic achievement. In other words, students who internally attribute the failure may feel more helpless and, in turn, they decrease their effort resulting in a lower achievement (Scott et al., 2004). Beyond this impact on academic achievement, a dysfunctional management of different academic stressors becomes one of the main reasons for leaving university (Blanc, DeBuhr, & Martin, 1983). In fact, we found that strategies focusing on a concrete problem perceived as threatening are within the first 10 predictors providing information to the neural network for a very accurate classification of the student abandoning or not his/her studies.

It is important to consider the relevance of the early prediction of a student’s GPA because this outcome becomes in itself a very crucial predictor for the other educational outcomes, such as to persevere (or not) in the career. In fact, GPA at the beginning of the second academic year is one of the first 10 predictors for the classification between students who drop out and those that are retained, with a very low GPA associated with the drop-out group of students. In addition, the GPA attained early in a student’s university studies has a high predictive weight on the attainment (or not) of a degree within the period studied.

Specific learning strategies have been identified in this research as very important variables predicting GPA and degree completion. Five of the first 10 predictors of GPA were learning strategies related with strategies of information processing, test taking strategies, strategies of monitoring and comprehension, and use of supports to learn or retain information. Moreover, the selection of main ideas (learning strategy) and the student’s regulation of the anxiety contributed in the prediction of degree completion. These results highlight the importance of the development of interventions to improve the learning strategies of students at risk, which has already been mentioned in the instructional research and educational literature of the last 30 years (e.g., Pintrich and De Groot 1990; Weinstein et al. 2000). In addition, two learning strategies related to motivation and attitude appear as very important predictors for GPA and academic retention from the very beginning of the academic trajectory. This result is consistent with previous research which found that the lack of intrinsic interest or apathy (Beck and Davidson 2001; Covington 2000) is a crucial factor determining the failure to achieve academic success.

If we compare the set of predictors for the three educational outcomes, learning strategies have a greater contribution for the prediction of GPA, but sociodemographic background variables have more predictive weight for the identification of students who will drop out or not from the university programs. Coping strategies contribute more to the classification between degree vs no-degree completion. Consistently with previous research (Astin & Oseguera, 2012), these comparative results suggest that achieving a university degree is a complex route where students have to make decisions applying different strategies to solve problems, beyond academic performance.

The total of completed courses, number of family members, and number of books in the home are the most important background variables for academic retention. This result is consistent with previous research that have identified sociodemographic and academic variables as potential predictors of drop-out (Kotsiantis et al. 2004; Kovacic 2010). For example, Herzog (2006) used neural networks to predict degree completion time finding that credit hours, student age, residency, and stop-out time were found as the most valuable predictors. In our study, sociodemographic background variables were not the most important factors predicting degree completion. Kovacic (2010) also found similar difficulties suggesting that background information collected during the enrollment of students does not contain sufficient information to classify with high accuracy successful and non-successful students.

Cognitive processes, especially reaction times, play a crucial role for the prediction of levels of GPA and degree completion. This result is consistent with previous studies using ANN (Musso et al. 2012, 2013) and with a large body of literature about the importance of these processes across several domains (Engle 2002; Gsanger et al. 2002).

According to the results of this study, the number of friends at the university is among the first 10 variables with the most predictive weight, not only for the prediction of GPA but also for academic retention. On the other hand, according to our results, the “support perceived from friends and other” has more weight for the predictive classification of degree completion. This result is not surprising if we consider that social support operates as a useful resource for the student facing increased stress during the transition to college (Fisher and Hood 1987). Scott et al. (2004), using multiple regression analyses and logistic regression, have also found that the total level of social support was a very significant independent predictor of academic achievement.

A main limitation of this study involves the sample used. First, students were enrolled in a private university, coming from a medium/high socioeconomic status, so the results can be generalized only to private Argentinian university students. Second, students were enrolled in social science disciplines, so it is unclear how well the findings would generalize to other disciplines such as natural and physical sciences, engineering, medicine, etc. Another limitation of this study has to do with the self-report measures used to collect data about learning and coping strategies, perceived social support, and health data. Although these instruments have satisfactory psychometric properties as it was reported in the method section, future research should replicate a similar study with online measures of metacognitive regulation during the performance of a task. Similarly, it should be expanded to a broader range of disciplines and a wider range of socioeconomic backgrounds. Nevertheless, the study demonstrates the usefulness of the approach and the potential for positive interventions at the university level.

Educational implications

The relevance of several coping strategies found in this research suggests the need for interventions that help students cope with adversity during academic transitions. A recent review of experimental interventions in higher education has shown a series of framing interventions at a general level across courses or domains (Harackiewicz and Priniski 2018). Framing interventions aim to improve academic achievement providing these type of coping strategies (Harackiewicz and Priniski 2018; Stephens et al. 2015; Stephens et al. 2014; Yeager et al. 2016). In addition, several studies suggest that coping strategies can be modifiable through interventions that teach how to focus on a problem perceived as controllable by an individual (Compas et al. 2001; MacCann et al. 2011).

We found that self-concept and self-satisfaction are two important predictors according to their predictive weight in the neural network classifying students that will withdraw from the university and students that will persevere the following academic year. One set of interventions reviewed in the literature involves studies on value affirmation highlighting the recursive effects of these types of intervention (Harackiewicz and Priniski 2018). In other words, an improvement in the students’ self-affirmation leads to a change in confidence in their coping skill that causes a better GPA over a 2-year period (Brady et al. 2016).

The importance of several learning strategies found in this research suggests the need for the development of effective programs directed to improve students’ learning strategies following cognitive models in order to enhance the transfer of strategies to various educational situations. Weinstein and Meyer (1991) had already suggested that cognitive strategies must be goal-directed, intentionally invoked, and effortful. Students have to apply the cognitive strategies on an ample and real set of tasks (Weinstein et al. 2000).

Conclusions

In sum, it is possible to identify the predictors of positive educational outcomes, as well as those that are related to failure to achieve these outcomes, with the high level of precision provided by the machine-learning approach used in this research. Once we have this information for the population of students in an institution, we can develop targeted programs that specifically address the factors which are identified in students at risk. Given that these predictive classifications have been successfully carried out with data collected more than one year ahead of the educational outcomes studied, it provides an opportunity for positive and early interventions that will have a chance to change expected negative outcomes for more favorable ones. An early-warning program could be set up and ongoing analyses like the ones in this research could provide institutions with very valuable actionable information for interventions.