Design and Validation of a Questionnaire for University Students’ Generic Competencies (COMGAU)

Introduction

Twenty-first century society is in the middle of a social and technological revolution which is producing new contexts that people must adapt to, contributing to the construction of processes that improve people’s overall quality of life. Sound training facilitates students’ transition into the world of work and lays the foundation for social harmony (Ambrosy, 2015), however, achieving that requires us to predict these new social, economic, and educational situations, and use methodologies that combine theory and practice to bring these contexts into the university environment. In the areas of economics and employment, Drittich (2016) called these transformations the Fourth Industrial Revolution. In this revolution, the knowledge society (with particular attention to education and creativity) has become especially important, and it has produced new dynamics between the educational system and the world of work, particularly due to the demand for academic training that is more suited to the needs of industry (Santos-Rego et al., 2018).

“In this way the mission of the university as a social institution emerges, as one of the most important elements of culture and society that must help in its transformation, maintaining quality and competitiveness and helping to respond to the general needs and problems in life that humanity is facing” (Rodríguez-Fernández et al., 2019, p. 7). The demands of modern society strengthen the need for students to have comprehensive training, they must take on responsibilities and develop key competencies for an ever-changing social and professional environment (Álvarez-Benítez and Asensio-Muñoz, 2020).

The Spanish response to the process of university convergence in Europe at the end of the 20th century was swift, with Spain adapting its university system to a new contextual framework. The new European Higher Education Area (EHEA) brought with it changes that Spanish universities took on board and which particularly had impacts on improving the teaching-learning process (encouraging initiatives so that students acquired competencies that met their aspirations and the needs of society), improving students’ employability (and their acquisition of appropriate competencies that allowed them more effective entry into the job market), the structural reform of undergraduate degrees, and the trend toward creating more inclusive systems due to the growing diversity of the European population at this time (Mella, 2020). The EHEA is therefore, according to Rodríguez-Fernández et al. (2019), an excellent opportunity to reflect on the model of university that we want to create.

Continuous training and competency-based learning are two of the mainstays of the new educational model at university (Gargallo, 2016). Universities have the fundamental role of producing graduates with appropriate skills and competencies to satisfy the needs of employers and the modern world (Pheko and Molefhe, 2017; Mtawa et al., 2019), producing rounded citizens (Archer and Chetty, 2013; Walker and Fongwa, 2017) aimed at the public good (Walker and McLean, 2013), creating opportunities for all and reinforcing inclusive, humanist values (Murris, 2016).

The focus on competencies arose because of the difficulties graduates had in entering and staying in the job market (Brice, 2018; Mtawa et al., 2019). From a pragmatic perspective, the conditions of work, globalization, and the knowledge society are factors that place a series of demands on universities explaining the importance of introducing competency-based teaching-learning approaches (González-Maura and González-Tirados, 2008; Mella, 2020). González and Wagenaar (2003), via their Tuning Project, distinguished between specific competencies (strictly academic in nature, associated with each area of study) and generic/transversal competencies (covered in all courses or areas of study). The latter are the product of an open university with close ties to external social agents (Mella, 2020).

Chell and Athayde (2011) confirmed that nowadays the main concern related to graduates moving to the world of work is to have given them the skills and competencies that the labor system needs, taking for granted the fact that specific (academic) competencies alone are not sufficient. Studies such as the Reflex Project (Allen and Van der Velden, 2007), the aforementioned Tuning Project (González and Wagenaar, 2003), and the Galician University Quality System Agency (Agencia para la Calidad del Sistema Universitario de Galicia [ACSUG], 2014) highlight the importance of generic skills and competencies and new requirements for graduates in the job market, emphasizing skills such as working in teams and communication (Archer and Davison, 2008).

These generic competencies must be flexible and adaptable (Allen and Van der Velden, 2011), multifunctional, multidimensional, and have a high level of mental complexity (Rychen and Salganik, 2003). Santos-Rego et al. (2018) summarized the different generic or transversal competencies and related them to what the labor market most often requested from university qualifications. They included instrumental competencies (which need the combination of manual skills and cognitive abilities needed for professional competence, including cognitive, methodological, technological, and linguistic skills), interpersonal competencies (personal and relationship skills such as working in teams, expression of social or ethical commitment), and systemic competencies (involving skills and abilities concerning systems as a whole, such as the capacity to learn, adaptability, creativity, and leadership skills).

Griesel and Parker (2009) and Hinchliffe and Jolly (2011) highlighted the consistency in meaning in the attributes of employability defined as generic competencies, which include diversity awareness, interpersonal skills, self-management, digital literacy, communication skills, critical thinking, and team working, among others. In addition Pheko and Molefhe (2017) identified 14 generic skills: literacy, communication, enthusiasm/commitment, arithmetic, timekeeping, personal appearance, working in a team, information technology in general, dealing with customers, problem solving, specific vocational skills for the job, advanced professional skills for the specific job, entrepreneurial awareness, and enterprising abilities.

As we can see, the restructuring demanded of universities is aimed at balancing the production of “useful machines” for the labor market and the education of citizens with values that contribute to the development of a fairer, more just society, and all of that with an educational approach that encourages the complete development of the individual. Recent graduates need to be agile enough to adapt to a rapidly changing work environment (Coffelt et al., 2016), they need the attention necessary to learn this new work, and the humility to realize that they have a lot to learn despite having a university degree (Beaton, 2017). The university should also seek solutions to reduce graduate unemployment by developing and encouraging generic competencies (Mella, 2020).

Service-Learning, Development of Generic Competencies, and Employability

Not crediting the minimum competencies valued and required in our society means great difficulties entering the job market and places people in our country in this situation at greater risk of social exclusion and vulnerability due to poverty (Sarasa and Sales, 2009; Boada et al., 2011; Gil-Flores, 2011). Service-learning is one of the educational approaches that best combines the social, ethical, and civic dimensions of professional profiles within the mix of professional competencies (Tejada, 2013). It is therefore an educational strategy with great potential to facilitate access to employment (Gallagher, 2007; Rodríguez-Izquierdo, 2018).

Sotelino et al. (2019) emphasized the notable impact of the service-learning methodology on participants and the community, with the greatest effects being in higher education (Chiva-Bartoll and Gil-Gómez, 2018). In addition, Santos-Rego et al. (2015) reiterated the suitability of service-learning for addressing the educational process in a comprehensive manner, from the acquisition of cognitive and academic skills to civic-social skills.

Brice (2018) and Mtawa et al. (2019) underlined the fact that, in addition to developing human skills, service-learning has great potential for improving graduates’ employability. It is also a useful teaching tool for producing competent graduates because it allows close connections between the academic and working worlds. Participating in service-learning projects increases students’ chances of finding and keeping a job (De Leon, 2014; Peterson et al., 2014; Hebert and Hauf, 2015). In addition to purely academic skills, service-learning develops students’ social awareness and active citizenship (Bringle and Clayton, 2012; Mayor-Paredes, 2019; Opazo et al., 2019) along with socio-relational skills such as empathy, managing groups, resilience, and solidarity (Whitley et al., 2017; An and Decker, 2019; Calvo et al., 2019). In terms of professional skills, Folgueiras et al. (2018) showed that students acquired numerous generic competencies through service-learning, particularly working in teams, ethical commitment, adaptability to new situations, and problem solving. Similarly Naval and Arbués (2016) confirmed the positive effect of service-learning on the development of generic competencies including professional communication skills, leadership and motivation, working in teams, project management, techniques for beginning work and professional development, negotiation skills, decision-making and problem-solving, dealing with customers, initiative, creativity, and change management.

The goal of this study is to obtain evidence to support maximizing the educational potential of service-learning in modern universities, which are tasked with an ever more demanding professionalizing function. In Spain there have been contributions from studies referring to the implementation of service-learning, such as Resch et al. (2020), but sufficiently thorough work using appropriate methodologies is scarce. For this reason we have proposed the construction and validation of a questionnaire to evaluate acquired generic competencies and university students’ potential employability from the practical educational approach offered by service-learning.

Specifically, our hypotheses guiding the study are as follows:

(a) The COMGAU (Questionnaire on University Students’ Generic Competencies based on the initials in the Spanish version) will demonstrate adequate factorial validity via a two-scale structure, a Generic Competencies scale and a scale of competencies directly related to employability.

(b) The scales will demonstrate adequate internal consistency, with Cronbach alpha values above 0.70.

(c) The factors from the questionnaire will stand out as predictors of acquired generic competencies and the level of university students’ potential employability.

Materials and Methods

Sample

Empirical analysis of the questionnaire’s reliability and validity was carried out using a convenience sample of 564 university students (67.2% women) aged between 18 and 59 years old, although most were between 18 and 22 (M = 20.96). The students were studying social and legal sciences (88.4%), engineering and architecture (9.6%), and lastly arts and humanities (2%). Data collection was carried out at the University of Santiago de Compostela (Spain), on the Lugo campus (40.4%) and the Santiago de Compostela campus (59.6%).

Instrument

The instrument we proposed was a questionnaire for evaluating acquired generic competencies and university students’ potential employability.

The final questionnaire was the result of the following phases:

(1) Literature review, performed by the research team and triangulated with external validation from five expert judges (national and international) in service-learning using a correction template. They were asked to assess the items in terms of validity criteria, location, intelligibility, and non-duality.

(2) A Pilot test with 50 students to validate the linguistic suitability of the items and their fit to the starting construct.

(3) Psychometric validation of the instrument, which is reported in this article.

The final questionnaire, called COMGAU (COMGAU based on the initials in the Spanish version) was made up of two scales:

(1) One generic competencies scale made up of 17 items (of the 33 formulated initially) grouped in five factors: leadership ability (five items), interpersonal skills (three items), intercultural ability (four items), collaborative online skills (three items), and analysis and summary skills (two items).

(2) A scale of items directly associated with employability (nine items).

Responses to the items were given on a Likert-type scale –as it is one of the most commonly used formats in social sciences, in addition to performing well (Blanco and Alvarado, 2005)— with five response alternatives, the optimum number as indicated by Matas (2018), where 1 is not at all and 5 is a lot.

Following the review and the analysis, prior to its psychometric assessment, the COMGAU was organized as follows: (1) the objective of the questionnaire, instructions, and thanks, (2) data on the person completing the questionnaire, (3) data on experience of service-learning, and (4) acknowledgments and contact details.

Procedure

The study was approved by the University of Santiago de Compostela Ethics Committee. Following that, the instrument was applied during the 2019/2020 academic year. It was distributed in the participating centers, accompanied by members of the research team who were present during the application. Prior to that a participation agreement was signed ensuring compliance with the ethical guidelines of the American Psychological Association [APA] (2009) with respect to consent, confidentiality, and anonymity of responses.

Students present at the time of the study participated voluntarily. They completed the questionnaire in approximately 15 min in a suitable, distraction-free environment. The study was performed via the application of a transversal survey design. This type of design allows the description of a population at a given point in time. In addition, it allows the establishment of relationships between variables and between the different segments making up the population (León and Montero, 2015).

To avoid falsified responses, we excluded any kind of question from the instrument that may have allowed subjects’ identities to be recognized and recorded an identifying code that respected respondents’ anonymity.

Data Analysis

To verify the psychometric properties of the questionnaire, we calculated the descriptive statistics of the sample for each of the scales, also analyzing the item properties. Following that, and also for each scale, we performed an Exploratory Factor Analysis (AFE) using IBM-SPSS.25, following the principal component extraction method with Varimax rotation. From that we obtained the reliability of each of the factors through Cronbach’s alpha, and finally, using structural equation models with the AMOS.20 program, we assessed model fit using the corresponding statistics (Byrne, 2001).

Results

Descriptive Statistics: Generic Competencies Scale

Table 1 gives the descriptive statistics and the item-total correlations which show: (1) acceptable values, (2) that symmetry and kurtosis indicate a sufficiently normal distribution, and (3) that the item-total correlations were all significant.

Exploratory Factor Analysis: Generic Competencies Scale

First we calculated the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy, Bartlett’s sphericity test was significant (p < 0.001), indicating the data was suitable. We performed the Kolmogorov–Smirnov test of normality, producing suitable values in all cases (p > 0.05). We performed an AFE by principal components and Varimax rotation, with the following descriptive statistics: KMO = 0.86; χ2₍₁₃₆₎ = 2490.867; p < 0.001. Table 2 shows the results, indicating a structure made up of five factors in which all of the items had factorial loadings above 0.40 (Mullan et al., 1997). We obtained eigenvalues above 1 and a total variance explained of 59.85%.

We used Cronbach’s alpha (Cronbach, 1951) to assess the questionnaire’s reliability which is considered the most appropriate for instruments whose final scores are found by additive processes or the accumulation of points (Nunnally, 1978). Analysis of the 17 items gave an alpha coefficient (Cronbach’s α) of 0.87 indicating high internal consistency, which did not increase on the elimination of some of the items. Table 3 shows the index of homogeneity and the alpha coefficient for each of the factors.

This first scale, the generic competencies scale, is made up of five factors. Factor I, composed of five items (E1_3, E1_8, E1_9, E1_22, and E1_26), Factor II with three items (E1_29, E1_30, and E1_31), Factor III with four items (E1_4, E1_5, E1_15, and E1_23), Factor IV with three items (E1_6, E1_11, and E1_32), and Factor V with two items (E1_1 and E1_2).

In terms of the internal consistency of the items making up Factor I, the α coefficient was 0.76 indicating good reliability. The values for the items making up Factors II and III was slightly lower (0.74 and 0.72 respectively), whereas the consistency for Factor IV was 0.57 and for Factor V it was 0.51 –which can be justified as the calculation of alpha is sensitive to the small number of items making up the factor–. Table 3 shows that none of the values of alpha increased if items were eliminated.

Confirmatory Factor Analysis: Generic Competencies Scale

In order to confirm the factorial structure produced by the exploratory analysis, we performed a confirmatory factor analysis, using the maximum likelihood estimation method together with the Bootstrapping procedure which ensured that the results were robust, and thus not affected by the lack of multivariate normality (Byrne, 2001). Considering that the value of χ² is very sensitive to small deviations of the hypothesized model when working with large samples (Jöreskog and Sörbom, 1993), we used the following indices of fit to assess the fit of the model: χ²/df (Chi-square/Degrees of Freedom), CFI (Comparative Fit Index), TLI (Tucker-Lewis Index), IFI (Incremental Fit Index), NFI (Normed Fit Index), GFI (Goodness of Fit Index), SRMR (Standardized Root Mean Residual), and RMSEA (Root Mean Square Error of Approximation). For χ²/df values below 5 are considered acceptable (Bentler, 1995), whereas Hu and Bentler (1999) consider values for GFI and the incremental indices (CFI, TLI, IFI, and NFI) above 0.90 to be acceptable, and values above 0.95 to be excellent. Finally, the model is considered to have a good fit if the SRMR is below 0.08 and the RMSEA is below 0.06 (Hu and Bentler, 1999).

As Figure 1 shows and given the indices in Table 4, we confirm that the proposed model is adequate, and is clearly consistent with the theoretical structure that guided the creation of the scale and the corresponding items.

Descriptive Statistics: Employability Scale

Following the analysis of reliability, this second scale was made up of the nine original items. Table 5 gives the descriptive statistics showing that the acceptable values, symmetry, and kurtosis indicate a sufficiently normal distribution and that all of the item-total correlations were significant.

Exploratory Factor Analysis: Employability Scale

We determined the dimensionality of the scale using AFE, following the method of principal component extraction and Varimax rotation, with the following descriptive statistics: KMO = 0.93; χ2₍₃₆₎ = 870.965; p < 0.001. The initial extraction produced a single significant factor which explained 54.83% of the variance (see Table 6).

Analysis of the internal consistency of the final version of the scale and all of its components required calculating Cronbach’s alpha coefficient. This analysis of the nine items gave an alpha coefficient (Cronbach’s α) of 0.895, indicating high internal consistency. Table 7 shows the index of homogeneity and the alpha coefficient for each factor making up the scale.

With a single factor, the employability scale was made up of nine items (E1_1, E1_2, E1_3, E1_4, E1_5, E1_6, E1_7, E1_8, and E1_9).

Confirmatory Factor Analysis: Employability Scale

Given the values cited for the first scale in this study and looking at the indices in Table 8, we confirm that the proposed model is adequate, and is clearly consistent with the theoretical structure that led the creation of the scale and its corresponding items (see Figure 2 and Table 8).

Discussion and Conclusion

Service-learning is not only a process that involves transformation of academic life, personal, social, and citizenship development, but is also one that involves employability (Matthews et al., 2015). In this study we focused on the creation and validation of a questionnaire aimed at assessing university students’ acquired generic competencies and their potential employability from practical service-learning education. We began from the conceptualization of professional competence as complex knowledge (Rychen and Salganik, 2003) linked to action, the socio-professional context, and experience, and how that is linked to service-learning.

As Tejada (2013) stated, the development of competencies means a close collaboration between what the individual brings to the work process and what the educational institution can provide for development of competencies (for example, time and space to reflect, the chance to take on appropriate levels of responsibility, etc.). Consequently, it is necessary to have criteria, evidence, and validated evaluation instruments that allow us to measure the acquisition of generic competencies that university students gain (Tobón et al., 2010) and their potential employability from the practical educational approach of service-learning (Brice, 2018; Rodríguez-Izquierdo, 2018; Folgueiras et al., 2018; Mtawa et al., 2019; Mella, 2020). We consider the factorial structure of the COMGAU to have a good theoretical fit.

Our results reinforce findings from previous studies such as Chiva-Bartoll and Gil-Gómez (2018); Rodríguez-Izquierdo (2018), and Sotelino et al. (2019), in which they concluded that service-learning notably facilitates the achievement of professional action skills in higher education institutions. This consistency means that we have a measuring instrument that can be used to check the acquisition of competencies.

To go into more detail, the proposed instrument has two scales: the generic competencies scale (17 items) and the employability scale (9 items), totaling 26 items. From the exploratory and confirmatory factor analysis, the solution is satisfactory both in the factorial structure of the scales and the high internal consistency, according to what various authors have stated in relation to the criteria to follow for confirming a questionnaire’s validity (Merenda, 2007; Jackson et al., 2009).

With respect to the first scale, generic competencies, it is made up of five factors: leadership skills, interpersonal skills, intercultural skills, collaborative online skills, and analysis and summarizing ability.

The items in the leadership skills factor describe the ability to plan, coordinate, and organize, the ability to solve problems and take decisions, leadership abilities (the ability to motivate others), and the ability to take the initiative.

Factor II is composed of items directly related to social interaction, which is why we have called it interpersonal skills. The items refer to the ability to present products, ideas, and reports in public, the ability to negotiate effectively, and written and oral communication skills.

Factor III refers to intercultural skills. This is basic general knowledge (general culture), the ability to write or speak in other languages, the ability to work in an international setting, and knowledge of cultures and customs in other countries.

Factor IV is related to collaborative online skills, it includes the use of ICT, working in teams, and the ability to work online with people and teams. These are all skills that make up this emerging paradigm, as noted by Villa and Poblete (2011) of online learning based on global interactivity, collaborative learning, and lifelong access to educational activities and resources (Harasim et al., 2000; Fisher, 2005; Guitert et al., 2007; Fainholc, 2008).

Finally, Factor V, called analytical and summarizing ability, as the name suggests, describes the ability to analyze and summarize.

All of the factors are consistent with the Tuning Project (González and Wagenaar, 2003), the Reflex project (Allen and Van der Velden, 2007), and the ACSUG project (Agencia para la Calidad del Sistema Universitario de Galicia [ACSUG], 2014). In addition, the factors refer to the availability of professional equipment and the use of necessary resources to perform a given activity, which is consistent with previous studies that have focused on generic competencies from this perspective (Santos-Rego et al., 2018). That means understanding the competencies examined in this study as a set of knowledge, procedures, and attitudes that are combined, integrated, and coordinated, as many previous studies have noted (Tejada, 1999, 2012; Echeverriìa, 2002; Naviìo, 2005; Rodríguez et al., 2010; Toboìn, 2007).

In addition, and owing to the fact that employers generally tend to be critical of the quality of recent graduates, blaming universities for the lack of training for the real world (Alonso et al., 2009; Brice, 2018; Santos-Rego et al., 2018; Mtawa et al., 2019), we present a scale which is directly associated with employability, with a single factor. In this way we refer to the connection between the content of university courses and the demands of the market, to personal development, and the preparation for a future occupation, the relationship between the academic and the professional trajectories, and also to the existing links between academic reality and the reality of work. Thus, through service-learning this gap between graduates’ skills and the demands of a competitive and changing world can be shrunk (Salgado et al., 2012; Brice, 2018; Rodríguez-Izquierdo, 2018; Santos-Rego et al., 2018; Folgueiras et al., 2018; Mtawa et al., 2019; Mella, 2020).

All of the factors in the two scales consider the analytical elements, indicated by Rychen and Salganik (2003), needed to define generic competencies: (a) they are transversal in different social fields; (b) they refer to a higher order of mental complexity, in other words they encourage the development of higher-order intellectual thinking such as critical and analytical thinking, reflection, and mental autonomy; (c) they are multifunctional, they are needed in a broad, diverse field of everyday, professional, and social situations, they are needed to achieve various goals and to resolve multiple problems in diverse contexts; and (d) they are multidimensional, as they address perceptive, normative, cooperative, and conceptual dimensions (Villarroel and Bruna, 2014).

In this way, our study maintains the connection with the objective of preparing citizens for the new challenges of the 21st century (Dearing, 1997; Bricall, 2000) along with the Tuning Project (González and Wagenaar, 2003), and the DeSeCo model (Definition and Selection of Competences) (Organization for Economic Cooperation and Development, 2001). In addition, it fills in a gap about how to incorporate, teach, and assess these competencies given that none of these systems are aimed at that (Villarroel and Bruna, 2014).

However, the results must be considered in the light of certain limitations. One is that the study only had data available from self-reports, which can mean biases due to possible intentionality in the responses (social desirability or magnification of difficulties for example).

There are also educational implications. In this study we have created and validated a scale allowing the assessment of training in professional competencies in higher education and the level of potential employability, via the service-learning educational approach.

Despite there being sufficient research evidence about the benefits of this learning methodology, one of the main criticisms made of it is the lack of specific instruments aimed at evaluating generic competencies acquired and the level of potential employability (Santos-Rego et al., 2018). For this reason, our study is a notable contribution to the body of knowledge, from a multidimensional perspective, which allows evidence to be obtained that may support maximizing the educational potential of service-learning as a methodological strategy in modern universities (Walker and McLean, 2013; Pheko and Molefhe, 2017; Walker and Fongwa, 2017; Mtawa et al., 2019).

Among these implications, the results of the COMGAU demonstrate its particular use as a tool for measuring the impact of service-learning on the self-perception of certain generic competencies commonly deemed necessary for quality professional development, and therefore as an instrument for studying the relationship between service-learning and self-perceived competency development. It is therefore not only useful for teachers but also for the students themselves.

Given that the validation of a questionnaire is a continuous process, our results are a preliminary step and future research should test the validity and reliability of the scale with samples in different cultures, as well as doing other types of analysis to confirm the suitability of the instrument, such as analyzing its temporal stability (Sánchez-Oliva et al., 2013). Not only would this improve the quality of the questionnaire’s reliability estimators, it would also allow its validity to be more thoroughly examined in culturally distinct contexts. So, with further expansion of the tool and standardization it can be useful for intercultural groups also. Another limitation is the sample, it is necessary that in future studies it is extended to other regions or countries and areas of knowledge.

In conclusion, our results support the hypothesis we posed, highlighting the COMGAU (COMGAU based on the initials in the Spanish version) as an instrument with evidence of validity and reliability in the analysis of generic competencies acquired via the service-learning educational approach and the potential employability of university students.

Data Availability Statement

The raw data supporting the conclusions of this article can be made available by the authors, with reservation.

Ethics Statement

The studies involving human participants were reviewed and approved by the University of Santiago de Compostela Ethics Committee. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

BR and JR-F contributed to the conceptualization, investigation, methodology, writing, and supervision of this study. JC contributed to conception and design of the study. MP-J organized the database. All authors contributed to manuscript revision, read, and approved the submitted version.

Funding

This work is derived from a project of the Spanish State Program of Research, Development and Innovation oriented toward the Challenges of Society (EDU2017-82629-R) and a Research Network of the Spanish State Program for Knowledge Generation and Scientific and Technological Strengthening of the R&D&I System (EDU2017-90651-REDT) – http://www.usc.es/apsuni/.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References