Integrating MOOCs in traditionally taught courses: achieving learning outcomes with blended learning

Abstract

It has been several years since Massive Open Online Courses (MOOC) have entered the higher education environment and many forms have emerged from this new way of acquiring knowledge. Teachers have been incorporating MOOCs with more or less success in a traditional classroom setting to support various learning preferences, introduce this new way of learning to students, and to make learning available to those who might not be able to follow traditional instructions. This paper researches a blended learning model where a MOOC has been integrated in a traditional classroom. A learning outcomes based approach was implemented, that supported a balanced student workload. Qualitative approach was used to analyse students’ learning diaries. Based on this research, benefits of integrating a MOOC with classroom based teaching were identified, as well as barriers that can hinder the successful implementation. Recommendations for teachers are provided.

Introduction

In today’s education, it would be challenging to find a teacher who has not heard of Massive Open Online Courses (MOOCs) and equally challenging to find one that has not yet or is not planning to implement a mode of blended learning environment to their classes. Several authors have confirmed that blended learning has a positive impact on teaching and learning (Garrison & Kanuka, 2004; Gilbert & Flores-Zambada, 2011; Morris, 2014; Sharpe, Benfield, Roberts, & Francis, 2006). After mapping trends over 35 years of publications in the journal Distance Education, MOOCs have been identified as one of the emerging themes between 2010 and 2014 (Zawacki-Richter & Naidu, 2016). Blended learning models in higher education do not have to be developed around MOOCs. However, there is a growing interest for exploring how MOOCs can enrich traditionally taught courses and act as a complementary resource in achieving teachers’ and students’ goals. In order to create a model that supports this, series of pedagogical and technical questions need to be addressed, along with exploring students’ experience with this mode of learning. Literature has confirmed that blended learning works, but is yet to demonstrate the best way to embed MOOCs to achieve learning outcomes, support various groups of learners, and overcome common students’ challenges.

Series of studies have been conducted to trial and evaluate the use of MOOCs in traditionally taught courses. Most of the studies covered enriching a classroom course with a MOOC (Bruff, Fisher, McEwen, & Smith, 2013; Firmin, Schiorring, Whitmer, Willett, Collins, et al., 2014; Ghadiri, Qayoumi, Junn, Hsu, & Sujitparapitaya, 2013; Griffiths, Mulhern, Spies, & Chingos, 2015; Holotescu, Grosseck, Crețu, & Naaji, 2014). Some researchers have worked on reviewing literature on embedding MOOCs in classroom (Bogdan, Bicen, & Holotescu, 2017; Israel, 2015). Although the aforementioned studies have explored embedding MOOCs in traditionally taught courses, as well as reviewed pass scores and student satisfaction, none of them were specifically based on learning outcomes and aligning MOOCs to them.

This study is important for several reasons. First, it builds upon previous research in creating a blended learning model that now relies on specific learning outcomes that are to be achieved, as well as on anticipated workload, providing guidance to teachers in creating their own blended environment. Secondly, it confirms that using MOOCs in blended learning supports part-time students in achieving their learning goals. Third, it explores students’ challenges and experiences with this program. Finally, it opens a research question on achieving a deep approach to learning among students in a blended learning environment, using MOOCs.

The purpose of this research is to understand student experience in a blended learning environment that is based on learning outcomes and workload, and to answer several research questions directed towards evaluating success of using MOOCs in blended learning. Exploring blended learning, specifically with a relatively open content such as a MOOC, is crucial for today’s position of higher education: “In the 21st century higher education faces the challenge of providing cost-effective, high quality learning experiences appropriate to the needs of an ever-increasing, culturally diverse student population and to meet the competency demands of a digital, knowledge driven society” (Torrisi-Steele & Drew, 2013). This is where a blended learning model with MOOCs could help face the challenges.

Literature review

Blended learning and massive open online courses

Blended learning has resulted in various similar definitions in literature as it emerged as an interesting research topic. It can be defined as the “use of technology with face to face teaching” (Torrisi-Steele & Drew, 2013) as well as a “thoughtful integration of classroom face-to-face learning experiences with online learning experiences” (Garrison & Kanuka, 2004). The same authors continue that complexity can rise from its design possibilities and the fact that blended learning can be applied to various contexts. Blended learning programs can differ based on goals, ratio of face to face and online learning, content providers, and methods of content delivery. Many blended learning programs in academic setting today are created around traditional courses that are enriched with online content and capabilities, leveraging the positive impact of blended learning on teaching and learning highlighted in various studies (Gilbert & Flores-Zambada, 2011; Morris, 2014; Sharpe et al., 2006).

The blended learning model covered in this paper has the goal to enhance learning processes and ultimately increase retention of acquired knowledge, as well as engage learners and fulfil their various learning needs and preferences. Morris (2014) states that “MOOCs are available to students to supplement their learning and personalized learning environments”, which is supported in this paper as well. Literature supports that the above goals are indeed possible to be achieved with MOOCs. For example, one study generated six benefits of incorporating MOOCs in traditionally taught courses: “replaying lectures, augmenting or replacing secondary materials, filling gaps in expertise, exposing students to other styles of teaching and class discussion, reinforcing key skills, and teaching students how to teach online” (Griffiths et al., 2015). Another benefit of embedding MOOCs is that it can help diminish downsides usually reported by researches, such as low completion rate, since high dropout rates in MOOCs can be alarming for traditional educators (Koller, Ng, Do, & Chen, 2013). There are also specific challenges that need to be kept in mind when evaluating embedding MOOCs in traditionally taught courses. For example, Israel (2015) emphasizes that integrating a course that is not designed to be a part of a blended learning program, holds its challenges, such as ensuring student engagement. Finally, computer literacy and technology acceptance are general challenges of online learning, as well as general teacher acceptance when it comes to new technologies.

As mentioned in the Introduction, previous research on blended learning programs with MOOCs mainly include research based on a particular experience with a project of that kind. To start, some authors have examined the use of MOOCs in campus-based courses and found no statistical difference in pass rate or final score, but the feedback concerning rating, interest, difficulty, and amount learned was better for traditionally taught classes (Griffiths et al., 2015). A pilot was conducted at San José State University (SJSU) with an edX course where a flipped classroom model included projects and quizzes. The program achieved “a high success rate with 90% of the students passing the final exam, as compared with 55% in the traditional class of the previous year” (Ghadiri et al., 2013) in (Yousef, Chatti, Schroeder, & Wosnitza, 2015). Another model was piloted at Vanderbilt University where a Coursera MOOC “Machine learning” was incorporated in a graduate level course on machine learning. Students’ experience was evaluated through a focus group and qualitative analysis. Feedback was overall positive, with students appreciating the possibility to learn at their own pace. Students did not participate in forums but have described them as useful, and have recognized the importance of self-discipline to stay on track with the MOOC (Bruff et al., 2013). Holotescu et al. (2014) developed a model in which “students’ participation in different MOOCs was integrated in a blended course ran on a social mobile LMS”. Goals were increasing knowledge on the topic and allowing students to be more familiar with the phenomenon of MOOCs. Finally, a case of blended learning with MOOCs was recorded at San José State University (SJSU), in which three college courses were enriched with Udacity content. Findings confirmed the importance of consistent student engagement to student success (Firmin et al., 2014).

There is an intuitive appeal to the concept of integrating the strengths of synchronous and asynchronous learning activities (Garrison & Kanuka, 2004), which is demonstrated in the current research and efforts of teachers and educational institutions worldwide in launching these programs. Bruff et al. (2013) have stated that “massive open online courses (MOOCs) present a new option for blended course design”, which is certainly confirmed with this research.

Deep, strategic, and surface approaches to learning

Terms of deep and surface approach to learning were introduced by Ference Marton and his research group, who were researching why there are differences in how students who read the same text understand it. They have found that the differences “hinged on the initial intention” (Entwistle & Peterson, 2004). After the initial research, many have followed and resulted in additional findings. Three fundamental approaches to learning have been developed: deep, surface, and strategic. Students who “seek to understand the issues and interact critically with the contents of particular teaching materials, relate ideas to previous knowledge and experience, and examine the logic of the arguments and relate the evidence presented to the conclusions” are the ones with the deep approach to learning (Beattie, Collins, & McInnes, 1997; Entwistle & Ramsden, 1982) in (Ak, 2008). The surface approach is characterized by memorization that is in isolation to other ideas and the strategic approach is related to a desire to achieve the highest grades and has an emphasis on student skills (Duff, Boyle, Dunleavy, & Ferguson, 2004). Fransson (1977) researched relation between the approaches to learning and learning motivation to find that intrinsic motivation, absence of threat, and absence of anxiety were associated with deep approach, while threat (extrinsic motivation), anxiety, and absence of intrinsic motivation were associated with the surface approach (Marton & Säljö, 2005). This is an indicator that, to support deep approach, educators should find a way to connect the teaching material to personal preferences of students. A relation between teachers’ approaches to teaching (student-focused and teacher-focused) and students’ approaches to learning has been established; authors have proven that when teachers describe their approach to teaching as teacher-focused, students are more likely to report that they adopt a surface approach to learning. Furthermore, when teachers report adopting the student-focused teaching, students report adopting the deep approach to learning (Trigwell, Prosser, & Waterhouse, 1999). A research gap exists when evaluating the approaches to teaching and learning in a blended learning environment. An example of a research in this area would be exploring approaches to teaching and learning when using a Virtual Learning Environment (VLE)s (Lameras, Levy, Paraskakis, & Webber, 2012). There is a significant research potential in this area; approaches to learning have been heavily researched in a traditional classroom setting and developing this phenomenon further in a blended learning environment is relevant in assessing its importance and effect it might have on teaching and learning. For example, a similar research has been conducted through online and blended communities of inquiry framework where the authors have found that students in both blended and online courses were able to reach high levels of cognitive presence and learning outcomes (Akyol & Garrison, 2011).

Learning outcomes

The blended learning model described in this paper relies heavily on learning outcomes. Learning outcomes approach is relevant in the European Higher Education Area (Bologna process) (Gil-Jaurena & Kucina Softic, 2016). Knowing that learning outcomes present statements of what the learner should know and understand at the end of a learning module, a blended learning model ought to take them into consideration, too. Some earlier research covered the important role of learning outcomes in the design of an effective blended learning environments (Mugenyi, Zhu, & Kagambe, 2017). Having learning outcomes is not enough; alignment must be in place to ensure their achievement. Term “constructive alignment” is coined by John Biggs (Biggs, 2003b) and it can be used as a tool for systematic assessment of learning outcomes. In order to guarantee the achievement of intended learning outcomes, they must be aligned with teaching and learning methods, assessment and student workload. Particularly important is the alignment of learning outcomes with assessments (Gil-Jaurena & Kucina Softic, 2016; Hamad, 2017; O’Neill, Birol, & Pollock, 2010), meaning that appropriate assessment method should be chosen to ensure that the learning outcomes are achieved. In a way, the learner would be “trapped” and it would be difficult for the learner to escape without learning what is intended to be learned (Biggs, 2003a). The constructive alignment approach is implemented in the model covered in this paper.

Methodology

In this section we explain our methodological approach and analyse research questions, context, methods, and results.

Research questions

Pilot research on the model we are presenting in this paper was published during the 9th EDEN Research Workshop: “Forging new pathways of research and innovation in open and distance learning”. In this paper, a full 3 year research is covered, accompanied by extended research questions. This paper also introduces and explores the concept of approaches to learning within this model. We researched the case study in order to answer the following research question:

• 1. Can the use of a MOOC help in giving students a positive learning experience in a virtual learning environment?

• 2. Can the use of a MOOC help part-time students in achieving particular learning outcomes?

• 3. How to align learning outcomes and student workload with the use of a MOOC in a specific course?

• 4. What are the main challenges for students in using a MOOC?

• 5. Does the use of a MOOC motivate students for deep approach to learning and further use of MOOCs?

Context

The course Discrete Mathematics with Graph Theory (DMGT) is taught in the first year of master level of study programs Information Systems and Software Engineering at University of Zagreb, Faculty of Organization and Informatics (FOI). It is taught as a blended learning course with both full-time and part-time students enrolled. The number of students varies between 80 and 130. The syllabus consists of two parts: first, different topics in discrete mathematics are covered and the second part is dedicated to graph theory and its application. Topics have sound foundation in mathematical theory but offer multitude of applications of the covered theory in computer science and business, e.g. problem solving exercises that are performed individually or in teams (Divjak, 2015). The DMGT course was enriched with a MOOC based on learning outcomes. The constructive alignment for the two learning outcomes (out of 7) for the course DMGT is presented in Table 1.

Table 1 Constructive alignment of two of the learning outcomes on the course DMGT. Adapted from (Bralić & Divjak, 2016)

A MOOC has been offered as an alternative activity to project work, credited towards the final grade in the course. Both activities are aligned to the same learning outcomes. There is a two-folded goal for the introduction of a MOOC in the course: to give students more online learning experience and to help part-time students, who are not able to fully participate in campus teaching, meet the + learning outcomes of the course. In this paper, use of a MOOC in the DMGT course is evaluated over three academic years: 2014/2015, 2015/2016, and 2016/2017. In the first academic year, students were asked to choose a course-related MOOC on the Coursera platform and the teacher needed to approve the choice. In the second and third academic year students were supposed to choose a course among those that were preselected and offered in the learning management system (LMS) Moodle to provide a more focused approach and increase quality of the program. This is also aligned with the aforementioned research on deep approach to learning being connected with intrinsic motivation; although several MOOCs were suggested, students were able to choose the MOOC that is most aligned with their personal interests.

Data collection and analysis

For the purpose of analysing the research model, qualitative approach has been chosen. As per Creswell (2014), qualitative approach is a good choice if “a concept or phenomenon needs to be explored and understood because little research has been done on it” and the same author quotes Morse (1991) who states that qualitative approach is useful when the topic is new in terms of exploring variables that should be examined, which is the case in this research. Creswell (2014) also lists possible types of data collection in qualitative research, one of which is working with qualitative documents, a type adopted in our research: while going through the MOOC, students were required to write a learning diary, supported by the Journal feature in Moodle. The purpose of the diary was to gather thoughts and elaborate on the experience of learning with the MOOC. Open ended questions were provided to students; they were required to write their diary entries based on those questions. In 2015/2016 the questions were fine tuned to gather more detailed information, and the requirement was to have the review in 800–1000 words, instead of in 400–800 like the year before. These new requirements were valid in 2016/2017 as well. Having open ended questions offer guidance and ensure loose uniformity in results, but allow flexibility to express personal experience and opinion. For the purpose of this paper, questions were grouped to make qualitative analysis easier, as demonstrated in the section Results as well. To students, questions were presented in a random order:

General questions on the MOOC, success, and weekly activities:

• (1) Which MOOC did you choose and what success did you achieve? Certificate/points upload.

• (2) Elaborate your weekly activities in the MOOC by explaining what you learned. Have you encountered this content earlier in your studies at FOI? (only in 2015/2016 and 2016/2017)

Connection between the MOOC and the DMGT course:

• (3) What did you learn? Please refer to the learning outcomes of the DMGT course.

• (4) How are the content and the methods covered in MOOC related to the DMGT course?

• (5) What changes could we introduce to the DMGT course based on your experience with the MOOC you took?

General experience with taking the MOOC:

• (6) How would you describe your experience with taking the MOOC?

Feedback on time required to complete the MOOC:

• (7) Estimate the time required to successfully complete the MOOC (personal opinion, not what is listed on the MOOC site)

Results

The sample in this research consists of graduate students enrolled in the DMGT course. The structure of this sample is shown in the table below(Table 2):

Table 2 Participants

The sample structure indicates several findings: after the first year when this method of teaching the DMGT course was introduced, there was a strong increase in the percentage of students who opted in for a MOOC. In the third year, that percentage has dropped again, and it is to explore whether the percentage of around 15% is to stay for future generations. The percentage of male students who chose a MOOC as their task varies significantly year over year, while the percentage of female students is between 12 and 18% in all 3 years. This could be related to the fact that overall the number of female students is smaller so the variabilities are less likely to be encountered and be significant, but it is an interesting conclusion coming out of the descriptive overview of the sample.

In all years, part-time and full-time students were enrolled in the course. Part-time students were particularly encouraged to take a MOOC since it is more difficult for them to participate in the team project work. Some trends are shown in the table below. It is interesting to see that part-time students are opting for a MOOC more and more, based on the data gathered over the years. In the current academic year, 44% of overall number of students who completed a MOOC have been part-time students (Table 3).

Table 3 Key metrics and comparison over years

Authors have been observing the final grades achieved by students. The average grade has been calculated by taking full points achieved by the students and dividing it by the maximum number of points (for full time students it was 100 points, and for part-time students it was 94 points). Average of all average grades has then been calculated for each academic year. In the year 2014/2015 a better average final grade was achieved by the students who completed the project work. The results changed in 2015/2016 when a better final grade was achieved by those students who chose a MOOC over the project work. The trend continued in 2016/2017, when, on average, a better final grade was again achieved by the students who chose a MOOC over project work. It is challenging to draw conclusions based on this data and the trends need to be tracked in the upcoming years to make sure definite conclusions are drawn from descriptive analysis. However, the numbers so far indicate that students who opt in for a MOOC do achieve higher final grade, particularly when having in mind the larger proportion of learners taking MOOC in years 2015/2016 and 2016/2017, i.e. more reliable sample. Finally, MOOC related tasks have been graded better in 2014/2015. This grade, given by a teacher, consists of two parts: 90% are based on the MOOC’s final grade received by students and 10% on their diary quality and a short presentation of the MOOC they watched to other students.

General questions on MOOC, achieved success, and weekly activities (questions 1–2)

As required, all students have uploaded a screenshot of their certificates of completion and shared the results they achieved. Similarly, a description of weekly activities was done, in more or less detail, by all students.

Connection between MOOC and DMGT course (questions 3–5)

Questions on the connection between a MOOC and the DMGT course, as well as courses taken throughout formal education have been answered in different ways. Students that took specific MOOCs were able to connect them with particular learning outcomes of DMGT, not necessarily the ones from Table 1:

• “I would link the Coursera course to a specific learning outcome of DMGT: to use mathematical literature from multiple sources, at least one tool for processing mathematical language, and an e-learning system, having specific characteristic of mathematics in mind.”

Some students found a loose connection between a MOOC content and the classroom taught content. Several have emphasized that a MOOC in fact covers practical implications of what is taught in the classroom taught course:

• “DMGT course and the MOOC are complementing each other very well. In DMGT course I received theoretical grounds and the MOOC helped be to understand the theory following practical examples”

• “In this MOOC I encountered topics that were mentioned in the DMGT course but the MOOC has covered them more in-depth”

General experience with using MOOCs (question 6)

Generally speaking, the experience of taking a MOOC has been positive and the students have developed interest for this type of course

• “As I was going through the course I selected, I have also browsed through the platform and located several other courses I plan to take at a certain point”

• “The entire experience (…) is very positive. This is the first time I have studied something this way, but it is definitely not the last one”

• “The experience with MOOC was great and I will continue to learn this way for my personal advancement”

In the first year of this program choosing a MOOC that fitted the DMGT course was students’ responsibility and the reactions were various. Some appreciated this approach: “A significant advantage was that we (students) were not limited by a certain topic, but only by an area that needs to be covered in a MOOC”, while some would have preferred to have a specific MOOC to take: “It would be good to have a specific course as a task, rather than being given the option to choose any course that fits DMGT. Coursera library is very extensive so it took some time to find the appropriate course”. Based on the feedback, practice changed in the second year and a list of offered MOOCs was shared in the LMS.

Self-paced learning was much appreciated. It also helped to have a structure in place to keep students on track:

• “I was able to plan my time dedicated to learning. The only element to have in mind was the quizzes deadline, where I had three attempts without time limit, which was more than fair”

• “The advantage of MOOCs is the possibility to access content anytime, when I was focused and motivated, and interested in that content. Thanks to this, I was able to master the content in a more efficient way – simple and fast”

• “I liked that I was able to listen to particular videos multiple times and stop them in case I wanted to write anything down. I was aware of the duration of videos in a particular week in advance so I was able to organize my learning for the week”

Students have had positive experience with the more frequent knowledge evaluation: “The more frequent knowledge evaluation is far more effective than having two exams per semester”.

One student shared that he “has encountered online courses previously but has never completed one either because of lack of time or being lazy.” He finally chose the MOOC over the project work to “make himself” complete an online course. It is important to highlight that an online course is not the same as a MOOC. Still, this is an important point to take into account when planning a blended learning project, where teachers can, with their structured approach, make a significant effect on students’ acceptance of online learning (and MOOCs) and the motivation behind it. One student commented that “the experience is positive, and if the time is well managed, it is possible to watch all the videos in time, as well as prepare for the weekly tests, without major issues or anxiety”. This is another example of students appreciating the self-paced learning model, creating a habit of learning, and leveraging the more frequent knowledge checks.

Since online learning can be challenging in terms of a continuous teacher and peer support system, it is important to have a well-structured program. As an example, a student wrote: “I was excited when I easily understood how to solve a problem through a practical example.” It is interesting to read a comment from one of the students saying that “unfortunately he does not feel that the knowledge gained in this MOOC will help with further professional development” which opens a question: although the point of this blended learning program is to supplement and enrich the learning process, how can we best ensure that the chosen MOOC does affect the intrinsic motivation? For example, in a research on motivation behind learning on MOOCs, the authors compared the motivation and the differences in motivators between students and professionals; it was discovered that students are motivated by a benefit to future career and professionals by building the skillset for current role (Milligan & Littlejohn, 2017). This is an interesting observation; the student in this case completed the course because it was a required task and has adopted the strategic approach to learning, which is different than the deep approach that is characterized by intrinsic motivation and connecting experiences.

Another student made a comment that “Although the lectures in the MOOC are very well designed, online lectures simply cannot be as interesting and motivating as face to face lectures”, which opens up another important question: How to motivate and work with students that might lean towards classroom learning more naturally? This is the main reason why blended learning is a good fit in academic setting, it does take the best out of two worlds and accommodates different learning needs, styles, and preferences.

Another student stated that she “used MOOCs earlier but due to university related commitments she was not able to complete the courses”. Embedding MOOCs in traditional face to face classes by assigning credit to its completion could partly solve drop out issue, mentioned above.

Finally, one of the most mentioned parts of this learning experience was the language. Learning in English was rather new for most students and the feedback was various:

• “I was sceptical because of the language barrier (…) and taking a MOOC was quite a challenge because it required combining English language and important content”

• “I liked this way of learning because, in addition to learning itself, I had a chance to practice my English skills and to think about this topic in English”

Not all language related feedback was positive:

• “As the MOOC was progressing, it took more time and effort to complete everything in the curriculum. The completion was additionally slowed down by English language”

• “I spent most of the time translating tasks to Croatian to understand what needs to be done”

Time required to successfully complete the MOOC (question 7)

To fit a MOOC in a classroom taught course properly, it is important to value the time spent to complete a MOOC. This aligns with ECTS points awarded for each traditionally taught course in formal education. As mentioned in Table 1, the goal was to cover MOOCs with estimated 40–50 h to completion, to fit approximately 30% of complete DMGT course ECTS load. The actual time spent on completing a MOOC occasionally differs from what is stated on MOOC providers’ websites. Still, most of the students have shared that the time it took them to complete the MOOC corresponds to what is stated on the MOOC homepage.

However, a common feedback was that the required time can prolong significantly depending on prior knowledge of the subject and consequently the speed of completing follow up tasks, as well as on efforts put into studying follow up literature:

• “In the beginning I was fast with solving problems (…) because I have encountered this content before (1h-2h/week). Later, it took me longer to solve the tasks and I needed to go through materials again (4h-5h/week)”

• “It took the same amount of hours as stated on Coursera site to complete the MOOC, but to rewind the videos and to fully understand the content, it took twice as much time as suggested”

The role of English as the language of all chosen MOOCs was also significant in the actual time required to complete the MOOC:

• “Some tasks were easy while some required significant effort to master mathematical cryptography terminology in English”

• “If a student understands English well, it is possible to follow the lectures at a higher playback speed”.

Challenging tasks are not merely a time-consuming activity; some students report that challenging tasks make the MOOC participation more interesting: “Tasks that trigger intensive thinking are the reason why I’m glad I chose MOOC”.

An interesting feedback in regards to the time spent on a MOOC was given by a student who stated that “5 hours per week is the optimal amount of time to dedicate to this type of learning, as it’s likely that individuals spend the same amount of time on activities that are not at all connected to university related tasks”.

Discussion

Answers to the research questions are based on a qualitative analysis and insights into the overall student performance.

• (1) Can the use of a MOOC help in giving students a positive learning experience in a virtual environment?

According to the feedback gathered in the learning diaries, MOOCs have supported the experience of learning in virtual environments, providing a new experience to majority of the students. To them, possibility to learn at their own pace was very important. Students have recognized the value of forums, discussions, and partnerships with others to achieve the best results. This that MOOC activity is indeed an alternative to team work, which is an essential part of one learning outcome (LO1), where students are expected to solve the real world problems individually and in collaboration. As an example, one student stated that he appreciated the professor answering a forum question himself. The student felt that this is a way of professor trying hard to make sure the students understand the content. As a result, the student recommends that forum participation is encouraged more in the DMGT course itself as it would have a positive influence on the learning experience. Research is also supportive of the fact that peer support is crucial in making learning successful; Brindley (2014) states that for students that join a MOOC, “peer support in the form of a buddy or learning group can be critical to deepening the learning experience”. Virtual knowledge assessments have also been accepted well by students; a general belief is that a regular knowledge assessment increases knowledge retention and reduces the stress related to adopting big amount of content.

• (2) Can the use of a MOOC help part-time students in achieving particular learning outcomes?

The feedback showed that the part-time students appreciate the opportunity to manage their learning. This opportunity was mentioned as a chance for improvement for traditionally taught classes as well. Part-time students have had success with completing the MOOC and the DMGT course which is a strong indicator that self-paced learning via MOOC is a good way to go to support their individual learning needs. The students were able to achieve particular learning outcomes and we believe that our model supports the previous findings on the effectiveness of blended learning in higher education: “blended learning is consistent with the values of traditional higher education institutions and has the proven potential to enhance both the effectiveness and efficiency of meaningful learning experiences,” (Garrison & Kanuka, 2004).

• (3) How to align learning outcomes and student workload with the use of a MOOC in specific course?

In order to introduce MOOCs into a traditional classroom, fine tuning of learning outcomes, assessment methods, and students’ workload is required. Special attention should be given to students’ workload having pre-knowledge and possible language barriers in mind; Actual students’ workload for non-native English speaking students is usually higher than what is listed on the official MOOC declaration. This means that a teacher should, in advance, carefully check all recommended MOOCs. This would help ensure that the workload is aligned with what the students should know before starting with a MOOC, but also that the language requirements are aligned with that is expected of a non-native English speaking student, in terms of time invested in the course..Interestingly, intended learning outcomes (Table 1) were not always recognized by students. The possible explanation is that students are not very interested in the pedagogical foundation of the course and the concept of learning outcomes. Students much more easily recognize abstract competences such as problem solving.

• (4) What are the main challenges for students in using MOOCs?

Language has been highlighted as a barrier and the main challenge for multiple students. Obviously, good command of English language can significantly contribute to easier completion of a MOOC. Still, the language was also characterized as a positive challenge. All students successfully finished the courses in spite of the potential language barrier. One student has noted that “after completing a lesson, she understood all terms, but she was unable to translate them directly and explain them in Croatian”. This comment is revealing in a way that students need to be prepared to think and gain knowledge in English. Further, students emphasized importance of previous knowledge (mathematics and programming) that enabled them to be successful in a MOOC despite of the declaration at the beginning of majority of the MOOCs that no specific pre-knowledge was required. Finally, students appreciate specific examples provided in the MOOCs, as well as more frequent knowledge evaluation. Students are also aware that self-motivation and completing tasks in time is required to successfully complete the MOOC, which requires thorough planning of MOOC related activities so that they fit in students’ schedule.

• (5) Does the use of a MOOC motivate students for deep approach to learning and further use of MOOCs?

The students were connecting ideas and topics to prior knowledge..and many of the students have connected the MOOC findings with the findings in the DMGT course and have thought critically about the material they have gone through, which are characteristics of a deep approach to learning However, extrinsic motivation in this blended learning model needs to be taken in consideration as a factor commonly related to the surface approach to learning, as well as the fact that some students have fulfilled their task by writing the diary and completing the MOOC without connecting anything what has been learned to prior knowledge or any known concepts. Also, the strategic approach with the intention to obtain good grades needs to be questioned further, as the MOOC completion was a part of the final grade on the subject; it is highly possible that this affected the approach to learning among students. Without the use of a formal questionnaire and further analysis, a definite response to this question does not exist. Still, aforementioned experience is important as it presents a foundation for further formal research in determining the approaches to learning students are adopting in a model like this.

Motivation to continue using MOOCs for further development is noted. For many of the students that participated in the program, this was the first time they tried learning with a MOOC and many of them stated that they will continue taking MOOCs. The next research step would be to evaluate whether the intention has transformed into actual further use. Finally, having a MOOC as a mandatory activity motivated the learners to complete it. Students struggled with completion when there was only intrinsic motivation related to the intention, which is a good indicator that external motivators are a powerful tool in making e-learning more successful.

Finally, part-time students had positive reactions to having a MOOC available instead of the project work.

Limitations and further research

The sample in this research is rather small. Having a larger sample would enable a deeper quantitative analysis which would fit the deep qualitative analysis and provide an objective analysis of correlations and influence, for example analysing ratios and results of full time and part-time students in detail. Further research should be done in the upcoming years of teaching this course to be able to detect more reliable results and trends. Further research could also include deeper text analysis and discovering key concepts and themes shared by the students. Finally, approaches to learning and teaching were not the primary focus of this research and further research with formal instruments is needed. However, a foundation is made to further introduce these concepts in the model of blended learning, which is one of the focus areas of authors.

Conclusion

Blending a MOOC in a traditionally taught course resulted in multiple findings and opened further research questions. Although the model described in this paper is similar to certain models and researches in blended learning, there are several key elements that make this model unique and successful.

Recommendations to teachers based on this research include:

• sourcing several interesting MOOCs for students and allowing them to choose one they are most interested in, which positively affects motivation,

• ECTS load should be carefully examined before suggesting and finalizing online portion of the content to ensure reasonable workload and expectations from students

• learning outcomes should be taken into considerations to properly connect online and offline learning and to create an environment that ensures achieving those outcomes

• if completion of a MOOC required, it tackles the problem of high drop-out rates in online learning, which could also motivate students and empower them to complete further MOOCs.

This research is a starting point for further research in blending MOOCs in traditionally taught courses, to detect trends, progress, and generate guidelines for a successful implementation of online content from strategic point of view, in a way that supports the deep approach to learning.