1 Introduction

Digital libraries (DLs) have been a perennial topic of research and education in the fields of both Computer Science (CS) and Information and Library Science (ILS). The concept of digital libraries is quite broad, covering a variety of aspects pertaining to information theories, technology, and applications, making it difficult to determine the material that needs to be taught about digital libraries in higher education [1, 2]. This has been a primary concern for graduates in both CS and ILS, in particular. DLs present the opportunity for graduates to further apply new technologies to development of information systems or to provide information services to diverse and widely distributed populations who need access to digital information.

The DL Curriculum Development project was launched in 2006, responding to an urgent need for consensus on DL curriculum across the fields [3–5]; it can serve as an integrative and firm foundation for DL education in both ILS and CS. Top-ranked universities of DL education in ILS and CS, i.e., the School of Information and Library Science at the University of North Carolina at Chapel Hill (UNC-CH) and the Department of Computer Science at Virginia Tech, have been leading this project, with funding from the National Science Foundation. Many experts in the area of DL have contributed to developing and evaluating the DL curriculum. Basically, the goal of this project was to develop and validate a curriculum covering the area of DLs. This led to the development of educational materials for the DL curriculum that are applicable in graduate-level DL classes for both CS and ILS departments. The project was officially completed in 2009 but the DL community has continued to further develop the curriculum.

This project first proposed a framework for a DL curriculum, including 10 core and 51 sub-topics in DLs, based on a review of past DL course syllabi obtained from both the CS and the ILS departments in various institutes. Within the framework, 13 sub-topics were initially selected to be developed into “modules” that would support DL instructors. In this context, modules are focused lesson plans, including learning objectives, body of knowledge, learning activities, resources, and logistics to manage a class session. The 13 modules were developed collaboratively by researchers at UNC-CH and Virginia Tech, and the advisory board of the project with about 50 experts and instructors who teach about digital libraries around the world. They have been described elsewhere [6–8] and are available on the project website and in Wikiversity.

Once the module development was completed, DL instructors and their students in real classrooms used the modules and evaluated the effectiveness of the modules in enhancing students’ learning about DLs. The purpose of this paper is to present the findings from this field testing of the modules in classrooms. Through the methods of instructor interviews and student questionnaires, 15 instructors and their students in DL courses participated in the field testing and shared their experiences utilizing one or more of the thirteen modules. Findings indicate that they provided positive feedback on the modules’ effectiveness and gave constructive suggestions for improvement.

This paper is organized as follows. Section 2 provides some background on DL education. The project overview is described in Sect. 3. Our methods and study results are presented in Sects. 4 and 5, respectively, followed by further discussions in Sect. 6. Section 7 covers implications, and we present further enhancements and conclusions in Sect. 8.

2 DL education

Research in DLs has been thriving since the early 1990s. A significant amount of funding from the NSF, DARPA, and NASA has been invested to support DL research and practice, in particular a federal program called Digital Library Initiatives (DLI) [9]. Many DL research institutions and universities have received benefits from the funding and have contributed to developing advanced information systems and applications for DLs [10, 11]. A number of DL research projects and programs have been conducted not only in the US, but also internationally, for DL services, systems, development, management, implementation, and evaluation [12]. Among these topics, DL education is crucial because it is highly related to the success or failure of future DLs. Without having appropriate training and education, DL developers may risk building software that is seriously flawed due to not being fully aware of critical system requirements [13], efficient and effective techniques for implementation [14–18], or key ingredients of success [19]. A poorly designed DL can cause problems in usability and interoperability [20–24] and this could jeopardize its suitability [25] or long-term visibility with regard to digital preservation [26].

Some progress in assessing the status and emerging demand of DL education has been made since the late 1990s. In 1999, Spink and Cool’s survey of faculty in ILS and CS schools worldwide [27], found that about 20 institutions from eight countries (12 in the US and each one from Canada, New Zealand, UK, Malaysia, Singapore, Australia, and Brazil) offered DL courses. Most were at the graduate level, covering a variety of DL topics, such as theoretical and historical foundations, technical infrastructure, knowledge organization, collection development, information access and utilization, social, economic and policy issues, and professional issues. Two years later, Saracevic and Dalbello found that, among 56 American Library Association accredited ILS schools, 15 programs offered independent DL courses and 32 programs included DLs as a topic of other courses to some degree in the US and Canada, with a focus on information technology, integrating DLs with foundations, knowledge presentation, and archives of ILS programs [1].

In the 2000s, the need for developing a formal curriculum with integrative and interdisciplinary topics in the fields of ILS and CS for DL education emerged and was widely discussed [1, 28, 29]. A number of textbooks [30–36] and other references were published in order to provide appropriate guidance for DL education [37]. The number of schools offering DL courses worldwide increased to 42 in 2003 [38]. Topics pertinent to DLs have been identified and compared across DL courses from various ILS and CS schools [28, 38–40]. The DL courses in these schools were consistent in that they cover topics in theory and technology, including organization, resource descriptions, intellectual property, preservation, collaboration, management, and access. DL courses in CS programs have focused on metadata, databases, information retrieval, and DL software systems in particular. The course objectives and descriptions were similar in face-to-face and online offerings, but the level of interaction with students (for example, having experiences of digitizing or creating digital libraries) differed.

In spite of the increasing number of institutions that offer DL courses, most of the courses have been stand-alone without integrating any of the wide range of materials available in the field [41]. A variety of approaches to developing DL education have been discussed internationally [29, 42–47] but there was neither a formal curriculum of study, which could be widely applied to achieve balance between theories and practice in DLs, nor research on effective ways of evaluating a DL curriculum. Most of the previous studies were focused on investigating the trends and major topics covered in DL courses using surveys and content analysis of existing DL course syllabi or programs [38–41]. Liu compared aspects such as course outlines, textbooks, and assignments, across DL courses, but this study was still limited to analyzing the course content [38]. No studies have involved DL instructors and their students in DL curriculum evaluation, nor have they investigated how the DL courses have been taught and how effectively the course contents and practice have influenced students’ learning about DLs.

Therefore, a DL curriculum development project was launched in 2006 in order to develop, evaluate/validate, and disseminate curricular and educational materials that are useful for training university graduate students in both ILS and CS. An environmental scan of the DL courses from this project provided an overview of the topics covered in DL courses and led to development of the core topics of the proposed DL curriculum. Modules of these topics have been designed and developed based on 5S theory [48] and practice in DLs. Additionally, the effectiveness of the modules has been thoroughly reviewed and tested by instructors and students of DL courses in the field. This paper first describes the development of the DL curriculum, then reports and discusses the findings from the field tests of specific modules.

3 DL curriculum development project overview

A versatile digital library curriculum was developed through the three phases of the project: (1) DL curriculum framework development, (2) module development and review, and (3) module field testing.

3.1 Phase 1: DL curriculum framework development

The DL curriculum framework was developed, considering both theoretical and empirical approaches to DL education. Theoretically, the framework was developed based on the 5S framework of DLs, which designated the five important aspects of DLs: streams, structures, spaces, scenarios, and societies [48]. In addition, the Computing Curriculum 2001 (a joint effort of ACM and IEEE-CS) included a digital libraries module in Information Management, and it was carefully examined [49]. For the empirical development of the framework, two types of analyses were conducted. First, the research literature was examined in order to identify research topics of interest; all the papers presented at the ACM International Conference on Digital Libraries and the Joint Conference on Digital Libraries, as well as all the papers published in D-Lib Magazine were analyzed [53]. In a second study, 25 DL syllabi from ILS and CS taught between 2006 and 2007 were collected and analyzed [50, 51]. Weekly topics and readings to be covered in each course were examined. A total of 1777 titles for readings were identified which included books, book chapters, journals, journal articles, reports, or online sources. There was significant consensus on reading assignments, as well as the topics to be covered in the courses. These analyses led to identifying 10 core topics (overview, digital objects, collection development, information/knowledge organization, architecture, user behaviors/interactions, services, preservation, management and evaluation, and DL education and research) and the 51 sub-topics in the framework [8]. Details of the DL curriculum framework are available at http://curric.dlib.vt.edu/DLcurric_images/ModuleFramework2008-08-23.pdf.

3.2 Phase 2: module development & review

Each of the 51 sub-topics in the DL curriculum framework was a candidate for development as a module. Prior to module development, a template for the modules was designed to specify the minimal critical components of the modules. With input from the project’s advisory board, the final version of the module template was adopted (see Fig. 1).

Fig. 1
figure 1

Module template (also available at: http://curric.dlib.vt.edu/DLcurric/moduleTemplate.html/)

Full size image

Among the 51 sub-topics, 13 topics were carefully selected after considering the frequency of topics discovered through the analysis of the syllabi of DL courses. The modules were primarily developed by the project team members. Once a module was developed, external experts in the field of DLs were invited to conduct a preliminary evaluation of it. Experts were selected from the DL research and teaching community, especially among those who have particular expertise in the topic covered by a certain module. They inspected the assigned module carefully, considering various aspects, such as: (1) its coverage of the topic, (2) the currency and appropriateness of the readings, and (3) any assignments or exercises associated with the topic. Their comments were recorded and shared on the project’s wiki site (a closed one) [52]. All together, 32 different experts evaluated the 13 modules. The results from these reviews were incorporated in updates of the modules.

3.3 Phase 3: module field testing

Once modules had been reviewed by experts and revised accordingly, they were ready for field testing. The field testing was performed with two sources of evaluation in mind, DL instructors and their students. Instructors of DL courses in universities around the world were invited to implement one or more modules within the context of the existing DL course that they teach. After completing class sessions with the modules, instructors were interviewed individually about their perspectives on each module’s usefulness. Compared to the preliminary evaluation of the experts, this module field testing emphasized feedback on modules based on instructors’ direct experiences with implementation of the modules in their class environments.

Additionally, students taught by the instructors in DL courses were invited to complete questionnaires through which they could share their perceptions and experiences of learning through the activities of the modules. The primary challenge of the student questionnaire was to disambiguate their perceptions of the modules from their perceptions of their interactions with the instructors in classes. Therefore, the student evaluation questionnaire focused on students’ evaluations of the module content and their efforts to learn the knowledge and practices presented in the module. This paper reports on the findings from this field testing of the modules provided by instructors and their students.Footnote 1

4 Methods

4.1 Interviews with instructors

An email invitation to instructors who teach courses on DLs at the graduate level in any institute in CS and ILS was circulated in the JESSE and the ASIS&T SIG DL listservs in late 2007. Fifteen instructors responded, indicating their willingness to participate in the module field test. A total of 15 different modules were tested in 43 implementations. Among the 15 instructors, seven of them volunteered to participate in semi-structured interviews and to discuss their experiences using the modules. Prior to the module field testing, no specific guidelines were given to instructors about how to implement the modules in their class sessions, in order to respect individual instructors’ teaching styles and to test the flexibility of the modules to be used. Prior to the interviews, a copy of an informed consent information sheet was sent to participating instructors via email and they provided oral informed consent at the time of the interview. During the interviews, the questions focused on the ways in which the instructors implemented the modules in their class sessions, with questions about the module’s learning objectives, the body of knowledge covered in the module, the suggested readings, the suggested learning activities, the logistics of implementing the module, and the overall structure of the module. The interview guide is available on the project’s Wikiversity site (http://en.wikiversity.org/wiki/Digital_Libraries). The interviews averaged 36.7 min each, several of which covered use of multiple modules; interview length ranged from 13 to 88 min. Two of the interviews were conducted face-to-face, and the remaining five by phone/Skype. At the end of the interviews, several instructors provided their lecture notes (e.g., PowerPoint slides) or class resources to show how the modules had been implemented in their class sessions. These materials helped to specify which parts of the modules had been used in their class sessions. The interviews were audio-recorded and later transcribed for a descriptive analysis for the purpose of identifying important aspects of how the modules were used in class, as well as which aspects of each module were considered useful or could be improved, from the instructor’s perspective. Other information shared by instructors, such as their class environments (e.g., face-to-face, online) and the benefits and challenges of using the modules in class, were marked and reported as well.

4.2 Student questionnaires

Immediately after the completion of classes that utilized modules, students were invited to complete an online questionnaire. The instructors were asked to encourage their students to complete the online questionnaire, using a standard script provided by the researchers. Students’ participation was voluntary and instructors did not know which students participated in the study. An invitation email sent directly to the students included an informed consent sheet which described the study procedure and contained a link to a Web-based questionnaire, hosted by Qualtrics at UNC-CH. Students provided their implicit consent when they accessed the link to the questionnaire. After the invitation email, students received two additional emails as reminders a week apart. After completing the questionnaire, one student from each class was randomly selected and received a $5 Starbucks gift card, delivered via email.

Fig. 2
figure 2

Overview of field testing participation

Full size image

The evaluation questionnaire was designed to be completed in two minutes, evaluating the module contents and students’ efforts using the modules in their class sessions. The questionnaire items were drawn from several existing measures. Snare’s end of semester questionnaire was used to evaluate whether class lectures, learning activities, and assignments were appropriate [53]. Students’ satisfaction with the module style of learning, course topics, and important concepts was evaluated with suggested questions from McGorry’s quality evaluation tool of online programs [54]. Additionally, the evaluation handbook by Flashlight [55] and course evaluation questionnaires in nursing [56] were reviewed and used to develop the evaluation questionnaire used in the current study. A total of 17 statements were provided and the students were asked to rate their agreement with each statement, using a 5-point Likert scale (from strongly disagree \(=\) 1 to strongly agree \(=\) 5). Their prior knowledge of the module topic was investigated on a 4-point rating scale (from no prior knowledge \(=\) 1 to full knowledge of the topics covered in the module \(=\) 4).

In order to provide a more reliable/stable representation of the students’ views, each module implementation was treated as a distinct case during data analysis. This procedure was especially important because the class sizes and the number of students responding in each class varied dramatically (e.g., from 2 responses from a class of 10, to 25 responses from a class of 36). One implication of this procedure is that only modules for which multiple classes had hosted a field test are included in our current analysis. For each class, a mean score was calculated for each questionnaire item. The class means for all implementations of a particular module were then averaged to obtain an aggregated mean rating on each item for each module. It should be noted that, for some implementations, specific questionnaire items were not applicable to that implementation (e.g., some instructors did not incorporate any of the learning activities in their offering of the module, so items 5–7 were not applicable). These were treated as having missing data in the class means and the aggregated means.

5 Results

During Spring, Summer, and Fall 2008, as well as Spring 2009, 15 instructors from 14 universities participated in field testing 15 modules. A total of 749 students enrolled in DL classes taught by the 15 instructors were invited to participate in the student surveys. Approximately half of the enrolled students participated in the survey (376 students, 50.20 %).

While 15 different modules were evaluated in the field tests, only nine modules were tested in more than one class. This paper reports findings from the 37 module field tests of those nine modules, including responses from the 329 students who were exposed to those modules. The relationship between the full set of field tests and those presented in this paper is illustrated in Fig. 2.

Among the 37 module field tests, nine were conducted in CS or engineering departments (by two of the participating instructors) and the remaining 28 field tests were conducted in ILS schools or departments.

Of the 14 universities participating in the field tests, 12 were in the US; one of the others was in Europe and one was in Asia. Of the 12 US universities, nine are public and three are private universities. The 2010 Carnegie Classification of Institutions of Higher Education was used to describe additional characteristics of the 12 US participating universities. Eight of the nine public universities are research-oriented (classified as either “very high” or “high” research activities) and one of them is teaching-oriented (classified as Master’s College & Universities). One of the three private universities is research-oriented and two of them are teaching-oriented. Eight of the 12 universities are classified as “large” institutions, with equal to or greater than 7500 full-year unduplicated credit headcounts; three are “medium,” with 2500–7500; and one is “small,” with less than 2500.

Table 1 shows the number of classes and the number of students participating in the evaluation of the nine modules. Module 3-b: Digitization and module 4-b: Metadata were the most popular modules for instructors, which were field tested seven times each.

Table 1 Module field test participation overview
Full size table

5.1 Interviews with instructors

Seven out of 15 instructors participated in the interviews and shared their teaching experiences with eight different modules. They used the modules in face-to-face, online, or blended (having both face-to-face and online sessions) classes. In general, instructors spent about 3 h per class session (i.e., 3 h were devoted to teaching a single module).

Instructors commented that the modules were appropriate for covering important topics that they taught in their DL courses. They also mentioned that the modules were well structured and very detailed, including learning objectives, topics, and specific guidelines for possible exercises. Additional comments included that the modules could be used as a checklist against the instructors’ own materials, comparing topics, readings, exercises, and assignments. The modules helped instructors to think about the topics that they could assign for class discussion. Instructors added things listed in the modules to their class materials. An instructor who taught more than one module in her class affirmed that the modules were well articulated with one another (i.e., it was easy to move from one module to the next). These modules could be useful as a teaching guide or resource for instructors who are teaching the course for the first time, as well. One instructor provided hard copies of the modules to students after class for use during their study time, utilizing additional resources provided in the modules.

While the instructors commented on many positive aspects of using the modules, they also noted that the modules did not always perfectly fit with the scope of their class sessions. Sometimes instructors separated them into more than one session (mostly one and a half or two) due to the time limits they have within a class session. Or, they divided the topics in a module into multiple parts and taught them in different sessions. Table 2 shows a brief summary of how each of the six modules was used and the suggestions from the instructors for improving the module.

Table 2 A summary of instructor interviews on each module
Full size table

5.2 Student questionnaires

The nine modules shown in Table 1 were taught in 37 classes. A total of 329 students provided their feedback on these modules using a 5-point Likert scale (1 \(=\) strongly disagree and 5 \(=\) strongly agree). The mean ratings on each statement in the questionnaire were aggregated across classes and presented in Table 3. Grand means in the right-end column of Table 3 were calculated by averaging the individual class mean ratings across the nine modules. Grand means in the bottom row of the table were calculated by averaging the individual class mean ratings across the questionnaire items. The student responses will be discussed by first comparing the ratings across modules and across questionnaire items, then discussing the student perceptions of each module.

Table 3 Aggregated mean ratings (standard deviations) of the students’ questionnaire responses
Full size table

Overall, 9-c: DL evaluation and user studies received the highest ratings (\(M = 4.0\)), followed by 1-b: History of digital libraries, 3-b: Digitization and 6-b: Information seeking and search strategies (each \(M = 3.9\)). Individual class means for these modules ranged from 3.5 to 4.2. 5-b: Application software received the lowest ratings (each \(M = 3.1\)). Individual class means for this module ranged from 2.7 to 3.6.

It is also useful to look at students’ ratings on item 17, “Overall, considering its content, design, and structure, this module was effective.” On this item, 1-b: History of digital libraries and9-c: DL evaluation and user studies received the high average rating of 4.1. Individual classes rated these modules on this item as high as 4.4 and as low as 3.5. 6-d: Interaction design received the lowest mean rating on this item (\(M = 3.4\)). Individual classes rated this module on this item as high as 3.9 and as low as 3.1.

The mean ratings (across multiple offerings of each module) on each item ranged from 3.5 to 3.8. The mean across all items and all modules was 3.7, and responses on item 17, “Overall, considering its content, design, and structure, this module was effective,” averaged 3.8. Thus, we can conclude that students generally found the modules to be at least moderately effective. Student perspectives on each of the modules are discussed below; the content of all the modules is available at (http://curric.dlib.vt.edu/).

Class means for 1-b: History of DL and library automation ranged from 3.5 to 4.2 across all the questionnaire items, averaging 3.9. It was among the highest-rated modules, as noted above. The lowest ratings related to the assignments associated with the module and whether students would learn useful professional skills. The assignments involved focused in-class discussions and writing a case study of a project, so could likely be augmented with learning activities of a more applied nature.

Class means for 3-b: Digitization ranged from 3.6 to 4.1 across all the questionnaire items, averaging 3.9. Like module \(1-b\), it was among the highest-ranked modules. The lowest ratings were for item 14, “This module stimulated me to think critically about the subject matter” and item 16 “I was very satisfied with this learning module.” Students gave particularly high ratings for the items related to clearly outlined objectives and outcomes, module organization, readings, learning activities, and their ability to learn basic concepts from the module. The learning activity for this module was to build a digital image collection, and it’s likely that the hands-on nature of this activity was appreciated by the students.

Class means for 4-b: Metadata ranged from 3.6 to 4.2 across all the questionnaire items, averaging 3.8. As with module \(1-b\), students did not have confidence that the module helped them understand what would be expected of them as a professional. However, they were quite satisfied with the appropriateness and the intellectual demands embodied in the accompanying learning activities (in-class exercises that involved assigning metadata to physical objects). Instructors, however, commented that the learning activities were entertaining but used up more class time than they expected.

Class means for 5-b: Application software ranged from 2.8 to 3.6, averaging 3.1. It was the lowest-rated module among those involved in the field testing. The students did not find the assigned readings to be helpful. In addition, the student ratings indicated that improvements could be made by clarifying the module’s objectives and rethinking the amount of work required to complete the module. Instructors commented in their interviews that this module provides a good overview of technical aspects of software but it is too comprehensive, covering many different applications. This perspective is consistent with students’ perceptions that they had difficulty carrying out all the activities and learning all the important topics included in the module.

Class means for 6-a: Information needs/relevance ranged from 3.1 to 4.1 across all the questionnaire items, averaging 3.6. Responses to two items were the lowest: one related to the organization of the module and the other related to students’ beliefs that they gained a good understanding of relevant concepts from this module. The students did believe they learned to interrelate important issues by completing this module, and the overall effectiveness of the module was rated more highly than almost any single aspect of it.

Class means for 6-b: Online information seeking and search strategy ranged from 3.3 to 4.2 across all the questionnaire items, averaging 3.9. These ratings place it among the highest-ranked modules involved in the field test. The lowest-rated items pertained to the learning activities (not stimulating enough), the inability of the module to help students understand their future roles as professionals, and their ability to learn professional skills by completing the module. One of the instructors criticized this module saying that the learning activities need to be more practical; they included exercises in transaction log analysis, interviewing DL users, and conducting think-aloud protocols.

Class means for 6-d: Interaction design and usability assessment ranged from 3.1 to 4.1 across all questionnaire items, averaging 3.6. The student ratings were lowest in relation to their beliefs that the module helped them understand what would be expected of them as a professional; they also gave low ratings to items related to their satisfaction with the module, whether it served their needs well, and its overall effectiveness. In contrast, they did find the module’s learning activities (developing user personas, developing user tasks, and conducting a heuristic evaluation) to be stimulating. One of the instructors pointed out that it may help students’ learning if the module includes learning activities for practical and professional skill development, such as how to design interfaces with specific guidelines and how to use software to develop an effective design for a DL interface.

Class means for 9-c: DL evaluation, user studies ranged from 3.7 to 4.4 across all questionnaire items, averaging 4.0; this was the highest-rated module involved in the field test. While all the ratings were high, the items receiving the very highest ratings (4.2–4.4) indicated that the students found the learning activities to be appropriate and to require thinking and understanding and the module’s objectives to be clear. The instructors agreed that this module covers the basic and important topics in DL evaluation, though one suggested that content and learning activities related to DL evaluation criteria would be a useful addition to this module.

Class means for 9-e: Intellectual property ranged from 3.2 to 3.7 across all questionnaire items, averaging 3.4. It was one of the the lowest-rated modules among those involved in the field testing. The students did not find the learning activities to be stimulating, and did not believe that they significantly increased their knowledge of intellectual property issues by completing the module. The item related to the overall effectiveness of the module was rated higher than any specific aspect of the module.

6 Discussion

Field testing of the modules in this project provided a rare opportunity to implement the modules in real classrooms and to receive critical evaluations on a variety of aspects from the DL instructors who are experts and have many years of experience teaching the courses. In addition, students’ feedback provided information about how the modules influenced students’ perceptions of ways to learn about important topics related to DLs. Surveying students was useful for obtaining their overall understanding and experiences of the learning modules in classrooms. Interviews with instructors provided details about the classroom environments that we were unable to learn from the student questionnaires. Instructors’ experiences with and suggestions for the modules were very helpful for improving the modules in five areas, specifically: the learning objectives, the body of knowledge, readings, learning activities, and logistics for offering each module.

  • Learning objectives: Instructors agreed that most of the learning objectives assigned to each module were reasonable. They also made suggestions to tweak them in a few cases, covering not only theoretical but practical aspects of the modules as they apply to the context of DLs, in particular.

  • Body of knowledge: Instructors liked the format of lecture notes designed to list important topics and provide additional information with bullet points. They believed that the body of knowledge helped students to achieve the class objectives by discussing important issues and problems associated with the topics of each module. In several cases, however, the subjects covered in the body of knowledge were too comprehensive to be taught in one class session, so instructors had to reorganize the contents and often divided them into multiple sessions.

  • Readings: Instructors used the assigned readings in the modules selectively, based on what they think important or the number of readings they could reasonably cover in one session, since there sometimes were too many readings in one module. Instructors also provided readings that they used in class but that had not been included in the modules. They emphasized the importance of including recent publications in the readings so as to inform students about up-to-date technology.

  • Learning activities: Some instructors liked that there are learning activities in each module that can help students apply what they learned in class to practical cases. The detailed instructions for the activities were useful, but it seemed to take longer for the students to complete the activities than the assigned time allowed in the modules. There were other instructors who did not choose to use the activities in the modules since they already offered similar activities in the format of a course project, which was covered not in one session but throughout the semester; instructors used their own activities as well, which were different from those in the modules.

  • Logistics, e.g., feasibility, level of effort required in class, prerequisites, class time, etc.: In general, instructors agreed that the modules were desirable for use in their current DL courses, and that the level of effort required for students to prepare class readings and to learn and demonstrate knowledge gained in the modules was appropriate. A couple of instructors indicated that most of the prerequisite knowledge for the DL course had been covered in the foundation courses, such as Information Organization and Research Methods, in their programs. In terms of the class time assigned in the modules, instructors noted that it needs to be modified to reflect the situations in classroom environments; for example, one and half hours of class time was sometimes not enough to cover the material in one module.

Instructors thoroughly reviewed the body of knowledge and readings within the modules in particular and provided evaluative comments on the topics in a module, addressing whether the module covered the basic concepts and important features and functions related to DL building or DL evaluation. The emphasis on topics could be in line with the fact that most of the previous studies on DL curriculum evaluation/reviews examined the primary topics covered in the courses and compared them across DL courses [38–40] in that there were certain topics that DL instructors believe are so important that they are considered as fundamentals of digital librarianship. DL courses have been known for offering many creative assignments in classes, including DL building projects [38].

In addition to the comments on the five areas above, several instructors suggested that discussion questions be developed on which students can work in and out of class so they can study critical issues related to the modules and present what they learned from the modules in their own words, individually, in small groups, or in class. This could be important in online classrooms, in particular. One instructor indicated that he requires students to learn class materials by themselves in online but asynchronous class environments, and to post answers to discussion questions every week.

The modules were originally designed to satisfy the needs of interdisciplinary education for graduate-level DL courses in both ILS and CS. The application of modules in classes of CS could be different from those tested in classes in the ILS programs, depending on the core areas of knowledge on which each discipline focuses [29]. One instructor who has a CS background but taught DL courses in an ILS program commented that the current module contents could better serve ILS students than CS students and suggested customized assignments for CS students, such as providing opportunities for CS students to fully install, configure, and contribute to developing software in open source communities.

Each module in this project includes lecture-style notes to be used in a class session to teach an assigned topic. This approach was intended to enhance flexibility; instructors can choose any number of modules from the framework and implement them for their own class purposes. Therefore, modules are independent from one another, although we noted relationships with other modules in the module template to suggest the order in which the modules can be taught when an instructor chooses to use multiple modules. In real class environments, however, it may not be wise to utilize modules independently all the time. For example, most DL instructors assigned a semester-long group project to students, to build a small-scale DL. It could be important to consider the big picture in module development, looking for ways to connect class lectures and activities with the assigned semester-long class projects.

For effective module field testing, we used both instructor interviews and student questionnaires to collect data, but there were limitations to our findings. Fifteen instructors taught one or more modules in their DL courses but interview data were collected from only seven instructors due to conflicts in scheduling the interviews. Therefore, the interview data may not represent the opinions of all DL instructors who used the modules. Due to the limited data collection through the interviews, there were cases in which students’ responses on questionnaires were not fully interpretable.

7 Implications

Results from this module field test have both theoretical and practical implications in DL curriculum development and evaluation. The module field tests were effective in validating the contents and the practicality of the modules by obtaining constructive and critical feedback from instructors and students who had experience using the modules. The modules have been updated based on the findings from the module field tests. Furthermore, the subjective comments related to the overall aspect of the module design have been reflected in the development of new modules. Given the results of the formative evaluations reported here and the additional improvements to the curriculum modules, it is clear that this project has made a substantive contribution to DL education.

In addition, the procedures and methods used for the module field tests can be replicated in evaluating new educational materials in DL education and other areas of CS and ILS curricula. Interviews with the instructors were extremely useful in further development of the modules. Those curricula that offer multiple sections of the same course may want to conduct debriefing interviews or meetings with the participating instructors as part of their curriculum improvement efforts. In addition, the student questionnaires focused on the value of the modules to the students, rather than on more generic aspects of students’ reactions to this instruction. Such feedback from students could be gathered in other areas of the curriculum and used to improve modules or courses in those areas.

Overall, the DL project contributed to the research and practice of DL education in several ways. The DL project was designed and developed to have a broader impact on educating DL professionals—graduate-level students as well as current DL designers and administrators—through improved learning experiences. The DL module framework identifies core topics that need to be taught for training current and future DL professionals. The modules are readily useable in DL classes and have been shared with DL instructors and those who would like to study DLs independently through a variety of channels; all of the modules and resources are available from the project website for public use (http://curric.dlib.vt.edu/). Long term, those graduates and learners of DL courses will have a stronger background of DL development and management. The DLs they create will benefit their stakeholders and users by allowing them to access and use valuable data about their resources with critical information.

Moreover, during the project time period of three years, this project put effort into developing a DL community and to raising its visibility by providing a variety of outreach programs, such as DL tutorials and workshops for module reviews and development at international conferences on DLs (e.g., TPDL, ICADL, and JCDL).Thanks to the DL community, module development has been continued through the Wikiversity site (http://en.wikiversity.org/wiki/Curriculum_on_Digital_Libraries).

8 Further work and conclusion

This paper mainly reports on the findings from a field test of a number of DL modules, which was the final phase of the three-year NSF-funded project for DL curriculum building. The modules are the essential outputs of this project. We put a great deal of effort into designing and developing the modules in collaboration with a number of DL experts. Prior to distributing the modules to the DL community, we wanted to test the modules’ functions and capabilities of delivering the primary content of knowledge and practice in DL to students in classrooms. Thanks to cooperation and support from many DL instructors and students, we were able to test our modules in a variety of class environments. We obtained fruitful feedback from the participants for improving the current and future modules. In general, this project is meaningful not only in building a DL curriculum and developing modules as core resources for DL education, but also for initiating a collaborative project involving many experts in DLs and for developing a DL community as part of the goal for better education. The project would not have been successful without the continuous guidance from our advisory board members, DL experts’ constructive feedback on the modules, and instructors who volunteered to use the modules in their classrooms and participated in the field testing.

Although funding for this project ended, the module development, review, and field testing have been continued by the DL community. First, in the past few years, nine more modules have been developed and are available for use. Therefore, a total of 24 modules are currently available for use in DL courses. Second, the modules developed from this project have been used as resources not only for DL courses and the DL community, but also for the Information Retrieval (IR) community. Ten of the DL modules have topics that overlap with IR, and each module is about a particular IR software package, such as Hadoop Map-Reduce, Lemur, NLTK, R, Solr, Weka, and WordNet. These modules were developed by teams of graduate students studying IR at Virginia Tech. They were field tested and refined in two successive years of offerings of that class. We expect that the modules field tested during the project will also continue to evolve, based on the feedback received. Third, new modules also contribute to research in Big Data. Six modules for LucidWorks Big Data software (http://www.lucidworks.com/lucidworks-big-data/) were proposed. These modules were developed in 2012 in the above-mentioned IR class, by project teams, and field tested and refined by other student teams. LucidWorks generously supported these efforts, and students thus gained both IR and Big Data experience with an integrated suite of relevant software.

Furthermore, new modules focusing on particular software packages related to multimedia have been developed and field tested in two offerings of the Multimedia, Hypertext, and Information Access course at Virginia Tech. One package connects with the software used in a popular undergraduate course, originally developed at Georgia Tech, on Media Computation. Two others relate to music software, in particular Audacity (http://audacity.sourceforge.net/) and PureData (http://puredata.info/). The final module in the set discusses the fingerprint analysis software from the National Institute of Standards and Technology Biometric Image Software team. Therefore, a total of 56 modules have been developed, more than three times the number that were initially proposed; they are now available for public use from the Wikiversity site (http://en.wikiversity.org/wiki/Curriculum_on_Digital_Libraries). We hope the broader digital libraries, information retrieval, and multimedia communities will find these to be useful, and will help us refine and enhance them further, so they can be helpful for both teachers and learners across the information disciplines.