PhD prepared: research skill development across the undergraduate years

Evaluation of use of the Research Skill Development framework

The Research Skill Development (RSD) framework has been used and evaluated in a number of individual semester-length course contexts in disciplines as different as Biomedical Sciences (Munns and Chilton, 2014), Business (Willison et al., 2009a, 2009b), Engineering (Cochrane et al., 2009), English (Osborn, 2012), Nursing (Pretorius et al., 2013) and Physics (Menke, 2013), as well as interdisciplinary studies (Venning and Buisman-Pijlman, 2013). A multi-institution study showed that use of the RSD framework could effectively help individual educators and small teams to design semester-length courses that developed students’ discipline-specific research skills in many disciplines and year levels (Willison, 2012). However, the study also showed that benefits of explicit research skill development could be lost unless there was follow-on explicit development. This paper presents the first study showing the impact of explicit and coherent research skill development and assessment embedded in the curriculum over the time span of an undergraduate degree programme (Willison, 2012). Such studies are required to determine the long-term efficacy of upskilling and motivating undergraduate students toward PhD studies (Salter and Atkins, 2014). A model that embeds research skill development in the curriculum is of particular interest because it has the potential to be more efficient and more equitable and has a broader reach than mentored undergraduate research models; there is also evidence that some research skills, such as posing researchable questions, may be better developed in-curriculum than in mentored models (Willison, 2012).

The research presented here focused on the effectiveness of in-curriculum use of the RSD during the undergraduate degree in preparing students for PhD studies. Students were enrolled in a research-orientated, non-compulsory, fourth and final year of a Bachelor of Health Science, called the Honours year. Many students were exposed to the RSD during their undergraduate degree from the first year on. The Honours year embedded an extended version of the conceptual model, called the Researcher Skill Development (RSD7) framework (Willison and O’Regan, 2008/2015). Next, the RSD7 is described, followed by the research context, methodology, results, discussion and conclusions.

Conceptual framework: the Researcher Skill Development framework

The RSD7 framework is based on the RSD (Willison and O’Regan, 2006/2015), with the only difference being an extension of the original five-level continuum to seven levels of autonomy, and so it is called the RSD7 for short. The framework was extended to seven levels to bring in the unequivocally “capital R” research, so that the whole university community would be on the same continuum, from a first-year student to a high-profile professor.

An immediate benefit was the uptake of the RSD7 by academics involved in the Bachelor of Health Science to guide the assessment during the Honours years. The structurally similar five-level version of the framework, had not been taken up at Honours despite its use for six years in the Bachelor of Health Science with first-year students.

The RSD7 is a conceptual framework that is deliberately general in nature, so academics can adapt it to their context. It is a conceptual framework rather than, say, a rubric because it can be operationalised in many ways. The advantage of providing a more generic structure that can be adapted into more explicit rubrics was used in the VALUE rubric which was validated in a large range of degrees and institutions. This rubric was developed by the Association of American Colleges and Universities (Rhodes and Finley, 2013), and its generic nature helped with the uptake in a large range of settings. The VALUE rubric does not provide a range of descriptors for the levels of student autonomy; however, the RSD7 was salient for the research-oriented year of this study due to its articulation of researcher autonomy.

The RSD7 framework has been used in numerous ways, such as facilitating conversations between PhD students and their supervisors (Velautham and Picard, 2009); conversations with indigenous students about empowerment and knowledge; introducing students to necessary aspects of research processes; and for analysing teaching, learning and assessments elements in curricula. In this study, the prevalent mode of use was to scaffold the creation of marking rubrics.

This study evaluated the use of the RSD and the RSD7 to frame research skill development and assessment across a degree programme to prepare students for PhD study.

Facets of research

The RSD7 (Figure 1) describes the processes associated with research in either disciplinary or interdisciplinary contexts as “facets” of research. The RSD7 facets describe what researchers do. These facets have underlying commonalities across disciplines, as oppose to research products that may vary markedly in structure and purpose. The facets are not generic skills, as this would imply a ready transferability, but rather they are overarching perspectives about the research processes that are common across disciplines. In use, these general descriptions are made real by academics who operationalise them as discipline-specific and context-sensitive descriptors (www.rsd.edu.au). The six facets are that researchers:

1. Embark and clarify: Researchers embark on research, determining and clarifying the need for knowledge while taking note of ethical, cultural and social/team considerations.

2. Find and generate: Researchers find information and generate data relevant to the research using appropriate methodologies.

3. Evaluate and reflect: Researchers evaluate information and data and reflect on all processes used.

4. Organise and manage: Researchers organise information and data, and manage the research process of individuals and of teams.

5. Analyse and synthesise: Researchers analyse information and data, and synthesise new knowledge to produce coherent individual and team understandings.

6. Communicate and apply: Researchers write, speak and perform the processes, understandings and applications of the research, and respond to feedback, mindful of ethical, cultural and social/team issues (based on Willison and O’Regan, 2006/2015).

The separation of research processes into six distinct facets is, of course, artificial, as these facets frequently co-occur and overlap. Moreover, research is often characterised by a lack of clarity, fuzziness and even messiness, and therefore, the process is certainly not linear. It may be that the neater, more coherent and linear final phase is most commonly reported in journal articles, but this hides the actual processes used in research into the unknown. However, delineating facets associated with research makes the processes involved explicit, and for many students this equates to being learnable. Delineation is a first step towards demystifying research.

Seven levels of student autonomy

A second step in demystification afforded by the RSD7 is to clarify how much scope students have in initiating research projects and determining processes, and in negotiating final solutions and communications. The specific task or assignment sets the boundaries of autonomy. Making student and supervisor expectations of scope and autonomy clear may be a major factor in a successful mentor-mentee relationship, and one for which the RSD7 has provided effective guidance (Velautham and Picard, 2009). A common anecdotal complaint from students in the Bachelor of Health Science was that they were initially only given very limited amounts of autonomy in undergraduate courses, while they were expected to show independent research skills in Honours. This was also evident in one (Drew et al., 2002) but not in another (Allan, 2012) study of Honours students.

In the RSD7, the amount of scope is called “Extent of Researcher Autonomy” and is represented as a continuum delineated by seven levels. Considering the end of the continuum with minimal autonomy, Prescribed Research (Level 1 of Figure 1), the supervisor models discipline-specific approaches of how to commence, including question framing, hypothesis posing and/or goal setting. The student-researcher follows highly prescriptive guidance, such as tightly defined procedures or guided readings.

Bounded Research (Level 2) requires some limited student-researcher decisions, for example a choice between two given methods. The shift from Prescribed Research to Bounded Research is not trivial, as some beginning researchers struggle when decisions are necessary or grey areas are to be considered.

With Scaffolded Research (Level 3), the parameters of the research are determined by the supervisor, but student-researchers will need to make numerous decisions about what do and so work with a high degree of independence within the parameters set.

The move to Level 4 is into the zone of “researcher instigated” and so has corresponding difficulties: Self-initiated Research describes a move that is instigated by the researcher, with support and advice provided by the supervisor. Researchers may, for example, identify gaps in the literature as the launching pad, select appropriate methodologies and apply analytical techniques, and the supervisor ensures that directions are attainable within the parameters of resources and time.

In Open Research (Level 5), the research is initiated and propelled by the researcher in his or her own direction using self-chosen methodologies and audiences. At this level of autonomy, supervisors provide advice rather than prescriptions. Parameters still exist in terms of disciplinary conventions, and so Open Research does not allow for lack of rigour, as the degree of rigour to be applied is appropriate to the year-level of study and disciplinary norms.

The move to Level 6 is into the realm of discipline leading. The distinctive feature of the move to Adopted Research is where directions chosen and outcomes of research are used by others in the field, for example, by citing, adopting methodologies, being informed by research outcomes or following gaps the researcher has identified.

Enlarging Research (Level 7) is where the researcher changes the nature of the conversation in the discipline, e.g. through a reconceptualisation of what is worth researching or in the development of major research methodologies. This is research that reshapes or enlarges the parameters of the discipline, or conceivably consolidates and refocuses its mandate. Individual researchers may participate in research teams that make a broad contribution that is widely cited or used, even when they themselves have their directions prescribed, bounded or scaffolded by supervisors. In a team then, different personnel may be operating at different levels of autonomy, depending on their roles. Here, individual contributions combine into something bigger than the sum of the parts because the research-leader is operating at a higher level, moving into areas that the field follows.

The six facets of research form the vertical axis of the RSD7 framework, and the seven levels of autonomy comprise the horizontal axis. The resulting table cells are populated with details appropriate to these facets for each specified level. The elaboration of the original framework to include Levels 6 and 7 and so bring in the “discipline leading” aspects of research was introduced in 2008 (Willison and O’Regan, 2008). The RSD7 has been used early in PhD candidature (Velautham and Picard, 2009), for doctoral education more generally (Matas, 2012) and masters level (Willison et al., 2009a, 2009b; Venning and Buisman-Pijlman, 2013), but not previously been considered in an undergraduate research context.

Data collection and analysis

During the semi-structured interviews, additional probing took place when an answer seemed incomplete, in conflict with previous answers or worth seeking additional information. The 21 interviews in total yielded a range of benefits and disadvantages of the School’s use of the RSD7 in Honours, and when no additional information emerged, suggesting near-saturation of data, no further interviews were arranged.

The top-level extraction of data included all student or academic comments that addressed the research questions: benefits and disadvantages of RSD7-framed rubrics in Honours and recommendations for RSD7 use in years one to three of Medical Science degree programmes. The first author identified emerging themes which broadly conveyed student and academic sentiments of the benefits. The second author fine-tuned the initial phrasing of categories so that they reflected more accurately the data, and this took place in an iterative process until there was agreement on coding between both authors. Further commentary on the internal consistencies of themes in the emerging paper was provided by four independent peers, which led to clearer phrasing of each theme. A presentation of data and preliminary analysis was provided to a School meeting attended by 35 academic, professional staff and students, and attendees were invited to give feedback on the analysis of the findings, a process which provided confirmation of emerging themes.

Benefits of the RSD7 implementation in Honours year

There were two major themes concerning benefits of the School’s use of the RSD7-based criteria.

Disadvantages of RSD7 implementation

The major theme emerging as a disadvantage, stated by 8 of the 14 Honours students and 4 of the 9 academics, concerned problems associated with rubrics based on the RSD7, and their use.

The most common statement from students was about specific assessment criteria:

I kind of find that sometimes they [School-wide RSD7-framed rubric criteria] don’t apply very well to the assignment […] our projects are vastly different […]. I don’t understand how the same marking scheme can apply when it’s so specific for what it’s looking for […] one of the ones for the lit review was use of up-to-date relevant literature. It’s like, I don’t have any […] (Student 11).

“Up-to-date literature” was a criteria not perceived to be relevant, as the pertinent literature base on the topic was from 30 years previously according to the student. More worrisome for this student was that the assessors might not appreciate the context of her project, and so would not know the current state of play in the relevant literature. One criterion that two students perceived to be impossible for markers to make valid and informed judgements about “for outstanding grade” was:

Objectives clear, focussed and innovative, extending past supervisor guidelines (Medical science, 2012).

One student remarked about this criterion:

Some of it, you think, how will assessors know that we’re doing that? […] Assessors] don’t know what we and our supervisors talk about in our meetings, so they don’t know whether or not we’re going beyond our supervisors. I think that’s a bit stupid at times (Student 5).

From the perspective of the student, this was a criterion that required mind reading on the part of examiners. As another reason for the lack of applicability of the RSD7-generated criteria, one student stated:

I’ve found out that assessors are also quite subjective. I have one assessor that says I am in level 4 and 5; I have another one that says I’m in level 2 and 3 […]. So I’ve found that this framework is really subjective […] although they make markers a lot more accountable (Student 4).

This student is understandably troubled by the perennial problem of inter-rater reliability, and there is no surprise that mere criteria do not guarantee reliable scores, as this is frequently reported in the literature (Sadler, 2009, 2014). While the student was perplexed by the need to deal with multiple subjective perspectives at once, she appreciated that the markers would at least be able to defend their decisions for where they allocated marks, even if they did not agree with other markers.

Two students noted another problem:

We get a rubric for every single assessment, and they’re all completely different. They’re very much for each assessment, and I hate them (Student 11).

These comments suggest that some students were not aware of the common framing of the RSD four months into Honours, seeing each rubric as separate “for each assessment”.

Two academics noted an issue affecting applicability based around their use of the RSD-based rubrics, especially in oral presentations:

[…] what looks really clear on a piece of paper in an assessment task list becomes a lot more complex. Say you’re in a seminar and someone is talking about a particular area and trying to tease out a quite complicated presentation so that you can tick off numerous little boxes can be quite difficult (Academic 6).

For some academics, assessing with complex criteria is clearly a barrier, especially in oral presentation setting where details like small font size could make the process untenable. Another disadvantage of the use of the RSD-framed rubrics, even though identified by only one academic, may nevertheless be important:

There’s also some attempt to explicitly link assessment items with the Research Skill Development Framework, but there’s also the issue, you’re trying to micromanage how people mark […] (Academic 6).

This touches on tensions around academic autonomy versus a shared conceptualisation, and this tension will be revisited in the discussion.

Recommendations for undergraduate from first year to third year

Notwithstanding the above problems identified, all but one Honours students, and all the academics stated that the RSD7 could be used in beneficial ways to frame undergraduate study before Honours. One student was able to reflect on lessons learned during explicit development of his research skills in two consecutive first-year human biology courses, and clarified how they benefited his Honours study three years later:

It [the RSD rubric] tells you a lot, because I remember one of the criteria was to be specific […] in my [First Year] Human Biology course […] so I think it’s very relevant to Honours, because when we were doing our first seminar, some of the topics were really broad. Automatically you think: I have to be specific otherwise it’s too broad […] (Student 7).

This is a simple but profound demonstration of the development of research skills, where the student internalised the detail to be specific as well as the carriage of the idea throughout the years of study. Like other student quotes below, and as noted in Section 5.1, he indicated that cognition around research processes had started to become innate, where “automatically you think”. Indeed, a majority of students interviewed (n = 9) asserted that the RSD should be incorporated from first year of the undergraduate programme and continued throughout:

I think that would have been a good thing, especially if it was consistent throughout the years, because by the third year it would be a lot more ingrained and it wouldn’t be a novel thing that you’re looking at (Student 2).

These students perceived a deeper, more implicit internalisation of the research processes through the explicit RSD7 framing in multiple courses in the programme:

Since the beginning [of First Year], they have given us assignments based on this criteria. You might not have liked the assignments, but because they have been consistently applying this structure to all of our assignments, we have come to think that way for science, in the perspective of science and writing […] (Student 4).

For this student, one of the ten who encountered RSD-framed rubrics in the first year of the degree, this previous rich exposure had deep and pervasive influence on his thinking all the way through to Honours year. Here the idea is one of being inculcated into research processes so that these processes become internalised, provoking the sentiment we have come to think that way for science and enabling the process of students becoming metacognitive researchers. Students provided practical advice of how to incorporate the RSD effectively in the earlier years: “The idea is good; it just needs to be, for a better term, dumbed down to first-year level” (Student 11). This student, who was generally against the RSD7-framed rubrics in Honours year, here states the process to develop research skills should be developed in the years previous to Honours, with tasks set at an appropriate degree of rigour and sophistication. Another student considered the need for incremental development over the years:

I always remember these [RSD-framed] marking criteria sheets, but I think if they were emphasised more and you know that they kind of reinforce these facets a little more strongly throughout every year, it would be really, really good, I reckon (Student 6).

In a similar way, an academic phrased incremental development in this way:

But if it’s dribbled through in these undergraduate years, it then makes the Honours a lot easier, and then makes the PhD a lot easier, because a lot of students struggle in their first year [of PhD] (Academic 2).

This “dribbling” also gives a sense of what academics thought was the best ways of administering RSD7: earlier than Honours, but not “injected” forcefully and not over-emphasised. In other words, there was a desire that the RSD7 be used to provide structure and scaffolding, but not to the extent that it was inflicted in ways counter to effective learning or to academic autonomy.

Four Honours students agreed that RSD7 would effectively be used earlier in the degree than Honours, but not in the first year, with two students suggesting second year and two students suggesting third year. One of these student’s hesitation on recommending RSD7 in earlier years concerned not her own experience, for which the RSD7 had clear relevance, but that of her friends:

I loved what I did [in First year], but I just remember a lot of my friends used to sit there complaining, just going, this rubric sheet doesn’t make much sense because why am I ever going to need to do any of this stuff […] (Student 10).

Two students commented that the benefits of the RSD-framed rubrics from first year were negligible, with one saying:

I didn’t pay attention to it [RSD7-framed rubric in First Year], because it was kind of at the back of the assessment. They were never kind of made a big deal, whereas in Honours we kind of get handed this, like, go find this (Student 12).

For this student, it is not in having the rubrics but in how they are used, discussed and emphasised that is educationally pivotal. One out of the 14 Honours Students stated that the use of the RSD was a not a good idea in any of the years before Honours, saying that no a priori framework would be superior because that:

[…] sort of helps us to be a bit more flexible […] [and] sort of makes you think more, I think, because other people have different ideas and expect different things (Student 8).

This student perceived that a pre-determined structure may negatively affect flexible thinking.

While one academic was against an enforced use, eight of the nine academics saw an advantage of the RSD7 to frame an approach in common across all years of undergraduate study including Honours:

So there has to be what I’ll call a program wide approach where all the academic staff need to get together, or as many as possible, so you have the First-Year group there and Second Year and Third Year, and they all learn from one another (Academic 2).

A large majority of students and academics interviewed perceived that the marking criteria framed by the RSD7, located in appropriate courses and assessments across the degree programme, had reinforcing and multiplying effects on learning discipline-specific research skills.

RQ1

In terms of RQ1, there are clear benefits of the RSD7 for consistently informing learning, teaching and assessment processes throughout the Honours year. These benefits included that the RSD7 framework was perceived to fit closely to the nature of research in Medical Science, enable students to think metacognitively as researchers and help build the research capacity of students, academics and the School as a whole. These points are in agreement with a study of student inquiry that found “when appropriately framed (especially with the relevant student motivation and autonomy), such lessons can become deeply internalized” (Allchin et al., 2014, p. 469, italics added). Disadvantages of the RSD7-framed rubrics mainly focussed on the specific criteria, where some were perceived to be too prescriptive, too vague, subjective or irrelevant, in keeping with research on pre-determined assessment criteria (Adcroft, 2011; Sadler, 2009, 2014).

For both advantages and disadvantages communication was pivotal to the learning enterprise, with effective communication not guaranteed by the use of the RSD7-based marking rubrics, but more likely enabled by conversations framed by them. For example, one student in Section 5.2 perceived that the criterion about going beyond supervisor’s guidelines was “stupid”; however, Section 5.1 presented a student who also referred to this same criteria and found it provided substantial insight after discussing with his supervisor, saying that in reflection “I need to make it very clear […]” The first student said how will assessors know? The second student agreed that assessors wouldn’t know; therefore, I need to make it very clear in my writing. This demonstrates that written criteria alone are insufficient for at least some students to clarify assessment requirements.

Another element exposed above is that the RSD7 prompted academics to make visible criteria that are frequently left implicit for students. The general facet of “embarks and clarifies” was operationalised as the problematic, but necessary, description of “going beyond supervisor’s guidelines”, an honest exposure of assessment of expectations of high-performing students in the initiation of research. However, using the framework to unearth and articulate implicit criteria explicitly does not make the words or ideas immediately sensible or palatable, but preferably would be part of a process, whereby students critically analyse the criteria rather than reactively thinking “stupid”. Based on the differential response between students noted above, the RSD-framed criteria need to be used as a learning and conversation tool at least as much as an assessment tool, so that more students have an it-got-me-thinking reaction. It has been found elsewhere that the RSD framework frequently promotes a shift to making explicit that which has often been left implicit (Willison, 2012).

That some students saw several criteria as irrelevant, such as “recent literature” suggests that these students could better apprehend the standards of the discipline, and take pains to explain why their specific research does not fit the common practice. The criterion should suggest to the student – initially, with guidance from the supervisor, the need to explain why she is relying on what looks like out-of-date material. Overall, the effective use of the RSD7 and its resulting rubrics seemed to require dialogue between students and academics, in keeping with the academic who suggested introducing the rubrics with pizza. It seems that the marking rubrics framed by the RSD7 were like a “frozen conversation”, where the dense articulations of what is being sought needed to be defrosted by human-to-human contact to enable more fluid understandings.

RQ 2

RQ2 was “What do Honours students and academics recommend about RSD7 use in the entire undergraduate degree programme?” All those interviewed, except one Honours student, stated that RSD7 could be used to advantage earlier in the undergraduate degree than Honours. There was a strong emerging sense that across-the-years, explicit research skill development informed by the RSD7 is a way to enable the education needed before and during Honours, and to provide the preparation for PhD study. This may be contrasted with another study which found that without explicit development, there was a risk that research skills may atrophy from one year to the next (Willison, 2012). An explicit and ongoing upskilling process is more likely to overcome the problem that without appropriate structure and guidance, by the end of their degree, students tend to perform open-inquiry projects at “the same level of sophistication as in their introductory core course” (Chaplin, 2003, p. 231).

Repeated exposures of the same framing of the RSD7 maximise the potential for student metacognition, where they make their cognition visible to themselves and, as Student 4 said, because across the years of the degree “they have been consistently applying this structure to all of our assignments, we have come to think that way for science”. Students’ awareness of their own thinking processes are heightened through repeated and diverse exposures to the same six facets of the RSD in multiple contexts, and their potential for self-propelled learning is enhanced as shown by Student 1’s statement “you can improve yourself”. Metacognition was noted to be of paramount importance for fostering professional growth by a study on inquiry skills in science (Michalsky, 2012), and to student growth as researchers in this present study. Student metacognition evident in this study resulted in a dawning appreciation that the specifics of rubric criteria change time and again, but the six facets of the research process are the same, and the boosted metacognition fundamentally improved most interviewed students’ approaches to researching.

The above concurs with a study on students as apprentice researchers (Feldman et al., 2013, p. 240), which found that:

[…] there is need for professors to be more proactive in helping their students gain intellectual proficiency not just as part of doctoral studies but also for undergraduates and master’s degree students (italics added).

The opposite of this are separate courses where students have difficulties seeing the connections in the development of their research skills in different contexts. A lack of conceptual connection may lead to assessment and feedback that does not build systematically, but rather to courses and rubrics that have separate and distinct foci for different products. This may run contrary to reinforced learning across the years:

[…] given the growth of ever more detailed marking schemes for assessments, does feedback become something which is too specific to a single episode of assessment rather than generalisable to the learning experience as a whole (Adcroft, 2011).

A large majority of students interviewed were able to generalise to the learning experience as a whole, where they were metacognitive and considered themselves being prepared as researchers. This reported skill development is also in keeping with other studies of the RSD7 at the postgraduate level, (Venning and Buisman-Pijlman, 2013; Velautham and Picard, 2011) which, together with this study, suggest that the RSD7 could be used in master’s degrees as well as Honours degrees to better prepare students for PhD study. Any programme-level implementation of the RSD would need to balance advantages, such as coherency in the programme, with the dangers associated with over-riding individual academic’s autonomy.

Limitations and biases of the study

This study was conducted with a proportion of academics from one school and students from one honours cohort, so the study cannot be generalised to all the School’s academics and students, or to other schools and honours cohorts. The study risked a conformation bias; however, there is evidence especially in Section 5.2 that disconfirming evidence was gathered and presented, to mitigate this potential bias to some extent. The study was conducted at an Australian university, and this raises the question about generalisation to other settings and counties. The bachelor and Honours year are comparable to, for example, the UK system or the longer undergraduate science degrees where time is reserved for extended periods of research internships. It is noteworthy that a grading rubric based on the RSD7 was used successfully in The International Master of Science in Addiction Studies, which is taught by partner universities in the USA, the UK and Australia (Venning and Buisman-Pijlman, 2013). The consistent use in this current study of a common conceptual framework to inform rubric construction over multiple year levels to inform the students of their progress and to provide guidance on how to improve seems to be innovative. This use of a conceptual framework may require a cultural shift for academics who may otherwise be focussed on collecting the data from rubrics for accreditation purposes (Kuh et al., 2014) without seeing the broader educational issue.

Notwithstanding the above comments, this study is an improvement on studies that have focussed on RSD implementation by several academics (Peirce et al., 2012; Pretorius et al., 2013; Willison et al., 2009a; 2009b) or many academics working individually or in small teams (Willison, 2012): these studies of early adopters risk “early implementer” bias. This current study has considered a whole-school context, where not all those involved in implementation are convinced about the approach. This more closely mirrors the second phase of implementation of an innovation, which, in education, typically is less successful than first-phase pilot studies. Broader studies are needed to reveal more fully the advantages, disadvantages, limitations and successful methods of implementation of the RSD7 at the multiple school and institution levels.