Research reportMethods of assessment used by osteopathic educational institutions
Introduction
Osteopathic teaching institutions are charged with the responsibility of using appropriate methods to assess their students by professional registration and accreditation bodies such as the Australian & New Zealand Osteopathic Council.1 Assessment of learning in osteopathic teaching institutions takes a number of forms, with the type of assessment often dictated by the learning outcomes to be assessed. Further discussion of some of the guiding principles of assessment in these teaching institutions is found in the recently released document Assessment 2020; Seven propositions for assessment reform in higher education.2 The present report provides a discussion of seven principles related to assessment for learning (Table 1).
Assessment is a well-researched area, particularly in medicine; however, there has been little research into assessment in the osteopathic profession. London3 describes the educational development of the osteopathic student as moving “…from knowledge acquisition to understanding and application…”. The methods of assessment within osteopathic programs change as the student progresses (i.e. written essay & multiple choice questions to objective structured clinical examination & long case assessment) with the methods of assessment having traditionally followed the model proposed by Miller4 (Fig. 1).
The ‘knows’ and ‘knows how’ of Miller's triangle (Fig. 1) are predominantly assessed in the early years of an osteopathic program with the ‘shows how’ and ‘does’ being assessed in the later years once the student has developed the “understanding and application”3 of the material and techniques presented throughout the program. There may often be overlap between levels on Miller's triangle (Fig. 1) with early year students often being assessed on their ability to perform basic osteopathic techniques on the ‘shows how’ level.4 This suggests that Miller's triangle may not be appropriate as a model for assessment in pre-registration osteopathy programs given students do not necessarily progress from the lower to upper levels of the triangle in sequential order.
Written examinations assessing the ‘knows’ and knows how’ levels of Millers triangle are typically set as multiple choice questions, short-answer questions, essays, or a combination of methods5 covering the clinical sciences and basic health and biomedical sciences. Clinical and practical examinations, such as the Objective Structured Clinical Examination (OSCE) and the long case examination, assess the students' ability to undertake a particular examination or technique (‘shows how’)5 or assess their ability to manage a patient or patients using their skill and knowledge set (‘does’). Towards the end of most osteopathic programs, it can reasonably be assumed that assessments are predominantly directed towards the clinical competency of the student and their readiness, or fitness, to practise.
Despite the limited research into assessment of competence in osteopathy, there is evidence of increasing interest in this area. Of note is the recent publication of a report that draws on the literature on assessment of clinical competence in medicine,6 and several other commentaries on this topic.3, 6, 7 The aim of the present study was to investigate the methods of assessment used by osteopathic teaching institutions through different stages of the program. The present study also investigated the issues surrounding these assessments and the ways in which these issues were managed. This project formed part of a larger study designed to inform the development of a model to assess overseas-trained osteopaths who wish to practise in Australia.
Section snippets
Study design
Semi-structured interviews and focus groups were employed to explore the assessment of clinical competence within osteopathy programs. An interview schedule and focus group schedule were developed (Table 2) based on the findings of a systematic search and critical review of the literature pertaining to assessment of health professionals,8 as well as preliminary documentary information collected from the institutions via their websites.
A semi-structured format was chosen so that information
Results
Eleven (n = 11) osteopathic educational institutions agreed to participate in the study. One (n = 1) Australian, one (n = 1) Italian, two (n = 2) Canadian and seven (n = 7) United Kingdom institutions participated. At each institution, between one and five senior representatives of the osteopathy academic staff were involved in the data collection phase of the study (Table 3). Representatives of the institutions were institution principals, course coordinators, senior lecturers and those
Assessing (Theme 1)
Educational institutions used a variety of assessment methods to assess graduating students for their preparedness to practise independently and to “cope in terms of competency, effectiveness and safety.” Three second-order themes were generated: Types of assessment (Theme 1.1), Risk assessment (Theme 1.2) and Other assessment areas (Theme 1.3) (Fig. 2).
Processes (Theme 2)
The assessment processes undertaken by institutions are governed by university and registration and/or accreditation body requirements. When considering the first-order theme of Processes 4 s-order themes were generated, namely Assessment procedures (Theme 2.1), Appeals (Theme 2.2), Review processes (Theme 2.3) and Reflection (Theme 2.4) (Fig. 3).
Special consideration (Theme 2.1.1) was afforded to students with special needs, including those with learning difficulties (e.g. extra time). One
Examining (Theme 3)
When considering the second order themes generated under Examining there was some general consensus of opinion on issues with regard to Marking (Theme 3.1) and Assessors (Theme 3.1.1) (Fig. 4).
Cost efficiency (Theme 4)
The cost of examinations was rarely raised as an issue when considering the efficacy of examinations. There seemed to be a belief that ‘logistics, cost, is a price we have to pay’. Fig. 5 presents the sub-themes generated under Cost efficiency.
Discussion
The current study explored how osteopathic institutions assessed their students. Using verbatim transcripts of interviews with pre-registration osteopathic teaching institutions, content analysis identified four interconnected themes: Assessing, Processes, Examining and Cost-efficiency, each with several lower order themes. The discussion of these themes will take place within the context of the Assessment 20202 propositions articulated previously (Table 1).
Conclusion
Osteopathic teaching institutions focussed on ongoing assessment over a period of years as the students developed their clinical competence, both in class and in clinics. The final examinations are but the culmination of this long process and the participants were therefore concerned about those who fall at this final hurdle. They were well aware of the need to reduce performance anxiety and of the steps that failed candidates could take to redeem themselves. Few revealed much knowledge of
Authorship statement
All authors were involved in the design of the study. VS and BV undertook the literature review. VS lead the focus groups and interviews. MW assisted with the focus groups and interviews. RG, CG, VS and BV analysed the data. GF and PMcL developed the discussion. All authors contributed to the compilation and review of the manuscript.
Statement of competing interests
Brett Vaughan is a member of the Editorial Committee of the Int J Osteopath Med but was not involved in review or editorial decisions regarding this manuscript.
Gary Fryer is a member of the Editorial Committee of the Int J Osteopath Med but was not involved in review or editorial decisions regarding this manuscript.
Acknowledgements
The authors would like to thank the participants in the focus groups for their time and contribution to the study.
The study was part of a grant provided by the Osteopaths Registration Board of Victoria (superseded by the Osteopathy Board of Australia at July 1, 2010) to develop alternative models of assessment of overseas-qualified osteopaths for their suitability to practice in Australia.
References (40)
The assessment of clinical practice in osteopathic education: Is there a need to define a gold standard?
Int J Osteopath Med
(2008)Assessing clinical competence in osteopathic education: analysis of outcomes of different assessment strategies at the British School of Osteopathy
Int J Osteopath Med
(2008)Clinical competence examination – improvement of validity and reliability
Int J Osteopath Med
(2008)Objective structured clinical examination (OSCE): review of literature and implications for nursing education
Nurse Educ Today
(2007)- et al.
Assessment of clinical competence
Lancet
(2001) - et al.
Assessment of osteopaths: developing a capability-based approach to reviewing readiness to practice
Int J Osteopath Med
(2011) - et al.
A universal global rating scale for the evaluation of technical skills in the operating room
Am J Surg
(2007) Accreditation policy: standards and procedures for the accreditation of osteopathic courses in Australia
(2010)- et al.
Assessment 2020: seven propositions for assessment reform in higher education
(2010) The assessment of clinical skills/competence/performance
Acad Med
(1990)
OSCE: the Dundee experience
Med Teach
Assessment of overseas-qualified osteopaths for the suitabilty to practice in Australia
Writing up qualitative research
Research methods in sport and exercise psychology: quantitative and qualitative issues
J Sports Sci
An expanded source book: qualitative data anlysis
Data collection methods: semi-structured interviews and focus groups
The ability of a long-case assessment in one discipline to predict students' performances on long-case assessments in other disciplines
Acad Med
Performance of students in the final examination in paediatrics: importance of the "short cases"
Arch Dis Child
The long case and its modifications: a literature review
Med Educ
The long case versus objective structured clinical examinations: the long case is a bit better, if time is equal
Br Med J
Cited by (15)
Pseudoscience: A skeleton in osteopathy's closet?
2024, International Journal of Osteopathic MedicineThe extent and quality of evidence for osteopathic education: A scoping review
2023, International Journal of Osteopathic MedicineHow does French initial osteopathic training value relational competency, patient education, and the competency-based approach? A cross-sectional survey
2021, International Journal of Osteopathic MedicineCitation Excerpt :Whether for clinical or in-course assessment, our findings indicate a poor development of assessment tools. This is in line with the findings of Vaughan et al., in 2012 regarding assessment methods in osteopathic institutes in other countries [29]. Based on the ILOs tied to relational competency as reported by the institutes, it can be argued that the institutes view patient education practices as a professional activity that is a part of relational competency.
Osteopathic clinical reasoning: An ethnographic study of perceptual diagnostic judgments, and metacognition
2018, International Journal of Osteopathic MedicineThe mini Clinical Evaluation Exercise (mini-CEX) in a pre-registration osteopathy program: Exploring aspects of its validity
2016, International Journal of Osteopathic MedicineAssessment of Australian osteopathic learners' clinical competence during workplace learning
2016, International Journal of Osteopathic Medicine