Research report
Methods of assessment used by osteopathic educational institutions

https://doi.org/10.1016/j.ijosm.2012.07.002Get rights and content

Abstract

Background

The methods used for assessment of students in osteopathic teaching institutions are not widely documented in the literature. A number of commentaries around clinical competency assessment have drawn on the health professional assessment literature, particularly in medicine.

Objective

To ascertain how osteopathic teaching institutions assess their students and to identify issues associated with the assessment process.

Design

A series of focus groups and interviews was undertaken with osteopathic teaching institutions.

Participants

Twenty-five participants across eleven osteopathic teaching institutions from the United Kingdom, Canada, Italy and Australia.

Results

Four themes were identified from the focus groups: Assessing; Processes; Examining; Cost Efficiency. Institutions utilised assessment types such as multiple choice questions and written papers in the early years of a program and progressed towards the long case assessment and Objective Structured Clinical Examination in the later stages of a program. Although examiner cost and training were common themes across all of the institutions, they were perceived to be necessary for developing and conducting assessments.

Conclusion

Most institutions relied on traditional assessment methods such as the long case assessment, however, there is increasing recognition of newer forms of assessment, such as the portfolio. The assessment methods employed were typically written assessments in the early years of a program, progressing to long case and Objective Structured Clinical Examination format assessments.

Introduction

Osteopathic teaching institutions are charged with the responsibility of using appropriate methods to assess their students by professional registration and accreditation bodies such as the Australian & New Zealand Osteopathic Council.1 Assessment of learning in osteopathic teaching institutions takes a number of forms, with the type of assessment often dictated by the learning outcomes to be assessed. Further discussion of some of the guiding principles of assessment in these teaching institutions is found in the recently released document Assessment 2020; Seven propositions for assessment reform in higher education.2 The present report provides a discussion of seven principles related to assessment for learning (Table 1).

Assessment is a well-researched area, particularly in medicine; however, there has been little research into assessment in the osteopathic profession. London3 describes the educational development of the osteopathic student as moving “…from knowledge acquisition to understanding and application…”. The methods of assessment within osteopathic programs change as the student progresses (i.e. written essay & multiple choice questions to objective structured clinical examination & long case assessment) with the methods of assessment having traditionally followed the model proposed by Miller4 (Fig. 1).

The ‘knows’ and ‘knows how’ of Miller's triangle (Fig. 1) are predominantly assessed in the early years of an osteopathic program with the ‘shows how’ and ‘does’ being assessed in the later years once the student has developed the “understanding and application”3 of the material and techniques presented throughout the program. There may often be overlap between levels on Miller's triangle (Fig. 1) with early year students often being assessed on their ability to perform basic osteopathic techniques on the ‘shows how’ level.4 This suggests that Miller's triangle may not be appropriate as a model for assessment in pre-registration osteopathy programs given students do not necessarily progress from the lower to upper levels of the triangle in sequential order.

Written examinations assessing the ‘knows’ and knows how’ levels of Millers triangle are typically set as multiple choice questions, short-answer questions, essays, or a combination of methods5 covering the clinical sciences and basic health and biomedical sciences. Clinical and practical examinations, such as the Objective Structured Clinical Examination (OSCE) and the long case examination, assess the students' ability to undertake a particular examination or technique (‘shows how’)5 or assess their ability to manage a patient or patients using their skill and knowledge set (‘does’). Towards the end of most osteopathic programs, it can reasonably be assumed that assessments are predominantly directed towards the clinical competency of the student and their readiness, or fitness, to practise.

Despite the limited research into assessment of competence in osteopathy, there is evidence of increasing interest in this area. Of note is the recent publication of a report that draws on the literature on assessment of clinical competence in medicine,6 and several other commentaries on this topic.3, 6, 7 The aim of the present study was to investigate the methods of assessment used by osteopathic teaching institutions through different stages of the program. The present study also investigated the issues surrounding these assessments and the ways in which these issues were managed. This project formed part of a larger study designed to inform the development of a model to assess overseas-trained osteopaths who wish to practise in Australia.

Section snippets

Study design

Semi-structured interviews and focus groups were employed to explore the assessment of clinical competence within osteopathy programs. An interview schedule and focus group schedule were developed (Table 2) based on the findings of a systematic search and critical review of the literature pertaining to assessment of health professionals,8 as well as preliminary documentary information collected from the institutions via their websites.

A semi-structured format was chosen so that information

Results

Eleven (n = 11) osteopathic educational institutions agreed to participate in the study. One (n = 1) Australian, one (n = 1) Italian, two (n = 2) Canadian and seven (n = 7) United Kingdom institutions participated. At each institution, between one and five senior representatives of the osteopathy academic staff were involved in the data collection phase of the study (Table 3). Representatives of the institutions were institution principals, course coordinators, senior lecturers and those

Assessing (Theme 1)

Educational institutions used a variety of assessment methods to assess graduating students for their preparedness to practise independently and to “cope in terms of competency, effectiveness and safety.” Three second-order themes were generated: Types of assessment (Theme 1.1), Risk assessment (Theme 1.2) and Other assessment areas (Theme 1.3) (Fig. 2).

Processes (Theme 2)

The assessment processes undertaken by institutions are governed by university and registration and/or accreditation body requirements. When considering the first-order theme of Processes 4 s-order themes were generated, namely Assessment procedures (Theme 2.1), Appeals (Theme 2.2), Review processes (Theme 2.3) and Reflection (Theme 2.4) (Fig. 3).

Special consideration (Theme 2.1.1) was afforded to students with special needs, including those with learning difficulties (e.g. extra time). One

Examining (Theme 3)

When considering the second order themes generated under Examining there was some general consensus of opinion on issues with regard to Marking (Theme 3.1) and Assessors (Theme 3.1.1) (Fig. 4).

Cost efficiency (Theme 4)

The cost of examinations was rarely raised as an issue when considering the efficacy of examinations. There seemed to be a belief that ‘logistics, cost, is a price we have to pay’. Fig. 5 presents the sub-themes generated under Cost efficiency.

Discussion

The current study explored how osteopathic institutions assessed their students. Using verbatim transcripts of interviews with pre-registration osteopathic teaching institutions, content analysis identified four interconnected themes: Assessing, Processes, Examining and Cost-efficiency, each with several lower order themes. The discussion of these themes will take place within the context of the Assessment 20202 propositions articulated previously (Table 1).

Conclusion

Osteopathic teaching institutions focussed on ongoing assessment over a period of years as the students developed their clinical competence, both in class and in clinics. The final examinations are but the culmination of this long process and the participants were therefore concerned about those who fall at this final hurdle. They were well aware of the need to reduce performance anxiety and of the steps that failed candidates could take to redeem themselves. Few revealed much knowledge of

Authorship statement

All authors were involved in the design of the study. VS and BV undertook the literature review. VS lead the focus groups and interviews. MW assisted with the focus groups and interviews. RG, CG, VS and BV analysed the data. GF and PMcL developed the discussion. All authors contributed to the compilation and review of the manuscript.

Statement of competing interests

Brett Vaughan is a member of the Editorial Committee of the Int J Osteopath Med but was not involved in review or editorial decisions regarding this manuscript.

Gary Fryer is a member of the Editorial Committee of the Int J Osteopath Med but was not involved in review or editorial decisions regarding this manuscript.

Acknowledgements

The authors would like to thank the participants in the focus groups for their time and contribution to the study.

The study was part of a grant provided by the Osteopaths Registration Board of Victoria (superseded by the Osteopathy Board of Australia at July 1, 2010) to develop alternative models of assessment of overseas-qualified osteopaths for their suitability to practice in Australia.

References (40)

  • M. Davis

    OSCE: the Dundee experience

    Med Teach

    (2003)
  • B. Vaughan et al.

    Assessment of overseas-qualified osteopaths for the suitabilty to practice in Australia

    (2010)
  • H.F. Wolcott

    Writing up qualitative research

    (2009)
  • S.J.H. Biddle et al.

    Research methods in sport and exercise psychology: quantitative and qualitative issues

    J Sports Sci

    (2001)
  • M.B. Miles et al.

    An expanded source book: qualitative data anlysis

    (1994)
  • M.C. Harrell et al.

    Data collection methods: semi-structured interviews and focus groups

    (2009)
  • L.G. Olson

    The ability of a long-case assessment in one discipline to predict students' performances on long-case assessments in other disciplines

    Acad Med

    (1999)
  • Z. Hijazi et al.

    Performance of students in the final examination in paediatrics: importance of the "short cases"

    Arch Dis Child

    (2002)
  • G.G. Ponnamperuma et al.

    The long case and its modifications: a literature review

    Med Educ

    (2009)
  • G.R. Norman

    The long case versus objective structured clinical examinations: the long case is a bit better, if time is equal

    Br Med J

    (2002)
  • Cited by (15)

    • Pseudoscience: A skeleton in osteopathy's closet?

      2024, International Journal of Osteopathic Medicine
    • How does French initial osteopathic training value relational competency, patient education, and the competency-based approach? A cross-sectional survey

      2021, International Journal of Osteopathic Medicine
      Citation Excerpt :

      Whether for clinical or in-course assessment, our findings indicate a poor development of assessment tools. This is in line with the findings of Vaughan et al., in 2012 regarding assessment methods in osteopathic institutes in other countries [29]. Based on the ILOs tied to relational competency as reported by the institutes, it can be argued that the institutes view patient education practices as a professional activity that is a part of relational competency.

    View all citing articles on Scopus
    View full text