BACKGROUND

As part of their training, US medical students are required to complete clinical clerkship rotations in several core specialty areas. The structure and organization of these clerkship experiences have varied over time and across institutions, reflecting differences in patient populations, medical technologies, competencies to be mastered, and the provision of patient care. Indeed, over a decade ago the Association of American Medical Colleges described the evolution of and variability in clerkship experiences across schools and offered recommendations for overarching standards for clinical curricula within undergraduate medical education.1

Today, US medical schools typically require rotations in family medicine, internal medicine, obstetrics and gynecology (ob/gyn), pediatrics, psychiatry, and surgery. These clerkships may occur in hospital settings, ambulatory settings, or both. Due to large class sizes, many schools use multiple diverse sites for both hospital-based and ambulatory training. In addition, rotation length commonly varies for different clerkships within a school, and the length of rotations for a given clerkship varies across schools. The sequence in which students complete clerkships also differs within and across schools. Most schools utilize a ‘block’ system in which each block includes one or more clerkships, and students complete one block before moving on to the next. Within a school, students do not complete blocks in the identical order; usually all blocks occur in parallel throughout the academic year. As a consequence, students may complete core clerkships in many different sequences, which may, in turn, differentially impact students’ achievement of important learning outcomes.

The National Board of Medical Examiners (NBME) subject examination program provides assessment tools for measuring students’ cognitive learning outcomes in the clinical sciences. This program includes discipline-specific, multiple-choice examinations used by many medical schools to assess student performance at the end of clerkships. Assembled according to a content outline developed in consultation with the Clerkship Directors in Internal Medicine, the NBME subject examination in internal medicine is widely used by clerkships around the country, with student performances commonly included in determination of clerkship grades.2 , 3

Previous research has examined the effect of clerkship experiences on end-of-clerkship assessments such as the NBME subject examinations. In general, this research has focused on (1) clerkship length, (2) the time during the academic year when students complete their clerkships, and (3) the order in which students complete their clerkships. Studies examining clerkship length have shown that for most specialty areas, students in longer rotations outperform those in shorter ones.4 – 6 In addition, previous research generally has shown stronger end-of-clerkship examination performance for rotations completed later in the academic year.7 – 12 For example, a student who performs well on an internal medicine examination completed at the end of the year may not have performed as well if they had taken the same clerkship earlier, likely reflecting an accumulation of clinical acumen over the course of the academic year.

A few studies at individual medical schools have investigated clerkship performance in relation to completion of other specific rotations. One study examined the effect of clerkship sequence on family medicine examination scores and found that students who completed an internal medicine rotation prior to family medicine outperformed students who completed a family medicine first or completed just a psychiatry rotation.10 A more recent study indicated that the specialty of the first clerkship completed impacts subsequent performance in other rotations, with the strongest positive effects found for initial rotations in internal medicine.13 No studies have been identified that examined clerkship sequence using national samples.

OBJECTIVES

The current study uses a national sample of US medical school graduates to examine the effects of prior clerkship experiences on performance on an end-of-clerkship examination in internal medicine, controlling for basic science achievement. It specifically addresses the relationships between scores on the internal medicine examination and 1) completion of prior rotations in family medicine, ob/gyn, pediatrics, psychiatry, and surgery; and 2) the length of the internal medicine clerkship.

PARTICIPANTS

The dataset used in this study included 16,091 students from 67 Liaison Committee on Medical Education (LCME)-accredited medical schools who (1) graduated from medical school in 2012 or 2013, (2) took Step 1 of the United States Medical Licensing Examination® (USMLE®), and (3) took the NBME subject examinations in internal medicine, family medicine, ob/gyn, pediatrics, psychiatry, and surgery for the first time as end-of-clerkship examinations under standard testing conditions. This study was reviewed by the American Institutes for Research Institutional Review Board and qualified for exempt status because it involved very minimal or no risk to study subjects.

MAIN MEASURES

NBME internal medicine subject examination first-attempt scores were treated as the dependent measure in all analyses. The two-and-a-half hour internal medicine subject examination includes 100 multiple-choice questions, all of which have been used previously on USMLE Step 2 Clinical Knowledge (CK). Scores on the internal medicine subject examination are scaled to have a projected mean of 70 and a standard deviation of 8 based on a 1993–1994 criterion group. Virtually all questions begin with a description of a patient care situation. On average, about 60–70 % of the questions describe patients in an ambulatory setting, with the remaining questions fairly evenly divided among patients presenting in the hospital and emergency department.

Independent variables at the student level included five dichotomous variables capturing whether or not students completed clerkship rotations in family medicine, pediatrics, psychiatry, ob/gyn, and surgery prior to completing a rotation in internal medicine. In addition, USMLE Step 1 first-attempt scores were used as a covariate to control for pre-clerkship differences in basic science achievement. In supplemental analyses, the total number of clerkships and the total number of weeks of training students completed (including training in internal medicine) at the time they took the internal medicine subject examination were used as student-level independent variables. School-level independent variables included the length (in weeks) of the internal medicine rotation and the school mean of USMLE Step 1 first-attempt scores.

DESIGN

The study data are hierarchically structured, with students nested in schools. Multilevel modeling techniques allow for an appropriate analysis of such nested data where the assumption of independence may be violated.14 , 15 For this study, a series of two-level multilevel models were estimated. These analyses included (1) a one-way random-effects analysis of variance (ANOVA) used to partition the overall variation in internal medicine subject examination scores into between-school and within-school components, (2) a series of random-coefficients models used to examine the relationships between previous clerkship experiences and internal medicine subject examination scores, controlling for Step 1 performance, and (3) a series of means-as-outcomes models used to investigate the effects of internal medicine clerkship length and average Step 1 scores at the school level.

At the student level, the primary random-coefficients model included all five dichotomous independent variables indicating completion of prior clerkship rotations. This random-coefficients model was used to build the final means-as-outcomes model where internal medicine clerkship length was used to predict school-to-school variation in average internal medicine scores (i.e., school intercepts from the final random-coefficients model).

Step 1 scores were used in two ways to control for differences in prior basic science academic achievement. Individual Step 1 scores, centered within each school to have a mean of 0, were used as a student-level covariate to control for within-school differences in examinee performance. In addition, mean Step 1 scores for each school were used as a school-level covariate to control for between-school differences in Step 1 performance.

To examine the effects of overall clerkship experience on internal medicine subject examination scores, two supplemental two-level random-coefficients models were estimated. In addition to Step 1 scores, the first included the total number of clerkships completed prior to taking the internal medicine subject examination (as a student-level independent variable), and the second included the total number of weeks of clinical training across all disciplines prior to completion of an internal medicine rotation (as a student-level independent variable).

KEY RESULTS

Table 1 provides summary statistics for the student-level variables. The mean internal medicine subject examination score was 77.6 with a standard deviation of 7.9, and the mean Step 1 score was 225 with a standard deviation of 21. Reflecting differences in rotation order, roughly 50 % of students completed each of the other clerkships before taking the internal medicine subject examination. This is not to say that 50 % of the students in the sample completed all of their other clerkships prior to completing their internal medicine clerkship, though some students likely completed multiple rotations prior to taking the internal medicine subject examination. On average, students completed 3.4 clerkships (standard deviation of 1.8) and received 26.1 weeks of training (standard deviation of 12.2) prior to completing their internal medicine rotation.

Table 1 Summary Statistics for Student-Level Variables (N = 16,091)
Full size table

Table 2 provides summary statistics for the school-level variables. The average of schools’ mean Step 1 scores was 224 with a standard deviation of 6. Clerkship lengths varied by specialty, with the longest rotations observed in internal medicine (mean = 10.0 weeks; SD = 2.0) and the shortest rotations observed in family medicine (mean = 5.4 weeks; SD = 1.5).

Table 2 Summary Statistics for School-Level Variables (N = 67)
Full size table

Results from the one-way random-effects ANOVA indicated a grand mean of 77.5 with a standard error of 0.2 for the internal medicine subject examination scores. The school-level variance was estimated to be 3.3 and the student-level variance was estimated to be 58.8. In other words, roughly 5 % of the variation in internal medicine scores was between schools and 95 % of the variation was within schools. The square root of 3.3—around 1.8—can be interpreted as the true standard deviation for school means on reported subject examination score scale, indicating the dispersion of school means.

Table 3 shows the results from the final means-as-outcomes model, including the five dichotomous student-level independent variables indicating completion status for prior clerkship rotations and Step 1 scores. In this model, the intercept (predicted school mean) was allowed to vary randomly across schools, and internal medicine clerkship length was used to predict school-to-school variation in intercepts.

Table 3 Estimates for the Final Multilevel Model Predicting Internal Medicine Subject Examination Scores
Full size table

A strong relationship was observed between internal medicine subject examination scores and Step 1 scores: each 10-point increase in Step 1 scores was associated with a 2.5-point increase in subject examination scores. The effects associated with completing rotations in family medicine, pediatrics, and surgery were all positive and statistically significant, after controlling for Step 1 scores, indicating that students with previous clerkship experiences in these clinical domains had higher internal medicine scores. (Coefficients for completion of ob/gyn and psychiatry were small and positive, but they were not statistically significant.) The coefficients indicate the score gains associated with completion of the clerkship, controlling for completion of other clerkships. Thus, for students that completed multiple clerkships prior to taking the internal medicine examination, the clerkship-specific effects can be summed to provide an estimate of the overall benefit of the clinical training received prior to completing the clerkship in internal medicine. Of the statistically significant effects, the largest benefit was found for surgery and the smallest benefit was found for family medicine. On average, students who completed surgery rotations prior to internal medicine scored 1.58 points higher, and students who completed family medicine rotations prior to internal medicine scored 0.87 points higher, after accounting for within-school differences in Step 1 performance and completion of other clerkships.

At the school level, mean internal medicine subject examination scores were positively related to internal medicine clerkship length, with each additional week of internal medicine training predicted to increase subject examination scores by 0.23 points. School means on the internal medicine subject examination were also related to mean Step 1 scores, with each 10-point increase in a school’s mean Step 1 score associated with a predicted increase in mean internal medicine subject examination performance of 2.4 points.

Based on the final means-as-outcomes model, the student-level variance was estimated to be 30.78. Compared to the estimated student-level variance from the one-way random-effects ANOVA, adding student-level predictors, including completion of clerkship rotations in family medicine, pediatrics, psychiatry, ob/gyn, and surgery, as well as USMLE Step 1 scores, reduced the within-school variance by 48 %. That is, these predictors accounted for approximately 48 % of the within-school variation in internal medicine subject examination scores. The addition of the two school-level predictors, internal medicine rotation length and USMLE Step 1 score school mean, reduced the between-school variance from 3.5 to 1.6, indicating that these predictors accounted for about 54 % of the school-level variance in internal medicine scores.

Table 4 presents the results of the two supplemental analyses. The first used the total number of clerkships completed as a student-level independent variable, with Step 1 scores as a covariate indicating prior basic science achievement. The second included total number of weeks of clinical training as a student-level independent variable, again with Step 1 scores used to control for prior achievement. In both analyses, these indicators of overall clinical experience were positively related to internal medicine examination scores. The first showed an expected 0.72-point score increase for each additional clerkship completed, and the second showed an expected 0.11-point score increase for each week of additional training. These findings are consistent with each other and the results for the other models: the coefficient for weeks of training is between 6 and 7 times the coefficient for clerkships completed, which is close to the average length of a clerkship. Furthermore, the sum of the clerkship-specific coefficients from the primary random-coefficients model is fairly similar to the results of the two supplemental models when based on the maximum number of clerkships completed (six rotations) and the maximum number of weeks of training (52 weeks).

Table 4 Estimates for the Supplemental Multilevel Models Predicting Internal Medicine Subject Examination Scores
Full size table

Table 5 provides information related to the estimated within-school and between-school variance components for each of the models. Larger reductions in within-school and between-school variance components indicate that a model explained more of the associated variance in internal medicine subject examination scores. Compared with the supplemental models, the primary random-coefficient model capturing the specific clerkships that students completed prior to taking the internal medicine subject examination explains slightly more of the within-school variance in internal medicine subject examination scores.

Table 5 Estimated Variance Components
Full size table

CONCLUSIONS

Using a national sample of over 16,000 medical students from more than 60 medical schools, this study examined the effects of clerkship sequence and clerkship length on performance on the NBME internal medicine subject examination when used as an end-of-clerkship assessment. Using multilevel modeling techniques, results demonstrate a significant positive association between internal medicine subject examination performance and prior completion of clerkship rotations in surgery, pediatrics and family medicine, as well as the length of the internal medicine clerkship. This generally is consistent with previous research from a single institution.13

While prior completion of three clerkships (surgery, pediatrics, and family medicine) was associated with better performance on the internal medicine examination, effect size estimates for completing surgery (+1.58 points) were larger than those for pediatrics (+ 0.91 points), and family medicine (+ 0.87 points) clerkships. Prior completion of ob/gyn and psychiatric clerkships appear to be associated with larger internal medicine subject test scores, but effect sizes were small and not statistically significant. There was a significant association between internal medicine subject test performance and rotation length: each additional week of internal medicine training was associated with an expected increase of 0.23 points on the medicine examination.

Results suggest that students interested in maximizing their performance on the internal medicine subject examination would benefit from completion of rotations in surgery, pediatrics, and family medicine prior to their internal medicine rotation and longer clinical training in internal medicine. As an example, the results in Table 3 can be used to compare the expected performance of 1) a student taking an 8-week internal medicine clerkship at the beginning of the academic year with 2) a student taking a 12-week clerkship at the end of the academic year after all rotations have been completed. If these two students have the same Step 1 score and attend medical schools with similar mean Step 1 scores, results indicate that the second student is expected to perform 4.54 points (a little more than half an SD) better than the first student on the internal medicine subject examination. A difference of this magnitude, depending on where a student is on the score scale, could affect the student’s overall clerkship grade.

If schools use the same standards for grading student performance across the academic year, students completing an internal medicine clerkship earlier may be adversely affected in terms of their subject examination score. Score interpretation materials from the NBME for the internal medicine subject examination provide norms (by time of year and clerkship length), allowing schools to adjust grading criteria based on these guidelines if they choose to do so. Preliminary analyses of subject examination scores in other disciplines have shown fairly similar patterns as those illustrated for internal medicine, and the NBME provides similar aids for score interpretation for these disciplines. Alternatively, if a school wishes to use the same grading criteria throughout the academic year and it is logistically feasible, schools may wish to give students some control over clerkship sequence, allowing students who plan to specialize in internal medicine to take their internal medicine clerkship after their surgery, pediatrics, and/or family medicine rotations.

While NBME subject examinations are widely used and contribute to clerkship grades at most US medical schools,2 , 3 different results might be observed for other learning outcomes. For example, variation in a standardized clinical clerkship evaluation form at a single institution was found to be related to clerkship sequence, but performance on an end-of-year objective structured clinical examination from the same institution was not.12 Moreover, other school-level factors not included in the current study, such as overall curricular approach and time spent on in-hospital versus ambulatory rotations, may also influence internal medicine examination performance. Thus, additional multilevel studies examining multiple and varied achievement measures as a function of a range of school-level characteristics for large national samples are needed to more fully understand the relationships between clerkship completion, clerkship length, and clerkship performance.

One limitation of the current study is that although a large national sample was analyzed, it was judged analytically desirable to remove from the sample students from schools that did not administer all subject examinations used in the analyses (internal medicine, family medicine, ob/gyn, pediatrics, psychiatry, and surgery). This allowed for a straightforward determination of clerkship sequence and helped to ensure that students did not gain clerkship experiences in other disciplines throughout the study period. Removal of these students reduced the number of schools included in the analyses to 67, likely because the family medicine subject examination is used by a somewhat smaller percentage of US schools than other core clerkship subject examinations. As a result, it is possible that the schools omitted from the analyses are systematically different than those that are included, though unreported analyses indicate that performance on both Step 1 and the internal medicine subject examination for the excluded schools are similar to performance on those examinations for the study sample.

The present study provides useful information related to the effects of prior clerkship experiences on clerkship performance in internal medicine based on a large national sample of US medical school graduates. Findings may have implications for curricular design and allocation of resources within medical schools, especially related to balancing the need for both hospital and ambulatory experiences, as well as incorporating emerging contemporary issues into internal medicine rotations.16