Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

How curricular changes influence medical students’ perceptions of basic science: A pilot study

  • Yousef Elfanagely ,

    Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Validation, Visualization, Writing – original draft, Writing – review & editing

    yelfanagely@gmail.com

    Affiliation Department of Internal Medicine, Brown University, Providence, RI, United States of America

  • Joshua Ray Tanzer,

    Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Resources, Software, Writing – original draft, Writing – review & editing

    Affiliation Department of Biostatistics, Brown University, Providence, RI, United States of America

  • Ricardo Pulido,

    Roles Conceptualization, Data curation, Methodology, Resources

    Affiliation Department of Otolaryngology-Head and Neck Surgery, University of Washington, Seattle, WA, United States of America

  • Hanin Rashid,

    Roles Conceptualization, Data curation, Investigation, Methodology, Supervision, Validation, Writing – original draft, Writing – review & editing

    Affiliation Department of Psychiatry, Rutgers Robert Wood Johnson Medical School, New Brunswick, New Jersey, United States of America

  • Liesel Copeland

    Roles Conceptualization, Data curation, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing

    Affiliation Department of Education, Rutgers Robert Wood Johnson Medical School, New Brunswick, New Jersey, United States of America

Abstract

Theory

The perceived value of study material may have implications on learning and long-term retention. This study compares the perceived value of basic science of medical students from schools with a traditional “2+2” curriculum and the USMLE Step 1 placed before core clerkships to those from medical schools that have undergone curricular revisions, resulting in shortened pre-clerkship curricula and administration of the USMLE Step 1 after core clerkships.

Hypothesis

We hypothesize that differences in curricula, particularly duration of pre-clerkship curriculum and timing of the USMLE Step 1, affect medical students’ perceived value of basic science.

Methods

A twenty item anonymous questionnaire using a 5-point Likert scale was developed to assess medical students’ perceptions of basic science. The questionnaire was distributed to third-year medical students across four medical schools. Generalized linear models and p-values were calculated comparing the perceived value and use of basic science between medical schools with the USMLE Step 1 before clerkships and 2-years of basic science (BC) and medical schools with the USMLE Step 1 after core clerkships and 1.5-years of basic science (AC).

Results

The questionnaire was distributed to 695 eligible students and completed by 287 students. Students at BC schools tended to view basic science as more essential for clinical practice than students at AC schools across both outcomes (rating independence of basic science and clinical practice, AC school mean = 2.97, BC school mean = 2.73, p = 0.0017; rating importance of basic science to clinical practice, AC school mean = 3.30, BC schools mean = 3.50, p = 0.0135).

Conclusions

Our study suggests that students who have a longer basic science curriculum tend to value basic science greater than students with a shorter basic science curriculum. The timing of the USMLE Step 1 may also influence this relationship. Curricular decisions, such as reductions in pre-clerkship curricula and administration of the USMLE Step 1 after clerkships, may impact medical students’ perceptions of the value of basic science to clinical practice. This can have implications on their future engagement with basic science and should be considered when modifying curriculum.

Introduction

The current structure of medical education can be attributed to the reform following early criticisms of medical schools in Flexner’s 1910 report. In his report, Flexner criticized the nonscientific approach and lack of emphasis on basic science in medical courses and research [13]. He argued that basic science played a fundamental role in the practice of medicine and should be reflected in medical training [4]. Since Flexner’s report, many schools have adopted a “2+2” model with medical students spending their first two years taking basic science courses, such as anatomy and biochemistry, and their last two years completing clinical training in a teaching hospital [5]. In support of Flexner’s criticisms, additional studies have provided evidence that basic science is integrated in clinical reasoning [610].

While preclinical basic science is a pillar of medical education, the literature assessing the attitudes medical students have towards basic science is incongruent [11, 12]. Some studies cite positive attitudes shared by medical students towards basic science [1318]. Other studies report students having increasingly negative opinions of basic science during their medical education [1921]. Assessing the attitudes and subjective value medical students have towards basic science is important because attitude and subjective value of study material influence motivation, deeper learning, and future engagement with material [2224]. Further investigation is warranted as to what factors may contribute to medical students’ perceived value of basic science.

One factor that may contribute to the subjective value of basic science is curriculum design. The two prominent curricular changes that differ from the traditional “2+2” model are the shortening of pre-clerkship courses and the administration of the USMLE Step 1 after completion of core clerkships [25, 26]. The USMLE Step 1 is a standardized exam representing the culmination of applied basic science knowledge acquired during medical school [27]. The USMLE Step 1 is not without its shortcomings. Some medical students adopt a “binge and purge” mentality when studying for the exam as they view its contents as “clinically irrelevant minutiae” [28]. Despite this, the USMLE Step 1 score is the most prominent basic science exam and can be viewed as an indirect measure of short-term retention of basic science material. Medical schools that have changed the timing of the USMLE Step 1 to post-clerkship and reduced the length of pre-clerkship courses have done so to promote more integrated basic science learning in a clinical context [25]. After their curriculum changes, these medical schools have reported higher mean USMLE Step 1 scores [25]. It is unclear, however, if the reported increase in USMLE Step 1 score is due to curriculum changes as National USMLE scores have steadily increased and the schools involved in the study have had USMLE Step 1 scores historically above average [29].

In this paper, the subjective value of basic science will be assessed in relation to the curriculum design of medical schools. Unlike the USMLE Step 1, which is a short-term goal influenced by career aspirations, the subjective value of basic science can have an important influence on behaviors, including long-term engagement with learning material and retention [24, 3032]. Long-term retention of basic science is practically important given the frequent licensing exams doctors are required to take, but also empirically important as the value of education depends largely upon its lifespan [33]. Additionally, despite the widespread belief that much of the factual knowledge learned in medical courses is quickly forgotten, this forgetfulness is more accurately described as “temporarily inaccessible,” and the information can be recalled when needed in clinical circumstances [33]. Ultimately, we hope to evaluate the relationship between curriculum structure and attitudes towards basic science, and as such contribute to the current debate about curriculum changes.

Methods

Materials

We developed a twenty item questionnaire (see Table 1) to assess medical students’ perceptions of the value of basic science. The items are presented in a 5-point Likert-scale with scale points labeled “strongly disagree” (1), “disagree” (2), “neutral” (3), “agree” (4), and “strongly agree” (5). The questionnaire and study protocol were reviewed and approved by the Rutgers University Institutional Review Board (IRB #Pro20160000954).

thumbnail
Table 1. Twenty item questionnaire used to assess student’s perceived importance of basic science in different settings such as career, classroom, and licensing exams.

https://doi.org/10.1371/journal.pone.0236365.t001

The items in the questionnaire addressed the role of basic science in medical education (see Table 2). Established techniques for developing questionnaires were used to ensure the items posed in our questionnaire were valid for the purpose of this study [3436]. Items within the questionnaire were inspired by the surveys constructed by Custers, Alam, and Gupta who also assessed medical students’ attitudes towards basic science [13, 19, 20]. After development, the questionnaire was piloted by twenty medical students from two of the four participating medical schools. Students were asked to assess the quality and length of the questionnaire and whether individual questions were repetitive, biased, ambiguous, or confusing. Overall, feedback was positive and only minor corrections were made.

thumbnail
Table 2. Two categories were developed to better encapsulate different attitude about basic sciences.

https://doi.org/10.1371/journal.pone.0236365.t002

Distribution

The questionnaire was anonymously distributed to third-year medical students across four medical schools. Names of the institutions were withheld due to the sensitive and exploratory nature of this data. Institution A and B were private medical schools. Institution C and D were public medical schools. The medical schools were chosen because of their different approaches in scheduling the USMLE Step 1 and their relative geographic proximity. All four medical schools were located in the Mid-Atlantic region of the United States (see Table 3). Two medical schools (Institution A and Institution B) have their allotted dedicated study period for the USMLE Step 1 after 12 months of clerkship experience (AC). The two remaining medical schools (Institution C and Institution D) have their allotted dedicated study period for the USMLE Step 1 immediately after 2 years of basic science material (BC). Distribution of the questionnaire occurred from March of third-year to October of fourth-year for medical students at all four medical schools. The timing of questionnaire distribution was chosen because all study participants would have completed their USMLE Step 1 and had at least 6 months of clerkship experience. The link to the on-line questionnaire was distributed via e-mail and promoted via Facebook class pages. Four reminder e-mails were sent after the initial email sharing the link to increase response rate.

thumbnail
Table 3. Clinical experiences prior to taking USMLE Step 1 of the four medical schools participating in this study.

https://doi.org/10.1371/journal.pone.0236365.t003

Analysis

The goal of the analysis was to identify whether or not AC schools (Institution A and Institution B) and BC schools (Institution C and Institution D) had meaningful differences in perceptions of the importance of the basic science curriculum. Generalized linear mixed modeling was used because clustered sampling from multiple schools violates the assumption of independence [37]. Random intercepts were estimated by each university to address correlations between errors. The means across the AC schools and BC schools were calculated.

To account for possible confounding, two covariates were added. First, a random effect covariate was added to account for the possibility of differences between AC schools (Institution A and Institution B). Institution B modified their curriculum more recently than Institution A. The random effect was included to allow for differences between these two institutions. Second, due to the ongoing sampling design, participants who completed the questionnaire at different times of the year may have had differing opinions due to the academic calendar. To account for this, a fixed effect of “calendar year” was included.

Before data were analyzed, a priori power analysis was performed based on the equations provided by Fitzmaurice, Laird, and Ware [38]. The sample size used in the analysis included 287 cases. Type one error rate alpha was fixed at the research standard 0.05. Small, medium, and large population effect conditions were assumed by varying the intraclass correlation coefficient (ICC; small = 0.50, medium = 0.30, large = 0.10) and difference between means (small = 0.20, medium = 0.50, and large = 0.80). These are standard recommendations for effect sizes from Cohen (1992). Even when population effects were assumed the hardest to detect (i.e. ICC = 0.50, difference between means = 0.20), power for the analysis was above the research standard of 0.80 (power = 0.90). This suggests that, if there was a true population effect, the analytic design was well equipped to identify it.

Results

The questionnaire was distributed to a total of 695 students, however only 287 students (41.29%) completed the questionnaire. There were a total of 133 (39.12%) participants from AC schools and 154 (42.25%) participants from BC schools (Table 4). A factor analysis was used to assess the underlying structure of the question items [39]. The first two eigenvalues were greater than 1, suggesting that there may be two clusters of question items. Examining the item loadings with two factors extracted the first set of items (items 1, 2, 5, 8, 10, 12, 19, and 20) which represented the belief that clinical practice may function independently of basic science understanding. The remaining items (items 3, 4, 6, 7, 9, 11, 13, 14, 15, 16, 17, and 18) represented the belief that basic science is important for clinical practice. The description of the categories was based on what each item represented. As a diagnostic check of scale reliability, McDonald’s coefficient omega was calculated within the sample using Rstudio psych package [40]. The sample estimate was within the recommended value of 0.80 (hierarchal omega = 0.76) [41]. These two subscales were taken as the outcome variables in the analysis.

Data distributions were examined to ensure that they were appropriately distributed for the analysis. For both subscales, skew and kurtosis was less than |1.0|, suggesting that they were appropriate for the analysis. Models were fit using the Rstudio lme4 package [42]. Residual plots were examined visually and appeared normally distributed; skewness and kurtosis for the residuals was less than |1.0|. Means for the AC and BC school are displayed in Table 5.

Significant differences were identified across both study variables. The rating scale for all items used a 3 as average. Students at AC schools tended to view basic science as less essential for clinical practice than students at BC schools across both outcomes (rating independence of basic science and clinical practice, AC school mean = 2.97, BC school mean = 2.73, p = 0.0017; rating importance of basic science to clinical practice, AC school mean = 3.30, BC schools mean = 3.50, p = 0.0135). The effect sizes of these differences were moderately small (rating independence of clinical practice, Cohen’s d = 0.35; rating importance of basic science to clinical practice, Cohen’s d = 0.29).

Next, differences were considered in relation to covariates. Differences between Institution A and Institution B were not large. The ICC for these groups was essentially zero (rating independence of basic science and clinical practice, ICC<0.01; rating importance of basic science to clinical practice, ICC<0.01). This suggests that there was no evidence of differences between Institution A and Institution B despite their differing histories of changing the USMLE Step 1 and clerkship timing. Relationships to calendar year were nonsignificant across both outcomes (rating independence of clinical practice, beta<0.01; p = 0.2337; rating importance of basic science to clinical practice, beta<0.01; p = 0.5359).

Discussion

In this pilot study involving four medical schools, we investigated whether curriculum structure has an influence on medical students’ perceptions of the value of basic science. Perceptions can have an important influence on behaviors, including acquisition and retention of learning material [30, 31]. Thus, even though self-perception may not always accurately reflect knowledge, this study can inform the current discussions surrounding medical school curricular structures [43].

Medical students who had 2 years of pre-clerkship courses and their USMLE Step 1 exam before clerkships more strongly endorsed the belief that basic science is essential for clinical practice. Results were independent of the timing of transition from the traditional curriculum and when medical students completed the questionnaire. Therefore, it is likely results were reflective of the differences in dedicated time learning basic science and time spent in the hospital.

We hypothesize that schools with a longer basic science curriculum before clerkship rotations may emphasize basic science to a greater extent. Alternatively, curricula with reduced time dedicated to learning basic science and earlier exposure to the hospital in clerkship rotations may reinforce the belief that preparation for clerkship rotations is less reliant on basic science. In Walsh et al, medical students demonstrated significant interest in learning about patients in the context of their disease for their own benefit [44]. More time spent in the hospital may expose students to other factors that aid in clinical decisions and patient-centered care. Students may begin to view clinical experience or evidence based medicine as the crux of clinical practice. Clinical experiences and patient preference can often dictate the management of difficult patient cases that do not follow the textbook features discussed in basic science courses [45].

We hope our findings can contribute to the ongoing discussion of curriculum reform as new challenges, such as changing the USMLE Step 1 exam to a pass/fail grading system, present themselves [46]. Many variables impact medical education, including student perceptions of basic science, which must be considered when adjusting a curriculum [47, 48]. Future studies should further explore the impact of curricular design on students’ perceived value. Ultimately, the perceived value of basic science is important because of its implications on long-term retention and processing new material. Moreover, it is critical that students and practitioners are able and willing to transfer their basic science knowledge to clinical practice when encountering novel problem solving [49]. If students do not perceive value in basic science, deeper learning and application may be affected [24].

Limitations

Our study demonstrates how changes in curricula affect the attitude medical students’ have towards basic science. We cannot delineate if the differences observed were secondary to the reduction in pre-clerkship courses or administration of the USMLE Step 1 after core clerkships as both medical schools underwent these curricular changes. Additionally, the study design was observational, which can be limited by confounding variables and cannot determine a cause-and-effect relationship [50]. Data analysis was exploratory and empirically driven. The two categories for our question items were created after a factor analysis. Several steps were taken during the data analysis to limit potential confounding variables. Only 287 students completed the questionnaire out of the 695 eligible students (41.29%). Analysis included measures to ensure nonresponse rate did not significantly influence data. However, low response rates can limit the ability to generalize findings [51]. Data collected also had a response bias in favor of Institution C, with 50.60% of eligible students completing the questionnaire. Demographics, such as age and gender of the participants, were not collected. Additional information as to who completed the questionnaire could have aided in the generalizability of this study. Our data analysis also included measures to ensure there was no statistically significant difference between medical schools with similar curricula, despite adjusting curriculum at different times. However, school culture may have influenced the results gathered. The AC medical schools were private institutions, while the BC medical schools were public institutions. Factors such as incoming undergraduate GPA, tuition, and the quality and volume of institutional biomedical research can also affect school culture. Furthermore, curricula design in and of itself is part of the culture of a school and changing this may alter the culture. Hence it is important to better understand the influence of curricular design on medical students’ values.

References

  1. 1. de Bruin AB, Schmidt HG, Rikers RMJAM: The role of basic science knowledge and clinical knowledge in diagnostic reasoning: a structural equation modeling approach. 2005, 80:765–773. pmid:16043534
  2. 2. Pawlina WJMt: Basic sciences in medical education: why? How? When? Where? 2009, 31:787–789. pmid:19811182
  3. 3. Cooke M, Irby DM, Sullivan W, Ludmerer KMJNEjom: American medical education 100 years after the Flexner report. 2006, 355:1339–1344. pmid:17005951
  4. 4. Finnerty EP, Chauvin S, Bonaminio G, Andrews M, Carroll RG, Pangaro LNJAM: Flexner revisited: the role and value of the basic sciences in medical education. 2010, 85:349–355. pmid:20107367
  5. 5. Woods NN, Neville AJ, Levinson AJ, Howey EH, Oczkowski WJ, Norman GRJAM: The value of basic science in clinical diagnosis. 2006, 81:S124–S127.
  6. 6. Dixon JA, Spinale FGJCHF: Large animal models of heart failure: a critical link in the translation of basic science to clinical practice. 2009, 2:262–271. pmid:19808348
  7. 7. Johansen JS, Harris AK, Rychly DJ, Ergul AJCd: Oxidative stress and the use of antioxidants in diabetes: linking basic science to clinical practice. 2005, 4:5. pmid:15862133
  8. 8. Campana V, Milano G, Pagano E, Barba M, Cicione C, Salonna G, et al: Bone substitutes in orthopaedic surgery: from basic science to clinical practice. 2014, 25:2445–2461. pmid:24865980
  9. 9. Akat K, Borggrefe M, Kaden JJJH: Aortic valve calcification: basic science to clinical practice. 2009, 95:616–623. pmid:18632833
  10. 10. Shiromani PJ: Post-traumatic stress disorder: basic science and clinical practice. Springer; 2009.
  11. 11. Grande JPJMt: Training of physicians for the twenty-first century: role of the basic sciences. 2009, 31:802–806. pmid:19811184
  12. 12. Woods NN, Brooks LR, Norman GRJMe: The value of basic science in clinical diagnosis: creating coherence among signs and symptoms. 2005, 39:107–112. pmid:15612907
  13. 13. Gupta S, Gupta AK, Verma M, Kaur H, Kaur A, Singh KJIJoA, Research BM: The attitudes and perceptions of medical students towards basic science subjects during their clinical years: A cross-sectional survey. 2014, 4:16.
  14. 14. Shankar R, Dubey A, Mishra P, Deshpande V, Chandrasekhar T, Shivananda PJEfH: Student attitudes towards communication skills training in a medical college in Western Nepal. 2006, 19:71–84.
  15. 15. Shah DK, Jha RK, Sah AK, Sah P, Paudel K, Paudel PJJCME: Students’ attitude and perception towards basic sciences in a medical School of Nepal: A Cross-sectional Study. 2015, 3:165–169.
  16. 16. Stevenson FT, Bowe CM, Gandour‐Edwards R, Kumari VGJMe: Paired basic science and clinical problem‐based learning faculty teaching side by side: do students evaluate them differently? 2005, 39:194–201. pmid:15679687
  17. 17. Kaufman DM, Mann KVJMe: Basic sciences in problem‐based learning and conventional curricula: students' attitudes. 1997, 31:177–180. pmid:9231134
  18. 18. Pruskil S, Burgwinkel P, Georg W, Keil T, Kiessling CJMt: Medical students’ attitudes towards science and involvement in research activities: a comparative study with students from a reformed and a traditional curriculum. 2009, 31:e254–e259.
  19. 19. Alam AJAoSm: How do medical students in their clinical years perceive basic sciences courses at King Saud University? 2011, 31:58–61. pmid:21245601
  20. 20. Custers EJ, Cate OTJTJMe: Medical students' attitudes towards and perception of the basic sciences: a comparison between students in the old and the new curriculum at the University Medical Center Utrecht, The Netherlands. 2002, 36:1142–1150.
  21. 21. Bhangu A, Boutefnouchet T, Yong X, Abrahams P, Joplin RJAse: A three‐year prospective longitudinal cohort study of medical students' attitudes toward anatomy teaching and their career aspirations. 2010, 3:184–190. pmid:20607859
  22. 22. Artino AR, La Rochelle JS, Durning SJJMe: Second‐year medical students’ motivational beliefs, emotions, and achievement. 2010, 44:1203–1212. pmid:21091760
  23. 23. Dennick RJBE: How much biochemistry should a good doctor know? An educationalist's perspective. 1996, 24:85–88.
  24. 24. Eccles JJEP: Who am I and What am I Going to do with my Life? 2007.
  25. 25. Daniel M, Fleming A, Grochowski COC, Harnik V, Klimstra S, Morrison G, et al: Why not wait? Eight institutions share their experiences moving United States Medical Licensing Examination Step 1 after core clinical clerkships. 2017, 92:1515–1524.
  26. 26. Jurich D, Daniel M, Paniagua M, Fleming A, Harnik V, Pock A, et al: Moving the United States Medical Licensing Examination Step 1 after core clerkships: an outcomes analysis. 2019, 94:371–377. pmid:30211755
  27. 27. Ling Y, Swanson DB, Holtzman K, Bucak SDJAM: Retention of basic science information by senior medical students. 2008, 83:S82–S85. pmid:18820508
  28. 28. Barbieri JSJJ: Basic science content in the USMLE Step 1. 2014, 311:1358–1359. pmid:24691619
  29. 29. FACTS Table 17. Association of American Medical Colleges. 2013.
  30. 30. Cavallo AM, Laubach TAJJoRiSTTOJotNAfRiST: Students' science perceptions and enrollment decisions in differing learning cycle classrooms. 2001, 38:1029–1062.
  31. 31. KHOURY G: A MODEL IDENTIFYING SELECTED FACTORS INFLUENCING SCIENCE ENROLLMENT PATTERNS OF HIGH SCHOOL STUDENTS (ATTITUDES, ACADEMIC ABILITIES, SEX DIFFERENCES). 1985.
  32. 32. Jones MD, Yamashita T, Ross RG, Gong JJBme: Positive predictive value of medical student specialty choices. 2018, 18:33. pmid:29523127
  33. 33. Custers EJ, ten Cate OTJMe: Very long‐term retention of basic science knowledge in doctors after graduation. 2011, 45:422–430. pmid:21401691
  34. 34. Abell N, Springer DW, Kamata A: Developing and validating rapid assessment instruments. Oxford University Press; 2009.
  35. 35. Kazi AM, Khalid WJJotPMA: Questionnaire designing and validation. 2012, 62:514.
  36. 36. Arora Z, Garber A, Thota PN: Risk factors for Barrett's esophagus. J Dig Dis. 2016, 17:215–221. pmid:26929263
  37. 37. Brown H, Prescott R: Applied mixed models in medicine. John Wiley & Sons; 2014.
  38. 38. Fitzmaurice GM, Laird NM, Ware JH: Applied longitudinal analysis. John Wiley & Sons; 2012.
  39. 39. Harlow LL: The essence of multivariate thinking: Basic themes and methods. Routledge; 2014.
  40. 40. Revelle WJAe: Procedures for psychological, psychometric, and personality research. 2012, 9.
  41. 41. Dunn TJ, Baguley T, Brunsden VJBjop: From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. 2014, 105:399–412. pmid:24844115
  42. 42. Bates D, Mächler M, Bolker B, Walker SJapa: Fitting linear mixed-effects models using lme4. 2014.
  43. 43. Rubio M, Sanchez-Ronco M, Mohedano R, Hernando AJPo: The impact of participatory teaching methods on medical students’ perception of their abilities and knowledge of epidemiology and statistics. 2018, 13. pmid:30133528
  44. 44. Walsh KJTUmj: E-learning for medical education: reflections of learners on patients. 2018, 87:46. pmid:29588558
  45. 45. Hill J, Rolfe IE, Pearson SA, Heathcote AJMe: Do junior doctors feel they are prepared for hospital practice? A study of graduates from traditional and non‐traditional medical schools. 1998, 32:19–24. pmid:9624395
  46. 46. Rozenshtein A, Mullins ME, Marx MVJAr: The USMLE step 1 pass/fail reporting proposal: the APDR position. 2019, 26:1400–1402. pmid:31383545
  47. 47. Kusurkar R, Ten Cate TJ, Van Asperen M, Croiset GJMt: Motivation as an independent and a dependent variable in medical education: a review of the literature. 2011, 33:e242–e262.
  48. 48. Giesler M, Karsten G, Ochsendorf F, Breckwoldt JJGjfme: Conditions for excellence in teaching in medical education: The Frankfurt Model to ensure quality in teaching and learning. 2017, 34. pmid:29085890
  49. 49. Castillo JM, Park YS, Harris I, Cheung JJ, Sood L, Clark MD, et al: A critical narrative review of transfer of basic science knowledge in health professions education. 2018, 52:592–604. pmid:29417600
  50. 50. Tofthagen C: Threats to validity in retrospective studies. Journal of the advanced practitioner in oncology. 2012, 3:181. pmid:25031944
  51. 51. Pinsonneault A, Kraemer KJJomis: Survey research methodology in management information systems: an assessment. 1993, 10:75–105.