Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Perspective
  • Published:

Reducing bias, increasing transparency and calibrating confidence with preregistration

Nature Human Behaviour volume 7, pages 15–26 (2023)Cite this article

Subjects

Abstract

Flexibility in the design, analysis and interpretation of scientific studies creates a multiplicity of possible research outcomes. Scientists are granted considerable latitude to selectively use and report the hypotheses, variables and analyses that create the most positive, coherent and attractive story while suppressing those that are negative or inconvenient. This creates a risk of bias that can lead to scientists fooling themselves and fooling others. Preregistration involves declaring a research plan (for example, hypotheses, design and statistical analyses) in a public registry before the research outcomes are known. Preregistration (1) reduces the risk of bias by encouraging outcome-independent decision-making and (2) increases transparency, enabling others to assess the risk of bias and calibrate their confidence in research outcomes. In this Perspective, we briefly review the historical evolution of preregistration in medicine, psychology and other domains, clarify its pragmatic functions, discuss relevant meta-research, and provide recommendations for scientists and journal editors.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Evidentiary and interpretative degrees of freedom.
Fig. 2: The preregistration continuum.

Similar content being viewed by others

References

  1. Bacon, F. Novum Organum (P. F. Collier, 1620).

  2. Gilovich, T. How We Know What Isn’t So: The Fallibility of Human Reason in Everyday Life (Free Press, 1991).

  3. Laplace, P. S. Essai Philosophique sur les Probabilités (Springer, 1825).

  4. Nosek, B. A., Spies, J. R. & Motyl, M. Scientific utopia: II. restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7, 615–631 (2012).

    Article  Google Scholar 

  5. Smaldino, P. E. & McElreath, R. The natural selection of bad science. R. Soc. Open Sci. 3, 160384 (2016).

    Article  Google Scholar 

  6. Young, N. S., Ioannidis, J. P. A. & Al-Ubaydli, O. Why current publication practices may distort science. PLoS Med. 5, e201 (2008).

    Article  Google Scholar 

  7. Glasziou, P. et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet 383, 267–276 (2014).

    Article  Google Scholar 

  8. Nissen, S. B., Magidson, T., Gross, K. & Bergstrom, C. T. Publication bias and the canonization of false facts. eLife 5, e21451 (2016).

    Article  Google Scholar 

  9. IJzerman, H. et al. Use caution when applying behavioural science to policy. Nat. Hum. Behav. 4, 1092–1094 (2020).

    Article  Google Scholar 

  10. Prasad, V. K. & Cifu, A. S. Ending Medical Reversal: Improving Outcomes, Saving Lives (John Hopkins Univ. Press, 2019).

  11. Christensen, G. & Miguel, E. Transparency, reproducibility, and the credibility of economics research. J. Econ. Lit. 56, 920–980 (2018).

    Article  Google Scholar 

  12. Pashler, H. & Wagenmakers, E.-J. Editors’ introduction to the special section on replicability in psychological science: a crisis of confidence? Perspect. Psychol. Sci. 7, 528–530 (2012).

  13. Baker, M. 1,500 scientists lift the lid on reproducibility. Nature 533, 452–454 (2016).

    Article  CAS  Google Scholar 

  14. Ioannidis, J. P. A. Why most published research findings are false. PLoS Med. 2, e124 (2005).

    Article  Google Scholar 

  15. Feynman, R. P. Cargo Cult Science (California Institute of Technology, 1974).

  16. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J. & Kievit, R. A. An agenda for purely confirmatory research. Perspect. Psychol. Sci. 7, 632–638 (2012).

    Article  Google Scholar 

  17. Nosek, B. A., Ebersole, C. R., DeHaven, A. C. & Mellor, D. T. The preregistration revolution. Proc. Natl Acad. Sci. USA 115, 2600–2606 (2018).

    Article  CAS  Google Scholar 

  18. de Groot, A. D. The meaning of “significance” for different types of research. Acta Psychol. 148, 188–194 (2014).

    Google Scholar 

  19. Keynes, J. M. Treatise on Probability (Macmillan & Co, 1921).

  20. Mill, J. S. A System of Logic (George Routledge & Sons, 1843).

  21. Peirce, C. S. in Studies in Logic (ed. Peirce, C. S.) 126–281 (Little, Brown, 1883).

  22. Bakan, D. The test of significance in psychological research. Psychol. Bull. 66, 423–437 (1966).

    Article  CAS  Google Scholar 

  23. Walster, G. W. & Cleary, T. A. A proposal for a new editorial policy in the social sciences. Am. Stat. 24, 16–19 (1970).

    Google Scholar 

  24. Wiseman, R., Watt, C. & Kornbrot, D. Registered reports: an early example and analysis. PeerJ 7, e6232 (2019).

    Article  Google Scholar 

  25. Spellman, B. A. A short (personal) future history of Revolution 2.0. Perspect. Psychol. Sci. 10, 886–899 (2015).

    Article  Google Scholar 

  26. Simes, R. J. Publication bias: the case for an international registry of clinical trials. J. Clin. Oncol. 4, 1529–1541 (1986).

    Article  CAS  Google Scholar 

  27. Serghiou, S., Axford, C. & Ioannidis, J. P. A. Lessons learnt from registration of biomedical research. Nat. Hum. Behav., https://doi.org/10.1038/s41562-022-01499-0 (2023).

  28. De Angelis, C. et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. N. Engl. J. Med. 351, 1250–1251 (2004).

    Article  Google Scholar 

  29. Dickersin, K. & Rennie, D. The evolution of trial registries and their use to assess the clinical trial enterprise. JAMA 307, 1861–1864 (2012).

    Article  CAS  Google Scholar 

  30. World Medical Association. World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA 310, 2191 (2013).

    Article  Google Scholar 

  31. Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C. & Altman, D. G. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 291, 2457–2465 (2004).

    Article  CAS  Google Scholar 

  32. Dwan, K. et al. Evidence for the selective reporting of analyses and discrepancies in clinical trials: a systematic review of cohort studies of clinical trials. PLoS Med. 11, e1001666 (2014).

    Article  Google Scholar 

  33. Zarin, D. A., Tse, T., Williams, R. J. & Rajakannan, T. Update on trial registration 11 years after the ICMJE policy was established. N. Engl. J. Med. 376, 383–391 (2017).

    Article  Google Scholar 

  34. Greenberg, L., Jairath, V., Pearse, R. & Kahan, B. C. Pre-specification of statistical analysis approaches in published clinical trial protocols was inadequate. J. Clin. Epidemiol. 101, 53–60 (2018).

    Article  Google Scholar 

  35. Chan, A.-W., Hróbjartsson, A., Jørgensen, K. J., Gøtzsche, P. C. & Altman, D. G. Discrepancies in sample size calculations and data analyses reported in randomised trials: comparison of publications with protocols. Brit. Med. J. 337, a2299 (2008).

    Article  Google Scholar 

  36. Humphreys, M., Sierra, R. Sdela & van der Windt, P. Fishing, commitment, and communication: a proposal for comprehensive nonbinding research registration. Polit. Anal. 21, 1–20 (2013).

    Article  Google Scholar 

  37. Dal-Ré, R. et al. Making prospective registration of observational research a reality. Sci. Transl. Med. 6, 1–4 (2014).

    Article  Google Scholar 

  38. Lash, T. L. Preregistration of study protocols is unlikely to improve the yield from our science, but other strategies might. Epidemiology 21, 612–613 (2010).

    Article  Google Scholar 

  39. Boccia, S. et al. Registration practices for observational studies on ClinicalTrials.gov indicated low adherence. J. Clin. Epidemiol. 70, 176–182 (2016).

    Article  Google Scholar 

  40. Weber, W. E. J., Merino, J. G. & Loder, E. Trial registration 10 years on. Brit. Med. J. 351, H3572 (2015).

    Article  Google Scholar 

  41. Nosek, B. A. et al. Preregistration is hard, and worthwhile. Trends Cogn. Sci. 23, 815–818 (2019).

    Article  Google Scholar 

  42. Szollosi, A. et al. Is preregistration worthwhile? Trends Cogn. Sci. 24, 94–95 (2020).

    Article  Google Scholar 

  43. Devezer, B., Navarro, D. J., Vandekerckhove, J. & Buzbas, E. O. The case for formal methodology in scientific reform. R. Soc. Open Sci. 8, 200805 (2020).

    Article  Google Scholar 

  44. Rubin, M. Does preregistration improve the credibility of research findings? Quant. Methods Psychol. 16, 15 (2020).

    Article  Google Scholar 

  45. Simmons, J. P., Nelson, L. D. & Simonsohn, U. Pre-registration: why and how. J. Consum. Psychol. 31, 151–162 (2021).

    Article  Google Scholar 

  46. Monogan, J. E. Research preregistration in political science: the case, counterarguments, and a response to critiques. PS Polit. Sci. Polit. 48, 425–429 (2015).

    Article  Google Scholar 

  47. Olken, B. A. Promises and perils of pre-analysis plans. J. Econ. Perspect. 29, 61–80 (2015).

    Article  Google Scholar 

  48. Casey, K., Glennerster, R. & Miguel, E. Reshaping institutions: evidence on aid impacts using a preanalysis plan*. Q. J. Econ. 127, 1755–1812 (2012).

    Article  Google Scholar 

  49. Coffman, L. C. & Niederle, M. Pre-analysis plans have limited upside, especially where replications are feasible. J. Econ. Perspect. 29, 81–98 (2015).

    Article  Google Scholar 

  50. Shiffrin, R. M. Commentary on “robust modeling in cognitive science: misunderstanding the goal of modeling”. Comput. Brain Behav. 2, 176–178 (2019).

    Article  Google Scholar 

  51. Crüwell, S. & Evans, N. J. Preregistration in diverse contexts: a preregistration template for the application of cognitive models. R. Soc. Open Sci. 8, 210155 (2021).

    Article  Google Scholar 

  52. Paul, M., Govaart, G. H. & Schettino, A. Making ERP research more transparent: guidelines for preregistration. Int. J. Psychophysiol. https://doi.org/10.1016/j.ijpsycho.2021.02.016 (2021).

    Article  Google Scholar 

  53. van den Akker, O. et al. Preregistration of secondary data analysis: a template and tutorial. https://psyarxiv.com/hvfmr/ (2019).

  54. Haven, T. L. & Grootel, D. L. V. Preregistering qualitative research. Account. Res. 26, 229–244 (2019).

    Article  Google Scholar 

  55. Ofosu, G. K. & Posner, D. N. Pre-analysis plans: an early stocktaking. Perspect. Polit. https://doi.org/10.1017/S1537592721000931 (2020).

  56. Nosek, B. A. et al. Replicability, robustness, and reproducibility in psychological science. Annu. Rev. Psychol. https://doi.org/10.1146/annurev-psych-020821-114157 (2022).

  57. Christensen, G. et al. Open science practices are on the rise: the state of social science (3s) survey. Preprint at https://doi.org/10.31222/osf.io/5rksu (2019).

  58. Wallach, J. D., Boyack, K. W. & Ioannidis, J. P. A. Reproducible research practices, transparency, and open access data in the biomedical literature, 2015–2017. PLoS Biol. 16, e2006930 (2018).

    Article  Google Scholar 

  59. Hardwicke, T. E. et al. Estimating the prevalence of transparency and reproducibility-related research practices in psychology (2014–2017). Perspect. Psychol. Sci. https://doi.org/10.1177/1745691620979806 (2021).

    Article  Google Scholar 

  60. Hardwicke, T. E. et al. An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017). R. Soc. Open Sci. 7, 190806 (2020).

    Article  Google Scholar 

  61. Chambers, C. D. & Tzavella, L. The past, present and future of Registered Reports. Nat. Hum. Behav. https://doi.org/10.1038/s41562-021-01193-7 (2021).

  62. Montoya, A. K., Krenzer, W. L. D. & Fossum, J. L. Opening the door to registered reports: census of journals publishing registered reports (2013–2020). Collabra Psychol. 7, 24404 (2021).

    Article  Google Scholar 

  63. Hardwicke, T. E. & Ioannidis, J. P. A. Mapping the universe of registered reports. Nat. Hum. Behav. 2, 793–796 (2018).

    Article  Google Scholar 

  64. Leamer, E. E. Let’s take the con out of econometrics. Am. Econ. Rev. 73, 31–43 (1983).

    Google Scholar 

  65. Page, M. J., McKenzie, J. E. & Forbes, A. Many scenarios exist for selective inclusion and reporting of results in randomized trials and systematic reviews. J. Clin. Epidemiol. 66, 524–537 (2013).

    Article  Google Scholar 

  66. Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011).

    Article  Google Scholar 

  67. Oberauer, K. & Lewandowsky, S. Addressing the theory crisis in psychology. Psychon. Bull. Rev. 26, 1596–1618 (2019).

    Article  Google Scholar 

  68. Szollosi, A. & Donkin, C. Arrested theory development: the misguided distinction between exploratory and confirmatory research. Perspect. Psychol. Sci. https://doi.org/10.1177/1745691620966796 (2021).

    Article  Google Scholar 

  69. Forstmeier, W., Wagenmakers, E.-J. & Parker, T. H. Detecting and avoiding likely false-positive findings – a practical guide. Biol. Rev. 92, 1941–1968 (2017).

    Article  Google Scholar 

  70. Gelman, A. & Loken, E. The statistical crisis in science. Am. Sci. 102, 460–465 (2014).

    Article  Google Scholar 

  71. Duhem, P. The Aim and Structure of Physical Theory (Princeton Univ. Press, 1954).

  72. Gigerenzer, G. Surrogates for theories. Theory Psychol. 8, 195–204 (1998).

    Article  Google Scholar 

  73. Meehl, P. E. Theory-testing in psychology and physics: a methodological paradox. Phil. Sci. 34, 103–115 (1967).

    Article  Google Scholar 

  74. Muthukrishna, M. & Henrich, J. A problem in theory. Nat. Hum. Behav. 3, 221–229 (2019).

    Article  Google Scholar 

  75. Sackett, D. L. Bias in analytic research. J. Chronic Dis. 32, 51–63 (1979).

    Article  CAS  Google Scholar 

  76. Higgins, J. P. T. & Green, S. (eds) Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (The Cochrane Collaboration, 2011).

  77. DeCoster, J., Sparks, E. A., Sparks, J. C., Sparks, G. G. & Sparks, C. W. Opportunistic biases: their origins, effects, and an integrated solution. Am. Psychol. 70, 499–514 (2015).

    Article  Google Scholar 

  78. Greenland, S. Analysis goals, error‐cost sensitivity, and analysis hacking: essential considerations in hypothesis testing and multiple comparisons. Paediatr. Perinat. Epidemiol. 35, 8–23 (2021).

    Article  Google Scholar 

  79. Scott, J. G. & Berger, J. O. Bayes and empirical-Bayes multiplicity adjustment in the variable-selection problem. Ann. Stat. 38, 2587–2619 (2010).

    Article  Google Scholar 

  80. Dienes, Z. Bayesian versus orthodox statistics: which side are you on? Perspect. Psychol. Sci. 6, 274–290 (2011).

    Article  Google Scholar 

  81. Howson, C. & Urbach, P. Scientific Reasoning: The Bayesian Approach (Open Court, 2006).

  82. Dienes, Z. How Bayes factors change scientific practice. J. Math. Psychol. 72, 78–89 (2016).

    Article  Google Scholar 

  83. Wasserstein, R. L. & Lazar, N. A. The ASA’s statement on p-values: context, process, and purpose. Am. Stat. 70, 129–133 (2016).

    Article  Google Scholar 

  84. Armitage, P., McPherson, C. K. & Rowe, B. C. Repeated significance tests on accumulating data. J. R. Stat. Soc. A 132, 235–244 (1969).

    Article  Google Scholar 

  85. Stefan, A. & Schönbrodt, F. Big little lies: a compendium and simulation of p-hacking strategies. Preprint at https://doi.org/10.31234/osf.io/xy2dk (2022).

  86. Austin, P. C., Mamdani, M. M., Juurlink, D. N. & Hux, J. E. Testing multiple statistical hypotheses resulted in spurious associations: a study of astrological signs and health. J. Clin. Epidemiol. 59, 964–969 (2006).

    Article  Google Scholar 

  87. Bennett, C., Miller, M. & Wolford, G. Neural correlates of interspecies perspective taking in the post-mortem Atlantic salmon: an argument for multiple comparisons correction. NeuroImage 47, S125 (2009).

    Article  Google Scholar 

  88. Steegen, S., Tuerlinckx, F., Gelman, A. & Vanpaemel, W. Increasing transparency through a multiverse analysis. Perspect. Psychol. Sci. 11, 702–712 (2016).

    Article  Google Scholar 

  89. Patel, C. J., Burford, B. & Ioannidis, J. P. A. Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations. J. Clin. Epidemiol. 68, 1046–1058 (2015).

    Article  Google Scholar 

  90. Simonsohn, U., Simmons, J. P. & Nelson, L. D. Specification curve analysis. Nat. Hum. Behav. https://doi.org/10.1038/s41562-020-0912-z (2020).

  91. Botvinik-Nezer, R. et al. Variability in the analysis of a single neuroimaging dataset by many teams. Nature 582, 84–88 (2020).

    Article  CAS  Google Scholar 

  92. Silberzahn, R. et al. Many analysts, one data set: making transparent how variations in analytic choices affect results. Adv. Methods Pract. Psychol. Sci. https://doi.org/10.1177/2515245917747646 (2018).

    Article  Google Scholar 

  93. van Dongen, N. N. N. et al. Multiple perspectives on inference for two simple statistical scenarios. Am. Stat. 73, 328–339 (2019).

    Article  Google Scholar 

  94. Klau, S., Hoffmann, S., Patel, C. J., Ioannidis, J. P. A. & Boulesteix, A.-L. Examining the robustness of observational associations to model, measurement and sampling uncertainty with the vibration of effects framework. Int. J. Epidemiol. 50, 266–278 (2021).

    Article  Google Scholar 

  95. Fanelli, D. How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE 4, e5738 (2009).

    Article  Google Scholar 

  96. Fraser, H., Parker, T., Nakagawa, S., Barnett, A. & Fidler, F. Questionable research practices in ecology and evolution. PLoS ONE 13, e0200303 (2018).

    Article  Google Scholar 

  97. John, L. K., Loewenstein, G. & Prelec, D. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23, 524–532 (2012).

    Article  Google Scholar 

  98. Wang, M. Q., Yan, A. F. & Katz, R. V. Researcher requests for inappropriate analysis and reporting: a U.S. survey of consulting biostatisticians. Ann. Intern. Med. 169, 554–558 (2018).

    Article  Google Scholar 

  99. Chavalarias, D., Wallach, J. D., Li, A. H. T. & Ioannidis, J. P. A. Evolution of reporting P values in the biomedical literature, 1990–2015. JAMA 315, 1141–1148 (2016).

    Article  CAS  Google Scholar 

  100. Fanelli, D. Negative results are disappearing from most disciplines and countries. Scientometrics 90, 891–904 (2012).

    Article  Google Scholar 

  101. Sterling, T. D. Publication decisions and their possible effects on inferences drawn from tests of significance–or vice versa. J. Am. Stat. Assoc. 54, 30–34 (1959).

    Google Scholar 

  102. Button, K. S. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013).

    Article  CAS  Google Scholar 

  103. Szucs, D. & Ioannidis, J. P. A. Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLoS Biol. 15, e2000797 (2017).

    Article  Google Scholar 

  104. Camerer, C. F. et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat. Hum. Behav. 2, 637–644 (2018).

    Article  Google Scholar 

  105. Camerer, C. F. et al. Evaluating replicability of laboratory experiments in economics. Science 351, 1433–1436 (2016).

    Article  CAS  Google Scholar 

  106. Open Science Collaboration. Estimating the reproducibility of psychological science. Science 349, aac416 (2015).

    Article  Google Scholar 

  107. Williamson, P. R., Gamble, C., Altman, D. G. & Hutton, J. L. Outcome selection bias in meta-analysis. Stat. Methods Med. Res. https://doi.org/10.1191/0962280205sm415oa (2016).

    Article  Google Scholar 

  108. Carter, E. C., Schönbrodt, F. D., Gervais, W. M. & Hilgard, J. Correcting for bias in psychology: a comparison of meta-analytic methods. Adv. Methods Pract. Psychol. Sci. 2, 115–144 (2019).

    Article  Google Scholar 

  109. Sterne, J. A. C. et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. Brit. Med. J. 343, d4002 (2011).

    Article  Google Scholar 

  110. O’Boyle, E. H., Banks, G. C. & Gonzalez-Mulé, E. The chrysalis effect: how ugly initial results metamorphosize into beautiful articles. J. Manage. 43, 376–399 (2017).

    Google Scholar 

  111. Mazzola, J. J. & Deuling, J. K. Forgetting what we learned as graduate students: HARKing and selective outcome reporting in I–O journal articles. Ind. Organ. Psychol. 6, 279–284 (2013).

    Article  Google Scholar 

  112. Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A. & Rosenthal, R. Selective publication of antidepressant trials and its influence on apparent efficacy. N. Engl. J. Med. 358, 252–260 (2008).

    Article  CAS  Google Scholar 

  113. Cooper, H., DeNeve, K. & Charlton, K. Finding the missing science: the fate of studies submitted for review by a human subjects committee. Psychol. Methods 2, 447–452 (1997).

    Article  Google Scholar 

  114. Hahn, S., Williamson, P. R. & Hutton, J. L. Investigation of within-study selective reporting in clinical research: follow-up of applications submitted to a local research ethics committee. J. Eval. Clin. Pract. 8, 353–359 (2002).

    Article  CAS  Google Scholar 

  115. Franco, A., Malhotra, N. & Simonovits, G. Underreporting in psychology experiments: evidence from a study registry. Soc. Psychol. Personal. Sci. 7, 8–12 (2016).

    Article  Google Scholar 

  116. Franco, A., Malhotra, N. & Simonovits, G. Publication bias in the social sciences: unlocking the file drawer. Science 345, 1502–1505 (2014).

    Article  CAS  Google Scholar 

  117. Becker, J. E., Krumholz, H. M., Ben-Josef, G. & Ross, J. S. Reporting of results in ClinicalTrials.gov and high-impact journals. JAMA 311, 1063–1065 (2014).

    Article  CAS  Google Scholar 

  118. Hartung, D. M. et al. Reporting discrepancies between the clinicaltrials.gov results database and peer-reviewed publications. Ann. Intern. Med. 160, 477–483 (2014).

    Article  Google Scholar 

  119. Fife, D. & Rodgers, J. L. Understanding the exploratory/confirmatory data analysis continuum: moving beyond the ‘replication crisis’. Am. Psychol. 77, 453–466 (2022).

    Article  Google Scholar 

  120. Kimmelman, J., Mogil, J. S. & Dirnagl, U. Distinguishing between exploratory and confirmatory preclinical research will improve translation. PLoS Biol. 12, e1001863 (2014).

    Article  Google Scholar 

  121. Sterne, J. A. C. et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. Brit. Med. J. 366, l4898 (2019).

    Article  Google Scholar 

  122. Turner, L., Boutron, I., Hróbjartsson, A., Altman, D. G. & Moher, D. The evolution of assessing bias in Cochrane systematic reviews of interventions: celebrating methodological contributions of the Cochrane Collaboration. Syst. Rev. 2, 79 (2013).

    Article  Google Scholar 

  123. Page, M. J., McKenzie, J. E. & Higgins, J. P. T. Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review. BMJ Open 8, e019703 (2018).

    Article  Google Scholar 

  124. MacEachern, S. N. & Zandt, T. V. Preregistration of modeling exercises may not be useful. Comput. Brain Behav. 2, 179–182 (2019).

    Article  Google Scholar 

  125. Tukey, J. W. We need both exploratory and confirmatory. Am. Stat. 34, 23–25 (1980).

    Google Scholar 

  126. Wagenmakers, E.-J., Dutilh, G. & Sarafoglou, A. The creativity-verification cycle in psychological science: new methods to combat old idols. Perspect. Psychol. Sci. https://doi.org/10.1177/1745691618771357 (2018).

  127. Reichenbach, H. Experience and Prediction. An Analysis of the Foundations and the Structure of Knowledge (Univ. of Chicago Press, 1938).

  128. Kerr, N. L. HARKing: hypothesizing after the results are known. Personal. Soc. Psychol. Rev. 2, 196–217 (1998).

    Article  CAS  Google Scholar 

  129. Scheel, A. M., Tiokhin, L., Isager, P. M. & Lakens, D. Why hypothesis testers should spend less time testing hypotheses. Perspect. Psychol. Sci. https://doi.org/10.1177/1745691620966795 (2020).

  130. Toth, A. A. et al. Study preregistration: an evaluation of a method for transparent reporting. J. Bus. Psychol. https://doi.org/10.1007/s10869-020-09695-3 (2020).

  131. Sarafoglou, A., Kovacs, M., Bakos, B. E., Wagenmakers, E.-J. & Aczel, B. A survey on how preregistration affects the research workflow: better science but more work. R. Soc. Open Sci. 9, 211997 (2021).

    Article  Google Scholar 

  132. Bosnjak, M. et al. A template for preregistration of quantitative research in psychology: report of the joint psychological societies preregistration task force. Am. Psychol. 77, 602–615 (2022).

    Article  Google Scholar 

  133. Nosek, B. A. & Lakens, D. Registered reports: a method to increase the credibility of published results. Soc. Psychol. 45, 137–141 (2014).

    Article  Google Scholar 

  134. Hardwicke, T. E. et al. Calibrating the scientific ecosystem through meta-research. Annu. Rev. Stat. Appl. 7, 11–37 (2020).

    Article  Google Scholar 

  135. Lilburn, S. D., Little, D. R., Osth, A. F. & Smith, P. L. Cultural problems cannot be solved with technical solutions alone. Comput. Brain Behav. 2, 170–175 (2019).

    Article  Google Scholar 

  136. Navarro, D. Paths in strange spaces: a comment on preregistration. Preprint at https://doi.org/10.31234/osf.io/wxn58 (2020).

  137. Nosek, B. A. et al. Promoting an open research culture. Science 348, 1422–1425 (2015).

    Article  CAS  Google Scholar 

  138. Mathieu, S., Chan, A.-W. & Ravaud, P. Use of trial register information during the peer review process. PLoS ONE 8, e59910 (2013).

    Article  CAS  Google Scholar 

  139. TARG Meta-Research Group & Collaborators. Discrepancy review: a feasibility study of a novel peer review intervention to reduce undisclosed discrepancies between registrations and publications. R. Soc. Open. Sci. 9, 220142 (2022).

  140. Brainerd, C. J. & Reyna, V. F. Replication, registration, and scientific creativity. Perspect. Psychol. Sci. https://doi.org/10.1177/1745691617739421 (2018).

    Article  Google Scholar 

  141. Appelbaum, M. et al. Journal article reporting standards for quantitative research in psychology: the APA Publications and Communications Board task force report. Am. Psychol. 73, 3–25 (2018).

    Article  Google Scholar 

  142. Simera, I., Moher, D., Hoey, J., Schulz, K. F. & Altman, D. G. A catalogue of reporting guidelines for health research. Eur. J. Clin. Invest 40, 35–53 (2010).

    Article  CAS  Google Scholar 

  143. Haven, T. L. et al. Preregistering qualitative research: a Delphi study. Int. J. Qual. Methods https://doi.org/10.1177/1609406920976417 (2020).

    Article  Google Scholar 

  144. Ioannidis, J. P. A. Handling the fragile vase of scientific practices. Addiction 110, 9–10 (2015).

    Article  Google Scholar 

  145. Chambers, C. D., Feredoes, E., Muthukumaraswamy, S. D. & Etchells, P. J. Instead of “playing the game” it is time to change the rules: registered reports at AIMS Neuroscience and beyond. AIMS Neurosci. 1, 4–17 (2014).

    Article  Google Scholar 

  146. Veldkamp, C. L. S., Hartgerink, C. H. J., van Assen, M. A. L. M. & Wicherts, J. M. Who believes in the storybook image of the scientist? Account. Res. 24, 127–151 (2017).

    Article  Google Scholar 

  147. Barber, T. X. Pitfalls in Human Research: Ten Pivotal Points (Pergamon Press, 1976).

  148. Bakker, M., van Dijk, A. & Wicherts, J. M. The rules of the game called psychological science. Perspect. Psychol. Sci. 7, 543–554 (2012).

    Article  Google Scholar 

  149. Rosenthal, R. The file drawer problem and tolerance for null results. Psychol. Bull. 86, 638–641 (1979).

    Article  Google Scholar 

  150. Giner-Sorolla, R. Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspect. Psychol. Sci. 7, 562–571 (2012).

    Article  Google Scholar 

  151. Brewer, W. F. & Chinn, C. A. Scientists’ responses to anomalous data: evidence from psychology, history, and philosophy of science. PSA Proc. Bienn. Meet. Phil. Sci. Assoc. 1994, 304–313 (1994).

    Article  Google Scholar 

  152. Edwards, K. & Smith, E. E. A disconfirmation bias in the evaluation of arguments. J. Personal. Soc. Psychol. 71, 5–24 (1996).

  153. Mynatt, C. R., Doherty, M. E. & Tweney, R. D. Confirmation bias in a simulated research environment: an experimental study of scientific inference. Q. J. Exp. Psychol. https://doi.org/10.1080/00335557743000053 (1977).

  154. Hoekstra, R. & Vazire, S. Aspiring to greater intellectual humility in science. Nat. Hum. Behav. https://doi.org/10.1038/s41562-021-01203-8 (2021).

  155. Nickerson, R. S. Confirmation bias: a ubiquitous phenomenon in many guises. Rev. Gen. Psychol. 2, 175–220 (1998).

    Article  Google Scholar 

  156. Kunda, Z. The case for motivated reasoning. Psychol. Bull. 108, 480–498 (1990).

    Article  CAS  Google Scholar 

  157. Fischhoff, B. Hindsight does not equal foresight: the effect of outcome knowledge on judgment under uncertainty. J. Exp. Psychol. Hum. Percept. Perform. 1, 288–299 (1975).

    Article  Google Scholar 

  158. Slovic, P. & Fischhoff, B. On the psychology of experimental surprises. J. Exp. Psychol. Hum. Percept. Perform. 3, 544–551 (1977).

    Article  Google Scholar 

  159. Gilovich, T., Vallone, R. & Tversky, A. The hot hand in basketball: on the misperception of random sequences. Cogn. Psychol. 17, 295–314 (1985).

    Article  Google Scholar 

  160. Pronin, E. Perception and misperception of bias in human judgment. Trends Cogn. Sci. 11, 37–43 (2007).

    Article  Google Scholar 

  161. Srivastava, S. Sound inference in complicated research: a multi-strategy approach. Preprint at https://doi.org/10.31234/osf.io/bwr48 (2018).

  162. Baldwin, J. R., Pingault, J.-B., Schoeler, T., Sallis, H. M. & Munafò, M. R. Protecting against researcher bias in secondary data analysis: challenges and potential solutions. Eur. J. Epidemiol. https://doi.org/10.1007/s10654-021-00839-0 (2022).

  163. Thabane, L. et al. A tutorial on sensitivity analyses in clinical trials: the what, why, when and how. BMC Med. Res. Methodol. 13, 92 (2013).

    Article  Google Scholar 

  164. Young, C. Model uncertainty and the crisis in science. Socius https://doi.org/10.1177/2378023117737206 (2018).

    Article  Google Scholar 

  165. Haaf, J. M., Hoogeveen, S., Berkhout, S., Gronau, Q. F. & Wagenmakers, E.-J. A Bayesian multiverse analysis of Many Labs 4: quantifying the evidence against mortality salience. Preprint at https://doi.org/10.31234/osf.io/cb9er (2020).

  166. Orben, A. & Przybylski, A. K. The association between adolescent well-being and digital technology use. Nat. Hum. Behav. 3, 173–182 (2019).

    Article  Google Scholar 

  167. Giudice, M. D. & Gangestad, S. A traveler’s guide to the multiverse: promises, pitfalls, and a framework for the evaluation of analytic decisions. Adv. Methods Pract. Psychol. Sci. https://doi.org/10.1177/2515245920954925 (2021).

    Article  Google Scholar 

  168. Klein, J. R. & Roodman, A. Blind analysis in nuclear and particle physics. Annu. Rev. Nucl. Part. Sci. 55, 141–163 (2005).

    Article  CAS  Google Scholar 

  169. Dutilh, G., Sarafoglou, A. & Wagenmakers, E.-J. Flexible yet fair: blinding analyses in experimental psychology. Synthese https://doi.org/10.1007/s11229-019-02456-7 (2019).

    Article  Google Scholar 

  170. MacCoun, R. & Perlmutter, S. Blind analysis: hide results to seek the truth. Nature 526, 187–189 (2015).

    Article  CAS  Google Scholar 

  171. Schorfheide, F. & Wolpin, K. I. On the use of holdout samples for model selection. Am. Econ. Rev. 102, 477–481 (2012).

    Article  Google Scholar 

  172. Lin, W. & Green, D. P. Standard operating procedures: a safety net for pre-analysis plans. Polit. Sci. Polit. 49, 495–500 (2016).

    Article  Google Scholar 

  173. McIntosh, R. D. Exploratory reports: a new article type for Cortex. Cortex 96, A1–A4 (2017).

    Article  Google Scholar 

  174. Button, K. S., Bal, L., Clark, A. & Shipley, T. Preventing the ends from justifying the means: withholding results to address publication bias in peer-review. BMC Psychol. 4, 1–7 (2016).

    Article  Google Scholar 

  175. DeVito, N. J., Bacon, S. & Goldacre, B. Compliance with legal requirement to report clinical trial results on ClinicalTrials.gov: a cohort study. Lancet 395, 361–369 (2020).

    Article  Google Scholar 

  176. Wieschowski, S. et al. Result dissemination from clinical trials conducted at German university medical centers was delayed and incomplete. J. Clin. Epidemiol. 115, 37–45 (2019).

    Article  Google Scholar 

  177. André, Q. Outlier exclusion procedures must be blind to the researcher’s hypothesis. J. Exp. Psychol. Gen. https://doi.org/10.1037/xge0001069 (2021).

  178. Goldacre, B. et al. COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time. Trials 20, 118 (2019).

    Article  Google Scholar 

  179. Cramer, A. O. J. et al. Hidden multiplicity in exploratory multiway ANOVA: prevalence and remedies. Psychon. Bull. Rev. 23, 640–647 (2016).

    Article  Google Scholar 

  180. White, T. K., Reiter, J. P. & Petrin, A. Imputation in U.S. manufacturing data and its implications for productivity dispersion. Rev. Econ. Stat. 100, 502–509 (2018).

    Article  Google Scholar 

  181. Wallach, J. D. et al. Evaluation of evidence of statistical support and corroboration of subgroup claims in randomized clinical trials. JAMA Intern. Med. 177, 554–560 (2017).

    Article  Google Scholar 

  182. Page, M. J. & Higgins, J. P. T. Rethinking the assessment of risk of bias due to selective reporting: a cross-sectional study. Syst. Rev. 5, 108 (2016).

    Article  Google Scholar 

  183. Babbage, C. Reflections on the Decline of Science in England, and on Some of its Causes (Franklin Classics, 1830).

  184. Goldacre, B. Bad Science (Fourth Estate, 2008).

  185. Barnes, E. C. The Paradox of Predictivism (Cambridge Univ. Press, 2008).

  186. Douglas, H. & Magnus, P. D. State of the field: why novel prediction matters. Stud. Hist. Phil. Sci. A 44, 580–589 (2013).

    Article  Google Scholar 

  187. Howson, C. in Scientific Theories (ed. Savage, C. W.) 224–244 (Univ. Minnesota Press, 1990).

  188. Rubin, M. When does HARKing hurt? Identifying when different types of undisclosed post hoc hypothesizing harm scientific progress. Rev. Gen. Psychol. 21, 308–320 (2017).

    Article  Google Scholar 

  189. Hitchcock, C. & Sober, E. Prediction versus accommodation and the risk of overfitting. Br. J. Phil. Sci. 55, 1–34 (2004).

    Article  Google Scholar 

  190. Lipton, P. Testing hypotheses: prediction and prejudice. Science 307, 219–221 (2005).

    Article  CAS  Google Scholar 

  191. Lakens, D. The value of preregistration for psychological science: a conceptual analysis. Jpn Psychol. Rev. 62, 272–280 (2019).

    Google Scholar 

  192. Mayo, D. G. Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars (Cambridge Univ. Press, 2018).

  193. Vanpaemel, W. The really risky registered modeling report: incentivizing strong tests and HONEST modeling in cognitive science. Comput. Brain Behav. 2, 218–222 (2019).

    Article  Google Scholar 

  194. Popper, K. Conjectures and Refutations: The Growth of Scientific Knowledge (Harper & Row, 1963).

  195. Lakatos, I. in Criticism and the Growth of Knowledge (eds Musgrave, A. & Lakatos, I.) 91–196 (Cambridge Univ. Press, 1970).

  196. Maher, P. Prediction, accommodation, and the logic of discovery. PSA Proc. Bienn. Meet. Phil. Sci. Assoc. 1988, 273–285 (1988).

    Article  Google Scholar 

  197. Worrall, J. Prediction and accommodation revisited. Stud. Hist. Phil. Sci. A 45, 54–61 (2014).

    Article  Google Scholar 

  198. Zou, C. X. et al. Registration, results reporting, and publication bias of clinical trials supporting FDA approval of neuropsychiatric drugs before and after FDAAA: a retrospective cohort study. Trials 19, 581 (2018).

    Article  Google Scholar 

  199. Phillips, A. T. et al. Association of the FDA Amendment Act with trial registration, publication, and outcome reporting. Trials 18, 333 (2017).

    Article  Google Scholar 

  200. Dal-Ré, R., Ross, J. S. & Marušić, A. Compliance with prospective trial registration guidance remained low in high-impact journals and has implications for primary end point reporting. J. Clin. Epidemiol. 75, 100–107 (2016).

    Article  Google Scholar 

  201. Abrams, E., Libgober, J. & List, J. A. Research registries: taking stock and looking forward. https://uploads.strikinglycdn.com/files/840dd740-4a8d-4f09-8dbd-e6498f5661c2/January2021Version.pdf (2021).

  202. Bakker, M. et al. Ensuring the quality and specificity of preregistrations. PLoS Biol. 18, e3000937 (2020).

    Article  CAS  Google Scholar 

  203. Claesen, A., Gomes, S., Tuerlinckx, F. & Vanpaemel, W. Comparing dream to reality: an assessment of adherence of the first generation of preregistered studies. R. Soc. Open Sci. 8, 211037 (2019).

    Article  Google Scholar 

  204. TARG Meta-Research Group.et al. Estimating the prevalence of discrepancies between study registrations and publications: a systematic review and meta-analyses. Preprint at https://doi.org/10.1101/2021.07.07.21259868 (2021).

  205. Tan, A. C. et al. Prevalence of trial registration varies by study characteristics and risk of bias. J. Clin. Epidemiol. 113, 64–74 (2019).

    Article  Google Scholar 

  206. Allen, C. & Mehler, D. M. A. Open science challenges, benefits and tips in early career and beyond. PLoS Biol. 17, e3000246 (2019).

    Article  CAS  Google Scholar 

  207. Papageorgiou, S. N., Xavier, G. M., Cobourne, M. T. & Eliades, T. Registered trials report less beneficial treatment effects than unregistered ones: a meta-epidemiological study in orthodontics. J. Clin. Epidemiol. 100, 44–52 (2018).

    Article  Google Scholar 

  208. Kaplan, R. M. & Irvin, V. L. Likelihood of null effects of large NHLBI clinical trials has increased over time. PLoS ONE 10, e0132382 (2015).

    Article  Google Scholar 

  209. Scheel, A. M., Schijen, M. & Lakens, D. An excess of positive results: comparing the standard psychology literature with registered reports. Adv. Methods Pract. Psychol. Sci. https://doi.org/10.1177/25152459211007467 (2020).

  210. Odutayo, A. et al. Association between trial registration and positive study findings: cross sectional study (Epidemiological Study of Randomized Trials—ESORT). Brit. Med. J. 356, j917 (2017).

    Article  Google Scholar 

Download references

Acknowledgements

T.E.H. received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 841188.

Author information

Authors and Affiliations

  1. Department of Psychology, University of Amsterdam, Amsterdam, the Netherlands

    Tom E. Hardwicke & Eric-Jan Wagenmakers

Authors
  1. Tom E. Hardwicke
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eric-Jan Wagenmakers
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Tom E. Hardwicke or Eric-Jan Wagenmakers.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Human Behaviour thanks Ana Marusic, Ulf Toelch, Charles Ebersole and Timothy Errington for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Tutorials, templates, case studies and registries.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hardwicke, T.E., Wagenmakers, EJ. Reducing bias, increasing transparency and calibrating confidence with preregistration. Nat Hum Behav 7, 15–26 (2023). https://doi.org/10.1038/s41562-022-01497-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41562-022-01497-2

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing