Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Bringing rigour to translational medicine

Nature Reviews Neurology volume 10, pages 37–43 (2014)Cite this article

Subjects

Key Points

  • The costs of treating brain diseases are high, because many of these conditions are common, and we lack good treatments for the majority

  • Drug development for these diseases is hampered by a paucity of experimental and reporting rigour and a lack of statistical power, leading to overoptimistic interpretation of the literature

  • Recognition that we have a problem is the first step towards finding a solution

  • We have started to provide transparency in experimental design and rigour of reporting in preclinical publications, and we now need to bring similar quantifiable measures to funding decisions

  • Multicentre, randomized, blinded, powered and appropriately governed preclinical trials conducted by expert consortia offer an opportunity to ensure that only the very best drug candidates reach clinical trial

Abstract

Translational neuroscience is in the doldrums. The stroke research community was among the first to recognize that the motivations inherent in our system of research can cause investigators to take shortcuts, and can introduce bias and reduce generalizability, all of which leads ultimately to the recurrent failure of apparently useful drug candidates in clinical trials. Here, we review the evidence for these problems in stroke research, where they have been most studied, and in other translational research domains, which seem to be bedevilled by the same issues. We argue that better scientific training and simple changes to the way that we fund, assess and publish research findings could reduce wasted investment, speed drug development, and create a healthier research environment. For 'phase III' preclinical studies—that is, those studies that build the final justification for conducting a clinical trial—we argue for a need to apply the same attention to detail, experimental rigour and statistical power in our animal experiments as in the clinical trials themselves.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Numbers of animals used in experimental stroke studies.
Figure 2: Infarct volume variability in different strains of rat after MCAo.

Similar content being viewed by others

References

  1. OECD Health Data 2012. OECD [online], (2012).

  2. Olesen, J. & Leonardi, M. The burden of brain diseases in Europe. Eur. J. Neurol. 10, 471–477 (2003).

    Article  CAS  Google Scholar 

  3. Olesen, J. et al. The economic cost of brain disorders in Europe. Eur. J. Neurol. 19, 155–162 (2012).

    Article  CAS  Google Scholar 

  4. Rafii, M. S. & Aisen, P. S. Recent developments in Alzheimer's disease therapeutics. BMC Med. 7, 7 (2009).

    Article  Google Scholar 

  5. McDonagh, M., Peterson, K., Carson, S., Fu, R. & Thakurta, S. Drug Class Review: Atypical Antipsychotic Drugs: Final Update 3 Report (Oregon Health and Science University, 2010).

    Google Scholar 

  6. National Institute on Drug Abuse. Principles of Drug Addiction Treatment: A Research-Based Guide 19 (NIH Publication No. 12-4180, 2012).

  7. National Institute of Mental Health. Breaking Ground, Breaking Through: The Strategic Plan for Mood Disorders Research 85 (NIH Publication No. 03-5121, 2003).

  8. Wang, Y. R., Alexander, G. C. & Stafford, R. S. Outpatient hypertension treatment, treatment intensification, and control in Western Europe and the United States. Arch. Intern. Med. 167, 141–147 (2007).

    Article  Google Scholar 

  9. Weber, M. A. & Giles, T. D. Inhibiting the renin–angiotensin system to prevent cardiovascular diseases: do we need a more comprehensive strategy? Rev. Cardiovasc. Med. 7, 45–54 (2006).

    PubMed  Google Scholar 

  10. Strong, K., Mathers, C. & Bonita, R. Preventing stroke: saving lives around the world. Lancet Neurol. 6, 182–187 (2007).

    Article  Google Scholar 

  11. Donnan, G. A. et al. How to make better use of thrombolytic therapy in acute ischemic stroke. Nat. Rev. Neurol. 7, 400–409 (2011).

    Article  CAS  Google Scholar 

  12. Lees, K. R. et al. Time to treatment with intravenous alteplase and outcome in stroke: an updated pooled analysis of ECASS, ATLANTIS, NINDS, and EPITHET trials. Lancet 375, 1695–1703 (2010).

    Article  CAS  Google Scholar 

  13. Krumbhaar, E. B. & Krumbhaar, H. D. The blood and bone marrow in Yellow Cross gas (mustard gas) poisoning: changes produced in the bone marrow of fatal cases. J. Med. Res. 40, 497–508.3 (1919).

    CAS  PubMed  PubMed Central  Google Scholar 

  14. Haddad, R. et al. Induction chemotherapy followed by concurrent chemoradiotherapy (sequential chemoradiotherapy) versus concurrent chemoradiotherapy alone in locally advanced head and neck cancer (PARADIGM): a randomised phase 3 trial. Lancet Oncol. 14, 257–264 (2013).

    Article  CAS  Google Scholar 

  15. [No authors listed] Tissue plasminogen activator for acute ischemic stroke. The National Institute of Neurological Disorders and Stroke rt-PA Stroke Study Group. N. Engl. J. Med. 333, 1581–1587 (1995).

  16. Stockwell, B. R. The Quest for the Cure: The Science and Stories Behind the Next Generation of Medicines (Columbia University Press, 2011).

    Book  Google Scholar 

  17. Kling, J. Fresh from the biologic pipeline—2010. Nat. Biotechnol. 29, 197–200 (2011).

    Article  CAS  Google Scholar 

  18. Morgan, S., Grootendorst, P., Lexchin, J., Cunningham, C. & Greyson, D. The cost of drug development: a systematic review. Health Policy 100, 4–17 (2011).

    Article  Google Scholar 

  19. Stroke Therapy Academic Industry Roundtable (STAIR). Recommendations for standards regarding preclinical neuroprotective and restorative drug development. Stroke 30, 2752–2758 (1999).

  20. Stroke Therapy Academic Industry Roundtable II (STAIR-II). Recommendations for clinical trial evaluation of acute stroke therapies. Stroke 32, 1598–1606 (2001).

  21. Fisher, M. & Stroke Therapy Academic Industry Roundtable. Recommendations for advancing development of acute stroke therapies: Stroke Therapy Academic Industry Roundtable 3. Stroke 34, 1539–1546 (2003).

    Article  CAS  Google Scholar 

  22. Fisher, M. et al. Enhancing the development and approval of acute stroke therapies: Stroke Therapy Academic Industry Roundtable. Stroke 36, 1808–1813 (2005).

    Article  Google Scholar 

  23. Fisher, M. et al. Recommendations from the STAIR V meeting on acute stroke trials, technology and outcomes. Stroke 38, 245–248 (2007).

    Article  Google Scholar 

  24. Fisher, M. et al. Update of the Stroke Therapy Academic Industry Roundtable preclinical recommendations. Stroke 40, 2244–2250 (2009).

    Article  Google Scholar 

  25. Saver, J. L. et al. Stroke Therapy Academic Industry Roundtable (STAIR) recommendations for extended window acute stroke therapy trials. Stroke 40, 2594–2600 (2009).

    Article  Google Scholar 

  26. O'Collins, V. E. et al. 1,026 experimental treatments in acute stroke. Ann. Neurol. 59, 467–477 (2006).

    Article  CAS  Google Scholar 

  27. Macleod, M. R., O'Collins, T., Howells, D. W. & Donnan, G. A. Pooling of animal experimental data reveals influence of study design and publication bias. Stroke 35, 1203–1208 (2004).

    Article  Google Scholar 

  28. Macleod, M. R., O'Collins, T., Horky, L. L., Howells, D. W. & Donnan, G. A. Systematic review and meta-analysis of the efficacy of melatonin in experimental stroke. J. Pineal Res. 38, 35–41 (2005).

    Article  CAS  Google Scholar 

  29. Macleod, M. R., O'Collins, T., Horky, L. L., Howells, D. W. & Donnan, G. A. Systematic review and metaanalysis of the efficacy of FK506 in experimental stroke. J. Cereb. Blood Flow Metab. 25, 713–721 (2005).

    Article  CAS  Google Scholar 

  30. Macleod, M. R. et al. Sources of bias in animal models of neurological disease. J. Neurol. Neurosurg. Psychiatry 77, 135–136 (2006).

    Google Scholar 

  31. van der Worp, H. B., Sena, E. S., Donnan, G. A., Howells, D. W. & Macleod, M. R. Hypothermia in animal models of acute ischaemic stroke: a systematic review and meta-analysis. Brain 130, 3063–3074 (2007).

    Article  Google Scholar 

  32. Macleod, M. R. et al. Evidence for the efficacy of NXY-059 in experimental focal cerebral ischaemia is confounded by study quality. Stroke 39, 2824–2829 (2008).

    Article  Google Scholar 

  33. Sena, E. S. et al. Factors affecting the apparent efficacy and safety of tissue plasminogen activator in thrombotic occlusion models of stroke: systematic review and meta-analysis. J. Cereb. Blood Flow Metab. 30, 1905–1913 (2010).

    Article  CAS  Google Scholar 

  34. Jerndal, M. et al. A systematic review and meta-analysis of erythropoietin in experimental stroke. J. Cereb. Blood Flow Metab. 30, 961–968 (2010).

    Article  CAS  Google Scholar 

  35. Janssen, H. et al. An enriched environment improves sensorimotor function post-ischemic stroke. Neurorehabil. Neural Repair 24, 802–813 (2010).

    Article  Google Scholar 

  36. O'Collins, V. E. et al. Preclinical drug evaluation for combination therapy in acute stroke using systematic review, meta-analysis, and subsequent experimental testing. J. Cereb. Blood Flow Metab. 31, 962–975 (2011).

    Article  CAS  Google Scholar 

  37. Steward, O., Popovich, P. G., Dietrich, W. D. & Kleitman, N. Replication and reproducibility in spinal cord injury research. Exp. Neurol. 233, 597–605 (2012).

    Article  Google Scholar 

  38. Scott, S. et al. Design, power, and interpretation of studies in the standard murine model of ALS. Amyotroph. Lateral Scler. 9, 4–15 (2008).

    Article  CAS  Google Scholar 

  39. O'Collins, V., Donnan, G., Macleod, M. & Howells, D. in Animal Models for the Study of Human Disease (ed. Conn, M.) 532–569 (Academic Press, 2013).

    Google Scholar 

  40. Sena, E. S., van der Worp, H. B., Bath, P. M., Howells, D. W. & Macleod, M. R. Publication bias in reports of animal stroke studies leads to major overstatement of efficacy. PLoS Biol. 8, e1000344 (2010).

    Article  Google Scholar 

  41. McCabe, C. et al. Differences in the evolution of the ischemic penumbra in stroke-prone spontaneously hypertensive and Wistar-Kyoto rats. Stroke 40, 3864–3868 (2009).

    Article  Google Scholar 

  42. Letourneur, A., Roussel, S., Toutain, J., Bernaudin, M. & Touzani, O. Impact of genetic and renovascular chronic arterial hypertension on the acute spatiotemporal evolution of the ischemic penumbra: a sequential study with MRI in the rat. J. Cereb. Blood Flow Metab. 31, 504–513 (2011).

    Article  Google Scholar 

  43. Vesterinen, H. M. et al. Improving the translational hit of experimental treatments in multiple sclerosis. Mult. Scler. 16, 1044–1055 (2010).

    Article  Google Scholar 

  44. Ankolekar, S., Rewell, S., Howells, D. W. & Bath, P. M. The influence of stroke risk factors and comorbidities on assessment of stroke therapies in humans and animals. Int. J. Stroke 7, 386–397 (2012).

    Article  Google Scholar 

  45. O'Collins, V., Donnan, G., Macleod, M. & Howells, D. Hypertension and experimental stroke therapies. J. Cereb. Blood Flow Metab. 33, 1141–1147 (2013).

    Article  CAS  Google Scholar 

  46. Howells, D. W. et al. Different strokes for different folks: the rich diversity of animal models of focal cerebral ischemia. J. Cereb. Blood Flow Metab. 30, 1412–1431 (2010).

    Article  CAS  Google Scholar 

  47. Wang-Fischer, Y. (Ed.) Manual of Stroke Models in Rats (CRC Press, 2008).

    Book  Google Scholar 

  48. Stroke Therapy Academic Industry Roundtable (STAIR). Recommendations for standards regarding preclinical neuroprotective and restorative drug development. Stroke 30, 2752–2758 (1999).

  49. Rewell, S. S. et al. Inducing stroke in aged, hypertensive, diabetic rats. J. Cereb. Blood Flow Metab. 30, 729–733 (2010).

    Article  Google Scholar 

  50. Zhu, H. et al. Annexin A2 combined with low-dose tPA improves thrombolytic therapy in a rat model of focal embolic stroke. J. Cereb. Blood Flow Metab. 30, 1137–1146 (2010).

    Article  Google Scholar 

  51. Tsilidis, K. et al. Evaluation of excess significance bias in animal studies of neurological diseases. PLoS Biol. 11, e1001609 (2013).

    Article  CAS  Google Scholar 

  52. Chalmers, I. & Glasziou, P. Avoidable waste in the production and reporting of research evidence. Lancet 374, 86–89 (2009).

    Article  Google Scholar 

  53. Howells, D. W., Sena, E. S., O'Collins, V. & Macleod, M. R. Improving the efficiency of the development of drugs for stroke. Int. J. Stroke 7, 371–377 (2012).

    Article  Google Scholar 

  54. Ioannidis, J. P., Ntzani, E. E., Trikalinos, T. A. & Contopoulos-Ioannidis, D. G. Replication validity of genetic association studies. Nat. Genet. 29, 306–309 (2001).

    Article  CAS  Google Scholar 

  55. Dirnagl, U. & Fisher, M. REPRINT: International, multicenter randomized preclinical trials in translational stroke research: it is time to act. Stroke 43, 1453–1454 (2012).

    Article  Google Scholar 

  56. Dirnagl, U. & Fisher, M. International, multicenter randomized preclinical trials in translational stroke research: it's time to act. J. Cereb. Blood Flow Metab. 32, 933–935 (2012).

    Article  Google Scholar 

  57. Reimer, K. A. et al. Animal models for protecting ischemic myocardium: results of the NHLBI Cooperative Study. Comparison of unconscious and conscious dog models. Circ. Res. 56, 651–665 (1985).

    Article  CAS  Google Scholar 

  58. Bailey, K. R. Detecting fabrication of data in a multicenter collaborative animal study. Control. Clin. Trials 12, 741–752 (1991).

    Article  CAS  Google Scholar 

  59. Bath, P. M., Macleod, M. R. & Green, A. R. Emulating multicentre clinical stroke trials: a new paradigm for studying novel interventions in experimental models of stroke. Int. J. Stroke 4, 471–479 (2009).

    Article  CAS  Google Scholar 

  60. Kochanek, P. M. et al. A novel multicenter preclinical drug screening and biomarker consortium for experimental traumatic brain injury: operation brain trauma therapy. J. Trauma 71 (Suppl.), S15–S24 (2011).

    Article  Google Scholar 

  61. O'Brien, T. J. et al. Proposal for a “phase II” multicenter trial model for preclinical new antiepilepsy therapy development. Epilepsia 54 (Suppl. 4), 70–74 (2013).

    Article  Google Scholar 

  62. Macleod, M. R. et al. Good laboratory practice: preventing introduction of bias at the bench. Stroke 40, e50–e52 (2009).

    Article  Google Scholar 

  63. Macleod, M. R. et al. Reprint: Good laboratory practice: preventing introduction of bias at the bench. J. Cereb. Blood Flow Metab. 29, 221–223 (2009).

    Article  Google Scholar 

  64. Macleod, M. R. et al. Reprint: Good laboratory practice: preventing introduction of bias at the bench. Int. J. Stroke 4, 3–5 (2009).

    Article  Google Scholar 

  65. Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M. & Altman, D. G. Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research. PLoS Biol. 8, e1000412 (2010).

    Article  Google Scholar 

  66. Landis, S. C. et al. A call for transparent reporting to optimize the predictive value of preclinical research. Nature 490, 187–191 (2012).

    Article  CAS  Google Scholar 

  67. Begg, C. et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 276, 637–639 (1996).

    Article  CAS  Google Scholar 

  68. Schulz, K. F., Altman, D. G., Moher, D. & the CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann. Intern. Med. 152, 726–732 (2010).

    Article  Google Scholar 

  69. [No authors listed] Announcement: reducing our irreducibility. Nature 496, 398 (2013).

  70. Kelner, K. Medicine: playing our part. Sci. Transl. Med. 5, 190 (2013).

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Florey Institute of Neuroscience and Mental Health, 245 Burgundy Street, Heidelberg, 3084, Vic, Australia

    David W. Howells

  2. Department of Clinical Neurosciences, University of Edinburgh, Western General Hospital, Edinburgh, EH4 2XU, UK

    Emily S. Sena & Malcolm R. Macleod

Authors
  1. David W. Howells
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Emily S. Sena
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Malcolm R. Macleod
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All three authors researched the data for the article, provided a substantial contribution to discussions of the content, wrote the article, and reviewed and/or edited the manuscript before submission.

Corresponding author

Correspondence to David W. Howells.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

PowerPoint slides

Rights and permissions

Reprints and permissions

About this article

Cite this article

Howells, D., Sena, E. & Macleod, M. Bringing rigour to translational medicine. Nat Rev Neurol 10, 37–43 (2014). https://doi.org/10.1038/nrneurol.2013.232

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nrneurol.2013.232

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing