Living systematic reviews: 3. Statistical methods for updating meta-analyses

doi:10.1016/j.jclinepi.2017.08.008

Journal of Clinical Epidemiology

Volume 91, November 2017, Pages 38-46

https://doi.org/10.1016/j.jclinepi.2017.08.008 Get rights and content

Abstract

A living systematic review (LSR) should keep the review current as new research evidence emerges. Any meta-analyses included in the review will also need updating as new material is identified. If the aim of the review is solely to present the best current evidence standard meta-analysis may be sufficient, provided reviewers are aware that results may change at later updates. If the review is used in a decision-making context, more caution may be needed. When using standard meta-analysis methods, the chance of incorrectly concluding that any updated meta-analysis is statistically significant when there is no effect (the type I error) increases rapidly as more updates are performed. Inaccurate estimation of any heterogeneity across studies may also lead to inappropriate conclusions. This paper considers four methods to avoid some of these statistical problems when updating meta-analyses: two methods, that is, law of the iterated logarithm and the Shuster method control primarily for inflation of type I error and two other methods, that is, trial sequential analysis and sequential meta-analysis control for type I and II errors (failing to detect a genuine effect) and take account of heterogeneity. This paper compares the methods and considers how they could be applied to LSRs.

Section snippets

Background

The key intention of a living systematic review (LSR, see Box 1), which differentiates it from a standard systematic review, is that it will be updated frequently, ideally as soon as any new relevant study is published or identified [1], [2], [3]. Over time the information available to be included may increase, requiring the review to be updated to ensure it is presenting the best available evidence. In many updates, this will require updating one or more of the meta-analyses included in the

Analysis methods for repeated meta-analyses

Updating a meta-analysis has some similarities with interim analyses of clinical trials [9], [10], [11]. Interim analyses are often performed in trials so the trial can be stopped early if there is convincing evidence that the intervention is beneficial or harmful. Methods have been developed to avoid type I and II errors and produce robust conclusions for these trial sequential analyses. These methods have been adapted for the analysis of repeated meta-analyses and more recently for the

Methods for network meta-analysis

A multivariate extension of the alpha-spending boundaries method has been proposed for updating network meta-analysis under the assumption of consistency [25]. Despite the computational complexity in the presence of multiple interventions, the approach is essentially the same as in pairwise meta-analysis. Relative treatment effects between the compared treatments need to be set so as to satisfy the consistency assumptions. Then successively, monitoring boundaries for a predefined level of power

Commentary on the methods

The key properties of each method are outlined in Table 1. Most of the methods for handling repeated meta-analysis are based on an analogy between repeating meta-analysis and sequential analysis of a single clinical trial. While this analogy is generally reasonable, it has some limitations because meta-analyses are based on multiple studies and are not a single controlled trial. Heterogeneity between studies is an obvious key difference. In all methods, if a random-effects meta-analysis is

Conclusions and recommendations

The aim of an LSR is to provide the best available evidence to support decision-making by updating frequently, potentially as soon as a single relevant new study is identified. As with conventional approaches to updating, it is to be expected that the findings of the meta-analyses may change between updates and so reviewers should be suitably cautious when drawing conclusions from a meta-analysis in an LSR, particularly when considering if a result is statistically significant.

The methods

Acknowledgments

The authors would like to thank the members of the Living Systematic Review Network for their comments on drafts of this paper, particularly Philippe Ravaud, Andrew Maas, Kurinichi Gurusamy, Laura Martinez, Joerg Meerpohl and Stefania Mondello.

References (31)

J.H. Elliott et al.
Living systematic reviews: 1. Introduction—the why, what, when and how
J Clin Epidemiol
(2017)
J. Thomas et al.
Living systematic reviews: 2. Combining human and machine effort
J Clin Epidemiol
(2017)
J.P. Ioannidis et al.
Recursive cumulative meta-analysis: a diagnostic for the evolution of total randomized evidence from group and individual patient data
J Clin Epidemiol
(1999)
E.A. Akl et al.
Living systematic reviews: 4. Living guideline recommendations
J Clin Epidemiol
(2017)
G.F. Borm et al.
Updating meta-analyses leads to larger type I errors than publication bias
J Clin Epidemiol
(2009)
J.M. Pogue et al.
Cumulating evidence from randomized trials: utilizing sequential monitoring boundaries for cumulative meta-analysis
Control Clin Trials
(1997)
J. Wetterslev et al.
Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis
J Clin Epidemiol
(2008)
D. Langan et al.
Graphical augmentations to the funnel plot assess the impact of additional evidence on a meta-analysis
J Clin Epidemiol
(2012)
C.S. Berkey et al.
Uncertainty of the time of first significance in random effects cumulative meta-analysis
Control Clin Trials
(1996)
J.H. Elliott et al.
Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap
PLoS Med
(2017)

A.J. Sutton et al.

Evidence-based sample size calculations based upon updated meta-analysis

Stat Med

(2007)

R.M. Turner et al.

The impact of study size on meta-analyses: examination of underpowered studies in Cochrane reviews

PLoS One

(2013)

K.K.G. Lan et al.

Discrete sequential boundaries for clinical-trials

Biometrika

(1983)

J. Whitehead

A unified theory for sequential clinical trials

Stat Med

(1999)

J. Brok et al.

Apparently conclusive meta-analyses may be inconclusive–trial sequential analysis adjustment of random error risk due to repetitive testing of accumulating data in apparently conclusive neonatal meta-analyses

Int J Epidemiol

(2009)

Cited by (100)

Living systematic reviews: aims and standards
2024, Clinical Microbiology and Infection
Methods for living guidelines: early guidance based on practical experience. Paper 1: Introduction
2023, Journal of Clinical Epidemiology
To introduce methods for living guidelines based on practical experiences by the Australian Living Evidence Consortium (ALEC), the National Institute of Health and Care Excellence (NICE), and the Infectious Diseases Society of America (IDSA), with methodological support from the US Grading of Recommendations, Assessment, Development and Evaluations (GRADE) Network.
Members of ALEC, NICE, and the US GRADE Network, convened a working group to share experiences of the methods used to develop living guidelines and outline the key differences between traditional and living guidelines methods.
The guidance includes the following steps: 1) deciding if the guideline is a priority for a living approach, 2) preparing for living guideline development, 3) literature surveillance and frequency of searching, 4) assessment and synthesis of the evidence, 5) publication and dissemination, and 6) transitioning recommendations out of living mode.
This paper introduces methods for living guidelines and provides examples of the similarities and differences in approach across multiple organizations conducting living guidelines. It also introduces a series of papers exploring methods for living guidelines based on our practical experiences, including consumer involvement, selecting and prioritizing questions, search decisions, and methods decisions.
Methods for living guidelines: early guidance based on practical experience. Paper 5: decisions on methods for evidence synthesis and recommendation development for living guidelines
2023, Journal of Clinical Epidemiology
Producing living guidelines requires making important decisions about methods for evidence identification, appraisal, and integration to allow the living mode to function. Clarifying what these decisions are and the trade-offs between options is necessary. This article provides living guideline developers with a framework to enable them to choose the most suitable model for their living guideline topic, question, or context.
We developed this guidance through an iterative process informed by interviews, feedback, and a consensus process with an international group of living guideline developers.
Several key decisions need to be made both before commencing and throughout the continual process of living guideline development and maintenance. These include deciding what approach is taken to the systematic review process; decisions about methods to be applied for the evidence appraisal process, including the use of unpublished data; and selection of “triggers” to incorporate new studies into living guideline recommendations. In each case, there are multiple options and trade-offs.
We identify trade-offs and important decisions to be considered throughout the living guideline development process. The most appropriate, and most sustainable, mode of development and updating will be dependent on the choices made in each of these areas.
From standard systematic reviews to living systematic reviews
2023, Zeitschrift fur Evidenz, Fortbildung und Qualitat im Gesundheitswesen
Systematic reviews (SRs) have become a central tool for evidence-based health care over the last 30 years. The number of SRs being published has increased steadily. However, concerns have been raised regarding the duplication of work, methodological flaws and the currency of many systematic reviews, also in the context of the COVID-19 pandemic. Living systematic reviews (LSRs) offer a new approach to updating systematic reviews, particularly in high-priority research fields that face the challenge of dynamically evolving and sometimes uncertain evidence. Continual updates serve to ensure that LSRs remain current and methodologically rigorous. As a new element of the evidence ecosystem, LSRs can inform living guidelines and recommendations, user-adapted formats, decisions at the patient and system level as well as gaps in primary research.
Systematische Reviews sind seit mehr als 30 Jahren ein zentrales Instrument in der evidenzbasierten Gesundheitsversorgung. Die Anzahl der veröffentlichten Arbeiten ist in dieser Zeit stetig gestiegen. Allerdings gibt es aufgrund von thematischen Überschneidungen und Duplikaten, fraglicher Qualität und mangelnder Aktualität Kritik an vielen systematischen Reviews, auch im Kontext der COVID-19-Pandemie. Sogenannte lebende Evidenzsynthesen sind ein neuerer Ansatz, um systematische Reviews auf dem aktuellen Stand der Forschung zu halten. Sie eignen sich für Fragestellungen, zu denen aktuell viele Studien durchgeführt werden, die hohe Priorität haben und deren Ergebnisse noch in erheblichem Maße unsicher sind. Ziel dieses Ansatzes ist es, durch kontinuierliche Sichtung und Einarbeitung neuer Evidenz Aktualität sowie methodische Qualität zu erreichen. Lebende Evidenzsynthesen können als neues Element eines systematisierten Evidenz-Ökosystems die Grundlage für lebende Leitlinien und Empfehlungen, zielgruppengerechte Informationsangebote, Entscheidungen auf Patienten- und Systemebene sowie für die Identifizierung von Evidenzlücken in der Primärforschung sein.
Editorial
2023, Zeitschrift fur Evidenz, Fortbildung und Qualitat im Gesundheitswesen
Methods for developing and reporting living evidence synthesis
2022, Journal of Clinical Epidemiology
Citation Excerpt :
When ordinary volumes are anticipated [7], authors suggest integrating evidence every 6–12 months or when new evidence has the potential to impact review conclusions [9,11,19]. Simmonds et al. [3] reported the most appropriate statistical approach for integrating new evidence into an updated MA. Authors distinguish between the “continued integration” of new evidence, expected when the LE synthesis is aimed to inform health decision-making, and the “periodic integration” of evidence conducted to provide an up-to-date evidence summary.
Identify currently methodological aspects proposed for planning, conducting, and reporting living evidence (LE) synthesis. Develop a preliminary checklist of key LE synthesis elements.
A survey of methodological articles describing or analyzing methods for the design, conduction, or reporting of LE synthesis.
Twelve methodological articles were identified and analyzed. Key elements were related to: i) definition of LE and characteristics of LE synthesis, ii) methods and tools for the living process, iii) new evidence integration (methods and considerations), iv) updates dissemination and publication, v) revisiting living parameters, and vi) protocol considerations for LE synthesis.
This survey displays basic methodological concepts that can drive the development of LE synthesis and identifies specific aspects with opportunities for development. The potential impact of the LE approach calls for a change in the current evidence synthesis updating processes to more open, collaborative, transparent, and efficient systems. LE approaches also challenge journal editors to shift toward more efficient processes for synthesis update dissemination, which minimizes the risks of reliability of published information.

View all citing articles on Scopus

: Funding: All the authors of this paper were funded to produce this research by a grant from the Cochrane Methods Innovation Fund. The Living Systematic Review Network is supported by funding from Cochrane and the Australian National Health and Medical Research Council (Partnership Project grant APP1114605). Georgia Salanti is supported by a Marie Skłodowska-Curie fellowship (MSCA-IF-703254).

View full text

Series: Living Systematic ReviewLiving systematic reviews: 3. Statistical methods for updating meta-analyses

Abstract

Section snippets

Background

Analysis methods for repeated meta-analyses

Methods for network meta-analysis

Commentary on the methods

Conclusions and recommendations

Acknowledgments

J Clin Epidemiol

J Clin Epidemiol

J Clin Epidemiol

J Clin Epidemiol

J Clin Epidemiol

Control Clin Trials

J Clin Epidemiol

J Clin Epidemiol

Control Clin Trials

Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap

PLoS Med

Evidence-based sample size calculations based upon updated meta-analysis

Stat Med

The impact of study size on meta-analyses: examination of underpowered studies in Cochrane reviews

PLoS One

Discrete sequential boundaries for clinical-trials

Biometrika

A unified theory for sequential clinical trials

Stat Med

Apparently conclusive meta-analyses may be inconclusive–trial sequential analysis adjustment of random error risk due to repetitive testing of accumulating data in apparently conclusive neonatal meta-analyses

Int J Epidemiol

Series: Living Systematic Review
Living systematic reviews: 3. Statistical methods for updating meta-analyses