Systematic Reviews: Challenges for Their Justification, Related Comprehensive Searches, and Implications

Scientific research in health sciences aims to enhance knowledge about diseases and attempt to find better and newer diagnostic and therapeutic tools. With economic growth worldwide, the number of published scientific papers is exponentially increasing.1 For the practicing clinician, it is important to be able to analyze and comprehend healthcare evidence in the shortest possible time. With the growth of scientific literature, the concept of EBM has garnered increasing attention in the past three decades. Defined as an attempt to most appropriately utilize the currently available and best evidence toward making individual patient care decisions,2 EBM has witnessed growth in parallel with the voluminous amounts of scientific literature being published today. Broadly speaking, four major steps are involved in EBM. It starts with the identification of a clinical problem, followed by an appropriate literature search to solve it. Synthesis of thereby-derived search results culminate in the utilization of these search results to identify the most appropriate solution to the aforementioned clinical problem.3

Review articles are attempts to synthesize information in a particular area of interest and present it to the reader (busy clinician), who can then utilize processed evidence-based reports to make informed choices relevant to their patient. Reviews can be narrative or systematic. Narrative reviews are broader in scope, with a flexible structure, allowing the authors to be more exploratory in their approach toward a particular problem.4 Often, the narrative review may reflect certain points based on the authors experience, especially in niches on which little or no evidence exist, for example in the treatment of rare diseases. Some journals may not consider it mandatory to have a search strategy, detailing how the literature was searched (keywords), time limits of searches, and bibliographic databases accessed. However, it is highly desirable to include a search strategy even in narrative reviews as it provides the readers the proof of a methodical approach, as well as a reference point in time from which another future narrative review may focus on newer literature, thereby limiting redundancy. Systematic reviews, on the other hand, are directed toward answering a specific question and focusing on a few citable points.

Systematic reviews have a pre-defined structure,5 based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.6 This starts with the development of a protocol detailing the plan of the review, which is preferably pre-published. Thereafter, a systematic literature review should be conducted, which must be extensive, based on strict inclusion and exclusion criteria, covering multiple databases and preferably, also attempt to look at unpublished literature by browsing abstracts of major conferences and ongoing clinical trials. The generation of a PRISMA flowchart to summarize the results of such a literature search is mandatory. This is followed by a qualitative synthesis of the selected studies. Should it be feasible to provide a quantitative estimate of the same by pooling data from different studies, this forms the next stage — meta-analysis. It is important to understand that performing a meta-analysis is not mandatory, and should only be done when appropriate, i.e., when data from different studies have looked at comparable outcomes and there is absence of significant heterogeneity. Should meta-analysis be undertaken for significantly heterogenous data, it is imperative to use the random effects model as opposed to the fixed effects model. In addition, one should also undertake analysis of the quality of included studies, utilizing tools such as the Newcastle-Ottawa scale for assessing observational studies (cohort and case-control studies)7 or the Grading of Recommendations, Assessment, Development and Evaluation profiler (GRADEpro)8 for interventional studies. Assessment of publication bias to minimize the skewing of results of a systematic review by missing unpublished negative studies is advisable, but generally only when ten or more studies are available.5

The PRISMA guidelines mention the requirement to present the complete search strategy for at least one full electronic database.9 However, the requirement to conduct a comprehensive search across databases, and through conference abstracts and clinical trial registries, is not adequately emphasized. In this regard, recommendations emphasizing the need to conduct thorough searches through multiple bibliographic databases need to be considered.4 Authors may consider combining these guidelines4 with PRISMA to enhance the comprehensiveness of their searches, as rightly noted in a recently published exemplary systematic review.10 Table 1 compares narrative and systematic reviews, based on previously published literature11, 12, 13 as well as the authors' own experience.

Table 1
Features of narrative and systematic reviews

• Click for larger image
• Click for full table
• Download as Excel file

Another question is whether the same topic can be explored in narrative and systematic reviews. It is quite possible, but systematic reviews are focused on highly specific questions with a limited number of references, as opposed to narrative reviews, which are broader in scope with several citable points and reference counts exceeding one hundred.11 If such a question arises whether to conduct a systematic or narrative review, we opine that it is better to perform a systematic one. Moreover, there is evidence suggesting that systematic reviews are likely to garner more citations than narrative reviews.14, 15, 16

Systematic reviews are of two major types: those on interventions and overviews of observational data. In the latter group, systematic reviews and meta-analyses of genetic data constitute a significant proportion.

Systematic reviews are possible to perform when there are multiple related primary studies, and there is a need to numerically summate available research results across articles to generate the highest possible evidence.17 Evidence derived from systematic reviews and meta-analyses of randomized controlled trials is considered the most definitive in guiding clinical decisions and healthcare policies. Such meta-analyses often form the cornerstone of practice guidelines. Moreover, such papers are likely to garner greater number of citations, hence, are of special interest to high-impact journals. As an example, a recent publication on highly-cited systematic reviews in medicine included a rheumatology paper with over 800 total citations (141 citations annually on average).18

The Cochrane Collaboration is a multinational collaborative network of experts in methodology of performing systematic reviews.19 Systematic reviews are those on interventional studies undertaken by the Cochrane Collaboration and published in the Cochrane Database of Systematic Reviews (CDSR). The CDSR is a registry and a high-impact peer reviewed journal at the same time. Various different Cochrane groups look at different areas of medicine.20 Reviews published in the CDSR initially have a published protocol, followed by a full systematic review, strictly adhering to the published protocol. Both protocols and full reviews are reviewed by experts and patients. Cochrane reviews are regularly updated to reflect advances in a particular area.19 Over the years, findings described in Cochrane reviews have driven healthcare policies in numerous guidelines.21

Cochrane reviews are well cited. For example, reviews in musculoskeletal medicine rank sixth in terms of citations among other subject-related Cochrane reviews.22

A systematic review should have a pre-published protocol. To understand the need for this, one needs to understand the concept of redundancy. The adjective “redundant” is defined by the Oxford dictionary as “Not or no longer needed or useful; superfluous.” A review published in an area where there already exists previously or recently published article is redundant in terms of contribution toward the scientific literature. Simultaneously performed reviews are likely to reach the same conclusion. Multiple reviews on a single topic may also confuse readers. Herein lies the importance of pre-registering such reviews, so that prospective authors can avoid wasting their resources by undertaking redundant studies. The scope of the problem of redundancy in published systematic reviews and meta-analyses has been addressed in a recent article.23

Titles of already registered reviews under the Cochrane Collaboration are available for everyone to view, in order to avoid duplicate efforts. However, conducting and publishing a systematic review under the umbrella of the Cochrane Collaboration, while being highly prestigious, involves a strictly modulated process. Should authors intend to embark on a review not yet registered in the Cochrane Library, they can register their review at the PROSPERO,24 which also reflects titles registered with Cochrane. PROSPERO registers only those reviews which have at least a single outcome of direct relevance to the care of a patient.24 Most journals nowadays expect systematic reviews to be pre-registered at PROSPERO to avoid potential redundancy.

Regarding reviews not directly dealing with health outcomes, such as systematic reviews on genetic data, such as polymorphisms and susceptibility towards disease, it may not be possible to register them under PROSPERO. There exists an unmet need to develop a register to pre-register such reviews to avoid redundancy in this sort of reviews.

To assess the current state of published systematic reviews in the field of rheumatology, we conducted a PubMed search utilizing Medical Subject Headings (MeSH) terms “Osteoarthritis,” “Arthritis, Rheumatoid,” “Lupus Erythematosus, Systemic,” (“Scleroderma, Systemic” OR “Scleroderma, Diffuse”), “Systemic Vasculitis,” “Spondyloarthropathies,” “Fibromyalgia,” “Myositis,” “Arthritis, Juvenile,” restricting the results to the article type “Systematic reviews.” We compared the numbers of articles published up to the end of 2007 to those published from 2008 until present (November 21, 2017). As shown in Fig. 1, the number of systematic reviews in each area has increased exponentially during the last ten years.

Fig. 1
Number of published systematic reviews in different rheumatic diseases — comparison before and after 2007. PubMed search (November 21, 2017) limited to article type “Systematic reviews” for each disease done separately.

• Click for larger image
• Download as PowerPoint slide

Identification of major problems in the published literature can result in retractions of published papers.25 To assess the extent of this problem in systematic reviews, we searched the PubMed database on November 21, 2017 limiting our search to publication type “Retraction of publication” and article type “Systematic reviews.” We identified 88 retracted reviews. Of these, one was a duplicate retraction notice, another one an accidental republication by the publisher. One article had been prematurely published accidentally by the publisher whereas in another one, it could not be clearly ascertained from the search or by visiting the journal page whether the paper had been retracted or not. Of the remaining 84 retractions, all except two had been retracted from 2009 onward, suggesting that this was an emerging problem. We analyzed them for the country of publication (address of corresponding author), and further compared them to identify what percentage of similar articles published from that country did this figure constitute. represents the results.

Fig. 2
Proportion of retracted systematic reviews out of total number of systematic reviews from these countries (× 10⁻⁵). Numbers above each bar represent the absolute number of such retracted systematic reviews. PubMed search (November 21, 2017) limited to article type “Systematic reviews” for each country done separately.

• Click for larger image
• Download as PowerPoint slide

Further, we did a Scopus search utilizing the search terms “systematic reviews” and “retraction” on December 1, 2017. We identified one more retracted systematic review. For the total 85 systematic reviews identified through the aforementioned PubMed and Scopus searches, we looked at the reasons for type of systematic reviews retracted, classifying them into reviews on interventions and observational data (separately for genetic data) (Fig. 3). Reviews of observational data accounted for 49 out of 85 retractions. A detailed analysis of six retracted systematic reviews in the field of rheumatology (Table 2)26, 27, 28, 29, 30, 31 suggested that methodological errors were responsible for 2 and compromised, possibly fabricated peer review process was the cause for retraction in 3. In the remaining 1, it was identified after publication that certain studies had been missed, therefore, the review was retracted and a corrected version republished. Again, a majority of these papers were retracted in the past three years.

Fig. 3
Number of retracted systematic reviews based on type of article. PubMed search (November 21, 2017) limited to article type “Systematic reviews.” Scopus search (December 1, 2017) using “Systematic reviews” AND “Retraction.”

SR = systematic reviews.

• Click for larger image
• Download as PowerPoint slide

Table 2
Retracted systematic reviews in rheumatology

• Click for larger image
• Click for full table
• Download as Excel file

In this context, it is important to highlight a recent problem that has become apparent in scientific publishing. Numerous papers were retracted from 2014 onward by major publishers since they detected fraudulent reviews had been provided for submitted articles by authors (utilizing false reviewer email accounts, thereby enabling them to control the peer review of their own articles) or external agencies in collusion with authors.32 As evident in Table 2, three of the retractions of systematic reviews in rheumatology were probably due to this very same reason. In this context, it is necessary to bring up a concern among editors worldwide, regarding systematic reviews and meta-analyses commissioned by commercial editing agencies. It must be emphasized that such practices are of doubtful ethical integrity. Should authors have indulged in such practices, they should transparently declare this while submitting their review article. In the absence of such a declaration, it is possible to reject or retract such a work.

Considering the explosion in the number of meta-analyses, especially on genetic data,33 it is also imperative for prospective authors to understand that proper methodology and lack of redundancy should remain major considerations.

The first step to avoid redundancy is to search the CDSR and PROSPERO whether a similar study has been already registered. This helps avoid duplication of author efforts.

Peer reviewers should be familiar with methodology of systematic reviews and meta-analyses. They should cross-check related databases and PROSPERO to judge whether the review is redundant or not. In case of pre-registered reviews, it is imperative to compare the methodology prescribed in the pre-published protocol and check whether it has been followed or not. The presence of an appropriate PRISMA flow diagram is essential, as well as employment of a comprehensive search strategy. Appropriate assessment of heterogeneity in included studies should be looked at, proceeding to the stage of a meta-analysis only when the date permits one to. The quality assessment and risk of bias analysis should be asked for in a systematic review if not already included.

Poorly conducted systematic reviews and misleading meta-analyses are a significant concern for editors worldwide, especially in start-up journals.34 Since the number of citations to articles from a journal indirectly reflect the quality of a journal,16 it may be tempting for eager editors to favorably consider a systematic review for publication. Editors must be careful, considering the aforementioned recent instances of peer review compromise by fraudulent methods, which led to retraction of numerous systematic reviews (Table 2).28, 29, 30, 32 Based on limited evidence (Fig. 2), it may be prudent to view systematic reviews from certain geographic regions with greater caution. Our literature search suggested that retracted systematic reviews are more likely to be those on observational data, hence, such articles, especially those looking at genetic data, should be more carefully looked at. Also, journal policies may consider making it mandatory for authors to declare beforehand if services from commercial editing agencies have been sought for in writing systematic reviews. Such papers might be subjected to more stringent editorial policies.

Editors should engage in increasing awareness about inappropriate practices in systematic reviews by arranging workshops or online training courses. We have identified certain points which may help guide editors to flag potentially fraudulent systematic reviews (Table 3).

Table 3
Pointers for identifying fraudulent/erroneous systematic reviews

• Click for larger image
• Click for full table
• Download as Excel file