Introduction

The findings of clinical research inform clinicians, guideline committee members and policy-makers on how to provide the best care of patients at the lowest possible cost. It is of utmost importance that such research findings are trustworthy. If not, we may harm patients and waste resources by the use of interventions with an unfavourable balance between risk and benefit. The risk of harm is real as indicated in a recent systematic review of all interventions shown to affect mortality in critical care trials [1]. Among the 15 interventions that affected mortality, eight increased mortality. Many of these harmful interventions were in clinical use at the time of testing, including tight glycaemic control, hydroxyethyl starch and high-frequency oscillatory ventilation [2,3,4]. Faced with the need to identify and promote interventions that may help patients and with the similar need to identify and avoid interventions that may harm patients, intensivists have used a variety of approaches. One such approach has been based on the use of databases or registries. Such an approach, however, carries important risks. In particular, the reliability of findings from such investigations is open to challenge.

Why are estimates of intervention effects unreliable from databases and registries?

Empiric evidence suggests that the results of observational studies based on databases or registries are unreliable [5, 6]. Overall, the observational data may overestimate the benefits of interventions as compared to results of randomised clinical trials (RCTs) [5, 6], but harm may also be overestimated as observed in analyses of observational studies vs. placebo-controlled RCTs on digoxin in patients with heart failure [5]. The reasons for the biased results are multiple and often include the design of the databases/registries, the quality of the data, confounding by indication, and the design of research studies based on such databases (Table 1). Moreover, if the database being assessed is local (single centre), as is the case for small studies, it may be more detailed, but it will also lack external validity. On the other hand, if a large dataset is being used, data “missingness” will be common and quality controls will often be limited.

Table 1 Problems with observational studies from databases or registries vs. randomised trials in critical care

Can the problems be fixed?

No, probably not. In the digoxin case, the overestimation of harm was largest in unadjusted analyses (76% relative risk increase in mortality with the use of digoxin) [5], but in adjusted analyses including propensity scoring, the overestimation remained. Thus, the propensity matched analysis overestimated the relative risk of harm from digoxin by 18% [5]. Moreover, inferences about critical care interventions from studies of databases and registries may be particularly prone to bias because of multiple, time-dependent exposures and many confounding co-interventions in critical care. Competing risk is another major challenge as many critical ill patients experience adverse events, which may mimic harmful effects of interventions, e.g., bleeding and kidney and hepatic injury.

In addition to the above problems, database-driven research is often asked to do more than it can do. Like all observational research, however, it can never deliver inferences of causality. Unfortunately, associations are often presented with a language that suggests such causality. For example, expressions like biomarker or intervention X “has an impact on” or “has an independent effect on” or “modifies” or “changes” a given outcome are often used. Such expressions subtly suggest causality where none can be inferred to exist. Moreover, phrases like “after adjustment for confounders” or “independent association” or “propensity score adjusted impact” are often used. However, associations can only be adjusted for given available variables; many variables that influence outcome (patient frailty, doctor’s judgement, the exact situation at the time of the intervention etc.) are actually not available. Thus, all adjustments are flawed by the very nature of database characteristics. Such associations as may be found, therefore, are only hypothesis generating and should never lead to practice change. It would be good for critical care clinicians to reflect that, so far, no studies of associations have led to the identification of an intervention, which was then subsequently confirmed to affect outcome in a large multicentre RCT.

Have databases and registries no place in critical care research?

High quality observational data are the main source for information about incidences, complications, outcomes, clinical practice and resource use and for the development of diagnostic criteria and prognostic scoring systems. Longitudinal databases can identify secular trends in the incidence and outcome of conditions of interest such as sepsis, trauma, acute kidney injury, and ARDS. There are many good and recent examples in critical care [7,8,9,10], which provide useful information for policy-making and resource allocations. Exploratory analyses of the associations of specific interventions and outcome and estimates regarding recruitment rates and trial size have value for the planning of RCTs [11]. Finally, databases can be used to identify illness severity adjusted potential performance outliers that may require investigation [12].

However, research from databases needs to evolve in term of analysis protocols. In particular, the pre-publication of statistical analysis plans is necessary to give the reader confidence that the analysis was truly driven by the desire to test a hypothesis, rather than by a data-mining exercise, with all the inherent biases of such an approach. Modern RCTs now demand such pre-publication of protocols [13, 14] and statistical analysis plans [15, 16]. Similar demands for observational database-driven investigations should be made by journals to help attenuate the flaws of database research [17].

Perspectives

Critical care has evolved tremendously over the last half-century. Many of the interventions that have formed and form critical care were introduced into clinical practice without evidence from RCTs and systematic reviews with overall low risk of bias. The lesson learned in recent years is that the use of interventions based on the results of biased research, including those of observational studies of databases or registries may at best be a waste of time and money, but we may also harm patients [1]. Thus, it is imperative for the critical care community to conduct low risk of bias RCTs to answer clinical questions on the balance between the benefits and harms of common interventions to provide the best and most cost-effective care for critically ill patients.