Models developed by three techniques did not achieve acceptable prediction of binary trauma outcomes

Abstract

Background and Objectives

To develop prediction models for outcomes following trauma that met prespecified performance criteria. To compare three methods of developing prediction models: logistic regression, classification trees, and artificial neural networks.

Methods

Models were developed using a 1996–2001 dataset from a major trauma center in Victoria, Australia. Developed models were subjected to external validation using the first year of data collection, 2001–2002, from a state-wide trauma registry for Victoria. Different authors developed models for each method. All authors were blinded to the validation dataset when developing models.

Results

Prediction models were developed for an intensive care unit stay following trauma (prevalence 23%) using information collected at the scene of the injury. None of the three methods gave a model that satisfied the performance criteria of sensitivity >80%, positive predictive value >50% in the validation dataset. Prediction models were also developed for death (prevalence 2.9%) using hospital-collected information. The performance criteria of sensitivity >95%, specificity >20% in the validation dataset were not satisfied by any model.

Conclusion

No statistical method of model development was optimal. Prespecified performance criteria provide useful guides to interpreting the performance of developed models.

Introduction

Interest in developing prognostic models for binary outcomes is widespread, and guidelines exist for their creation [1] and validation [2] as applied to health outcomes. A key element of model creation is the use of a large database containing variables that will be available if the new model is applied in routine practice. This creation step involves trade off between (1) complexity of model; the more complex the model, the more attuned it becomes to observed features of the database from which it is developed, and (2) transportability of the model, that is, how well it performs with different databases and when it is used in practice. Validation involves two critical elements. First, “external validation”: the performance of the newly created model must be tested on a second dataset that was not a part of model creation but that is representative of the same population [3], [4]. Second, the model must be validated in the context of prespecified utility scores for correct predictions and/or indicators that represent acceptable model performance when used in routine practice [2]. These indicators represent minimum performance thresholds for the model to be of practical use when applied.

Numerous statistical methods have been applied to create prognostic models for binary outcomes. Many comparisons between different methods exist [4], [5], [6], [7], [8], [9], [10], [11]; however, there is no consensus as to an optimal method. It is therefore prudent to explore different methods.

We present a case study of prognostic model development for binary outcomes that involves separate model-creation and validation datasets and prespecified performance indicators. Three statistical methods were employed for the creation of a prognostic model; logistic regression, classification trees, and neural networks. Logistic regression has become a mainstay of medical research; the latter two techniques are commonly used with data involving interactions and nonlinear relationships [12] and are long established [13], [14]. Each method was applied by a different investigator, blinded to the validation dataset when creating their models.