Profiting from pilot studies: Analysing mortality using Bayesian models with informative priors

Abstract

Pilot studies are often used to help design ecological studies. Ideally the pilot data are incorporated into the full-scale study data, but if the pilot study's results indicate a need for major changes to experimental design, then pooling pilot and full-scale study data is difficult. The default position is to disregard the preliminary data. But ignoring pilot study data after a more comprehensive study has been completed forgoes statistical power or costs more by sampling additional data equivalent to the pilot study's sample size. With Bayesian methods, pilot study data can be used as an informative prior for a model built from the full-scale study dataset. We demonstrate a Bayesian method for recovering information from otherwise unusable pilot study data with a case study on eucalypt seedling mortality. A pilot study of eucalypt tree seedling mortality was conducted in southeastern Australia in 2005. A larger study with a modified design was conducted the following year. The two datasets differed substantially, so they could not easily be combined. Posterior estimates from pilot dataset model parameters were used to inform a model for the second larger dataset. Model checking indicated that incorporating prior information maintained the predictive capacity of the model with respect to the training data. Importantly, adding prior information improved model accuracy in predicting a validation dataset. Adding prior information increased the precision and the effective sample size for estimating the average mortality rate. We recommend that practitioners move away from the default position of discarding pilot study data when they are incompatible with the form of their full-scale studies. More generally, we recommend that ecologists should use informative priors more frequently to reap the benefits of the additional data.

Zusammenfassung

Pilotstudien werden oft genutzt, um das Design von ökologischen Untersuchungen zu bestimmen. Idealerweise werden die Daten aus der Pilotstudie in den Datensatz der Hauptstudie inkorporiert, aber wenn die Pilotstudie die Notwendigkeit größerer Veränderungen an der Versuchsanlage anzeigt, ist das Zusammenführen von Daten aus Pilot- und Hauptstudie schwierig. Die normale Entscheidung ist dann, die vorläufigen Daten nicht zu berücksichtigen. Aber die Ergebnisse aus der Pilotstudie zu ignorieren, nachdem die Hauptstudie abgeschlossen wurde, bedeutet, auf Teststärke zu verzichten, oder der Aufwand steigt durch das Sammeln zusätzlicher Daten, die den Probenumfang der Pilotstudie ausgleichen. Mit Bayesschen Methoden können Daten aus der Pilotstudie als informative a-priori-Verteilung (‘informative prior’) für ein Modell genutzt werden, das aus dem Datensatz der Hauptstudie hergestellt wird. Wir demonstrieren eine Bayessche Methode zur Gewinnung von Information aus anders nicht nutzbaren Pilotstudiendaten anhand einer Fallstudie zur Mortalität von Eukalyptussetzlingen. Eine Pilotstudie zur Mortalität von Eukalyptussetzlingen wurde 2005 in SO-Australien durchgeführt. Eine größere Studie mit einem modifizierten Design wurde im Folgejahr durchgeführt. Die beiden Datensätze unterschieden sich erheblich, so dass sie nicht ohne weiteres zusammengeführt werden konnten. A-posteriori-Schätzwerte der Modellparameter für die Pilotstudie wurden einem Modell für den zweiten, größeren Datensatz zugrundegelegt. Die Überprüfung des Modells zeigte, dass die Hinzunahme einer informativen a-priori-Verteilung die Vorhersagekraft des Models in Bezug auf die Trainingsdaten erhielt. Die Hinzunahme einer informativen a-priori-Verteilung verbesserte die Genauigkeit des Modells für die Vorhersage eines Validierungsdatensatzes und steigerte Genauigkeit und effektive Probengröße für die Bestimmung der durchschnittlichen Mortalitätsrate. Wir empfehlen, dass Praktiker von der Standardpraxis abrücken sollten, Daten aus Pilotstudien zu verwerfen, wenn diese mit ihrer Hauptstudie inkompatibel sind. Ganz allgemein empfehlen wir, dass Ökologen informative a-priori-Verteilungen häufiger einsetzen sollten, um die Vorteile zusätzlicher Daten zu nutzen.

Introduction

The ability of Bayesian analyses to formally incorporate prior information has been little exploited in ecological research despite its distinct appeal in a world where data and resources for research are limited and the impetus for rapid learning is great. Many textbooks on Bayesian methods for ecologists introduce the concept of informative priors in the first few pages (e.g., Kéry 2010; McCarthy 2007), yet researchers typically use very vague priors. In effect, these researchers assert that they have no prior knowledge of model parameters. Informally and formally, researchers use prior information to determine questions, sampling regimes, and model structures, and to interpret results. Whereas use of informative priors is rare, perhaps because it is hard to express prior knowledge as a probability distribution (Clyde 1999) or because informative priors are perceived as overly subjective (Dennis 1996), though subjectivity is not a requirement of priors (Hobbs & Hilborn 2006). Another reason why informative priors are not used is a fear that they could reduce model accuracy. A prior not only affects the precision of estimates, but also the location of the posterior and therefore, potentially the predictive accuracy.

Here we extend the domain of using informative priors in ecological modelling (see Choy, O’Leary, & Mengersen 2009; Dupuis & Joachim 2006; McCarthy & Masters 2005; McCarthy, Citroen, & McCall 2008) with pilot study data. The primary goal of a pilot study is to inform the design of the subsequent full-scale study. Pilot studies are small studies aimed to help reduce important uncertainties, and reveal the sample size needed to detect particular effects. Or, they can reveal major drivers of system variation and help identify at what spatial and temporal scales the variation is propagated, or help refine a set of predictors. However, the data generated by a pilot study may not simply inform the design of the full-scale study, but may also help address the fundamental research question. Many ecologists’ default position is to disregard the preliminary data, a stance recommended by texts on data collection and analysis (e.g. Green 1979). However, treating the results of a pilot study as an informative prior using Bayesian methods provides a formal and transparent way to combine two otherwise incompatible data sources, improving the cost-effectiveness of the research.

We illustrate the use of Bayesian informative priors to recover the inferential and predictive power of otherwise unusable pilot study data with a case study on eucalypt tree seedling mortality. Mortality rate is a key demographic parameter to be estimated. Understanding how and why it varies, is key to describing and learning about population dynamics of all species (Zens & Peart 2003). However, the mortality events from which rates are calculated are often rare in absolute terms, and in many systems may also be episodic. Combined, these issues make mortality rates difficult to characterise, so large datasets that include many individuals and span large spatial and temporal ranges are often required.

Including prior information in a Bayesian model will increase the precision of relevant parameter estimates and posterior predictive distributions (McCarthy 2007). The effect on model accuracy is harder to define though no less important and can only be done via model validation. Effects of priors on accuracy have received limited attention in ecology. If a prior disagrees with the likelihood then the model will be less accurate with respect to the training data than it would without the prior information. But the cost to predictive accuracy specific to the training data may be outweighed by the increased generality of the model, as it can include information from a wider range of sources than the training data alone. In this paper, we demonstrate how to treat the knowledge learned during a pilot study on eucalypt seedling mortality as an informative Bayesian prior. We express this source of prior knowledge as the degree to which the full-scale study budget would need to increase to recover the loss of information by not including the prior. The particular demonstration of using informative priors we provide here highlights their general benefit.