Elsevier

Environmental Modelling & Software

Volume 100, February 2018, Pages 146-158
Environmental Modelling & Software

Minimising biases in expert elicitations to inform environmental management: Case studies from environmental flows in Australia

https://doi.org/10.1016/j.envsoft.2017.11.020Get rights and content

Highlights

  • We present a framework for expert elicitation that minimizes expert biases.

  • It brings together three hitherto separate methods to maximize effectiveness.

  • It results in a fully functional Bayesian network model to inform decision making.

  • We illustrate the framework using examples from environmental water management.

  • The model can be updated with empirical data within an adaptive management cycle.

Abstract

Environmental managers often do not have sufficient empirical data to inform decisions, and instead must rely on expert predictions. However, the informal methods often used to gather expert opinions are prone to cognitive and motivational biases. We developed a structured elicitation protocol, where opinions are directly incorporated into Bayesian Network (BBN) models. The 4-stage protocol includes approaches to minimise biases during pre-elicitation, workshop facilitation and output analysis; and results in a fully functional BBN model. We illustrate our protocol using examples from environmental flow management in Australia, presenting models of vegetation responses to changes in riverine flow regimes. The reliance on expert opinion and the contested nature of many environmental management decisions mean that our structured elicitation protocol is potentially of great value for developing robust environmental recommendations. This method also lends itself to effective adaptive management, because the expert-populated ecological response models can be readily updated with field data.

Introduction

Decision making informed by expert opinion is common in environmental management and can form a basis for urgent management and policy decisions when stakes are too high to postpone such choices (Krueger et al., 2012, Martin et al., 2012). In complex systems, when formal theories and/or measured data may be scarce, expert opinion can help to assess whether information is consistent and where evidence may be lacking (Martin et al., 2012). Informal techniques that are implicit, unstructured and undocumented—such as ‘roundtable discussions’—are commonly used for extracting expert knowledge (Fidler et al., 2012, McBride and Burgman, 2012). Whilst discussion itself is not necessarily problematic, the knowledge provided by such unstructured group expert opinion is usually based on subjective judgements prone to cognitive and motivational biases (Fidler et al., 2012, Garthwaite et al., 2005, O'Hagan et al., 2006, O'Hagan and Oakley, 2004, Speirs-Bridge et al., 2010, Tversky and Kahneman, 1975). Cognitive biases occur as a result of a failure to adequately process, aggregate or integrate relevant information due to limitations on human processing ability (McBride and Burgman, 2012, Wilson, 1994). Overconfidence biases, for example, undermine expert judgments by underestimating uncertainty (Soll and Klayman, 2004). Other examples of cognitive biases include the availability bias, where familiarity with one particular driving factor may lead an expert to believe it to be more important than it actually is (Kynn, 2008); and anchoring biases, where an initial value is used to calculate another value by adjusting it up or down (Jacowitz and Kahneman, 1995, Speirs-Bridge et al., 2010). Motivational biases are conscious or subconscious adjustments in an expert's responses that depend on their particular context, personal beliefs and experiences, and from what the expert stands to gain or lose personally form a decision (Garthwaite et al., 2005).

Biases are inherent in heuristic processing. The utility of expert knowledge is therefore dependent on the rigour with which it is elicited (Martin et al., 2012). The choice of an expert elicitation method needs to account for, and minimise the risk of, bias affecting expert judgements and resulting elicited data (Speirs-Bridge et al., 2010). Formal procedures to elicit expert knowledge have been designed to account for such biases in order to increase credibility, repeatability and transparency (Hanea et al., 2017, McBride and Burgman, 2012). Structured elicitation often employs a framework combining foundational elements of decision theory and mathematics with procedures for minimising biases (Keeney and von Winterfeldt, 1991, O'Hagan et al., 2006). Recently, the IDEA (Investigate Discuss Estimate Aggregate) structured protocol by Hanea et al. (2017) has combined different elements from established approaches to optimise expert knowledge elicitation. The success of expert elicitation depends on facilitation and the way expert judgements are collected and compiled. Therefore, only well managed, systematic elicitation protocols can return high-quality, transparent and repeatable predictions from experts (Cook et al., 2012, Hanea et al., 2017, Knol et al., 2010, Martin et al., 2012, Runge et al., 2011).

One area of environmental management where expert opinion is widely used, is the provision of environmental flows (water released from storage solely to benefit the environment; Horne et al., 2017b) to restore regulated river systems (Stewardson and Webb, 2010). Restoring more natural flow regimes is a key issue around the world (Poff et al., 1997), with substantial investments from governments in an attempt to manage degraded river systems (Acreman and Ferguson, 2010, Skinner and Langford, 2013). Effective restoration and management of these systems rely on an understanding of the relationships between the stressors (changes in flow regimes) and ecological responses (Poff and Zimmerman, 2010, Webb et al., 2013). However, while general principles of the ecological effects of changes in flow regime are well accepted (Poff and Zimmerman, 2010), there are few quantitative predictions about how different components of degraded ecosystems will respond to flow restoration (Arthington and Pusey, 2003, Souchon et al., 2008).

Despite this paucity of quantified relationships, flow management decisions must be made. Therefore, of the many environmental flow assessment techniques that have been developed (Tharme, 2003), those that make direct predictions of ecological effects have relied to a great degree on expert opinion (Stewardson and Webb, 2010). In Australia, expert opinion is a major determinant of how the approximately $15 billion investment in environmental water under the Basin Plan and other initiatives will be used (Commonwealth of Australia, 2014). It is therefore surprising that systematic approaches to elicit expert knowledge are less common than informal methods (McBride and Burgman, 2012). The reliance on expert opinion and the contested nature of environmental flows means that structured elicitation should be of great potential value for developing environmental flow recommendations (Webb et al., 2015).

Bayesian Belief Networks (BBNs; Pearl, 2000) is a modelling technique that is commonly used in natural resource management applications (McCann et al., 2006), and has also been used in environmental flows applications (Chan et al., 2012, Horne et al., 2017c, Shenton et al., 2011, Shenton et al., 2014). One of the often-cited advantages of BBNs is that they can incorporate multiple data types, including expert knowledge (Horne et al., 2017a). However, this advantage also leaves them open to being populated by poorly elicited expert opinion, affected by bias and overconfidence. The quantitative outputs from these models may give a false sense of security in the results if they are based on poor-quality expert-derived data.

This paper presents a 4-stage formal expert elicitation protocol, where opinions are directly incorporated into BBN models, providing improved rigour in the relationships and subsequent predictions. The 4 stages include pre-elicitation, workshop facilitation, output analysis and BBN model building. The first 3 stages are based on the IDEA framework by Hanea et al. (2017), combining different established approaches to minimise biases of elicited opinions. The final stage consists of discretising the elicited probability distributions in order to populate the conditional probability tables in a BBN model. This model becomes an ideal vehicle for later updating with empirical data.

We first present the general principles underpinning the protocol, providing a justification of the approaches used at each stage (section 2). We then illustrate application of the protocol using two case studies that elicited quantitative predictions of ecological responses to changes in riverine flow regimes and other environmental factors under environmental flow recommendations (section 3). The two case studies elicited expert predictions of expected changes in (i) terrestrial vegetation encroachment into river channels, or undesirable vegetation cover (hereafter “Encroachment”) and (ii) abundance of native riparian species on river banks, or desirable vegetation cover (hereafter “Native banks”). The paper closes by assessing strengths and weaknesses of the protocol, especially as experienced through the case studies.

Section snippets

Expert elicitation protocol: general principles

Our approach to minimising cognitive and motivational biases include (i) using a question protocol to reduce overconfidence and availability biases during pre-elicitation; (ii) requiring experts to answer questions independently and in isolation to minimise motivational biases, but also allowing multiple rounds of judgement to ensure transparency and expert comprehension during workshop facilitation; (iii) using mathematical accumulation of opinion and interpolation as analysis tools to avoid

Case studies on vegetation responses to environmental flows

We illustrate the above protocol using the example of two elicitation projects on Encroachment and Native banks to quantitatively predict the response of in-channel and riparian vegetation to inundation by environmental flows under a number of different scenarios.

Discussion

Environmental management requires quantitative predictions of ecological responses to management actions. In the absence of field data, the use of a formalised, expert elicitation protocol can produce transparent, methodologically rigorous and defensible estimates of such relationships (Cook et al., 2012, Knol et al., 2010, Martin et al., 2012, Runge et al., 2011). Our protocol brings together for the first time three previously-published methods for eliciting and synthesizing expert knowledge:

Acknowledgements

Funding was provided by the Australian Research Council (LP100200170). We thank Yacov Salomon for his help with the minimum cross entropy calculator, and all the experts for their participation in the workshops.

References (64)

  • M. Burgman et al.

    Redefining expertise and improving ecological judgment

    Conserv. Lett.

    (2011)
  • M.A. Burgman

    Risks and Decisions for Conservation and Environmental Management

    (2005)
  • M.A. Burgman

    Trusting Judgements: How to Get the Best Out of Experts

    (2015)
  • J. Cain

    Planning Improvements in Natural Resources Management: Guidelines for Using Bayesian Networks to Support the Planning and Management of Development Programmes in the Water Sector and beyond

    (2001)
  • T.U. Chan et al.

    Bayesian network models for environmental flow decision making in the Daly River, Northern Territory, Australia

    River Res. Appl.

    (2012)
  • R. Clemen et al.

    Aggregating probability distributions

  • Commonwealth of Australia

    Water Recovery Strategy for the Murray-Darling Basin

    (2014)
  • I. Cresswell et al.

    Scoping the Development of a Method to Assess the Relative Environmental Benefits of Complementary Measures: a Report to the Murray-Darling Basin Authority

    (2017)
  • N. Dalkey et al.

    An experimental application of the Delphi method to the use of experts

    Manag. Sci.

    (1963)
  • F. Fidler et al.

    Groups Making Wise Judgements

    (2012)
  • P.H. Garthwaite et al.

    Statistical methods for eliciting probability distributions

    J. Am. Stat. Assoc.

    (2005)
  • D.J. Hand

    Statistics: a Very Short Introduction

    (2008)
  • R.M. Hogarth

    Cognitive processes and the assessment of subjective probability distributions

    J. Am. Stat. Assoc.

    (1975)
  • A. Horne et al.

    Water for the Environment: from Policy and Science to Implementation and Management

    (2017)
  • A.C. Horne et al.

    Informing environmental water management decisions: using conditional probability networks to address the information needs of planning and implementation cycles

    Environ. Manage

    (2017)
  • K.E. Jacowitz et al.

    Measures of anchoring in estimation tasks

    Personality Soc. Psychol. Bull.

    (1995)
  • R.L. Keeney et al.

    Eliciting probabilities from experts in complex technical problems

    IEEE Trans. Eng. Manag.

    (1991)
  • A.B. Knol et al.

    The use of expert elicitation in environmental health impact assessment: a seven step procedure

    Environ. Health

    (2010)
  • K.B. Korb et al.

    Bayesian Artificial Intelligence

    (2010)
  • B. Kraan et al.

    Probabilistic inversion of expert judgments in the quantification of model uncertainty

    Manag. Sci.

    (2005)
  • P.M. Kuhnert et al.

    A guide to eliciting and using expert knowledge in Bayesian ecological models

    Ecol. Lett.

    (2010)
  • M. Kynn

    The 'heuristics and biases' bias in expert elicitation

    J. R. Stat. Soc. Ser. a-Statistics Soc.

    (2008)
  • Cited by (0)

    1

    Present address: Healesville Sanctuary, Badger Creek Rd, Healesville 3777, Victoria, Australia.

    View full text