Minimising biases in expert elicitations to inform environmental management: Case studies from environmental flows in Australia
Introduction
Decision making informed by expert opinion is common in environmental management and can form a basis for urgent management and policy decisions when stakes are too high to postpone such choices (Krueger et al., 2012, Martin et al., 2012). In complex systems, when formal theories and/or measured data may be scarce, expert opinion can help to assess whether information is consistent and where evidence may be lacking (Martin et al., 2012). Informal techniques that are implicit, unstructured and undocumented—such as ‘roundtable discussions’—are commonly used for extracting expert knowledge (Fidler et al., 2012, McBride and Burgman, 2012). Whilst discussion itself is not necessarily problematic, the knowledge provided by such unstructured group expert opinion is usually based on subjective judgements prone to cognitive and motivational biases (Fidler et al., 2012, Garthwaite et al., 2005, O'Hagan et al., 2006, O'Hagan and Oakley, 2004, Speirs-Bridge et al., 2010, Tversky and Kahneman, 1975). Cognitive biases occur as a result of a failure to adequately process, aggregate or integrate relevant information due to limitations on human processing ability (McBride and Burgman, 2012, Wilson, 1994). Overconfidence biases, for example, undermine expert judgments by underestimating uncertainty (Soll and Klayman, 2004). Other examples of cognitive biases include the availability bias, where familiarity with one particular driving factor may lead an expert to believe it to be more important than it actually is (Kynn, 2008); and anchoring biases, where an initial value is used to calculate another value by adjusting it up or down (Jacowitz and Kahneman, 1995, Speirs-Bridge et al., 2010). Motivational biases are conscious or subconscious adjustments in an expert's responses that depend on their particular context, personal beliefs and experiences, and from what the expert stands to gain or lose personally form a decision (Garthwaite et al., 2005).
Biases are inherent in heuristic processing. The utility of expert knowledge is therefore dependent on the rigour with which it is elicited (Martin et al., 2012). The choice of an expert elicitation method needs to account for, and minimise the risk of, bias affecting expert judgements and resulting elicited data (Speirs-Bridge et al., 2010). Formal procedures to elicit expert knowledge have been designed to account for such biases in order to increase credibility, repeatability and transparency (Hanea et al., 2017, McBride and Burgman, 2012). Structured elicitation often employs a framework combining foundational elements of decision theory and mathematics with procedures for minimising biases (Keeney and von Winterfeldt, 1991, O'Hagan et al., 2006). Recently, the IDEA (Investigate Discuss Estimate Aggregate) structured protocol by Hanea et al. (2017) has combined different elements from established approaches to optimise expert knowledge elicitation. The success of expert elicitation depends on facilitation and the way expert judgements are collected and compiled. Therefore, only well managed, systematic elicitation protocols can return high-quality, transparent and repeatable predictions from experts (Cook et al., 2012, Hanea et al., 2017, Knol et al., 2010, Martin et al., 2012, Runge et al., 2011).
One area of environmental management where expert opinion is widely used, is the provision of environmental flows (water released from storage solely to benefit the environment; Horne et al., 2017b) to restore regulated river systems (Stewardson and Webb, 2010). Restoring more natural flow regimes is a key issue around the world (Poff et al., 1997), with substantial investments from governments in an attempt to manage degraded river systems (Acreman and Ferguson, 2010, Skinner and Langford, 2013). Effective restoration and management of these systems rely on an understanding of the relationships between the stressors (changes in flow regimes) and ecological responses (Poff and Zimmerman, 2010, Webb et al., 2013). However, while general principles of the ecological effects of changes in flow regime are well accepted (Poff and Zimmerman, 2010), there are few quantitative predictions about how different components of degraded ecosystems will respond to flow restoration (Arthington and Pusey, 2003, Souchon et al., 2008).
Despite this paucity of quantified relationships, flow management decisions must be made. Therefore, of the many environmental flow assessment techniques that have been developed (Tharme, 2003), those that make direct predictions of ecological effects have relied to a great degree on expert opinion (Stewardson and Webb, 2010). In Australia, expert opinion is a major determinant of how the approximately $15 billion investment in environmental water under the Basin Plan and other initiatives will be used (Commonwealth of Australia, 2014). It is therefore surprising that systematic approaches to elicit expert knowledge are less common than informal methods (McBride and Burgman, 2012). The reliance on expert opinion and the contested nature of environmental flows means that structured elicitation should be of great potential value for developing environmental flow recommendations (Webb et al., 2015).
Bayesian Belief Networks (BBNs; Pearl, 2000) is a modelling technique that is commonly used in natural resource management applications (McCann et al., 2006), and has also been used in environmental flows applications (Chan et al., 2012, Horne et al., 2017c, Shenton et al., 2011, Shenton et al., 2014). One of the often-cited advantages of BBNs is that they can incorporate multiple data types, including expert knowledge (Horne et al., 2017a). However, this advantage also leaves them open to being populated by poorly elicited expert opinion, affected by bias and overconfidence. The quantitative outputs from these models may give a false sense of security in the results if they are based on poor-quality expert-derived data.
This paper presents a 4-stage formal expert elicitation protocol, where opinions are directly incorporated into BBN models, providing improved rigour in the relationships and subsequent predictions. The 4 stages include pre-elicitation, workshop facilitation, output analysis and BBN model building. The first 3 stages are based on the IDEA framework by Hanea et al. (2017), combining different established approaches to minimise biases of elicited opinions. The final stage consists of discretising the elicited probability distributions in order to populate the conditional probability tables in a BBN model. This model becomes an ideal vehicle for later updating with empirical data.
We first present the general principles underpinning the protocol, providing a justification of the approaches used at each stage (section 2). We then illustrate application of the protocol using two case studies that elicited quantitative predictions of ecological responses to changes in riverine flow regimes and other environmental factors under environmental flow recommendations (section 3). The two case studies elicited expert predictions of expected changes in (i) terrestrial vegetation encroachment into river channels, or undesirable vegetation cover (hereafter “Encroachment”) and (ii) abundance of native riparian species on river banks, or desirable vegetation cover (hereafter “Native banks”). The paper closes by assessing strengths and weaknesses of the protocol, especially as experienced through the case studies.
Section snippets
Expert elicitation protocol: general principles
Our approach to minimising cognitive and motivational biases include (i) using a question protocol to reduce overconfidence and availability biases during pre-elicitation; (ii) requiring experts to answer questions independently and in isolation to minimise motivational biases, but also allowing multiple rounds of judgement to ensure transparency and expert comprehension during workshop facilitation; (iii) using mathematical accumulation of opinion and interpolation as analysis tools to avoid
Case studies on vegetation responses to environmental flows
We illustrate the above protocol using the example of two elicitation projects on Encroachment and Native banks to quantitatively predict the response of in-channel and riparian vegetation to inundation by environmental flows under a number of different scenarios.
Discussion
Environmental management requires quantitative predictions of ecological responses to management actions. In the absence of field data, the use of a formalised, expert elicitation protocol can produce transparent, methodologically rigorous and defensible estimates of such relationships (Cook et al., 2012, Knol et al., 2010, Martin et al., 2012, Runge et al., 2011). Our protocol brings together for the first time three previously-published methods for eliciting and synthesizing expert knowledge:
Acknowledgements
Funding was provided by the Australian Research Council (LP100200170). We thank Yacov Salomon for his help with the minimum cross entropy calculator, and all the experts for their participation in the workshops.
References (64)
- et al.
Managers consider multiple lines of evidence important for biodiversity management decisions
J. Environ. Manag.
(2012) - et al.
I nvestigate D iscuss E stimate A ggregate for structured expert judgement
Int. J. Forecast.
(2017) - et al.
Using optimization to develop a “designer” environmental flow regime
Environ. Model. Softw.
(2017) - et al.
Elicitator: an expert elicitation tool for regression in ecology
Environ. Model. Softw.
(2010) - et al.
The role of expert opinion in environmental modelling
Environ. Model. Softw.
(2012) - et al.
Probability is perfect, but we can't elicit it perfectly
Reliab. Eng. Syst. Saf.
(2004) - et al.
Which uncertainty? Using expert elicitation and expected value of information to design an adaptive program
Biol. Conserv.
(2011) - et al.
Why groups perform better than individuals at quantitative judgment tasks: group-to-individual transfer as an alternative to differential weighting
Organ. Behav. Hum. Decis. Process.
(2012) - et al.
Environmental flows and the European water framework directive
Freshw. Biol.
(2010) - et al.
Flow restoration and protection in Australian rivers
River Res. Appl.
(2003)
Redefining expertise and improving ecological judgment
Conserv. Lett.
Risks and Decisions for Conservation and Environmental Management
Trusting Judgements: How to Get the Best Out of Experts
Planning Improvements in Natural Resources Management: Guidelines for Using Bayesian Networks to Support the Planning and Management of Development Programmes in the Water Sector and beyond
Bayesian network models for environmental flow decision making in the Daly River, Northern Territory, Australia
River Res. Appl.
Aggregating probability distributions
Water Recovery Strategy for the Murray-Darling Basin
Scoping the Development of a Method to Assess the Relative Environmental Benefits of Complementary Measures: a Report to the Murray-Darling Basin Authority
An experimental application of the Delphi method to the use of experts
Manag. Sci.
Groups Making Wise Judgements
Statistical methods for eliciting probability distributions
J. Am. Stat. Assoc.
Statistics: a Very Short Introduction
Cognitive processes and the assessment of subjective probability distributions
J. Am. Stat. Assoc.
Water for the Environment: from Policy and Science to Implementation and Management
Informing environmental water management decisions: using conditional probability networks to address the information needs of planning and implementation cycles
Environ. Manage
Measures of anchoring in estimation tasks
Personality Soc. Psychol. Bull.
Eliciting probabilities from experts in complex technical problems
IEEE Trans. Eng. Manag.
The use of expert elicitation in environmental health impact assessment: a seven step procedure
Environ. Health
Bayesian Artificial Intelligence
Probabilistic inversion of expert judgments in the quantification of model uncertainty
Manag. Sci.
A guide to eliciting and using expert knowledge in Bayesian ecological models
Ecol. Lett.
The 'heuristics and biases' bias in expert elicitation
J. R. Stat. Soc. Ser. a-Statistics Soc.
Cited by (0)
- 1
Present address: Healesville Sanctuary, Badger Creek Rd, Healesville 3777, Victoria, Australia.