Skip to main content
SpringerLink
Log in
Book cover

Elicitation pp 171–210Cite as

Eliciting Multivariate Uncertainty from Experts: Considerations and Approaches Along the Expert Judgement Process

  • Chapter
  • First Online:

Part of the book series: International Series in Operations Research & Management Science ((ISOR,volume 261))

Abstract

In decision and risk analysis problems, modelling uncertainty probabilistically provides key insights and information for decision makers. A common challenge is that uncertainties are typically not isolated but interlinked which introduces complex (and often unexpected) effects on the model output. Therefore, dependence needs to be taken into account and modelled appropriately if simplifying assumptions, such as independence, are not sensible. Similar to the case of univariate uncertainty, which is described elsewhere in this book, relevant historical data to quantify a (dependence) model are often lacking or too costly to obtain. This may be true even when data on a model’s univariate quantities, such as marginal probabilities, are available. Then, specifying dependence between the uncertain variables through expert judgement is the only sensible option. A structured and formal process to the elicitation is essential for ensuring methodological robustness. This chapter addresses the main elements of structured expert judgement processes for dependence elicitation. We introduce the processes’ common elements, typically used for eliciting univariate quantities, and present the differences that need to be considered at each of the process’ steps for multivariate uncertainty. Further, we review findings from the behavioural judgement and decision making literature on potential cognitive fallacies that can occur when assessing dependence as mitigating biases is a main objective of formal expert judgement processes. Given a practical focus, we reflect on case studies in addition to theoretical findings. Thus, this chapter serves as guidance for facilitators and analysts using expert judgement.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   229.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   299.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   299.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    The study is also known as WASH-1400 and as the Rasmussen Report due to Norman Carl Rasmussen. At that time, the use of expert opinion for assessing uncertainties was often viewed highly sceptical, however a main challenge was that until then no nuclear plant accident had been observed. Therefore, the report, together with its use of expert opinion, was only revived due to the Three Mile Island accident (1979). After the incident, the report’s results were prescient. In particular, the inclusion of human error as a source of risk made the case for expert judgement.

  2. 2.

    There has been ongoing philosophical debate about the meaning of causation. While some refuted the concept of causation in science altogether (Russell 1912), others focused on specific aspects. For us, probabilistic causation (Suppes 1970) and its perception/inference are of interest. Hume (1748/2000) proposes one of the most established accounts for that. He proposes a (unobservable) causal mechanism which is inferred through the regularity of an effect following a cause.

  3. 3.

    Bayes’ Theorem is named after Thomas Bayes (1701–1761) who first proposed it. Since then it has been further developed and had its impact in a variety of problem contexts (see Bertsch McGrayne 2011 for a historical overview). In its simplest form, for events X and Y, it is defined as \(P(X\vert Y ) = \frac{P(Y \vert X)P(X)} {P(Y )}\) whereas P(Y ) ≠ 0.

  4. 4.

    When, + indicates the presence of variables X and Y, and − their absence, the quadrants can be presented as:

  5. 5.

    In the literature on event trees and influence diagrams, the idea of decomposition is often mentioned as it describes a “divide and conquer” technique (Hora 2007) that allows to ease the assessment in particular of conditional probabilities (see e.g. Kleinmuntz et al. 1996).

References

  • Abou-Moustafa KT, De La Torre F, Ferrie FP (2010) Designing a metric for the difference between Gaussian densities. In: Angeles J, Boulet B, Clark J J, Kövecses J, Siddiqi K (eds) Brain, body and machine. Springer, Berlin, pp 57–70

    Chapter  Google Scholar 

  • Aczél J, Wagner C (1980) A characterisation of weighted arithmetic means. SIAM J Algebr Discrete Methods 1(3):259–260

    Article  Google Scholar 

  • Ajzen I (1977) Intuitive theories of events and the effects of base-rate information on prediction. J Pers Soc Psychol 35(5):303–314

    Article  Google Scholar 

  • Allan LG (1980) A note on measurement of contingency between two binary variables in judgment tasks. Bull Psychon Soc 15(3):147–149

    Article  Google Scholar 

  • Balakrishnan N, Nevzorov VB (2004) A primer on statistical distributions. Wiley, Hoboken

    Google Scholar 

  • Bar-Hillel M (1980) The base-rate fallacy in probability judgments. Acta Psychol 44(3):211–233

    Article  Google Scholar 

  • Batanero C, Díaz C (2012) Training school teachers to teach probability: reflections and challenges. Chilean J Stat 3(1):3–13

    Google Scholar 

  • Bechlivanidis C, Lagnado DA (2013) Does the “why” tell us the “when”? Psychol Sci 24(8):1563–1572

    Article  Google Scholar 

  • Bedford T, Cooke RM (2001) Probabilistic risk analysis: foundations and methods. Cambridge University Press, Cambridge

    Book  Google Scholar 

  • Bertsch McGrayne S (2011) The theory that would not die. Yale University Press, New Haven

    Google Scholar 

  • Bes B, Sloman S, Lucas CG and Raufaste E (2012) Non-Bayesian inference: causal structure trumps correlation. Cogn Sci 36(7):1178–1203

    Article  Google Scholar 

  • Bolger F, Wright G (1994) Assessing the quality of expert judgment: issues and analysis. Decis Support Syst 11(1):1–24

    Article  Google Scholar 

  • Borovcnik M, Kapadia R (2014) From puzzles and paradoxes to concepts in probability. In: Chernoff EJ, Sriraman B (eds) Probabilistic thinking. Springer, Dordrecht, pp 35–73

    Chapter  Google Scholar 

  • Bradley R, Dietrich F, List C (2014) Aggregating causal judgments. Philos Sci 81(4):491–515

    Article  Google Scholar 

  • Browne GJ, Curley SP, Benson PG (1997) Evoking information in probability assessment: knowledge maps and reasoning-based directed questions. Manag Sci 43(1):1–14

    Article  Google Scholar 

  • Carranza P, Kuzniak A (2009) Duality of probability and statistics teaching in French education. In: Batanero C (ed) Joint ICMI/IASE study: teaching statistics in school mathematics, p 5

    Google Scholar 

  • Chapman LJ, Chapman JP (1969) Illusory correlation as an obstacle to the use of valid psychodiagnostic signs. J Abnorm Psychol 74(3):271–280

    Article  Google Scholar 

  • Clemen RT, Reilly T (1999) Correlations and copulas for decision and risk analysis. Manag Sci 45(2):208–224

    Article  Google Scholar 

  • Clemen RT, Reilly T (2014) Making hard decisions with decision tools. South-Western, Mason

    Google Scholar 

  • Clemen RT, Fischer GW, Winkler RL (2000) Assessing dependence: some experimental results. Manag Sci 46(8):1100–1115

    Article  Google Scholar 

  • Cooke RM (1991) Experts in uncertainty: opinion and subjective probability in Science. Oxford University Press, New York

    Google Scholar 

  • Cooke RM (2013) Uncertainty analysis comes to integrated assessment models for climate change and conversely. Climatic Change 117(3):467–479

    Article  Google Scholar 

  • Cooke RM, Goossens LHJ (1999) Procedures guide for structured expert judgment. Commission of the European Communities, Brussels

    Google Scholar 

  • Cooke RM, Goossens LHJ (2004) Expert judgement elicitation for risk assessments of critical infrastructures. J Risk Res 7(6):643–656

    Article  Google Scholar 

  • Cooke RM, Goossens LHJ (2008) TU Delft expert judgment data base. Reliab Eng Syst Saf 93(5):657–674

    Article  Google Scholar 

  • Costello FJ (2009) How probability theory explains the conjunction fallacy. J Behav Decis Mak 22(3):213–234

    Article  Google Scholar 

  • Dawes RM (1988) Rational choice in an uncertain world. Harcourt, Brace, Jovanovich, New York

    Google Scholar 

  • DeGroot MH (1974) Reaching a consensus. J Am Stat Assoc 69(345):118–121

    Article  Google Scholar 

  • Díaz C, Batanero C, Contreras JM (2010) Teaching independence and conditional probability. Boletín de Estadística e Investigación Operativa 26(2):149–162

    Google Scholar 

  • Dietrich F, List C (2016) Probabilistic Opinion Pooling. In: Hajek A, Hitchcock C (eds) The oxford handbook of probability and philosophy. Oxford handbooks. Oxford University Press, Oxford, pp 179–207

    Google Scholar 

  • DuCharme WM (1970) Response bias explanation of conservative human inference. J Exp Psychol 85(1):66–74

    Article  Google Scholar 

  • Eddy DM (1982) Probabilistic reasoning in clinical medicine: problems and opportunities. In: Kahneman D, Slovic P and Tversky A (eds) Judgment under uncertainty: heuristics and biases. Cambridge University Press, New York, pp 249–267

    Chapter  Google Scholar 

  • Eder AB, Fiedler K, Hamm-Eder S (2011) Illusory correlations revisited: the role of pseudocontingencies and working-memory capacity. Q J Exp Psychol 64(3):517–532

    Article  Google Scholar 

  • Edwards W (1965) Optimal strategies for seeking information: models for statistics, choice reaction times, and human information processing. J Math Psychol 2(2):312–329

    Article  Google Scholar 

  • EFSA=European Food and Safety Authority (Bolger F, Hanea AM, O’Hagan A, Mosbach-Schulz O, Oakley J, Rowe G, Wenholt M) (2014) Guidance on expert knowledge elicitation in food and feed safety risk assessment. EFSA J 12(6):3734

    Google Scholar 

  • Eichler A, Vogel M (2014) Three Approaches for Modelling Situations with Randomness. In: Chernoff EJ, Sriraman B (eds) Probabilistic thinking. Springer, Dordrecht, pp 75–99

    Chapter  Google Scholar 

  • Einhorn HJ, Hogarth RM (1986) Judging probable cause. Psychol Bull 99(1):3–19

    Article  Google Scholar 

  • Falk R (1983) Conditional probabilities: insights and difficulties. In: Tall D (ed) Proceedings of the third international conference for the psychology of mathematics education, Warwick, 1983

    Google Scholar 

  • Fenton NE, Neil M (2013) Risk Assessment and Decision Analysis with Bayesian Networks. Taylor and Francis Group, Boca Raton

    Google Scholar 

  • Fenton NE, Neil M, Caballero JG (2007) Using Ranked Nodes to Model Qualitative Judgments in Bayesian Networks. IEEE Trans Knowl Data Eng 19(10):1420–1432

    Article  Google Scholar 

  • Fiedler K, Brinkmann B, Betsch T, Wild B (2000) A sampling approach to biases in conditional probability judgments: beyond base rate neglect and statistical format. J Exp Psychol Gen 129(3):399–418

    Article  Google Scholar 

  • Flores MJ, Nicholson AE, Brunskill A, Korb KB, Mascaro S (2011) Incorporating expert knowledge when learning Bayesian network structure: a medical case study. Artif Intell Med 53(3):181–204

    Article  Google Scholar 

  • Fountain J, Gunby P (2011) Ambiguity, the certainty illusion, and the natural frequency approach to reasoning with inverse probabilities. N Z Econ Pap 45(1-2):195–207

    Google Scholar 

  • French S (2011) Aggregating expert judgement. Revista de la Real Academia de Ciencias Exactas, Fisicas y Naturales. Ser A Mate 105(1):181–206

    Google Scholar 

  • Gal I (2002) Adults’ statistical literacy: meanings, components, responsibilities. Int Stat Rev 70(1):1–25

    Article  Google Scholar 

  • Garthwaite PH, Kadane JB, O’Hagan A (2005) Statistical methods for eliciting probability distributions. J Am Stat Assoc 100(470):680–701

    Article  Google Scholar 

  • Gavanski I, Hui C (1992) Natural sample spaces and uncertain belief. J Pers Soc Psychol 63(5):766–780

    Article  Google Scholar 

  • Gilovich T, Griffin D, Kahneman D (2002) Heuristics and biases: the psychology of intuitive judgment. Cambridge University Press, New York

    Book  Google Scholar 

  • Gokhale DV, Press SJ (1982) Assessment of a prior distribution for the correlation coefficient in a bivariate normal distribution. J R Stat Soc Ser A (General) 145:237–249

    Article  Google Scholar 

  • Goldstein M, Wooff D (2007) Bayes linear statistics, theory and methods. Wiley, Chichester

    Book  Google Scholar 

  • Hamilton DL (2015) Cognitive processes in stereotyping and intergroup behavior. Psychology Press, New York

    Google Scholar 

  • Hanea A, McBride MF, Burgman MA, Wintle BC, Fidler F, Flander L, Twardy CR, Manning B, Mascaro S (2017) Investigate discuss estimate aggregate for structured expert judgement. Int J Forecast 33:267–279

    Article  Google Scholar 

  • Hanea A, Morales Nápoles O, Ababei D (2015) Non-parametric Bayesian networks: improving theory and reviewing applications. Reliab Eng Syst Saf 144:265–284

    Article  Google Scholar 

  • Hanea D, Ale B (2009) Risk of human fatality in building fires: a decision tool using Bayesian networks. Fire Saf J 44(5):704–710

    Article  Google Scholar 

  • Hastie R (2016) Causal thinking in judgments. In: Keren G, Wu G (eds) The Wiley Blackwell handbook of judgment and decision making. Wiley, Chichester, pp 590–628

    Google Scholar 

  • Hastie R, Dawes RM (2001) Rational choice in an uncertain world. Sage, Thousand Oaks

    Google Scholar 

  • Henrion M (1989) Some practical issues in constructing belief networks. In: Kanal L, Levitt T, Lemmer J (eds) Uncertainty in artificial intelligence. Elsevier Science Publishing Company, New York, pp 132–139

    Google Scholar 

  • Hora SC (2007) Eliciting Probabilities from Experts. In: Edwards W, Miles RF Jr, Von Winterfeldt D (eds) Advances in decision analysis - from foundations to applications. Cambridge University Press, New York, pp 129–163

    Chapter  Google Scholar 

  • Hora SC, Kardeş E (2015) Calibration, sharpness and the weighting of experts in a linear opinion pool. Ann Oper Res 229(1):429–450

    Article  Google Scholar 

  • Howard RA (1989) Knowledge maps. Manag Sci 35(8):903–922

    Article  Google Scholar 

  • Howard RA, Matheson JE (2005) Influence diagrams. Decis Anal 2(3):127–143

    Article  Google Scholar 

  • Hume D (1748/2000) An enquiry concerning human understanding. Beauchamp TL (ed) Oxford University Press, New York

    Google Scholar 

  • Joe H (2014) Dependence modeling with copulas. Chapman and Hall, Boca Raton

    Google Scholar 

  • Kadane J, Wolfson LJ (1998) Experiences in elicitation. J R Stat Soc Ser D (Stat) 47(1):3–19

    Article  Google Scholar 

  • Kahneman D, Frederick S (2002) Representativeness revisited: attribute substitution in intuitive judgment. In: Gilovich T, Griffin D, Kahneman D (eds) Heuristics of intuitive judgment: extensions and applications, 1st edn. Cambridge University Press, New York, pp 19–49

    Google Scholar 

  • Kahneman D, Tversky A (1972) Subjective probability: a judgment of representativeness. Cogn Psychol 3(3):430–454

    Article  Google Scholar 

  • Kahneman D, Tversky A (1973) On the psychology of prediction. Psychol Rev 80(4):237–251

    Article  Google Scholar 

  • Kao SF, Wasserman EA (1993) Assessment of an information integration account of contingency judgment with examination of subjective cell importance and method of information presentation. J Exp Psychol Learn Mem Cogn 19(6):1363–1386

    Article  Google Scholar 

  • Keeney RL, Von Winterfeldt D (1991) Eliciting probabilities from experts in complex technical problems. IEEE Trans Eng Manag 38(3):191–201

    Article  Google Scholar 

  • Keren G, Teigen KH (2006) Yet another look at Heuristics and biases approach. In: Koehler DJ, Harvey N (eds) Blackwell handbook of judgment and decision making, 1st edn. Blackwell Publishing, Oxford, p 89–109

    Google Scholar 

  • Kleinmuntz DN, Fennema MG, Peecher ME (1996) Conditioned assessment of subjective probabilities: Identifying the benefits of decomposition. Organ Behav Hum Decis Process 66(1):1–15

    Article  Google Scholar 

  • Koehler JJ (1996) The base rate fallacy reconsidered: descriptive, normative, and methodological challenges. Behav Brain Sci 19(1):1–17

    Article  Google Scholar 

  • Kraan BCP (2002) Probabilistic inversion in uncertainty analysis: and related topics. PhD Thesis, Delft University of Technology, The Netherlands

    Google Scholar 

  • Kruskal WH (1958) Ordinal measures of association. J Am Stat Assoc 53(284):814–861

    Article  Google Scholar 

  • Kurowicka D, Cooke RM (2006) Uncertainty analysis with high dimensional dependence modelling. Wiley, Chichester

    Book  Google Scholar 

  • Kynn M (2008) The “heuristics and biases” bias in expert elicitation. J R Stat Soc Ser A (Stat Soc) 171(1):239–264

    Google Scholar 

  • Lad F (1996) Operational subjective statistical methods: a mathematical, philosophical, and historical introduction. Wiley-Interscience, New York

    Google Scholar 

  • Lagnado DA, Shanks DR (2002) Probability judgment in hierarchical learning: a conflict between predictiveness and coherence. Cognition 83(1):81–112

    Article  Google Scholar 

  • Lagnado DA, Sloman SA (2006) Inside and outside probability judgment. In: Koehler DJ, Harvey N (eds) Blackwell handbook of judgment and decision making, 1st edn. Blackwell Publishing, Oxford, pp 157–176

    Google Scholar 

  • Liu TC, Lin YC (2010) The application of Simulation Assisted Learning Statistics (SALS) for correcting misconceptions and improving understanding of correlation. J Comput Assis Learn 26(2):143–158

    Article  Google Scholar 

  • Mandel DR, Lehman DR (1998) Integration of contingency information in judgments of cause, covariation, and probability. J Exp Psychol Gen 127(3):269–285

    Article  Google Scholar 

  • McConway KJ (1981) Marginalization and linear opinion pools. J Am Stat Assoc 76(374):410–414

    Article  Google Scholar 

  • McKenzie CR, Mikkelsen LA (2007) A Bayesian view of covariation assessment. Cogn Psychol 54(1):33–61

    Article  Google Scholar 

  • Meder B, Gigerenzer G (2014) Statistical thinking: no one left behind. In: Chernoff EJ, Sriraman B (eds) Probabilistic thinking. Springer, Dordrecht, pp 127–148

    Chapter  Google Scholar 

  • Medin DL, Coley JD, Storms G, Hayes BL (2003) A relevance theory of induction. Psychon Bull Rev 10(3):517–532

    Article  Google Scholar 

  • Meehl PE, Rosen A (1955) Antecedent probability and the efficiency of psychometric signs, patterns, and cutting scores. Psychol Bull 52:194–216

    Article  Google Scholar 

  • Merkhofer MW (1987) Quantifying judgmental uncertainty: Methodology, experiences, and insights. IEEE Trans Syst Man Cybern 17(5):741–752

    Article  Google Scholar 

  • Mitchell CJ, De Houwer J, Lovibond PF (2009) The propositional nature of human associative learning. Behav Brain Sci 32(2):183–198

    Article  Google Scholar 

  • Mkrtchyan L, Podofillini L, Dang VN (2015) Bayesian belief networks for human reliability analysis: a review of applications and gaps. Reliab Eng Syst Saf 139:1–16

    Article  Google Scholar 

  • Montibeller G, Von Winterfeldt D (2015) Cognitive and motivational biases in decision and risk analysis. Risk Anal 35(7):1230–1251

    Article  Google Scholar 

  • Morales-Nápoles O (2010) Bayesian belief nets and vines in aviation safety and other applications. PhD Thesis, Delft University of Technology, The Netherlands

    Google Scholar 

  • Morales-Nápoles O, Worm DTH (2013) Hypothesis testing of Multidimensional probability distributions. WP4 GAMES2R TNO Report No. 0100003764, TNO, The Netherlands

    Google Scholar 

  • Morales-Nápoles O, Hanea AM, Worm DTH (2013) Experimental results about the assessments of conditional rank correlations by experts: Example with air pollution estimates. In: Proceedings of the 22nd European safety and reliability conference safety, reliability and risk analysis. University of Amsterdam, Amsterdam, 2013

    Google Scholar 

  • Morales-Nápoles O, Worm DTH, Abspoel LM, Huibregtse E, Courage W (2015) Trends and uncertainties regarding rain intensity in the Netherlands. TNO 2015 R10009, TNO

    Google Scholar 

  • Morales-Nápoles O, Paprotny D, Worm DTH, Abspoel LM, Courage W (2016a) Characterization of precipitation through copulas and expert judgement for risk assessment of infrastructure. In: Proceedings of the 25th European safety and reliability conference safety, reliability and risk analysis. Eidgenössische Technische Hochschule Zürich, Zürich, 2016

    Google Scholar 

  • Morales-Nápoles O, Paprotny D, Worm D, Abspoel-Bukman L, Courage W (2017) Characterization of precipitation through copulas and expert judgement for risk assessment of infrastructure. ASCE-ASME J Risk and Uncertain Eng Syst A Civ Eng 3(4), 04017012

    Article  Google Scholar 

  • Morgan MG, Henrion M (1990) Uncertainty: a guide to dealing with uncertainty in quantitative risk and policy analysis. Cambridge University Press, New York

    Book  Google Scholar 

  • Mosleh A, Apostolakis G (1986) The assessment of probability distributions from expert opinions with an application to seismic fragility curves. Risk Anal 6(4):447–461

    Article  Google Scholar 

  • Norrington L, Quigley J, Russell A, Van der Meer R (2008) Modelling the reliability of search and rescue operations with Bayesian belief networks. Reliab Eng Syst Saf 93(7):940–949

    Article  Google Scholar 

  • O’Hagan A, Buck CE, Daneshkhah A, Eiser JR, Garthwaite PH, Jenkinson DJ, Oakley JE, Rakow T (2006) Uncertain judgements: eliciting experts’ probabilities. Wiley, Chichester

    Book  Google Scholar 

  • Over D (2004) Rationality and the normative/descriptive distinction. In: Koehler DJ, Harvey N (eds) Blackwell handbook of judgment and decision making, 1st edn. Blackwell Publishing, Oxford, pp 3–18

    Google Scholar 

  • Pearl J (1988) Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan Kaufmann, San Francisco

    Google Scholar 

  • Pearl J (2009) Causality: models, reasoning, and inference. Cambridge University Press, Cambridge

    Book  Google Scholar 

  • Pollatsek A, Well AD, Konold C, Hardiman P, Cobb G (1987) Understanding conditional probabilities. Organ. Behav Hum Decis Process 40(2):255–269

    Article  Google Scholar 

  • Pollino CA, Woodberry O, Nicholson A, Korb K, Hart BT (2007) Parameterisation and evaluation of a Bayesian network for use in an ecological risk assessment. Environ Model Softw 22(8):1140–1152

    Article  Google Scholar 

  • Pólya G (1941) Heuristic reasoning and the theory of probability. Am Math Mon 48(7):450–465

    Article  Google Scholar 

  • Ramsey FP (1926) Truth and probability. In: Braithwaite RB (ed) Ramsey FP (1931) The foundations of mathematics and other logical essays. Kegan, Paul, Trench, Trubner and Co, London, pp 156–198

    Google Scholar 

  • Revie M, Bedford T, Walls L (2010) Evaluation of elicitation methods to quantify Bayes linear models. J Risk Reliab 224(4):322–332

    Google Scholar 

  • Rottman BM, Hastie R (2014) Reasoning about causal relationships: inferences on causal networks. Psychol Bull 140(1):109–139

    Article  Google Scholar 

  • Rowe G, Wright G (2001) Expert opinions in forecasting: the role of the Delphi technique. In: Armstrong JS (ed) Principles of forecasting. Springer, New York, pp 125–144

    Chapter  Google Scholar 

  • Russell B (1912) On the notion of cause. Proc Aristot Soc 13:1–26

    Article  Google Scholar 

  • Shachter RD (1988) Probabilistic inference and influence diagrams. Oper Res 36(4):589–604

    Article  Google Scholar 

  • Simon HA (1957) Models of man: social and rational. Wiley, New York

    Google Scholar 

  • Sklar M (1959) Fonctions de répartitionà n dimensions et leurs marges. Publ Inst Statist Univ Paris 8:229–231

    Google Scholar 

  • Sloman SA, Lagnado D (2005) The problem of induction. In: Holyoak KJ, Morrison RG (eds) The Cambridge handbook of thinking and reasoning. Cambridge University Press, New York, pp 95–116

    Google Scholar 

  • Smedslund J (1963) The concept of correlation in adults. Scand J Psychol 4(1):165–173

    Article  Google Scholar 

  • Spetzler CS, Staël von Holstein CA (1975) Exceptional paper-probability encoding in decision analysis. Manag Sci 22(3):340–358

    Article  Google Scholar 

  • Spirtes P, Glymour CN, Scheines R (2000) Causation, prediction, and search. MIT Press, Cambridge

    Google Scholar 

  • Staël von Holstein CA, Matheson JE (1979) A manual for encoding probability distributions. In: Defense advanced research projects agency, decisions and designs. SRI International, Menlo Park, CA

    Google Scholar 

  • Stanovich KE, West RF (2000) Individual differences in reasoning: implications for the rationality debate? Behav Brain Sci 23:645–726

    Article  Google Scholar 

  • Stone M (1961) The opinion pool. Ann Math Stat 32(4):1339–1342

    Article  Google Scholar 

  • Suppes P (1970) A probabilistic theory of causality. North-Holland Publishing, Amsterdam

    Google Scholar 

  • Swain AD, Guttman HE (1983) Handbook of human reliability analysis with emphasis on nuclear power plant applications. US Nuclear Regulatory Commission. NUREG/CR-1278, Washington, DC

    Google Scholar 

  • Tentori K, Crupi V, Russo S (2013) On the determinants of the conjunction fallacy: probability versus inductive confirmation. J Exp Psychol Gen 142(1):235–255

    Article  Google Scholar 

  • Tversky A, Kahneman D (1973) Availability: a heuristic for judging frequency and probability. Cogn Psychol 5(2):207–232

    Article  Google Scholar 

  • Tversky A, Kahneman D (1980) Causal schemas in judgments under uncertainty. Progress Soc Psychol 1:49–72

    Google Scholar 

  • Tversky A, Kahneman D (1983) Extensional versus intuitive reasoning: the conjunction fallacy in probability judgment. Psychol Rev 90(4):293–315

    Article  Google Scholar 

  • USNRC = US Nuclear Regulatory Commission (1975) Reactor safety study: An assessment of accident risks in U.S. commercial nuclear power plants. NUREG 75–014, Washington, DC

    Google Scholar 

  • USNRC = US Nuclear Regulatory Commission (1987) Reactor risk reference document. NUREG-1150, Washington, DC

    Google Scholar 

  • Utts J (2003) What educated citizens should know about statistics and probability. Am Stat 57(2):74–79

    Article  Google Scholar 

  • Villejoubert G, Mandel DR (2002) The inverse fallacy: an account of deviations from Bayes’s theorem and the additivity principle. Mem Cogn 30(2):171–178

    Article  Google Scholar 

  • Walls L, Quigley J (2001) Building prior distributions to support Bayesian reliability growth modelling using expert judgement. Reliab Eng Syst Saf 74(2):117–128

    Article  Google Scholar 

  • Werner C, Bedford T, Cooke RM, Hanea AM, Morales Nápoles, O (2017) Expert judgement for dependence in probabilistic modelling: a systematic literature review and future research directions. Eur J Oper Res 258(3):801–819

    Article  Google Scholar 

  • Wilson KJ (2017) An investigation of dependence in expert judgement studies with multiple experts. Int J Forecast 33(1):325–336

    Article  Google Scholar 

  • Winkler RL, Clemen RT (2004) Multiple experts vs. multiple methods: combining correlation assessments. Decis Anal 1(3):167–176

    Article  Google Scholar 

  • Wright G, Goodwin P (2009) Decision making and planning under low levels of predictability: Enhancing the scenario method. Int J Forecast 25(4):813–825

    Article  Google Scholar 

  • Wunderlich K, Symmonds M, Bossaerts P, Dolan RJ (2011) Hedging your bets by learning reward correlations in the human brain. Neuron 71(6):1141–1152

    Article  Google Scholar 

  • Zwirglmaier K, Straub D (2016) Approaches to Bayesian network structure elicitation. In: Walls L, Revie M, Bedford T (eds) Risk, reliability and safety: innovating theory and practice. Proceedings of the 26th European safety and reliability conference, ESREL 2016, Glasgow, September 2016. Taylor and Francis Group, London, p 333

    Google Scholar 

Download references

Acknowledgements

We would like to thank the editors for the feedback on the previous version of this chapter.

Author information

Authors and Affiliations

  1. Department of Management Science, University of Strathclyde, Glasglow, UK

    Christoph Werner

  2. CEBRA, University of Melbourne, Parkville, VIC, Australia

    Anca M. Hanea

  3. Faculty of Civil Engineering and Geosciences, Technological University Delft, Delft, The Netherlands

    Oswaldo Morales-Nápoles

Authors
  1. Christoph Werner
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anca M. Hanea
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Oswaldo Morales-Nápoles
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christoph Werner .

Editor information

Editors and Affiliations

  1. Faculty of Economics, CeBER and INESC Coimbra, University of Coimbra, Coimbra, Portugal

    Luis C. Dias

  2. Department of Management Science, University of Strathclyde, Scotland, United Kingdom

    Alec Morton

  3. Department of Management Science, University of Strathclyde, Scotland, United Kingdom

    John Quigley

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Werner, C., Hanea, A.M., Morales-Nápoles, O. (2018). Eliciting Multivariate Uncertainty from Experts: Considerations and Approaches Along the Expert Judgement Process. In: Dias, L., Morton, A., Quigley, J. (eds) Elicitation. International Series in Operations Research & Management Science, vol 261. Springer, Cham. https://doi.org/10.1007/978-3-319-65052-4_8

Download citation

Publish with us

Policies and ethics

Search

Navigation