An aim of much statistical research is to wring as much from data as is possible. Improved usage of expert opinion can add significantly more information than a slight improvement in efficiency through better data analysis techniques. This book presents a range of tried and tested elicitation methods to enable statisticians to get make the most of expert opinion
An elicitation method forms a bridge between an experts opinion and an expression of these points in a statistically useful form. This book, written by a group of expert statisticians and psychologists provides an introduction to the subject and a detailed overview of the existing literature. The book guides the reader through the design of an elicitation method and details examples from a cross section of literature in the statistics, psychology, engineering and health sciences disciplines.
Preface.
1 Fundamentals of Probability and Judgement.
1.1 Introduction.
1.2 Probability and elicitation.
1.2.1 Probability.
1.2.2 Random variables and probability distributions.
1.2.3 Summaries of distributions.
1.2.4 Joint distributions.
1.2.5 Bayes' theorem.
1.2.6 Elicitation.
1.3 Uncertainty and the interpretation of probability.
1.3.1 Aleatory and epistemic uncertainty.
1.3.2 Frequency and personal probabilities.
1.3.3 An extended example.
1.3.4 Implications for elicitation.
1.4 Elicitation and the psychology of judgement.
1.4.1 Judgement - absolute or relative?
1.4.2 Beyond perception.
1.4.3 Implications for elicitation.
1.5 What use are such judgements?
1.5.1 Normative theories of probability.
1.5.2 Coherence.
1.5.3 Do elicited probabilities have the desired interpretation?
1.6 Conclusions.
1.6.1 Elicitation practice.
1.6.2 Research questions.
2 The elicitation Context.
2.1 How and who?
2.1.1 Choice of format.
2.2 What is an expert?
2.3 The elicitation process.
2.3.1 Roles within the elicitation process.
2.3.2 A model for the elicitation process.
2.4 Conventions in Chapters 3 to 9.
2.5 Conclusions.
2.5.1 Elicitation practice.
2.5.2 Research questions.
3 The psychology of Judgement Under Uncertainty.
3.1 Introduction.
3.1.1 Why psychology?
3.1.2 Chapter overview.
3.2 Understanding the task and the expert.
3.2.1 Cognitive capabilities: the proper view of human information processing?
3.2.2 Constructive processes: the proper view of the process?
3.3 Understanding research on human judgement.
3.3.1 Experts versus the rest: the proper focus or research?
3.3.2 Early research on subjective probability: 'conservatism' in Bayesian probability revision.
3.4 The heuristic and biases research programme.
3.4.1 Availability.
3.4.2 Representativeness.
3.4.3 Do frequency representations remove the biases attributed to availability and representativeness?
3.4.4 Anchoring and adjusting.
3.4.5 Support Theory.
3.4.6 The affect heuristic.
3.4.7 Critique of the heuristics and biased approach.
3.5 Experts and expertise.
3.5.1 The heuristics and biased approach.
3.5.2 The cognitive science approach.
3.5.3 'The middle way'.
3.6 Three meta theories of judgement.
3.6.1 The cognitive continuum.
3.6.2 The inside versus the outside view.
3.6.3 The naive intuitive statistician metaphor.
3.7 Conclusions.
3.7.1 Elicitation practice.
3.7.2 Research questions.
4 The Elicitation of Probabilities.
4.1 Introduction.
4.2 The Calibration of Subjective Probabilities.
4.2.1 Research methods in calibration research.
4.2.1 Calibration research: General findings.
4.2.3 Calibration research in applied setting.
4.2.4 A case study in probability judgement: calibration research in medicine.
4.3 The calibration in subjective probabilities: theories and explanations.
4.3.1 Explanation of probability judgement in calibration tasks.
4.3.2 Theories of the calibration of subjective probabilities.
4.4 Representations and methods.
4.4.1 Different modes for representing uncertainty.
4.4.2 Different formats for eliciting responses.
4.4.3 Key lessons.
4.5 Debiasing.
4.5.1 General principles for debiasing judgement.
4.5.2 'Managing noise'.
4.5.3 Redressing insufficient regressiveness in prediction.
4.5.4 A caveat concerning post hoc corrections.
4.6 Conclusions.
4.6.1 Elicitation practice.
4.6.2 Research questions.
5 Eliciting Distributions - General.
5.1 From probabilities to distributions.
5.1.1 From a few to infinity.
5.1.2 Summaries.
5.1.3 Fitting.
5.1.4 Overview.
5.2 Eliciting univariate distributions.
5.2.1 Summaries based on probabilities.
5.2.2 Proportions.
5.2.3 Other summaries.
5.3 Eliciting multivariate distributions.
5.3.1 Structuring.
5.3.2 Eliciting association.
5.3.3 Joint and conditional probabilities.
5.3.4 Regression.
5.3.5 Many variables.
5.4 Uncertainty and imprecision.
5.4.1 Qualifying alicitation error.
5.4.2 Sensivity analysis.
5.4.3 Feedback and overfitting.
5.5 Conclusions.
5.5.1 Elicitation practice.
5.5.2 Research questions.
6 Eliciting and Fitting a Parametric Distribution.
6.1 Introduction.
6.2 Outline of this chapter.
6.3 Eliciting opinion about a proportion.
6.4 Eliciting opinion about a general scalar quantity.
6.5 Eliciting opinion about a set of proportions.
6.6 Eliciting opinion about the parameters of a multivariate normal distribution.
6.7 Eliciting opinion about the parameters of a linear regression model.
6.8 Eliciting opinion about the parameters of a generalized linear model.
6.9 Elicitation methods for other problems.
6.10 Deficiencies in existing research.
6.11 Conclusions.
6.11.1 Elicitation practice.
6.11.2 Research questions.
7 Eliciting distributions - Uncertainty and Imprecision.
7.1 Introduction.
7.2 Imprecise probabilities.
7.3 Incomplete information.
7.4 Summary.
7.5 Conclusions.
7.5.1 Elicitation practice.
7.5.2 Research questions.
8 Evaluating Elicitation.
8.1 Introduction.
8.1.1 Good elicitation.
8.1.2 Innacurate knowledge.
8.1.3 Automatic calibration.
8.1.4 Lessons of the psychological literature.
8.1.5 Outline of this chapter.
8.2 Scoring rules.
8.2.1 Scoring rules for discrete probability distributions.
8.2.2 Scoring rules for continuous probability distributions.
8.3 Coherence, feedback and overfitting.
8.3.1 Coherence and calibration.
8.3.2 Feedback and overfitting.
8.4 Conclusions.
8.4.1 Elicitation practice.
8.4.2 Research questions.
9 Multiple experts.
9.1 Introduction.
9.2 Mathematical aggregation.
9.2.1 Bayesian methods.
9.2.2 Opinion pooling.
9.2.3 Cooke's method.
9.2.4 Performance of mathematical aggregation.
9.3 Behavioural aggregation.
9.3.1 Group elicitation.
9.3.2 Other methods of behavioural aggregation.
9.3.3 Performance of behavioural methods.
9.4 Discussion.
9.5 Elicitation practice.
9.6 Research questions.
10 Published Examples of the Formal Elicitation of Expert Opinion.
10.1 Some applications.
10.2 An example of an elicitation interview - eliciting engine sales.
10.3 Medicine.
10.3.1 Diagnosis and treatment decisions.
10.3.2 Clinical trials.
10.3.3 Survival analysis.
10.3.4 Clinical psychology.
10.4 The nuclear industry.
10.5 Veterinary science.
10.6 Agriculture.
10.7 Meteorology.
10.8 Business studies, economics and finance.
10.9 Other professions.
10.10 Other examples of the elicitation of subjective probabilities.
11 Guidance on Best Practice.
12 Areas for Research.
Bibliography.
Glossary.
