Informing Reimbursement Decisions Using Cost-Effectiveness Modelling: A Guide to the Process of Generating Elicited Priors to Capture Model Uncertainties

Abstract

In informing decisions, utilising health technology assessment (HTA), expert elicitation can provide valuable information, particularly where there is a less-developed evidence-base at the point of market access. In these circumstances, formal methods to elicit expert judgements are preferred to improve the accountability and transparency of the decision-making process, help reduce bias and the use of heuristics, and also provide a structure that allows uncertainty to be expressed. Expert elicitation is the process of transforming the subjective and implicit knowledge of experts into their quantifiable expressions. The use of expert elicitation in HTA is gaining momentum, and there is particular interest in its application to diagnostics, medical devices and complex interventions such as in public health or social care. Compared with the gathering of experimental evidence, elicitation constitutes a reasonably low-cost source of evidence. Given its inherent subject nature, the potential biases in elicited evidence cannot be ignored and, due to its infancy in HTA, there is little guidance to the analyst wishing to conduct a formal elicitation exercise. This article attempts to summarise the stages of designing and conducting an expert elicitation, drawing on key literature and examples, most of which are not in HTA. In addition, we critique their applicability to HTA, given its distinguishing features. There are a number of issues that the analyst should be mindful of, in particular the need to appropriately characterise the uncertainty associated with model inputs and the fact that there are often numerous parameters required, not all of which can be defined using the same quantities. This increases the need for the elicitation task to be as straightforward as possible for the expert to complete.

Appendices

Appendix

See Boxes 1, 2 and 3.

Box 1: Uses of elicitation in cost-effectiveness modelling

• Generating an appropriate set of comparators.

• Identifying appropriate patient pathways and relevant events.

• Describing parameters and their associated uncertainty.

• Quantifying the extent of bias, or improving generalisability from one context to another.

• Characterising structural uncertainties either through generating differential weights for scenarios or by eliciting distributions of parameterized uncertainties.

• Validating or calibrating model estimates.

Box 2: Application of the histogram method

Box 3: Examples of biases in elicitation

Biases in elicitation can include:

• Biases associated with experts.

  • Motivation biases, e.g. when experts have an incentive (for example, financial) to reach a certain conclusion.

  • Cognitive biases: these commonly involve the use of heuristics, or ‘rules of thumb’, to help reach decisions, solve problems or form judgements quickly. Examples are:

    • Conjunction fallacy: When the probability of conjunction (combined) events is judged to be more likely than either of its constituents.

    • Availability: Where easy-to-recall events (such as natural disasters) are judged to have a high probability of occurring.

    • Hindsight bias: The tendency to overestimate the predictability of past events.

    • Anchoring effect: The tendency to rely on an anchor value that does not provide any information about the actual value.

• Biases associated with elicitation methods

  • Structuring elicitation questions: Biases may arise from how the question is framed, for example if relevant events have been omitted, experts are less likely to consider these in replying. But biases can also occur when scales are used; for example, contraction bias occurs when the full range of a scale has not been presented to the expert.

  • Elicitation medium (e.g. interview or email survey) or aggregation method: As an example, experts in group meetings (typically conducted when consensus aggregation methods are applied) tend to adopt a stronger position, often resulting in overconfident statements.

  • Fitting probability distributions: The encoding of the summaries elicited from the expert as a distribution usually implies assumptions referring to the shape of the distribution that may differ from what the expert intended. For example an expert, when fitting distributions to his/her own beliefs, may be driven by familiar probability distribution shapes, in particular bell-shaped curves (familiarity bias).

    While the literature suggests that biases cannot be completely avoided, it is good practice to be aware of possible biases and to employ strategies to mitigate against these (debiasing) in both designing and conducting the elicitation exercise.