Advertisement

Exploring the Feasibility of Comprehensive Uncertainty Assessment in Health Economic Modeling: A Case Study

Open AccessPublished:March 30, 2021DOI:https://doi.org/10.1016/j.jval.2021.01.004

      Highlights

      • Health economic models rarely systematically identify and subsequently parameterize or acknowledge all uncertainties, which can lead to biased results and hence suboptimal allocation decisions. Systematic identification and modeling of uncertainties is a step forward in addressing this issue. This article uses a case study to illustrate this type of comprehensive uncertainty assessment and explores the feasibility.
      • Existing tools were used to systematically identify uncertainties (Transparent Uncertainty Assessment Tool) and to parameterize uncertainties based on expert opinion (EXPLICIT). The Transparent Uncertainty Assessment Tool assessed the potential impact of uncertainties on the analysis results, supporting the prioritization of uncertainties for modeling. The remote Excel-based elicitation of parameters using EXPLICIT helped address unavailability and indirectness of evidence regarding treatment effectiveness, utility, and cost parameters. Further research on these tools could enhance their application for comprehensive uncertainty assessment.
      • Improvements in uncertainty assessment were feasible; however, we were not able to perform a comprehensive uncertainty assessment. Barriers to this were time and resource constraints of the research team and the involved clinical experts, and a lack of guidance regarding methodologies including the framing of expert elicitation questions, the aggregation of elicited data, the pooling of scenario analyses, and the transparent reporting. Further guidance would be helpful to support analysts performing comprehensive uncertainty assessment.

      Abstract

      Objectives

      Decision makers adopt health technologies based on health economic models that are subject to uncertainty. In an ideal world, these models parameterize all uncertainties and reflect them in the cost-effectiveness probability and risk associated with the adoption. In practice, uncertainty assessment is often incomplete, potentially leading to suboptimal reimbursement recommendations and risk management. This study examines the feasibility of comprehensive uncertainty assessment in health economic models.

      Methods

      A state transition model on peripheral arterial disease treatment was used as a case study. Uncertainties were identified and added to the probabilistic sensitivity analysis if possible. Parameter distributions were obtained by expert elicitation, and structural uncertainties were either parameterized or explored in scenario analyses, which were model averaged.

      Results

      A truly comprehensive uncertainty assessment, parameterizing all uncertainty, could not be achieved. Expert elicitation informed 8 effectiveness, utility, and cost parameters. Uncertainties were parameterized or explored in scenario analyses and with model averaging. Barriers included time and resource constraints, also of clinical experts, and lacking guidance regarding some aspects of expert elicitation, evidence aggregation, and handling of structural uncertainty. The team’s multidisciplinary expertise and existing literature and tools were facilitators.

      Conclusions

      While comprehensive uncertainty assessment may not be attainable, improvements in uncertainty assessment in general are no doubt desirable. This requires the development of detailed guidance and hands-on tutorials for methods of uncertainty assessment, in particular aspects of expert elicitation, evidence aggregation, and handling of structural uncertainty. The issue of benefits of uncertainty assessment versus time and resources needed remains unclear.

      Keywords

      Introduction

      Health economic models are frequently used to estimate the cost-effectiveness of a healthcare technology compared to its comparator(s), to quantify uncertainty in the cost-effectiveness estimate and risk associated with the adoption decision, and to indicate areas of further research.
      • Briggs A.
      • Sculpher M.
      An introduction to Markov modelling for economic evaluation.
      Developers of clinical guidelines and policy decision makers use health economic models to ensure their recommendations and adoption decisions maximize the efficiency of healthcare. Health economic models allow for uncertainties to be made explicit and to assess their impact on results. In an ideal world, health economic models identify and parameterize all uncertainties and produce one probabilistic cost-effectiveness estimate that reflects all uncertainty. On this basis, a risk estimate can be obtained, which allows for risk management. Nevertheless, in practice this is rarely the case,
      • Grimm S.E.
      • Pouwels X.
      • Ramaekers B.L.T.
      • et al.
      Development and validation of the TRansparent Uncertainty ASsessmenT (TRUST) Tool for assessing uncertainties in health economic decision models.
      leading to potentially suboptimal recommendations and risk management.
      • Grimm S.E.
      • Strong M.
      • Brennan A.
      • et al.
      The HTA Risk Analysis Chart: visualising the need for and potential value of managed entry agreements in health technology assessment.
      ,
      • Rothery C.
      • Strong M.
      • Koffijberg H.
      • et al.
      Value of information analytical methods: report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force.
      To understand reasons for which uncertainties are not appropriately reflected in health economic models, consider that health economic models reduce complex real-world circumstances to a simplified version of reality, and therefore rely on assumptions.
      • Sampson C.J.
      • Wrightson T.
      Model registration: a call to action.
      This often entails subjective decisions regarding methodology, evidence, and model structure.
      • Arnold R.J.G.
      • Ekins S.
      Time for cooperation in health economics among the modelling community.
      As a consequence, there may be alternative valid modeling choices that lead to different results—often referred to as structural uncertainty.
      • Briggs A.H.
      • Weinstein M.C.
      • Fenwick E.A.L.
      • et al.
      Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group–6.
      Uncertainty affects various elements of any given health economic model, which is importantly not limited to imprecision of the model parameter estimates. Other sources of uncertainty include indirectness, unavailability of evidence, transparency issues, and methodological issues.
      • Grimm S.E.
      • Pouwels X.
      • Ramaekers B.L.T.
      • et al.
      Development and validation of the TRansparent Uncertainty ASsessmenT (TRUST) Tool for assessing uncertainties in health economic decision models.
      These can be present in different model aspects: the scope of the model (ie, the definition of the patients, intervention, comparators, outcomes, time horizon, and perspective), the model structure (ie, health states, events, and their relationships), the evidence used ([relative] effectiveness, adverse events, utilities, and costs), the technical model implementation, and the reported outcomes.
      • Grimm S.E.
      • Pouwels X.
      • Ramaekers B.L.T.
      • et al.
      Development and validation of the TRansparent Uncertainty ASsessmenT (TRUST) Tool for assessing uncertainties in health economic decision models.
      ,
      • Briggs A.H.
      • Weinstein M.C.
      • Fenwick E.A.L.
      • et al.
      Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group–6.
      ,
      • Walker W.E.
      • Harremoës P.
      • Rotmans J.
      • et al.
      Defining uncertainty: a conceptual basis for uncertainty management in model-based decision support.
      Together, these uncertainties result in uncertain cost-effectiveness results, which is often referred to as decision uncertainty,
      • Briggs A.H.
      • Weinstein M.C.
      • Fenwick E.A.L.
      • et al.
      Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group–6.
      but is also understood as the risk associated with a decision problem.
      • Grimm S.E.
      • Strong M.
      • Brennan A.
      • et al.
      The HTA Risk Analysis Chart: visualising the need for and potential value of managed entry agreements in health technology assessment.
      A first step in the assessment of uncertainties is to identify them and make them explicit. Based on this, uncertainties can be reflected in the model. There is guidance on uncertainty assessment, but this is still falling short of comprehensively addressing all abovementioned sources of uncertainty. Well-established methods and guidelines exist for addressing issues with transparency: the Consolidated Health Economic Evaluation Reporting Standards checklist for reporting of health economic models, and the International Society for Pharmacoeconomics and Outcomes Research guidelines that address model transparency and validation.
      • Eddy D.M.
      • Hollingworth W.
      • Caro J.J.
      • et al.
      Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force–7.
      ,
      • Husereau D.
      • Drummond M.
      • Petrou S.
      • et al.
      Consolidated Health Economic Evaluation Reporting Standards (CHEERS)–7 explanation and elaboration: a report of the ISPOR Health Economic Evaluation Publication Guidelines Good Reporting Practices Task Force.
      For some countries, a reference case, that is, a generic set of best practices for modeling including the handling of different uncertainties, is available.
      Zorginstituut Nederland
      Guideline for economic evaluations in healthcare.
      ,
      National Institute for Health and Care Excellence
      Guide to the methods of technology appraisal 2013 (PMG9). Process and methods.
      Whether international or country-specific, guidelines recommend the use of probabilistic input parameters and of sensitivity analyses to address imprecision, also called statistical uncertainty.
      • Briggs A.H.
      • Weinstein M.C.
      • Fenwick E.A.L.
      • et al.
      Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group–6.
      ,
      Zorginstituut Nederland
      Guideline for economic evaluations in healthcare.
      ,
      National Institute for Health and Care Excellence
      Guide to the methods of technology appraisal 2013 (PMG9). Process and methods.
      Methods and guidelines for handling uncertainty related to bias, indirectness, and unavailability of evidence are often less detailed and focused on scenario analyses.
      • Briggs A.H.
      • Weinstein M.C.
      • Fenwick E.A.L.
      • et al.
      Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group–6.
      ,
      • Walker W.E.
      • Harremoës P.
      • Rotmans J.
      • et al.
      Defining uncertainty: a conceptual basis for uncertainty management in model-based decision support.
      ,
      Zorginstituut Nederland
      Guideline for economic evaluations in healthcare.
      National Institute for Health and Care Excellence
      Guide to the methods of technology appraisal 2013 (PMG9). Process and methods.
      • Strong M.
      • Oakley J.E.
      • Chilcott J.
      Managing structural uncertainty in health economic decision models: a discrepancy approach.
      The recent International Society for Pharmacoeconomics and Outcomes Research guideline on value of information analysis additionally recommends model averaging and expert elicitation to obtain additional, or adjust existing, parameter estimates.
      • Rothery C.
      • Strong M.
      • Koffijberg H.
      • et al.
      Value of information analytical methods: report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force.
      Indirectness, unavailability, and methodological uncertainties are considered more likely to be left out of models in practice.
      • Grimm S.E.
      • Pouwels X.
      • Ramaekers B.L.T.
      • et al.
      Development and validation of the TRansparent Uncertainty ASsessmenT (TRUST) Tool for assessing uncertainties in health economic decision models.
      ,
      • O’Hagan A.
      • Oakley J.E.
      Probability is perfect, but we can’t elicit it perfectly.
      ,
      • Bilcke J.
      • Beutels P.
      • Brisson M.
      • et al.
      Accounting for methodological, structural, and parameter uncertainty in decision-analytic models: a practical guide.
      Assessing these uncertainties can be challenging and labor-intense.
      • O’Hagan A.
      • Oakley J.E.
      Probability is perfect, but we can’t elicit it perfectly.
      ,
      • Bilcke J.
      • Beutels P.
      • Brisson M.
      • et al.
      Accounting for methodological, structural, and parameter uncertainty in decision-analytic models: a practical guide.
      The aim of this study was to explore the feasibility of comprehensive uncertainty assessment in health economic models. For this, we used a health economic model previously developed by our group on treatment of patients with peripheral arterial disease (PAD) with dual platelet inhibition therapy (DPI) with rivaroxaban and aspirin. The article describes the application of comprehensive uncertainty assessment consisting of the systematic identification of uncertainties, and the use of expert elicitation, uncertainty parameterization, scenario analyses, and model averaging. The results of the uncertainty assessment and findings regarding feasibility, barriers, and facilitators are presented.

      Methods

       Case Study: Pharmacological Treatment of PAD Cost-Effectiveness Model

      The case study modeled the costs and effects of 3 competing treatments for PAD, and the resulting incremental cost per incremental quality-adjusted life-year. The treatments were DPI with rivaroxaban plus aspirin, and aspirin and clopidogrel single antiplatelet inhibition.
      • Petersohn S.
      • Pouwels X.
      • Ramaekers B.
      • et al.
      Rivaroxaban plus aspirin for the prevention of ischaemic events in patients with cardiovascular disease: a cost-effectiveness study.
      The state transition model reflected the effects of the treatments on the probability of cardiovascular and ischemic limb events over a lifetime horizon. The effects of these diseases were modeled using health states reflective of PAD severity and tunnel states reflective of cardiovascular history. This reflected the assumption that PAD progression and cardiovascular events have lasting impacts on quality of life and healthcare consumption (see Appendix Fig 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.01.004 for the model structure). Effectiveness data used stemmed from the COMPASS trial
      • Anand S.S.
      • Bosch J.
      • Eikelboom J.W.
      • et al.
      Rivaroxaban with or without aspirin in patients with stable peripheral or carotid artery disease: an international, randomised, double-blind, placebo-controlled trial.
      comparing DPI to aspirin, and from the CAPRIE trial (a randomised, blinded trial of clopidogrel versus aspirin in patients at risk of ischaemic events)
      The CAPRIE Steering Committee
      A randomised, blinded, trial of clopidogrel versus aspirin in patients at risk of ischaemic events (CAPRIE).
      comparing clopidogrel to aspirin; and utilities stemmed from a large UK-based study.
      • Sullivan P.W.
      • Slejko J.F.
      • Sculpher M.J.
      • et al.
      Catalogue of EQ-5D scores for the United Kingdom.
      The analysis modeled drug costs, costs relating to the treatment of PAD and cardiovascular events, which stemmed from Dutch sources.
      • Oostenbrink J.B.
      • Tangelder M.J.D.
      • Busschbach J.J.V.
      • et al.
      Cost-effectiveness of oral anticoagulants versus aspirin in patients after infrainguinal bypass grafting surgery.
      • van Stel H.F.
      • Busschbach J.J.V.
      • Hunink M.G.M.
      • et al.
      Impact of secondary cardiovascular events on health status.
      • Greving J.
      • Visseren F.
      • de Wit G.
      • et al.
      Statin treatment for primary prevention of vascular disease: whom to treat? Cost-effectiveness analysis.
      • van Asselt A.D.
      • Nicolai S.P.
      • Joore M.A.
      • et al.
      Cost-effectiveness of exercise therapy in patients with intermittent claudication: supervised exercise therapy versus a ‘go home and walk’ advice.
      • van Hout B.A.
      • Simoons M.L.
      Cost-effectiveness of HMG coenzyme reductase inhibitors. Whom to treat?.
      • Spronk S.
      • Bosch J.L.
      • den Hoed P.T.
      • et al.
      Cost-effectiveness of endovascular revascularization compared to supervised hospital-based exercise training in patients with intermittent claudication: a randomized controlled trial.
      All utility, cost, and effectiveness parameters were entered into the model using uncertainty distributions. Probabilistic sensitivity analyses (PSAs) and deterministic sensitivity analyses, scenario analyses, and subgroup analyses were implemented to handle uncertainty. This model was selected as the case study, because uncertainties affected different aspects of the model and stemmed from different sources.

       Identification of Uncertainties

      Uncertainties in the model-based economic evaluation were identified with the International Society for Pharmacoeconomics and Outcomes Research TRUST tool.
      • Grimm S.E.
      • Pouwels X.
      • Ramaekers B.L.T.
      • et al.
      Development and validation of the TRansparent Uncertainty ASsessmenT (TRUST) Tool for assessing uncertainties in health economic decision models.
      It was assessed whether the uncertainties were fully reflected in the model’s uncertainty analyses, in how far existing parameters contributed to the overall uncertainty, and in how far uncertainties not fully reflected were expected to influence the incremental cost-effectiveness results (see Appendix Fig. 2 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.01.004 for the completed TRUST tool).

       Approaches to Comprehensive Uncertainty Assessment

      In accordance with best practices as described by Briggs et al,
      • Briggs A.H.
      • Weinstein M.C.
      • Fenwick E.A.L.
      • et al.
      Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group–6.
      uncertainties identified as potentially influential and not fully reflected were included in the PSA (Table 1). If not possible with the available evidence (owing to indirectness or unavailability), expert opinion was elicited to obtain (alternative) estimates and probability distributions to enable inclusion in the PSA, following guidance in the literature.
      • Grigore B.
      • Peters J.
      • Hyde C.
      • et al.
      EXPLICIT: a feasibility study of remote expert elicitation in health technology assessment.
      • Soares M.O.
      • Sharples L.
      • Morton A.
      • et al.
      Experiences of structured elicitation for model-based cost-effectiveness analyses.
      • Iglesias C.P.
      • Thompson A.
      • Rogowski W.H.
      • et al.
      Reporting guidelines for the use of expert judgement in model-based economic evaluations.
      • Bojke L.
      • Claxton K.
      • Bravo-Vergel Y.
      • et al.
      Eliciting distributions to populate decision analytic models.
      • Bojke L.
      • Grigore B.
      • Jankovic D.
      • et al.
      Informing reimbursement decisions using cost-effectiveness modelling: a guide to the process of generating elicited priors to capture model uncertainties.
      • Knol A.B.
      • Slottje P.
      • van der Sluijs J.P.
      • et al.
      The use of expert elicitation in environmental health impact assessment: a seven step procedure.
      If relevant evidence was used in the original model, this evidence was aggregated with the expert opinion. The model after inclusion of additional uncertainties in the PSA is referred to as the improved model. In line with the literature,
      • Ghabri S.
      • Cleemput I.
      • Josselin J.-M.
      Towards a new framework for addressing structural uncertainty in health technology assessment guidelines.
      ,
      • Bojke L.
      • Claxton K.
      • Sculpher M.
      • et al.
      Characterizing structural uncertainty in decision analytic models: a review and application of methods.
      uncertainty pertaining to the model structure, that is, the use of PAD health states, was explored through scenario analysis. Other model assumptions were explored using parameterization of alternative assumptions or further scenario analyses. All were integrated using model averaging; this model is referred to as the integrated model. We calculated the expected value of perfect information (EVPI) and the partial EVPI (EVPPI) of groups of parameters, calculated with the Sheffield accelerated value of information framework.
      • Strong M.
      • Oakley J.E.
      • Brennan A.
      Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach.
      Table 1Uncertainties and assessment approaches.
      Source of uncertaintyDescription of uncertaintyOriginal approachNew approach
      Improved modelScenarios (averaged in integrated model)
      Model structure
      Methodological issuesNo reference case for PAD model structureModel structure with mild, moderate, and severe PAD health statesScenario 1:

      Model structure without PAD health states, progression modeled as events
      Transition probabilities
      UnavailabilityIndividual TTE data for (non)fatal CV events were lacking.Proportion of fatal events increases from age 75 to 85 to 100% (deterministic).Probabilistic modeling of the shift
      Methodological issuesNo reference case for model structuresA proportion of patients had moderate PAD at baseline (COMPASS trial
      National Institute for Health and Care Excellence
      Guide to the methods of technology appraisal 2013 (PMG9). Process and methods.
      data, deterministic).
      Probabilistic modeling of the proportion.
      UnavailabilityLack of TTE data for ALI and CLIEqual progression risk upon ALI and CLI (probabilistic).

      Proportions of ALI and CLI within endpoint MALE constant over time.
      Scenario 2:

      Individual progression risks.

      Scenarios 3 and 4:

      Proportions of ALI and CLI decrease and increase over time.
      Relative effectiveness
      IndirectnessSource of aspirin and rivaroxaban effectiveness data provides no clopidogrel data.HR of clopidogrel vs aspirin from the CAPRIE trial was used.
      The CAPRIE Steering Committee
      A randomised, blinded, trial of clopidogrel versus aspirin in patients at risk of ischaemic events (CAPRIE).
      Elicitation of expert opinion on the effectiveness of clopidogrel on MACE (Q1)
      UnavailabilityNo evidence available regarding effectiveness of clopidogrel on MALEEffectiveness of clopidogrel was assumed equal to effectiveness of aspirin.Elicitation of expert opinion on the effectiveness of clopidogrel on MALE (Q2)
      Adverse events
      UnavailabilityNo evidence available regarding bleeding risks of clopidogrelBleeding risks of clopidogrel were assumed equal to ticagrelor.Elicitation of expert opinion on bleeding risk of clopidogrel (Q3).
      Methodological issuesViolation of best practice; no TTE data available regarding bleedsBleeds were modeled with a constant probability over time.Bleeding risk increases with age based on COMPASS subgroup data.
      Utilities
      IndirectnessNo utility data were available from the main trial.Utilities from literature2 sets of utilities from literature for mild and moderate PAD are randomly used in the PSA.
      • Petersohn S.
      • Ramaekers B.L.T.
      • Olie R.H.
      • et al.
      Comparison of three generic quality-of-life metrics in peripheral arterial disease patients undergoing conservative and invasive treatments.
      UnavailabilityNo utility data were available for health states with multiple diseases (eg, PAD and MI).PAD health state utility and event disutility were assumed to be additive.Elicitation of expert opinion on PAD patients’ loss of QoL upon acute and post-acute CV events (Q4-7)
      Costs
      Indirectness, imprecisionNo cost data were available from the main trial.PAD health state costs stemmed from other publications, unknown SE imputed as 25% of the mean.Elicitation of expert opinion on health state costs of severe PAD (Q8)
      UnavailabilityIschemic and hemorrhagic stroke costs not available, instead major and minor stroke costs.All strokes were assumed to be 50% major and minor strokes (deterministic).Hemorrhagic strokes can be 10% more severe (probabilistic).
      UnavailabilityCosts for health states entailing multiple diseases not available.PAD health state and event costs were assumed to be additive.2 sets of costs (additive and not additive) randomly used in the PSA.
      ALI indicates acute limb ischemia; CAPRIE, a randomised, blinded trial of clopidogrel versus aspirin in patients at risk of ischaemic events; CLI, chronic limb ischemia; CV, cardiovascular; HR, hazard ratio; HS, hemorrhagic stroke; IS, ischemic stroke; MACE, major adverse cardiovascular event (fatal or non-fatal MI, IS, or HS); MALE, major adverse limb event (ALI, CLI, and major vascular amputation); MI, myocardial infarction; PAD, peripheral arterial disease; PSA, probabilistic sensitivity analysis; Q1-8, questions 1 to 8 of the expert elicitation; QoL, quality of life; SE, standard error; TTE, time-to-event.

       Elicitation of Expert Beliefs

      This section summarizes the expert elicitation methods in line with reporting guidelines.
      • Iglesias C.P.
      • Thompson A.
      • Rogowski W.H.
      • et al.
      Reporting guidelines for the use of expert judgement in model-based economic evaluations.
      The elicitation protocol is available in the Appendix (see Appendix in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.01.004).

       Expert selection

      Experts were selected with diverse backgrounds that could provide different perspectives on the elicitation questions asked, and to have tangible knowledge in the disease area.
      • Bojke L.
      • Grigore B.
      • Jankovic D.
      • et al.
      Informing reimbursement decisions using cost-effectiveness modelling: a guide to the process of generating elicited priors to capture model uncertainties.
      Vascular surgeons, cardiologists, internal medicine specialists, vascular medicine specialists, general practitioners, and vascular surgery nurses treating PAD patients in Dutch hospitals were considered eligible (see the Appendix in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.01.004 for details about the selection of experts).

       Quantities elicited

      We elicited 8 quantities in 8 elicitation questions (Q1-8, see Table 1 and Appendix Table 2 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.01.004). Some model parameters of interest could not be elicited owing to their conceptual complexity, for example, relative risks. Instead, observable and intuitive values were elicited.
      • Knol A.B.
      • Slottje P.
      • van der Sluijs J.P.
      • et al.
      The use of expert elicitation in environmental health impact assessment: a seven step procedure.
      To keep the cognitive complexity of the elicitation exercise to a minimum, questions were kept as similar as possible regarding the types of quantities asked: proportions on a scale from 0 to 1, and one absolute number were elicited.
      • Bojke L.
      • Grigore B.
      • Jankovic D.
      • et al.
      Informing reimbursement decisions using cost-effectiveness modelling: a guide to the process of generating elicited priors to capture model uncertainties.

       Elicitation methods

      The elicitation of all quantities was conducted with the EXPLICIT tool, which enabled remote elicitation without researcher support.
      • Grigore B.
      • Peters J.
      • Hyde C.
      • et al.
      EXPLICIT: a feasibility study of remote expert elicitation in health technology assessment.
      Multiple coauthors reviewed the instrument and wording of questions, and 2 experts pilot-tested the tool.
      • Knol A.B.
      • Slottje P.
      • van der Sluijs J.P.
      • et al.
      The use of expert elicitation in environmental health impact assessment: a seven step procedure.
      The experts received a background information package including key literature on PAD treatment and information about biases and heuristics. For each quantity elicited, the tool elicited the lower limit, the upper limit, and the mode, that is, the most likely estimate, concepts shown to be widely understood by clinical experts.
      • Soares M.O.
      • Sculpher M.J.
      • Claxton K.
      Health opportunity costs: assessing the implications of uncertainty using elicitation methods with experts.
      A beta PERT distribution was fitted and presented. The estimates could be adjusted subsequently, including the possibility to create flatter or more pointed curves reflecting the level of certainty. Experts were encouraged to skip questions they felt they did not have expert knowledge about.

       Validation

      The elicited values were compared with plausible ranges and checked for inconsistency with comments made in comment boxes. Responses seemingly based on misunderstanding of the elicitation exercise were excluded based on consensus among the authors, or when possible, experts were re-contacted to validate and potentially re-elicit their estimates.

       Pooling of expert beliefs

      The literature reports different methods for pooling expert beliefs, with none of the methods being presented as best practice.
      • Bojke L.
      • Claxton K.
      • Bravo-Vergel Y.
      • et al.
      Eliciting distributions to populate decision analytic models.
      • Bojke L.
      • Grigore B.
      • Jankovic D.
      • et al.
      Informing reimbursement decisions using cost-effectiveness modelling: a guide to the process of generating elicited priors to capture model uncertainties.
      • Knol A.B.
      • Slottje P.
      • van der Sluijs J.P.
      • et al.
      The use of expert elicitation in environmental health impact assessment: a seven step procedure.
      We applied linear pooling, a commonly used method with clear assumptions; no relationship is assumed between the expert responses—this results in wider confidence intervals and more uncertainty. Each of the 4 parameters of the beta PERT distribution, the lower and upper limit, the mode, and the shape parameter, were aggregated using equal weights per expert, that is, averaging.

       Aggregation of expert opinion and evidence from the literature

      The probability distribution elicited for Q1 (effectiveness of clopidogrel on major adverse cardiovascular event [MACE]) was aggregated with evidence from the literature (effectiveness data from the CAPRIE trial
      The CAPRIE Steering Committee
      A randomised, blinded, trial of clopidogrel versus aspirin in patients at risk of ischaemic events (CAPRIE).
      ). The pooled beta PERT distribution of events estimated by the experts (clopidogrel) and the number of events observed in the COMPASS trial (aspirin)
      • Anand S.S.
      • Bosch J.
      • Eikelboom J.W.
      • et al.
      Rivaroxaban with or without aspirin in patients with stable peripheral or carotid artery disease: an international, randomised, double-blind, placebo-controlled trial.
      were used to obtain the expert-elicited relative risk (RR) distribution; the RR distribution (beta PERT) was log-transformed for normality and multiplied with the normal distribution of the RR estimated from the CAPRIE trial
      The CAPRIE Steering Committee
      A randomised, blinded, trial of clopidogrel versus aspirin in patients at risk of ischaemic events (CAPRIE).
      to obtain the posterior distribution of RR.

       Scenario Analyses and Model Averaging

      Scenario analyses exploring alternative assumptions relating to the methodology and assumptions of the PAD health states were performed conditional on the improved model (see Table 1). The results were model-averaged with the improved model using equal weights (20% for each scenario and the improved model), resulting in the integrated model.
      • Rothery C.
      • Strong M.
      • Koffijberg H.
      • et al.
      Value of information analytical methods: report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force.

       Exploring the Impact of Uncertainties on Cost-Effectiveness Analysis Results

      To evaluate the impact of the assessed uncertainties, we compared results of the original, improved, and integrated models. Incremental cost-effectiveness ratios and planes, cost-effectiveness acceptability curves, EVPI and EVPPI per patient, and the EVPI accrued for the Dutch population over 5 years at a threshold of 50 000€ per quality-adjusted life-year are presented for the 3 models.

       Feasibility of Uncertainty Assessment

      The research team discussed potential barriers and facilitators within each step and recorded approximate time and expertise needed per step. Feedback by the experts and meeting notes were also taken into account.

      Results

       Uncertainty Assessment in This Case

      The steps of uncertainty assessment undertaken in this case study, the expertise of people involved, as well as observed barriers and facilitators are presented in Table 2 and described in the following.
      Table 2Summary of the tasks, expertise needed, and barriers and facilitators identified during the uncertainty assessment.
      StepApproachExpertiseBarriersFacilitatorsOutcome
      1. Uncertainty identificationCompletion of the TRUST toolModel developers, clinical expertComplexity of the model, number of uncertainties to considerTRUST tool helpful in systematic identificationThe initial framework of the TRUST tool was extended with additional rows to allow for the description of all uncertainties.
      2. Inclusion and assessment approaches for identified uncertaintiesSeeking consensus on uncertainties assessed and methods usedModel developersSome methodologies were not feasible owing to limited time available (eg, conducting a systematic literature review).Available methods for different uncertainties, including for structural uncertaintiesPragmatic assessment choices were made, affected by the perceived relevance of uncertainty. This can introduce bias and favor an incomplete uncertainty assessment.
      3. Expert elicitationSelection of the elicitation toolModel developers, health economist experienced with expert elicitationChoice of elicitation tools and literature on how to design protocols are still limited.Good documentation on EXPLICIT toolThe EXPLICIT tool appeared to be a good choice for remote elicitation.
      Designing of the elicitation questionsLittle guidance on framing of questions—this may influence the use of heuristics by experts.

      Number of questions limited by acceptable burden to experts
      Literature on heuristics and biasesBackground information was provided to facilitate a correct understanding of the questions, while avoiding information that would potentially introduce bias, for example, anchor and adjustment bias. For one question, its framing did introduce a bias, which was then resolved by a second round of elicitation.

      To reduce complexity for experts, elicited quantities were defined in a way that they were knowable. This meant that quantities had to be transformed to be used in the model. Not all uncertain quantities were elicited to reduce burden on experts.
      Expert elicitation piloting and amendment of protocolHealth economist experienced with expert elicitation, clinical expert, pilot testersComplementing areas of expertise within the multidisciplinary research team, literature on expert elicitationEnabled an assessment of the elicitation exercise from clinical and methodological standpoints
      Invitation of expertsClinical experts, model developersLimited number of experts in the field; 6 experts is considered the minimum.Access to the network of collaborating cliniciansIt can be challenging to reach a sufficient number and diversity of available experts.
      Remote elicitation processClinical expertsLack of expert-analyst interaction may pose problems when clinical experts have questions or misinterpret a question.EXPLICIT tool allows remote elicitation, which was feasible for all experts involved in this exercise, was relatively fast and reduced time involved by analyst. Parameterization of beta PERT using lowest and highest values and mode is more easily understandable for clinical experts than other methods (eg, tertiles).Elicitations took an average of 17 minutes per expert (7-33 minutes). Five experts indicated the elicitation exercise was difficult; no questions were skipped.

      Comment boxes were scarcely used; the motivations behind the answers remained unclear. This complicated the plausibility and consistency screening as previously acknowledged.
      • Grigore B.
      • Peters J.
      • Hyde C.
      • et al.
      EXPLICIT: a feasibility study of remote expert elicitation in health technology assessment.


      Two experts created unexpected triangular distributions with the mode at the extreme end; these were not excluded as the experts created, saw, and confirmed the curves. Such decisions can introduce bias.
      Processing and validation of elicitation resultsModel developers, clinical experts, health economist experienced with expert elicitationSubjectivity in the identification of implausible or inconsistent responses

      Expert opinion voiced during the validation might not reflect the opinion of all experts.
      Teamwork in the plausibility and consistency screening could mitigate potential biases induced by subjectivity.5.2% of answers were excluded from the analysis; 2 answers were re-elicited by the expert upon identification of a misunderstanding in a validation exercise. Bias cannot be ruled out as the exclusion was not objective, and the validation was conducted in a subset of participating experts.
      Evidence aggregationHealth economist experienced with expert elicitation, model developersLimited availability of hands-on guidance on methodologies for pooling of expert opinion and aggregation of expert opinion and other evidenceLiterature on methods for pooling and aggregation nevertheless helpfulPooling with equal and unequal weights and random-effects meta-analysis was explored. Assumptions and weaknesses were considered.
      • Soares M.O.
      • Sharples L.
      • Morton A.
      • et al.
      Experiences of structured elicitation for model-based cost-effectiveness analyses.
      It remained unclear which methodology was preferred.

      Aggregation with Bayesian updating was considered but discarded owing to its complexity.
      • Spiegelhalter D.
      • Abrams K.
      • M J.
      Bayesian Approaches to Clinical Trials and Health-Care Evaluation.
      Alternative approaches may have led to different results.
      4. Integrating updated estimates in cost-effectiveness analysisAdjusting the PSA, implementing scenarios and parameters based on aggregated expert opinionModel developersThere is a lack of formal guidance, hands-on tutorials, and best practices, especially concerning structural uncertainty.Literature on structural uncertainty was helpful.Scenarios could have been aggregated and weighted differently. Alternative approaches, such as discrepancy modeling approaches, may have led to different results.
      Value of information analysisComputational burden and analysis timeMethods deemed easy to understand and likely to be used by other modelers were selected.Alternative analyses could have been conducted.
      5. ReportingReflecting assessment methods and presenting relevant resultsModel developersThere is a lack of formal guidance to the transparent and concise reporting of uncertainty assessment.General guidelines on reporting of cost-effectiveness analysis were helpful.The transparent reporting of methods and their justifications was complicated by the complexity of the underlying considerations, which can be perceived as strategic.
      PSA indicates probabilistic sensitivity analysis; TRUST, TRansparent Uncertainty ASsessmenT.

       Identification of uncertainty

      Uncertainties that were not fully reflected in the original model and identified as potentially influential are presented in Table 1. The TRUST tool was helpful in identifying uncertainties because it allowed going through all model aspects systematically.

       Selection of approaches to assess influential uncertainties

      We disregarded uncertainties that were not fully reflected but not likely influential, such as searches not being systematic or exhaustive in identifying evidence. The research team felt that incorporating these uncertainties would not be feasible owing to time constraints. Table 1 summarizes the approaches taken for each identified uncertainty and the improved and integrated model (see Appendix Table 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.01.004 for details).

       Expert elicitation

      The elicitation tool was chosen based on available documentation and possibility for remote elicitation. The EXPLICIT tool was easy to use for the research team and the experts. Few tools were available and drawbacks with other tools that are well documented (Sheffield Elicitation Framework) include that remote elicitation may be more challenging (as per our perception because this is programmed in R). Nevertheless, the EXPLICIT tool is less flexible, because it currently only supports the beta PERT distribution.
      The protocol design posed challenges to enable quantities to be elicited in a way experts think about them, rather than in the way that they are incorporated in the model (number of events in a sample were elicited rather than RRs). Furthermore, while literature on heuristics and potential biases exists, there is no detailed guidance on how to frame elicitation questions to avoid these. Sufficient time and team members need to be budgeted in for the designing of an elicitation protocol to make sure background information and questions do not induce bias.
      Despite careful phrasing of elicitation questions, we realized that 1 question (about MACE and major adverse limb events with clopidogrel treatment) was too open for experts to give their best responses. The way experts activated their knowledge about MACE and major adverse limb events with clopidogrel treatment was in comparison to the same events with aspirin treatment. We did not provide this reference point in the question, and this led to experts considering their own responses invalid when they were presented with the evidence for aspirin in an attempt at validating their responses. When eliciting expert beliefs to inform an RR, it is therefore advisable to either include available background information on the comparator—and therefore stimulate anchor and adjustment, thus risking biased responses—or to elicit events with both treatments, which is the approach we chose (see Aggregation of evidence section for more detail).
      Of 21 invited experts, 12 participated in the elicitation exercise within a week of invitation or after a reminder sent 2 weeks after the invitation. Four respondents were vascular surgeons, 2 internal medicine specialists, 3 vascular medicine specialists, 1 general practitioner, and 2 physician assistants in vascular surgery. All experts worked in the south of The Netherlands in 3 different hospitals. Their years of experience with PAD patients varied between 2 and 30 years, with a mean of 14 years. Information saturation was reached with the addition of expert 12, changing the pooled estimate little to not at all for all elicited values except for Q8. The individual and pooled distributions obtained are presented in Figure 1.
      Figure thumbnail gr1
      Figure 1Individual and pooled probability distributions. Distributions obtained from expert elicitation. See (in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.01.004) for the values elicited.
      Experts can only reasonably be asked a limited number of questions before time runs out or attention span and patience wear. Therefore, it may not be feasible to elicit all quantities from a limited number of experts, also considering that a minimum of 6 experts is required for each question.
      • Knol A.B.
      • Slottje P.
      • van der Sluijs J.P.
      • et al.
      The use of expert elicitation in environmental health impact assessment: a seven step procedure.
      The analysis of elicitation results was complicated by the beta PERT distribution, which uses different parameters than the beta distribution, challenging the incorporation with existing evidence. Nevertheless, this distribution is intuitive for respondents.
      • O’Leary R.A.
      • Low-Choy S.
      • Fisher R.
      • et al.
      Characterising uncertainty in expert assessments: encoding heavily skewed judgements.
      Despite this, 2 experts provided answers that were not in line with our own expectations, with modes being located at the extreme ends of their own ranges. These responses were not excluded, because the experts confirmed their distributions.
      Remote elicitation poses particular challenges, because experts do not have easy opportunity for clarifying any issues, and misinterpretations are therefore more likely. Comment boxes were provided but experts hardly made use of them. Perhaps this could be encouraged more. The advantage of remote elicitation was that turnaround was fast and time involvement of the researcher can be minimized.

       Aggregation of evidence

      Limited availability of hands-on guidance on methodologies for pooling of expert opinion and aggregation of expert opinion and other evidence made this challenging. We performed linear pooling with equal and unequal weights as well as random-effects meta-analyses, which were associated with different results, but it was unclear which method was preferable. We considered Bayesian updating as too complex and did not perform this, but any practical tutorials may have proved us wrong. Table 3 shows the pooled expert opinion, trial evidence (CAPRIE),
      The CAPRIE Steering Committee
      A randomised, blinded, trial of clopidogrel versus aspirin in patients at risk of ischaemic events (CAPRIE).
      and posterior mean, as well as results from the validation study: interviews with 2 experts revealed that the obtained RRs in Q1 and Q2 were considered inconsistent with underlying beliefs that clopidogrel treatment was superior to aspirin treatment. In response, the alternative approach to estimating the relative treatment effect was based on elicited number of events with clopidogrel and aspirin treatment. The number of events in the aspirin arm was elicited from 5 experts and the RR distribution was calculated from the beta PERT distributions of events and subsequently pooled with the RR observed in the CAPRIE trial
      The CAPRIE Steering Committee
      A randomised, blinded, trial of clopidogrel versus aspirin in patients at risk of ischaemic events (CAPRIE).
      for Q1 (see Appendix Table 4 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.01.004 for the elicited values). The validated estimates were used in the improved model (see Appendix Fig. 3 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.01.004 for the prior, likelihood, and posterior distributions).
      Table 3Posterior distributions, RR of clopidogrel vs aspirin.
      Q1 MACEQ2 MALE
      Original pooled expert opinion2.5045.198
      Trial evidence (CAPRIE
      The CAPRIE Steering Committee
      A randomised, blinded, trial of clopidogrel versus aspirin in patients at risk of ischaemic events (CAPRIE).
      )
      0.776NA
      Original posterior0.949
      Validated pooled expert opinion0.8501.169
      Validated posterior0.777
      Note. Original pooled expert opinion was not used in line with expert opinion that it did not have face validity. The validated posterior was elicited in response and used in the analysis.
      MACE indicates major adverse cardiovascular event (fatal or non-fatal myocardial infarction, ischemic stroke, or hemorrhagic stroke); MALE, major adverse limb event (acute limb ischemia, chronic limb ischemia, and major vascular amputation); RR, relative risk.

       Impact on cost-effectiveness results

      There is a lack of formal guidance by health technology assessment (HTA) authorities on how they wish to have uncertainties explored beyond incorporation in the PSA. For example, it remains unclear whether it is preferable to present scenarios separately or model averaged. We present cost-effectiveness results in Table 4, for the original, improved, and integrated models (see Appendix Table 5 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.01.004 for individual results of scenarios 1-4). Figure 2 shows the incremental cost-effectiveness planes and the cost-effectiveness acceptability curves of the 3 models. The improved and integrated models showed increased ICERs, and reduced QALYs and costs for all treatments compared to the original model, presumably owing to use of 2 sets of cost and utility parameters in the improved and integrated models. The cost-effectiveness probability of DPI was increased relative to the original model and the EVPI of the improved model was reduced, while the EVPI of the integrated model was closer to the original value (see Appendix in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.01.004 for EVPI at different willingness-to-pay thresholds). This reduction in EVPI and increased cost-effectiveness probability of rivaroxaban in the improved model may be reflective of reduced decision uncertainty, whereas the additional inclusion of structural uncertainty in the integrated model increased the decision uncertainty and the EVPI. The EVPPI for groups of parameters was highest for utilities in the original model, whereas in the improved and integrated models, the EVPPIs of utilities, hazard ratios of clopidogrel, and costs were similar. This underlines that uncertainty relating to the relative effectiveness of clopidogrel and costs was reflected better in the improved and integrated models.
      Table 4Probabilistic results.
      LYsQALYsCostsIncrementsProbability CE (%)
      Considering a willingness-to-pay threshold of €50 000/QALY.
      EVPI Dutch population, over 5 years
      Considering a willingness-to-pay threshold of €50 000/QALY.
      ,
      Considering a PAD population of 227 499 patients.
      EVPI per patientEVPPI per patient
      Considering a willingness-to-pay threshold of €50 000/QALY.
      LYsQALYsCostsICERtransition probabilitiesUtilitiesHRs clopidogrelCosts
      Original results€2.7 billion€2351€11€908€0€154
      Aspirin11.667.49€156 854cheapest7.8
      Clopidogrel12.117.76€165 1860.450.28€8332ext. domin.34.3
      DPI12.137.88€166 9630.470.39€10 109€26 22157.9
      Improved model, incorporating uncertainties in the PSA€1.4 billion€1243€0€265€342€199
      Aspirin11.587.40€154 388cheapest7.3
      Clopidogrel12.047.67€164 5320.460.28€10 144ext. domin.23.7
      DPI12.067.82€165 8040.480.42€11 416€27 18069.1
      Integrated model (averaging improved model and scenarios 1-4)€1.8 billion€1549€1€361€400€350
      Aspirin11.547.40€152 400cheapest8.7%
      Clopidogrel12.017.69€162 5730.470.28€10 173ext. domin.27.5%
      DPI12.037.82€164 2290.490.41€11 829€28 69863.8%
      CE indicates cost-effectiveness; DPI, dual pathway inhibition; ext. domin., extendedly dominated; EVPI, expected value of perfect information; HR, hazard ratio; ICER, incremental cost-effectiveness ratio; LYs, life-years; PAD, peripheral arterial disease; QALYs, quality-adjusted life-years.
      Considering a willingness-to-pay threshold of €50 000/QALY.
      Considering a PAD population of 227 499 patients.
      Figure thumbnail gr2
      Figure 2Analyses results: (A) incremental cost-effectiveness planes and (B) cost-effectiveness acceptability curves.

      Discussion

      This article explored the feasibility of comprehensive uncertainty assessment in health economic modeling. The uncertainties identified in the case study on the treatment of patients with peripheral arterial disease with DPI
      • Petersohn S.
      • Pouwels X.
      • Ramaekers B.
      • et al.
      Rivaroxaban plus aspirin for the prevention of ischaemic events in patients with cardiovascular disease: a cost-effectiveness study.
      stemmed from imprecision, unavailability or indirectness of evidence, and methodological issues. Expert elicitation and scenario analyses with model averaging were used to model uncertainties. This caused an increase of the ICER and cost-effectiveness probability, and a decrease of the EVPI, whereas the EVPPI of utilities decreased, and of HRs of clopidogrel and costs increased. Although this article demonstrates that improvements in uncertainty assessment were feasible, we were not able to perform comprehensive uncertainty assessment. The main barriers to this were time and resource constraints on the part of our research team and on the part of involved clinical experts. Furthermore, a lack of guidance and hands-on tutorials regarding key methodologies, including the framing of expert elicitation questions, the aggregation of elicited data, the pooling of scenario analyses, and the transparent reporting, caused difficulty in performing these tasks. Perceived facilitators were the use of existing tools for the systematic identification of uncertainties and elicitation of expert opinion, the literature on expert elicitation, handling of structural uncertainty and value of information, and the complementing expertise within the research team.

       Implications

      This study adds to the literature on uncertainty identification and assessment by exploring the feasibility of comprehensive uncertainty assessment by means of different recommended methods.
      • Grimm S.E.
      • Pouwels X.
      • Ramaekers B.L.T.
      • et al.
      Development and validation of the TRansparent Uncertainty ASsessmenT (TRUST) Tool for assessing uncertainties in health economic decision models.
      ,
      • Briggs A.H.
      • Weinstein M.C.
      • Fenwick E.A.L.
      • et al.
      Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group–6.
      ,
      • Eddy D.M.
      • Hollingworth W.
      • Caro J.J.
      • et al.
      Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force–7.
      ,
      • Ghabri S.
      • Cleemput I.
      • Josselin J.-M.
      Towards a new framework for addressing structural uncertainty in health technology assessment guidelines.
      ,
      • Claxton K.
      Exploring uncertainty in cost-effectiveness analysis.
      To our knowledge, it is the first of its kind employing and demonstrating a range of methods with the aim of exploring the feasibility of comprehensive uncertainty assessment in a case study. The obtained insights may provide practical support to analysts who encounter similar challenges, and provide an impetus for further research into methods, tools, and hands-on tutorials that can facilitate uncertainty assessment within HTA. As for the case study, EVPI and EVPPI analyses indicated that further research may be indicated on utilities, the effectiveness of clopidogrel, as well as costs. To establish the value of research, further evaluation of the EVSI and net gain of the research versus its cost would be necessary.
      • Willan A.R.
      • Eckermann S.
      Optimal clinical trial design using value of information methods with imperfect implementation.
      There should be no doubt that a good understanding of uncertainty is desirable in health-technology decision making. Of course, the benefit of this understanding depends on the magnitude of the risk associated with an assessment and the quality of the uncertainty assessment and is therefore difficult to quantify precisely and generalize. Ideally, HTA authorities would require more uncertainty assessment from companies that prepare reimbursement applications or analysts preparing health economic models because they are best placed to steer the identification and assessment of uncertainties. Time and resources required for a good-quality uncertainty assessment may be significant but depend heavily on the expertise of the involved analysts and the model and condition complexity. Also, good-quality uncertainty assessments are still hampered by a lack of guidance and hands-on tutorials. Comprehensive uncertainty assessment may be challenging or even impossible: (1) owing to resource and time constraints on the part of clinical experts, and (2) some uncertainties may not even be known to analysts or clinical experts (so-called unknown unknowns
      • Grutters J.P.
      • van Asselt M.B.
      • Chalkidou K.
      • et al.
      Healthy decisions: towards uncertainty tolerance in healthcare policy.
      or total ignorance
      • Ghabri S.
      • Hamers F.F.
      • Josselin J.M.
      Exploring uncertainty in economic evaluations of drugs and medical devices: lessons from the first review of manufacturers’ submissions to the French National Authority for Health.
      ). Nevertheless, in our opinion comprehensive uncertainty identification should be a prerequisite for good quality health technology assessment. And the selection of uncertainties for assessment should be fully transparent and justified.

       Strengths and Weaknesses

      One limitation of this study is that only 1 case study was used to examine the feasibility of comprehensive uncertainty assessment. Using multiple health economic models with different characteristics, and developed by different model developers, may have added further experiences and insights. Furthermore, our team’s experience and expertise shaped our feasibility assessment, which is therefore somewhat subjective. We acknowledge that our findings are not exhaustive but highlight particular challenges and need for more guidance in the field of uncertainty assessment that other researchers may also encounter. The analysis resulted in important implications for further research as well as practical recommendations for other researchers attempting comprehensive uncertainty assessment. Secondly, alternative methods to the ones chosen by us were available, for example, instead of the EXPLICIT
      • Grigore B.
      • Peters J.
      • Hyde C.
      • et al.
      EXPLICIT: a feasibility study of remote expert elicitation in health technology assessment.
      expert elicitation tool, the SHELF tool

      O’Hagan A, Oakley J. SHELF: the Sheffield Elicitation Framework.

      could have been chosen, which is also frequently used in the field of health economic modeling. We chose to use the EXPLICIT
      • Grigore B.
      • Peters J.
      • Hyde C.
      • et al.
      EXPLICIT: a feasibility study of remote expert elicitation in health technology assessment.
      tool owing to its ability to support remote elicitation. Instead of model averaging, we could have explored the model discrepancy approach for modeling structural uncertainty.
      • Strong M.
      • Oakley J.E.
      • Chilcott J.
      Managing structural uncertainty in health economic decision models: a discrepancy approach.
      The advantage of this is that it does not assume that the true model is captured by the model so far. Nevertheless, it appears to be more difficult and time-consuming in its implementation and potentially requires further expert elicitation to quantify discrepancies between the model and the real world. Further guidance on the choice of analytical approach to structural uncertainty would be most helpful.
      • Ghabri S.
      • Cleemput I.
      • Josselin J.-M.
      Towards a new framework for addressing structural uncertainty in health technology assessment guidelines.
      We used Sheffield accelerated value of information for EVPPI analysis because we were familiar with it and it was convenient. Other tools are available that may offer greater precision and gains in computational efficiency.
      • Kunst N.
      • Wilson E.C.F.
      • Glynn D.
      • et al.
      Computing the expected value of sample information efficiently: practical guidance and recommendations for four model-based methods.
      The use of alternative tools and methods may result in alternative challenges and maybe even different results. This should be explored in future research. Furthermore, for time and resource constraints, we did not perform expected value of sample information analysis to investigate the value of further research on specific model parameters. This could be considered for further research.

      Conclusions

      Comprehensive uncertainty assessment in health economic models, that is, systematic identification and modeling of uncertainties, promises improved reimbursement decision making and risk management. Barriers to its routine use are time and resource constraints and a lack of guidance regarding methodologies used, such as the prioritization of uncertainties to model, the avoidance of bias and heuristics in framing expert elicitation questions, and the aggregation of elicited data. To facilitate the routine use of comprehensive uncertainty assessment, the use of existing tools for uncertainty identification and expert elicitation should be considered, and further research should be conducted on key methodologies. The issue of benefits of uncertainty assessment versus time and resources needed remains unclear.

      Article and Author Information

      Author Contributions: Concept and design: Petersohn, Grimm, Ramaekers, Joore
      Acquisition of data: Petersohn
      Analysis and interpretation of data: Petersohn, Grimm, Ramaekers, Joore
      Drafting of the manuscript: Petersohn, Grimm, ten Cate-Hoek
      Critical revision of the paper for important intellectual content: Petersohn, Grimm, Ramaekers, ten Cate-Hoek, Joore
      Statistical analysis: Petersohn
      Provision of study materials or patients: ten Cate-Hoek
      Obtaining funding: ten Cate-Hoek
      Supervision: Grimm, Ramaekers, ten Cate-Hoek, Joore
      Conflict of Interest Disclosures: The authors reported no conflicts of interest.
      Funding/Support: The authors received no financial support for this research.

      Acknowledgment

      Renske Olie, MD, helped testing the pilot version of the elicitation tool. Twelve experts participated in the expert elicitation and shared their knowledge with us.

      Supplemental Material

      References

        • Briggs A.
        • Sculpher M.
        An introduction to Markov modelling for economic evaluation.
        Pharmacoeconomics. 1998; 13: 397-409
        • Grimm S.E.
        • Pouwels X.
        • Ramaekers B.L.T.
        • et al.
        Development and validation of the TRansparent Uncertainty ASsessmenT (TRUST) Tool for assessing uncertainties in health economic decision models.
        Pharmacoeconomics. 2020; 38: 205-216
        • Grimm S.E.
        • Strong M.
        • Brennan A.
        • et al.
        The HTA Risk Analysis Chart: visualising the need for and potential value of managed entry agreements in health technology assessment.
        Pharmacoeconomics. 2017; 35: 1287-1296
        • Rothery C.
        • Strong M.
        • Koffijberg H.
        • et al.
        Value of information analytical methods: report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force.
        Value Health. 2020; 23: 277-286
        • Sampson C.J.
        • Wrightson T.
        Model registration: a call to action.
        Pharmacoecon Open. 2017; 1: 73-77
        • Arnold R.J.G.
        • Ekins S.
        Time for cooperation in health economics among the modelling community.
        Pharmacoeconomics. 2010; 28: 609-613
        • Briggs A.H.
        • Weinstein M.C.
        • Fenwick E.A.L.
        • et al.
        Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group–6.
        Med Decis Making. 2012; 32: 722-732
        • Walker W.E.
        • Harremoës P.
        • Rotmans J.
        • et al.
        Defining uncertainty: a conceptual basis for uncertainty management in model-based decision support.
        Integrated Assessment. 2003; 4: 5-17
        • Eddy D.M.
        • Hollingworth W.
        • Caro J.J.
        • et al.
        Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force–7.
        Med Decis Making. 2012; 32: 733-743
        • Husereau D.
        • Drummond M.
        • Petrou S.
        • et al.
        Consolidated Health Economic Evaluation Reporting Standards (CHEERS)–7 explanation and elaboration: a report of the ISPOR Health Economic Evaluation Publication Guidelines Good Reporting Practices Task Force.
        Value Health. 2013; 16: 231-250
        • Zorginstituut Nederland
        Guideline for economic evaluations in healthcare.
        2016
        • National Institute for Health and Care Excellence
        Guide to the methods of technology appraisal 2013 (PMG9). Process and methods.
        National Institute for Health and Care Excellence, 2013
        • Strong M.
        • Oakley J.E.
        • Chilcott J.
        Managing structural uncertainty in health economic decision models: a discrepancy approach.
        J R Stat Soc Ser C Appl Stat. 2012; 61: 25-45
        • O’Hagan A.
        • Oakley J.E.
        Probability is perfect, but we can’t elicit it perfectly.
        Reliab Eng Syst Saf. 2004; 85: 239-248
        • Bilcke J.
        • Beutels P.
        • Brisson M.
        • et al.
        Accounting for methodological, structural, and parameter uncertainty in decision-analytic models: a practical guide.
        Med Decis Making. 2011; 31: 675-692
        • Petersohn S.
        • Pouwels X.
        • Ramaekers B.
        • et al.
        Rivaroxaban plus aspirin for the prevention of ischaemic events in patients with cardiovascular disease: a cost-effectiveness study.
        Eur J Prev Cardiol. 2020; 27: 1354-1365
        • Anand S.S.
        • Bosch J.
        • Eikelboom J.W.
        • et al.
        Rivaroxaban with or without aspirin in patients with stable peripheral or carotid artery disease: an international, randomised, double-blind, placebo-controlled trial.
        Lancet. 2018; 391: 219-229
        • The CAPRIE Steering Committee
        A randomised, blinded, trial of clopidogrel versus aspirin in patients at risk of ischaemic events (CAPRIE).
        Lancet. 1996; 348: 1329-1339
        • Sullivan P.W.
        • Slejko J.F.
        • Sculpher M.J.
        • et al.
        Catalogue of EQ-5D scores for the United Kingdom.
        Med Decis Making. 2011; 31: 800-804
        • Oostenbrink J.B.
        • Tangelder M.J.D.
        • Busschbach J.J.V.
        • et al.
        Cost-effectiveness of oral anticoagulants versus aspirin in patients after infrainguinal bypass grafting surgery.
        J Vasc Surg. 2001; 34: 254-262
        • van Stel H.F.
        • Busschbach J.J.V.
        • Hunink M.G.M.
        • et al.
        Impact of secondary cardiovascular events on health status.
        Value Health. 2012; 15: 175-182
        • Greving J.
        • Visseren F.
        • de Wit G.
        • et al.
        Statin treatment for primary prevention of vascular disease: whom to treat? Cost-effectiveness analysis.
        BMJ. 2011; 342: d1672
        • van Asselt A.D.
        • Nicolai S.P.
        • Joore M.A.
        • et al.
        Cost-effectiveness of exercise therapy in patients with intermittent claudication: supervised exercise therapy versus a ‘go home and walk’ advice.
        Eur J Vasc Endovasc Surg. 2011; 41: 97-103
        • van Hout B.A.
        • Simoons M.L.
        Cost-effectiveness of HMG coenzyme reductase inhibitors. Whom to treat?.
        Eur Heart J. 2001; 22: 751-761
        • Spronk S.
        • Bosch J.L.
        • den Hoed P.T.
        • et al.
        Cost-effectiveness of endovascular revascularization compared to supervised hospital-based exercise training in patients with intermittent claudication: a randomized controlled trial.
        J Vasc Surg. 2008; 48: 1472-1480
        • Grigore B.
        • Peters J.
        • Hyde C.
        • et al.
        EXPLICIT: a feasibility study of remote expert elicitation in health technology assessment.
        BMC Med Inform Decis Making. 2017; 17: 131
        • Soares M.O.
        • Sharples L.
        • Morton A.
        • et al.
        Experiences of structured elicitation for model-based cost-effectiveness analyses.
        Value Health. 2018; 21: 715-723
        • Iglesias C.P.
        • Thompson A.
        • Rogowski W.H.
        • et al.
        Reporting guidelines for the use of expert judgement in model-based economic evaluations.
        Pharmacoeconomics. 2016; 34: 1161-1172
        • Bojke L.
        • Claxton K.
        • Bravo-Vergel Y.
        • et al.
        Eliciting distributions to populate decision analytic models.
        Value Health. 2010; 13: 557-564
        • Bojke L.
        • Grigore B.
        • Jankovic D.
        • et al.
        Informing reimbursement decisions using cost-effectiveness modelling: a guide to the process of generating elicited priors to capture model uncertainties.
        Pharmacoeconomics. 2017; 35: 867-877
        • Knol A.B.
        • Slottje P.
        • van der Sluijs J.P.
        • et al.
        The use of expert elicitation in environmental health impact assessment: a seven step procedure.
        Environ Health. 2010; 9: 19
        • Ghabri S.
        • Cleemput I.
        • Josselin J.-M.
        Towards a new framework for addressing structural uncertainty in health technology assessment guidelines.
        Pharmacoeconomics. 2018; 36: 127-130
        • Bojke L.
        • Claxton K.
        • Sculpher M.
        • et al.
        Characterizing structural uncertainty in decision analytic models: a review and application of methods.
        Value Health. 2009; 12: 739-749
        • Strong M.
        • Oakley J.E.
        • Brennan A.
        Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach.
        Med Decis Making. 2014; 34: 311-326
        • Soares M.O.
        • Sculpher M.J.
        • Claxton K.
        Health opportunity costs: assessing the implications of uncertainty using elicitation methods with experts.
        Med Decis Making. 2020; 40: 448-459
        • O’Leary R.A.
        • Low-Choy S.
        • Fisher R.
        • et al.
        Characterising uncertainty in expert assessments: encoding heavily skewed judgements.
        PLoS One. 2015; 10e0141697
        • Claxton K.
        Exploring uncertainty in cost-effectiveness analysis.
        Pharmacoeconomics. 2008; 26: 781-798
        • Willan A.R.
        • Eckermann S.
        Optimal clinical trial design using value of information methods with imperfect implementation.
        Health Econ. 2010; 19: 549-561
        • Grutters J.P.
        • van Asselt M.B.
        • Chalkidou K.
        • et al.
        Healthy decisions: towards uncertainty tolerance in healthcare policy.
        Pharmacoeconomics. 2015; 33: 1-4
        • Ghabri S.
        • Hamers F.F.
        • Josselin J.M.
        Exploring uncertainty in economic evaluations of drugs and medical devices: lessons from the first review of manufacturers’ submissions to the French National Authority for Health.
        Pharmacoeconomics. 2016; 34: 617-624
      1. O’Hagan A, Oakley J. SHELF: the Sheffield Elicitation Framework.

        • Kunst N.
        • Wilson E.C.F.
        • Glynn D.
        • et al.
        Computing the expected value of sample information efficiently: practical guidance and recommendations for four model-based methods.
        Value Health. 2020; 23: 734-742
        • Petersohn S.
        • Ramaekers B.L.T.
        • Olie R.H.
        • et al.
        Comparison of three generic quality-of-life metrics in peripheral arterial disease patients undergoing conservative and invasive treatments.
        Qual Life Res. 2019; 28: 2257-2279
        • Spiegelhalter D.
        • Abrams K.
        • M J.
        Bayesian Approaches to Clinical Trials and Health-Care Evaluation.
        John Wiley & Sons Ltd, Chichester2004