Advertisement

Dealing With Uncertainty in Early Health Technology Assessment: An Exploration of Methods for Decision Making Under Deep Uncertainty

Open AccessPublished:October 14, 2022DOI:https://doi.org/10.1016/j.jval.2022.08.012

      Highlights

      • In early health economic modeling, there is an increasing need to move away from a predictive approach. Nevertheless, proper methods that are able to deal with the many unknown and uncertain factors often present in the earliest stages of innovation are lacking.
      • This article is a first step to translate exploratory modeling (EM) methods from the decision making under deep uncertainty field to early health technology assessment. This field uses EM to help decision makers make sound decisions under conditions of deep uncertainty: when stakeholders do not know, or cannot agree on, the system model, the probability distributions to place over the inputs to these models, which consequences to consider, and their relative importance.
      • EM might be useful in the very early stages of innovation, ideally when concrete solutions do not yet exist. Here, EM can explore the problem and the associated uncertainty and investigate many potential policies. Subsequently, health economic modeling could further evaluate promising policies.

      Objectives

      In early stages, the consequences of innovations are often unknown or deeply uncertain, which complicates early health economic modeling (EHEM). The field of decision making under deep uncertainty uses exploratory modeling (EM) in situations when the system model, input probabilities/distributions, and consequences are unknown or debated. Our aim was to evaluate the use of EM for early evaluation of health technologies.

      Methods

      We applied EM and EHEM to an early evaluation of minimally invasive endoscopy-guided surgery (MIS) for acute intracerebral hemorrhage and compared these models to derive differences, merits, and drawbacks of EM.

      Results

      EHEM and EM differ fundamentally in how uncertainty is handled. Where in EHEM the focus is on the value of technology, while accounting for the uncertainty, EM focuses on the uncertainty. EM aims to find robust strategies, which give relatively good outcomes over a wide range of plausible futures. This was reflected in our case study. EHEM provided cost-effectiveness thresholds for MIS effectiveness, assuming fixed MIS costs. EM showed that a policy with a population in which most patients had severe intracerebral hemorrhage was most robust, regardless of MIS effectiveness, complications, and costs.

      Conclusions

      EHEM and EM were found to complement each other. EM seems most suited in the very early phases of innovation to explore existing uncertainty and many potential strategies. EHEM seems most useful to optimize promising strategies, yet EM methods are complex and might only add value when stakeholders are willing to consider multiple solutions to a problem and adopt flexible research and adoption strategies.

      Keywords

      Introduction

      Healthcare technologies are increasingly evaluated in early stages of development.
      • IJzerman M.J.
      • Koffijberg H.
      • Fenwick E.
      • Krahn M.
      Emerging use of early health technology assessment in medical product development: a scoping review of the literature.
      Health economic modeling has a prominent, exploratory role in these early health technology assessments (HTAs).
      • Grutters J.P.C.
      • Govers T.M.
      • Nijboer J.
      • Tummers M.
      • van der Wilt G.J.
      • Rovers M.M.
      Exploratory, participatory and iterative assessment of value: a response to recent commentaries.
      Modeling is used to explore gaps in current clinical practice and circumstances under which the innovation has added value.
      • IJzerman M.J.
      • Koffijberg H.
      • Fenwick E.
      • Krahn M.
      Emerging use of early health technology assessment in medical product development: a scoping review of the literature.
      These explorations can subsequently guide further development of the innovation and inform research. This is appealing, given that large investments have not yet been made in these early stages of innovation and the opportunity to steer the development of an innovation in a way that it adds most value to patients and users is still there.
      • Greenhalgh T.
      • Fahy N.
      • Shaw S.
      The bright elusive butterfly of value in health technology development comment on “providing value to new health technology: the early contribution of entrepreneurs, investors, and regulatory agencies”.
      • Grutters J.P.C.
      • Govers T.
      • Nijboer J.
      • Tummers M.
      • van der Wilt G.J.
      • Rovers M.M.
      Problems and promises of health technologies: the role of early health economic modeling.
      • Lehoux P.
      • Mille R.F.
      • Daudelin G.
      • Denis J.L.
      Why learning how to chase butterflies matters: a response to recent commentaries.
      • Lehoux P.
      • Miller F.A.
      • Daudelin G.
      • Denis J.L.
      Providing value to new health technology: the early contribution of entrepreneurs, investors, and regulatory agencies.
      • Markiewicz K.
      • van Til J.A.
      • IJzerman M.J.
      Medical devices early assessment methods: systematic literature review.
      Traditionally, methods for standard health economic modeling have been applied to early health economic modeling (EHEM), mostly using assumptions to deal with the unknown model parameters and distributions (eg, unknown innovation effectiveness, target population, and costs).
      • Briggs A.H.
      • Weinstein M.C.
      • Fenwick E.A.
      • et al.
      Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group-6.
      Nevertheless, these assumptions do not necessarily represent the true uncertainty and limit the exploratory function of EHEM. This could lead to a false sense of confidence, which might subsequently not accurately inform research and development decisions.
      • Grutters J.P.
      • van Asselt M.B.
      • Chalkidou K.
      • Joore M.A.
      Healthy decisions: towards uncertainty tolerance in healthcare policy.
      • Love-Koh J.
      How useful are early economic models? Comment on “problems and promises of health technologies: the role of early health economic modelling”.
      • Sarewitz D.
      • Pielke R.A.
      • Byerly Jr., R.
      Prediction: Science, Decision Making, and the Future of Nature.
      • Gandjour A.
      Deterministic sensitivity analysis under ignorance.
      There is an increasing need to move away from this approach. Nevertheless, proper methods that are able to deal with parameter and structural uncertainties caused by the many unknown and uncertain factors often present in the earliest stages of innovation are lacking.
      In contrast, the field of “decision making under deep uncertainty (DMDU)” uses exploratory modeling (EM) approaches to help decision makers make sound decisions in states of deep uncertainty.
      • Marchau V.
      • Walker W.
      • Bloemen P.
      • et al.
      Decision Making Under Deep Uncertainty.
      A situation is deemed deeply uncertain when stakeholders do not know, or cannot agree on, the system model, the probability distributions to place over the inputs to these models, which consequences to consider, and their relative importance. DMDU systematically challenges assumptions, seeks for the set of conditions that represent desirable functioning of the system, and pursues robust and adaptive strategies instead of optimal, fixed strategies.
      • Bankes S.
      Exploratory modeling for policy analysis.
      Such an approach seems useful for early HTA, where, compared with traditional HTA, data are generally limited or absent. Therefore, this study aimed to use EM methods and explore whether and how they can enhance the early evaluation of health technologies. To that end, we applied EM methods on a case study of minimally invasive endoscopy-guided surgery (MIS) in patients with intracerebral hemorrhage (ICH), for which we previously conducted a “traditional” EHEM study. We first explain the case study and then summarize the methods and results of our traditional EHEM study, followed by the methods and results of EM. In the last section, we compare and discuss our findings.

      Case Study

      ICH is the deadliest stroke subtype with a 30-day case fatality of 40%.
      • Samarasekera N.
      • Fonville A.
      • Lerpiniere C.
      • et al.
      Influence of intracerebral hemorrhage location on incidence, characteristics, and outcome: population-based study.
      ,
      • van Asch C.J.
      • Luitse M.J.
      • Rinkel G.J.
      • van der Tweel I.
      • Algra A.
      • Klijn C.J.
      Incidence, case fatality, and functional outcome of intracerebral haemorrhage over time, according to age, sex, and ethnic origin: a systematic review and meta-analysis.
      Of patients surviving, only a few gain independence.
      • van Asch C.J.
      • Luitse M.J.
      • Rinkel G.J.
      • van der Tweel I.
      • Algra A.
      • Klijn C.J.
      Incidence, case fatality, and functional outcome of intracerebral haemorrhage over time, according to age, sex, and ethnic origin: a systematic review and meta-analysis.
      ,
      • Poon M.T.
      • Fonville A.F.
      • Al-Shahi Salman R.
      Long-term prognosis after intracerebral haemorrhage: systematic review and meta-analysis.
      Prognosis has not improved over the past decades.
      • van Asch C.J.
      • Luitse M.J.
      • Rinkel G.J.
      • van der Tweel I.
      • Algra A.
      • Klijn C.J.
      Incidence, case fatality, and functional outcome of intracerebral haemorrhage over time, according to age, sex, and ethnic origin: a systematic review and meta-analysis.
      ,
      • Jolink W.M.
      • Klijn C.J.
      • Brouwers P.J.
      • Kappelle L.J.
      • Vaartjes I.
      Time trends in incidence, case fatality, and mortality of intracerebral hemorrhage.
      Despite many attempts, there are no medical or surgical treatments with any proven benefit besides stroke unit care and possibly early control of elevated blood pressure.
      • Anderson C.S.
      • Heeley E.
      • Huang Y.
      • et al.
      Rapid blood-pressure lowering in patients with acute intracerebral hemorrhage.
      • Langhorne P.
      • Fearon P.
      • Ronning O.M.
      • et al.
      Stroke unit care benefits patients with intracerebral hemorrhage: systematic review and meta-analysis.
      • Qureshi A.I.
      • Palesch Y.Y.
      • Barsan W.G.
      • et al.
      Intensive blood-pressure lowering in patients with acute cerebral hemorrhage.
      Theoretically, surgical removal of the hemorrhage could relieve pressure on surrounding structures, which might subsequently reduce symptoms.
      • Sondag L.
      • Schreuder F.
      • Boogaarts H.D.
      • et al.
      Neurosurgical intervention for supratentorial intracerebral hemorrhage.
      That is why currently a research strategy is being set up to study minimally invasive endoscopy-guided treatment early after symptom onset.
      • Klijn C.J.D.R.
      • Schreuder F.B.H.M.
      • Wilting F.
      • Sondag L.
      DUTCH-ICH surgery trial.
      The first stages of this innovation process are characterized by deep uncertainty,
      • Marchau V.
      • Walker W.
      • Bloemen P.
      • et al.
      Decision Making Under Deep Uncertainty.
      given that stakeholders do not know (or at least disagree) whether MIS will be effective or not, in how many patients, and to what extent. Additionally, they do not know (or disagree) whether it could cause any extra harm to patients and what the probabilities and consequences of harm are. In addition, stakeholders do not know which patients might have benefit from MIS and what outcomes should be studied. This results in deep uncertainty regarding MIS effectiveness, MIS costs, MIS population, and MIS complications, for which data are currently unavailable.

      Traditional EHEM Study

      Specifying the Model

      In our EHEM study, we aimed to assess the conditions that need to be met to reach potential cost-effectiveness of MIS compared with usual care.
      • Schreuder F.H.B.M.
      • Scholte M.
      • Ulehake M.J.
      • et al.
      Identifying the conditions for cost-effective minimally invasive neurosurgery in spontaneous supratentorial intracerebral hemorrhage.
      Therefore, we developed a Markov state-transition model with 7 health states that represented functional outcome after ICH based on the modified Rankin Scale (mRS) score (Appendix Fig. 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.08.012).
      • Bamford J.M.
      • Sandercock P.A.
      • Warlow C.P.
      • Slattery J.
      Interobserver agreement for the assessment of handicap in stroke patients.
      The mRS is a widely used functional outcome assessment scale, which ranges from 0 (no symptoms) to 6 (death). An mRS score of 3 indicates moderate disability where the patient requires some help but is able to walk unassisted whereas an mRS score of 5 indicates severe disability where the patient requires constant nursing care and attention and is bedridden and incontinent.
      • van Swieten J.C.
      • Koudstaal P.J.
      • Visser M.C.
      • Schouten H.J.
      • van Gijn J.
      Interobserver agreement for the assessment of handicap in stroke patients.
      Input parameters related to current care were well studied and can be found in Appendix Tables 1 to 4 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.08.012. Model outcomes were quality-adjusted life-years and direct healthcare costs over a lifetime horizon.
      Consequences of MIS for different patient populations were explored using 3 patient cohorts that had varying age of onset and severity of ICH. Effectiveness of MIS was modeled as the percentage of patients who improved from a bad functional outcome (mRS 4-6) to a good functional outcome (mRS of 0-3). We used 11% improvement as a base case example, based on expert opinion. Costs of MIS were assumed to be €10 000 in our base case, based on expert opinion. No information about complications was available, and given that we aimed to determine the minimal conditions for MIS to be cost-effective, we assumed there were no complications.

      Analyses and Results

      We conducted 4 types of analyses, threshold analysis, two-way sensitivity analysis, scenario analysis, and probabilistic sensitivity analysis, for 3 patient populations with varying age and severity of ICH. We found that MIS had the potential to be cost-effective in our threshold analyses, even with relatively low effectiveness. Assuming €10 000 of surgical costs, MIS would become cost-effective when at least 0.7% (95% confidence interval [CI] 0.5%-0.9%) to 1.3% (95% CI 1.2%-1.5%) of mRS 4 to 6 patients improve to mRS 0 to 3 due to MIS compared with usual care, depending on the population characteristics. When 11% of patients improve to mRS 0 to 3, surgical costs may be up to €83 301 (95% CI €71 633-€93 496) to €164 382(95% CI €120 973-€203 160). Two-way sensitivity analyses showed that the cost-effectiveness of MIS was mainly driven by its effectiveness. Scenario analysis showed that MIS needed to be more effective to be cost-effective in lower mRS states compared with higher mRS states. Probabilistic sensitivity analysis included all parameters related to current ICH care; MIS parameters were incorporated as fixed values. All abovementioned analyses were based on probabilistic results.

      EM Methods

      DMDU is a relatively new field of research and is typically involved in decision making related to the consequences of climate change, long-term mobility planning, the planning of mega projects, or large disrupting events such as COVID-19.
      • Marchau V.
      • Walker W.
      • Bloemen P.
      • et al.
      Decision Making Under Deep Uncertainty.
      ,
      • Mitcham J.
      • Keisler J.
      Multi-attribute COVID-19 policy evaluation under deep uncertainty.
      ,
      • Nascimento de Lima P.
      • Lempert R.
      • Vardavas R.
      • et al.
      Reopening California: seeking robust, non-dominated COVID-19 exit strategies.
      Besides COVID-19, few examples of DMDU in healthcare are available.
      • Cassidy R.
      • Manski C.F.
      Tuberculosis diagnosis and treatment under uncertainty.
      We applied DMDU EM to determine under which conditions the MIS model produces (un)desirable cost-effectiveness outcomes and to seek for robust MIS policies.
      • Kwakkel J.H.
      The Exploratory Modeling Workbench: an open source toolkit for exploratory modeling, scenario discovery, and (multi-objective) robust decision making.
      Robust policies perform well (in this case: are cost-effective), compared with the alternatives, over a wide range of plausible futures. There are 4 main differences between EM and traditional methods in health economic modeling. First, where traditional modeling methods aim to predict cost-effectiveness, EM aims to explore circumstances under which the system (in this case the ICH care pathway) produces desirable outcomes. Second, to influence the system’s behavior, EM regards certain input parameters as policy levers (“control knobs” of the system) to create many potential policies, instead of evaluating 1 fixed strategy. Third, EM avoids making assumptions, by exploring a very large ensemble of plausible inputs by randomly sampling from uniform distributions ranging from 0 to 1 for input parameters. Finally, EM seeks for robust policies that perform well over a range of scenarios, instead of maximizing incremental net monetary benefit (iNMB) (Table 1 and Fig. 1).
      Table 1Overview of differences between traditional EHEM and DMDU EM for the ICH case study.
      FeaturesTraditional EHEMDMDU EMComparison
      Aim
      To assess the conditions that need to be met to reach potential cost-effectiveness of MIS in comparison with usual care.To determine under what conditions the MIS model produces (un)desirable cost-effectiveness outcomes and seek robust MIS policies. Robust policies perform well, compared with the alternatives, over a wide range of plausible futures.EM has a system focus, whereas EHEM has a technology focus. Therefore, EM seeks for conditions that represent desirable functioning of the system (ie, the healthcare pathway), whereas EHEM tries to find the conditions under which the technology produces desirable outcomes.



      EM aims for robust policies, whereas EHEM aims for cost-effective strategies.

      In this way, studying uncertainty is the main focus of EM, whereas EHEM focuses on expected or potential cost-effectiveness, under conditions of uncertainty.
      Model structure
      State-transition model

      Seven health states represented functional outcome after ICH: mRS0 (no symptoms) to mRS6 (death).

      Lifetime horizon, with 3 monthly cycles in the first year and yearly cycles thereafter.
      Decision tree + state-transition model

      The system was modeled using the same state-transition model as in the traditional modeling study, and a decision tree was added for more flexible modeling of MIS effectiveness and complications.
      To use a model for EM, the model structure should be flexible to allow for many policies and scenarios to be explored. EM typically takes a broader scope in their models (eg, the entire care system) and uses different model types (eg, system dynamics models).
      Input parameters
      See Appendix Tables 1-4 in Supplemental Materials found at https://dx.doi.org/10.1016/j.jval.2022.08.012 for a detailed overview of input parameters.See Appendix Tables 1-4 in Supplemental Materials found at https://dx.doi.org/10.1016/j.jval.2022.08.012 for a detailed overview of input parameters.EM divides input parameters into uncertainties and policy levers. The effectiveness of MIS, chance of complications, and costs of MIS were regarded as deep uncertainties. The target population and treatment policy were regarded policy levers. Due to limitations in the EMA workbench, it was not possible to include distributions for well-studied parameters. Ideally, these should be included.



      Dividing input parameters in policy levers and uncertainties allows for the evaluation of many new policies, instead of 1 fixed strategy.
      Patient population
      Three patient populations who had varying age and severity of ICH were modeled.Patient population was regarded a policy lever, meaning that this can be influenced by decision makers to steer the system toward more desirable functioning. Therefore, no assumptions regarding the initial distribution over the mRS0 to mRS6 health states were made. Instead, uniform distributions ranging from 0.00 to 1.00 were assigned to all 7 parameters. In addition, 3 treatment policies were modeled, where surgery was offered to (1) all patients, (2) only mRS0 to 3, or (3) only mRS 4 to 6.
      Transition probabilities
      Current strategyAfter ICH, patients transit to 1 of the 7 mRS scores. In the first year after ICH, patients could change health states, after the first year the mRS score was assumed to be stable for the remaining lifetime. Patients had a probability to die of natural causes or causes related to their ICH.

      Transition probabilities related to current care were well studied; therefore, point estimates and probability distributions were derived from literature.
      Transition probabilities related to current care were well studied and therefore the same as in the traditional modeling study. We modeled all well-studied parameters as fixed values, given that beta distributions were not available in the EMA workbench.
      New strategyTransition probabilities related to the new strategy, that is, effectiveness of MIS and chance of complications, were unknown. Therefore, the effectiveness of MIS was assumed to be an absolute improvement in mRS 0 to 3 for patients treated with MIS compared with usual care, using 11% as base case example for cost threshold analysis. Complications were not incorporated in the model.The effectiveness of MIS and chance of complications were regarded as uncertainties. Effectiveness was modeled via 2 parameters. First, the percentage of the population who benefits from MIS was modeled as a uniform distribution ranging from 0.00 to 1.00. Second, the degree of improvement was modeled as the number of mRS states that patients improved, which ranged from 1 to 6. We modeled the chance of complications as a uniform distribution between 0.00 and 1.00. As a consequence of the complication, the effectiveness of MIS was negated and functional outcome worsened by 1 mRS state.
      Effects
      Effectiveness was measured in QALYs. QALYs were well studied; therefore, point estimates and related beta distributions were derived from literature.QALYs were well studied and were therefore the same. They were modeled as fixed values.
      Costs
      Costs of MIS were assumed to be €10 000 as base case for the MIS effectiveness threshold analysis. Costs related to ICH care were well studied; therefore, point estimates and associated gamma distributions were derived from literature and our own patient cohort.Costs of MIS were unknown; therefore, a uniform distribution was applied ranging from €0 to €50 000 surgical costs. Costs related to ICH care were well studied and were therefore the same. They were modeled as fixed values.
      Outcome parameters
      Incremental effectiveness, incremental costs, and the incremental net monetary benefit were used as outcome parameters.Outcome parameters were the same.
      Analyses
      Threshold analysis was used to determine the minimal effectiveness and maximum cost of MIS to be cost-effective, using the assumptions above as base case scenario.



      Two-way sensitivity analysis to determine which strategy is cost-effective at each combination of cost and effect.



      A combination of scenario and threshold analysis was used to determine the minimal effectiveness of MIS for each health state, for example, how effective should MIS be if it were only effective in bringing mRS5 patients to mRS4.



      All abovementioned analyses were based on the results of a probabilistic sensitivity analysis with 5000 simulations.



      All analyses were performed for all 3 patient populations.
      First, we generated policy options and states of the world. By randomly sampling policy levers and uncertainties, we respectively created 1000 scenarios and 1000 policies to evaluate.



      Next, vulnerability analysis was used to explore which input parameters (policy levers and uncertainties) drive the system into desirable outcomes, that is, a positive iNMB. Using feature scoring, we studied the influence of the input parameters on the outcome parameters. Next, the PRIM algorithm was used to obtain specific parameters, parameter threshold values, and accuracy measures.



      Finally, we searched for robust policies, that is, policies that perform well (that is, have a positive iNMB), compared with the alternatives, over a wide range of scenarios. By using our set of 1000 random policies and stress testing them over the scenarios, we identified the policies that resulted in positive iNMB in > 95% of all samples. The policy levers of the 5 policies with the highest robustness were examined in closer detail.
      EHEM often uses a base case scenario from which conditions under which the new strategy is cost-effective are explored. A few important assumptions need to be selected for further analysis, as analyses quickly become intangible.



      In EM, there is no base case scenario; instead, it starts with many policies and scenarios. Then, the conditions under which the system drives into (un)desirable outcomes are explored.



      The threshold, two-way sensitivity analysis, and scenario analysis were based on probabilistic results, but still underestimate uncertainty due to fixed assumptions.



      EM in the EMA workbench underestimates uncertainty in well-studied parameters, given that distributions other than the uniform are currently not supported.
      DMDU indicates decision making under deep uncertainty; EHEM, early health economic modeling; EM, exploratory modeling; EMA, EM and analysis; ICH, intracerebral hemorrhage; iNMB, incremental net monetary benefit; MIS, minimally invasive endoscopy-guided surgery; mRS, modified Rankin Scale; PRIM, patient rule induction method; QALY, quality-adjusted life-year.
      Figure thumbnail gr1
      Figure 1Overview of model development in traditional early health economic modeling and DMDU EM. ∗In general, more flexible models are needed for DMDU EM. Adding a decision tree is not always necessary, but was helpful for our case study. This combination of decision tree and state-transition model could also be used in traditional early health economic modeling.
      DMDU indicates decision making under deep uncertainty; EM, exploratory modeling; IC, incremental cost; IE, incremental effectiveness; iNMB, incremental net monetary benefit; MIS, minimally invasive endoscopy-guided surgery; mRS, modified Rankin Scale.

      Specifying the Model

      The situation explained in the case study was a situation of deep uncertainty, given that we, and consulted stakeholders, had difficulties defining the system model and most input probabilities and probability distributions related to MIS were unknown. EM typically uses models that evaluate the entire system (eg, all factors related to ICH care, such as consequences for hospital bed capacities and nursing home facilities). We used an adapted version of the abovementioned state-transition model that has a more narrow scope, to facilitate the comparison of outcomes between methods. Deeply uncertain factors in our model comprised MIS effectiveness, consequences of complications, which patient populations to consider, and MIS costs.
      When specifying the model, EM specifies the system model, inputs, and outputs. Input parameters are divided into 2 categories: uncertainties (eg, safety, costs, and effectiveness of MIS) and policy levers (eg, the population that receives MIS). Policy levers are parameters that can be influenced by decision makers to steer the system toward more desirable outcomes. In our case, a decision tree was added to the already existing state-transition model to allow for more policies and uncertainties to be explored (Appendix Fig. 2 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.08.012). Input parameters related to current care were well studied and therefore remained the same as in the traditional modeling study, although we implemented these as fixed values given that distributions other than the uniform distribution were not supported by the EM and analysis (EMA) workbench. Outcome parameters were also identical, that is, incremental costs, incremental effectiveness, and iNMB.
      Although in traditional EHEM studies specific assumptions are made about deeply uncertain input parameters, EM considers a very large ensemble of plausible assumptions. Therefore, MIS effectiveness was modeled via 2 parameters. First, the percentage of the population who benefits from MIS was modeled, using assumptions ranging from 0% to 100%. Second, the degree of improvement was modeled as the number of mRS states that patients improved after MIS, which ranged from 1 to 6 (every possible mRS state). We modeled the chance of complications, ranging from 0% to 100%. As a consequence of the complication, the effectiveness of MIS was negated, and functional outcome would worsen by 1 mRS state.

      Analyses and Results

      To conduct EM, the EMA workbench was used, which provides support for performing DMDU EM with models developed in various modeling packages and environments.
      • Kwakkel J.H.
      The Exploratory Modeling Workbench: an open source toolkit for exploratory modeling, scenario discovery, and (multi-objective) robust decision making.
      We show the key features of EM that may be useful for early HTA: generation of policy options and states of the world, vulnerability analysis, and robustness search.

      Generation of policy options and states of the world

      First, we simultaneously explored uncertainties and policy levers to see how they jointly affected the outcomes. By randomly sampling uncertainties and policy levers, we respectively created scenarios (often called simulations in health economic modeling) and policies (often called strategies in health economic modeling). Using Latin hypercube sampling (Appendix in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.08.012 for further information), 1000 policies were generated, which were evaluated over 1000 scenarios. This set was used for all analyses.
      The policies had different performance in terms of incremental effectiveness and cost, but also in terms of how vulnerable they were to uncertainty. Figure 2 shows the results of 4 randomly selected policies, each evaluated over the same 1000 scenarios. The orange policy was much less sensitive to uncertainty compared with the yellow and pink policies; nevertheless, all were not robust (ie, did not always result in a positive iNMB). In contrast, the dark blue policy seemed to be robust.
      Figure thumbnail gr2
      Figure 2Cost-effectiveness plane displaying the results of 4 randomly selected policies in the EM approach. Each policy was evaluated over the same 1000 scenarios. The blue line indicates the willingness to pay threshold of €80 000 per QALY. The policies (in different colors) differ in terms of patient population and which patients undergo surgery. The dark blue policy is relatively insensitive to uncertainty compared with the other strategies. Compared with the other policies, where all patients or only patients in mRS states 0 to 3 receive surgery, the blue policy offers surgery to patients with mRS states 4 to 6 only.
      EM indicates exploratory modeling; mRS, modified Rankin Scale; QALY, quality-adjusted life-year.

      Vulnerability analysis

      Subsequently, the relations between input and output parameters were explored via feature scoring using the Extra Trees algorithm (Fig. 3 and Technical Appendix in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.08.012 further information).
      • Geurts P.
      • Ernst D.
      • Wehenkel L.
      Extremely randomized trees.
      The percentage of patients in mRS5 (severe disability) and the population who received MIS had the most influence on incremental costs, whereas the percentage of patients who benefited from MIS, the chance of complications, and the population who received MIS had the most influence on incremental effectiveness and the iNMB.
      Figure thumbnail gr3
      Figure 3EM: feature scoring plot. This plot shows the relative influence of the input parameters (displayed on the y axis) on the outcomes (displayed on the x axis). Blue colors indicate low influence; yellow colors indicate high influence. The mRS 0 to 6 parameters relate to the proportion of patients in that particular health state, and the MIS population parameter determines which part of the population (ie, only low mRS states, only high mRS states, or all patients) receive MIS.
      EM indicates exploratory modeling; IC, incremental cost; IE, incremental effectiveness; iNMB, incremental net monetary benefit; MIS, minimally invasive endoscopy-guided surgery; mRS, modified Rankin Scale.
      Next, we analyzed which parameters drove the system into desired outcomes, that is, a positive iNMB, in more detail. For this analysis, we used the patient rule induction method (PRIM) to identify in what parameter ranges a positive iNMB was most likely. PRIM is a nonparametric method for subgroup discovery that looks at specific regions of the input space where the outcome of interest is larger or smaller than average (see Technical Appendix and Appendix Fig. 3 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.08.012 for further information).
      • Nannings B.
      • Abu-Hanna A.
      • de Jonge E.
      Applying PRIM (patient rule induction method) and logistic regression for selecting high-risk subgroups in very elderly ICU patients.
      PRIM analyses showed that if the effectiveness of MIS is higher than 5% (ie, 5% of patients benefit from MIS), 51% of simulations resulted in cost-effective outcomes. When the chance of complications is lower than 66% and the percentage of patients who benefit from MIS is higher than 10%, 71% of simulations resulted in cost-effective outcomes. Further increasing the percentage of cost-effective simulations to 98% might be possible when the chance of complications is lower than 28%, costs of MIS are lower than €47 600, percentage of patients who benefit from MIS is higher than 40%, and the number of mRS states these patients improve when MIS is effective is higher than 1.

      Robustness search

      EM seeks for robust policies that perform well, compared with the alternatives, over a wide range of scenarios. A policy was considered robust if > 95% of scenarios resulted in a positive iNMB (Fig. 4). In the robust policies with on average the highest iNMB, relatively most patients were in mRS5 (Table 2). This makes sense, because patients with an mRS score of 5 have a negative quality of life and had high healthcare spending over lifetime, which means that both benefiting from MIS and experiencing a complication resulted in better outcomes. In the policy with the highest average iNMB, all patients receive MIS, whereas in the other 4 robust policies only patients with a high mRS score receive MIS. Robustness was reflected in the outcomes, as the 95% CIs of the iNMB of the robust policies did not include negative values.
      Figure thumbnail gr4
      Figure 4EM: robustness plot. Each line represents 1 policy. The light gray lines are policies that are not robust; that is, the percentage of scenarios that results in a positive iNMB is lower than 95% in these policies. The orange and dark blue policies are robust. In the dark blue policies, all patients receive MIS, whereas in the orange policies only patients in mRS states 4 to 6 receive MIS.
      EM indicates exploratory modeling; IC, incremental cost; IE, incremental effectiveness; iNMB, incremental net monetary benefit; MIS, minimally invasive endoscopy-guided surgery; mRS, modified Rankin Scale.
      Table 2Policy levers and outcomes of the 5 robust strategies with the highest average incremental net monetary benefit.
      PolicyOutcomes
      Patients who receive MIS% of population in mRS0% of population in mRS1% of population in mRS2% of population in mRS3% of population in mRS4% of population in mRS5% of population in mRS6IC, mean (95% CI)IE, mean (95% CI)iNMB, mean (95% CI)
      mRS0-60.010.280.020.110.050.450.08−€131 045 (−€204 749 to −€10 691)1.68 (−0.21 to 6.73)€265 533 (€40 613-€725 512)
      mRS4-60.020.140.050.270.020.360.14−€118 916 (−€177 344 to −€13 459)1.57 (0.28-5.11)€244 910 (€44 005-€570 818)
      mRS4-60.270.130.020.140.050.350.04−€120 779 (−€173 539 to −€21 749)1.38 (0.25-4.42)€231 115 (€46 111-€516 606)
      mRS4-60.150.010.110.150.130.370.08−€109 698 (−€177 221 to −€14 986)1.52 (0.11-5.56)€230 968 (€44 641-€616 637)
      mRS4-60.070.090.100.070.220.370.08−€99 256 (−€184 239 to −€12 613)1.55 (−0.19 to 6.26)€223 503 (€36 981-€679 684)
      CI indicates confidence interval; IC, incremental cost; IE, incremental effectiveness; iNMB, incremental net monetary benefit; MIS, minimally invasive endoscopy-guided surgery; mRS, modified Rankin Scale.

      Discussion

      In this study, we explored the deep uncertainty paradigm and determined whether and how EM methods could enhance early HTA. Accordingly, we have performed both traditional EHEM and EM using an empirical example of MIS for patients with ICH. Using EHEM, we showed that cost-effectiveness of MIS is mainly determined by its effectiveness: MIS becomes cost-effective when at least 0.7% to 1.3% of patients improve from a poor to a good functional outcome compared with standard medical management. In lower mRS states, MIS needs to be more effective to become cost-effective compared with higher mRS states. Using EM, we found that MIS effectiveness, the chance of complications, and the patient mix have a large influence on (cost-) effectiveness whereas the patient mix has a large influence on the surrounding uncertainty. Policies with a population in which most patients had severe ICH are most robust, meaning that they perform well compared with the alternatives over a wide range of scenarios.

      Strengths and Limitations

      The major strength of this article is that it is a first step to bridge different paradigms and translate the way of thinking and methodology from DMDU to early HTA. The HTA community needs approaches that can deal with the unknown and uncertain factors often present in the earliest stages of innovation. Therefore, it is important that other fields of research are explored for new, “out of the box” methods to deal with uncertainty.
      • Grutters J.P.
      • van Asselt M.B.
      • Chalkidou K.
      • Joore M.A.
      Healthy decisions: towards uncertainty tolerance in healthcare policy.
      This article shows that EM might be promising in this context.
      Our study also has some limitations. This is the first attempt to translate the DMDU mindset and EM methods to early HTA. Therefore, this article is far from exhaustive and only touches upon some of the key concepts of DMDU. Second, the DMDU mindset and methods were applied to a typical early HTA case study, which is quite different from a typical DMDU case study, for example, climate change or water management. The scale of the problem in the case study was a lot smaller than a typical DMDU problem. In contrast, the definition of deep uncertainty did apply to our HTA case study. Third, the EHEM and EM model structures were not entirely comparable, given that we added a decision tree to the EM model and incorporated complications. Ideally, we would have used same model structure, but two-way sensitivity analyses limited the number of parameters we could incorporate. Nevertheless, EM approaches easily compared a multitude of parameters, allowing us to broaden the scope of our model. Finally, to conduct EM, we used the EMA workbench in Python, which makes the exploratory analyses more accessible to newcomers and provides support for models developed in various modeling packages and environments (eg, Excel and Vensim).
      • Kwakkel J.H.
      The Exploratory Modeling Workbench: an open source toolkit for exploratory modeling, scenario discovery, and (multi-objective) robust decision making.
      Nevertheless, in the EMA workbench, it is not yet possible to include distributions other than the uniform distribution (eg, beta or gamma), although the developers are currently working on making this functionality available. In addition, given that many health economic modeling studies are conducted in R, other toolboxes such as the open Multiobjective Robust Decision Making toolkit for R might also be applicable.
      • Hadka D.
      • Herman J.
      • Reed P.
      • Keller K.
      An open source framework for many-objective robust decision making.

      Implications

      We found 2 clear advantages of EM compared with EHEM for our MIS case study. First, EM methods better embrace uncertainty. In our traditional model, we had to make assumptions for MIS effectiveness, MIS costs, and the population who received MIS, and could only calculate thresholds for 1 of these parameters while keeping the other parameters fixed. EM sought for robust results while taking all these uncertainties into account. Second, EM separated the model inputs into uncertainties (eg, MIS effectiveness) and policy levers (eg, population). Although EHEM showed that MIS effectiveness had substantial influence on MIS cost-effectiveness, this result is only of limited value when planning a future clinical trial. This is because information whether MIS is sufficiently effective to become cost-effective only becomes available after the trial is conducted. Instead, EM methods use parameters that can be controlled before or during the study, such as the population who receives MIS, to increase the chances of reaching a certain effectiveness bar.
      Although these EM features sound promising, we felt that the analyses were complex to perform and that our results were not directly useful for research and clinical practice. We had difficulties interpreting and communicating the EM findings to stakeholders. For example, the finding that policies where patients ended up in certain mRS states were most robust was difficult to translate to clinical trial design, given that mRS scores can only be poorly predicted when ICH patients arrive at the hospital. Results of our traditional EHEM study, although not directly useful for clinical trial design, were more in line with their thought processes as they gave clear cutoff points for MIS effectiveness. EM results might be better interpretable when a broader set of policy levers can be identified, for example, more innovative treatment options instead of only MIS.
      We found that EM methods show some important advantages over traditional EHEM methods: the use of policy levers to evaluate many policies, the systematic avoidance of assumptions, the search for robust policies that perform well over a myriad of scenarios, and the incorporation of uncertainty in all analyses. Nevertheless, its methods were also complex, require a different way of thinking, and cannot be properly executed without involving experts from the DMDU field. It is questionable whether this can be aligned with recent developments early HTA, where a simple approach to modeling is advocated for the earliest stages of innovation.
      • Grutters J.P.C.
      • Govers T.M.
      • Nijboer J.
      • Tummers M.
      • van der Wilt G.J.
      • Rovers M.M.
      Exploratory, participatory and iterative assessment of value: a response to recent commentaries.
      Finally, to really benefit from these methods and to make the results more interpretable, EM needs to be linked to the decision making process. To achieve such a link, a broader DMDU approach, including also adaptive strategies and joint sense making, is required.
      • Marchau V.
      • Walker W.
      • Bloemen P.
      • et al.
      Decision Making Under Deep Uncertainty.
      ,
      • Haasnoot M.
      • Kwakkel J.H.
      • Walker W.E.
      • ter Maat J.
      Dynamic adaptive policy pathways: a method for crafting robust decisions for a deeply uncertain world.
      These ideas align with the idea that HTA is an iterative process throughout the innovation lifecycle and not a 1-time exercise to determine cost-effectiveness. Furthermore, there must be a policy window and key stakeholders should be willing to consider multiple solutions to a problem, adopt adaptive strategies, and involve a broad range of stakeholders throughout this process. The current case might have been more straightforward than other, more complex, problems. These problems, such as the COVID-19 crisis or sustainability of healthcare, might be more suitable given that they have (inter)national, potentially impactful, consequences for the total population. Recently, the first DMDU projects for COVID-19 have emerged, exploring different management strategies.
      • Mitcham J.
      • Keisler J.
      Multi-attribute COVID-19 policy evaluation under deep uncertainty.
      ,
      • Nascimento de Lima P.
      • Lempert R.
      • Vardavas R.
      • et al.
      Reopening California: seeking robust, non-dominated COVID-19 exit strategies.
      ,
      • Olde Rikkert M.G.M.
      • Wertheim H.
      • Quax R.
      • Marchau V.A.W.J.
      • Rouwette E.A.J.A.
      • Korzilius H.P.L.M.
      Adaptive systems based pandemic management to improve resilience of healthcare and society. Dutch Research Council (NWO).
      HTA is a technology-driven approach, which often focuses on a single technology of which the development can often no longer be stopped, whereas DMDU is a problem-driven approach. Some DMDU features, such as the generation of policies and adaptive strategies, cannot reach their full potential within the current scope of HTA. By broadening the scope of EHEM, for example, by looking at the size of the problem the innovation aims to address and other solutions to it, HTA might become better in directing technological developments.

      Conclusion

      We explored the differences, merits, and drawbacks of EM versus EHEM. Both approaches seem valuable to the early innovation process in their own way. EM seems most suited in the very early phases of innovation where it can explore existing uncertainty and many potential strategies. When potential strategies are selected, EHEM seems useful to monitor and optimize these strategies, yet EM methods are complex and might only add value when a policy window exists that facilitates flexible research and adoption strategies.

      Article and Author Information

      Author Contributions: Concept and design: Scholte, Grutters
      Acquisition of data: Scholte, Grutters
      Analysis and interpretation of data: Scholte, Marchau, Kwakkel, Klijn, Rovers, Grutters
      Drafting of the manuscript: Scholte, Grutters
      Critical revision of the paper for important intellectual content: Marchau, Kwakkel, Klijn, Rovers, Grutters
      Statistical analysis: Scholte, Kwakkel
      Obtaining funding: Rovers, Supervision: Marchau, Klijn, Rovers, Grutters
      Conflict of Interest Disclosures: Dr Rovers reported receiving support for the present manuscript in terms of a VICI grant from NWO/ZonMw and a nonrestricted grant from Siemens Healthineers for a project not related to the current manuscript. Dr Rovers is a member of several research grant committees and the IDEAL council. Dr Grutters reported receiving several unrestricted grants from The Netherlands Organization for Health Research and Development and the National Health Care Institute in addition to a research grant from Edwards Lifesciences. Dr Grutters is a member of the Lorentz center advisory “Life Sciences and Medicine Board,” the Zorginstituut Nederland expert committee on proton therapy, and several committees of The Netherlands Organization for Health Research and Development. No other disclosures were reported.
      Funding/Support: Dr Klijn reported receiving grants from the Dutch Heart Foundation (grant no. 2012T077) and The Netherlands Organization for Health Research and Development, ZonMw (grant no. 015008048); support of the Netherlands Cardiovascular Research Initiative, which is supported by the Dutch Heart Foundation, CVON2015–01: CONTRAST; and the support of the Brain Foundation Netherlands (HA2015·01·06). The collaboration project is additionally financed by the Ministry of Economic Affairs by means of the Public-private partnerships. Allowance made available by the Top Sector Life Sciences & Health to stimulate public-private partnerships (LSHM17016). This work was funded in part through unrestricted funding by Stryker, Medtronic, and Cerenovus. Radboud UMC and Erasmus MC received additional unrestricted funding on behalf of CONTRAST, for the execution of the Dutch ICH Surgery Trial pilot study from Penumbra Inc. In addition, Radboud UMC and Erasmus MC received funding from Pneumbra Inc and from ZonMw (80-86200-08-25001) for the Dutch ICH Surgery Trial. Dr Klijn is member of the DSMB of ENCHANTED2 (NCT04140110), MOSES (NCT03961334), OPTIMAS (NCT03759938), and EMITT (NCT05318612). This study was supported by an unrestricted VICI grant from the Dutch research council (grant no. 91818617).
      Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
      Data Availability: The data supporting the findings in this study are available within the article and its supplemental materials. The R and Python code used for modeling are available from the corresponding author upon reasonable request.

      Acknowledgment

      The authors thank R. Dammers, F.H.B.M. Schreuder, and L. Sondag, who, together with C.J.M. Klijn, have led the DIST pilot study for the use of their minimally invasive endoscopy-guided surgery in ICH case study (www.dutch-ich.nl) and their contributions to the discussions. We also thank the lolaHESG community for the valuable discussions on an earlier version of this article. A special thanks to H. Broekhuizen for his very helpful and detailed comments.

      Supplemental Material

      References

        • IJzerman M.J.
        • Koffijberg H.
        • Fenwick E.
        • Krahn M.
        Emerging use of early health technology assessment in medical product development: a scoping review of the literature.
        Pharmacoeconomics. 2017; 35: 727-740
        • Grutters J.P.C.
        • Govers T.M.
        • Nijboer J.
        • Tummers M.
        • van der Wilt G.J.
        • Rovers M.M.
        Exploratory, participatory and iterative assessment of value: a response to recent commentaries.
        Int J Health Policy Manag. 2021; 10: 42-44
        • Greenhalgh T.
        • Fahy N.
        • Shaw S.
        The bright elusive butterfly of value in health technology development comment on “providing value to new health technology: the early contribution of entrepreneurs, investors, and regulatory agencies”.
        Int J Health Policy Manag. 2018; 7: 81-85
        • Grutters J.P.C.
        • Govers T.
        • Nijboer J.
        • Tummers M.
        • van der Wilt G.J.
        • Rovers M.M.
        Problems and promises of health technologies: the role of early health economic modeling.
        Int J Health Policy Manag. 2019; 8: 575-582
        • Lehoux P.
        • Mille R.F.
        • Daudelin G.
        • Denis J.L.
        Why learning how to chase butterflies matters: a response to recent commentaries.
        Int J Health Policy Manag. 2018; 7: 286-287
        • Lehoux P.
        • Miller F.A.
        • Daudelin G.
        • Denis J.L.
        Providing value to new health technology: the early contribution of entrepreneurs, investors, and regulatory agencies.
        Int J Health Policy Manag. 2017; 6: 509-518
        • Markiewicz K.
        • van Til J.A.
        • IJzerman M.J.
        Medical devices early assessment methods: systematic literature review.
        Int J Technol Assess Health Care. 2014; 30: 137-146
        • Briggs A.H.
        • Weinstein M.C.
        • Fenwick E.A.
        • et al.
        Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group-6.
        Med Decis Making. 2012; 32: 722-732
        • Grutters J.P.
        • van Asselt M.B.
        • Chalkidou K.
        • Joore M.A.
        Healthy decisions: towards uncertainty tolerance in healthcare policy.
        Pharmacoeconomics. 2015; 33: 1-4
        • Love-Koh J.
        How useful are early economic models? Comment on “problems and promises of health technologies: the role of early health economic modelling”.
        Int J Health Policy Manag. 2020; 9: 215-217
        • Sarewitz D.
        • Pielke R.A.
        • Byerly Jr., R.
        Prediction: Science, Decision Making, and the Future of Nature.
        Island Press, Washington, DC2000
        • Gandjour A.
        Deterministic sensitivity analysis under ignorance.
        Pharmacoeconomics. 2021; 39: 1197-1198
        • Marchau V.
        • Walker W.
        • Bloemen P.
        • et al.
        Decision Making Under Deep Uncertainty.
        1st ed. Springer, Cham, Switzerland2019
        • Bankes S.
        Exploratory modeling for policy analysis.
        Oper Res. 1993; 41: 435-449
        • Samarasekera N.
        • Fonville A.
        • Lerpiniere C.
        • et al.
        Influence of intracerebral hemorrhage location on incidence, characteristics, and outcome: population-based study.
        Stroke. 2015; 46: 361-368
        • van Asch C.J.
        • Luitse M.J.
        • Rinkel G.J.
        • van der Tweel I.
        • Algra A.
        • Klijn C.J.
        Incidence, case fatality, and functional outcome of intracerebral haemorrhage over time, according to age, sex, and ethnic origin: a systematic review and meta-analysis.
        Lancet Neurol. 2010; 9: 167-176
        • Poon M.T.
        • Fonville A.F.
        • Al-Shahi Salman R.
        Long-term prognosis after intracerebral haemorrhage: systematic review and meta-analysis.
        J Neurol Neurosurg Psychiatry. 2014; 85: 660-667
        • Jolink W.M.
        • Klijn C.J.
        • Brouwers P.J.
        • Kappelle L.J.
        • Vaartjes I.
        Time trends in incidence, case fatality, and mortality of intracerebral hemorrhage.
        Neurology. 2015; 85: 1318-1324
        • Anderson C.S.
        • Heeley E.
        • Huang Y.
        • et al.
        Rapid blood-pressure lowering in patients with acute intracerebral hemorrhage.
        N Engl J Med. 2013; 368: 2355-2365
        • Langhorne P.
        • Fearon P.
        • Ronning O.M.
        • et al.
        Stroke unit care benefits patients with intracerebral hemorrhage: systematic review and meta-analysis.
        Stroke. 2013; 44: 3044-3049
        • Qureshi A.I.
        • Palesch Y.Y.
        • Barsan W.G.
        • et al.
        Intensive blood-pressure lowering in patients with acute cerebral hemorrhage.
        N Engl J Med. 2016; 375: 1033-1043
        • Sondag L.
        • Schreuder F.
        • Boogaarts H.D.
        • et al.
        Neurosurgical intervention for supratentorial intracerebral hemorrhage.
        Ann Neurol. 2020; 88: 239-250
        • Klijn C.J.D.R.
        • Schreuder F.B.H.M.
        • Wilting F.
        • Sondag L.
        DUTCH-ICH surgery trial.
        2018
        https://www.dutch-ich.nl/
        Date accessed: October 6, 2022
        • Schreuder F.H.B.M.
        • Scholte M.
        • Ulehake M.J.
        • et al.
        Identifying the conditions for cost-effective minimally invasive neurosurgery in spontaneous supratentorial intracerebral hemorrhage.
        Front Neurol. 2022; 13: 830614
        • Bamford J.M.
        • Sandercock P.A.
        • Warlow C.P.
        • Slattery J.
        Interobserver agreement for the assessment of handicap in stroke patients.
        Stroke. 1989; 20: 828
        • van Swieten J.C.
        • Koudstaal P.J.
        • Visser M.C.
        • Schouten H.J.
        • van Gijn J.
        Interobserver agreement for the assessment of handicap in stroke patients.
        Stroke. 1988; 19: 604-607
        • Mitcham J.
        • Keisler J.
        Multi-attribute COVID-19 policy evaluation under deep uncertainty.
        Ann Oper Res. 2022; : 1-38
        • Nascimento de Lima P.
        • Lempert R.
        • Vardavas R.
        • et al.
        Reopening California: seeking robust, non-dominated COVID-19 exit strategies.
        PLoS One. 2021; 16e0259166
        • Cassidy R.
        • Manski C.F.
        Tuberculosis diagnosis and treatment under uncertainty.
        Proc Natl Acad Sci USA. 2019; 116: 22990-22997
        • Kwakkel J.H.
        The Exploratory Modeling Workbench: an open source toolkit for exploratory modeling, scenario discovery, and (multi-objective) robust decision making.
        Environ Modell Softw. 2017; 96: 239-250
        • Geurts P.
        • Ernst D.
        • Wehenkel L.
        Extremely randomized trees.
        Mach Learn. 2006; 63: 3-42
        • Nannings B.
        • Abu-Hanna A.
        • de Jonge E.
        Applying PRIM (patient rule induction method) and logistic regression for selecting high-risk subgroups in very elderly ICU patients.
        Int J Med Inform. 2008; 77: 272-279
        • Hadka D.
        • Herman J.
        • Reed P.
        • Keller K.
        An open source framework for many-objective robust decision making.
        Environ Modell Softw. 2015; 74: 114-129
        • Haasnoot M.
        • Kwakkel J.H.
        • Walker W.E.
        • ter Maat J.
        Dynamic adaptive policy pathways: a method for crafting robust decisions for a deeply uncertain world.
        Glob Environ Change. 2013; 23: 485-498
        • Olde Rikkert M.G.M.
        • Wertheim H.
        • Quax R.
        • Marchau V.A.W.J.
        • Rouwette E.A.J.A.
        • Korzilius H.P.L.M.
        Adaptive systems based pandemic management to improve resilience of healthcare and society. Dutch Research Council (NWO).