Advertisement
Value in Health
ISPOR
Advanced searchSave search
Methodology| Volume 23, ISSUE 6, P734-742, June 2020

Download started.

Ok

Computing the Expected Value of Sample Information Efficiently: Practical Guidance and Recommendations for Four Model-Based Methods

Open ArchivePublished:May 27, 2020DOI:https://doi.org/10.1016/j.jval.2020.02.010
Computing the Expected Value of Sample Information Efficiently: Practical Guidance and Recommendations for Four Model-Based Methods
Previous ArticleResearch and Policy Impact of Trials Published by the UK National Institute of Health Research (2006-2015)
Next ArticleDeveloping Markov Models From Real-World Data: A Case Study of Heart Failure Modeling Using Administrative Data

      Highlights

      • •
        The expected value of sample information (EVSI) can be used to prioritize research and design future studies to reduce decision uncertainty for policy makers. Four recently published methods have overcome the computational issues associated with EVSI analysis, but practical guidance on using and distinguishing these methods is lacking.
      • •
        Members of the Collaborative Network on Value of Information (ConVOI) reviewed 4 EVSI computation methods to understand their required skills and inputs. They also provided step-by-step guides and recommendations about which method to use based on features of the decision problem and proposed future study.
      • •
        By comparing these methods, ConVOI provides practical guidance for analysts looking to compute EVSI and use it for study design. Analysts now have useful information to confidently select the most appropriate EVSI estimation method for their application and expertise.

      Abstract

      Value of information (VOI) analyses can help policy makers make informed decisions about whether to conduct and how to design future studies. Historically a computationally expensive method to compute the expected value of sample information (EVSI) restricted the use of VOI to simple decision models and study designs. Recently, 4 EVSI approximation methods have made such analyses more feasible and accessible. Members of the Collaborative Network for Value of Information (ConVOI) compared the inputs, the analyst’s expertise and skills, and the software required for the 4 recently developed EVSI approximation methods. Our report provides practical guidance and recommendations to help inform the choice between the 4 efficient EVSI estimation methods. More specifically, this report provides: (1) a step-by-step guide to the methods’ use, (2) the expertise and skills required to implement the methods, and (3) method recommendations based on the features of decision-analytic problems.

      Introduction

      Decisions on which research studies to fund are intrinsically economic in nature as public, private, and third sector (research charity) funders have finite resources. Thus their decisions bear an opportunity cost, implying the prioritization of research studies that are expected to yield the greatest benefit.
      A value of information analysis (VOI) can determine the value of research studies based on how much they reduce decision uncertainty about the optimal intervention for use in a population of interest. Reducing decision uncertainty has value because a suboptimal decision reduces the potential health improvements in the population. The probability of making the wrong decision multiplied by the size of the loss incurred by that decision yields the expected loss associated with uncertainty.
      • Alarid-Escudero F.
      • Enns E.A.
      • Kuntz K.M.
      • Michaud T.L.
      • Jalal H.
      “Time traveling is just too dangerous” but some methods are worth revisiting: the advantages of expected loss curves over cost-effectiveness acceptability curves and frontier.
      Value Health. 2019; 22: 611-618
      Additional data are expected to reduce decision uncertainty and so reduce the expected loss. The expected value of sample information (EVSI) measures the “expected reduction in expected loss” from a given research study. Scaled up to the relevant population, this can be expressed in health terms of life-years, or quality-adjusted life-years (QALYs), or in monetary units.
      • McCabe C.
      • Claxton K.
      • Culyer A.J.
      The NICE cost-effectiveness threshold: what it is and what that means.
      Pharmacoeconomics. 2008; 26: 733-744
      ,
      National Institute for Health and Care Excellence
      Guide to the methods of technology appraisal 2013.
      https://www.ncbi.nlm.nih.gov/books/NBK395867/pdf/Bookshelf_NBK395867.pdf
      The difference between the EVSI and the cost of the research is the expected net benefit of sampling (ENBS).
      An increasing number of authorities and Health Technology Assessment (HTA) agencies either acknowledge
      Norwegian Medicines Agency
      Guidelines for the submission of documentation for single technology assessment (STA) of pharmaceuticals.
      https://legemiddelverket.no/Documents/English/Public%20funding%20and%20pricing/Documentation%20for%20STA/Guidelines%20151018.pdf
      Date accessed: March 25, 2019
      , 
      National Institute of Health and Care Excellence
      Developing NICE guidelines: the manual.
      https://www.nice.org.uk/process/pmg20/chapter/introduction-and-overview
      , 
      National Institute of Health and Care Excellence
      Research recommendations.
      https://www.nice.org.uk/Media/Default/About/what-we-do/Research-and-development/Research-Recommendation-Process-and-Methods-Guide-2015.pdf
      or recommend
      CADTH
      Guidelines for the Economic Evaluation of Health Technologies: Canada. 4th Edition.
      https://www.cadth.ca/about-cadth/how-we-do-it/methods-and-guidelines/guidelines-for-the-economic-evaluation-of-health-technologies-canada
      VOI analysis. Thus, EVSI calculations will soon be required from analysts. Until recently, EVSI calculations were extremely computationally expensive, potentially taking weeks or months,
      • Heath A.
      • Kunst N.
      • Jackson C.
      • et al.
      Calculating the expected value of sample information in practice: Considerations from three case studies.
      Medical Decision Making. 2020; (In press)https://doi.org/10.1177/0272989X20912402
      because they required nested simulation methods.
      • Ades A.E.
      • Lu G.
      • Claxton K.
      Expected value of sample information calculations in medical decision modeling.
      Med Decis Making. 2004; 24: 207-227
      Thus most VOI analyses were restricted to computing the expected value of perfect information (EVPI) or the expected value of partial perfect information (EVPPI), for which several efficient methods are available.
      • Felli J.C.
      • Hazen G.B.
      Sensitivity analysis and the expected value of perfect information.
      Med Decis Making. 1998; 18: 95-109
      , 
      • Brennan A.
      • Kharroubi S.
      • O'Hagan A.
      • Chilcott J.
      Calculating partial expected value of perfect information via Monte Carlo sampling algorithms.
      Med Decis Making. 2007; 27: 448-470
      , 
      • Madan J.
      • Ades A.E.
      • Price M.
      • et al.
      Strategies for efficient computation of the expected value of partial perfect information.
      Med Decis Making. 2014; 34: 327-342
      , 
      • Oakley J.
      Value of information for complex cost-effectiveness models.
      University of Sheffield, Sheffield, UK2002
      , 
      • Sadatsafavi M.
      • Bansback N.
      • Zafari Z.
      • Najafzadeh M.
      • Marra C.
      Need for speed: an efficient algorithm for calculation of single-parameter expected value of partial perfect information.
      Value Health. 2013; 16: 438-448
      , 
      • Strong M.
      • Oakley J.E.
      An efficient method for computing single-parameter partial expected value of perfect information.
      Med Decis Making. 2013; 33: 755-766
      , 
      • Strong M.
      • Oakley J.E.
      • Brennan A.
      Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach.
      Med Decis Making. 2014; 34: 311-326
      , 
      • Giles M.B.
      • Goda T.
      Decision-making under uncertainty: using MLMC for efficient estimation of EVPPI.
      Statistics and Computing. 2019; 29: 739-751
      New algorithms and associated software,
      • Welton N.J.
      • Madan J.J.
      • Caldwell D.M.
      • Peters T.J.
      • Ades A.E.
      Expected value of sample information for multi-arm cluster randomized trials with binary outcomes.
      Med Decis Making. 2013; 34: 352-365
      , 
      • Strong M.
      • Oakley J.E.
      • Brennan A.
      • Breeze P.
      Estimating the expected value of sample information using the probabilistic sensitivity analysis sample: a fast, nonparametric regression-based method.
      Med Decis Making. 2015; 35: 570-583
      , 
      • Menzies N.A.
      An efficient estimator for the expected value of sample information.
      Med Decis Making. 2016; 36: 308-320
      , 
      • Jalal H.
      • Alarid-Escudero F.
      A Gaussian approximation approach for value of information analysis.
      Med Decis Making. 2018; 38: 174-188
      , 
      • Jalal H.
      • Goldhaber-Fiebert J.D.
      • Kuntz K.M.
      Computing expected value of partial sample information from probabilistic sensitivity analysis using linear regression metamodeling.
      Med Decis Making. 2015; 35: 584-595
      , 
      • Heath A.
      • Manolopoulou I.
      • Baio G.
      Efficient Monte Carlo estimation of the expected value of sample information using moment matching.
      Med Decis Making. 2018; 38: 163-173
      , 
      • Heath A.
      • Baio G.
      Calculating the expected value of sample information using efficient nested Monte Carlo: a tutorial.
      Value Health. 2018; 21: 1299-1304
      , 
      • Heath A.
      • Manolopoulou I.
      • Baio G.
      Bayesian curve fitting to estimate the expected value of sample information using moment matching across different sample sizes.
      Med Decis Making. 2018; ([in press])
      , 
      • Heath A.
      • Baio G.
      EVSI: a suite of functions for the calculation and presentation of the EVSI.
      https://github.com/annaheath/EVSI
      Date accessed: October 4, 2019
      however, enable the efficient computation of EVSI for different study designs using complex decision-analytic models.
      Recently, the accuracy and computational time of these methods were compared using 3 different case studies, and researchers concluded that none of the methods were systematically superior in terms of accuracy, precision, or computational time.
      • Heath A.
      • Kunst N.
      • Jackson C.
      • et al.
      Calculating the expected value of sample information in practice: Considerations from three case studies.
      Medical Decision Making. 2020; (In press)https://doi.org/10.1177/0272989X20912402
      Still, these methods differ in their approach, requiring different expertise and skills. Nevertheless, because there is no structured comparison of the practical steps required to use them, it is challenging for analysts to determine which method is appropriate for their situation and expertise. Additionally, each method has different properties that could make it more suitable for a given decision problem. This report provides practical guidance and recommendations for computing EVSI using 4 recently developed approximation methods. For each method, we provide: (1) a step-by-step guide to its use, (2) the expertise and skills required to implement it, and (3) its recommended usage based on the features of decision-analytic problems.

      Four EVSI Approximation Methods

      The Collaborative Network for Value of Information (ConVOI)
      ConVOI
      The collaborative network for value of information.
      https://www.convoi-group.org/
      Date accessed: June 9, 2019
      is an international group of researchers that focuses on the application and development of VOI calculation methods. In November 2018, members of this network prepared a questionnaire (N.K., A.H., D.G.) completed by the developers of 4 recently developed EVSI computation methods (M.S., N.M., H.J., F.A.E., A.H.). The questionnaire results were reviewed, summarized (N.K., A.H., D.G., E.W.), and discussed (by all authors) to find: (1) the inputs, (2) the skills and expertise, and (3) the software required to use the reviewed methods. The questionnaire also identified the situations in which each method was appropriate. Any disagreements were resolved by consensus (for the questionnaire, see Appendix 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.02.010).
      The reviewed methods included a regression-based (RB) method,
      • Strong M.
      • Oakley J.E.
      • Brennan A.
      • Breeze P.
      Estimating the expected value of sample information using the probabilistic sensitivity analysis sample: a fast, nonparametric regression-based method.
      Med Decis Making. 2015; 35: 570-583
      an importance sampling (IS) method,
      • Menzies N.A.
      An efficient estimator for the expected value of sample information.
      Med Decis Making. 2016; 36: 308-320
      a Gaussian approximation (GA) method,
      • Jalal H.
      • Alarid-Escudero F.
      A Gaussian approximation approach for value of information analysis.
      Med Decis Making. 2018; 38: 174-188
      ,
      • Jalal H.
      • Goldhaber-Fiebert J.D.
      • Kuntz K.M.
      Computing expected value of partial sample information from probabilistic sensitivity analysis using linear regression metamodeling.
      Med Decis Making. 2015; 35: 584-595
      and a moment matching (MM) method.
      • Heath A.
      • Manolopoulou I.
      • Baio G.
      Efficient Monte Carlo estimation of the expected value of sample information using moment matching.
      Med Decis Making. 2018; 38: 163-173
      These methods were selected because they place limited restrictions on the complexity of the underlying decision-analytic model or data collection exercise when calculating EVSI.

      Inputs Required for Efficient EVSI Calculation

      The considered EVSI approximation methods have diverse requirements and make different assumptions. Nevertheless, all require a decision-analytic model on which a probabilistic analysis is conducted. The IS and MM methods also require calculation of the EVPPI for the parameter(s) to be evaluated and updated in the proposed study. If studies informing different parameters or groups of parameters are considered, then EVPPI would need to be computed for each. The following section briefly outlines these requirements and assumptions (Table 1).
      Table 1The inputs required to compute EVSI with the regression-based, importance sampling, Gaussian approximation, and moment matching methods.
      #RequirementsRBISGAMM
      Inputs
      1Decision-analytic model
      EVSI estimation involves rerunning the decision-analytic model and thus access to that model is required.
      		✓
      2Probabilistic analysis results✓✓✓✓
      3Simulations of the expected net benefit conditional on φ (required to compute EVPPI)✓✓
      Note. ✓ indicates that the skill/input is required.
      GA indicates Gaussian approximation method; IS, importance sampling method; MM, moment matching method; RB, regression-based method.
      ∗ EVSI estimation involves rerunning the decision-analytic model and thus access to that model is required.

      Decision-Analytic Model

      VOI requires an objective function that should be optimized to determine the best course of action.
      • Raiffa H.
      • Schlaifer H.
      Applied Statistical Decision Theory.
      Harvard University Press, Boston, MA1961
      Economic evaluations in healthcare typically define this in terms of net health or monetary benefit for each of T interventions,
      • Stinnett A.A.
      • Mullahy J.
      Net health benefits: a new framework for the analysis of uncertainty in cost-effectiveness analysis.
      Med Decis Making. 1998; 18: S68-S80
      which uses a “willingness-to-pay” threshold to put health consequences on the same scale as costs for the different interventions.
      • Stinnett A.A.
      • Mullahy J.
      Net health benefits: a new framework for the analysis of uncertainty in cost-effectiveness analysis.
      Med Decis Making. 1998; 18: S68-S80
      Decision-analytic models are mathematical models often used for cost-effectiveness analyses to compute this net benefit (NB). These models require a set of inputs, denoted by θ. These inputs include disease prevalence, treatment effectiveness, background mortality, health-related utility weights, and costs. The model can take many forms, typically a decision tree, Markov model, or microsimulation model.
      • Briggs A.
      • Sculpher M.
      • Claxton K.
      Decision Modelling for Health Economic Evaluation.
      Oxford University Press, Oxford2006
      Conceptually, the decision-analytic model maps a set of inputs to the output net benefit (either in monetary or health units). Assuming risk neutrality,
      • Arrow K.J.
      • Lind R.C.
      Uncertainty and the evaluation of public investment decisions.
      Am Econ Rev. 1970; 60
      the intervention with the highest expected NB should be implemented in the wider population.

      Probabilistic Analysis

      Probabilistic analysis (PA), sometimes known as probabilistic sensitivity analysis, propagates parameter uncertainty to the model output under each decision alternative and thereby quantifies decision uncertainty.
      • Baio G.
      • Dawid A.P.
      Probabilistic sensitivity analysis in health economics.
      Stat Methods Med Res. 2015; 24: 615-634
      PA simultaneously varies all parameters for which there is uncertainty. Uncertainty in the model inputs is characterized using probability distributions, p(θ).
      • Briggs A.H.
      Handling uncertainty in cost-effectiveness models.
      Pharmacoeconomics. 2000; 17: 479-500
      PA is often conducted using Monte Carlo methods where S parameter sets are drawn from p(θ) for the whole set of model parameters. The decision-analytic model is evaluated at all θs, s=1,…,S to estimate the costs and health outcomes of each of the T strategies. This produces a distribution for the NB for each strategy, which we denote by NBtθ, for t=1,…,T. In non-linear models, PA is required to estimate the expected NB of each strategy, Eθ[NBtθ].
      To compute EVSI, we require the PA simulations of the model inputs θs and the corresponding NB simulations NBtθs for s=1,…,S and t=1,…,T. These simulations should be saved in a matrix or spreadsheet form, often called a PA data set (the expression “data set” is not used in the traditional sense; here, the PA data set simply contains simulated values from distributions representing the uncertainty in the parameter estimates), where the columns contain the input parameters and the outputs from a decision-analytic model. Each row contains the parameter sets drawn from their distributions and their corresponding simulated results from a decision-analytic model output. An example of PA simulations is provided in the Supplemental Materials.
      Although PA results are necessary for all of the 4 efficient EVSI methods, 2 of the methods (RB and GA) require the PA data set from a traditional cost-effectiveness analysis whereas the other 2 (IS and MM) require the augmented PA data set presented in the following section.

      Expected Value of Partial Perfect Information

      The EVPPI computes the value of eliminating all uncertainty about a subset of the model parameters.
      • Felli J.C.
      • Hazen G.B.
      Sensitivity analysis and the expected value of perfect information.
      Med Decis Making. 1998; 18: 95-109
      ,
      • Coyle D.
      • Oakley J.
      Estimating the expected value of partial perfect information: a review of methods.
      Eur J Health Econ. 2008; 9: 251-259
      ,
      • Heath A.
      • Manolopoulou I.
      • Baio G.
      A review of methods for analysis of the expected value of information.
      Med Decis Making. 2017; 37: 747-758
      Specifically, the model parameters are split into two subsets θ=(φ,ψ), where we propose to gather further information about the model parameters φ. Typically, a proposed study does not collect information about all the underlying parameters in a decision-analytic model, and therefore ψ indicates the parameters that will not be directly informed by the proposed data collection.
      Mathematically, the EVPPI for the parameters φ is defined as:
      EVPPI=Eφ[maxtEψ|φ[NBtθ]]−maxtEθ[NBtθ],
      (1)


      where the inner expectation in the first term of equation (1) calculates the NB for each intervention conditional on φ. The second term calculates the value of the decision made with current information (ie, the expected NB of the treatment with the highest expected NB).
      The PA outcomes for φ are simulated and the EVPPI is estimated by computing the NB for each intervention conditional φ=φs for s=1,…,S. Approximation methods are available to estimate the conditional NB for each value φs.
      • Felli J.C.
      • Hazen G.B.
      Sensitivity analysis and the expected value of perfect information.
      Med Decis Making. 1998; 18: 95-109
      ,
      • Sadatsafavi M.
      • Bansback N.
      • Zafari Z.
      • Najafzadeh M.
      • Marra C.
      Need for speed: an efficient algorithm for calculation of single-parameter expected value of partial perfect information.
      Value Health. 2013; 16: 438-448
      , 
      • Strong M.
      • Oakley J.E.
      An efficient method for computing single-parameter partial expected value of perfect information.
      Med Decis Making. 2013; 33: 755-766
      , 
      • Strong M.
      • Oakley J.E.
      • Brennan A.
      Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach.
      Med Decis Making. 2014; 34: 311-326
      ,
      • Brennan A.
      • Kharroubi S.A.
      Efficient computation of partial expected value of sample information using Bayesian approximation.
      J Health Econ. 2007; 26: 122-148
      ,
      • Heath A.
      • Manolopoulou I.
      • Baio G.
      Estimating the expected value of partial perfect information in health economic evaluations using integrated nested Laplace approximation.
      Stat Med. 2016; 35: 4264-4280
      These have been assessed
      • Coyle D.
      • Oakley J.
      Estimating the expected value of partial perfect information: a review of methods.
      Eur J Health Econ. 2008; 9: 251-259
      ,
      • Heath A.
      • Manolopoulou I.
      • Baio G.
      A review of methods for analysis of the expected value of information.
      Med Decis Making. 2017; 37: 747-758
      and can be obtained using available software.
      • Baio G.
      • Berardi A.
      • Heath A.
      Bayesian Cost Effectiveness Analysis with the R Package BCEA.
      Springer, 2017
      , 
      • Strong M.
      • Breeze P.
      • Thomas C.
      • Brennan A.
      SAVI - Sheffield accelerated value of information, release version 1.013 (2014-12-11).
      http://savi.shef.ac.uk/SAVI/
      Date accessed: June 9, 2019
      , 
      • Strong M.
      Partial EVPPI functions.
      http://www.shef.ac.uk/polopoly_fs/1.305039!/file/R_functions.txt
      Date accessed: June 9, 2019
      The MM and IS methods require a PA data set with T additional columns that contain Monte Carlo estimates of the expected NB conditional on φs for s=1,…,S, but averaged over the uncertain values of ψ. We denote these simulations by ηts.

      EVSI Methods

      EVSI for a proposed research strategy that collects additional data, denoted by X, is defined as:
      EVSI=EX[maxtEθ|X[NBtθ]]−maxtEθ[NBtθ].
      (2)


      In this setup, X are observable but not yet observed. Computing EVSI helps us determine whether we should defer making a decision on the optimal intervention, among the T possible options, and collect X instead.
      In line with the Bayesian approach, which underpins VOI analyses, the distribution of X is defined by p(X,θ)=p(θ)p(X|θ), where p(θ) is the PA distribution of θ and p(X|θ) is the sampling distribution of the data. In this setting, p(θ) is a “prior” distribution that represents the current uncertainty level for the model parameters before observing the additional data X. This distribution can be based on the observed data that are used to construct the decision-analytic model. We note that X will give information about the subset of model parameters φ, where φ could be the whole set θ. By definition, this implies that ψ and X are independent given φ and that p(X|θ)=p(X|φ).
      The second term of equation (2) can be estimated from the initial PA. To obtain simulations for X, we simulate potential study outcomes Xs,s=1,…, S, for each row of the PA data set (ie, we generate a single sample from p(X|φs) for s=1,…,S). Traditionally, EVSI has been computationally demanding and methodologically challenging because a large number of simulations are needed to estimate the posterior mean of the NB for each Xs. This requires a Bayesian model for the distribution of the data and the parameters, where the posterior mean is given by:
      μt(Xs)=Eθ|Xs[NBtθ].
      (3)


      The EVSI approximation methods in this report estimate μt(X) with a reduced computational burden. For computational simplicity and stability,
      • Strong M.
      • Oakley J.E.
      • Brennan A.
      Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach.
      Med Decis Making. 2014; 34: 311-326
      VOI calculations should be undertaken using incremental NB (INB) defined, without loss of generality, as:
      INBtθ=NBtθ−NB1θ,


      for t=2,…,T. We could also use the opportunity loss, defined as the INB of all treatments from the intervention that is currently associated with the maximum expected NB (ie, the optimal strategy). By estimating the posterior mean of the INB, we only need to add T−1 columns containing the posterior mean conditional on Xs to our PA data set. Once these simulations, denoted by μt(Xs), are available, the EVSI can be estimated by:
      1S∑s=1Smaxt{μt(Xs)}−maxt{1S∑s=1Sμt(Xs)},
      (4)


      where μ1(Xs)=0 for all s=1,…,S.

      EVSI Calculation

      We present step-by-step guides for the implementation of the 4 EVSI methods in Box 1, Box 2, Box 3, Box 4 through 4. Box 1 contains the RB algorithm
      • Strong M.
      • Oakley J.E.
      • Brennan A.
      • Breeze P.
      Estimating the expected value of sample information using the probabilistic sensitivity analysis sample: a fast, nonparametric regression-based method.
      Med Decis Making. 2015; 35: 570-583
      to estimate EVSI where the fitted values from a regression model between the simulated INB and a low-dimensional summary statistic from the simulated data set Xs are used to calculate μt(X). Box 2 contains the IS algorithm
      • Menzies N.A.
      An efficient estimator for the expected value of sample information.
      Med Decis Making. 2016; 36: 308-320
      where the simulations to compute EVPPI are reweighted using the statistical likelihood to calculate μt(X). Box 3 presents the GA alogrithm
      • Jalal H.
      • Alarid-Escudero F.
      A Gaussian approximation approach for value of information analysis.
      Med Decis Making. 2018; 38: 174-188
      where the prior distribution for φ are rescaled and combined using a linear meta-model to estimate μt(X). Finally, Box 4 presents the MM algorithm
      • Heath A.
      • Manolopoulou I.
      • Baio G.
      Efficient Monte Carlo estimation of the expected value of sample information using moment matching.
      Med Decis Making. 2018; 38: 163-173
      where the simulations to compute EVPPI are rescaled by a factor estimated using efficient nested MC sampling to calculate μt(X).
      Step-by-step guide for regression-based method.
      • 1. Perform PA simulation to obtain θs and INBtθs, s = 1,…,S.
      • 2. For each s = 1,…,S:
        • (a) Simulate a dataset Xs from p(X|ϕs).
        • (b) Summarise this dataset, producing the quantity or quantities that would be estimated from it in a trial, such as the mean. We denote this summary of the dataset W(Xs).
      • 3. Fit T−1 regression models with INBtθs as the outcome and W(Xs) as the covariates.
      • 4. Extract the fitted values from these regressions to estimate μt(Xs).
      The algorithm used to estimate EVSI using the RB method,
      • Strong M.
      • Oakley J.E.
      • Brennan A.
      • Breeze P.
      Estimating the expected value of sample information using the probabilistic sensitivity analysis sample: a fast, nonparametric regression-based method.
      Med Decis Making. 2015; 35: 570-583
      based on fitting a regression model and determining a low-dimensional summary of the simulated dataset X.
      EVSI indicates expected value of sample information; INB, incremental net benefit; PA, probabilistic analysis; RB, regression-based; θ, set of inputs in a decision-analytic model; p(θ), probability distributions to characterize uncertainty in the model inputs; S, number of parameter sets that are drawn from p(θ) in PA; T, number of considered interventions; θ=(φ,ψ), φ are model parameters for which we are aiming to collect further information and ψ are parameters that will not be directly informed by the proposed data collection; X, additional data proposed to be collected; μ, posterior mean; W(X), a summary measure for the data.
      Step-by-step guide for importance sampling method.
      • 1. Perform PA simulation to obtain θs and INBtθs, s = 1,…,S.
      • 2. Estimate EVPPI to obtain simulations of ηts, the inner expectation in the first term of equation:
      EVPPI=Eϕ[maxtEψ|ϕ[NBtθ]]−maxtEθ[NBtθ].


      • 3. For each s = 1,…,S:
        • (a) Simulate a dataset Xs from p(X|ϕs).
        • (b) Compute the likelihood of Xs conditional on every PA simulation for θ, i.e., calculate the likelihood Lr of Xs for each parameter set θr with r = 1,…,S.
        • (c) Compute lr=Lr∑r=1SLr so lr sums to 1.
        • (d) Calculate the weighted sum of ηtr, ∑r=1Slrηtr.
      • 4. Each weighted sum estimates μt(Xs).
      The algorithm used to estimate EVSI using the IS method,
      • Menzies N.A.
      An efficient estimator for the expected value of sample information.
      Med Decis Making. 2016; 36: 308-320
      based on calculating weighted sums of ηts using weights calculated from the likelihood of the data given the parameters. This algorithm is listed as Method 2 in the original IS development but represents the most accurate method for EVSI estimation.
      • Menzies N.A.
      An efficient estimator for the expected value of sample information.
      Med Decis Making. 2016; 36: 308-320
      EVPPI indicates expected value of partial perfect information; EVSI, expected value of sample information; INB, incremental net benefit; IS, importance sampling; PA, probabilistic analysis; θ, set of inputs in a decision-analytic model; p(θ), probability distributions to characterize uncertainty in the model inputs; S, number of parameter sets that are drawn from p(θ) in PA; T, number of considered interventions; θ=(φ,ψ), φ are model parameters for which we are aiming to collect further information and ψ are parameters that will not be directly informed by the proposed data collection; X, additional data proposed to be collected; μ, posterior mean; η, PA simulations used to compute EVPPI.
      Step-by-step guide for Gaussian approximation method.
      • 1. Perform PA simulation to obtain θs and INBtθs, s = 1,…,S.
      • 2. Fit T − 1 regression models with INBtθs as outcomes and ϕs as covariates.
      • 3. For each element of ϕ denoted by ϕp for p = 1,…,P:
        • (a) Determine the prior effective sample size n0p. Methods to estimate the prior effective sample size are presented in [21].
        • (b) For a proposed study with N participants, compute a weighted sum of the simulations of ϕpand ϕp¯, the mean of the pth parameter, by multiplying ϕp by NN+n0p and multiplying ϕp¯ by (1−NN+n0p).
      • 4. Using the regression models from Step 2, predict the model outcomes for the rescaled ϕ simulations.
      • 5. The fitted values from Step 4 estimate μt(Xs).
      The algorithm used to estimate EVSI using the GA method,
      • Jalal H.
      • Alarid-Escudero F.
      A Gaussian approximation approach for value of information analysis.
      Med Decis Making. 2018; 38: 174-188
      based on rescaling the PA distributions for φ and estimating the INB from these rescaled distribution using the meta-model. The original GA algorithm worked with opportunity loss rather than INB
      • Jalal H.
      • Alarid-Escudero F.
      A Gaussian approximation approach for value of information analysis.
      Med Decis Making. 2018; 38: 174-188
      ; in this article we have used INB for consistency across the reviewed methods.
      EVSI indicates expected value of sample information; GA, Gaussian approximation; INB, incremental net benefit; PA, probabilistic analysis; θ, set of inputs in a decision-analytic model; S, number of parameter sets that are drawn from p(θ) in PA; T, number of considered interventions; θ=(φ,ψ), φ are model parameters for which we are aiming to collect further information and ψ are parameters that will not be directly informed by the proposed data collection; X, additional data proposed to be collected; μ, posterior mean.
      Step-by-step guide for moment matching method.
      • 1. Perform PA simulation to obtain θs and INBtθs, s = 1,…,S.
      • 2. Estimate EVPPI to obtain simulations of ηts, the inner expectation in the first term of equation:
      EVPPI=Eϕ[maxtEψ|ϕ[NBtθ]]−maxtEθ[NBtθ].


      • 3. Extract Q, with 30 < Q < 50, sample quantiles from the simulations of ϕ, denoted by ϕq. To estimate EVSI across sample size, you also require Q proposed sample sizes denoted by Nq for q = 1,…,Q.
      • 4. For q = 1,…,Q:
        • (a) Simulate a future dataset, with sample size Nq if required, from the sampling distribution p(X|ϕq).
        • (b) Use Bayesian methods to update the distribution of the model parameters.
        • (c) Rerun the probabilistic sensitivity analysis to update the distribution of the net monetary benefit.
        • (d) Calculate the variance of the net monetary benefit, denoted by σq2.
      • 5. Calculate the rescaling factor σ2
        • (a) If estimating EVSI for a single sample size, σ2=Var[NBtθs]−1Q∑q=1Qσq2.
        • (b) If estimating EVSI across sample size, σ2 for a given N is estimated by estimating h in the non-linear regression function NN+h with Var[NBtθs]−σq2 as the dependant variable and Nq as the independent variable.
      • 6. Rescale the simulations of ηts so their variance is equal to σ2 using:
      σ2Var(ηts)ηts+(1−σ2Var(ηts))ηt¯,


      where ηt¯ is the mean of ηt simulations and Var(ηts) is the sample variance of ηt simulations.
      • 7. These rescaled simulations estimate μt(Xs).
      The algorithm used to estimate EVSI using the MM method,
      • Heath A.
      • Manolopoulou I.
      • Baio G.
      Efficient Monte Carlo estimation of the expected value of sample information using moment matching.
      Med Decis Making. 2018; 38: 163-173
      based on rescaling the simulations of ηts. In this method, the rescaling factor is estimated using Q nested Monte Carlo simulations, where Q is usually between 30 and 50 and should be as high as possible whilst allowing for a feasible computational burden.
      • Heath A.
      • Manolopoulou I.
      • Baio G.
      Efficient Monte Carlo estimation of the expected value of sample information using moment matching.
      Med Decis Making. 2018; 38: 163-173
      Functions are available in the EVSI package in R to perform steps 2-7.
      EVPPI indicates expected value of partial perfect information; EVSI, expected value of sample information; INB, incremental net benefit; MM, moment matching; PA, probabilistic analysis; θ, set of inputs in a decision-analytic model; p(θ), probability distributions to characterize uncertainty in the model inputs; S, number of parameter sets that are drawn from p(θ) in PA; T, number of considered interventions; θ=(φ,ψ), φ are model parameters for which we are aiming to collect further information and ψ are parameters that will not be directly informed by the proposed data collection; X, additional data proposed to be collected; μ, posterior mean; η, PA simulations used to compute EVPPI; Q, number of times PA is performed.

      Expertise and Skills Recommended for EVSI Calculation

      We have identified 3 skills that are recommended for analysts before computing EVSI using the reviewed approximation methods (Table 2). No method requires all 3 skills but each method requires at least 1 skill.
      Table 2The recommended skills and expertise to compute EVSI with the regression-based, importance sampling, Gaussian approximation, and moment matching methods.
      #Recommended skills and expertiseRBISGAMM
      1Regression methods✓✓
      2Specification of likelihoods✓✓
      The skill/input may be required.
      3Bayesian updating
      The skill/input may be required.
      		✓
      Note. ✓ indicates that the skill/input is required.
      GA indicates Gaussian approximation method; IS, importance sampling method; MM, moment matching method; RB, regression-based method.
      ∗ The skill/input may be required.

      Regression Methods

      Regression methods that model the relationship between a set of predictors and an outcome of interest are required for both the RB and GA methods. Both methods use regression metamodeling to model the INB as a function of model inputs or quantities related to model inputs. The 2 methods differ because they require alternative covariates.
      Both of these EVSI methods model potentially complex relationships using models that require limited information about the functional relationship between the independent and dependent variables. To account for potential nonlinear relationships between these variables, flexible regression methods using generalized additive models (GAMs)
      • Wood S.N.
      Thin plate regression splines.
      J Royal Stat Soc. 2003; 65: 95-114
      are the predominant regression methods for EVSI calculation. Standard software is available to fit GAMs,
      • Wood S.
      • Pya N.
      • Kneib T.
      • et al.
      mgcv: mixed GAM computation vehicle with GCV/AIC/REML smoothness estimation and GAMMs by REML/PQL.
      University of Bath, 2016
      , 
      • Royston P.
      • Ambler G.
      Gam: stata module for generalised additive models.
      Boston College Department of Economics, 2012
      , 
      • Cai W.
      Fitting generalized additive models with the gam procedure in SAS 9.2. SAS Global Forum.
      https://support.sas.com/resources/papers/proceedings/pdfs/sgf2008/378-2008.pdf
      but the EVSI estimate can be affected by the structure of the GAM model and, thus, an understanding of these models is recommended. Gaussian Processes
      • Rasmussen C.
      • Williams C.
      Gaussian processes for machine learning.
      http://www.gaussianprocess.org/gpml/chapters/RW.pdf
      have also been suggested for problems with greater than 6 covariates when using the RB method,
      • Strong M.
      • Oakley J.E.
      • Brennan A.
      • Breeze P.
      Estimating the expected value of sample information using the probabilistic sensitivity analysis sample: a fast, nonparametric regression-based method.
      Med Decis Making. 2015; 35: 570-583
      so knowledge of these methods is also recommended.
      If regression is used to estimate EVSI, it is important to assess whether the regression model has correctly captured the relationship between the independent and dependent variables.
      • Strong M.
      • Oakley J.E.
      • Brennan A.
      • Breeze P.
      Estimating the expected value of sample information using the probabilistic sensitivity analysis sample: a fast, nonparametric regression-based method.
      Med Decis Making. 2015; 35: 570-583
      Regression methods assume that the residual error is uncorrelated with the fitted values, has zero mean, and has a trivial covariance structure. These assumptions can be confirmed by observing no clusters, systematic features, or outliers in the plot of the residuals against the fitted values.
      • Hastie T.
      • Tibshirani R.
      Generalized Additive Models.
      CRC Press, Boca Raton, FL1990

      Specification of Data Generating Distribution

      EVSI computes the expected value of collecting information in a future research study. The future research study would collect a potential data set, X, with NO different clinical, health, or economic outcomes from N participants. For example, a clinical trial would collect the primary and secondary outcomes from each participant.
      All EVSI computation methods require the simulation of potential data sets (except the GA method in specific circumstances; see the section on Bayesian Updating). This is achieved by specifying the assumed data generating distribution p(X|φ). Thus it is recommended that analysts have knowledge of probability distributions.
      For the RB method, the simulated datasets Xs for s=1,…,S must be summarized to reflect how the data would be analyzed after the trial. As such, expertise in the statistical analysis of study results, such as maximum likelihood procedures, is recommended. The GA method can also be based on a summary statistic for specific data collection exercises (see Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.02.010 for more information).
      For the IS method, the analytic likelihood for X must be specified and coded as a function. Likelihood is used in its proper statistical meaning and, therefore, this specification requires statistical expertise. For this likelihood specification, it is recommended that analysts have knowledge of statistical theory, function development, and coding.

      Bayesian Updating

      Bayesian methods are a statistical paradigm in which conclusions are formally updated as evidence accumulates.
      Food and Drug Administration
      Guidance for the use of Bayesian statistics in medical device clinical trials.
      https://www.fda.gov/regulatory-information/search-fda-guidance-documents/guidance-use-bayesian-statistics-medical-device-clinical-trials
      A full Bayesian model consists of probability distributions specified for the model inputs p(θ), the prior distribution defined using the PA distributions, and the sampling distribution for the proposed data collection. Based on these distributions, Bayesian methods update the distribution of the parameters conditional on the data X to produce a posterior distribution. In practice, this is undertaken using specialized software such as BUGS, JAGS, or Stan.
      • Lunn D.J.
      • Thomas A.
      • Best N.
      • Spiegelhalter D.
      WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility.
      Statistics and Computing. 2000; 10: 325-337
      , 
      • Plummer M.
      rjags: Bayesian graphical models using MCMC: R package version 4-4.
      https://cran.r-project.org/web/packages/rjags/rjags.pdf
      , 
      • Gelman A.
      • Lee D.
      • Guo J.
      • Stan
      A probabilistic programming language for Bayesian inference and optimization.
      J Educ Behav Stat. 2015; 40: 530-543
      The MM method is explicitly based on Bayesian analysis and thus we recommend that analysts have expertise in one of these programs and in the specification of Bayesian models.
      Additionally, the GA method requires an estimate of the prior effective sample size.
      • Morita S.
      • Thall P.F.
      • Muller P.
      Determining the effective sample size of a parametric prior.
      Biometrics. 2008; 64: 595-602
      The prior effective sample size, denoted by n0, is the number of participants required in a trial identical to the currently proposed study that would need to be studied to obtain the level of information in the prior. If the prior and sampling distribution for X are conjugate to each other, then the prior effective sample size can be estimated directly (see Supplemental Materials for more information).
      • Morita S.
      • Thall P.F.
      • Muller P.
      Determining the effective sample size of a parametric prior.
      Biometrics. 2008; 64: 595-602
      If this is not possible, 2 algorithms to compute the prior effective sample size are available.
      • Jalal H.
      • Alarid-Escudero F.
      A Gaussian approximation approach for value of information analysis.
      Med Decis Making. 2018; 38: 174-188
      The first of these can be used if a single summary statistic is a good estimator of the model parameter and the second, based on a full Bayesian analysis, can be used in all settings. Thus the complexity of the prior effective sample size estimation is based on the complexity of the decision-analytic model and data collection exercise. See Supplemental Materials for more details on these methods for prior effective sample size estimation.

      EVSI Methods Recommendations Based on Features of the Decision-Analytic Problem

      Because each method is different, the most suitable method for EVSI calculation will depend on the decision-analytic model, the sampling distribution for the data, the expertise of the analyst, and the amount of computation time available. The following section describes features of the decision problem and recommendations about which method should be used in these settings to help analysts select the most appropriate method (Table 3).
      Table 3The recommendations for EVSI approximation methods based on features of the decision-analytic problem.
      No.FeaturesRBISGAMM
      1Designing complex studies collecting a large number of outcomes (more than 6 outcomes)✓✓
      2Having only probabilistic analysis results without access to the decision-analytic model
      EVSI estimation does not require rerunning the decision-analytic model. This may be advantagous if the analyst does not have access to the decision-analytic model or if the decision-analytic model is computationally expensive.
      		✓✓✓
      4Identifying optimal sample size by examining different study sizes with same computational cost✓✓
      5Including uncertainty quantification in the estimate✓✓✓
      6Proposing studies with a large sample size
      The IS method can be computationally challenging, and inaccurate, to estimate EVSI for proposed studies with a sample size over 1,000.8
      		✓✓✓
      7Proposing studies with small prior effective sample size✓✓✓
      8Proposing studies with small sample size✓✓✓
      Note. ✓ indicates that the method is appropriate for the given feature.
      GA indicates Gaussian approximation method; IS, importance sampling method; MM, moment matching method; RB, regression-based method.
      ∗ EVSI estimation does not require rerunning the decision-analytic model. This may be advantagous if the analyst does not have access to the decision-analytic model or if the decision-analytic model is computationally expensive.
      † The IS method can be computationally challenging, and inaccurate, to estimate EVSI for proposed studies with a sample size over 1,000.
      • Heath A.
      • Kunst N.
      • Jackson C.
      • et al.
      Calculating the expected value of sample information in practice: Considerations from three case studies.
      Medical Decision Making. 2020; (In press)https://doi.org/10.1177/0272989X20912402

      Challenging to Run Decision-Analytic Model

      In some settings, analysts may not have access to the decision-analytic model and only have access to the PA results. Additionally, if the decision-analytic model with a full PA is computationally expensive to run, the analyst may wish to avoid rerunning the model. In these settings, we recommend using the RB, GA, or IS methods because these methods only require access to the PA data set.

      Optimizing the Sample Size of the Future Study

      EVSI can be used to help determine the optimal design for the future study, including optimizing the sample size. The GA and MM methods can be used to calculate EVSI across different sample sizes with a similar computational cost to a single EVSI estimate and are therefore recommended if the analyst is looking to optimize the overall sample size. Of note, the RB method typically has a low computational cost
      • Heath A.
      • Kunst N.
      • Jackson C.
      • et al.
      Calculating the expected value of sample information in practice: Considerations from three case studies.
      Medical Decision Making. 2020; (In press)https://doi.org/10.1177/0272989X20912402
      so it can be used repeatedly to calculate EVSI across sample sizes.

      Designing Studies with Multiple Outcomes

      GAM regression, used for the RB and GA methods, can be challenging to implement with 6 or more covariates.
      • Strong M.
      • Oakley J.E.
      • Brennan A.
      Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach.
      Med Decis Making. 2014; 34: 311-326
      Therefore, the IS and MM methods are recommended for settings where the future study collects more than 6 outcomes. Alternatives to GAM regression, such as the Gaussian Process regression, can be used to implement the RB method in these settings but are more computationally expensive and challenging to implement. Additionally, linear regression can be used to implement the GA method, but this may be insufficiently accurate for EVSI estimation. Note that although EVPPI estimation, required for MM and IS methods, is often based on regression methods, this estimation is only required once to estimate EVSI. Consequently, the computational burden of using more complex regression methods is reduced significantly for the IS and MM methods.

      Designing Studies With Extreme Sample Sizes

      The sample size of the proposed study affects the considered EVSI approximation methods. First, the IS method is not recommended for studies where the sample size of the future data is over 1,000
      • Heath A.
      • Kunst N.
      • Jackson C.
      • et al.
      Calculating the expected value of sample information in practice: Considerations from three case studies.
      Medical Decision Making. 2020; (In press)https://doi.org/10.1177/0272989X20912402
      because of computational issues when the likelihood tends to 0 as the sample size increases. Second, the MM method is not recommended for studies collecting information on under 10 patients because the moment matching approximation is not sufficiently accurate.
      • Heath A.
      • Manolopoulou I.
      • Baio G.
      Efficient Monte Carlo estimation of the expected value of sample information using moment matching.
      Med Decis Making. 2018; 38: 163-173
      Finally, the GA method is not recommended when the prior effective sample size n0 is under 10 because the Gaussian approximation is not accurate.
      • Jalal H.
      • Alarid-Escudero F.
      A Gaussian approximation approach for value of information analysis.
      Med Decis Making. 2018; 38: 174-188

      Quantifying Uncertainty in EVSI Estimates

      The RB, GA, and MM methods can be used to estimate the uncertainty associated with the EVSI estimation procedure. Specifically, for the RB and GA methods, the uncertainty can be obtained using an algorithm developed by Strong et al,
      • Strong M.
      • Oakley J.E.
      • Brennan A.
      • Breeze P.
      Estimating the expected value of sample information using the probabilistic sensitivity analysis sample: a fast, nonparametric regression-based method.
      Med Decis Making. 2015; 35: 570-583
      provided the regression model gives good model fit. For the MM method, the uncertainty estimates are obtained in the process of computing EVSI across different sample sizes.
      • Heath A.
      • Manolopoulou I.
      • Baio G.
      Estimating the expected value of sample information across different sample sizes using moment matching and nonlinear regression.
      Med Decis Making. 2019; 39: 347-359

      Limitations for the EVSI Calculation Methods

      EVSI estimation will be poor for the RB and GA methods if the regression model does not capture the relationship of interest accurately, as assessed using model checking procedures. The IS method will be inaccurate if the likelihood is not specified correctly or cannot be estimated owing to numerical issues. Finally, the MM method is less accurate for studies that will have limited impact on the underlying uncertainty in the decision-analytic model (ie, the EVPPI of φ should be high compared with the value of reducing all model uncertainty [such as EVPI] to use the MM method,
      • Heath A.
      • Baio G.
      Calculating the expected value of sample information using efficient nested Monte Carlo: a tutorial.
      Value Health. 2018; 21: 1299-1304
      ideally greater than 40%).

      EVSI Methods in Practice

      Resources for the EVSI Methods

      An online application Sheffield Accelerated Value of Information (SAVI)
      • Strong M.
      • Breeze P.
      • Thomas C.
      • Brennan A.
      SAVI - Sheffield accelerated value of information, release version 1.013 (2014-12-11).
      http://savi.shef.ac.uk/SAVI/
      Date accessed: June 9, 2019
      computes the EVPPI using the RB method. It fits a regression model between the INBtθs and the parameters of interest φ. Because of the similarity between the EVPPI and EVSI calculation methods, SAVI can be used to compute EVSI once the future data sets have been summarized: W(Xs), s=1,…,S. This is achieved by augmenting the PA data set with (a) column(s) containing the data summaries, which must be saved and uploaded into SAVI. EVSI is then equal to the “EVPPI” calculated for the column(s) containing the data summary.
      An online repository is available for the GA method (https://zenodo.org/record/3263876), which contains a function that calculates the fitted values μt(Xs) from a GAM regression model
      • Wood S.N.
      Thin plate regression splines.
      J Royal Stat Soc. 2003; 65: 95-114
      and estimates of n0p for p=1,…,P.
      The EVSI package in R has been developed to implement the MM method based on a Bayesian decision-analytic model. The manual is available
      • Heath A.
      • Baio G.
      Calculating value of information measures for health economic evaluations within R.
      https://github.com/annaheath/EVSI/blob/master/VoI%20using%20R%20Manual.pdf
      and the EVSI package can be installed in R using the command: devtools::install_github(“annaheath/EVSI”).

      Examples of the EVSI Methods Application

      To aid the implementation of the reviewed EVSI methods, we have created a comprehensive GitHub repository that presents the code used to compute EVSI in the original publications, available at https://github.com/convoigroup/EVSIExample. This repository also contains a suite of practical examples that demonstrate EVSI calculations across several real-world applications using common decision-analytic model structures developed in R.
      • Kunst N.R.
      • Alarid-Escudero F.
      • Paltiel A.D.
      • Wang S.
      A value of information analysis of research on the 21-gene assay for breast cancer management.
      Value Health. 2019; 22: 1102-1110
      Our GitHub repository demonstrates how the reviewed methods can be used in practice and will help analysts to implement VOI methods in their own work and increase the feasibility and accessibility of the EVSI methods as they become an important and required tool.

      EVSI Results Presentation

      The EVSI R package contains several graphical displays to present EVSI and related quantities.
      • Heath A.
      • Baio G.
      Calculating value of information measures for health economic evaluations within R.
      https://github.com/annaheath/EVSI/blob/master/VoI%20using%20R%20Manual.pdf
      EVSI results can be loaded into the EVSI package, irrespective of the computation method used. These graphics can be displayed directly in R or explored using a dynamic graphical display launched from within R. Analysts unfamiliar with R can explore these graphics using an online interface at https://egon.stats.ucl.ac.uk/projects/EVSI/Test/. The optimal sample size estimated using VOI methods can also be presented in form of a curve of optimal sample size (COSS).
      • Jutkowitz E.
      • Alarid-Escudero F.
      • Kuntz K.M.
      • Jalal H.
      The curve of optimal sample size (COSS): a graphical representation of the optimal sample size from a value of information analysis.
      Pharmacoeconomics. 2019; 37: 871-877
      The COSS graphically displays optimal sample sizes for proposed study designs over a range of willingness-to-pay thresholds, including the impact of variation and uncertainty in VOI parameters on those optimal sample sizes.

      Limitations

      EVSI estimation is a rapidly developing subject, and future research may propose new efficient methods than those covered in this report. For example, promising results have been shown for multilevel Monte Carlo methods.
      • Hironaka T.
      • Giles M.B.
      • Goda T.
      • Thom H.
      Multilevel Monte Carlo estimation of the expected value of sample information.
      2019
      Nonetheless, we believe the recommendations for analysts’ expertise and skills necessary for EVSI estimation and other included recommendations will remain relevant and may be applicable to new EVSI approximation methods. Next, the present recommendations only consider model-based EVSI methods. Although model-based VOI is most commonly used, minimal modeling approaches can be applied when decision makers have access to information from a clinical study on comprehensive measures of health outcomes and their incorporated uncertainty.
      • Meltzer D.O.
      • Hoomans T.
      • Chung J.W.
      • Basu A.
      Minimal modeling approaches to value of information analysis for health research.
      Med Decis Making. 2011; 31: E1-E22
      Nevertheless, because these minimal modeling approaches are only feasible in certain circumstances, they cannot be compared directly with the reviewed EVSI methods. Furthermore, our EVSI definition is based on an assumption of risk-neutral decision makers who base their decision on maximizing expected values following the normative theory presented by Arrow and Lind.
      • Briggs A.
      • Sculpher M.
      • Claxton K.
      Decision Modelling for Health Economic Evaluation.
      Oxford University Press, Oxford2006
      ,
      • Arrow K.J.
      • Lind R.C.
      Uncertainty and the evaluation of public investment decisions.
      Am Econ Rev. 1970; 60
      Lastly, these EVSI methods do not consider structural uncertainty, and this is one of the recommended future research directions as indicated by the ISPOR VOI Good Practice Task Force.
      • Rothery C.
      • Strong M.
      • Koffijberg H.
      • et al.
      Value of information analytical methods: report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force.
      Value Health. 2020; 23: 277-286

      Conclusion

      VOI analysis has the potential to guide policy makers in the prioritization and design of future research studies, thereby improving decision making. Increasingly, HTA agencies are acknowledging the potential of VOI analyses and may start recommending EVSI to prioritize and design future research. In this study, members of ConVOI have provided practical guidance and recommendations to facilitate the implementation of 4 EVSI approximation methods. Our report outlines the inputs and software required to use each of these methods and also summarizes the recommended analysts’ skills and expertise needed to implement them based on the features of decision-analytic problems. The direction of future research in VOI is highlighted in Rothery et al,
      • Rothery C.
      • Strong M.
      • Koffijberg H.
      • et al.
      Value of information analytical methods: report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force.
      Value Health. 2020; 23: 277-286
      but future research should also focus on improving VOI implementation and its communication in clinical practice.

      Acknowledgments

      We thank Dr Nicky Welton for her comments and discussions as part of the ConVOI group, and Yidi Jiang for her help preparing the EVSI application examples. Last but not least, we thank the Editor and 5 anonymous Reviewers for their valuable comments. N.K. was funded by the Research Council of Norway (276146 and 304034) and LINK Medical Research. EW was funded by Norwich Medical School. F.A.E. was funded by the National Cancer Institute (U01- CA-199335) as part of the Cancer Intervention and Surveillance Modeling Network (CISNET). G.B. was partially funded by a research grant sponsored by Mapi/ICON at University College London. A.B. was funded through Higher Education Funding Council for England and a range of UK and International research grant award bodies. M.F. was funded by the Stanford Interdisciplinary Graduate Fellowship. D.G. has no funding to declare. J.D.G.F. was funded in part by a grant from Stanford’s Precision Health and Integrated Diagnostics Center (PHIND). C.J. was funded the UK Medical Research Council program MCUU00002/11. M.S. has no funding to declare. H.J. was funded by NIH/NCATS grant 1KL2TR0001856. N.M. was supported by National Institutes of Health (NIH) [R01AI112438-02.]. H.T. was funded by the NIHR Biomedical Research Centre at the University Hospitals Bristol NHS Foundation Trust and the University of Bristol. A.H. was funded through an Innovative Clinical Trials Multi-year Grant from the Canadian Institutes of Health Research (funding reference number MYG-151207; 2017-2020). The funding agreement ensured the authors’ independence in designing the study, interpreting the data, writing, and publishing the report.

      Supplemental Material

      References

        • Alarid-Escudero F.
        • Enns E.A.
        • Kuntz K.M.
        • Michaud T.L.
        • Jalal H.
        “Time traveling is just too dangerous” but some methods are worth revisiting: the advantages of expected loss curves over cost-effectiveness acceptability curves and frontier.
        Value Health. 2019; 22: 611-618
        View in Article
        • McCabe C.
        • Claxton K.
        • Culyer A.J.
        The NICE cost-effectiveness threshold: what it is and what that means.
        Pharmacoeconomics. 2008; 26: 733-744
        View in Article
        • National Institute for Health and Care Excellence
        Guide to the methods of technology appraisal 2013.
        https://www.ncbi.nlm.nih.gov/books/NBK395867/pdf/Bookshelf_NBK395867.pdf
        (Accessed March 25, 2019)
        View in Article
        • Norwegian Medicines Agency
        Guidelines for the submission of documentation for single technology assessment (STA) of pharmaceuticals.
        https://legemiddelverket.no/Documents/English/Public%20funding%20and%20pricing/Documentation%20for%20STA/Guidelines%20151018.pdf
        Date accessed: March 25, 2019
        View in Article
        • National Institute of Health and Care Excellence
        Developing NICE guidelines: the manual.
        https://www.nice.org.uk/process/pmg20/chapter/introduction-and-overview
        (Accessed March 25, 2019)
        View in Article
        • National Institute of Health and Care Excellence
        Research recommendations.
        https://www.nice.org.uk/Media/Default/About/what-we-do/Research-and-development/Research-Recommendation-Process-and-Methods-Guide-2015.pdf
        (Accessed March 25, 2019)
        View in Article
        • CADTH
        Guidelines for the Economic Evaluation of Health Technologies: Canada. 4th Edition.
        https://www.cadth.ca/about-cadth/how-we-do-it/methods-and-guidelines/guidelines-for-the-economic-evaluation-of-health-technologies-canada
        (Accessed March 25, 2019)
        View in Article
        • Heath A.
        • Kunst N.
        • Jackson C.
        • et al.
        Calculating the expected value of sample information in practice: Considerations from three case studies.
        Medical Decision Making. 2020; (In press)https://doi.org/10.1177/0272989X20912402
        View in Article
        • Ades A.E.
        • Lu G.
        • Claxton K.
        Expected value of sample information calculations in medical decision modeling.
        Med Decis Making. 2004; 24: 207-227
        View in Article
        • Felli J.C.
        • Hazen G.B.
        Sensitivity analysis and the expected value of perfect information.
        Med Decis Making. 1998; 18: 95-109
        View in Article
        • Brennan A.
        • Kharroubi S.
        • O'Hagan A.
        • Chilcott J.
        Calculating partial expected value of perfect information via Monte Carlo sampling algorithms.
        Med Decis Making. 2007; 27: 448-470
        View in Article
        • Madan J.
        • Ades A.E.
        • Price M.
        • et al.
        Strategies for efficient computation of the expected value of partial perfect information.
        Med Decis Making. 2014; 34: 327-342
        View in Article
        • Oakley J.
        Value of information for complex cost-effectiveness models.
        University of Sheffield, Sheffield, UK2002
        View in Article
        • Sadatsafavi M.
        • Bansback N.
        • Zafari Z.
        • Najafzadeh M.
        • Marra C.
        Need for speed: an efficient algorithm for calculation of single-parameter expected value of partial perfect information.
        Value Health. 2013; 16: 438-448
        View in Article
        • Strong M.
        • Oakley J.E.
        An efficient method for computing single-parameter partial expected value of perfect information.
        Med Decis Making. 2013; 33: 755-766
        View in Article
        • Strong M.
        • Oakley J.E.
        • Brennan A.
        Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach.
        Med Decis Making. 2014; 34: 311-326
        View in Article
        • Giles M.B.
        • Goda T.
        Decision-making under uncertainty: using MLMC for efficient estimation of EVPPI.
        Statistics and Computing. 2019; 29: 739-751
        View in Article
        • Welton N.J.
        • Madan J.J.
        • Caldwell D.M.
        • Peters T.J.
        • Ades A.E.
        Expected value of sample information for multi-arm cluster randomized trials with binary outcomes.
        Med Decis Making. 2013; 34: 352-365
        View in Article
        • Strong M.
        • Oakley J.E.
        • Brennan A.
        • Breeze P.
        Estimating the expected value of sample information using the probabilistic sensitivity analysis sample: a fast, nonparametric regression-based method.
        Med Decis Making. 2015; 35: 570-583
        View in Article
        • Menzies N.A.
        An efficient estimator for the expected value of sample information.
        Med Decis Making. 2016; 36: 308-320
        View in Article
        • Jalal H.
        • Alarid-Escudero F.
        A Gaussian approximation approach for value of information analysis.
        Med Decis Making. 2018; 38: 174-188
        View in Article
        • Jalal H.
        • Goldhaber-Fiebert J.D.
        • Kuntz K.M.
        Computing expected value of partial sample information from probabilistic sensitivity analysis using linear regression metamodeling.
        Med Decis Making. 2015; 35: 584-595
        View in Article
        • Heath A.
        • Manolopoulou I.
        • Baio G.
        Efficient Monte Carlo estimation of the expected value of sample information using moment matching.
        Med Decis Making. 2018; 38: 163-173
        View in Article
        • Heath A.
        • Baio G.
        Calculating the expected value of sample information using efficient nested Monte Carlo: a tutorial.
        Value Health. 2018; 21: 1299-1304
        View in Article
        • Heath A.
        • Manolopoulou I.
        • Baio G.
        Bayesian curve fitting to estimate the expected value of sample information using moment matching across different sample sizes.
        Med Decis Making. 2018; ([in press])
        View in Article
        • Heath A.
        • Baio G.
        EVSI: a suite of functions for the calculation and presentation of the EVSI.
        https://github.com/annaheath/EVSI
        Date accessed: October 4, 2019
        View in Article
        • ConVOI
        The collaborative network for value of information.
        https://www.convoi-group.org/
        Date accessed: June 9, 2019
        View in Article
        • Raiffa H.
        • Schlaifer H.
        Applied Statistical Decision Theory.
        Harvard University Press, Boston, MA1961
        View in Article
        • Stinnett A.A.
        • Mullahy J.
        Net health benefits: a new framework for the analysis of uncertainty in cost-effectiveness analysis.
        Med Decis Making. 1998; 18: S68-S80
        View in Article
        • Briggs A.
        • Sculpher M.
        • Claxton K.
        Decision Modelling for Health Economic Evaluation.
        Oxford University Press, Oxford2006
        View in Article
        • Arrow K.J.
        • Lind R.C.
        Uncertainty and the evaluation of public investment decisions.
        Am Econ Rev. 1970; 60
        View in Article
        • Baio G.
        • Dawid A.P.
        Probabilistic sensitivity analysis in health economics.
        Stat Methods Med Res. 2015; 24: 615-634
        View in Article
        • Briggs A.H.
        Handling uncertainty in cost-effectiveness models.
        Pharmacoeconomics. 2000; 17: 479-500
        View in Article
        • Coyle D.
        • Oakley J.
        Estimating the expected value of partial perfect information: a review of methods.
        Eur J Health Econ. 2008; 9: 251-259
        View in Article
        • Heath A.
        • Manolopoulou I.
        • Baio G.
        A review of methods for analysis of the expected value of information.
        Med Decis Making. 2017; 37: 747-758
        View in Article
        • Brennan A.
        • Kharroubi S.A.
        Efficient computation of partial expected value of sample information using Bayesian approximation.
        J Health Econ. 2007; 26: 122-148
        View in Article
        • Heath A.
        • Manolopoulou I.
        • Baio G.
        Estimating the expected value of partial perfect information in health economic evaluations using integrated nested Laplace approximation.
        Stat Med. 2016; 35: 4264-4280
        View in Article
        • Baio G.
        • Berardi A.
        • Heath A.
        Bayesian Cost Effectiveness Analysis with the R Package BCEA.
        Springer, 2017
        View in Article
        • Strong M.
        • Breeze P.
        • Thomas C.
        • Brennan A.
        SAVI - Sheffield accelerated value of information, release version 1.013 (2014-12-11).
        http://savi.shef.ac.uk/SAVI/
        Date accessed: June 9, 2019
        View in Article
        • Strong M.
        Partial EVPPI functions.
        http://www.shef.ac.uk/polopoly_fs/1.305039!/file/R_functions.txt
        Date accessed: June 9, 2019
        View in Article
        • Wood S.N.
        Thin plate regression splines.
        J Royal Stat Soc. 2003; 65: 95-114
        View in Article
        • Wood S.
        • Pya N.
        • Kneib T.
        • et al.
        mgcv: mixed GAM computation vehicle with GCV/AIC/REML smoothness estimation and GAMMs by REML/PQL.
        University of Bath, 2016
        View in Article
        • Royston P.
        • Ambler G.
        Gam: stata module for generalised additive models.
        Boston College Department of Economics, 2012
        View in Article
        • Cai W.
        Fitting generalized additive models with the gam procedure in SAS 9.2. SAS Global Forum.
        https://support.sas.com/resources/papers/proceedings/pdfs/sgf2008/378-2008.pdf
        (Accessed September 25, 2019)
        View in Article
        • Rasmussen C.
        • Williams C.
        Gaussian processes for machine learning.
        http://www.gaussianprocess.org/gpml/chapters/RW.pdf
        (Accessed September 25, 2019)
        View in Article
        • Hastie T.
        • Tibshirani R.
        Generalized Additive Models.
        CRC Press, Boca Raton, FL1990
        View in Article
        • Food and Drug Administration
        Guidance for the use of Bayesian statistics in medical device clinical trials.
        https://www.fda.gov/regulatory-information/search-fda-guidance-documents/guidance-use-bayesian-statistics-medical-device-clinical-trials
        (Accessed September 25, 2019)
        View in Article
        • Lunn D.J.
        • Thomas A.
        • Best N.
        • Spiegelhalter D.
        WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility.
        Statistics and Computing. 2000; 10: 325-337
        View in Article
        • Plummer M.
        rjags: Bayesian graphical models using MCMC: R package version 4-4.
        https://cran.r-project.org/web/packages/rjags/rjags.pdf
        (Accessed September 25, 2019)
        View in Article
        • Gelman A.
        • Lee D.
        • Guo J.
        • Stan
        A probabilistic programming language for Bayesian inference and optimization.
        J Educ Behav Stat. 2015; 40: 530-543
        View in Article
        • Morita S.
        • Thall P.F.
        • Muller P.
        Determining the effective sample size of a parametric prior.
        Biometrics. 2008; 64: 595-602
        View in Article
        • Heath A.
        • Manolopoulou I.
        • Baio G.
        Estimating the expected value of sample information across different sample sizes using moment matching and nonlinear regression.
        Med Decis Making. 2019; 39: 347-359
        View in Article
        • Heath A.
        • Baio G.
        Calculating value of information measures for health economic evaluations within R.
        https://github.com/annaheath/EVSI/blob/master/VoI%20using%20R%20Manual.pdf
        (Accessed September 25, 2019)
        View in Article
        • Kunst N.R.
        • Alarid-Escudero F.
        • Paltiel A.D.
        • Wang S.
        A value of information analysis of research on the 21-gene assay for breast cancer management.
        Value Health. 2019; 22: 1102-1110
        View in Article
        • Jutkowitz E.
        • Alarid-Escudero F.
        • Kuntz K.M.
        • Jalal H.
        The curve of optimal sample size (COSS): a graphical representation of the optimal sample size from a value of information analysis.
        Pharmacoeconomics. 2019; 37: 871-877
        View in Article
        • Hironaka T.
        • Giles M.B.
        • Goda T.
        • Thom H.
        Multilevel Monte Carlo estimation of the expected value of sample information.
        2019 (arXiv: 190512013 [Preprint])
        View in Article
        • Meltzer D.O.
        • Hoomans T.
        • Chung J.W.
        • Basu A.
        Minimal modeling approaches to value of information analysis for health research.
        Med Decis Making. 2011; 31: E1-E22
        View in Article
        • Rothery C.
        • Strong M.
        • Koffijberg H.
        • et al.
        Value of information analytical methods: report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force.
        Value Health. 2020; 23: 277-286
        View in Article

      Article info

      Publication history

      Published online: May 27, 2020

      Identification

      DOI: https://doi.org/10.1016/j.jval.2020.02.010

      Copyright

      © 2020 ISPOR-The Professional Society for Health Economics and Outcomes Research. Published by Elsevier Inc.

      User license

      Elsevier user license |
      How you can reuse Information Icon
      Elsevier user license

      Permitted

      For non-commercial purposes:

      • Read, print & download
      • Text & data mine
      • Translate the article

      Not Permitted

      • Reuse portions or extracts from the article in other works
      • Redistribute or republish the final article
      • Sell or re-use for commercial purposes

      Elsevier's open access license policy

      ScienceDirect

      Access this article on ScienceDirect
      We use cookies to help provide and enhance our service and tailor content. To update your cookie settings, please visit the for this site.
      Copyright © 2023 Elsevier Inc. except certain content provided by third parties. The content on this site is intended for healthcare professionals.


      RELX