## Highlights

- •The expected value of sample information (EVSI) can be used to prioritize research and design future studies to reduce decision uncertainty for policy makers. Four recently published methods have overcome the computational issues associated with EVSI analysis, but practical guidance on using and distinguishing these methods is lacking.
- •Members of the Collaborative Network on Value of Information (ConVOI) reviewed 4 EVSI computation methods to understand their required skills and inputs. They also provided step-by-step guides and recommendations about which method to use based on features of the decision problem and proposed future study.
- •By comparing these methods, ConVOI provides practical guidance for analysts looking to compute EVSI and use it for study design. Analysts now have useful information to confidently select the most appropriate EVSI estimation method for their application and expertise.

## Abstract

Value of information (VOI) analyses can help policy makers make informed decisions about whether to conduct and how to design future studies. Historically a computationally expensive method to compute the expected value of sample information (EVSI) restricted the use of VOI to simple decision models and study designs. Recently, 4 EVSI approximation methods have made such analyses more feasible and accessible. Members of the Collaborative Network for Value of Information (ConVOI) compared the inputs, the analyst’s expertise and skills, and the software required for the 4 recently developed EVSI approximation methods. Our report provides practical guidance and recommendations to help inform the choice between the 4 efficient EVSI estimation methods. More specifically, this report provides: (1) a step-by-step guide to the methods’ use, (2) the expertise and skills required to implement the methods, and (3) method recommendations based on the features of decision-analytic problems.

## Introduction

Decisions on which research studies to fund are intrinsically economic in nature as public, private, and third sector (research charity) funders have finite resources. Thus their decisions bear an opportunity cost, implying the prioritization of research studies that are expected to yield the greatest benefit.

A value of information analysis (VOI) can determine the value of research studies based on how much they reduce decision uncertainty about the optimal intervention for use in a population of interest. Reducing decision uncertainty has value because a suboptimal decision reduces the potential health improvements in the population. The probability of making the wrong decision multiplied by the size of the loss incurred by that decision yields the expected loss associated with uncertainty. Additional data are expected to reduce decision uncertainty and so reduce the expected loss. The expected value of sample information (EVSI) measures the “expected reduction in expected loss” from a given research study. Scaled up to the relevant population, this can be expressed in health terms of life-years, or quality-adjusted life-years (QALYs), or in monetary units.

^{1}

^{2}

^{,}The difference between the EVSI and the cost of the research is the expected net benefit of sampling (ENBS).An increasing number of authorities and Health Technology Assessment (HTA) agencies either acknowledge because they required nested simulation methods. Thus most VOI analyses were restricted to computing the expected value of perfect information (EVPI) or the expected value of partial perfect information (EVPPI), for which several efficient methods are available. New algorithms and associated software, however, enable the efficient computation of EVSI for different study designs using complex decision-analytic models.

^{4}

^{, }^{, }or recommend VOI analysis. Thus, EVSI calculations will soon be required from analysts. Until recently, EVSI calculations were extremely computationally expensive, potentially taking weeks or months,^{8}

- Heath A.
- Kunst N.
- Jackson C.
- et al.

Calculating the expected value of sample information in practice: Considerations from three case studies.

*Medical Decision Making.*2020; (In press)https://doi.org/10.1177/0272989X20912402

^{9}

^{10}

^{, }^{11}

^{, }^{12}

^{, }^{13}

^{, }^{14}

^{, }^{15}

^{, }^{16}

^{, }^{17}

^{18}

^{, }^{19}

^{, }^{20}

^{, }^{21}

^{, }^{22}

^{, }^{23}

^{, }^{24}

^{, }^{25}

^{, }^{26}

- Heath A.
- Baio G.

EVSI: a suite of functions for the calculation and presentation of the EVSI.

https://github.com/annaheath/EVSI

Date accessed: October 4, 2019

Recently, the accuracy and computational time of these methods were compared using 3 different case studies, and researchers concluded that none of the methods were systematically superior in terms of accuracy, precision, or computational time. Still, these methods differ in their approach, requiring different expertise and skills. Nevertheless, because there is no structured comparison of the practical steps required to use them, it is challenging for analysts to determine which method is appropriate for their situation and expertise. Additionally, each method has different properties that could make it more suitable for a given decision problem. This report provides practical guidance and recommendations for computing EVSI using 4 recently developed approximation methods. For each method, we provide: (1) a step-by-step guide to its use, (2) the expertise and skills required to implement it, and (3) its recommended usage based on the features of decision-analytic problems.

^{8}

- Heath A.
- Kunst N.
- Jackson C.
- et al.

Calculating the expected value of sample information in practice: Considerations from three case studies.

*Medical Decision Making.*2020; (In press)https://doi.org/10.1177/0272989X20912402

## Four EVSI Approximation Methods

The Collaborative Network for Value of Information (ConVOI) is an international group of researchers that focuses on the application and development of VOI calculation methods. In November 2018, members of this network prepared a questionnaire (N.K., A.H., D.G.) completed by the developers of 4 recently developed EVSI computation methods (M.S., N.M., H.J., F.A.E., A.H.). The questionnaire results were reviewed, summarized (N.K., A.H., D.G., E.W.), and discussed (by all authors) to find: (1) the inputs, (2) the skills and expertise, and (3) the software required to use the reviewed methods. The questionnaire also identified the situations in which each method was appropriate. Any disagreements were resolved by consensus (for the questionnaire, see Appendix 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.02.010).

^{27}

ConVOI

The collaborative network for value of information.

The collaborative network for value of information.

https://www.convoi-group.org/

Date accessed: June 9, 2019

The reviewed methods included a regression-based (RB) method, an importance sampling (IS) method, a Gaussian approximation (GA) method, and a moment matching (MM) method. These methods were selected because they place limited restrictions on the complexity of the underlying decision-analytic model or data collection exercise when calculating EVSI.

^{19}

^{20}

^{21}

^{,}^{22}

## Inputs Required for Efficient EVSI Calculation

The considered EVSI approximation methods have diverse requirements and make different assumptions. Nevertheless, all require a decision-analytic model on which a probabilistic analysis is conducted. The IS and MM methods also require calculation of the EVPPI for the parameter(s) to be evaluated and updated in the proposed study. If studies informing different parameters or groups of parameters are considered, then EVPPI would need to be computed for each. The following section briefly outlines these requirements and assumptions (Table 1).

Table 1The inputs required to compute EVSI with the regression-based, importance sampling, Gaussian approximation, and moment matching methods.

# | Requirements | RB | IS | GA | MM |
---|---|---|---|---|---|

Inputs | |||||

1 | Decision-analytic model | ✓ | |||

2 | Probabilistic analysis results | ✓ | ✓ | ✓ | ✓ |

3 | Simulations of the expected net benefit conditional on $\phi $ (required to compute EVPPI) | ✓ | ✓ |

*Note.*✓ indicates that the skill/input is required.

GA indicates Gaussian approximation method; IS, importance sampling method; MM, moment matching method; RB, regression-based method.

∗ EVSI estimation involves rerunning the decision-analytic model and thus access to that model is required.

### Decision-Analytic Model

VOI requires an objective function that should be optimized to determine the best course of action. Economic evaluations in healthcare typically define this in terms of net health or monetary benefit for each of which uses a “willingness-to-pay” threshold to put health consequences on the same scale as costs for the different interventions.

^{28}

*T*interventions,^{29}

Decision-analytic models are mathematical models often used for cost-effectiveness analyses to compute this net benefit (NB). These models require a set of inputs, denoted by Conceptually, the decision-analytic model maps a set of inputs to the output net benefit (either in monetary or health units). Assuming risk neutrality, the intervention with the highest expected NB should be implemented in the wider population.

*θ*. These inputs include disease prevalence, treatment effectiveness, background mortality, health-related utility weights, and costs. The model can take many forms, typically a decision tree, Markov model, or microsimulation model.^{30}

### Probabilistic Analysis

Probabilistic analysis (PA), sometimes known as probabilistic sensitivity analysis, propagates parameter uncertainty to the model output under each decision alternative and thereby quantifies decision uncertainty. PA simultaneously varies all parameters for which there is uncertainty. Uncertainty in the model inputs is characterized using probability distributions,

^{32}

*p*(*θ*).PA is often conducted using Monte Carlo methods where

*S*parameter sets are drawn from*p*(*θ*) for the whole set of model parameters. The decision-analytic model is evaluated at all*θ**s*, $s=1,\dots ,S$ to estimate the costs and health outcomes of each of the*T*strategies. This produces a distribution for the NB for each strategy, which we denote by${\text{NB}}_{t}^{\theta}$, for $t=1,\dots ,T$. In non-linear models, PA is required to estimate the expected NB of each strategy, ${\text{E}}_{\theta}\left[{\text{NB}}_{t}^{\theta}\right]$.To compute EVSI, we require the PA simulations of the model inputs ${\theta}_{s}$ and the corresponding NB simulations ${\text{NB}}_{t}^{{\theta}_{s}}$ for $s=1,\dots ,S$ and $t=1,\dots ,T$. These simulations should be saved in a matrix or spreadsheet form, often called a PA data set (the expression “data set” is not used in the traditional sense; here, the PA data set simply contains simulated values from distributions representing the uncertainty in the parameter estimates), where the columns contain the input parameters and the outputs from a decision-analytic model. Each row contains the parameter sets drawn from their distributions and their corresponding simulated results from a decision-analytic model output. An example of PA simulations is provided in the Supplemental Materials.

Although PA results are necessary for all of the 4 efficient EVSI methods, 2 of the methods (RB and GA) require the PA data set from a traditional cost-effectiveness analysis whereas the other 2 (IS and MM) require the augmented PA data set presented in the following section.

### Expected Value of Partial Perfect Information

The EVPPI computes the value of eliminating Specifically, the model parameters are split into two subsets $\theta =(\phi ,\psi )$, where we propose to gather further information about the model parameters $\phi $. Typically, a proposed study does not collect information about all the underlying parameters in a decision-analytic model, and therefore $\psi $ indicates the parameters that will not be directly informed by the proposed data collection.

*all*uncertainty about a subset of the model parameters.^{10}

^{,}^{34}

^{,}^{35}

Mathematically, the EVPPI for the parameters $\phi $ is defined as:

where the inner expectation in the first term of equation (1) calculates the NB for each intervention conditional on $\phi $. The second term calculates the value of the decision made with current information (ie, the expected NB of the treatment with the highest expected NB).

$\text{EVPPI}={\text{E}}_{\phi}\left[\underset{t}{\text{max}}\phantom{\rule{0.25em}{0ex}}{\text{E}}_{\psi |\phi}\left[{\text{NB}}_{t}^{\mathit{\theta}}\right]\right]-\underset{t}{\text{max}}\phantom{\rule{0.25em}{0ex}}{\text{E}}_{\theta}\left[{\text{NB}}_{t}^{\mathit{\theta}}\right],$

(1)

where the inner expectation in the first term of equation (1) calculates the NB for each intervention conditional on $\phi $. The second term calculates the value of the decision made with current information (ie, the expected NB of the treatment with the highest expected NB).

The PA outcomes for $\phi $ are simulated and the EVPPI is estimated by computing the NB for each intervention conditional $\phi ={\phi}_{s}$ for $s=1,\dots ,S$. Approximation methods are available to estimate the conditional NB for each value ${\phi}_{s}$. These have been assessed and can be obtained using available software. The MM and IS methods require a PA data set with $T$ additional columns that contain Monte Carlo estimates of the expected NB conditional on ${\phi}_{s}$ for $s=1,\dots ,S$, but averaged over the uncertain values of $\psi $. We denote these simulations by ${\eta}_{t}^{s}$.

^{10}

^{,}^{14}

^{, }^{15}

^{, }^{16}

^{,}^{36}

^{,}^{37}

^{34}

^{,}^{35}

^{38}

^{, }^{39}

- Strong M.
- Breeze P.
- Thomas C.
- Brennan A.

SAVI - Sheffield accelerated value of information, release version 1.013 (2014-12-11).

http://savi.shef.ac.uk/SAVI/

Date accessed: June 9, 2019

^{, }^{40}

- Strong M.

Partial EVPPI functions.

http://www.shef.ac.uk/polopoly_fs/1.305039!/file/R_functions.txt

Date accessed: June 9, 2019

## EVSI Methods

EVSI for a proposed research strategy that collects additional data, denoted by $X$, is defined as:

$\text{EVSI}={E}_{X}\left[\underset{t}{\text{max}}\phantom{\rule{0.25em}{0ex}}{\text{E}}_{\theta |X}\left[{\text{NB}}_{t}^{\theta}\right]\right]-\underset{t}{\text{max}}\phantom{\rule{0.25em}{0ex}}{\text{E}}_{\theta}\left[{\text{NB}}_{t}^{\theta}\right].$

(2)

In this setup, $X$ are observ

*able*but not yet observ*ed*. Computing EVSI helps us determine whether we should defer making a decision on the optimal intervention, among the $T$ possible options, and collect $X$ instead.In line with the Bayesian approach, which underpins VOI analyses, the distribution of $X$ is defined by $p(X,\theta )=p\left(\theta \right)p\left(X\right|\theta )$, where $p\left(\theta \right)$ is the PA distribution of $\theta $ and $p\left(X\right|\theta )$ is the sampling distribution of the data. In this setting, $p\left(\theta \right)$ is a “prior” distribution that represents the current uncertainty level for the model parameters

*before*observing the additional data $X$. This distribution can be based on the observed data that are used to construct the decision-analytic model. We note that $X$ will give information about the subset of model parameters $\phi $, where $\phi $ could be the whole set $\theta $. By definition, this implies that $\psi $ and $X$ are independent given $\phi $ and that $p(X\left|\theta \right)=p\left(X\right|\phi )$.The second term of equation (2) can be estimated from the initial PA. To obtain simulations for $X$, we simulate potential study outcomes ${X}_{s},\phantom{\rule{0.25em}{0ex}}s=1,\dots ,\text{}S$, for each row of the PA data set (ie, we generate a single sample from $p\left(X\right|{\phi}_{s})$ for $s=1,\dots ,S$). Traditionally, EVSI has been computationally demanding and methodologically challenging because a large number of simulations are needed to estimate the posterior mean of the NB for each ${X}_{s}$. This requires a Bayesian model for the distribution of the data and the parameters, where the posterior mean is given by:

${\mu}_{t}\left({X}_{s}\right)={\text{E}}_{\theta |{X}^{s}}\left[{\text{NB}}_{t}^{\theta}\right].$

(3)

The EVSI approximation methods in this report estimate ${\mu}_{t}\left(X\right)$ with a reduced computational burden. For computational simplicity and stability, VOI calculations should be undertaken using

for $t=2,\dots ,T$. We could also use the opportunity loss, defined as the INB of all treatments from the intervention that is currently associated with the maximum expected NB (ie, the optimal strategy). By estimating the posterior mean of the INB, we only need to add $T-1$ columns containing the posterior mean conditional on ${X}_{s}$ to our PA data set. Once these simulations, denoted by ${\mu}_{t}\left({X}_{s}\right)$, are available, the EVSI can be estimated by:

where ${\mu}_{1}\left({X}_{s}\right)=0$ for all $s=1,\dots ,S$.

^{16}

*incremental*NB (INB) defined, without loss of generality, as:${\text{INB}}_{t}^{\theta}=\phantom{\rule{0.25em}{0ex}}{\text{NB}}_{t}^{\theta}-\phantom{\rule{0.25em}{0ex}}{\text{NB}}_{1}^{\theta},$

for $t=2,\dots ,T$. We could also use the opportunity loss, defined as the INB of all treatments from the intervention that is currently associated with the maximum expected NB (ie, the optimal strategy). By estimating the posterior mean of the INB, we only need to add $T-1$ columns containing the posterior mean conditional on ${X}_{s}$ to our PA data set. Once these simulations, denoted by ${\mu}_{t}\left({X}_{s}\right)$, are available, the EVSI can be estimated by:

$\frac{1}{S}\sum _{s=1}^{S}\underset{t}{\text{max}}\left\{{\mu}_{t}\left({X}_{s}\right)\right\}-\underset{t}{\text{max}}\left\{\frac{1}{S}\sum _{s=1}^{S}{\mu}_{t}\left({X}_{s}\right)\right\},$

(4)

where ${\mu}_{1}\left({X}_{s}\right)=0$ for all $s=1,\dots ,S$.

## EVSI Calculation

We present step-by-step guides for the implementation of the 4 EVSI methods in Box 1, Box 2, Box 3, Box 4 through 4. Box 1 contains the RB algorithm to estimate EVSI where the fitted values from a regression model between the simulated INB and a low-dimensional summary statistic from the simulated data set ${X}_{s}$ are used to calculate ${\mu}_{t}\left(X\right)$. Box 2 contains the IS algorithm where the simulations to compute EVPPI are reweighted using the statistical likelihood to calculate ${\mu}_{t}\left(X\right)$. Box 3 presents the GA alogrithm where the prior distribution for $\phi $ are rescaled and combined using a linear meta-model to estimate ${\mu}_{t}\left(X\right)$. Finally, Box 4 presents the MM algorithm where the simulations to compute EVPPI are rescaled by a factor estimated using efficient nested MC sampling to calculate ${\mu}_{t}\left(X\right)$.

^{19}

^{20}

^{21}

^{23}

Box 1

**Step-by-step guide for regression-based method.**

- 1. Perform PA simulation to obtain
*θ*_{s}and ${\text{INB}}_{t}^{{\mathit{\theta}}_{s}}$,*s*= 1,…,*S.* - 2. For each
*s*= 1,…,*S*:- (a) Simulate a dataset
*X*_{s}from*p*(|*X**ϕ*_{s}). - (b) Summarise this dataset, producing the quantity or quantities that would be estimated from it in a trial, such as the mean. We denote this summary of the dataset
*W*(*X*_{s}).

- 3. Fit
*T*−1 regression models with ${\text{INB}}_{t}^{{\mathit{\theta}}_{s}}$ as the outcome and*W*(*X*_{s}) as the covariates. - 4. Extract the fitted values from these regressions to estimate ${\mu}_{t}\left({\mathit{X}}_{s}\right)$.

The algorithm used to estimate EVSI using the RB method, based on fitting a regression model and determining a low-dimensional summary of the simulated dataset

^{19}

*X*.EVSI indicates expected value of sample information; INB, incremental net benefit; PA, probabilistic analysis; RB, regression-based;

*θ*, set of inputs in a decision-analytic model;*p(θ)*, probability distributions to characterize uncertainty in the model inputs;*S*, number of parameter sets that are drawn from p(θ) in PA;*T*, number of considered interventions; $\theta =(\phi ,\psi )$, $\phi $ are model parameters for which we are aiming to collect further information and $\psi $ are parameters that will not be directly informed by the proposed data collection;*X*, additional data proposed to be collected;*μ*, posterior mean;*W(X)*, a summary measure for the data.Box 2

**Step-by-step guide for importance sampling method.**

- 1. Perform PA simulation to obtain ${\mathit{\theta}}_{s}$ and ${\text{INB}}_{t}^{{\mathit{\theta}}_{s}}$,
*s*= 1,…,*S.* - 2. Estimate EVPPI to obtain simulations of ${\eta}_{t}^{s}$, the inner expectation in the first term of equation:

$\text{EVPPI}={\text{E}}_{\varphi}\left[\underset{t}{\text{max}}{\text{E}}_{\mathit{\psi}|\varphi}\left[{\text{NB}}_{t}^{\mathit{\theta}}\right]\right]-\phantom{\rule{0.25em}{0ex}}\underset{t}{\text{max}}{\text{E}}_{\theta}\left[{\text{NB}}_{t}^{\mathit{\theta}}\right].$

- 3. For each
*s*= 1,…,*S*:- (a) Simulate a dataset
*X*_{s}from $p\left(\mathit{X}|{\varphi}_{s}\right)$. - (b) Compute the likelihood of
*X*_{s}conditional on every PA simulation for $\mathit{\theta}$, i.e., calculate the likelihood*L*_{r}of*X*_{s}for each parameter set ${\mathit{\theta}}_{r}$ with*r*= 1,…,*S.* - (c) Compute ${l}_{r}=\frac{{L}_{r}}{{\sum}_{r=1}^{S}{L}_{r}}$ so
*l*_{r}sums to 1. - (d) Calculate the weighted sum of ${\eta}_{t}^{r}$, ${\sum}_{r=1}^{S}{l}_{r}{\eta}_{t}^{r}.$

- 4. Each weighted sum estimates ${\mu}_{t}\left({\mathit{X}}_{s}\right)$.

The algorithm used to estimate EVSI using the IS method, based on calculating weighted sums of ${\eta}_{t}^{s}$ using weights calculated from the likelihood of the data given the parameters. This algorithm is listed as Method 2 in the original IS development but represents the most accurate method for EVSI estimation.

^{20}

EVPPI indicates expected value of partial perfect information; EVSI, expected value of sample information; INB, incremental net benefit; IS, importance sampling; PA, probabilistic analysis;

*θ*, set of inputs in a decision-analytic model;*p(θ)*, probability distributions to characterize uncertainty in the model inputs;*S*, number of parameter sets that are drawn from p(θ) in PA;*T*, number of considered interventions; $\theta =\left(\phi ,\psi \right),$ $\phi $ are model parameters for which we are aiming to collect further information and $\psi $ are parameters that will not be directly informed by the proposed data collection;*X*, additional data proposed to be collected;*μ*, posterior mean;*η*, PA simulations used to compute EVPPI.Box 3

**Step-by-step guide for Gaussian approximation method.**

- 1. Perform PA simulation to obtain ${\mathit{\theta}}_{s}$ and ${\text{INB}}_{t}^{{\mathit{\theta}}_{s}}$,
*s*= 1,…,*S.* - 2. Fit
*T*− 1 regression models with ${\text{INB}}_{t}^{{\mathit{\theta}}_{s}}$ as outcomes and ${\varphi}_{s}$ as covariates. - 3. For each element of $\varphi $ denoted by ${\varphi}^{p}$ for
*p*= 1,…,*P*:- (a) Determine the prior effective sample size ${n}_{0}^{p}$. Methods to estimate the prior effective sample size are presented in [21].
- (b) For a proposed study with
*N*participants, compute a weighted sum of the simulations of ${\varphi}^{p}$and $\overline{{\varphi}^{p}}$, the mean of the*p*^{th}parameter, by multiplying ${\varphi}^{p}$ by $\sqrt{\frac{N}{N+{n}_{0}^{p}}}$ and multiplying $\overline{{\varphi}^{p}}$ by $\left(1-\sqrt{\frac{N}{N+{n}_{0}^{p}}}\right)$.

- 4. Using the regression models from Step 2, predict the model outcomes for the rescaled $\varphi $ simulations.
- 5. The fitted values from Step 4 estimate ${\mu}_{t}\left({\mathit{X}}_{s}\right)$.

The algorithm used to estimate EVSI using the GA method, based on rescaling the PA distributions for $\phi $ and estimating the INB from these rescaled distribution using the meta-model. The original GA algorithm worked with opportunity loss rather than INB; in this article we have used INB for consistency across the reviewed methods.

^{21}

EVSI indicates expected value of sample information; GA, Gaussian approximation; INB, incremental net benefit; PA, probabilistic analysis;

*θ*, set of inputs in a decision-analytic model;*S*, number of parameter sets that are drawn from p(θ) in PA;*T*, number of considered interventions; $\theta =\left(\phi ,\psi \right),$ $\phi $ are model parameters for which we are aiming to collect further information and $\psi $ are parameters that will not be directly informed by the proposed data collection;*X*, additional data proposed to be collected;*μ*, posterior mean.Box 4

**Step-by-step guide for moment matching method.**

- 1. Perform PA simulation to obtain ${\mathit{\theta}}_{s}$ and ${\text{INB}}_{t}^{{\mathit{\theta}}_{s}}$,
*s*= 1,…,*S.* - 2. Estimate EVPPI to obtain simulations of ${\eta}_{t}^{s}$, the inner expectation in the first term of equation:

$\text{EVPPI}={\text{E}}_{\varphi}\left[\underset{t}{\text{max}}{\text{E}}_{\mathit{\psi}|\varphi}\left[{\text{NB}}_{t}^{\mathit{\theta}}\right]\right]-\phantom{\rule{0.25em}{0ex}}\underset{t}{\text{max}}{\text{E}}_{\theta}\left[{\text{NB}}_{t}^{\mathit{\theta}}\right].$

- 3. Extract
*Q*, with 30 <*Q*< 50, sample quantiles from the simulations of $\varphi $, denoted by ${\varphi}_{q}$. To estimate EVSI across sample size, you also require*Q*proposed sample sizes denoted by*N*_{q}for*q*= 1,…,*Q.* - 4. For
*q*= 1,…,*Q*:- (a) Simulate a future dataset, with sample size
*N*_{q}if required, from the sampling distribution $p\left(\mathit{X}|{\varphi}_{q}\right)$. - (b) Use Bayesian methods to update the distribution of the model parameters.
- (c) Rerun the probabilistic sensitivity analysis to update the distribution of the net monetary benefit.
- (d) Calculate the
*variance*of the net monetary benefit, denoted by ${\sigma}_{q}^{2}$.

- 5. Calculate the rescaling factor ${\sigma}^{2}$
- (a) If estimating EVSI for a single sample size, ${\sigma}^{2}=\mathrm{V}\mathrm{a}\mathrm{r}\left[\mathrm{N}{\mathrm{B}}_{t}^{{\theta}_{s}}\right]-\frac{1}{Q}{\sum}_{q=1}^{Q}{\sigma}_{q}^{2}$.
- (b) If estimating EVSI across sample size, ${\sigma}^{2}$ for a given
*N*is estimated by estimating*h*in the non-linear regression function $\frac{N}{N+h}$ with $\text{Var}[{\text{NB}}_{t}^{{\mathit{\theta}}_{s}}]-{\sigma}_{q}^{2}$ as the dependant variable and*N*_{q}as the independent variable.

- 6. Rescale the simulations of ${\eta}_{t}^{s}$ so their variance is equal to ${\sigma}^{2}$ using:

$\sqrt{\frac{{\sigma}^{2}}{\text{Var}\left({\eta}_{t}^{s}\right)}}{\eta}_{t}^{s}+\left(1-\sqrt{\frac{{\sigma}^{2}}{\text{Var}\left({\eta}_{t}^{s}\right)}}\right)\overline{{\eta}_{t}},$

where $\overline{{\eta}_{t}}$ is the mean of ${\eta}_{t}$ simulations and $\text{Var}\left({\eta}_{t}^{s}\right)$ is the sample variance of ${\eta}_{t}$ simulations.

- 7. These rescaled simulations estimate ${\mu}_{t}\left({\mathit{X}}_{s}\right)$.

The algorithm used to estimate EVSI using the MM method, based on rescaling the simulations of ${\eta}_{t}^{s}$. In this method, the rescaling factor is estimated using Q nested Monte Carlo simulations, where Q is usually between 30 and 50 and should be as high as possible whilst allowing for a feasible computational burden. Functions are available in the EVSI package in R to perform steps 2-7.

^{23}

EVPPI indicates expected value of partial perfect information; EVSI, expected value of sample information; INB, incremental net benefit; MM, moment matching; PA, probabilistic analysis;

*θ*, set of inputs in a decision-analytic model;*p(θ)*, probability distributions to characterize uncertainty in the model inputs;*S*, number of parameter sets that are drawn from p(θ) in PA;*T*, number of considered interventions; $\theta =\left(\phi ,\psi \right),$ $\phi $ are model parameters for which we are aiming to collect further information and $\psi $ are parameters that will not be directly informed by the proposed data collection; $X$, additional data proposed to be collected;*μ*, posterior mean;*η*, PA simulations used to compute EVPPI;*Q*, number of times PA is performed.## Expertise and Skills Recommended for EVSI Calculation

We have identified 3 skills that are recommended for analysts before computing EVSI using the reviewed approximation methods (Table 2). No method requires all 3 skills but each method requires at least 1 skill.

Table 2The recommended skills and expertise to compute EVSI with the regression-based, importance sampling, Gaussian approximation, and moment matching methods.

# | Recommended skills and expertise | RB | IS | GA | MM |
---|---|---|---|---|---|

1 | Regression methods | ✓ | ✓ | ||

2 | Specification of likelihoods | ✓ | ✓ | ||

3 | Bayesian updating | ✓ |

*Note.*✓ indicates that the skill/input is required.

GA indicates Gaussian approximation method; IS, importance sampling method; MM, moment matching method; RB, regression-based method.

∗ The skill/input

*may be*required.### Regression Methods

Regression methods that model the relationship between a set of predictors and an outcome of interest are required for both the RB and GA methods. Both methods use regression metamodeling to model the INB as a function of model inputs or quantities related to model inputs. The 2 methods differ because they require alternative covariates.

Both of these EVSI methods model potentially complex relationships using models that require limited information about the functional relationship between the independent and dependent variables. To account for potential nonlinear relationships between these variables, flexible regression methods using generalized additive models (GAMs) are the predominant regression methods for EVSI calculation. Standard software is available to fit GAMs, but the EVSI estimate can be affected by the structure of the GAM model and, thus, an understanding of these models is recommended. Gaussian Processes have also been suggested for problems with greater than 6 covariates when using the RB method, so knowledge of these methods is also recommended.

^{42}

^{, }^{43}

^{, }^{44}

If regression is used to estimate EVSI, it is important to assess whether the regression model has correctly captured the relationship between the independent and dependent variables. Regression methods assume that the residual error is uncorrelated with the fitted values, has zero mean, and has a trivial covariance structure. These assumptions can be confirmed by observing no clusters, systematic features, or outliers in the plot of the residuals against the fitted values.

^{19}

### Specification of Data Generating Distribution

EVSI computes the expected value of collecting information in a future research study. The future research study would collect a potential data set, $X$, with ${N}_{\mathcal{O}}$ different clinical, health, or economic outcomes from $N$ participants. For example, a clinical trial would collect the primary and secondary outcomes from each participant.

All EVSI computation methods require the simulation of potential data sets (except the GA method in specific circumstances; see the section on Bayesian Updating). This is achieved by specifying the assumed data generating distribution $p\left(X\right|\phi )$. Thus it is recommended that analysts have knowledge of probability distributions.

For the RB method, the simulated datasets ${X}_{s}$ for $s=1,\dots ,S$ must be summarized to reflect how the data would be analyzed after the trial. As such, expertise in the statistical analysis of study results, such as maximum likelihood procedures, is recommended. The GA method can also be based on a summary statistic for specific data collection exercises (see Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.02.010 for more information).

For the IS method, the analytic likelihood for $X$ must be specified and coded as a function. Likelihood is used in its proper statistical meaning and, therefore, this specification requires statistical expertise. For this likelihood specification, it is recommended that analysts have knowledge of statistical theory, function development, and coding.

### Bayesian Updating

Bayesian methods are a statistical paradigm in which conclusions are formally updated as evidence accumulates. A full Bayesian model consists of probability distributions specified for the model inputs $p\left(\theta \right)$, the prior distribution defined using the PA distributions, and the sampling distribution for the proposed data collection. Based on these distributions, Bayesian methods update the distribution of the parameters conditional on the data $X$ to produce a posterior distribution. In practice, this is undertaken using specialized software such as BUGS, JAGS, or Stan.

^{47}

^{48}

^{, }^{, }The MM method is explicitly based on Bayesian analysis and thus we recommend that analysts have expertise in one of these programs and in the specification of Bayesian models.Additionally, the GA method requires an estimate of the prior effective sample size. The prior effective sample size, denoted by ${n}_{0}$, is the number of participants required in a trial identical to the currently proposed study that would need to be studied to obtain the level of information in the prior. If the prior and sampling distribution for $X$ are conjugate to each other, then the prior effective sample size can be estimated directly (see Supplemental Materials for more information). If this is not possible, 2 algorithms to compute the prior effective sample size are available. The first of these can be used if a single summary statistic is a good estimator of the model parameter and the second, based on a full Bayesian analysis, can be used in all settings. Thus the complexity of the prior effective sample size estimation is based on the complexity of the decision-analytic model and data collection exercise. See Supplemental Materials for more details on these methods for prior effective sample size estimation.

^{51}

^{51}

## EVSI Methods Recommendations Based on Features of the Decision-Analytic Problem

Because each method is different, the most suitable method for EVSI calculation will depend on the decision-analytic model, the sampling distribution for the data, the expertise of the analyst, and the amount of computation time available. The following section describes features of the decision problem and recommendations about which method should be used in these settings to help analysts select the most appropriate method (Table 3).

Table 3The recommendations for EVSI approximation methods based on features of the decision-analytic problem.

No. | Features | RB | IS | GA | MM |
---|---|---|---|---|---|

1 | Designing complex studies collecting a large number of outcomes (more than 6 outcomes) | ✓ | ✓ | ||

2 | Having only probabilistic analysis results without access to the decision-analytic model | ✓ | ✓ | ✓ | |

4 | Identifying optimal sample size by examining different study sizes with same computational cost | ✓ | ✓ | ||

5 | Including uncertainty quantification in the estimate | ✓ | ✓ | ✓ | |

6 | Proposing studies with a large sample size | ✓ | ✓ | ✓ | |

7 | Proposing studies with small prior effective sample size | ✓ | ✓ | ✓ | |

8 | Proposing studies with small sample size | ✓ | ✓ | ✓ |

*Note.*✓ indicates that the method is appropriate for the given feature.

GA indicates Gaussian approximation method; IS, importance sampling method; MM, moment matching method; RB, regression-based method.

∗ EVSI estimation does not require rerunning the decision-analytic model. This may be advantagous if the analyst does not have access to the decision-analytic model or if the decision-analytic model is computationally expensive.

† The IS method can be computationally challenging, and inaccurate, to estimate EVSI for proposed studies with a sample size over 1,000.

^{8}

- Heath A.
- Kunst N.
- Jackson C.
- et al.

Calculating the expected value of sample information in practice: Considerations from three case studies.

*Medical Decision Making.*2020; (In press)https://doi.org/10.1177/0272989X20912402

### Challenging to Run Decision-Analytic Model

In some settings, analysts may not have access to the decision-analytic model and only have access to the PA results. Additionally, if the decision-analytic model with a full PA is computationally expensive to run, the analyst may wish to avoid rerunning the model. In these settings, we recommend using the RB, GA, or IS methods because these methods only require access to the PA data set.

### Optimizing the Sample Size of the Future Study

EVSI can be used to help determine the optimal design for the future study, including optimizing the sample size. The GA and MM methods can be used to calculate EVSI across different sample sizes with a similar computational cost to a single EVSI estimate and are therefore recommended if the analyst is looking to optimize the overall sample size. Of note, the RB method typically has a low computational costCalculating the expected value of sample information in practice: Considerations from three case studies. so it can be used repeatedly to calculate EVSI across sample sizes.

^{8}

- Heath A.
- Kunst N.
- Jackson C.
- et al.

*Medical Decision Making.*2020; (In press)https://doi.org/10.1177/0272989X20912402

### Designing Studies with Multiple Outcomes

GAM regression, used for the RB and GA methods, can be challenging to implement with 6 or more covariates. Therefore, the IS and MM methods are recommended for settings where the future study collects more than 6 outcomes. Alternatives to GAM regression, such as the Gaussian Process regression, can be used to implement the RB method in these settings but are more computationally expensive and challenging to implement. Additionally, linear regression can be used to implement the GA method, but this may be insufficiently accurate for EVSI estimation. Note that although EVPPI estimation, required for MM and IS methods, is often based on regression methods, this estimation is only required once to estimate EVSI. Consequently, the computational burden of using more complex regression methods is reduced significantly for the IS and MM methods.

### Designing Studies With Extreme Sample Sizes

The sample size of the proposed study affects the considered EVSI approximation methods. First, the IS method is not recommended for studies where the sample size of the future data is over 1,000Calculating the expected value of sample information in practice: Considerations from three case studies. because of computational issues when the likelihood tends to 0 as the sample size increases. Second, the MM method is not recommended for studies collecting information on under 10 patients because the moment matching approximation is not sufficiently accurate. Finally, the GA method is not recommended when the prior effective sample size ${n}_{0}$ is under 10 because the Gaussian approximation is not accurate.

^{8}

- Heath A.
- Kunst N.
- Jackson C.
- et al.

*Medical Decision Making.*2020; (In press)https://doi.org/10.1177/0272989X20912402

^{23}

### Quantifying Uncertainty in EVSI Estimates

The RB, GA, and MM methods can be used to estimate the uncertainty associated with the EVSI estimation procedure. Specifically, for the RB and GA methods, the uncertainty can be obtained using an algorithm developed by Strong et al, provided the regression model gives good model fit. For the MM method, the uncertainty estimates are obtained in the process of computing EVSI across different sample sizes.

^{19}

### Limitations for the EVSI Calculation Methods

EVSI estimation will be poor for the RB and GA methods if the regression model does not capture the relationship of interest accurately, as assessed using model checking procedures. The IS method will be inaccurate if the likelihood is not specified correctly or cannot be estimated owing to numerical issues. Finally, the MM method is less accurate for studies that will have limited impact on the underlying uncertainty in the decision-analytic model (ie, the EVPPI of $\phi $ should be

*high*compared with the value of reducing all model uncertainty [such as EVPI] to use the MM method, ideally greater than 40%).## EVSI Methods in Practice

### Resources for the EVSI Methods

An online application Sheffield Accelerated Value of Information (SAVI) computes the EVPPI using the RB method. It fits a regression model between the ${\text{INB}}_{t}^{{\mathit{\theta}}_{\mathit{s}}}$ and the parameters of interest $\phi $. Because of the similarity between the EVPPI and EVSI calculation methods, SAVI can be used to compute EVSI once the future data sets have been summarized: $W\left({X}_{s}\right)$, $s=1,\dots ,S$. This is achieved by augmenting the PA data set with (a) column(s) containing the data summaries, which must be saved and uploaded into SAVI. EVSI is then equal to the “EVPPI” calculated for the column(s) containing the data summary.

^{39}

- Strong M.
- Breeze P.
- Thomas C.
- Brennan A.

SAVI - Sheffield accelerated value of information, release version 1.013 (2014-12-11).

http://savi.shef.ac.uk/SAVI/

Date accessed: June 9, 2019

An online repository is available for the GA method (https://zenodo.org/record/3263876), which contains a function that calculates the fitted values ${\mu}_{t}\left({X}_{s}\right)$ from a GAM regression model and estimates of ${n}_{0}^{p}$ for $p=1,\dots ,P$.

The EVSI package in R has been developed to implement the MM method based on a Bayesian decision-analytic model. The manual is available and the EVSI package can be installed in R using the command: devtools::install_github(“annaheath/EVSI”).

### Examples of the EVSI Methods Application

To aid the implementation of the reviewed EVSI methods, we have created a comprehensive GitHub repository that presents the code used to compute EVSI in the original publications, available at https://github.com/convoigroup/EVSIExample. This repository also contains a suite of practical examples that demonstrate EVSI calculations across several real-world applications using common decision-analytic model structures developed in R. Our GitHub repository demonstrates how the reviewed methods can be used in practice and will help analysts to implement VOI methods in their own work and increase the feasibility and accessibility of the EVSI methods as they become an important and required tool.

### EVSI Results Presentation

The EVSI R package contains several graphical displays to present EVSI and related quantities. EVSI results can be loaded into the EVSI package, irrespective of the computation method used. These graphics can be displayed directly in R or explored using a dynamic graphical display launched from within R. Analysts unfamiliar with R can explore these graphics using an online interface at https://egon.stats.ucl.ac.uk/projects/EVSI/Test/. The optimal sample size estimated using VOI methods can also be presented in form of a curve of optimal sample size (COSS). The COSS graphically displays optimal sample sizes for proposed study designs over a range of willingness-to-pay thresholds, including the impact of variation and uncertainty in VOI parameters on those optimal sample sizes.

^{53}

## Limitations

EVSI estimation is a rapidly developing subject, and future research may propose new efficient methods than those covered in this report. For example, promising results have been shown for multilevel Monte Carlo methods. Nonetheless, we believe the recommendations for analysts’ expertise and skills necessary for EVSI estimation and other included recommendations will remain relevant and may be applicable to new EVSI approximation methods. Next, the present recommendations only consider model-based EVSI methods. Although model-based VOI is most commonly used, minimal modeling approaches can be applied when decision makers have access to information from a clinical study on comprehensive measures of health outcomes and their incorporated uncertainty. Nevertheless, because these minimal modeling approaches are only feasible in certain circumstances, they cannot be compared directly with the reviewed EVSI methods. Furthermore, our EVSI definition is based on an assumption of risk-neutral decision makers who base their decision on maximizing expected values following the normative theory presented by Arrow and Lind. Lastly, these EVSI methods do not consider structural uncertainty, and this is one of the recommended future research directions as indicated by the ISPOR VOI Good Practice Task Force.

^{56}

^{57}

^{30}

^{,}^{31}

## Conclusion

VOI analysis has the potential to guide policy makers in the prioritization and design of future research studies, thereby improving decision making. Increasingly, HTA agencies are acknowledging the potential of VOI analyses and may start recommending EVSI to prioritize and design future research. In this study, members of ConVOI have provided practical guidance and recommendations to facilitate the implementation of 4 EVSI approximation methods. Our report outlines the inputs and software required to use each of these methods and also summarizes the recommended analysts’ skills and expertise needed to implement them based on the features of decision-analytic problems. The direction of future research in VOI is highlighted in Rothery et al, but future research should also focus on improving VOI implementation and its communication in clinical practice.

## Acknowledgments

We thank Dr Nicky Welton for her comments and discussions as part of the ConVOI group, and Yidi Jiang for her help preparing the EVSI application examples. Last but not least, we thank the Editor and 5 anonymous Reviewers for their valuable comments. N.K. was funded by the Research Council of Norway (276146 and 304034) and LINK Medical Research. EW was funded by Norwich Medical School. F.A.E. was funded by the National Cancer Institute (U01- CA-199335) as part of the Cancer Intervention and Surveillance Modeling Network (CISNET). G.B. was partially funded by a research grant sponsored by Mapi/ICON at University College London. A.B. was funded through Higher Education Funding Council for England and a range of UK and International research grant award bodies. M.F. was funded by the Stanford Interdisciplinary Graduate Fellowship. D.G. has no funding to declare. J.D.G.F. was funded in part by a grant from Stanford’s Precision Health and Integrated Diagnostics Center (PHIND). C.J. was funded the UK Medical Research Council program MCUU00002/11. M.S. has no funding to declare. H.J. was funded by NIH/NCATS grant 1KL2TR0001856. N.M. was supported by National Institutes of Health (NIH) [R01AI112438-02.]. H.T. was funded by the NIHR Biomedical Research Centre at the University Hospitals Bristol NHS Foundation Trust and the University of Bristol. A.H. was funded through an Innovative Clinical Trials Multi-year Grant from the Canadian Institutes of Health Research (funding reference number MYG-151207; 2017-2020). The funding agreement ensured the authors’ independence in designing the study, interpreting the data, writing, and publishing the report.

## Supplemental Material

- Appendix 1

## References

- “Time traveling is just too dangerous” but some methods are worth revisiting: the advantages of expected loss curves over cost-effectiveness acceptability curves and frontier.
*Value Health.*2019; 22: 611-618 - The NICE cost-effectiveness threshold: what it is and what that means.
*Pharmacoeconomics.*2008; 26: 733-744 - Guide to the methods of technology appraisal 2013.(Accessed March 25, 2019)
- Guidelines for the submission of documentation for single technology assessment (STA) of pharmaceuticals.
- Developing NICE guidelines: the manual.(Accessed March 25, 2019)
- Research recommendations.(Accessed March 25, 2019)
- Guidelines for the Economic Evaluation of Health Technologies: Canada. 4th Edition.(Accessed March 25, 2019)
- Calculating the expected value of sample information in practice: Considerations from three case studies.
*Medical Decision Making.*2020; (In press)https://doi.org/10.1177/0272989X20912402 - Expected value of sample information calculations in medical decision modeling.
*Med Decis Making.*2004; 24: 207-227 - Sensitivity analysis and the expected value of perfect information.
*Med Decis Making.*1998; 18: 95-109 - Calculating partial expected value of perfect information via Monte Carlo sampling algorithms.
*Med Decis Making.*2007; 27: 448-470 - Strategies for efficient computation of the expected value of partial perfect information.
*Med Decis Making.*2014; 34: 327-342 - Value of information for complex cost-effectiveness models.University of Sheffield, Sheffield, UK2002
- Need for speed: an efficient algorithm for calculation of single-parameter expected value of partial perfect information.
*Value Health.*2013; 16: 438-448 - An efficient method for computing single-parameter partial expected value of perfect information.
*Med Decis Making.*2013; 33: 755-766 - Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach.
*Med Decis Making.*2014; 34: 311-326 - Decision-making under uncertainty: using MLMC for efficient estimation of EVPPI.
*Statistics and Computing.*2019; 29: 739-751 - Expected value of sample information for multi-arm cluster randomized trials with binary outcomes.
*Med Decis Making.*2013; 34: 352-365 - Estimating the expected value of sample information using the probabilistic sensitivity analysis sample: a fast, nonparametric regression-based method.
*Med Decis Making.*2015; 35: 570-583 - An efficient estimator for the expected value of sample information.
*Med Decis Making.*2016; 36: 308-320 - A Gaussian approximation approach for value of information analysis.
*Med Decis Making.*2018; 38: 174-188 - Computing expected value of partial sample information from probabilistic sensitivity analysis using linear regression metamodeling.
*Med Decis Making.*2015; 35: 584-595 - Efficient Monte Carlo estimation of the expected value of sample information using moment matching.
*Med Decis Making.*2018; 38: 163-173 - Calculating the expected value of sample information using efficient nested Monte Carlo: a tutorial.
*Value Health.*2018; 21: 1299-1304 - Bayesian curve fitting to estimate the expected value of sample information using moment matching across different sample sizes.
*Med Decis Making.*2018; ([in press]) - EVSI: a suite of functions for the calculation and presentation of the EVSI.https://github.com/annaheath/EVSIDate accessed: October 4, 2019
- The collaborative network for value of information.https://www.convoi-group.org/Date accessed: June 9, 2019
- Applied Statistical Decision Theory.Harvard University Press, Boston, MA1961
- Net health benefits: a new framework for the analysis of uncertainty in cost-effectiveness analysis.
*Med Decis Making.*1998; 18: S68-S80 - Decision Modelling for Health Economic Evaluation.Oxford University Press, Oxford2006
- Uncertainty and the evaluation of public investment decisions.
*Am Econ Rev.*1970; 60 - Probabilistic sensitivity analysis in health economics.
*Stat Methods Med Res.*2015; 24: 615-634 - Handling uncertainty in cost-effectiveness models.
*Pharmacoeconomics.*2000; 17: 479-500 - Estimating the expected value of partial perfect information: a review of methods.
*Eur J Health Econ.*2008; 9: 251-259 - A review of methods for analysis of the expected value of information.
*Med Decis Making.*2017; 37: 747-758 - Efficient computation of partial expected value of sample information using Bayesian approximation.
*J Health Econ.*2007; 26: 122-148 - Estimating the expected value of partial perfect information in health economic evaluations using integrated nested Laplace approximation.
*Stat Med.*2016; 35: 4264-4280 - Bayesian Cost Effectiveness Analysis with the R Package BCEA.Springer, 2017
- SAVI - Sheffield accelerated value of information, release version 1.013 (2014-12-11).http://savi.shef.ac.uk/SAVI/Date accessed: June 9, 2019
- Partial EVPPI functions.http://www.shef.ac.uk/polopoly_fs/1.305039!/file/R_functions.txtDate accessed: June 9, 2019
- Thin plate regression splines.
*J Royal Stat Soc.*2003; 65: 95-114 - mgcv: mixed GAM computation vehicle with GCV/AIC/REML smoothness estimation and GAMMs by REML/PQL.University of Bath, 2016
- Gam: stata module for generalised additive models.Boston College Department of Economics, 2012
- Fitting generalized additive models with the gam procedure in SAS 9.2. SAS Global Forum.(Accessed September 25, 2019)
- Gaussian processes for machine learning.(Accessed September 25, 2019)
- Generalized Additive Models.CRC Press, Boca Raton, FL1990
- Guidance for the use of Bayesian statistics in medical device clinical trials.(Accessed September 25, 2019)
- WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility.
*Statistics and Computing.*2000; 10: 325-337 - rjags: Bayesian graphical models using MCMC: R package version 4-4.(Accessed September 25, 2019)
- A probabilistic programming language for Bayesian inference and optimization.
*J Educ Behav Stat.*2015; 40: 530-543 - Determining the effective sample size of a parametric prior.
*Biometrics.*2008; 64: 595-602 - Estimating the expected value of sample information across different sample sizes using moment matching and nonlinear regression.
*Med Decis Making.*2019; 39: 347-359 - Calculating value of information measures for health economic evaluations within R.(Accessed September 25, 2019)
- A value of information analysis of research on the 21-gene assay for breast cancer management.
*Value Health.*2019; 22: 1102-1110 - The curve of optimal sample size (COSS): a graphical representation of the optimal sample size from a value of information analysis.
*Pharmacoeconomics.*2019; 37: 871-877 - Multilevel Monte Carlo estimation of the expected value of sample information.2019 (arXiv: 190512013 [Preprint])
- Minimal modeling approaches to value of information analysis for health research.
*Med Decis Making.*2011; 31: E1-E22 - Value of information analytical methods: report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force.
*Value Health.*2020; 23: 277-286

## Article info

### Publication history

Published online: May 27, 2020

### Identification

### Copyright

© 2020 ISPOR-The Professional Society for Health Economics and Outcomes Research. Published by Elsevier Inc.

### User license

Elsevier user license | How you can reuse

Elsevier's open access license policy

Elsevier user license

## Permitted

### For non-commercial purposes:

- Read, print & download
- Text & data mine
- Translate the article

## Not Permitted

- Reuse portions or extracts from the article in other works
- Redistribute or republish the final article
- Sell or re-use for commercial purposes

Elsevier's open access license policy