Advertisement

Prioritizing Additional Data Collection to Reduce Decision Uncertainty in the HIV/AIDS Response in 6 US Cities: A Value of Information Analysis

Open ArchivePublished:October 03, 2020DOI:https://doi.org/10.1016/j.jval.2020.06.013

      Highlights

      • Achieving the ambitious goals set by the US Ending the HIV Epidemic initiative necessitates localized combination implementation strategies. Economic models provide useful tools to inform health decision making. Nevertheless, the inherent uncertainty underlying model parameters may propagate into decision uncertainty regarding what constitutes the optimal response. Value of information analysis estimates the value of collecting additional data to reduce decision uncertainty and helps guide efforts of data collection to focus on the most influential parameters.
      • We demonstrated a pragmatic approach in applying a metamodeling approach in value of information analysis in a dynamic transmission model. This study presents one of the few examples of applying advanced metamodeling approaches, including a deep learning method, in estimating the expected value of partial perfect information for large groups of parameters and many competing strategies.
      • We found substantial value in further data collection to reduce decision uncertainty in Baltimore, Los Angeles, and New York City. We also identified the primary contributors to the uncertainty in the decisions and recommended further data collection on HIV risk behaviors, probability of HIV transmission, health service engagement, HIV-related mortality, health utility weights, and healthcare costs.

      Abstract

      Objectives

      The ambitious goals of the US Ending the HIV Epidemic initiative will require a targeted, context-specific public health response. Model-based economic evaluation provides useful guidance for decision making while characterizing decision uncertainty. We aim to quantify the value of eliminating uncertainty about different parameters in selecting combination implementation strategies to reduce the public health burden of HIV/AIDS in 6 US cities and identify future data collection priorities.

      Methods

      We used a dynamic compartmental HIV transmission model developed for 6 US cities to evaluate the cost-effectiveness of a range of combination implementation strategies. Using a metamodeling approach with nonparametric and deep learning methods, we calculated the expected value of perfect information, representing the maximum value of further research to eliminate decision uncertainty, and the expected value of partial perfect information for key groups of parameters that would be collected together in practice.

      Results

      The population expected value of perfect information ranged from $59 683 (Miami) to $54 108 679 (Los Angeles). The rank ordering of expected value of partial perfect information on key groups of parameters were largely consistent across cities and highest for parameters pertaining to HIV risk behaviors, probability of HIV transmission, health service engagement, HIV-related mortality, health utility weights, and healthcare costs. Los Angeles was an exception, where parameters on retention in pre-exposure prophylaxis ranked highest in contributing to decision uncertainty.

      Conclusions

      Funding additional data collection on HIV/AIDS may be warranted in Baltimore, Los Angeles, and New York City. Value of information analysis should be embedded into decision-making processes on funding future research and public health intervention.

      Keywords

      Introduction

      Despite over $20 billion of annual federal funding directed toward the domestic human immunodeficiency virus and acquired immunodeficiency syndrome (HIV/AIDS) response, 38 000 new cases were diagnosed in 2017 in the United States.
      Centers for Disease Control and Prevention (CDC)
      ,
      Kaiser Family Foundation
      U.S. federal funding for HIV/AIDS: trends over time.
      The recently announced Ending the HIV Epidemic: A Plan for America initiative has set ambitious goals to reduce new HIV infections by 75% within 5 years and by 90% within 10 years.
      US Department of Health and Human Services
      Ending the HIV epidemic: a plan for America.
      Achieving these goals necessitates “an infusion of resources to employ strategic practices in the right places targeted to the right people to maximize impact and end the HIV epidemic in America.”
      US Department of Health and Human Services
      Ending the HIV epidemic: a plan for America.
      Although a range of tools to end the HIV epidemic are available, resources—both financial and human—are scarce and compete with other priorities outside HIV.
      • Schwartlander B.
      • Stover J.
      • Hallett T.
      • et al.
      Towards an improved investment approach for an effective response to HIV/AIDS.
      As the global public health response moves toward localized combination implementation strategies to combat HIV/AIDS,
      • Schwartlander B.
      • Stover J.
      • Hallett T.
      • et al.
      Towards an improved investment approach for an effective response to HIV/AIDS.
      • Jones A.
      • Cremin I.
      • Abdullah F.
      • et al.
      Transformation of HIV from pandemic to low-endemic levels: a public health approach to combination prevention.
      • Panagiotoglou D.
      • Olding M.
      • Enns B.
      • et al.
      Building the case for localized approaches to HIV: structural conditions and health system capacity to address the HIV/AIDS epidemic in six US cities.
      • Hankins C.A.
      • de Zalduondo B.O.
      Combination prevention: a deeper understanding of effective HIV prevention.
      economic models are increasingly used to inform health decision making in the care and prevention of HIV/AIDS. They provide a unified framework to quantify the health and economic value of different strategies while accounting for their synergistic effects.
      • Garnett G.P.
      • Cousens S.
      • Hallett T.B.
      • et al.
      Mathematical models in the evaluation of health programmes.
      • Garnett G.P.
      An introduction to mathematical models in sexually transmitted disease epidemiology.
      • Briggs A.
      • Claxton K.
      • Sculpher M.
      Decision Modelling for Health Economic Evaluation.
      Adequately characterizing city-level HIV microepidemics in a simulation modeling framework is data-intensive, particularly given the substantial disparities in access to services and health outcomes observed across HIV risk and race/ethnic groups in the United States.
      • Zang X.
      • Krebs E.
      • Wang L.
      • et al.
      Structural design and data requirements for simulation modelling in HIV/AIDS: a narrative review.
      The inherent uncertainty in estimating many of the required parameters, which span population composition and dynamics, health service engagement, disease progression, HIV transmission, cost, and utility weights,
      • Zang X.
      • Krebs E.
      • Wang L.
      • et al.
      Structural design and data requirements for simulation modelling in HIV/AIDS: a narrative review.
      ,
      • Krebs E.
      • Enns B.
      • Wang L.
      • et al.
      Developing a dynamic HIV transmission model for 6 U.S. cities: an evidence synthesis.
      may obscure value-based recommendations and lead to potentially suboptimal decisions.
      The United States has invested in a range of multisite surveillance studies that provide vital information for simulation modeling, including HIV risk behaviors from the National HIV Behavioral Surveillance (NHBS), antiretroviral therapy (ART) surveillance data from Medical Monitoring Project from the US Centers for Disease Control and Prevention, and clinical data from the HIV Research Network.
      • Krebs E.
      • Enns B.
      • Wang L.
      • et al.
      Developing a dynamic HIV transmission model for 6 U.S. cities: an evidence synthesis.
      Nonetheless, our previous study found a paucity of representative, high-quality local administrative data, particularly on rates of HIV testing and ART engagement.
      • Krebs E.
      • Enns B.
      • Wang L.
      • et al.
      Developing a dynamic HIV transmission model for 6 U.S. cities: an evidence synthesis.
      This is in many cases owing to structural barriers where data are either not systematically collected or are organized in isolated silos, creating barriers to access and linkage with other data sources.
      • Grossman C.
      • Goolsby W.A.
      • Olsen L.
      • et al.
      Clinical data as the basic staple of health learning: creating and protecting a public good.
      The infrastructure to facilitate data collection is costly to construct, and not all model parameters will affect recommendations equally. Efforts to collect additional data should focus on those with the most potential to reduce uncertainty in funding allocation decisions.
      • Shepherd K.
      • Hubbard D.
      • Fenton N.
      • et al.
      Policy: development goals should enable decision-making.
      Value of information (VoI) analysis quantifies the opportunity cost of acting on suboptimal recommendations owing to uncertainty underlying model parameters. It also helps guide efforts to ensure the limited resources available for data collection are focused on the most influential parameters for a given decision.
      • Shepherd K.
      • Hubbard D.
      • Fenton N.
      • et al.
      Policy: development goals should enable decision-making.
      Expected value of perfect information (EVPI) and expected value of partial perfect information (EVPPI) estimate the value of eliminating uncertainty from all parameters jointly and from a subset of parameters, respectively, providing guidance for setting research priorities. The expected value of sample information (EVSI) measures the expected value of sample (imperfect) information from a given study of a particular design with a known sample size, which can be used to inform future research design. The conventional approach to EVPPI and EVSI is based on a nested Monte Carlo procedure that is computationally costly, particularly for a complex model with many competing interventions.
      • Strong M.
      • Oakley J.E.
      • Brennan A.
      Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach.
      • rabideau dj
      • pei pp
      • walensky rp
      • et al.
      implementing generalized additive models to estimate the expected value of sample information in a microsimulation model: results of three case studies.
      • Jalal H.
      • Goldhaber-Fiebert J.D.
      • Kuntz K.M.
      Computing expected value of partial sample information from probabilistic sensitivity analysis using linear regression metamodeling.
      Metamodeling, a tool that simplifies the relationship between the inputs and outputs of a simulation model,
      • Jalal H.
      • Dowd B.
      • Sainfort F.
      • et al.
      Linear regression metamodeling as a tool to summarize and present simulation model results.
      has been increasingly used as an alternative method to expedite VoI analysis.
      • Strong M.
      • Oakley J.E.
      • Brennan A.
      Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach.
      ,
      • Heath A.
      • Manolopoulou I.
      • Baio G.
      A review of methods for analysis of the expected value of information.
      A previous EVPPI analysis using a metamodeling approach shortened the computation time from weeks to minutes while consistently yielding similar estimates.
      • rabideau dj
      • pei pp
      • walensky rp
      • et al.
      implementing generalized additive models to estimate the expected value of sample information in a microsimulation model: results of three case studies.
      On the basis of a comprehensive evidence synthesis,
      • Krebs E.
      • Enns B.
      • Wang L.
      • et al.
      Developing a dynamic HIV transmission model for 6 U.S. cities: an evidence synthesis.
      a systematic model calibration and validation,
      • Zang X.
      • Krebs E.
      • Min J.E.
      • et al.
      Development and calibration of a dynamic HIV transmission model for 6 US cities.
      and an extensive economic evaluation of 16 evidence-based interventions individually
      • Krebs E.
      • Zang X.
      • Enns B.
      • et al.
      The impact of localized implementation: determining the cost-effectiveness of HIV prevention and care interventions across six United States cities.
      and in combination
      • Nosyk B.
      • Zang X.
      • Krebs E.
      • et al.
      Ending the HIV epidemic in the USA: an economic modelling study in six cities.
      for 6 US cities with diverse HIV microepidemics, we identified combination strategies to address the HIV/AIDS epidemic in each city along with uncertainty underlying the selection of the optimal (highest-valued) strategy. A decomposition of the sources of this uncertainty, coupled with a judgment on the value of eliminating it in each setting, can guide future research efforts focused explicitly on reducing decision uncertainty. Our objective is to quantify the value of eliminating uncertainty about different groups of parameters to reduce decision uncertainty regarding the cost-effectiveness of HIV combination implementation strategies in 6 US cities. We aimed to identify future research priorities through EVPPI analysis on key groups of uncertain parameters using a metamodeling approach with advanced regression techniques.

      Methods

      Model Description

      We adapted a previously published dynamic compartmental HIV transmission model
      • Zang X.
      • Krebs E.
      • Min J.E.
      • et al.
      Development and calibration of a dynamic HIV transmission model for 6 US cities.
      ,
      • Nosyk B.
      • Min J.E.
      • Krebs E.
      • et al.
      The cost-effectiveness of HIV testing and treatment engagement initiatives in British Columbia, Canada: 2011-2013.
      • Nosyk B.
      • Zang X.
      • Min J.E.
      • et al.
      Relative effects of antiretroviral therapy and harm reduction initiatives on HIV incidence in British Columbia, Canada, 1996-2013: a modelling study.
      • Nosyk B.
      • Min J.E.
      • Lima V.D.
      • et al.
      Cost-effectiveness of population-level expansion of highly active antiretroviral treatment for HIV in British Columbia, Canada: a modelling study.
      to simulate the HIV microepidemics in 6 US cities: Atlanta, Georgia; Baltimore, Maryland; Los Angeles (LA), CA; Miami, Florida; New York City (NYC), New York; and Seattle, Washington. Each city features distinct epidemiological characteristics and public health responses to HIV.
      • Panagiotoglou D.
      • Olding M.
      • Enns B.
      • et al.
      Building the case for localized approaches to HIV: structural conditions and health system capacity to address the HIV/AIDS epidemic in six US cities.
      In the model, the adult population aged 15 to 64 in each city was partitioned by biological sex, HIV risk group, race/ethnicity, sexual risk behavior level (high- vs. low-risk), and access to opioid agonist treatment among people who inject drugs, for a total of 42 population groups. Individuals within each group progress through 19 health states illustrated in Appendix Figure 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.06.013. We considered dynamic HIV transmission through heterosexual contact, homosexual contact, or needle sharing. The model captured heterogeneity in the risk of HIV transmission, maturation, mortality, and accessing health and prevention services. Full details about the model are available in prior articles.
      • Zang X.
      • Krebs E.
      • Min J.E.
      • et al.
      Development and calibration of a dynamic HIV transmission model for 6 US cities.
      • Krebs E.
      • Zang X.
      • Enns B.
      • et al.
      The impact of localized implementation: determining the cost-effectiveness of HIV prevention and care interventions across six United States cities.
      • Nosyk B.
      • Zang X.
      • Krebs E.
      • et al.
      Ending the HIV epidemic in the USA: an economic modelling study in six cities.
      The model’s input parameters were grouped in 7 categories: (1) initial population estimates and population dynamics (N = 1074); (2) parameters used to calculate the probability of HIV transmission (N = 201); (3) screening, diagnosis, treatment, and HIV disease progression (N = 312); (4) HIV prevention programs (N = 23); (5) the costs of medical care (N = 30); (6) health utility weights (N = 26); and (7) intervention parameters (eg, effect, scale of delivery, and cost) (N = 184). The derivations, point estimates, uncertainty distributions, and quality of evidence were reported for all model parameters in an article detailing the evidence synthesis for this project.
      • Krebs E.
      • Enns B.
      • Wang L.
      • et al.
      Developing a dynamic HIV transmission model for 6 U.S. cities: an evidence synthesis.
      ,
      • Krebs E.
      • Zang X.
      • Enns B.
      • et al.
      The impact of localized implementation: determining the cost-effectiveness of HIV prevention and care interventions across six United States cities.
      For parameters with lower quality of evidence where little information was available, we allowed for prior distributions with higher variance.
      • Briggs A.H.
      • Weinstein M.C.
      • Fenwick E.A.
      • et al.
      Model parameter estimation and uncertainty: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-6.
      Although parameters were assumed to be independent, for potentially correlated parameters with an empirically established rank ordering (eg, quality-adjusted life-year [QALY] weights, costs for different health states), we used a recently proposed method
      • Goldhaber-Fiebert J.D.
      • Jalal H.J.
      Some health states are better than others: using health state rank order to improve probabilistic analyses.
      to establish joint uncertainty distributions from parameters’ marginal distributions and their ordinal preferences.

      Cost-Effectiveness Analysis

      We evaluated the cost-effectiveness of 16 evidence-based HIV interventions and their combinations within 3 domains: (1) HIV prevention, including syringe service programs, medication for opioid use disorder, either buprenorphine or methadone, and targeted pre-exposure prophylaxis (PrEP) for high-risk men who have sex with men; (2) HIV testing, including opt-out testing in the hospital emergency room and in primary care, electronic medical record testing offer reminder, nurse-initiated rapid testing, and medication for opioid use disorder initiated rapid testing; and (3) treatment engagement, including case management (antiretroviral treatment and access to services) for ART initiation, care coordination for ART retention, targeted care coordination (CD4 < 200), electronic medical record ART engagement reminder, rapid ART initiation, enhanced personal contact for ART reinitiation, and ART re-linkage program. We considered combinations of the 16 interventions for each city, excluding any that would not practically be implemented jointly, for a total of 23 040 unique combinations.
      Consistent with our previous studies, we maintained the same evaluation time horizon of 20 years (2020-2040) to capture the long-term individual benefits of ART and second-order transmission effects. All costs were reported in 2018 US dollars, and we chose a base cost-effectiveness threshold of $100 000/QALY while varying this threshold in 3 discrete levels ($50 000/QALY, $100 000/QALY, and $150 000/QALY) for EVPI analysis. Decisions were made by choosing the strategyi with the highest net monetary benefit (NMB), as estimated by:
      NMBi=QALYi×ThresholdCosti


      Probabilistic Sensitivity Analysis

      The probabilistic sensitivity analysis (PSA) dataset for input parameters was constructed by randomly sampling all uncertain parameters (including initial population estimates) simultaneously from their prior distributions or fitted joint distributions. For each sample of the model inputs, corresponding costs and QALYs for the given combination implementation strategies were estimated, representing 1 PSA simulation. As a most of the over 23 000 competing strategies we estimated had a zero or near-zero probability of being the health-maximizing strategy for a given threshold, we performed PSA only on the subset of combination implementation strategies with a non-zero probability of being selected.

      VoI Analysis

      VoI analysis provides a systematic approach to quantify the value of research in reducing decision uncertainty. EVPI represents the maximum a decision maker should be willing to pay to eliminate uncertainty from all model parameters. For a given cost-effectiveness threshold, we first evaluated and ranked the NMB of all combination strategies setting all input parameters at their point estimates as the base case (ie, a deterministic) analysis and identified the 25 combination strategies with the highest NMB for each city. We then generated 2000 random samples of model inputs, conducted PSA on these 25 highest-valued strategies, and estimated their corresponding NMB. We chose 25 to make the problem tractable given computation limitations, and the final number was determined through an iterative process: at each threshold level, we gradually expanded the inclusion of top-ranked strategies (from 5 to 25 in an increment of 5) while assessing their probabilities of being the highest-valued (ie, with highest NMB) in the 2000 PSA simulations. We proceeded with the process until we found that strategies with NMB from the deterministic analysis ranked below 20 had nearly zero probability to be selected. We then calculated the opportunity loss for a given simulation as the difference of NMB between the optimal strategy (ie, with highest NMB) in a given simulation and the overall optimal strategy—the combination strategy with the highest expected NMB across all simulations. Finally, EVPI was estimated by averaging the opportunity loss across all simulations. Given that our model is developed at the population level, the computed EVPI directly represents a population estimate, thus requiring no extrapolation. We also expressed this uncertainty as a cost per million residents to clarify relative differences across cities.
      EVPPI provides an estimate of the value of eliminating uncertainty for specific parameters or groups of parameters. The value of collecting additional information for particular parameters depends on both their influence on the cost-effectiveness of competing strategies and the extent of uncertainty.
      • Oostenbrink J.B.
      • Al M.J.
      • Oppe M.
      • et al.
      Expected value of perfect information: an empirical example of reducing decision uncertainty by conducting additional research.
      Using a metamodeling approach,
      • Strong M.
      • Oakley J.E.
      • Brennan A.
      Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach.
      ,
      • Heath A.
      • Manolopoulou I.
      • Baio G.
      A review of methods for analysis of the expected value of information.
      we modeled the opportunity loss of each strategy (same as in EVPI) as a function of the parameters of interest at the base threshold of $100 000/QALY. We adopted 2 advanced metamodeling methods with a generalized additive model (GAM) and a deep learning artificial neural network (ANN) model to approximate the inner expectation of the conventional nested Monte Carlo approach.
      • Strong M.
      • Oakley J.E.
      • Brennan A.
      Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach.
      • rabideau dj
      • pei pp
      • walensky rp
      • et al.
      implementing generalized additive models to estimate the expected value of sample information in a microsimulation model: results of three case studies.
      • Jalal H.
      • Goldhaber-Fiebert J.D.
      • Kuntz K.M.
      Computing expected value of partial sample information from probabilistic sensitivity analysis using linear regression metamodeling.
      In simulation models like ours where the relationship between opportunity loss and parameters is nonlinear, these methods provide highly flexible functions requiring no parametric assumptions and are found to outperform linear modeling approaches.
      • Jalal H.
      • Alarid-Escudero F.
      A Gaussian approximation approach for value of information analysis.
      ,
      • Alam M.F.
      • Briggs A.
      Artificial neural network metamodel for sensitivity analysis in a total hip replacement health economic model.
      The GAM is efficient and more straightforward to implement with high precision, but is constrained by the size of regression parameters because the required PSA sample size increases exponentially.
      • Strong M.
      • Oakley J.E.
      • Brennan A.
      Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach.
      We therefore used GAM for each individual parameter and the ANN model, which can approximate complex, nonlinear relationships in data with high dimensionality, to estimate the full EVPPIs for parameters in groups. Key groups of parameters were selected by having at least 1 parameter whose EVPPI > (0.05 × EVPI), and groupings were based on which parameters were likely to be informed by any one single study in practice. Given the large number of input parameters, we required a larger number of PSA runs to avoid overfitting and achieve more precise EVPPI estimates. The outputs from GAM for each individual parameter were used to inform the selection of key groups of parameters for ANN metamodel analysis. Following a procedure displayed in Appendix Figure 2 (in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.06.013), we expanded the PSA samples to 20 000 and performed EVPPI analysis with the 2 aforementioned methods. We then compared the resulting EVPPI estimates to inform targets for prioritizing future data collection. We constructed the ANN for each metamodel with 4 hidden layers and 10 hidden nodes per layer.
      • Fonseca D.J.
      • Navaresse D.O.
      • Moynihan G.P.
      Simulation metamodeling through artificial neural networks.
      To address the randomness in generating initial weights for the ANN algorithm, we repeated each estimation 10 times and averaged the results for the final EVPPI values. We described the process for result validation and tuning for the ANN model in detail in the Appendix (see Appendix in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.06.013). We executed model simulation using R version 3.6.1 (www.r-project.org) and built the GAM and ANN metamodels using the R package mgcv (version 1.8-31)
      • Wood S.
      mgcv: Mixed GAM computation vehicle with automatic smoothness estimation. R package version 1.8-31.
      and keras (version 2.2.5.0),
      • Falbel D.
      • Allaire J.
      • Chollet F.
      keras: R interface to ‘Keras’. R package version 2.2.5.0.
      respectively.

      Results

      Uncertainty in Cost-Effectiveness

      Results from our PSA demonstrate that the optimal combination implementation strategies, that is, with the highest NMB, varied across cities (Fig 1, Fig 2). At the base threshold of $100 000/QALY, the probability of the most probable strategies to be highest-valued ranged from 30% (LA) to 56% (Atlanta). The uncertainty came from the decision on choosing between competing combinations that differed by only 1 or 2 interventions. This uncertainty may result in opportunity costs owing to acting on the suboptimal decision and is strongly associated with the magnitude of EVPI and EVPPI. Results for other thresholds (see Appendix Figs. 4 and 5 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.06.013) indicated that the level of uncertainty generally decreased with higher cost-effectiveness thresholds.
      Figure thumbnail gr1
      Figure 1The probabilities of competing combination strategies representing the highest value at a cost-effectiveness threshold of $100 000 per QALY.
      QALY indicates quality-adjusted life-year.
      Figure thumbnail gr2
      Figure 2The composition of the highest-valued combination strategies at a cost-effectiveness threshold of $100 000 per QALY.
      LA indicates Los Angeles; NYC, New York City; MOUD, medication for opioid use disorder; PrEP, pre-exposure prophylaxis; MSM, men who have sex with men; ER, emergency room; EMR, electronic medical record; ART, antiretroviral therapy; QALY, quality-adjusted life-year.

      Expected Value of Perfect Information

      Figure 3 shows the EVPI for each city at the base cost-effectiveness threshold. The city-level EVPI was found to be small in Miami ($59 683), Atlanta ($92 684), and Seattle ($340 439) and relatively greater in Baltimore ($1 348 364), NYC ($7 649 541), and LA ($54 108 679). Per individual EVPI, after standardizing with city population size, it was generally consistent with the city-level results and was highest in LA ($8 009 105 per million residents) and lowest in Atlanta ($20 880 per million residents). These upper bounds on the return of future research demonstrate a higher chance to exceed the costs of feasible research for improving data precision. Through estimating EVPI at different thresholds (see Appendix 6 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.06.013), we found that values became smaller with increasing threshold levels when the decision uncertainty was reduced, except for LA. The large increase in EVPI for LA from $50 000/QALY to $100 000/QALY reflected uncertainty surrounding the inclusion of PrEP in the combination implementation strategy (an effective yet very costly strategy) at the higher threshold.
      Figure thumbnail gr3
      Figure 3Expected value of perfect information (EVPI) results at a cost-effectiveness threshold of $100 000 per QALY. (A) The city-level EVPI. (B) The EVPI per million residents. Population size (aged 15-64) used to calculate the EVPI per million residents: Atlanta: 4 447 524; Baltimore: 1 844 759; Los Angeles: 6 755 896; Miami: 1 946 897; New York City: 5 894 751; Seattle: 1 542 132.

      Expected Value of Partial Perfect Information

      We presented in Appendix Figure 2 (see Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.06.013) the distribution of EVPPI values we estimated at a threshold of $100 000/QALY from metamodeling analysis with a GAM on each individual parameter for each city. Many parameters were found to have an EVPPI of $0, implying acquiring additional information on them is unlikely to affect the allocation decision. We found a high degree of consistency across cities on the parameters that were most influential on decision uncertainty and we grouped them accordingly: (1) mortality rate estimates for people living with HIV, stratified by injection drug use status and ART receipt (N = 58); (2) the probability of direct ART engagement (within 1 month of diagnosis) (N = 15); (3) the rate of ART initiation (N = 15); (4) the rate of ART dropout (N = 54); (5) the number of sexual partners, both heterosexual and homosexual (N = 25); (6) HIV transmission probabilities per heterosexual partnership (N = 6); (7) HIV transmission probabilities per homosexual partnership (N = 3); (8) HIV transmission probabilities per shared injection contact (N = 3); (9) the HIV testing rate across strata (N = 30); (10) QALY weights (N = 11); and (11) healthcare costs for individuals receiving ART (N = 18) (see Appendix Table 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.06.013). Some additional groups of influential parameters specific to individual cities included (12) injection frequency among people who inject drugs (N = 2), (13) the effectiveness of syringe service programs (N = 1) in reducing the probability of shared injection (Atlanta, Baltimore, and Miami), and (14) the PrEP retention rate (LA, N = 1). Figure 4 compares the EVPPI for these key groups of parameters, showing that the VoI associated with reducing parameter uncertainty was highest for the number of sexual partners (ranged from $36 695 in Miami to $19 619 277 in LA), the probability of immediate ART engagement (ranged from $32 380 in Miami to $8 660 878 in LA), and HIV testing parameters (ranged from $34 094 in Miami to $9 665 386 in LA) except in LA where the EVPPI for PrEP retention rate ($22 861 668) exceeded all other parameters.
      Figure thumbnail gr4
      Figure 4The city-level expected value of partial perfect information (EVPPI) for identified parameter groups at a cost-effectiveness threshold of $100 000 per QALY. Bracketed group indexes correspond to the ones defined in Results section. Parameter groups have been ordered according to the ranking of estimated EVPPI values. The error bars represent the mean ± 1.96 × SD of the estimated values from the 10 repeated estimations. Parameter groups with 2 or fewer parameters were estimated by ANN model and thus had no error bars.
      ANN indicates artificial neural network; N, number of parameters in the given group.

      Discussion

      We performed a VoI analysis to determine the potential value of additional data collection in reducing decision uncertainty on the cost-effectiveness of localized HIV combination implementation strategies in 6 US cities. The high EVPIs we estimated for Baltimore, New York City, and LA suggest there is substantial value in further data collection to reduce decision uncertainty in these cities. Through an extensive EVPPI analysis, we identified the primary contributors to the uncertainty in these decisions and recommended further data collection on HIV risk behaviors, probability of HIV transmission per sexual partnership or shared injection contact, health service engagement, HIV-related mortality, health utility weights, and healthcare costs. Given existing data collection infrastructure in the United States for most of the identified parameters groups, improving the validity and precision of data will likely rely on enhancing current efforts rather than launching new research efforts at greater costs.
      The low EVPI values we estimated for Atlanta, Miami, and Seattle at a $100 000/QALY threshold were a result of their relatively smaller populations and lower levels of decision uncertainty. Although this may suggest that funding further data collection may not be justified, these findings should be interpreted with caution, because they reflect the current status of a given microepidemic (and our knowledge of it) and thus may change over time. For example, the current high incidence of HIV in Miami makes most interventions highly cost-effective despite substantial uncertainty in expected costs and benefits of implementation. Nevertheless, if incidence declines in the future, the probability of a suboptimal implementation decision—and accordingly, the EVPI—will increase. EVPI calculations should thus be updated alongside prospective modeling efforts.
      EVPPI analysis provides insight into the value of reducing uncertainty for model parameters that are most influential to decision uncertainty. We identified several key groups of parameters that were consistently detected as potential targets across cities for future research, including HIV risk behaviors, probability of HIV transmission, health service engagement; HIV-related mortality, health utility weights, and healthcare costs (for people living with HIV on ART). Many model parameters within these groups were also found to be highly influential on epidemiological targets in a global sensitivity analysis in our previous model calibration work.
      • Zang X.
      • Krebs E.
      • Min J.E.
      • et al.
      Development and calibration of a dynamic HIV transmission model for 6 US cities.
      The value of other parameters was low or negligible. This does not mean that their uncertainty is unimportant, but that gathering more accurate information about these parameters would be unlikely to help us reduce decision uncertainty and narrow the ultimate decision. In particular, improving estimates on the rate of PrEP retention was valued at over $20 million in LA but was negligible in other cities. Though effective in preventing HIV transmission, PrEP is a costly intervention whose value is highly dependent on recipients’ persistence,
      • Cambiano V.
      • Miners A.
      • Dunn D.
      • et al.
      Cost-effectiveness of pre-exposure prophylaxis for HIV prevention in men who have sex with men in the UK: a modelling study and health economic evaluation.
      and the evidence informing PrEP retention was characterized by a wide uncertainty range (20.6 months, range: 13.1-49.6),
      • Krebs E.
      • Enns B.
      • Wang L.
      • et al.
      Developing a dynamic HIV transmission model for 6 U.S. cities: an evidence synthesis.
      resulting in substantial variation in PrEP persistence. Nevertheless, in all cities except LA, this intervention had either 100% or 0% probability of entering as a component in the optimal strategy and thus had negligible impact on decision uncertainty, resulting in a stark contrast in valuation of EVPPI.
      Collectively, parameters pertaining to the number of sexual partners contributed to a high degree of decision uncertainty across all cities. The importance of these behavioral parameters has also been noted in many prior modeling studies.
      • Shafer L.A.
      • Nsubuga R.N.
      • Chapman R.
      • et al.
      The dual impact of antiretroviral therapy and sexual behaviour changes on HIV epidemiologic trends in Uganda: a modelling study.
      ,
      • Hallett T.B.
      • Gregson S.
      • Mugurungi O.
      • et al.
      Assessing evidence for behaviour change affecting the course of HIV epidemics: a new mathematical modelling approach and application to data from Zimbabwe.
      Our current model parameter values were informed by evidence from NHBS

      Centers for Disease Control and Prevention. HIV risk, prevention, and testing behaviors - National HIV Behavioral Surveillance System: men who have sex with men, 20 US cities, 20 HIV Special Report 8: Centers for Disease Control and Prevention. 2014.

      and the National Survey of Family Growth.

      Centers for Disease Control and Prevention. Public use data file documentation. 2011-2013. National Survey of Family Growth. User’s guide. Hyattsville, MD: Centers for Disease Control and Prevention, National Center for Health Science; 2014.

      Although these population-based surveillance databases provide the best available evidence on this topic, these surveys featured several threats to internal and external validity that could potentially be improved on with greater investments. For example, National Survey of Family Growth only provided regional rather than city-level data while in NHBS, where city-specific data were available, sample sizes for some population groups were small, resulting in wide uncertainty ranges. Equally importantly, the framing of the survey questions did not always correspond to how the model parameters in this particular application were encoded, which may have inevitably introduced additional uncertainty from data triangulation. Enhancing behavioral evidence on the basis of the current surveillance system should focus on subpopulations with high-risk behaviors.
      • Zaba B.
      • Slaymaker E.
      • Urassa M.
      • et al.
      The role of behavioral data in HIV surveillance.
      An ongoing project may provide an example for a multisite community-engaged study that engages a high-coverage sample of men who have sex with men recruited using respondent-driven sampling.
      • Wang L.
      • Moqueet N.
      • Lambert G.
      • et al.
      Population-level sexual mixing by HIV status and pre-exposure prophylaxis use among men who have sex with men in Montreal, Canada: implications for HIV prevention.
      Notably, in this application researchers also applied standardized adjustment methods to generate population-representative estimates.
      • Volz E.
      • Heckathorn D.D.
      Probability based estimation theory for respondent driven sampling.
      Health service engagement was another parameter category found to be influential. Good-quality evidence for population-level HIV testing rates in the United States is sparse. We derived these testing parameters from NHBS,
      Centers for Disease Control and Prevention
      HIV infection, Risk, prevention, and testing behaviors among persons who inject drugs—national HIV behavioral surveillance: injection drug use, 20 U.S. cities, 2012. HIV Surveillance Special Report 11. Revised edition.
      ,
      Centers for Disease Control and Prevention
      HIV infection risk, prevention, and testing behaviors among men who have sex with men—national HIV behavioral surveillance, 20 U.S. cities, 2014. HIV Surveillance Special Report 15.
      the Behavioral Risk Factor Surveillance System,
      Centers for Disease Control and Prevention
      Behavioral Risk Factor Surveillance System Survey Data.
      and other sources in the peer-reviewed literature. Further, in the absence of city-specific data, ART engagement parameters were derived from corresponding regional HIV Research Network data
      The HIV Research Network (HIVRN)
      Goals of the HIV Research Network.
      and corresponding primary analysis.
      • Wang L.
      • Krebs E.
      • Min J.E.
      • et al.
      Combined estimation of disease progression and retention on antiretroviral therapy among treated individuals with HIV in the USA: a modelling study.
      These data, available from sample-based studies in selected sites, may have been unrepresentative of the general population. Establishing a centralized public health database to systematically collect and combine surveillance and health administrative data across different jurisdictions and agencies could substantially reduce the bias and uncertainty of population-level evidence. Although the Centers for Disease Control and Prevention is making efforts to enhance the collection, analysis, visualization, and dissemination of its surveillance data, challenges exist in coordinating information sharing, developing a broader surveillance workforce, and building health information systems to integrate data collected from various parties.
      • Richards C.L.
      • Iademarco M.F.
      • Atkinson D.
      • et al.
      Advances in public health surveillance and information dissemination at the Centers for Disease Control and Prevention.
      • Sharfstein J.M.
      Using healthcare data to track and improve public health.
      • Fojo A.T.
      • Dowdy D.W.
      Ending the human immunodeficiency virus epidemic: towards an evidence-based approach.
      Although several modern methods have been developed to replace the conventional nested Monte Carlo procedure in VoI estimation,
      • Heath A.
      • Manolopoulou I.
      • Baio G.
      A review of methods for analysis of the expected value of information.
      few applications using dynamic transmission models have been published.
      • rabideau dj
      • pei pp
      • walensky rp
      • et al.
      implementing generalized additive models to estimate the expected value of sample information in a microsimulation model: results of three case studies.
      Further, this study presents one of the few examples of applying advanced metamodeling approaches, including a deep learning method, in estimating the EVPPI for large groups of parameters and many competing strategies. We demonstrated a pragmatic approach in applying VoI analysis while maintaining a high level of rigor. We also found limited guidance on several critical steps of VoI analysis. One question we encountered was whether to include calibrated parameters with their posterior calibrated values or their prior uncertainty ranges in the PSA. We chose the latter option because we felt that the posterior ranges on calibrated parameters may arbitrarily shrink decision uncertainty and undervalue the estimated EVP(P)I. VoI analysis using calibrated values does not truly reflect our current knowledge about the uncertainty of parameters being calibrated and their impact on decision uncertainty. Further, the posterior ranges on calibrated parameters are subject to uncertainty stemming from a number of choices made in the calibration process, including the selected calibration targets, the goodness-of-fit measure, the parameters chosen for calibration, and the calibration algorithm.
      • Zang X.
      • Krebs E.
      • Min J.E.
      • et al.
      Development and calibration of a dynamic HIV transmission model for 6 US cities.
      We note that this was a departure from our prior analysis,
      • Nosyk B.
      • Zang X.
      • Krebs E.
      • et al.
      Ending the HIV epidemic in the USA: an economic modelling study in six cities.
      and explains the differences in the PSA results presented herein. Questions also remain about to what extent these VoI results are transferable beyond their current setting. For parameters that are based on common evidence and can be generalizable across locations, for example, transmission probabilities, it may be more meaningful to sum the relevant populations across locations when computing the population-level EVPPI to inform the true value of collecting additional data on those parameters.
      This analysis had several limitations. First, the procedure we undertook to only perform PSA on a subset of the highest-valued combination strategies identified in the deterministic base case might inevitably omit several strategies that had very small probabilities of being chosen as optimal. We felt these omissions had a negligible impact on the VoI while significantly improving computational efficiency in this application. Second, we did not consider potential developments in HIV interventions. Emerging evidence, which may comprise longer-acting ART and PrEP formulations or substantial cost reductions from the introduction of generic formulations, could alter the VoI estimates. Third, we did not attempt to calculate the EVSI, which remains a topic for further inquiry. Finally, the structural uncertainty attributable to our model was outside the scope of this VoI analysis, which focuses explicitly on parameter uncertainty. As decisions about model structure may also influence decision uncertainty, cross-model comparisons can provide a robust validation of the recommendations of a given model and highlight additional priorities on the nature of data that should be incorporated to reduce decision uncertainty.

      Conclusion

      We found uncertainty on the combination implementation strategies that would be most cost-effective for HIV prevention. This uncertainty is associated with high opportunity cost where benefits are likely to outweigh the cost of future research to improve parameter precision in 3 of 6 cities we considered in this project. To refine decision making for HIV, priority should be given to data collection on several key groups of parameters consistently found to be associated with the greatest decision uncertainty, including HIV risk behaviors, health service engagement, probability of HIV transmission, HIV-related mortality, health utility weights, and healthcare costs. Value of information analysis, combined with other information, such as study feasibility, relevance, and the cost of research, should be embedded into decision-making processes on funding and identifying targets for data acquisition efforts.

      Article and Author Information

      Accepted for Publication: June 30, 2020
      Published Online: October 3, 2020
      Author Contributions: Concept and design: Zang, Jalal, Pandya, Nosyk
      Acquisition of data: Zang, Enns
      Analysis and interpretation of data: Zang, Jalal, Krebs, Pandya, Zhou
      Drafting of the manuscript: Zang, Krebs, Enns, Nosyk
      Critical revision of the paper for important intellectual content: Zang, Krebs, Pandya, Zhou, Enns, Nosyk
      Statistical Analysis: Jalal, Zhou
      Obtaining funding: Nosyk
      Supervision: Jalal, Nosyk
      Conflicts of Interest Disclosures: The authors reported no conflicts of interest.
      Funding/Support: This study was funded by the National Institute on Drug Abuse (NIDA grant no. R01DA041747). The funders had no direct role in the conduct of the analysis or the decision to submit the manuscript.
      Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

      Supplemental Material

      References

        • Centers for Disease Control and Prevention (CDC)
        HIV Surveillance Report, 2017. 29. US Department of Health and Human Services, Atlanta, GA2018
        • Kaiser Family Foundation
        U.S. federal funding for HIV/AIDS: trends over time.
        2019
        • US Department of Health and Human Services
        Ending the HIV epidemic: a plan for America.
        2019
        • Schwartlander B.
        • Stover J.
        • Hallett T.
        • et al.
        Towards an improved investment approach for an effective response to HIV/AIDS.
        Lancet. 2011; 377: 2031-2041
        • Jones A.
        • Cremin I.
        • Abdullah F.
        • et al.
        Transformation of HIV from pandemic to low-endemic levels: a public health approach to combination prevention.
        Lancet. 2014; 384: 272-279
        • Panagiotoglou D.
        • Olding M.
        • Enns B.
        • et al.
        Building the case for localized approaches to HIV: structural conditions and health system capacity to address the HIV/AIDS epidemic in six US cities.
        AIDS Behavior. 2018; 22: 3071-3082
        • Hankins C.A.
        • de Zalduondo B.O.
        Combination prevention: a deeper understanding of effective HIV prevention.
        AIDS. 2010; 24: S70-S80
        • Garnett G.P.
        • Cousens S.
        • Hallett T.B.
        • et al.
        Mathematical models in the evaluation of health programmes.
        Lancet. 2011; 378: 515-525
        • Garnett G.P.
        An introduction to mathematical models in sexually transmitted disease epidemiology.
        Sex Transm Infect. 2002; 78: 7-12
        • Briggs A.
        • Claxton K.
        • Sculpher M.
        Decision Modelling for Health Economic Evaluation.
        1st ed. Oxford University Press, London, England2006
        • Zang X.
        • Krebs E.
        • Wang L.
        • et al.
        Structural design and data requirements for simulation modelling in HIV/AIDS: a narrative review.
        Pharmacoeconomics. 2019; 37: 1219-1239
        • Krebs E.
        • Enns B.
        • Wang L.
        • et al.
        Developing a dynamic HIV transmission model for 6 U.S. cities: an evidence synthesis.
        PloS One. 2019; 14e0217559
        • Grossman C.
        • Goolsby W.A.
        • Olsen L.
        • et al.
        Clinical data as the basic staple of health learning: creating and protecting a public good.
        Institute of Medicine, Washington, DC2011
        • Shepherd K.
        • Hubbard D.
        • Fenton N.
        • et al.
        Policy: development goals should enable decision-making.
        Nature. 2015; 523: 152-154
        • Strong M.
        • Oakley J.E.
        • Brennan A.
        Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach.
        Med Decis Making. 2014; 34: 311-326
        • rabideau dj
        • pei pp
        • walensky rp
        • et al.
        implementing generalized additive models to estimate the expected value of sample information in a microsimulation model: results of three case studies.
        Med Decis Making. 2018; 38: 189-199
        • Jalal H.
        • Goldhaber-Fiebert J.D.
        • Kuntz K.M.
        Computing expected value of partial sample information from probabilistic sensitivity analysis using linear regression metamodeling.
        Med Decis Making. 2015; 35: 584-595
        • Jalal H.
        • Dowd B.
        • Sainfort F.
        • et al.
        Linear regression metamodeling as a tool to summarize and present simulation model results.
        Med Decis Making. 2013; 33: 880-890
        • Heath A.
        • Manolopoulou I.
        • Baio G.
        A review of methods for analysis of the expected value of information.
        Med Decis Making. 2017; 37: 747-758
        • Zang X.
        • Krebs E.
        • Min J.E.
        • et al.
        Development and calibration of a dynamic HIV transmission model for 6 US cities.
        Med Decis Making. 2020; 40: 3-16
        • Krebs E.
        • Zang X.
        • Enns B.
        • et al.
        The impact of localized implementation: determining the cost-effectiveness of HIV prevention and care interventions across six United States cities.
        AIDS. 2020; 34: 447-458
        • Nosyk B.
        • Zang X.
        • Krebs E.
        • et al.
        Ending the HIV epidemic in the USA: an economic modelling study in six cities.
        Lancet HIV. 2020; 7: 491-503
        • Nosyk B.
        • Min J.E.
        • Krebs E.
        • et al.
        The cost-effectiveness of HIV testing and treatment engagement initiatives in British Columbia, Canada: 2011-2013.
        Clin Infectious Dis. 2018; 66: 765-777
        • Nosyk B.
        • Zang X.
        • Min J.E.
        • et al.
        Relative effects of antiretroviral therapy and harm reduction initiatives on HIV incidence in British Columbia, Canada, 1996-2013: a modelling study.
        Lancet HIV. 2017; 4: e303-e310
        • Nosyk B.
        • Min J.E.
        • Lima V.D.
        • et al.
        Cost-effectiveness of population-level expansion of highly active antiretroviral treatment for HIV in British Columbia, Canada: a modelling study.
        Lancet HIV. 2015; 2: e393-e400
        • Briggs A.H.
        • Weinstein M.C.
        • Fenwick E.A.
        • et al.
        Model parameter estimation and uncertainty: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-6.
        Value Health. 2012; 15: 835-842
        • Goldhaber-Fiebert J.D.
        • Jalal H.J.
        Some health states are better than others: using health state rank order to improve probabilistic analyses.
        Med Decis Making. 2016; 36: 927-940
        • Oostenbrink J.B.
        • Al M.J.
        • Oppe M.
        • et al.
        Expected value of perfect information: an empirical example of reducing decision uncertainty by conducting additional research.
        Value Health. 2008; 11: 1070-1080
        • Jalal H.
        • Alarid-Escudero F.
        A Gaussian approximation approach for value of information analysis.
        Med Decis Making. 2018; 38: 174-188
        • Alam M.F.
        • Briggs A.
        Artificial neural network metamodel for sensitivity analysis in a total hip replacement health economic model.
        Expert Rev Pharmacoecon Outcomes Res. 2019; : 1-12
        • Fonseca D.J.
        • Navaresse D.O.
        • Moynihan G.P.
        Simulation metamodeling through artificial neural networks.
        Eng Appl Artif Intell. 2003; 16: 177-183
        • Wood S.
        mgcv: Mixed GAM computation vehicle with automatic smoothness estimation. R package version 1.8-31.
        https://CRAN.R-project.org/package=keras
        Date accessed: March 10, 2020
        • Falbel D.
        • Allaire J.
        • Chollet F.
        keras: R interface to ‘Keras’. R package version 2.2.5.0.
        https://CRAN.R-project.org/package=keras
        Date accessed: March 25, 2020
        • Cambiano V.
        • Miners A.
        • Dunn D.
        • et al.
        Cost-effectiveness of pre-exposure prophylaxis for HIV prevention in men who have sex with men in the UK: a modelling study and health economic evaluation.
        Lancet Infect Dis. 2018; 18: 85-94
        • Shafer L.A.
        • Nsubuga R.N.
        • Chapman R.
        • et al.
        The dual impact of antiretroviral therapy and sexual behaviour changes on HIV epidemiologic trends in Uganda: a modelling study.
        Sex Transm Infect. 2014; 90: 423-429
        • Hallett T.B.
        • Gregson S.
        • Mugurungi O.
        • et al.
        Assessing evidence for behaviour change affecting the course of HIV epidemics: a new mathematical modelling approach and application to data from Zimbabwe.
        Epidemics. 2009; 1: 108-117
      1. Centers for Disease Control and Prevention. HIV risk, prevention, and testing behaviors - National HIV Behavioral Surveillance System: men who have sex with men, 20 US cities, 20 HIV Special Report 8: Centers for Disease Control and Prevention. 2014.

      2. Centers for Disease Control and Prevention. Public use data file documentation. 2011-2013. National Survey of Family Growth. User’s guide. Hyattsville, MD: Centers for Disease Control and Prevention, National Center for Health Science; 2014.

        • Zaba B.
        • Slaymaker E.
        • Urassa M.
        • et al.
        The role of behavioral data in HIV surveillance.
        AIDS. 2005; 19: S39-S52
        • Wang L.
        • Moqueet N.
        • Lambert G.
        • et al.
        Population-level sexual mixing by HIV status and pre-exposure prophylaxis use among men who have sex with men in Montreal, Canada: implications for HIV prevention.
        Am J Epidemiol. 2020; 189: 44-54
        • Volz E.
        • Heckathorn D.D.
        Probability based estimation theory for respondent driven sampling.
        J Off Stat. 2008; 24: 79
        • Centers for Disease Control and Prevention
        HIV infection, Risk, prevention, and testing behaviors among persons who inject drugs—national HIV behavioral surveillance: injection drug use, 20 U.S. cities, 2012. HIV Surveillance Special Report 11. Revised edition.
        • Centers for Disease Control and Prevention
        HIV infection risk, prevention, and testing behaviors among men who have sex with men—national HIV behavioral surveillance, 20 U.S. cities, 2014. HIV Surveillance Special Report 15.
        • Centers for Disease Control and Prevention
        Behavioral Risk Factor Surveillance System Survey Data.
        Centers for Disease Control and Prevention, Atlanta, GA2014
        • The HIV Research Network (HIVRN)
        Goals of the HIV Research Network.
        https://cds.johnshopkins.edu/hivrn/
        Date accessed: August 15, 2017
        • Wang L.
        • Krebs E.
        • Min J.E.
        • et al.
        Combined estimation of disease progression and retention on antiretroviral therapy among treated individuals with HIV in the USA: a modelling study.
        Lancet HIV. 2019; 6: e531-e539
        • Richards C.L.
        • Iademarco M.F.
        • Atkinson D.
        • et al.
        Advances in public health surveillance and information dissemination at the Centers for Disease Control and Prevention.
        Public Health Rep. 2017; 132: 403-410
        • Sharfstein J.M.
        Using healthcare data to track and improve public health.
        JAMA. 2015; 313: 2012-2013
        • Fojo A.T.
        • Dowdy D.W.
        Ending the human immunodeficiency virus epidemic: towards an evidence-based approach.
        Clin Infect Dis. 2019; 69: 2199-2200