Advertisement

The Use of Cost-Effectiveness Thresholds for Evaluating Health Interventions in Low- and Middle-Income Countries From 2015 to 2020: A Review

  • Joseph Kazibwe
    Affiliations
    Global Health and Development Group, School of Public Health, Imperial College London, Norfolk Place, London, England, UK

    International Decision Support Initiative, Center for Global Development, London, England, UK

    MRC Centre for Global Infectious Disease Analysis and the Abdul Latif Jameel Institute for Disease and Emergency Analytics, School of Public Health, Imperial College London, England, UK
    Search for articles by this author
  • Adrian Gheorghe
    Affiliations
    Global Health and Development Group, School of Public Health, Imperial College London, Norfolk Place, London, England, UK

    International Decision Support Initiative, Center for Global Development, London, England, UK

    MRC Centre for Global Infectious Disease Analysis and the Abdul Latif Jameel Institute for Disease and Emergency Analytics, School of Public Health, Imperial College London, England, UK
    Search for articles by this author
  • David Wilson
    Affiliations
    Bill & Melinda Gates Foundation, London, England, UK
    Search for articles by this author
  • Francis Ruiz
    Affiliations
    Global Health and Development Group, School of Public Health, Imperial College London, Norfolk Place, London, England, UK

    International Decision Support Initiative, Center for Global Development, London, England, UK

    MRC Centre for Global Infectious Disease Analysis and the Abdul Latif Jameel Institute for Disease and Emergency Analytics, School of Public Health, Imperial College London, England, UK
    Search for articles by this author
  • Kalipso Chalkidou
    Affiliations
    Global Health and Development Group, School of Public Health, Imperial College London, Norfolk Place, London, England, UK

    International Decision Support Initiative, Center for Global Development, London, England, UK

    MRC Centre for Global Infectious Disease Analysis and the Abdul Latif Jameel Institute for Disease and Emergency Analytics, School of Public Health, Imperial College London, England, UK
    Search for articles by this author
  • Y-Ling Chi
    Correspondence
    Correspondence: Y-Ling Chi, PhD, International Decision Support Initiative, Center for Global Development, London, England, United Kingdom SW1P 3SE.
    Affiliations
    International Decision Support Initiative, Center for Global Development, London, England, UK
    Search for articles by this author
Open AccessPublished:October 28, 2021DOI:https://doi.org/10.1016/j.jval.2021.08.014

      Highlights

      • The assessment of cost-effectiveness of interventions ex ante can ensure that cost-effective interventions that highly benefit population health are consistently prioritized over interventions that offer comparatively less value.
      • Cost-effectiveness thresholds (CETs) are essential in determining whether an intervention under consideration is cost-effective or not; nevertheless, there is no international consensus on how they should be determined by countries.
      • The World Health Organization has distanced itself from the use of 1 to 3× gross domestic product (GDP) per capita per disability-adjusted life-year averted (earlier recommended by World Health Organization-CHOICE) as a CET but they remain the most popular and commonly used thresholds.
      • More than a third of studies apply the 1 to 3× GDP per capita CET use 3× GDP per capita per disability-adjusted life-year averted yet 3× GDP per capita is a lot higher than other CET.
      • Using 1 to 3× GDP per capita as a CET has resulted in a higher proportion of study interventions being found as cost-effective in comparison with studies where other types of CET are used. This may lead to a loss of would-be health gains from deployed resources for health.
      • There is an urgent need to move away from 1 to 3× GDP per capita as a CET and a more appropriate replacement should be found.

      Abstract

      Objectives

      Evidence-informed priority setting, in particular cost-effectiveness analysis (CEA), can help target resources better to achieve universal health coverage. Central to the application of CEA is the use of a cost-effectiveness threshold. We add to the literature by looking at what thresholds have been used in published CEA and the proportion of interventions found to be cost-effective, by type of threshold.

      Methods

      We identified CEA studies in low- and middle-income countries from the Global Health Cost-Effectiveness Analysis Registry that were published between January 1, 2015, and January 6, 2020. We extracted data on the country of focus, type of interventions under consideration, funder, threshold used, and recommendations.

      Results

      A total of 230 studies with a total 713 interventions were included in this review; 1 to 3× gross domestic product (GDP) per capita was the most common type of threshold used in judging cost-effectiveness (84.3%). Approximately a third of studies (34.2%) using 1 to 3× GDP per capita applied a threshold at 3× GDP per capita. We have found that no study used locally developed thresholds. We found that 79.3% of interventions received a recommendation as “cost-effective” and that 85.9% of studies had at least 1 intervention that was considered cost-effective. The use of 1 to 3× GDP per capita led to a higher proportion of study interventions being judged as cost-effective compared with other types of thresholds.

      Conclusions

      Despite the wide concerns about the use of 1 to 3× GDP per capita, this threshold is still widely used in the literature. Using this threshold leads to more interventions being recommended as “cost-effective.” This study further explore alternatives to the 1 to 3× GDP as a decision rule.

      Keywords

      Introduction

      The regular assessment of the cost-effectiveness of interventions ex ante can ensure that cost-effective interventions that highly benefit population health are consistently prioritized over interventions that offer comparatively less value in coverage decisions.
      Strategizing national health in the 21st century: a handbook chapter 4: priority-setting for national health policies, strategies and plans. World Health Organization.
      Cost-effectiveness analyses (CEAs) compare the relative costs and health gains (eg, measured through disability-adjusted life-years [DALYs] or quality-adjusted life-years) from the introduction of an intervention. Central to CEA is the incremental cost-effectiveness ratio (ICER), which summarizes the expected incremental costs to the incremental benefits of a given intervention or program compared with an alternative (eg, standard/usual care).
      • Leech A.A.
      • Kim D.D.
      • Cohen J.T.
      • Neumann P.J.
      Use and misuse of cost-effectiveness analysis thresholds in low- and middle-income countries: trends in cost-per-DALY studies.
      ICERs can be compared with a decision rule called the cost-effectiveness threshold (CET). If the ICER is below the CET, then the intervention or program under consideration is deemed cost-effective, and if it is above the threshold, then the intervention or program is regarded as not cost-effective. A report by the World Health Organization (WHO) indicates that critically assessing interventions and selecting optimal intervention mix in countries could lead to achieving the same health gains with between 16% and 99% of current spending.
      The world health report: Health systems financing the path to universal coverage. World Health Organization.
      There is no global CET or consensus on how countries should develop their CET to inform the utilization of public/pooled resources for health (see Appendix 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.08.014 for information on methods).
      • Chi Y.L.
      • Blecher M.
      • Chalkidou K.
      • et al.
      What next after GDP-based cost-effectiveness thresholds?.
      Ideally, CETs should be contextualized to local health constraints, in particular with regard to available resources or population preferences. For instance, what defines a cost-effective service in the United-States (where public spending on health per capita in 2019 was $9386
      Government at a glance. OCED iLibrary.
      ) will be very different from what may be cost-effective in Uganda (where public spending on health per capita in 2017 was $24
      Domestic general government health expenditure per capita, PPP (current international $) – Uganda. World Bank.
      ). Countries with a local CET include Ireland and England,
      • Santos A.S.
      • Guerra-Junior A.A.
      • Godman B.
      • Morton A.
      • Ruas C.M.
      Cost-effectiveness thresholds: methods for definition and cases around the world.
      but with the exception of a few (eg, Thailand
      • Thavorncharoensap M.
      • Teerawattananon Y.
      • Natanant S.
      • Kulpeng W.
      • Yothasamut J.
      • Werayingyong P.
      Estimating the willingness to pay for a quality- adjusted life year in Thailand: does the context of health gain matter?.
      ), local thresholds are rare in low- and middle-income countries (LMICs). Because of this gap, CETs that have been used internationally such as the 1 to 3× gross domestic product (GDP) per capita rule have grown in importance, both in research and for consideration by public payers.
      For the good part of the last decade, GDP-based thresholds have been considered unrealistic because they were based on flawed technical evidence
      • Robinson L.A.
      • Hammitt J.K.
      • Chang A.Y.
      • Resch S.
      Understanding and improving the one and 3 times GDP per capita cost-effectiveness thresholds.
      in addition to being completely disconnected with health opportunity costs or local budgets constraints.
      • Meyer-Rath G.
      • van Rensburg C.
      • Larson B.
      • Jamieson L.
      • Rosen S.
      Revealed willingness-to-pay versus standard cost-effectiveness thresholds: evidence from the South African HIV Investment Case.
      ,
      • Bertram M.Y.
      • Lauer J.A.
      • De Joncheere K.
      • et al.
      Cost-effectiveness thresholds: pros and cons.
      One big concern is that GDP-based thresholds are too easily attainable
      • Woods B.
      • Revill P.
      • Sculpher M.
      • Claxton K.
      Country-level cost-effectiveness thresholds: initial estimates and the need for further research.
      and therefore push for the inclusion of interventions that may not be affordable (or cost-effective from perspective of health opportunity costs).
      • Loganathan T.
      • Ng C.W.
      • Lee W.S.
      • Hutubessy R.C.W.
      • Verguet S.
      • Jit M.
      Thresholds for decision-making: informing the cost-effectiveness and affordability of rotavirus vaccines in Malaysia.
      This may lead to misallocation of resources and cause losses in health.
      • Robinson L.A.
      • Hammitt J.K.
      • Chang A.Y.
      • Resch S.
      Understanding and improving the one and 3 times GDP per capita cost-effectiveness thresholds.
      It is worth noting that, despite those increasing concerns (including from staff at the WHO), the 1 to 3× GDP threshold is still in use in WHO-CHOICE. A review conducted by Leech et al
      • Leech A.A.
      • Kim D.D.
      • Cohen J.T.
      • Neumann P.J.
      Use and misuse of cost-effectiveness analysis thresholds in low- and middle-income countries: trends in cost-per-DALY studies.
      found that 66% of the 381 CEAs published in LMICs between 2000 and 2015 use GDP-based thresholds, with an increasing trend over that period (23% did not report any threshold). Nevertheless, Leech et al
      • Leech A.A.
      • Kim D.D.
      • Cohen J.T.
      • Neumann P.J.
      Use and misuse of cost-effectiveness analysis thresholds in low- and middle-income countries: trends in cost-per-DALY studies.
      looked at the broad categories of GDP-based thresholds, so it is not clear what the exact GDP rule used in the studies they considered is. Specifying the rule clearly is important, because, by definition, 1 to 3× GDP per capita is a 3-fold range.
      The aim of this study is 2-fold. First, we seek to document what CETs have been used in published CEAs in detail (eg, whether it is 0.5, 1, 2, or 3× GDP per capita) and, second, how that correlated with the recommendations made by the authors (cost-effective/not cost-effective). Moreover, the period adopted in this review goes beyond the timeframe covered by Leech et al,
      • Leech A.A.
      • Kim D.D.
      • Cohen J.T.
      • Neumann P.J.
      Use and misuse of cost-effectiveness analysis thresholds in low- and middle-income countries: trends in cost-per-DALY studies.
      which will allow us to compare our results to trends identified by their study, in particular after the growing critique of the GDP-based CET around the cutoff date (Marseille et al
      • Marseille E.
      • Larson B.
      • Kazi D.S.
      • Kahn J.G.
      • Rosen S.
      Thresholds for the cost–effectiveness of interventions: alternative approaches.
      and Bertram et al
      • Bertram M.Y.
      • Lauer J.A.
      • De Joncheere K.
      • et al.
      Cost-effectiveness thresholds: pros and cons.
      ).

      Methods

      This review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.
      • Moher D.
      • Liberati A.
      • Tetzlaff J.
      • Altman D.G.
      The PRISMA Group
      Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
      This review relies on the Global Health Cost-Effectiveness Analysis (GH CEA) Registry managed by the Center for the Evaluation of Value and Risk in Health at Tufts Medical Center.
      About. Center for the Evaluation of Value and Risk in Health.
      The registry is a repository of all studies in English language majorly using the “cost-per-DALY averted” metric to measure the efficacy of health interventions around the globe.
      GH CEA registry. Center for the Evaluation of Value and Risk in Health.
      ,
      • Neumann P.J.
      • Thorat T.
      • Zhong Y.
      • et al.
      A systematic review of cost-effectiveness studies reporting cost-per-DALY averted.
      Therefore, it was considered because of its comprehensiveness
      GH CEA registry. Center for the Evaluation of Value and Risk in Health.
      and coverage of CEAs in LMICs. We searched the GH CEA Registry on January 6, 2020, for studies published since January 1, 2015. All the studies (indexed in PubMed) that came up were extracted in an excel file and reviewed for inclusion into the study. The GH CEA Registry is organized around interventions, not studies.
      Studies that met the following criteria were included: (1) presents cost-effectiveness findings measured in costs per effect outcome such as DALYs and (2) has been conducted within/for a LMIC as categorized by the World Bank as of January 6, 2020. The description of the threshold used and recommendation was extracted manually. In this study, we define a recommendation as the conclusion drawn about cost-effectiveness of an intervention following the use of a given threshold, typically after the “Results” section.

      Results

      A total of 232 cost-effectiveness studies published in respective journals between January 1, 2015, and January 6, 2020, were identified from the GH CEA database. Using the inclusion and exclusion criteria, 2 studies were excluded because they focused on high income countries. A total of 230 studies were included in the review and underwent full-text reading and data extraction.

       Characteristics of Studies

      The summary of characteristics of the studies that have been assessed and analyzed in this study is presented in Table 1. Both the studies (N = 230) and their respective interventions (N = 713) are considered as most studies looked at >1 intervention.
      Table 1Characteristics of the studies and interventions.
      Number of studies (N = 230)FrequencyPercentage
      Studies by number of countries studied
       One country18480.0
       Multiple countries4620.0
      Regional distribution of studies
       Sub-Saharan Africa9039.1
       Multiple regions4117.8
       South Asia3515.2
       East Asia and Pacific3113.5
       Latin America and Caribbean208.7
       Middle East and North Africa93.9
       Europe and Central Asia41.7
      Funding body/institution by study
       Foundation8034.8
       Government6026.1
       Not mentioned4620.0
       None167.0
       Academic institution104.4
       Professional member organization52.2
       Manufacturer41.7
       Healthcare organization31.3
       Others62.6
      Disease area
       HIV3113.5
       TB and malaria2812.2
       Maternal child health2510.9
       NCDs except cancer and mental health219.1
       Cancer62.6
       Mental health31.3
       Other communicable diseases7733.5
       Others
      Others included physical disabilities, agronomic biofortification, and nondisease policy areas such as health financing mechanisms.
      3917.0
      Types of intervention
       Immunization6026.1
       Pharmaceutical5423.5
       Screening3515.2
       Public health intervention excluding screening and immunization167.0
       Surgical146.1
       Others
      Others included voucher systems, medical devices, taxation, expansion of access to a given service, aquaculture, and health financing mechanisms.
      5122.2
      HIV indicates human immunodeficiency virus; NCD, noncommunicable disease; TB, tuberculosis.
      Others included physical disabilities, agronomic biofortification, and nondisease policy areas such as health financing mechanisms.
      Others included voucher systems, medical devices, taxation, expansion of access to a given service, aquaculture, and health financing mechanisms.
      Most studies were single-country studies (80%). Most studies were conducted in the Sub-Saharan African region (39.1%), followed by South Asia (15.2%). Communicable diseases (eg, tuberculosis, malaria, human immunodeficiency virus) were the focus of most studies (59.1%).
      Foundations, such as the Bill and Melinda Gates Foundation, were cited as the funding source most frequently (34.8%), followed by governments (26.1%). A considerable number of studies (20.0%) did not mention/report the source of funding, and 7.0% of the studies reported not having received any funding.

       Thresholds in Use

      We found several CETs in use that can broadly be classified as (1) GDP based, (2) willingness to pay (WTP) or “demand side” thresholds, and (3) opportunity cost or “supply side” thresholds (see Box 1 for definitions).
      ▪▪▪
      Thresholds in use
      In this review, the 0.5× GDP per capita was classified as an opportunity cost CET. All authors that used 0.5× GDP per capita referred to opportunity cost threshold estimates and were thus categorized as opportunity cost thresholds in this paper. A total of 3 studies use opportunity cost thresholds. For instance, Olney et al
      The world health report: Health systems financing the path to universal coverage. World Health Organization.
      estimate the CEA of different strategies used to improve HIV care outcomes along the cascade of care in Kenya and use 0.5× GDP as the threshold. Opportunity cost-based thresholds are often based on an estimate of the marginal productivity of the healthcare system that describe the amount of money in the present health system, needed to produce a unit of health due to the resource constraint. Two cross-country estimates from Woods et al
      • Woods B.
      • Revill P.
      • Sculpher M.
      • Claxton K.
      Country-level cost-effectiveness thresholds: initial estimates and the need for further research.
      (2016) and Ochalek et al
      • Ochalek J.
      • Lomas J.
      • Claxton K.
      Estimating health opportunity costs in low-income and middle-income countries: a novel approach and evidence from cross-country data.
      (2018) have found that estimates average 0.5 GDP per capita, which is why this threshold is still defined against GDP per capita.
      GDP-based thresholds refer to the 1 to 3× GDP per capita rule that is commonly referred to as WHO recommended threshold. If the ICER of an intervention is equal to or less than 1× GDP per capita per DALY averted, then the intervention is deemed highly cost effective and if the ICER is between 1 and 3× GDP per capita per DALY averted then the intervention is considered cost effective. Above 3× GDP per capita, the intervention is considered not cost effective.
      Another method often referenced is the WTP CET, which is based on aspirational values, representing a specific view of what value ought to be placed on a unit improvement in health. It can be estimated using preference data collected through a WTP survey from the population to elicit the average maximum amount of money individuals are willing to pay for a unit of health gain.
      CET indicates cost-effectiveness threshold; DALY, disability-adjusted life-year; GDP, gross domestic product; ICER, incremental cost-effectiveness ratio; WHO, World Health Organization; WTP, willingness to pay.

       Thresholds Used by Intervention and Study

      GDP-based CETs are the most commonly (84.3% of studies) used CET decision rules in CEA studies in LMICs in the period studied. As noted in Box 1, this category includes only thresholds within the range of 1 to 3× GDP per capita. Notably, 3 studies use opportunity cost thresholds; 7.3% of studies did not state a rule. Others (0.9% studies) include articles that used regional thresholds (Price et al
      • Price J.T.
      • Wheeler S.B.
      • Stranix-Chibanda L.
      • et al.
      Cost-effectiveness of pre-exposure HIV prophylaxis during pregnancy and breastfeeding in Sub-Saharan Africa.
      [2016]) and comparison of regional ICERs (Kaucley and Levy
      • Kaucley L.
      • Levy P.
      Cost-effectiveness analysis of routine immunization and supplementary immunization activity for measles in a health district of Benin.
      [2015]). Interestingly, we have found no study using WTP thresholds.
      Additionally, we have found that some studies (eg, Nayagam et al
      • Nayagam S.
      • Conteh L.
      • Sicuri E.
      • et al.
      Cost-effectiveness of community-based screening and treatment for chronic hepatitis B in the Gambia: an economic modelling analysis.
      [2016]) used multiple thresholds (GDP-based threshold and $240 per DALY).
      • Nayagam S.
      • Conteh L.
      • Sicuri E.
      • et al.
      Cost-effectiveness of community-based screening and treatment for chronic hepatitis B in the Gambia: an economic modelling analysis.
      It is worth noting that not all studies need a CET. In this review, 14 studies found the interventions under consideration cost saving, dominated, or dominant. Please see Appendix 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.08.014 for the table illustrating CET used by intervention and study.

       GDP-Based Thresholds

      We further explored what GDP-based threshold was used (see Appendix 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.08.014).
      Moreover, 65.4% of studies used 1× GDP per capita, and 3× GDP per capita was used by more than a third of studies (34.2%).

       Recommendations on Cost-Effectiveness

      Here, we cover the recommendation formulated by authors of the respective studies about cost-effectiveness. We show 2 main values: by intervention and by study (we look at whether a given study finds least 1 cost-effective intervention when several interventions are considered), 79.3% of interventions were found cost-effective. This rate was higher when considering studies rather than intervention: 85.2% of studies reported that at least one of the interventions under consideration was cost-effective.
      Recommendations about cost-effectiveness were not stated in 11.6% of studies. For instance, Tolla et al
      • Tolla M.T.
      • Norheim O.F.
      • Memirie S.T.
      • et al.
      Prevention and treatment of cardiovascular disease in Ethiopia: a cost-effectiveness analysis.
      (2016) highlighted the controversy surrounding the use of 1 to 3× GDP per capita, and on that basis, the authors decided not to issue a recommendation for cost-effectiveness. Only considering studies where a CET was applied and a recommendation was made (ie, excluding “not stated”), the proportion of interventions that were found to be cost-effective was 89.7% (see Appendix 2 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.08.014).

       Recommendations on Cost-Effectiveness, by Type of Threshold Used

      The recommendations on cost-effectiveness, by type of threshold used, are presented in Table 2.
      Table 2Recommendation on cost-effectiveness, by type of threshold.
      By interventionRecommendationGDP-based threshold (n = 522)Non-GDP-based threshold (n = 115)
      FrequencyPercentageFrequencyPercentage
      CE47290.43328.7
      Not CE509.686.9
      Not stated007464.4
      Total522100115100
      By studiesRecommendationGDP-based threshold (n = 194)Non-GDP-based threshold (n = 36)
      FrequencyPercentageFrequencyPercentage
      At least 1 CE intervention18394.31336.1
      No CE intervention115.72363.9
      Total19410036100
      Note. Intervention section: only includes interventions whose CE recommendation was clearly stated. Interventions that were reported as dominant, dominated, or cost saving were excluded from the table above. Non-GDP-based threshold included opportunity cost CET and also included interventions where the CET was not stated.
      CE indicates cost-effective; CET, cost-effectiveness threshold; GDP, gross domestic product.
      We find that only 36.1% of studies using a non-GDP-based threshold reported at least 1 cost-effective intervention, versus 94.3% of studies using a GDP-based threshold. The proportion of studies not stating a recommendation was high in the non-GDP-based threshold category: 64.4% of studies.

      Discussion

      The use of a CET is aligned with the goal of maximizing health in a budget constrained setting.
      • Culyer A.
      Cost-effectiveness thresholds in health care: a bookshelf guide to their meaning and use. Centre for Health Economics, University of York.
      By using thresholds that seem to be set too high and reject a small proportion of interventions, the probability of allocating resources to interventions that are not good value for money increases, resulting in displacement of health and net health losses to the society.
      • Culyer A.
      Cost-effectiveness thresholds in health care: a bookshelf guide to their meaning and use. Centre for Health Economics, University of York.
      This review reveals that several CET approaches are in use in LMICs, including GDP-based thresholds and opportunity cost-based thresholds. GDP-based thresholds remain the most frequently used CET in CE studies in LIMCs (84.3%), and we even observe an upward trend compared with the Leech et al
      • Leech A.A.
      • Kim D.D.
      • Cohen J.T.
      • Neumann P.J.
      Use and misuse of cost-effectiveness analysis thresholds in low- and middle-income countries: trends in cost-per-DALY studies.
      review (66% from 2000 to 2015).
      This is a surprising result given the numerous criticisms of the 1 to 3× GDP per capita thresholds,
      • Robinson L.A.
      • Hammitt J.K.
      • Chang A.Y.
      • Resch S.
      Understanding and improving the one and 3 times GDP per capita cost-effectiveness thresholds.
      ,
      • Meyer-Rath G.
      • van Rensburg C.
      • Larson B.
      • Jamieson L.
      • Rosen S.
      Revealed willingness-to-pay versus standard cost-effectiveness thresholds: evidence from the South African HIV Investment Case.
      including recent warnings from WHO staff.
      • Bertram M.Y.
      • Lauer J.A.
      • De Joncheere K.
      • et al.
      Cost-effectiveness thresholds: pros and cons.
      ,
      • Marseille E.
      • Larson B.
      • Kazi D.S.
      • Kahn J.G.
      • Rosen S.
      Thresholds for the cost–effectiveness of interventions: alternative approaches.
      This study finds that those criticisms seem to have had limited impact in discouraging their use, at least in published research. We found no record of local thresholds in our review. To further compound matters, we have found that in some countries where a local threshold exists, the GDP-based thresholds superseded the local rule. For example, Suraratdecha et al
      • Suraratdecha C.
      • Stuart R.M.
      • Manopaiboon C.
      • et al.
      Cost and cost-effectiveness analysis of pre-exposure prophylaxis among men who have sex with men in two hospitals in Thailand.
      (2018) used 3× GDP per capita as a threshold in Thailand, despite the fact that there is an established threshold in the country (equivalent to 0.3× GDP per capita and 1.5× GDP per capita depending on severity).
      Another surprising result is that 34.2% of studies applying GDP-based thresholds use 3× GDP per capita. Studies have criticized the use of 1 GDP per capita threshold in LMICs as being “too high” and 1 to 3× GDP per capita for having a wide cost-effectiveness range.
      • Robinson L.A.
      • Hammitt J.K.
      • Chang A.Y.
      • Resch S.
      Understanding and improving the one and 3 times GDP per capita cost-effectiveness thresholds.
      ,
      • Woods B.
      • Revill P.
      • Sculpher M.
      • Claxton K.
      Country-level cost-effectiveness thresholds: initial estimates and the need for further research.
      As an illustration, for Kenya, the difference in cutoff between those 2 rules would mean applying 1816.5 or 5449.5 US dollar (based on 2019 GDP per capita) per DALY averted as a CET.
      GDP per capita (current US$) - Kenya. The World Bank.
      If we consider estimates produced by Woods et al
      • Woods B.
      • Revill P.
      • Sculpher M.
      • Claxton K.
      Country-level cost-effectiveness thresholds: initial estimates and the need for further research.
      and Ochalek et al
      • Ochalek J.
      • Lomas J.
      • Claxton K.
      Estimating health opportunity costs in low-income and middle-income countries: a novel approach and evidence from cross-country data.
      (2018) on health opportunity costs (which seem to average to 0.5× GDP per capita for every DALY averted), then it would suggest including introducing interventions that avert a DALY at 3× GDP per capita is likely to displace up to 5 times more health than the intervention creates. In other words, if an intervention averts a single DALY at 3× GDP per capita and the true cost of averting a DALY in the country is equal to approximately 0.5× GDP per capita, then the introduction of that intervention would in principle lead to a net loss of health equivalent to 5 DALYs averted.
      Overall, we find that a high proportion of interventions (79.3%) were found to be cost-effective, and 85.2% of studies had at least 1 cost-effective intervention. There is no objective metric for assessing whether the proportion of negative recommendations found in this study is “too low.” Nevertheless, given the significant budget constraints, for example, in some Sub-Saharan African countries (where 39.1% of studies were conducted), we would postulate that this proportion of negative recommendations is too low, and therefore, the application of the 1 to 3× GDP per capita (without consideration of other evidence) may not enable decision makers to discriminate effectively between cost-effective and not cost-effective interventions. It is worth noting that factors other than the CET may lead to low proportion of negative recommendations (eg, publication bias); nevertheless, we were not able to investigate those in this study.
      The reasons for the continued use of 1 to 3× GDP per capita as a threshold cannot be confirmed in this review; nevertheless, it should be noted that if ease of use (eg, the fact that it is a ready to use, off the shelf solution) makes it attractive, then the opportunity cost threshold (or 0.5× GDP per capita) is also similarly easy to use. We recommend, at the very least, that researchers include a justification and appropriate caveating when using 1 to 3× GDP per capita. Moreover, whenever a local threshold exists, there is no reason why it should not be used by study authors. It essential for CEA publications to have a clear and transparent use of appropriate thresholds and recommendations to allow such evidence to easily be used by the decision makers. Further research should be conducted to determine how decision makers use the CEA recommendation.

       Policy Implications

      Together, these findings reinforce the view that 1 to 3× GDP thresholds are not a good rule for making recommendations about cost-effectiveness. Researchers should prioritize using local thresholds when available or explore alternatives to this rule. Funders and development agencies should consider discouraging its use in the future.

       Study Limitations

      This article focused on studies that had reported DALYs; therefore, CETs based on quality-adjusted life-years or other health-related outcome measures were not included in this study. In addition, only 1 database was considered in identifying articles for inclusion. Although the GH CEA database is known for its comprehensiveness, there may be studies that are missed out because they were not uploaded onto the GH CEA database. Manual data recording (eg, on the threshold use or the recommendations) may have been prone to random errors. We only considered the use of CETs in published, English-language, research articles.
      In addition, we were not able to review the use of thresholds by decision makers when making coverage decisions, because those are not always documented and may be difficult to identify.

      Article and Author Information

      Author Contributions: Concept and design: Kazibwe, Gheorghe, Ruiz, Chalkidou, Chi
      Acquisition of data: Kazibwe
      Analysis and interpretation of data: Kazibwe, Gheorghe, Wilson, Chi
      Drafting of the manuscript: Kazibwe, Wilson, Ruiz, Chi
      Critical revision of the paper for important intellectual content: Kazibwe, Gheorghe, Wilson, Ruiz, Chalkidou, Chi
      Statistical analysis: Kazibwe, Chi
      Obtaining funding: Chalkidou
      Administrative, technical, or logistic support: Kazibwe, Chi
      Supervision: Ruiz, Chi
      Conflict of Interest Disclosures: Dr Wilson is employed by the Bill & Melinda Gates Foundation, which includes funding work in health economic evaluations. Dr Chalkidou is an editor for Value in Health and had no role in the peer review process of this article. No other disclosures were reported.
      Funding/Support: This study was funded by the Bill & Melinda Gates Foundation through the International Decision Support Initiative Plus grant (grant number: INV-006987).
      Role of the Funder/Sponsor: The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

      Supplemental Material

      References

      1. Strategizing national health in the 21st century: a handbook chapter 4: priority-setting for national health policies, strategies and plans. World Health Organization.
        • Leech A.A.
        • Kim D.D.
        • Cohen J.T.
        • Neumann P.J.
        Use and misuse of cost-effectiveness analysis thresholds in low- and middle-income countries: trends in cost-per-DALY studies.
        Value Heal. 2018; 21: 759-761
      2. The world health report: Health systems financing the path to universal coverage. World Health Organization.
        • Chi Y.L.
        • Blecher M.
        • Chalkidou K.
        • et al.
        What next after GDP-based cost-effectiveness thresholds?.
        Gates Open Res. 2020; 4: 176
      3. Government at a glance. OCED iLibrary.
      4. Domestic general government health expenditure per capita, PPP (current international $) – Uganda. World Bank.
        • Santos A.S.
        • Guerra-Junior A.A.
        • Godman B.
        • Morton A.
        • Ruas C.M.
        Cost-effectiveness thresholds: methods for definition and cases around the world.
        Expert Rev Pharmacoecon Outcomes Res. 2018; 18: 277-288
        • Thavorncharoensap M.
        • Teerawattananon Y.
        • Natanant S.
        • Kulpeng W.
        • Yothasamut J.
        • Werayingyong P.
        Estimating the willingness to pay for a quality- adjusted life year in Thailand: does the context of health gain matter?.
        Clinicoecon Outcomes Res. 2013; 5: 29-36
        • Robinson L.A.
        • Hammitt J.K.
        • Chang A.Y.
        • Resch S.
        Understanding and improving the one and 3 times GDP per capita cost-effectiveness thresholds.
        Health Policy Plan. 2017; 32: 141-145
        • Meyer-Rath G.
        • van Rensburg C.
        • Larson B.
        • Jamieson L.
        • Rosen S.
        Revealed willingness-to-pay versus standard cost-effectiveness thresholds: evidence from the South African HIV Investment Case.
        PLoS One. 2017; 12e0186496
        • Bertram M.Y.
        • Lauer J.A.
        • De Joncheere K.
        • et al.
        Cost-effectiveness thresholds: pros and cons.
        Bull World Health Organ. 2016; 94: 925-930
        • Woods B.
        • Revill P.
        • Sculpher M.
        • Claxton K.
        Country-level cost-effectiveness thresholds: initial estimates and the need for further research.
        Value Heal. 2016; 19: 929-935
        • Loganathan T.
        • Ng C.W.
        • Lee W.S.
        • Hutubessy R.C.W.
        • Verguet S.
        • Jit M.
        Thresholds for decision-making: informing the cost-effectiveness and affordability of rotavirus vaccines in Malaysia.
        Heal Policy Plan. 2018; 33: 204-214
        • Marseille E.
        • Larson B.
        • Kazi D.S.
        • Kahn J.G.
        • Rosen S.
        Thresholds for the cost–effectiveness of interventions: alternative approaches.
        Bull World Health Organ. 2015; 93: 118-124
        • Moher D.
        • Liberati A.
        • Tetzlaff J.
        • Altman D.G.
        • The PRISMA Group
        Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
        PLoS Med. 2009; 6: e1000097
      5. About. Center for the Evaluation of Value and Risk in Health.
        https://cevr.tuftsmedicalcenter.org/about
        Date accessed: July 20, 2020
      6. GH CEA registry. Center for the Evaluation of Value and Risk in Health.
        • Neumann P.J.
        • Thorat T.
        • Zhong Y.
        • et al.
        A systematic review of cost-effectiveness studies reporting cost-per-DALY averted.
        PLoS One. 2016; 11e0168512
        • Price J.T.
        • Wheeler S.B.
        • Stranix-Chibanda L.
        • et al.
        Cost-effectiveness of pre-exposure HIV prophylaxis during pregnancy and breastfeeding in Sub-Saharan Africa.
        J Acquir Immune Defic Syndr. 2016; 72: S145-S153
        • Kaucley L.
        • Levy P.
        Cost-effectiveness analysis of routine immunization and supplementary immunization activity for measles in a health district of Benin.
        Cost Eff Resour Alloc. 2015; 13: 14
        • Nayagam S.
        • Conteh L.
        • Sicuri E.
        • et al.
        Cost-effectiveness of community-based screening and treatment for chronic hepatitis B in the Gambia: an economic modelling analysis.
        Lancet Glob Health. 2016; 4: e568-e578
        • Tolla M.T.
        • Norheim O.F.
        • Memirie S.T.
        • et al.
        Prevention and treatment of cardiovascular disease in Ethiopia: a cost-effectiveness analysis.
        Cost Eff Resour Alloc. 2016; 14: 1-14
        • Culyer A.
        Cost-effectiveness thresholds in health care: a bookshelf guide to their meaning and use. Centre for Health Economics, University of York.
        http://eprints.whiterose.ac.uk/135882/
        Date accessed: August 15, 2020
        • Suraratdecha C.
        • Stuart R.M.
        • Manopaiboon C.
        • et al.
        Cost and cost-effectiveness analysis of pre-exposure prophylaxis among men who have sex with men in two hospitals in Thailand.
        J Int AIDS Soc. 2018; 21e25129
      7. GDP per capita (current US$) - Kenya. The World Bank.
        • Ochalek J.
        • Lomas J.
        • Claxton K.
        Estimating health opportunity costs in low-income and middle-income countries: a novel approach and evidence from cross-country data.
        BMJ Global Health. 2018; 3 (Erratum in: BMJ Glob Health. 2019 Jun 4;4(3):e000964corr1): e000964