Health Policy Analysis| Volume 24, ISSUE 5, P691-698, May 01, 2021

# 10 Years of End-of-Life Criteria in the United Kingdom

Open AccessPublished:March 20, 2021

## Abstract

### Objectives

In January 2009, the National Institute for Health and Care Excellence introduced supplementary guidance for end-of-life (EoL) treatments, which allowed treatments with an incremental cost-effectiveness ratio over the regular threshold (£20 000-£30 000) to be recommended, if they satisfied the EoL criteria. The aims of this study were (1) to systematically review 10 years of EoL supplementary guidance implementation and explore how it could be improved, and (2) to create a framework for incorporating the uncertainty relating to EoL criteria satisfaction into model-based cost-effectiveness analyses for decision making.

### Methods

All appraisals between January 2009 and 2019 were screened for EoL discussions. Data were extracted on the EoL criteria and cost-effectiveness assessment details. Additionally, a quantitative method was developed to include the EoL criteria satisfaction uncertainty into model-based cost-effectiveness analyses. A stylized example was created to provide a case study for the inclusion of EoL criteria satisfaction uncertainty.

### Results

An EoL discussion was identified in 35% of appraisals, 57% of which led to a positive EoL decision. Only 5.7% of technologies with positive EoL decisions were not recommended, versus 43.8% of technologies with negative EoL decisions. EoL criteria assessment was often reported insufficiently and evaluated inconsistently and nontransparently. A total of 54.9% of EoL decisions were made while at least 1 criterion was surrounded by considerable uncertainty. By applying the proposed quantitative method, this EoL criteria satisfaction uncertainty was accounted for in decision making. The stylized example demonstrated that the impact of EoL criteria satisfaction uncertainty can be substantial enough to reverse the reimbursement decision.

### Conclusions

To improve consistency/transparency and correct reimbursement decisions’ likelihood, new guidelines on the implementation of the EoL criteria are needed.

## Introduction

In 2009, the National Institute for Health and Care Excellence (NICE) introduced supplementary guidance for end of life (EoL) technologies.
National Institute for Health and Clinical Excellence, National Institute for Health and Care Excellence
Appraising life-extending, end of life treatments. National Institute for Health and Care Excellence.
This guidance allows the Appraisal Committee (AC) to give additional weight to quality-adjusted life-years (QALYs) benefits achieved at the EoL as they see fit, up to a maximum of 1.7. This effectively allows EoL technologies with incremental cost-effectiveness ratios (ICERs) greater than the standard £20 000 to £30 000 NICE threshold range to be recommended for use within the National Health Service (NHS), up to a maximum of £50 000 per QALY. While the EoL threshold can be lower than £50 000 owing to the committee’s discretion to choose the additional weight assigned to QALY gains, in practice this £50 000 maximum threshold has become the standard EoL threshold. To be considered an EoL technology, several criteria must be met: it should be indicated for patients with a short life expectancy (<24 months) and should extend life by at least 3 months compared to current NHS treatment.
National Institute for Health and Clinical Excellence, National Institute for Health and Care Excellence
Appraising life-extending, end of life treatments. National Institute for Health and Care Excellence.
The original guidance contained a third criterion, stating that the treatment should be indicated for a small patient population, but NICE removed this criterion in 2016.
National Institute for Health and Care Excellence
Consultation on Proposals for a New Cancer Drugs Fund (CDF) Operating Model from 1st April 2016.
The assumption underlying the higher ICER threshold for EoL technologies is that society values time at the end of life more highly; however, a strong evidence base supporting this assumption is lacking.
• Appleby J.
• Maybin J.
Topping up NHS care.
• Shah K.K.
• Tsuchiya A.
• Wailoo A.J.
Valuing health at the end of life: a stated preference discrete choice experiment.
• Linley W.G.
• Hughes D.A.
Societal views on NICE, cancer drugs fund and value-based pricing criteria for prioritising medicines: a cross-sectional survey of 4118 adults in Great Britain.
• McHugh N.
• van Exel J.
• Mason H.
• et al.
Are life-extending treatments for terminal illnesses a special case? Exploring choices and societal viewpoints.
Despite this lack of evidence, over a decade has passed since the EoL supplementary guidance was issued, and it is now firmly enshrined in NICE decision making, without any sign of abandonment.
National Institute for Health and Care Excellence Decision Support Unit. End of life.
,
National Institute for Health and Care Excellence
Guide to the methods of technology appraisal 2013.
Collins and Latimer calculated that the still relatively few EoL technologies approved between 2009 and 2011 led to an annual additional cost of over £500 million or an annual health loss of 15 098 QALYs (assuming that on average the NHS spends £30 000 per QALY displaced by EoL technologies).
• Collins M.
• Latimer N.
NICE’s end of life decision making scheme: impact on population health.
Since 2011, many more EoL technologies have been approved. Therefore, the impact of the EoL supplementary guidance on NHS health outcomes is expected to be profound.
Considering the continuing and ever-growing impact of EoL guidance on the NHS, it is vital that this guidance is implemented with precision and care. Given the potential health losses that can result from recommending less efficient technologies than those that are displaced, the AC needs to be certain that the new technology does indeed meet the EoL criteria and is deserving of the higher EoL threshold. Yet, despite the guidance having been implemented regularly over the last decade, there are no recent or elaborate reviews of the implementation of the EoL supplementary guidance by NICE. This study therefore sets out to systematically review the implementation of the EoL supplementary guidance over the past 10 years and to identify how the implementation can be improved, by giving particular attention to the uncertainty associated with EoL decisions and how this is dealt with by the AC. This type of EoL satisfaction uncertainty has been historically underexplored in technology appraisals (TAs), despite its potential contribution to overall decision uncertainty. Hence, besides conducting a systematic review on the EoL implementation, we proposed a method to incorporate EoL uncertainty in model-based cost-effectiveness analysis, which quantifies the impact of the overlooked EoL uncertainty on the cost-effectiveness of a technology. This strategy would improve the implementation of the EoL supplementary guidance, by ensuring that the EoL criteria satisfaction uncertainty is accounted for in the cost-effectiveness analysis, which would ultimately increase the probability that the correct reimbursement decision is made, improving the NICE reimbursement decision-making process and the economic efficiency of resource allocation within NHS.

## Methods

### Systematic Review of TAs

A systematic review of TA guidance was performed to provide insight into the implementation of the EoL supplementary guidance and to identify the types of uncertainty surrounding EoL decisions. Details of the review strategy are given in Appendix 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.11.015. The review focused on single TAs (STAs), since it is the most frequently conducted appraisal type, and other types either follow a different process (ie, in multiple or diagnostic TAs, independent assessment groups prepare the evidence dossier) or different criteria are used in decision making (ie, in ultra-orphan drug appraisals, a broader range of factors are included and different QALY weighting methods are used).

### Incorporating EoL Criteria Satisfaction Uncertainty Into Economic Evaluation

To improve the implementation of the EoL supplementary guidance, a method to integrate EoL decision uncertainty into model-based cost-effectiveness analyses was developed and the impact of integrating this uncertainty was investigated using a stylized example. The developed method is similar to the method explained by Versteegh et al, which incorporated the uncertainty surrounding being in different severity categories, for which different cost-effectiveness thresholds are applied in The Netherlands, using probabilistic sensitivity analysis (PSA).
• Versteegh M.M.
• Ramos I.C.
• Ansaripour A.
• Reckers-Droog V.T.
• Brouwer W.B.
Severity-adjusted probability of being cost effective.

#### Modeling EoL decision uncertainty

In current model-based economic evaluations, if an EoL technology is considered to meet both EoL criteria, the net monetary benefit (NMB) is calculated for each simulation iteration of the PSA using the effective EoL cost-effectiveness threshold of £50 000. However, in some PSA simulation iterations, one or more EoL criteria can be violated. For those iterations in which the EoL criteria are violated, the regular cost-effectiveness threshold range of £20 000 to £30 000 should be used in the NMB calculations. To reflect this uncertainty, we introduce three outcomes that can be presented alongside the other cost-effectiveness results:
• EoL criteria satisfaction probability (proportion of PSA iterations that satisfied EoL criteria, explained in box 1 in Appendix 3.1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.11.015)
• $EoLiNMB$: EoL-adjusted incremental NMB (average iNMB from all iterations, where the threshold used in each iteration is dependent on whether the EoL criteria were satisfied or not, explained in box 2 in Appendix 3.1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.11.015)
• $EoL_PCE$: EoL-adjusted probability of being cost-effective (proportion of PSA iterations with positive $EoLiNMB$, explained in box 3 in Appendix 3.1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.11.015)
From these outcomes, one can quantify the risk of EoL criteria violation and the impact of the incorporation of EoL criteria satisfaction on cost-effectiveness outcomes ($EoLiNMB$and $EoL_PCE$). These EoL-adjusted cost-effectiveness outcomes would enable the inclusion of the EoL decision uncertainty in technologies that are considered to meet EoL criteria in cost-effectiveness analyses, instead of the standard cost-effectiveness outcomes, which use a single cost-effectiveness threshold (eg, £50 000 per QALY gained) and assume that EoL criteria are satisfied in all PSA iterations. A typical oncology model (details in Appendix 3.2 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.11.015) was used as an example case study to demonstrate the impact of incorporating the EoL uncertainty in cost-effectiveness analysis.

## Results

### Overview of the EoL Supplementary Guidance Implementation

Of 350 screened TAs, 121 (35%) included an EoL discussion. Figure 1 reveals a large increase in the EoL discussions after 2016, when the small patient population criterion was removed.
Of the 121 TAs with an EoL discussion, 111 STAs covered 74 technologies. Notably, all these technologies targeted oncology indications. In 12 STAs, different EoL decisions were given for different subpopulations. These EoL discussions were analyzed separately, bringing the total number of analyzed EoL discussions to 123, from which data were extracted.
Approximately 20% of EoL assessments in the identified appraisals were initiated without any company claim. It was unclear what triggered these ACs to consider the EoL criteria without any company EoL claim.
Of the 123 STAs in which the EoL criteria were discussed, 70 (57%) resulted in a positive EoL decision and 48 (39%) resulted in a negative EoL decision. In 4 STAs the EoL discussion was concluded to be irrelevant, because ICERs were below £30 000, and in 1 STA the EoL discussion was deemed inconclusive owing to clinical data immaturity.

#### The EoL decision and the reimbursement decision

A strong correlation emerged between a positive EoL decision and the chance of reimbursement (Fig. 2). If a positive EoL decision was made, in only 6% (4 STAs
National Institute for Health and Care Excellence
TA403: Ramucirumab for previously treated locally advanced or metastatic non-small-cell lung cancer.
National Institute for Health and Care Excellence
TA414: Cobimetinib in combination with vemurafenib for treating unresectable or metastatic BRAF V600 mutation-positive melanoma.
National Institute for Health and Care Excellence
TA515: Eribulin for treating locally advanced or metastatic breast cancer after 1 chemotherapy regimen.
National Institute for Health and Care Excellence
TA530: Nivolumab for treating locally advanced unresectable or metastatic urothelial cancer after platinum-containing chemotherapy.
) of STAs was a negative recommendation given. Conversely, if a technology was deemed not to satisfy the EoL criteria, in 44% (21 STAs) of cases, a negative reimbursement recommendation was given. This large difference (6% vs. 44%) demonstrates that the judgment on the EoL criteria satisfaction is highly correlated with the reimbursement decision of the AC.
Additionally, Figure 2 shows that the proportion of assessments with an ICER (used for AC decision making) below £30 000 was much higher among submissions with a negative EoL decision (versus submissions with a positive EoL decision): (31% vs 4%). This could indicate that when companies were convinced to meet the EoL criteria, they adjusted their pricing according to the higher EoL cost-effectiveness threshold, since most technology assessments (71%) with a positive EoL decision had an ICER between £30 000 and £50 000 per QALY gained.

#### EoL criteria evaluation

Table 1 presents the EoL criteria evaluation outcomes. The life expectancy criterion was not met in 56% of negative EoL decisions, while the incremental life-years criterion was not met in 46% of negative EoL decisions. Only in 86% of EoL discussions were both EoL criteria assessed. Since the failure of one criterion is sufficient for a negative EoL decision, the second criterion was often not discussed after the first criterion was not met.
Table 1EoL criteria outcomes underlying EoL decisions and uncertainty surrounding the EoL criteria satisfaction, as signaled by the discussions in the FAD.
EoL criterion evaluations
Life expectancy criterion →MetNot metNot evaluatedTotal
Incremental life-years criterion ↓
Met70
Positive EoL decisions.
10080
Not met184022
Not evaluated013
In 1 EoL decision not evaluated owing to immature data, in 12 EoL decisions owing to life expectancy criterion not being met.
3
Both criteria not evaluated owing to patient population criterion not being met.
16
Total88273118
Uncertainty in EoL criterion evaluations
Life expectancy criterion evaluations (%) (n = 115)Incremental life-years criterion evaluations (%) (n = 102)
No FAD discussion signaling uncertainty82 (94)58 (59)
FAD discussion signaling uncertainty18 (21)42 (43)
Conclusion criterion met62 (13)77 (33)
Conclusion criterion not met38 (8)23 (10)
EoL indicates end of life; FAD, final appraisal document.
Positive EoL decisions.
In 1 EoL decision not evaluated owing to immature data, in 12 EoL decisions owing to life expectancy criterion not being met.
Both criteria not evaluated owing to patient population criterion not being met.

### Uncertainties Surrounding the EoL Decision

#### Reported EoL decision uncertainty in the final appraisal documents

For all identified EoL decisions, the final appraisal documents (FAD) and committee papers were screened to detect if any ambiguity or uncertainty on the satisfaction of each criterion was reported. Life expectancy criterion evaluations were less often (18%) associated with uncertainty compared to incremental life-years criterion evaluations (42%), as shown in Table 1.
In 55% of EoL decisions, in which both EoL criteria were evaluated, the FAD included a discussion signaling uncertainty. Often these discussions centered on uncertainty surrounding the incremental life-years criterion (Appendix Fig. A2 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.11.015). This could be because there are often more data available on life expectancy estimates, because they are related to the indication studied and not specific to the new technology, unlike the incremental life-years estimates, often based only on the trials/economic model carried out by the manufacturer.
In 77% of incremental life-years evaluations indicating uncertainty, the AC concluded that the criterion was met (Table 1). For the life expectancy criterion, this percentage was 62%. These figures indicate that the AC was inclined to give the benefit of the doubt even in the presence of uncertainty surrounding criteria satisfaction.

#### Uncertainty in the estimates used in the EoL criteria evaluation

This section focuses on how the estimates used in the EoL criteria evaluation were generated, in the information source (eg, trial, economic model, literature, clinical opinion) and the measure used (eg, median OS, mean OS).
A total of 64% of life expectancy criterion evaluations were based on a single estimate, with the rest based on multiple estimates. These estimates were calculated from a wide variety of information sources and measure combinations (Table 2). Life expectancy was most frequently evaluated by either median best supportive care (BSC) OS from the trial and the mean BSC OS from the economic model. In many evaluations, the information source and/or measure used was not described. In addition, expert opinion and literature sources were most often used (58% and 75%, respectively) in combination with other information sources, as supporting evidence.
Table 2Methods used in the criterion evaluations.
Life expectancy criterion (n = 166)Incremental life-years criterion (n = 166)
MethodNumber of times appliedMethodNumber of times applied
Median OS from trial46Mean OS from economic model42
Mean OS from economic model39Median OS from trial38
Life expectancy from experts14Not specified from trial8
Life expectancy from literature12Not specified from economic model6
Not specified from trial11Crossover adjusted methods
Different crossover adjustment methods were aggregated owing to small size of category.
from trial
5
Not specified from not specified11Not specified from indirect comparison4
Not specified from economic model10Median OS from economic model3
Median OS from literature10Median PFS from trial2
Not specified from literature7Restricted mean OS from trial2
Median OS from economic model3Other combinations6
Mean OS from literature3
OS indicates overall survival; PFS, progression-free survival.
Different crossover adjustment methods were aggregated owing to small size of category.
Eight-five percent of the incremental life-years criterion evaluations were based on a single estimate, with the rest based on multiple estimates. These estimates were mostly based on either the mean incremental OS from the economic model or the median incremental OS from the trial (Table 2).
The review of the EoL discussions from the STAs revealed a lack of consistency and transparency in how the EoL criteria were assessed. The estimates used in the assessments were obtained using different measure types and originated from various information sources. The uncertainty surrounding the EoL criteria satisfaction was not formally included in reimbursement decision making.

### Impact of EoL Decision Uncertainty on Model-Based Cost-Effectiveness Analyses

In this section, the potential impact of incorporating the EoL criteria satisfaction uncertainty on the reimbursement decisions is demonstrated, using the stylized example of a 3-state model, typical to oncology appraisals, described further in Appendix 3 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.11.015.
In this example, the life expectancy criterion assessment was based on the modeled mean OS BSC, and the incremental life-years criterion was based on the modeled mean OS benefit. Table 3 shows that both the life expectancy (modeled mean OS BSC less than 2 years) and the incremental life-years (modeled mean OS benefit larger than 3 months) criteria could be considered met, leading to a cost-effectiveness threshold of £50 000.
Table 3Deterministic and the probabilistic results from the economic model.
Discounted costsDiscounted QALYsDiscounted NMB (λ = £50 000)Discounted NMB (λ = £20 000)Undiscounted LYs (Mean OS)
Deterministic
Treatment£31 643.041.41£38 901.53−£3425.212.24
BSC£22 361.601.22£38 648.53£2042.451.98
The estimates relevant to the life expectancy and incremental life-years criteria.
Increment£9281.440.19£253.00−£5467.660.26
The estimates relevant to the life expectancy and incremental life-years criteria.
ICER£48 673.23
Probabilistic
Treatment£32 190.721.43£39 193.16−£3637.172.26
BSC£22 564.021.23£39 081.62£2094.232.00
The estimates relevant to the life expectancy and incremental life-years criteria.
,
BSC LYs = 1.9996, slightly less than 2 years.
Increment£9626.700.20£111.54-£5731.400.26
The estimates relevant to the life expectancy and incremental life-years criteria.
ICER£49 427.32
Probability of treatment being cost-effective (λ = £50 000)0.50
Note. BSC indicates best supportive care; ICER, incremental cost-effectiveness ratio; LYs, life-years; NMB, net monetary benefit; OS, overall survival; QALYs, quality-adjusted life-years.
The estimates relevant to the life expectancy and incremental life-years criteria.
BSC LYs = 1.9996, slightly less than 2 years.
The deterministic and probabilistic ICER were both below £50 000 and the iNMB was thus positive at the £50 000 EoL threshold (Table 3). In Figure 3, the cost-effectiveness acceptability curve demonstrates the EoL technology probability of being cost-effective at a threshold of £50 000 was 50%. These results suggest that the oncology treatment would be cost-effective, considering the EoL threshold.
However, this EoL decision based on the deterministic and/or mean PSA values does not incorporate the EoL decision uncertainty. The impact of adjusting for this EoL decision uncertainty was investigated by using the economic model described in Appendix 3 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.11.015.
From the model, the EoL criteria satisfaction probability was calculated to be 38.98% (green dots in Appendix Fig. A8 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.11.015). The $EoLiNME$ was calculated to be -£1894 and finally the $EoL_PCE$ for the new oncology drug was 33%; these EoL-adjusted results suggest that the technology would not be considered cost-effective. Hence, the incorporation of the EoL criteria satisfaction uncertainty in the cost-effectiveness analysis can yield different conclusions in the reimbursement decision for a new technology, compared to the standard case when EoL criteria satisfaction uncertainty is not included in the analysis, and it is assumed that EoL criteria are met based on the mean PSA results.

## Discussion

### Key Findings

This study aimed to systematically review the implementation of EoL supplementary guidance and to explore how its implementation could be improved. Therefore, the EoL discussions in STAs in the 10 years following its inception were analyzed and a de novo method was developed to incorporate EoL decision uncertainty into the cost-effectiveness analysis conducted by means of an economic model.
Discussions on the EoL criteria were identified in 121 STAs, all in oncology. Particularly after the patient population criterion was removed in 2016, the proportion of STAs discussing the EoL criteria increased tremendously. Once the drug was deemed to be an EoL technology, the chance of receiving a positive reimbursement decision was higher.
In practice, the EoL supplementary guidance has led to a shift in the cost-effectiveness threshold for EoL technologies from £20 000 to £30 000 to £50 000.
• Griffiths E.
NICE’s criteria for end-of-life therapies: is there a fourth hurdle to overcome?.
• Hamerslag L.
• Haynes S.
• Kusel J.
• Costello S.
RS2 Cost-effectiveness of end-of-life, life-extending interventions: NICE’s cost-effectiveness threshold explored.
• Moïse P.
• Fassler P.
PCN105 Estimating NICE’s cost effective threshold for end-of-life cancer treatments.
In our review, we observed that most EoL technologies were priced between £30 000 and £50 000 per QALY gained, which could suggest that companies adjusted their pricing strategy to the higher EoL cost-effectiveness threshold.
In the implementation of the EoL supplementary guidance, some lack of consistency and transparency was observed. Firstly, there was a lack of transparency on what had triggered the ACs to initiate an EoL discussion. Often it could not be inferred from the FAD whether the company had made an EoL claim or the ACs initiated the EoL discussion themselves without a company claim.
A more profound level of inconsistency and lack of transparency was observed in how the estimates used in the EoL criteria assessment were evaluated and reported in FADs. A wide variety of methods and information sources were alternately used in the evaluation of the EoL criteria. In many STAs, the estimates used in the EoL assessments were not reported transparently. Often, the life expectancy and incremental life-years criteria were assessed using the mean OS from the economic model and median OS from the Kaplan-Meier curves from the trial data; however, many other methods and information sources were also consulted. The EoL supplementary guidance does not specify whether to use mean OS or median OS and whether to use the trial, economic model, literature, or experts as the information source, but leaves room for interpretation for the ACs.
National Institute for Health and Clinical Excellence, National Institute for Health and Care Excellence
Appraising life-extending, end of life treatments. National Institute for Health and Care Excellence.
The results of the review imply that the room for interpretation is too broad, causing inconsistencies across different EoL decisions.
Half of the EoL discussions in the identified FADs signaled some level of uncertainty in the EoL criteria satisfaction. These discussions were primarily centered around the incremental life-years criterion. Despite these uncertainties, the AC gave the benefit of the doubt in well over half of cases, and the uncertainty surrounding the EoL criteria satisfaction was not formally included in the cost-effectiveness decisions.
This study introduced 3 measures to quantitatively account for EoL uncertainty in appraisals where both EoL criteria are argued to be satisfied: (1) EoL criteria satisfaction probability, to quantify the level of EoL criteria satisfaction uncertainty; (2) EoL-adjusted incremental net monetary benefit; and (3) EoL-adjusted probability of being cost-effective, which quantify the inclusion of the EoL uncertainty on the cost-effectiveness estimate and its uncertainty. These measures can be calculated easily with a few adjustments to a standard economic model. As demonstrated in the case study, incorporating the uncertainty in the EoL criteria satisfaction can change the cost-effectiveness results from borderline cost-effective at the maximum EoL threshold of £50 000 to being convincingly not cost-effective.
Making the optimal reimbursement decisions and maximizing the efficiency of the NHS budget are of utmost importance given the substantial impact of the EoL guidance on the economic efficiency of the NHS demonstrated in the literature and the lack of empirical evidence that the public truly places a larger weight on QALYs at the end of life.
• Shah K.K.
• Tsuchiya A.
• Wailoo A.J.
Valuing health at the end of life: a stated preference discrete choice experiment.
• Linley W.G.
• Hughes D.A.
Societal views on NICE, cancer drugs fund and value-based pricing criteria for prioritising medicines: a cross-sectional survey of 4118 adults in Great Britain.
• McHugh N.
• van Exel J.
• Mason H.
• et al.
Are life-extending treatments for terminal illnesses a special case? Exploring choices and societal viewpoints.
,
• Collins M.
• Latimer N.
NICE’s end of life decision making scheme: impact on population health.
The tendency for benefit of the doubt and use of the maximum threshold as the norm further increases the impact on the NHS. However, without further guidance on how best to implement the EoL guidance and without methods for the quantification of the uncertainty surrounding the EoL claim, the ACs have had little choice until now but to adopt a more lenient strategy.
Some may argue that the impact of this lenience is somewhat mitigated by the fact that the AC can provide a temporary recommendation for new and uncertain technologies within the Cancer Drug Fund, subject to continued data collection to reduce uncertainty in an updated appraisal. However, funding many borderline and uncertain technologies at an ICER of £50 000, even for a limited period, will still have a substantial economic impact on the NHS and given the current methods for EoL appraisal, the final decision regarding reimbursement will still be made without accounting for uncertainty relating to the EoL criteria, meaning that technologies that are borderline on the EoL criteria may still be inappropriately funded at the maximum £50 000 when they are in fact not cost-effective.
The timing of this study coincides with the ongoing NICE methods update, which provides an opportune moment for additional recommendations relating to the EoL appraisal process. This methods update has been long awaited and much anticipated as many advances have been made in the field of economic evaluation since it was last revised in 2013. During the wait, there has been discussion in the literature on the current weaknesses in the NICE methods guidelines and the updates that would provide the most value. Two of the biggest methods challenges that have emerged are how best to deal with limited and uncertain evidence as well as contestability of normative judgements about benefits.
• Vemer P.
• Ramos I.C.
• Van Voorn G.A.K.
• Al M.J.
• Feenstra T.L.
AdViSHE: a validation-assessment tool of health-economic models for decision makers and model users.
This study makes an important step in advancing the methods for quantifying uncertainty in relation to the normative EoL judgment of benefits.

### Limitations

There are several important limitations to this study. In the review, it was assumed that the reported discussions in the FAD documents accurately reflected the discussions on the EoL criteria in the appraisal process. Upon unclarity, committee papers were consulted; however, if important details of the EoL discussions were omitted completely in the FAD, this would have implications for the sensitivity of our review in identifying the EoL discussions. Additionally, some important data were missing in some appraisals (eg, the company submission, precise ICER or OS benefit). The lack of a precise OS benefit in some appraisals hindered accurate analysis of the uncertainty, but as it was rather infrequent, it is not expected to influence the validity of the results.
In this study, we propose that the EoL assessment should be based on the mean OS calculated from a valid cost-effectiveness model. Ideally, a valid cost-effectiveness model should also include relevant external evidence, and the model outcomes should be in line with the external evidence, allowing an accurate estimation of the mean OS for each treatment. However, in practice, we are aware that this might not be the case. For this purpose, it is essential that validation and verification efforts are sufficiently conducted and transparently reported, following the best practices and guidelines published in the literature.
• Vemer P.
• Ramos I.C.
• Van Voorn G.A.K.
• Al M.J.
• Feenstra T.L.
AdViSHE: a validation-assessment tool of health-economic models for decision makers and model users.
,
• Büyükkaramikli N.C.
• Rutten-van Mölken M.P.M.H.
• Severens J.L.
• Al M.
TECH-VER: a verification checklist to reduce errors in models and improve their credibility.
The methods proposed in this study, to incorporate the EoL criteria satisfaction uncertainty into the PSA in an economic model, would account for EoL adjustment for the decision making, only taking the parameterized uncertainty into consideration. We are aware that the structure of a model is dependent on the model developer’s judgments concerning the decision problem, and the choice of any type of model structure would lead to structural uncertainty, since every model approximates real-world processes and relationships. Many international good practice recommendations in the literature recommend identifying and possibly quantifying the inherent structural uncertainties in a model, using techniques such as scenario analysis, model parameterization, model averaging, or model discrepancy analysis.
• Rothery C.
• Strong M.
• Koffijberg H.
• et al.
Value of information analytical methods: report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force.
• Strong M.
• Oakley J.E.
When is a model good enough? Deriving the expected value of model improvement via specifying internal model discrepancies.
• Jones L.
• Griffin S.
• Palmer S.
• et al.
Clinical effectiveness and cost-effectiveness of clopidogrel and modified-release dipyridamole in the secondary prevention of occlusive vascular events: a systematic review and economic evaluation.
• Bojke L.
• Claxton K.
• Sculpher M.
• Palmer S.
Characterizing structural uncertainty in decision analytic models: a review and application of methods.
• Jackson C.H.
• Thompson S.G.
• Sharples L.D.
Accounting for uncertainty in health economic decision models by using model averaging.
• Price M.J.
• Welton N.J.
• Briggs A.H.
Model averaging in the presence of structural uncertainty about treatment effects: influence on treatment decision and expected value of information.
• Strong M.
• Oakley J.E.
• Chilcott J.
Managing structural uncertainty in health economic decision models: a discrepancy approach.
Once this type of structural uncertainty is quantified, the EoL adjustments can be incorporated as well, such as running EoL-adjusted probabilistic analysis for each scenario analysis or running the EoL-adjusted probabilistic analysis, where the variables of the model parameterization are sampled alongside the other parameters in the model.
Another potential limitation is that the ACs might consider threshold ranges at their discretion (eg, between £20 000 and £30 000), whereas the method proposed in this study requires prespecified thresholds. This limitation can be addressed in further methodological research, by extending the method proposed in this article to threshold ranges. Additionally, the ACs might not currently focus on the results from probabilistic analysis in their decision making. However, the authors would argue that this quantification of uncertainty within the adjusted probabilistic analyses provides the AC with a flexible method to assess the cost-effectiveness under uncertainty and whether an uncertain EoL technology warrants the maximum EoL threshold or whether a lower threshold is more appropriate, enabling the AC to maximize the efficiency of the NHS budget in the context of the EoL guidance.

## Conclusions

This review of the EoL supplementary guidance implementation showed that it is not always consistently conducted or transparently reported and that uncertainty relating of EoL criteria satisfaction is insufficiently accounted for in decision making. To improve the consistency and transparency of the implementation process and, more importantly, the likelihood that correct reimbursement decisions are made, new guidelines should be published on the implementation of the EoL supplementary guidance.
To optimize the implementation process in efficiency and consistency, firstly, we recommend that the EoL assessment in an appraisal always should be initiated by the company. Future EoL guidance should make clear that this is the responsibility of the company to avoid confusion and inconsistent treatment of appraisals, as in some cases where an EoL assessment may be relevant but has not been initiated by the company, the ACs may initiate a claim and in other cases the potential for a claim may be missed. For sake of efficiency during tightly scheduled committee meetings within a packed NICE system, the authors recommend that each EoL assessment should begin with the life expectancy criterion and only if this is satisfied should the incremental life-years criterion be discussed. Additionally, we suggest that the estimates used in EoL criteria assessments should be obtained from the mean OS from the economic model, and the other estimates from other methods/information sources can be consulted as supplementary evidence.
To optimize reimbursement decision making, if it is decided that a technology satisfies both EoL criteria, the EoL criteria satisfaction uncertainty should be incorporated into the cost-effectiveness analyses using the methods/measures described in this study. This quantitative assessment of EoL uncertainty can provide valuable guidance to the AC on whether the technology should be considered cost-effective at the maximum EoL threshold of £50 000 or whether, as stated in the EoL guidance, a lower weighting (and therefore threshold) would be more appropriate on a case-by-case basis. This will assist the AC in providing recommendations consistent with the intention of the EoL guidance. The methods presented are generalizable to other contexts, where differential cost-effectiveness thresholds are used in reimbursement decision making. We suggest the metrics introduced in this article are included as supplementary analyses for future evaluations, where differential cost-effectiveness thresholds are at place. We would like to emphasize that further equity weighting research and the challenges of incorporating equity weighting in economic evaluations should be conducted to complement the findings from our study.
Author Contributions: Concept and design: Bovenberg, Buyukkaramikli
Acquisition of data: Bovenberg
Analysis and interpretation of data: Bovenberg, Penton, Buyukkaramikli
Drafting of the manuscript: Bovenberg, Penton, Buyukkaramikli
Critical revision of the paper for important intellectual content: Penton, Buyukkaramikli
Conflict of Interest Disclosures: The authors reported no conflicts of interest.
Funding/Support: The authors received no financial support for this research.

## Supplemental Material

• Supplementary Material

## References

• National Institute for Health and Clinical Excellence, National Institute for Health and Care Excellence
Appraising life-extending, end of life treatments. National Institute for Health and Care Excellence.
• National Institute for Health and Care Excellence
Consultation on Proposals for a New Cancer Drugs Fund (CDF) Operating Model from 1st April 2016.
2016
• Appleby J.
• Maybin J.
Topping up NHS care.
BMJ. 2008; 337: a2449
• Shah K.K.
• Tsuchiya A.
• Wailoo A.J.
Valuing health at the end of life: a stated preference discrete choice experiment.
Soc Sci Med. 2015; 124: 48-56
• Linley W.G.
• Hughes D.A.
Societal views on NICE, cancer drugs fund and value-based pricing criteria for prioritising medicines: a cross-sectional survey of 4118 adults in Great Britain.
Health Econ. 2013; 22: 948-964
• McHugh N.
• van Exel J.
• Mason H.
• et al.
Are life-extending treatments for terminal illnesses a special case? Exploring choices and societal viewpoints.
Soc Sci Med. 2018; 198: 61-69
1. National Institute for Health and Care Excellence Decision Support Unit. End of life.
• National Institute for Health and Care Excellence
Guide to the methods of technology appraisal 2013.
2013
• Collins M.
• Latimer N.
NICE’s end of life decision making scheme: impact on population health.
BMJ. 2013; 346: f1363
• Versteegh M.M.
• Ramos I.C.
• Ansaripour A.
• Reckers-Droog V.T.
• Brouwer W.B.
Severity-adjusted probability of being cost effective.
Pharmacoeconomics. 2019; 37: 1155-1163
• National Institute for Health and Care Excellence
TA403: Ramucirumab for previously treated locally advanced or metastatic non-small-cell lung cancer.
2016
• National Institute for Health and Care Excellence
TA414: Cobimetinib in combination with vemurafenib for treating unresectable or metastatic BRAF V600 mutation-positive melanoma.
2016
• National Institute for Health and Care Excellence
TA515: Eribulin for treating locally advanced or metastatic breast cancer after 1 chemotherapy regimen.
2018
• National Institute for Health and Care Excellence
TA530: Nivolumab for treating locally advanced unresectable or metastatic urothelial cancer after platinum-containing chemotherapy.
2018
• Griffiths E.
NICE’s criteria for end-of-life therapies: is there a fourth hurdle to overcome?.
Value Health. 2016; 19: A489
• Hamerslag L.
• Haynes S.
• Kusel J.
• Costello S.
RS2 Cost-effectiveness of end-of-life, life-extending interventions: NICE’s cost-effectiveness threshold explored.
Value Health. 2011; 14: A243
• Moïse P.
• Fassler P.
PCN105 Estimating NICE’s cost effective threshold for end-of-life cancer treatments.
Value Health. 2011; 14: A173
• Vemer P.
• Ramos I.C.
• Van Voorn G.A.K.
• Al M.J.
• Feenstra T.L.
AdViSHE: a validation-assessment tool of health-economic models for decision makers and model users.
Pharmacoeconomics. 2016; 34: 349-361
• Büyükkaramikli N.C.
• Rutten-van Mölken M.P.M.H.
• Severens J.L.
• Al M.
TECH-VER: a verification checklist to reduce errors in models and improve their credibility.
Pharmacoeconomics. 2019; 37: 1391-1408
• Rothery C.
• Strong M.
• Koffijberg H.
• et al.
Value of information analytical methods: report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force.
Value Health. 2020; 23: 277-286
• Strong M.
• Oakley J.E.
When is a model good enough? Deriving the expected value of model improvement via specifying internal model discrepancies.
SIAM/ASA J Uncertain Quantif. 2014; 2: 106-125
• Jones L.
• Griffin S.
• Palmer S.
• et al.
Clinical effectiveness and cost-effectiveness of clopidogrel and modified-release dipyridamole in the secondary prevention of occlusive vascular events: a systematic review and economic evaluation.
Health Technol Assess. 2004; 8 (iii-iv, 1-196)
• Bojke L.
• Claxton K.
• Sculpher M.
• Palmer S.
Characterizing structural uncertainty in decision analytic models: a review and application of methods.
Value Health. 2009; 12: 739-749
• Jackson C.H.
• Thompson S.G.
• Sharples L.D.
Accounting for uncertainty in health economic decision models by using model averaging.
J R Stat Soc Ser A Stat Soc. 2009; 172: 383-404
• Price M.J.
• Welton N.J.
• Briggs A.H.