If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Correspondence: Thomas Ward, MSc, Health Economics Group, College of Medicine and Health, University of Exeter, St Luke’s Campus, Heavitree Road, Exeter EX1 2LU England, United Kingdom.
The availability of novel, more efficacious cancer therapies is increasing, resulting in significant treatment effect heterogeneity and complicated treatment and disease pathways. Technology appraisals (TAs) evaluate clinical and economic evidence to inform reimbursement decisions and resource allocation. Through critical appraisal of UK cancer TAs, we identify areas where considerations of heterogeneity can be improved. We focus on 3 cancer sites: colorectal, lung, and ovarian cancer, encompassing variation in screening, diagnostic and treatments pathways.
•
All TAs in this review used decision analytic modeling. The majority used partitioned survival models and evaluated aggregate outcomes of clinical trial populations. Only 2 models explicitly considered real-world patient heterogeneity in disease progression estimates. Moreover, predetermined subgroup analyses contained within the clinical studies that informed the TAs were rarely exploited in economic analyses.
•
This review highlights a paucity of information relating to the assessment of heterogeneity in colorectal, lung, and ovarian cancer TAs. We conclude that future cancer TAs should consider more flexible modeling approaches and apply real-world data to explore heterogeneity within their economic analyses, especially if the complexity of treatment and disease pathways is to be reflected.
Abstract
Objectives
The availability of novel, more efficacious and expensive cancer therapies is increasing, resulting in significant treatment effect heterogeneity and complicated treatment and disease pathways. The aim of this study is to review the extent to which UK cancer technology appraisals (TAs) consider the impact of patient and treatment effect heterogeneity.
Methods
A systematic search of National Institute for Health and Care Excellence TAs of colorectal, lung and ovarian cancer was undertaken for the period up to April 2020. For each TA, the pivotal clinical studies and economic evaluations were reviewed for considerations of patient and treatment effect heterogeneity. The study critically reviews the use of subgroup analysis and real-world translation in economic evaluations, alongside specific attributes of the economic modeling framework.
Results
The search identified 49 TAs including 49 economic models. In total, 804 subgroup analyses were reported across 69 clinical studies. The most common stratification factors were age, gender, and Eastern Cooperative Oncology Group performance score, with 15% (119 of 804) of analyses demonstrating significantly different clinical outcomes to the main population; economic subgroup analyses were undertaken in only 17 TAs. All economic models were cohort-level with the majority described as partitioned survival models (39) or Markov/semi-Markov models. The impact of real-world heterogeneity on disease progression estimates was only explored in 2 models.
Conclusion
The ability of current modeling approaches to capture patient and treatment effect heterogeneity is constrained by their limited flexibility and simplistic nature. This study highlights a need for the use of more sophisticated modeling methods that enable greater consideration of real-world heterogeneity.
Between 2015 and 2017, an estimated 2.5 million people were living with cancer in the United Kingdom, with an estimated annual incidence of 367 000 and, despite general improvements in population health, incidence and prevalence are predicted to increase.
Nevertheless, while the United Kingdom falls behind other high-income countries, in recent years there has been improvement in mortality rates across most cancers, driven by an ever-evolving therapeutic landscape and earlier diagnoses.
Global surveillance of trends in cancer survival 2000-14 (CONCORD-3): analysis of individual records for 37 513 025 patients diagnosed with one of 18 cancers from 322 population-based registries in 71 countries.
The introduction of several nationwide screening policies, the emergence of targeted therapies and an increasing focus on personalized care have all contributed to such improvements.
These changes have ushered in the potential for significant treatment outcome variability, compounded by inherent increases in patient and treatment effect heterogeneity.
Patient heterogeneity typically refers to the variability of particular characteristics (eg, age, sex, etc) among patients in a given population, while treatment effect heterogeneity refers to the nonrandom, explainable variability in the direction and magnitude of treatment effects for individuals within a population.
Estimation and reporting of heterogeneity of treatment effects.
in: Developing a Protocol for Observational Comparative Effectiveness Research: A User's Guide. Agency for Healthcare Research and Quality (US),
Rockville (MD)2013
Treatment effect heterogeneity can be measured in relative or absolute terms and patient heterogeneity may conventionally be represented by variation in outcomes under the status quo, while treatment effect heterogeneity would be operationalized as the variation in the difference in outcomes between the new treatment and the status quo.
Measures of patient and treatment effect heterogeneity seem particularly applicable to a disease area such as cancer, where the treatment landscape is rapidly evolving, and the availability of novel and more efficacious therapies is increasing. This is even more relevant when considering that newer cancer therapies are often targeted to specific patient groups, such as those with particular gene mutations or treatment and clinical histories. These targeted treatment recommendations arise from the significant patient and treatment effect heterogeneity observed among patients with cancer, naturally resulting in complicated treatment and disease pathways.
Indeed, reimbursement decisions are typically made based on average population-level results of clinical and economic evaluations, which potentially conceal important sources of outcome variability, particularly within large clinically heterogeneous populations.
Linked to these issues are the growing concerns related to inequalities across socioeconomic groups, particularly with respect to cancer survival.
The effect of socioeconomic deprivation on the association between an extended measurement of unhealthy lifestyle factors and health outcomes: a prospective analysis of the UK Biobank cohort.
People in the most income-deprived areas in England are more likely to have their cancer diagnosed at a later stage, present with more comorbidities and observe different treatment pathways to those in less deprived areas, and perhaps as a consequence, observe lower life expectancy.
Knowledge about variation in patient outcomes and their association with clinical and socioeconomic characteristics would enable efficient and equitable healthcare resource allocation.
The objective of this study is to review the extent to which UK cancer TAs consider the impact of patient and treatment effect heterogeneity and to evaluate the suitability of current modeling approaches with respect to their ability to capture such heterogeneity. Through critical appraisal, this review aims to identify areas where the consideration of patient and treatment effect heterogeneity may be improved, and to move toward recommendations on best practice for future economic evaluations.
Methods
A search of published National Institute for Health and Care Excellence (NICE) cancer TAs was undertaken. Focus was given to 3 cancer sites: colorectal, lung, and ovarian cancer, to encompass a range of screening, diagnostic and treatment practices. Full details of the search are provided in see Appendix 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.12.022. In brief, the review was undertaken according to best practices as described by the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines.
Searches were conducted on the April 12, 2020; no date restrictions were applied. For each TA, the clinical studies describing the effectiveness of the intervention under assessment, and any associated economic analyses, were retrieved and reviewed.
Within the context of economic evaluation specifically, this review explores the use of subgroup analyses and real-world translation, alongside specific attributes of the underlying economic modeling frameworks. Each component is critically reviewed from the perspective of their ability to incorporate patient and treatment effect heterogeneity.
Subgroup Analysis
Randomized clinical trials often assess the impact of treatments in specific groups of patients through predefined subgroup or stratification factor analyses. Subgroup analysis is also a common approach used to explore heterogeneity implications in cost-effectiveness analyses. Espinoza et al develops a general framework to guide the use of subgroup cost-effectiveness analysis for decision making in a collectively funded health system.
With this framework in mind, we consider to what extent TAs have included subgroup analysis in clinical and economic sections of the submission.
Patient and treatment effect heterogeneity were initially explored by extracting data relating to the presentation of subgroup analyses in the pivotal clinical studies. Subgroup analyses undertaken within the clinical studies that presented treatment effect hazard ratios (HR) for either progression-free survival (PFS) or overall survival (OS) were recorded, alongside subgroup stratifications. In addition, we recorded the number of subgroup analyses where the HR in the subgroup population significantly differed from the HR in the intention-to-treat, or overall population. Here, a significant difference is defined by an opposing effect in each population, for example, instances where the HR is greater than one favoring the comparator in the subgroup analysis population, whilst the HR is lower than one favoring the intervention in the overall population.
Economic Modeling
Economic modeling in TAs is used to estimate lifetime clinical and economic outcomes associated with a particular treatment, where direct experimental or observational data are unavailable or incomplete. Modeling frameworks provide a natural environment to assess the impact of patient and treatment effect heterogeneity and their associated uncertainty. The ability of models to incorporate such aspects can be highly dependent on their structural form and the statistical analysis used to manipulate and evaluate the underlying data.
We draw on Brennan et al and Briggs et al and describe a modified taxonomy of models (Appendix 2 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.12.022) to critically review the ability of identified models to incorporate patient and treatment effect heterogeneity.
Both are typically cohort-level and predict outcomes based on the average patient and treatment effect in a population. A PSM follows a cohort as they move between a set of exhaustive and mutually exclusive health states, relying on the use of independent survival functions to estimate state occupancy. Similarly, a Markov model follows a cohort as they move between exhaustive and mutually exclusive health states, but relies on static, cyclic transition rates. These Markov transitions enable the cohort to move back into health states that have already been visited. However, to incorporate time-dependency of transitions through relaxation of the Markov assumption, the use of tunnel states or semi-Markov models is required.
Patient-level models are an alternative to cohort-level models and estimate outcomes for each individual patient, enabling individual patient histories to be recorded and the ability to capture (first order) heterogeneity in the patient population. Patient-level models require more data than cohort models, and their ability to capture patient histories therefore comes at a cost which may or may not be necessary for solving a decision problem. Although it is suggested that patient-level models are the preferred choice for incorporating heterogeneity considerations due to their inherent flexibility, heterogeneity may be incorporated in PSMs and Markov models using extra health states to stratify patients by clinical or treatment characteristics.
The following model components are therefore reviewed and appraised:
•
Modeled population
•
Model type
•
Health states
•
Health state transitions and their derivation
•
Treatment pathway and its influence on outcomes
Real-World Translation
Trial populations often differ from those they are deemed to represent in routine clinical practice, with trial participants often being younger and healthier.
Trials undertaken in different regions or at different times can also differ significantly with respect to the patients they recruit and treatment management. These differences are particularly important for establishing the external validity of economic findings, with subgroup analysis a natural first test.
For example, in a trial that has recruited patients younger than those observed in routine clinical practice, and where the intervention demonstrates reduced effectiveness in the elderly subgroup, showing the generalizability of the trial findings to the proportions of the elderly found in routine clinical practice is akin to extending the heterogeneity of the trial subgroup to the overall clinical population. We explored the TAs acknowledgement of differences between trial populations and routine clinical practice and their approaches to real-world data translation.
Firstly, we extracted patient characteristic data for age, sex, Eastern Cooperative Oncology performance status (ECOG-PS) and ethnicity from the pivotal clinical studies. Where multiple clinical studies were included for a single TA, the range of results was presented and discussed. Secondly, for clinical studies with a National Clinical Trial identification number we reviewed the exclusion criteria described on the ClinicalTrials.gov website. The extent to which exclusion criteria would reduce the comparability between trial and routine practice populations was discussed. Finally, the TA submissions were reviewed for explicit acknowledgements of differences between trial and routine clinical practice, and economic analyses were reviewed for analytical methods that accounted for these differences.
Results
Included Studies
A total of 49 TAs, published between 2003 and 2020, were included in the review; 38 evaluated a targeted therapy (Fig. 1). The included TAs were dominated by lung (L) cancer appraisals (32/49), of which 31 were for non-small cell lung cancer; there were 8 colorectal (C) and 9 ovarian (O) cancer TAs. Details of excluded studies are presented in Appendix 3 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.12.022, alongside an overview of each included TA (Fig.1).
The clinical evidence across all TAs was informed by a total of 94 (C: 22; L: 55; O: 17) clinical studies. Among the TAs, a total of 49 (C: 9; L: 31; O: 9) cost-effectiveness models were available for review. A total of 41 cost-effectiveness analyses were undertaken by the submitting pharmaceutical company, with 8 undertaken by academic review groups.
Subgroup Analysis
Subgroup analyses assessing either PFS or OS were reported for 72 (C: 13; L: 44; O: 15) of the clinical studies in either the clinical section of the TA or in the main clinical study publication cited in the TA. A total of 804 subgroup analyses were described among these 72 clinical studies. The most common stratification factors were age (62 studies), sex (46 studies), and ECOG-PS (50 studies). Across all reported clinical subgroup analyses, 14.8% (119/804) observed results that differed to those of the overall population. Figure 2 contrasts the number of subgroup analyses presented as clinical evidence to the number of subgroup analyses undertaken within economic evaluations. Subgroup analysis in the economic evaluations was only conducted in 17 TAs. In 8 TAs the conclusions from at least one economic subgroup analysis differed to those of the main population, based on cost-effectiveness criteria described by the analysis authors. The most common subgroups included histology (5 lung cancer studies) and mutation status (8 lung cancer studies; Fig. 2).
Figure 2Overview of subgroup analyses presented in clinical and economic submissions. The number in brackets next to the TA number indicates how many clinical studies reported at least some information relating to the use of subgroup analyses. A number of subgroup analysis and results were unavailable for review either due to absence or redaction.
Table 1 describes the structures of the 49 cost-effectiveness models. The majority of models were described as PSMs (total: 39; C: 5; L: 27; O: 7), Markov models (total: 5; C: 3; L: 2; O: 0), or semi-Markov models (total: 4; C: 0; L: 2; O: 2). Figure 3 describes the health states included in each of the models. Health states of partitioned survival models typically reflected PFS, progression and death (37 models), with 6 of these models including response- or treatment-based substates. In contrast, Markov and semi-Markov structures described a range of health states reflecting treatment and clinical status. All models were cohort-level, and given the majority included between 2 and 4 health states only, there was little consideration of individual patient clinical heterogeneity or variability within the model structures themselves (Fig. 3).
Same survival model form chosen for each treatment arm.
TA620
Spline model with treatment covariate
Parametric survival model with treatment covariate
TA483
Model type
Modeled health states
Modeled health state transitions
Outcomes informing health state transitions
Nondeath transitions
Death transitions
Technology assessment
Markov model
1st line, 2nd line, 3rd line, postresection, death
1st line -> post resection; 1st line -> 2nd line; 1st line -> death; post resection -> death; 2nd line -> 3rd line; 2nd line -> death; 3rd line -> death
Progression-free, first subsequent treatment, second subsequent treatment, death
PFS -> First subsequent treatment; PFS -> death; first subsequent treatment -> second subsequent treatment; first subsequent treatment -> death; second subsequent treatment -> death
ToT, OS
Multivariable parametric survival models with or without treatment coefficients
Multivariable parametric survival models with or without treatment coefficients
TA381
Progression-free, progressed, death
Progression-free -> progression; progression-free -> death; progression -> death
Survival models were only produced for one arm with indirect treatment comparison results used to inform disease progression in other arms.
Exponential transition rates
TA258
Accounting exercise
-
-
OS
-
-
TA61
ACM indicates all-cause mortality; DFS, disease-free survival; OS: overall survival; PFS, progression-free survival; PPS, postprogression survival; PSM, partitioned survival model; TA, technology appraisal; ToT, time on treatment; TTD, time to discontinuation.
TA192, TA402 and TA411: Model structure believed to be incorrectly described as Markov model in submission.
TA118: Model structure not described but assumed to be PSM based on description of parameters.
TA192 and TA258: PFS health state stratified in to 2 substates (“treatment response” and “stable disease”) based on proportions at model initiation.
TA242: Some (but not all) comparator survival estimates informed by survival ratios applied to parametric curves of other arms (holding shape parameters constant).
TA212, TA307 and TA611: PFS health state stratified in to 2 substates (“on treatment” and “post-treatment”) using parametric survival models.
TA374: Two populations modeled where piecewise spline models used for one population and piecewise parametric models used for one population.
TA381: Unclear whether single models were used for certain transitions (ie, treatment independent transitions) as the submission contains contradictory statements; the external review group report states that apart from time to first event, all other transitions were set the same for each treatment arm.
TA411: PFS health state stratified in to 3 substates (“on induction treatment,” “off treatment,” “receiving maintenance treatment”) using parametric survival models.
TA484: TTD used as proxy for PFS.
TA528: Model states that mean survival estimates are used (therefore not strictly a PSM as an area-under-the-curve approach not utilized); however, parametric survival curves are used to assess PFS and OS so it has been included in the PSM section.
TA536: PFS stratified in to 2 substates (“patients with brain metastases” and “patients without brain metastases”) although it is unclear how patients are stratified.
∗ Underlined transitions denote those that are modeled with different rates across each treatment arm.
† Same survival model form chosen for each treatment arm.
‡ Survival models were only produced for one arm with indirect treatment comparison results used to inform disease progression in other arms.
With respect to treatment heterogeneity, 42 (C: 4; L: 31; O: 7) economic analyses used data from a single clinical study relating to the first modeled line of treatment only, relying on either clinical expert opinion or validation against published studies with longer-term follow-up to justify extrapolation choices. Appendix 4 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.12.022 demonstrates the growing importance of accurate clinical extrapolation; across the 72 clinical studies for which information on the maturity of clinical data was available, 46% (33 of 72) of studies had observed events in <50% of patients at the time of analysis (17% [12 of 72] had observed events in <25% of patients). Furthermore, there appears to be no discernible relationship between the length of follow-up of the clinical studies and the choice of modeling structure, with lung cancer studies, as expected due to their comparatively lower survival rate, observing the shortest periods of follow-up on average.
The modeling of treatment pathways is described in Table 2, alongside additional context with respect to the modeled population. Despite real-world potential for multiple subsequent therapies across many of the reviewed indications, 20% (10 of 49) of models did not include subsequent therapy at all and 71% (35 of 49) included only one explicit subsequent line of therapy (not including best supportive care). Of those models that included the impact of subsequent therapies, this impact was limited to cost accrual in 77% (30 of 39) of models and to cost accrual and utility values in 18% (7 of 39) of models; subsequent therapy impacted disease progression, cost accrual and utility values in the remaining 2 models (Tables 1 and 2).
Subsequent therapy is defined as either targeted therapy, chemotherapy, surgery, or radiotherapy and does not capture the modeling of best supportive care.
For NSCLC, “advanced or metastatic” includes “recurrent disease” in TA347 and, for ovarian cancer, “advanced” includes one appraisal that looked at stage III/IV disease (TA284).
Mutation-specific
CRC
No
Dukes stage C
No
1
Cost only
Mean treatment duration
TA100
Metastatic
No
None
-
Explicit number of cycles capped by OS
TA61
ToT KM curve
TA212
Yes
1
Cost, utility, disease progression
Initial treatment modeled with own health state
TA439 (external review group)
2
Cost, utility, disease progression
Initial treatment modeled with own health state
TA439 (company)
Yes
Metastatic
No
None
-
Treatment to progression or mean treatment duration
TA242
No
1
Cost only
Mean treatment duration
TA307
No
1
Cost only
Treatment to progression
TA405
Yes
2
Cost only
Mean treatment duration
TA118
NSCLC
Both
Advanced or metastatic
Yes
1
Cost and utility
TTD modeled independently using KM data
TA529
Cost only
Parametric ToT model
TA584
No
Advanced or metastatic
No
1
Cost only
Cyclic discontinuation rate capped by specific no. of cycles
TA181
Parametric ToT model
TA557
ToT KM curve
TA411
Treatment to progression
TA600
Yes
None
-
Parametric ToT model
TA258
1
Cost and utility
Mean treatment duration beyond progression
TA406, TA621
Treatment to progression
TA310
Cost only
Parametric ToT model
TA500
Treatment to progression
TA192, TA531, TA536
2
Cost only
Treatment to progression
TA595
Yes
Advanced or metastatic
No
None
-
Cyclic discontinuation rate capped by specific number of cycles
TA124
Treatment to progression
TA374
1
Cost and utility
Parametric ToT model
TA520
Treatment to progression
TA483, TA484
Cost only
Cyclic discontinuation rate capped by progression
TA347
Parametric ToT model
TA402
ToT KM curve
TA578
Treatment to progression
TA190, TA227, TA403
Yes
None
Cost only
Independent mean duration beyond progression
TA571
1
Cost only
Treatment to progression
TA416, TA428
Any
Yes
None
-
Treatment to progression
TA395
SCLC
Yes
Relapsed
No
None
-
Specific number of treatment cycles
TA184
Ovarian cancer
Any
Any
No
-
-
-
TA55
No
Advanced
No
None
Cost only
Mean treatment duration
TA284 (Model 1)
1
Cost only
Mean treatment duration
TA285
TA284 (Model 2)
Yes
Any
No
1
Cost only
Treatment to progression
TA389
Advanced
Yes
1
Cost only
Parametric ToT model
TA598
High-grade
No
1
Cost only
Parametric ToT model
TA528, TA611
Yes
1
Cost only
Parametric ToT model
TA620
2
Cost only
Initial treatment modeled with own health state
TA381
CRC indicates colorectal cancer; KM, Kaplan-Meier; NSCLC, non-small cell lung cancer; SCLC, small cell lung cancer; TA, technology appraisal; ToT, time on treatment.
TA284 (Model 2), TA416 and TA528: Average number of subsequent therapies received was greater than 1 but modeled within one subsequent line of therapy.
TA307: Despite including substates within the progression-free health state describing treatment status (“on treatment” versus “off treatment”), treatment costs were modeled based on a mean treatment duration.
TA406: Only a proportion of patients were assumed to receive therapy postprogression; remaining patients ceased treatment at progression.
TA439: Patients could also receive curative resection after first line treatment, independent of other treatment lines.
∗ For NSCLC, “advanced or metastatic” includes “recurrent disease” in TA347 and, for ovarian cancer, “advanced” includes one appraisal that looked at stage III/IV disease (TA284).
† Subsequent therapy is defined as either targeted therapy, chemotherapy, surgery, or radiotherapy and does not capture the modeling of best supportive care.
The majority of TAs (32 of 49) acknowledged differences between the patient characteristics or treatment pathways used in the clinical studies and routine clinical practice. Furthermore, within individual TAs, patient heterogeneity was particularly noticeable in those that included more than one pivotal clinical study (Appendix 5 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.12.022). In such TAs, where data were reported, the average range of median ages was 4.9 years (C: 4.9; L: 5.7; O: 3.0), with average ranges of 16% (C: 12%; L: 18%), 10% (C: 5%; L: 11%; O: 14%), and 21% (C: 31%; L: 20%; O: 12%) for the proportion of patients that were male, had an ECOG-PS of 0 or 1 and were white or Caucasian, respectively; the largest ranges in any single TA were 18.8 years (C: 18.5; L: 18.8; O: 3.1), 65% (C: 29%; L: 65%), 57% (C: 11%; L: 57%; O: 37%), and 99% (C: 58%; L: 99%; O: 13%), respectively.
Figure 4 describes the most common exclusion criteria used by the clinical studies and gives an overall impression of the selective nature of clinical trials and how the trial populations might differ from those found in routine clinical practice. A total of 73 of 94 clinical studies described exclusion criteria. The most common criteria not related to the treatment indication (eg, treatment history, histology, mutation status, etc) were a history of other malignancies (35 of 73 studies) and a history of cardiac problems (31 of 73). Such exclusion criteria would likely ostracize a significant proportion of cancer patients in UK clinical practice that are expected to have comorbid conditions (Fig. 4).
Finally, few TAs attempted to investigate the impact of clinical heterogeneity through disease progression modeling, with only 5 models (all evaluating targeted therapies), including clinical covariates within their estimation of PFS and OS disease progression estimates (Table 1). Although not exclusive to these TAs, patients in the clinical studies associated with 4 out of the 5 TAs were systematically different to the routine clinical practice patients they were representing with respect to their ethnic origin. Of these, 2 TAs (TA406 and TA529) used methods to account for differences between trial and clinical practice populations. These lung cancer TAs generated disease progression survival models that included clinical covariates based on data from the clinical study. Subsequently, these survival models were used to predict clinical outcomes for the cost-effectiveness model at covariate values corresponding to those observed in published studies deemed representative of UK clinical practice. In both cases, the following clinical covariates were included in the survival models: race (Asian/non-Asian), ECOG-PS (0 or 1/2), brain metastases at baseline (yes/no), age (≥65/<65 years), sex, smoking status (never smoked/former or current smoker), and adenocarcinoma at baseline (yes/no).
Discussion
This is the first review to consider patient and treatment effect heterogeneity in UK TAs of colorectal, lung, and ovarian cancer. The review highlighted that whilst many clinical studies undertook subgroup analyses, only a small number of economic evaluations considered these subgroups further in modeling analyses. Although, such findings must be caveated with the potential for publication bias and the under-reporting of negative results in economic submissions. This lack of representation in economic analyses was notable as several clinical subgroup analyses presented results contradicting the overall population findings, although statistical significance was rare. In addition, it was common to find a significant and positive treatment effect in the overall population analysis with subgroup analyses failing to demonstrate the same effect (or achieve significance). Statistical significance was not a focus of this study due to inconsistent definitions across TAs, underreporting of results (eg, commercial-in-confidence redaction) and small sample sizes. Subgroup analyses provide evidence for improved allocation of healthcare resources, with the potential to tailor reimbursement recommendations to specific patient groups where evidence for effectiveness is either very strong or very weak. Guidance is available on when to apply subgroup analysis in cost-effectiveness evaluation, with such analyses continuing to be a preferred first step to evaluate patient and treatment heterogeneity.
Review of the economic models further showed that the ability to incorporate heterogeneity in the economic evaluations was hampered by an overreliance on relatively simplistic cohort-based modeling structures. It was found for example, that most economic models used univariable disease progression estimates and represented disease progression through just 3 health states. Of particular note is what may be perceived as a systemic reliance on PSMs to demonstrate the economic impact of new cancer therapies. PSMs are designed for use with near complete clinical data and relatively simplistic treatment and disease pathways.
Inherently, models with simple structures lack flexibility and therefore do not lend themselves to the modeling of heterogeneity, particularly those of patient characteristics and treatment effects.
The NICE Decision Support Unit technical documentation suggests patient-level simulation should be considered when the number of categories required to define patient groups with homogeneous outcomes becomes large.
NICE decision support unit technical support documents.
in: NICE DSU Technical Support Document 15: Cost-Effectiveness Modeling Using Patient-Level Simulation. National Institute for Health and Care Excellence (NICE),
London2014
Patient-level simulation is also advocated for consideration when the likelihood of future events (eg, death) are dependent on the time since previous events (eg, disease progression). Notably, these criteria are true of certain TAs in this review, with the latter being particularly relevant to cancers for which curative treatment is available (colorectal and ovarian cancer) and those where disease progression is particularly influential over patient prognosis (lung cancer). However, patient-level models often have greater computational requirements, with respect to the data required, the time taken to run analyses and the complexity of such analyses. As such, trade-offs between analyst time, computation time, and the requirements of the decision problem may be required. To justify final model selection, a checklist approach could be used to characterize the decision problem, the data, computational limitations, and other relevant issues. This approach offers several advantages over algorithmic model selection, including the ability to summarize strengths and weaknesses of modeling approaches within the context of the decision problem aiding critical appraisal of model choice, and the avoidance of prescriptive decisions that create the illusion that only one model type suits a particular decision problem.
The review additionally identified weaknesses in the methods used to extrapolate clinical endpoints to policy-relevant time horizons. Extrapolations rarely considered clinical mechanisms for estimating disease progression, and instead relied predominantly on within-trial statistical goodness-of-fit output, visual inspection, and comparison to historical data. A potential solution is to use risk equations to aid in the extrapolation of outcomes beyond the trial phase using clinical and treatment history data. This approach is commonly applied in other disease areas such as diabetes, cardiovascular disease, and chronic kidney disease.
The derivation of these risk equations is typically undertaken from large real-world observational datasets and may also assist in alleviating concerns over the real-world applicability of outcome extrapolation. These methods may have previously been overlooked in cancer due to the potential for low quality of recording of data, the propensity for cancer treatments to fundamentally change the course of disease and for the prevalence of highly unique cancer subpopulations defined by genetic variation.
Combining risk equations with more flexible and sophisticated modeling methods will provide greater consideration, and understanding, of real-world patient and treatment effect heterogeneity, and go some way to addressing historical limitations.
Finally, while appraisals acknowledged differences between the clinical studies from which their evidence was based and routine clinical practice, few summarized these differences quantitatively. Clearly there are tensions between the representativeness of clinical trials and the necessity of a trial to have homogeneous groups of patients to enable comparison between groups.
However, homogeneity does not need to come at the expense of the natural heterogeneity observed in the population of interest, which may become the case when extensive exclusion criteria are applied. Addressing such issues is not straightforward, with patient safety, ethical issues, and sample size considerations at the forefront of concern. As a relatively simple and practical initial step, we suggest that trial investigators could improve reporting by making available more evidence on clinical outcomes of stratification subgroups, alongside encouraging access to individual patient data (IPD) for research. Subsequently, we would advocate the addition of a more explicit and structured comparison of routine clinical practice and trial patient populations within TAs. Such a comparison might take the form of a quantitative side-by-side summary of the clinical and demographic characteristics of patients from both groups, based on relevant UK clinical practice datasets and the clinical studies informing the appraisal. We would encourage the adoption of such an approach as standard practice within TAs to provide relevant parties with a transparent overview of both the relevance and the extent of any differences.
Furthermore, very few TAs used methods to adjust cancer outcomes to account for differences between trial and routine clinical practice, even in the most recent TAs. This is particularly relevant given observed differences between cancer outcomes in these settings, particularly amongst PFS and OS outcomes,
Predicting real-world effectiveness of cancer therapies using overall survival and progression-free survival from clinical trials: empirical evidence for the ASCO value framework.
and the need for policymakers to understand these differences to inform policy recommendations and guidelines. Two TAs used a form of simulated treatment comparison (STC), generating survival models that included clinical covariates based on IPD from the clinical study, and subsequently using these to predict clinical outcomes for the cost-effectiveness model at covariate values deemed representative of UK clinical practice. Methods such as STC and matching-adjusted indirect comparison, aim to reduce bias in treatment comparisons by using IPD from the clinical studies to provide indicative estimates of the likely outcomes in different settings, and may be used to address the above concerns.
Matching-adjusted indirect comparison adjusts average population-level outcomes by applying weights to IPD from the clinical study, using larger weights for patients that more closely match those of routine clinical practice, while STC uses regression equations to adjust estimates.
Clearly these suggestions should acknowledge the current constraints of the NICE review process, which is subject to strict timelines. For example, IPD needed for patient-level simulation or risk equation development may not be available to researchers. This raises the questions of how NICE should resource future TAs to enable them to better incorporate heterogeneity and related equity concerns. Another limitation of the review is the pragmatic decision to consider 3 cancer sites. Additional research is required before we can generalize across all cancers and across economic evaluations of cancer outside the remit of NICE TAs.
Conclusion
This study highlights a relative paucity of information relating to the assessment of heterogeneity in UK cancer TAs and identifies a mostly unjustified reliance on relatively simplistic modeling frameworks. If heterogeneity considerations are to be included in TA frameworks, and the complexity of treatment and disease pathways reflected in economic analyses, there is a requirement to embrace more flexible modeling approaches and to further research real-world heterogeneity.
Article and Author Information
Author Contributions:Concept and design: Ward, Medina-Lara, Mujica-Mota, Spencer
Acquisition of data: Ward, Medina-Lara, Mujica-Mota
Analysis and interpretation of data: Ward
Drafting of the manuscript: Ward, Medina-Lara, Mujica-Mota, Spencer
Critical revision of the paper for important intellectual content: Ward, Medina-Lara, Mujica-Mota, Spencer
Provision of study materials or patients: Ward, Medina-Lara, Mujica-Mota
Obtainingfunding: Spencer
Administrative, technical, or logisticsupport: Ward
Supervision: Medina-Lara, Mujica-Mota, Spencer
Conflict of Interest Disclosures: Mr Ward and Dr Spencer reported receiving grants from the Dennis and Mereille Gillings Foundation during the conduct of the study. No other disclosures were reported.
Funding/Support: This research has been supported by a PhD studentship grant for Mr Ward from the Dennis and Mereille Gillings Foundation.
Role of theFunder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Acknowledgment
This research arises from the Dennis and Mireille Gillings Foundation funded ERICA trial that explores the use of electronic risk assessments to help identify possible cancers in primary care. Thomas Ward is a postgraduate researcher and Anne Spencer is a co-applicant on this trial. This research is linked to the CanTest Multi-institution Collaborative, which is funded by Cancer Research UK (C8640/A23385), of which Mr Ward is an affiliated postgraduate researcher and Dr Spencer is senior faculty. We would like to thank Willie Hamilton, MD, Professor of Primary Care Diagnostics, University of Exeter, Exeter, England, UK, for his continued support and clinical advice on the choice of cancer areas as principal investigator of the ERICA trial.
Global surveillance of trends in cancer survival 2000-14 (CONCORD-3): analysis of individual records for 37 513 025 patients diagnosed with one of 18 cancers from 322 population-based registries in 71 countries.
Estimation and reporting of heterogeneity of treatment effects.
in: Developing a Protocol for Observational Comparative Effectiveness Research: A User's Guide. Agency for Healthcare Research and Quality (US),
Rockville (MD)2013
The effect of socioeconomic deprivation on the association between an extended measurement of unhealthy lifestyle factors and health outcomes: a prospective analysis of the UK Biobank cohort.
NICE decision support unit technical support documents.
in: NICE DSU Technical Support Document 15: Cost-Effectiveness Modeling Using Patient-Level Simulation. National Institute for Health and Care Excellence (NICE),
London2014
Predicting real-world effectiveness of cancer therapies using overall survival and progression-free survival from clinical trials: empirical evidence for the ASCO value framework.
00146-7&id=gr1.jpg){kind=link}
00146-7&id=gr2.jpg){kind=link}
00146-7&id=gr3.jpg){kind=link}
00146-7&id=gr4.jpg){kind=link}