If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
A Comparison of Alternative Network Meta-Analysis Methods in the Presence of Nonproportional Hazards: A Case Study in First-Line Advanced or Metastatic Renal Cell Carcinoma
Network meta-analysis (NMA) of time-to-event outcomes based on constant hazard ratios can result in biased findings when the proportional hazards (PHs) assumption does not hold in a subset of the trials. Several NMA methods that do not rely on the PH assumption have been proposed. Nevertheless, their application to health technology assessment submissions has been limited, possibly due to a lack of familiarity among researchers performing cost-effectiveness analysis of oncology drugs.
•
NMA methods that allow for non-PH may provide relative treatment effect estimates that better reflect the evidence. We provide a systematic overview and comparison of methods using a case study to increase familiarity. Given the impact of NMA model choice on findings, we propose a stepwise process to select appropriate NMA models. This is especially important when considering the most flexible NMA models.
•
When relative treatment effects in terms of time-to-event outcomes need to be estimated based on an NMA of randomized controlled trials and the PH assumption is uncertain in one or more of the randomized controlled trials, alternative parametric NMA methods that do not rely on this assumption are recommended.
Abstract
Objectives
Network meta-analysis (NMA) of time-to-event outcomes based on constant hazard ratios can result in biased findings when the proportional hazards (PHs) assumption does not hold in a subset of trials. We aimed to summarize the published non-PH NMA methods for time-to-event outcomes, demonstrate their application, and compare their results.
Methods
The following non-PH NMA methods were compared through an illustrative case study in oncology of 4 randomized controlled trials in terms of progression-free survival and overall survival: (1) 1-step or (2) 2-step multivariate NMAs based on traditional survival distributions or fractional polynomials, (3) NMAs with restricted cubic splines for baseline hazard, and (4) restricted mean survival NMA.
Results
For progression-free survival, the PH assumption did not hold across trials and non-PH NMA methods better reflected the relative treatment effects over time. The most flexible models (fractional polynomials and restricted cubic splines) fit better to the data than the other approaches. Estimated hazard ratios obtained with different non-PH NMA methods were similar at 5 years of follow-up but differed thereafter in the extrapolations. Although there was no strong evidence of PH violation for overall survival, non-PH NMA methods captured this uncertainty in the relative treatment effects over time.
Conclusions
When the PH assumption is questionable in a subset of the randomized controlled trials, we recommend assessing alternative non-PH NMA methods to estimate relative treatment effects for time-to-event outcomes. We propose a transparent and explicit stepwise model selection process considering model fit, external constraints, and clinical validity. Given inherent uncertainty, sensitivity analyses are suggested.
In oncology, the efficacy of a new intervention is often assessed using time-to-event outcomes, such as progression-free survival (PFS) and overall survival (OS), and summarized with Kaplan-Meier (KM) curves. The relative treatment effect of one intervention versus another is typically expressed as a hazard ratio (HR) obtained with a Cox model assuming proportional hazards (PHs).
Other relative treatment effect measures that may be used include the difference in median survival, the odds ratio of survival at a specific time point, or the difference in restricted mean survival time (RMST) up to a specific time point (t∗).
Only the HR and RMST (dependent on time point selected) capture the relative treatment effect over the complete follow-up of the trial.
Many health technology assessment agencies require manufacturers to undertake cost-effectiveness analysis (CEA) of their new interventions versus the current standard of care in the respective healthcare system and versus other competing interventions not included in the randomized controlled trial (RCT) program. For interventions aiming to improve survival, this frequently requires extrapolation of the RCT data over time using parametric survival distributions to obtain estimates of expected survival. When the evidence base consists of multiple RCTs, each comparing a subset of the interventions of interest, we need to perform a network meta-analysis (NMA) to obtain relative treatment effects of each intervention versus an overall reference treatment. These estimates can subsequently be applied to the PFS and OS curves of this reference treatment, to predict the PFS and OS curves for all interventions in the evaluation. It is important to realize that both the PFS and OS curves of the reference treatment and the PFS and OS relative treatment effect estimates may need to be extrapolated over time.
NMAs of time-to-event outcomes can be performed based on the trial-specific reported HRs, but these estimates may be biased when the PH assumption does not hold. Trials comparing treatments with different mechanisms of action, or short-term versus long-term benefits, tend to violate the PH assumption.
We aimed to summarize the published methods for NMA of time-to-event outcomes that allow for time-varying treatment effects, compare their application with a case study, and discuss the implications for CEA.
NMA Methods That Do Not Rely on the PH Assumption
Four distinct NMA methods for time-to-event outcomes that do not rely on the PH assumption were identified from the literature
Individual patient data network meta-analysis using either restricted mean survival time difference or hazard ratios: is there a difference? A case study on locoregionally advanced nasopharyngeal carcinomas.
Comparative survival benefit of currently licensed second or third line treatments for epidermal growth factor receptor (EGFR) and anaplastic lymphoma kinase (ALK) negative advanced or metastatic non-small cell lung cancer: a systematic review and secondary analysis of trials.
Programmed death-1 or programmed death ligand-1 blockade in patients with platinum-resistant metastatic urothelial cancer: a systematic review and meta-analysis.
Individual patient data network meta-analysis using either restricted mean survival time difference or hazard ratios: is there a difference? A case study on locoregionally advanced nasopharyngeal carcinomas.
All methods use time-to-event data based on individual patients either reconstructed from Kaplan-Meier using Guyot algorithm or based on observed individual patient data.
Relative treatment effect and how is non-PH addressed
Uses a multivariate relative treatment effect as an alternative to the synthesis of the trial-specific constant HRs. The hazard functions of the interventions in a trial are modeled using parametric distribution and the difference in the parameters are considered the multidimensional relative treatment effect, which are synthesized (and indirectly compared) across studies
Weibull, Gompertz, log-normal, log-logistic
One-step (trial level–specific relative treatment effects and pooled effects are estimated simultaneously)
Bayesian
Approximation with piecewise constant hazards (discrete hazards) according to a binomial likelihood
Multivariate relative treatment effect parameters regarding scale and shape related factors of the survival distribution/function. These relative treatment effect parameters are used to describe time-varying HRs (or odds ratios in case of log-logistic models)
“Allow relative treatment effects to vary by covariates independently of the other treatments in the network of evidence. The relative treatment effect remains constant for any treatment not specified within a hierarchical exchangeable structure… In addition, where possible, different doses also were included as a hierarchical structure with an overall treatment class effect. Constraints were imposed to ensure that the efficacy increased with dose intensity.” Adapted from Owen et al.17
For each arm of every RCT in the network, (recreated), IPD are used to estimate alternative survival distributions. Next, for each distribution, its scale and shape parameters are included in a multivariate NMA to obtain time-varying estimates of relative treatment effects between competing interventions
Weibull, Gompertz, log-normal, log-logistic describing the ln-hazards over time
Two-step (arm-specific survival function parameters are estimated first. Subsequently, these are incorporated in the multivariate NMA)
Step 1 – Frequentist; Step 2 – Bayesian
Exact likelihood corresponding to survival distribution selected
An IPD Royston-Parmar Bayesian NMA model, which provides flexible alternative modeling approach that can accommodate time-dependent effects. The baseline log-cumulative hazards are modeled with restricted cubic splines. HRs are either fixed over time or can be modeled as a function of ln(time)
Restricted cubic splines describing the baseline log-cumulative hazard of each trial
Two-step (described as 1-step but requires orthogonalized basis function of study-specific splines as input for NMA)
Bayesian framework for NMA but first step in frequentist framework
General likelihood using “zeros trick” using probability density function of Poisson distribution
If we wish to implement a likelihood representing the flexible fractional polynomials or cubic splines in WinBUGS, we can use the “zeros” trick17 where a data set comprising entirely of zeros is given a Poisson distribution with its parameter defined equal to the negative log-likelihood (plus a sufficiently large constant); the log-likelihood function corresponding to the fractional polynomial or spline is then written algebraically in the WinBUGS code.
Constant HRs represented with a single basic parameter by treatment. As an extension, HR can vary over time by adding extra parameters for the interaction between treatment and ln(time)
Individual patient data network meta-analysis using either restricted mean survival time difference or hazard ratios: is there a difference? A case study on locoregionally advanced nasopharyngeal carcinomas.
Comparative survival benefit of currently licensed second or third line treatments for epidermal growth factor receptor (EGFR) and anaplastic lymphoma kinase (ALK) negative advanced or metastatic non-small cell lung cancer: a systematic review and secondary analysis of trials.
Programmed death-1 or programmed death ligand-1 blockade in patients with platinum-resistant metastatic urothelial cancer: a systematic review and meta-analysis.
RMS estimated based on (1) pseudo-values based on KM; (2) Poisson-gamma frailty model
Frequentist
No
No
No
AUC indicates area under the curve; HR, hazard ratio; IPD, individual patient data; KM, Kaplan-Meier; NMA, network meta-analysis; PH, proportional hazard; RCT, randomized controlled trial; RMS, restricted mean survival; RMST, restricted mean survival time.
∗ All methods use time-to-event data based on individual patients either reconstructed from Kaplan-Meier using Guyot algorithm or based on observed individual patient data.
† “Allow relative treatment effects to vary by covariates independently of the other treatments in the network of evidence. The relative treatment effect remains constant for any treatment not specified within a hierarchical exchangeable structure… In addition, where possible, different doses also were included as a hierarchical structure with an overall treatment class effect. Constraints were imposed to ensure that the efficacy increased with dose intensity.” Adapted from Owen et al.
where a data set comprising entirely of zeros is given a Poisson distribution with its parameter defined equal to the negative log-likelihood (plus a sufficiently large constant); the log-likelihood function corresponding to the fractional polynomial or spline is then written algebraically in the WinBUGS code.
proposed a multivariate relative treatment effect measure that describes how the HRs change over time. Parametric survival functions are used to model the hazard functions of the interventions in each trial, and the multivariate relative treatment effect is based on the difference in its parameters, which are pooled and indirectly compared across trials. Using multiple parameters for the relative treatment effects avoids the need for the PH assumption and provides flexibility to fit the data closely.
presented a variation by assuming exchangeable covariate interaction effects across treatments (ie, histology, programmed death ligand 1 expression, and mutation status), rather than treatment-specific or constant covariate interaction effects. We label these methods as “1-step” approaches, where trial-specific survival curve parameters and pooled relative treatment effects across studies are estimated simultaneously. These analyses are based on the discrete hazards (or conditional survival probabilities) derived from individual patient data (IPD)
With this approach, parametric survival distributions are fitted for each arm of each trial in the first stage. In the second stage, these parameters are synthesized with a multivariate NMA model.
In the first step, an exact likelihood distribution for survival times, for example, Weibull, Gompertz, log-logistic, and log-normal distributions, is used. Chan et al
Flexible parameteric proportional-hazards and proportional-odds models for censored survival data, with application to prognostic modelling and estimation of treatment effects.
which can be estimated more efficiently as an alternative to a Cox-based NMA of IPD. The method focused primarily on constant HRs but can be extended with an interaction between treatment and ln(time) to estimate time-varying HRs, where HR develops as a linear function of ln(time). As such, this model does not use RCS to describe the HRs over time.
Individual patient data network meta-analysis using either restricted mean survival time difference or hazard ratios: is there a difference? A case study on locoregionally advanced nasopharyngeal carcinomas.
Comparative survival benefit of currently licensed second or third line treatments for epidermal growth factor receptor (EGFR) and anaplastic lymphoma kinase (ALK) negative advanced or metastatic non-small cell lung cancer: a systematic review and secondary analysis of trials.
Programmed death-1 or programmed death ligand-1 blockade in patients with platinum-resistant metastatic urothelial cancer: a systematic review and meta-analysis.
use a 2-step process. First, the differences in RMST between treatments in a trial are estimated based on (reconstructed) IPD. Next, these estimates are synthesized with a standard NMA model, assuming the differences in RMST within each trial are normally distributed. The choice for the specific time point (t∗) is suggested to maximize the use of available data, while minimizing the need for extrapolation.
If extrapolation is not required, the difference in the RMST is estimated nonparametrically using the area under the KM curve for each trial, and the delta method is used to estimate the variance of the difference in the RMST.
Our illustrative example concerns a comparison of immunotherapies (IOs) with tyrosine kinase inhibitors for the first-line treatment of relapsed or stage IV clear cell renal cell carcinoma. The PH assumption of the RCTs, which formed a star-shaped network (Fig. 1), has been questioned by health technology assessment agencies.
Cabozantinib versus everolimus, nivolumab, axitinib, sorafenib and best supportive care: a network meta-analysis of progression-free survival and overall survival in second line treatment of advanced renal cell carcinoma.
Axitinib, cabozantinib, everolimus, nivolumab, sunitinib and best supportive care in previously treated renal cell carcinoma: a systematic review and economic evaluation.
Comparative efficacy of first-line immune-based combination therapies in metastatic renal cell carcinoma: a systematic review and network meta-analysis.
Indirect treatment comparisons including network meta-analysis: lenvatinib plus everolimus for the second-line treatment of advanced/metastatic renal cell carcinoma.
Survival outcomes and independent response assessment with nivolumab plus ipilimumab versus sunitinib in patients with advanced renal cell carcinoma: 42-month follow-up of a randomized phase 3 clinical trial.
Updated efficacy results from the JAVELIN Renal 101 trial: first-line avelumab plus axitinib versus sunitinib in patients with advanced renal cell carcinoma.
Pembrolizumab plus axitinib versus sunitinib monotherapy as first-line treatment of advanced renal cell carcinoma (KEYNOTE-426): extended follow-up from a randomised, open-label, phase 3 trial.
All NMA models evaluated were implemented assuming fixed effects rather than random effects, given the small number of studies in the network.
NMA based on constant HRs
An NMA assuming constant HRs was performed for comparative purposes.
One-step multivariate NMA
In the multivariate NMA using traditional parametric models (ie, Weibull [equivalent to a first-order FP with a log(time) transformation (P1 = 0)], Gompertz [first-order FP without transformation of time (P1 = 1)], log-logistic, and log-normal), relative treatment effects were applied to the scale and shape parameters. For the second-order FPs, we focused on models that were extensions of Weibull or Gompertz (P1 = 0 or 1) models to facilitate interpretation, limiting power transformations of time for this additional shape parameter P2 from −1, −0.5, 0, 0.5, 1 with P2 = 0 defined as ln(time).
We explored alternative parameterizations of the relative treatment effects for the second-order FP models with the treatment having an impact on (1) only the scale (ie, constant HR), (2) scale and only 1 shape, and (3) scale and both shape parameters. A binomial likelihood was used for the discrete hazards data.
The 2-step models evaluated were consistent with the 1-step multivariate NMA. However, for second-order FPs, we assumed the treatment had an impact on the scale and both shape parameters. An exact likelihood (ie, Weibull, Gompertz, log-logistic, log-normal, or FP) was used for the trial-specific analysis of the pseudo-IPD in the first step, followed by a multivariate normal likelihood for the NMA model in the second step.
NMAs with RCS for baseline hazard
For the NMAs with RCS (RCS NMA), the knots were placed at equally spaced percentiles of the uncensored survival times based on the pseudo-IPD for each trial, with boundary knots at the minimum and maximum values of the uncensored survival times.
Flexible parameteric proportional-hazards and proportional-odds models for censored survival data, with application to prognostic modelling and estimation of treatment effects.
The relative treatment effects were either (1) constant HR over time or (2) time-varying HR based on an interaction between HR and ln(time). The exact likelihood was specified according to the pseudo-IPD assuming RCS.
Restricted mean survival time NMA
For the RMST NMA, the area under the curve was defined up to t∗ at 2 or 5 years according to the shortest (JAVELIN) and longest (CheckMate 214) follow-up across trials, respectively. For t∗ = 2 years, RMST was estimated based on KM curves (ie, no extrapolation required). For t∗ = 5 years, the best-fitting RCS function per arm (1, 2, or 3 internal knots) was used. RCS was selected to provide most flexible fit to the observed data. A normal likelihood was used in the second step to combine the differences in RMST.
A Bayesian framework was used for all NMA models, with parameters estimated using Markov chain Monte Carlo implemented in the JAGS (v4.1) or Stan software package (RStan v2.21.2). R (v3.6.1) packages were used for trial-specific estimates in the 2-step NMA (flexsurv), the RCS NMA (flexsurvspline), and RMST NMA (survRM2). Additional details regarding the likelihoods, data setup, and priors used for the implementation of the NMA methods for the case study are presented in Appendix D in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.017.
Model selection by NMA method
We identified the most appropriate 1 or 2 models for each NMA method using the criteria in Table 2.
Individual patient data network meta-analysis using either restricted mean survival time difference or hazard ratios: is there a difference? A case study on locoregionally advanced nasopharyngeal carcinomas.
Comparative survival benefit of currently licensed second or third line treatments for epidermal growth factor receptor (EGFR) and anaplastic lymphoma kinase (ALK) negative advanced or metastatic non-small cell lung cancer: a systematic review and secondary analysis of trials.
Programmed death-1 or programmed death ligand-1 blockade in patients with platinum-resistant metastatic urothelial cancer: a systematic review and meta-analysis.
NICE DSU technical support document 14: undertaking survival analysis for economic evaluations alongside clinical trials - extrapolation with patient-level data. National Institute for Health and Care Excellence (NICE).
to assess the development of the hazard over time and whether the PH assumption held for each trial. Next, the goodness of fit of the competing models was assessed. For the 1-step multivariate NMA method, model selection was based on the complete evidence base using the deviance information criterion (DIC).
For the other 3 methods, alternative survival models were fitted to (each arm of) the individual trials and the model of choice was based on the sum of the Akaike information criterion across trials. For the multivariate NMA and the RMST NMA, DICs can be used to compare between fixed- and random-effects models. Nevertheless, given the small number of studies in the network, only a fixed-effects NMA was considered. For the RCS NMA, the DIC was used to decide between constant and time-varying HRs. We also visually compared age-matched general mortality rates
with the extrapolated survival and hazards plots up to 5 and 40 years obtained with the alternative models to assess the upper bound of the extrapolations that would be possible.
Table 2Overview of the NMA model selection process.
Individual patient data network meta-analysis using either restricted mean survival time difference or hazard ratios: is there a difference? A case study on locoregionally advanced nasopharyngeal carcinomas.
Comparative survival benefit of currently licensed second or third line treatments for epidermal growth factor receptor (EGFR) and anaplastic lymphoma kinase (ALK) negative advanced or metastatic non-small cell lung cancer: a systematic review and secondary analysis of trials.
Programmed death-1 or programmed death ligand-1 blockade in patients with platinum-resistant metastatic urothelial cancer: a systematic review and meta-analysis.
); (2) Quantile-quantile plot of times of survival percentiles; (3) Log-cumulative hazard plots vs log(time); and (4) Plot of smoothed empirical hazard vs time
Select arm or trial-level model
Not applicable since study-level estimates and indirect comparisons performed simultaneously
Not applicable since study-level estimates and indirect comparisons performed simultaneously
Arm-level model: AIC per arm and across full network and visual inspection per arm
Arm-level model: AIC per arm and across full network and visual inspection per arm
Trial-level models: AIC per trial and visual inspection per trial
Arm-level models when extrapolation is required for (B): AIC per arm and visual inspection per arm
Given the star-shaped network with only 1 trial per contrast, we were able to use the relative treatment effects as estimated from the NMA model to predict trial-specific hazards and survival (by applying them to trial-specific baseline), which were visually inspected and compared with the observed hazards and survival. Nevertheless, this step is not proposed by authors in the original papers and would not be feasible if there were > 1 study informing each comparison or if there was indirect evidence.
DICs across FP models can be compared since uses same data and same likelihood to inform model and parameterization of relative treatment effects (PH vs non-PH)
Given the star-shaped network with only 1 trial per contrast, we were able to use the relative treatment effects as estimated from the NMA model to predict trial-specific hazards and survival (by applying them to trial-specific baseline), which were visually inspected and compared with the observed hazards and survival. Nevertheless, this step is not proposed by authors in the original papers and would not be feasible if there were > 1 study informing each comparison or if there was indirect evidence.
DICs across parametric models cannot be compared since data (ie, parameters of different distributions) is not the same across models (likelihood is same)
DICs can be used to compare FE versus RE models versus with or without treatment by covariate interactions.
DICs across spline models can be compared since the model uses same data and likelihood to inform model and parameterization of relative treatment effects (ie, PH vs non-PH)
Only 1 RMST model evaluated for extrapolated survival (B); If alternative extrapolations were used to inform RMST, DICs across RMST models could not be compared (data would differ)
DICs can be used to compare FE versus RE models versus with or without treatment by covariate interactions.
Green checkmark indicates selection criteria was applicable; Red 'x' indicates the selection criteria was not applicable.
AIC indicates Akaike information criterion; DIC, deviance information criterion; FE, fixed effect model; FP, fractional polynomials; HR, hazard ratio; KM, Kaplan-Meier; NMA, network meta-analysis; PH, proportional hazards; RCS, restricted cubic splines; RE, random effects model; RMST, restricted mean survival time.
∗ DICs across NMA methods cannot be compared because data and likelihoods differ.
† Given the star-shaped network with only 1 trial per contrast, we were able to use the relative treatment effects as estimated from the NMA model to predict trial-specific hazards and survival (by applying them to trial-specific baseline), which were visually inspected and compared with the observed hazards and survival. Nevertheless, this step is not proposed by authors in the original papers and would not be feasible if there were > 1 study informing each comparison or if there was indirect evidence.
‡ DICs can be used to compare FE versus RE models versus with or without treatment by covariate interactions.
Comparison of NMA methods regarding treatment effects
The alternative NMA methods were compared based on the selected models regarding the following short-term (5 years, longest trial follow-up) and long-term (40 years, lifetime horizon
) estimates: (1) relative treatment effects for each intervention versus SUN and uncertainty based on the 95% credible intervals, (2) survival proportions by treatment (obtained by multiplying the average hazard function of SUN across the trials in the network by the estimated HRs), and (3) difference in RMST per intervention versus SUN
Selection of the specific NMA models for each of the NMA methods is summarized in Appendix Tables E2 and E3 for PFS and in Tables E4 and E5 for OS in Appendix E in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.017. For both 1- and 2-step multivariate NMA methods, models based on the log-normal and Gompertz distributions were selected among the traditional survival models. When the set of competing models to consider for the 1-step NMA method was expanded to include the more flexible second-order FPs, the models describing ln-hazards over time as a function of ln(time) and 1/time (FP P1 = 0, P2 = −1) with a relative treatment effect on the scale and either the first or second shape parameter were best fitting. For the RCS NMA method, a model with study-specific number of knots and a time-varying HR was selected (number of knots: CheckMate 214, 2; COMPARZ, 2; JAVELIN, 3; KEYNOTE-426, 2). Please note that Akaike information criteria/DICs could not be compared across NMA methods because of differences in data (ie, pseudo-IPD vs discrete hazards derived from pseudo-IPD).
Comparison of NMA methods regarding relative treatment effects
Estimated time-varying PFS HRs of the competing interventions versus SUN up to 5 years (longest follow-up in the trials) obtained with different non-PH NMA methods were relatively similar, except for HRs obtained with the multivariate NMA methods assuming Gompertz-distributed survival times (Fig. 2). Applying the time-varying HRs from each NMA method to the average SUN hazard function results in the corresponding PFS curves by treatment (Appendix Fig. F1 and Table F1 in Appendix F in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.017). The difference in RMST per intervention relative to SUN up to 5 years consistently suggested that PEM + AXI and AVE + AXI were the most efficacious treatment combinations, followed closely by NIVO + IPI, although confidence intervals were wide and overlapping for these combinations with all models. PAZ was the least efficacious for all models (Table F2 in Appendix F in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.017).
Figure 2NMA models for progression-free survival: HRs versus sunitinib up to 5 years (trial period).
Substantial differences in the time-varying HRs were observed between NMA methods when estimates were fully extrapolated (up to 40 years) (Fig. 3). NIVO + IPI was most efficacious according to the RCS NMA method, multivariate NMA Gompertz-based models, and the multivariate NMA FP-based models with a treatment effect on the scale and first shape parameters (ie, ln[time]). Nevertheless, when the multivariate NMA log-normal-based models and the multivariate NMA FP-based models with a treatment effect on the scale and second shape parameters were used, PEM + AXI and AVE + AXI were most efficacious. The time-varying HRs obtained with the former NMA models resulted in a NIVO + IPI PFS curve that crossed PEM + AXI and AVE + AXI around 40 to 60 months and flattened in the long run. The time-varying HRs obtained with the latter models, which are less flexible, did not result in this crossing of curves or flattening of the NIVO + IPI PFS curve (Fig. F2 in Appendix F in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.017) and were characterized by narrower 95% credible intervals. The difference in RMST per intervention relative to SUN up to 40 years favored NIVO + IPI for some of the models, although credible intervals were wide and overlapping for the IO combinations irrespective of model chosen (Table F2 in Appendix F in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.017).
Figure 3NMA models for progression-free survival: progression-free survival up to 40 years (fully extrapolated).
For OS (Figs. F4-F8 and Tables F4 and F5 in Appendix F in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.017), similar findings were observed; as for PFS; short-term estimates (5 years) were relatively consistent across methods, whereas fully extrapolated relative treatment effects diverged. NIVO + IPI was most efficacious when time-varying HRs were used, whereas PEM + AXI was most efficacious when a constant HR was used, followed by either AVE + AXI or PEM + AXI and PAZ or SUN. Estimates of uncertainty (ie, 95% credible intervals) were wider for models that assumed time-varying HRs as opposed to constant HRs.
Discussion
Our objective was to summarize and compare published non-PH NMA methods for the analysis of time-to-event outcomes through a case study: 1-step
Individual patient data network meta-analysis using either restricted mean survival time difference or hazard ratios: is there a difference? A case study on locoregionally advanced nasopharyngeal carcinomas.
In addition to estimating time-varying relative treatment effects with the different NMA methods, we also derived extrapolated PFS and OS for each intervention by applying these relative treatment effects to an average hazard function of SUN across the trials. This is equivalent to the process to incorporate NMA findings into a model-based CEA.
Findings From the Illustrative Example
In our case study, PFS hazards increased sharply initially and then decreased gradually over time, except for KEYNOTE-426 with an increase in the tail. Violation of the PH assumption was identified for PFS in CheckMate 214 driven by the comparison of IO combination therapy relative to tyrosine kinase inhibitors. OS hazards were relatively stable over time. For the parametric NMA methods, we need to use a single type of survival distribution applicable to all trials. Therefore, sufficient flexibility is required to capture a diverse set of hazard patterns across trials, especially when drugs of different classes are compared. Given the contrast in patterns of the hazard over time between studies, multivariate NMA models assuming a second-order FP and the RCS NMA models tended to provide improved fit to the observed hazards compared with the traditional parametric models.
Although FP models provide the most flexibility, they may be sensitive to extreme random fluctuations toward the end of follow-up,
As our case study illustrates, limiting the relative treatment effects to the scale and only 1 of the 2 shape parameters led to less extreme extrapolations. In general, it is important to explore alternative parameterizations and constraints informed by external evidence or expert opinion.
Time-varying HR estimates obtained with FP or RCS NMA models were more uncertain than those obtained with traditional parametric distributions given the larger number of parameters with the former. Nevertheless, it is important to realize that with the more flexible approaches (ie, FP and RCS) the actual uncertainty in hazard patterns over time is less likely to be underestimated. With simpler models relying on stronger structural assumptions, a certain degree of uncertainty is not reflected in the credible intervals of the HRs.
Limitations of the Illustrative Example
Although the included trials were large, the number of trials in our case study was small, so it was not feasible to estimate between-study heterogeneity.
Motzer RJ, Tannir NM, McDermott DF, et al. Conditional survival and 5-year follow-up in CheckMate 214: First-line nivolumab plus ipilimumab (N+I) versus sunitinib (S) in advanced renal cell carcinoma (aRCC). 661P. Presented at: European Society for Medical Oncology, September 16-21, 2021; Virtual Congress.
was published after we completed the SLR (June 2020). As such, the results should not be overinterpreted from a clinical perspective. To assess the plausibility of our extrapolations, an updated analysis is of interest. There may be value in exploring how well alternative non-PH NMA methods predict results based on different follow-up, generalizing the approach by Klijn et al
What did time tell us? A comparison and retrospective validation of different survival extrapolation methods for immuno-oncologic therapy in advanced or metastatic renal cell carcinoma.
to NMAs and focusing on prediction of time-varying HRs.
Another limitation was that we focused on RCS models to estimate the RMST up to 5 years, rather than exploring all possible models. Although the extrapolated period was minimal for most trials, a more comprehensive approach is advisable.
Challenges of modelling approaches for network meta-analysis of time-to-event outcomes in the presence of non-proportional hazards to aid decision making: application to a melanoma network.
published a case study in melanoma evaluating select NMA methods for time-to-event outcomes. Like our study, they found that different NMA methods can result in different relative treatment effect estimates over time. Nevertheless, our study takes a comprehensive perspective to review and assess published NMA methods. In particular, we (1) performed an SLR to identify methods specifically proposed to avoid reliance on the PH assumption or constant relative treatment effects; (2) presented the alternative methods using a consistent notation in line with Dias et al
NICE DSU technical support document 6: embedding evidence synthesis in probabilistic cost-effectiveness analysis: software choices. National Institute for Clinical Excellence (NICE).
to compare and contrast the different approaches; (3) implemented the FP NMA methods using monthly discrete hazards as proposed in the original papers, rather than using longer term intervals; (4) explored frequently overlooked modifications of the second-order FP NMAs to constraint how time-varying relative treatment effects can vary between treatments; (5) compared 1- and 2-step implementation approaches; (6) provided a template to illustrate how to integrate NMA estimates from these methods an Excel-based cost-effectiveness model; and (7) proposed an algorithm for NMA model selection that is generalizable across methods. Freeman et al
Challenges of modelling approaches for network meta-analysis of time-to-event outcomes in the presence of non-proportional hazards to aid decision making: application to a melanoma network.
discussed 2 approaches that we did not identify in our SLR: piecewise NMA models and generalized gamma NMA with a treatment effect on only the scale (not published). The first is a valid approach but requires clinically valid decisions regarding the relevant time points, which need to be the same across studies, where the HRs are allowed to jump. The second approach still assumes a single treatment effect and is therefore still constrained in its ability to capture differences in time-varying patterns in relative treatment effects.
Proposed Model Selection Process for Non-PH NMA Methods
Given the importance of selecting an appropriate NMA model for the estimation of time-varying relative treatment effects to be used in a model-based CEA, we propose a process for such selection process, as depicted in Figure 4. This first attempt to formalize a general NMA model selection process was informed by the findings of the case study, our general experience with using NMA survival methods, and builds upon the steps proposed by Latimer 2013
Survival analysis for economic evaluations alongside clinical trials--extrapolation with patient-level data: inconsistencies, limitations, and a practical guide.
for survival analyses to inform CEA in the context of a single RCT. All of the alternative non-PH NMA methods fit this process. Nevertheless, the RMST NMA does not align with the CEA model framework and should arguable not be considered
NICE DSU technical support document 6: embedding evidence synthesis in probabilistic cost-effectiveness analysis: software choices. National Institute for Clinical Excellence (NICE).
NICE DSU technical support document 19: partitioned survival analysis for decision modelling in health care: a critical review. National Institute for Clinical Excellence (NICE).
; RMST NMA is not designed to extrapolate relative treatment effects or facilitate discounting needed for an economic evaluation. We developed an economic model template to illustrate use of results obtained with the other NMA approaches to encourage uptake of these methods.
The 1-step multivariate NMA estimates the trial-specific parameters and pooled estimates across studies simultaneously. Therefore, model selection is based on the full network of evidence, which may imply an overreliance on goodness of fit in the absence of trial-specific extrapolations that can be visually inspected. Methods to improve efficiency of model selection for 1-step NMAs have been proposed by Wiksten et al.
One- and 2-step implementations of the multivariate NMA using the same underlying survival functions and choices regarding the parameterization of the treatment effects result in similar relative treatment effect estimates over time. We prefer the 2-step method
given that the fit of alternative models to each trial can be visually inspected to eliminate any distributions that do not yield plausible extrapolations. It also allows evaluating alternative structural assumptions whether treatment affects a single or multiple parameters describing hazard functions at the trial level. This is especially important for second-order FP and RCS models.
Flexible parameteric proportional-hazards and proportional-odds models for censored survival data, with application to prognostic modelling and estimation of treatment effects.
Furthermore, given that the 2-step approach uses the actual time-to-event likelihood, established packages (flexsurv in R) can be used to overcome barriers identified in the application of non-PH NMA methods.
Despite our preference for a 2-step approach, its implementation for RCS based NMA models has not been published (yet). This highlights potential challenges regarding acceptance of the 2-step NMA approach for all potential models describing (cumulative) hazards over time.
Our case study illustrated the importance of going beyond the goodness-of-fit statistics to assess the plausibility of extrapolations, as emphasized in our proposed NMA model selection algorithm. For PFS there was a clear PH violation in CheckMate 214, which justified exploring non-PH alternative methods. For OS, there was uncertainty regarding the PH assumption in KEYNOTE-426, and therefore, non-PH methods were also explored. Beyond the trial-specific diagnostics, we also evaluated alternative trial-specific survival models using different parametrizations of the relative treatment effects to eliminate approaches that did not fit the data well or result in realistic extrapolations. This trial-level information makes the process of selecting the most appropriate NMA model across the trials more efficient. The case study identified several NMA models (FP and RCS models) for OS based on model fit criteria and visual inspection of extrapolated curves. This illustrates the importance of further considering additional criteria for model selection to make final decisions, including the consistency between extrapolated PFS and OS, as well as the use of external information from observational studies,
First-line sunitinib versus pazopanib in metastatic renal cell carcinoma: results from the International Metastatic Renal Cell Carcinoma Database Consortium.
NICE DSU technical support document 14: undertaking survival analysis for economic evaluations alongside clinical trials - extrapolation with patient-level data. National Institute for Health and Care Excellence (NICE).
Survival analysis for economic evaluations alongside clinical trials--extrapolation with patient-level data: inconsistencies, limitations, and a practical guide.
How uncertain is the survival extrapolation? A study of the impact of different parametric survival models on extrapolated uncertainty about hazard functions, lifetime mean survival and cost effectiveness.
Despite formally summarizing the overall NMA model selection steps we applied with the case study with Figure 4, it is important to highlight that this proposed algorithm is just an initial step. There is significant uncertainty associated with applying such an algorithm with many considerations to be factored in. A formal evaluation of the proposal algorithm would be worthwhile and may result in modifications.
Conclusion
When the PH assumption is questionable in a subset of the RCTs, we recommend assessing alternative non-PH NMA methods to estimate relative treatment effects for time-to-event outcomes, especially when these analyses are required for a CEA. We further recommend following a transparent and explicit stepwise considering model fit, external constraints, and clinical validity. Given the inherent uncertainties associated with such a process, sensitivity analysis using findings from more than a single NMA model should be considered.
Article and Author Information
Author Contributions:Concept and design: Cope, Chan, Chen, Borrill, May, Malcolm, Kupas, Jansen
Acquisition of data: Cope, Chan, Chen, May
Analysis and interpretation of data: Cope, Chan, Campbell, Chen, May, Malcolm, Branchoux, Kupas, Jansen
Drafting of the manuscript: Cope, Chan, Campbell, Chen, Borrill, May, Branchoux
Critical revision of the paper for important intellectual content: Cope, Chan, Chen, Borrill, May, Malcolm, Branchoux, Kupas, Jansen
Statistical analysis: Chan, Campbell
Obtaining funding: Borrill
Administrative, technical, or logistic support: Chen
Supervision: Cope, Jansen
Conflict of Interest disclosures: Mss Cope and Chen, Mr Chan, and Drs Campbell and Jansen are employees of PRECISIONheor, a consulting firm that received funding from Bristol Myers Squibb for this study. Messrs Borrill and Malcolm and Drs May, Branchoux, and Kupas are employees of and have stock ownership in Bristol Myers Squibb. No other disclosures were reported.
Funding/Support: This article was supported by Bristol Myers Squibb.
Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.
Acknowledgment
The authors acknowledge the helpful feedback from the journal reviewers, Eleanor Paul and Andrea Berardi, for their contribution to the economic model template and Julie Park, Kevin Towle, and Dieter Ayers for their contribution to the literature review to identify relevant NMA methods.
Individual patient data network meta-analysis using either restricted mean survival time difference or hazard ratios: is there a difference? A case study on locoregionally advanced nasopharyngeal carcinomas.
Comparative survival benefit of currently licensed second or third line treatments for epidermal growth factor receptor (EGFR) and anaplastic lymphoma kinase (ALK) negative advanced or metastatic non-small cell lung cancer: a systematic review and secondary analysis of trials.
Programmed death-1 or programmed death ligand-1 blockade in patients with platinum-resistant metastatic urothelial cancer: a systematic review and meta-analysis.
Flexible parameteric proportional-hazards and proportional-odds models for censored survival data, with application to prognostic modelling and estimation of treatment effects.
Cabozantinib versus everolimus, nivolumab, axitinib, sorafenib and best supportive care: a network meta-analysis of progression-free survival and overall survival in second line treatment of advanced renal cell carcinoma.
Axitinib, cabozantinib, everolimus, nivolumab, sunitinib and best supportive care in previously treated renal cell carcinoma: a systematic review and economic evaluation.
Comparative efficacy of first-line immune-based combination therapies in metastatic renal cell carcinoma: a systematic review and network meta-analysis.
Indirect treatment comparisons including network meta-analysis: lenvatinib plus everolimus for the second-line treatment of advanced/metastatic renal cell carcinoma.
Survival outcomes and independent response assessment with nivolumab plus ipilimumab versus sunitinib in patients with advanced renal cell carcinoma: 42-month follow-up of a randomized phase 3 clinical trial.
Updated efficacy results from the JAVELIN Renal 101 trial: first-line avelumab plus axitinib versus sunitinib in patients with advanced renal cell carcinoma.
Pembrolizumab plus axitinib versus sunitinib monotherapy as first-line treatment of advanced renal cell carcinoma (KEYNOTE-426): extended follow-up from a randomised, open-label, phase 3 trial.
NICE DSU technical support document 14: undertaking survival analysis for economic evaluations alongside clinical trials - extrapolation with patient-level data. National Institute for Health and Care Excellence (NICE).
Motzer RJ, Tannir NM, McDermott DF, et al. Conditional survival and 5-year follow-up in CheckMate 214: First-line nivolumab plus ipilimumab (N+I) versus sunitinib (S) in advanced renal cell carcinoma (aRCC). 661P. Presented at: European Society for Medical Oncology, September 16-21, 2021; Virtual Congress.
What did time tell us? A comparison and retrospective validation of different survival extrapolation methods for immuno-oncologic therapy in advanced or metastatic renal cell carcinoma.
Challenges of modelling approaches for network meta-analysis of time-to-event outcomes in the presence of non-proportional hazards to aid decision making: application to a melanoma network.
NICE DSU technical support document 6: embedding evidence synthesis in probabilistic cost-effectiveness analysis: software choices. National Institute for Clinical Excellence (NICE).
Survival analysis for economic evaluations alongside clinical trials--extrapolation with patient-level data: inconsistencies, limitations, and a practical guide.
NICE DSU technical support document 19: partitioned survival analysis for decision modelling in health care: a critical review. National Institute for Clinical Excellence (NICE).
First-line sunitinib versus pazopanib in metastatic renal cell carcinoma: results from the International Metastatic Renal Cell Carcinoma Database Consortium.
How uncertain is the survival extrapolation? A study of the impact of different parametric survival models on extrapolated uncertainty about hazard functions, lifetime mean survival and cost effectiveness.
04749-0&id=gr1.jpg){kind=link}
04749-0&id=gr2.jpg){kind=link}
04749-0&id=gr3.jpg){kind=link}
04749-0&id=gr4.jpg){kind=link}