Advertisement

A Comparison of Alternative Network Meta-Analysis Methods in the Presence of Nonproportional Hazards: A Case Study in First-Line Advanced or Metastatic Renal Cell Carcinoma

Open AccessPublished:December 08, 2022DOI:https://doi.org/10.1016/j.jval.2022.11.017

      Highlights

      • Network meta-analysis (NMA) of time-to-event outcomes based on constant hazard ratios can result in biased findings when the proportional hazards (PHs) assumption does not hold in a subset of the trials. Several NMA methods that do not rely on the PH assumption have been proposed. Nevertheless, their application to health technology assessment submissions has been limited, possibly due to a lack of familiarity among researchers performing cost-effectiveness analysis of oncology drugs.
      • NMA methods that allow for non-PH may provide relative treatment effect estimates that better reflect the evidence. We provide a systematic overview and comparison of methods using a case study to increase familiarity. Given the impact of NMA model choice on findings, we propose a stepwise process to select appropriate NMA models. This is especially important when considering the most flexible NMA models.
      • When relative treatment effects in terms of time-to-event outcomes need to be estimated based on an NMA of randomized controlled trials and the PH assumption is uncertain in one or more of the randomized controlled trials, alternative parametric NMA methods that do not rely on this assumption are recommended.

      Abstract

      Objectives

      Network meta-analysis (NMA) of time-to-event outcomes based on constant hazard ratios can result in biased findings when the proportional hazards (PHs) assumption does not hold in a subset of trials. We aimed to summarize the published non-PH NMA methods for time-to-event outcomes, demonstrate their application, and compare their results.

      Methods

      The following non-PH NMA methods were compared through an illustrative case study in oncology of 4 randomized controlled trials in terms of progression-free survival and overall survival: (1) 1-step or (2) 2-step multivariate NMAs based on traditional survival distributions or fractional polynomials, (3) NMAs with restricted cubic splines for baseline hazard, and (4) restricted mean survival NMA.

      Results

      For progression-free survival, the PH assumption did not hold across trials and non-PH NMA methods better reflected the relative treatment effects over time. The most flexible models (fractional polynomials and restricted cubic splines) fit better to the data than the other approaches. Estimated hazard ratios obtained with different non-PH NMA methods were similar at 5 years of follow-up but differed thereafter in the extrapolations. Although there was no strong evidence of PH violation for overall survival, non-PH NMA methods captured this uncertainty in the relative treatment effects over time.

      Conclusions

      When the PH assumption is questionable in a subset of the randomized controlled trials, we recommend assessing alternative non-PH NMA methods to estimate relative treatment effects for time-to-event outcomes. We propose a transparent and explicit stepwise model selection process considering model fit, external constraints, and clinical validity. Given inherent uncertainty, sensitivity analyses are suggested.

      Keywords

      Introduction

      In oncology, the efficacy of a new intervention is often assessed using time-to-event outcomes, such as progression-free survival (PFS) and overall survival (OS), and summarized with Kaplan-Meier (KM) curves. The relative treatment effect of one intervention versus another is typically expressed as a hazard ratio (HR) obtained with a Cox model assuming proportional hazards (PHs).
      • Moher D.
      • Hopewell S.
      • Schulz K.F.
      • et al.
      CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials.
      ,
      • Batson S.
      • Greenall G.
      • Hudson P.
      Review of the reporting of survival analyses within randomised controlled trials and the implications for meta-analysis.
      Other relative treatment effect measures that may be used include the difference in median survival, the odds ratio of survival at a specific time point, or the difference in restricted mean survival time (RMST) up to a specific time point (t∗).
      • Royston P.
      • Parmar M.K.
      Restricted mean survival time: an alternative to the hazard ratio for the design and analysis of randomized trials with a time-to-event outcome.
      Only the HR and RMST (dependent on time point selected) capture the relative treatment effect over the complete follow-up of the trial.
      Many health technology assessment agencies require manufacturers to undertake cost-effectiveness analysis (CEA) of their new interventions versus the current standard of care in the respective healthcare system and versus other competing interventions not included in the randomized controlled trial (RCT) program. For interventions aiming to improve survival, this frequently requires extrapolation of the RCT data over time using parametric survival distributions to obtain estimates of expected survival. When the evidence base consists of multiple RCTs, each comparing a subset of the interventions of interest, we need to perform a network meta-analysis (NMA) to obtain relative treatment effects of each intervention versus an overall reference treatment. These estimates can subsequently be applied to the PFS and OS curves of this reference treatment, to predict the PFS and OS curves for all interventions in the evaluation. It is important to realize that both the PFS and OS curves of the reference treatment and the PFS and OS relative treatment effect estimates may need to be extrapolated over time.
      NMAs of time-to-event outcomes can be performed based on the trial-specific reported HRs, but these estimates may be biased when the PH assumption does not hold. Trials comparing treatments with different mechanisms of action, or short-term versus long-term benefits, tend to violate the PH assumption.
      • Royston P.
      • Parmar M.K.
      Restricted mean survival time: an alternative to the hazard ratio for the design and analysis of randomized trials with a time-to-event outcome.
      • Schadendorf D.
      • Hodi F.S.
      • Robert C.
      • et al.
      Pooled analysis of long-term survival data from phase II and phase III trials of ipilimumab in unresectable or metastatic melanoma.
      • Li H.
      • Han D.
      • Hou Y.
      • Chen H.
      • Chen Z.
      Statistical inference methods for two crossing survival curves: a comparison of methods.
      To overcome these limitations, alternative NMA methods have been proposed.
      • Ouwens M.J.
      • Philips Z.
      • Jansen J.P.
      Network meta-analysis of parametric survival curves.
      • Jansen J.P.
      Network meta-analysis of survival data with fractional polynomials.
      • Cope S.
      • Chan K.
      • Jansen J.
      Multivariate network meta-analysis of survival function parameters.
      • Freeman S.
      • Carpenter J.
      Bayesian one-step IPD network meta-analysis of time-to-event data using Royston-Parmar models.
      Nevertheless, their application has been limited, possibly due to a lack of awareness how to perform these analyses.
      • Freeman S.C.
      • Sutton A.J.
      • Cooper N.J.
      Uptake of methodological advances for synthesis of continuous and time-to-event outcomes would maximize use of the evidence base.
       We aimed to summarize the published methods for NMA of time-to-event outcomes that allow for time-varying treatment effects, compare their application with a case study, and discuss the implications for CEA.

      NMA Methods That Do Not Rely on the PH Assumption

      Four distinct NMA methods for time-to-event outcomes that do not rely on the PH assumption were identified from the literature
      • Cope S.
      • Chen J.
      • Ayers D.
      • et al.
      PNS92 a targeted review of indirect treatment comparison synthesis methods that do not rely on the proportional hazard assumption.
      and summarized in Table 1
      • Ouwens M.J.
      • Philips Z.
      • Jansen J.P.
      Network meta-analysis of parametric survival curves.
      • Jansen J.P.
      Network meta-analysis of survival data with fractional polynomials.
      • Cope S.
      • Chan K.
      • Jansen J.
      Multivariate network meta-analysis of survival function parameters.
      • Freeman S.
      • Carpenter J.
      Bayesian one-step IPD network meta-analysis of time-to-event data using Royston-Parmar models.
      ,
      • Jansen J.P.
      • Cope S.
      Meta-regression models to address heterogeneity and inconsistency in network meta-analysis of survival outcomes.
      • Vickers A.D.
      • Winfree K.B.
      • Cuyun Carter G.
      • et al.
      Relative efficacy of interventions in the treatment of second-line non-small cell lung cancer: A systematic review and network meta-analysis.
      • Petit C.
      • Blanchard P.
      • Pignon J.P.
      • Lueza B.
      Individual patient data network meta-analysis using either restricted mean survival time difference or hazard ratios: is there a difference? A case study on locoregionally advanced nasopharyngeal carcinomas.
      • Connock M.
      • Armoiry X.
      • Tsertsvadze A.
      • et al.
      Comparative survival benefit of currently licensed second or third line treatments for epidermal growth factor receptor (EGFR) and anaplastic lymphoma kinase (ALK) negative advanced or metastatic non-small cell lung cancer: a systematic review and secondary analysis of trials.
      • Niglio S.A.
      • Jia R.
      • Ji J.
      • et al.
      Programmed death-1 or programmed death ligand-1 blockade in patients with platinum-resistant metastatic urothelial cancer: a systematic review and meta-analysis.
      • Lunn D.
      • Jackson C.
      • Best N.
      • Thomas A.
      • Spiegelhalter D.
      The BUGS Book.
      • Owen R.
      • Tincello D.
      • Abrams K.
      Network meta-analysis: development of a three-level hierarchical modeling approach incorporating dose-related constraints.
      : (1) 1-step multivariate NMA (Ouwens et al,
      • Ouwens M.J.
      • Philips Z.
      • Jansen J.P.
      Network meta-analysis of parametric survival curves.
      Jansen
      • Jansen J.P.
      Network meta-analysis of survival data with fractional polynomials.
      ), (2) 2-step multivariate NMA (Cope et al
      • Cope S.
      • Chan K.
      • Jansen J.
      Multivariate network meta-analysis of survival function parameters.
      ), (3) NMA with restricted cubic splines (RCS) for baseline hazard (Freeman and Carpenter
      • Freeman S.
      • Carpenter J.
      Bayesian one-step IPD network meta-analysis of time-to-event data using Royston-Parmar models.
      ), and (4) NMA with RMST (RMST NMA) (Petit et al
      • Petit C.
      • Blanchard P.
      • Pignon J.P.
      • Lueza B.
      Individual patient data network meta-analysis using either restricted mean survival time difference or hazard ratios: is there a difference? A case study on locoregionally advanced nasopharyngeal carcinomas.
      ). Details regarding the systematic literature review (SLR) process and the NMA methods are provided in Appendices A (Tables A1-A7; Figs. A1 and A2) and B (Tables B1 and B2; Boxes 1-15) in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.017.
      Table 1Summary of evidence synthesis methods for indirect comparisons that do not rely on PHs assumption.
      MethodsDescription of NMA modelSurvival distribution/functionOne- or 2-stepFrameworkLikelihood
      All methods use time-to-event data based on individual patients either reconstructed from Kaplan-Meier using Guyot algorithm or based on observed individual patient data.
      Relative treatment effect and how is non-PH addressedBetween-study heterogeneityInconsistency modelsMeta-regression
      Ouwens et al
      • Ouwens M.J.
      • Philips Z.
      • Jansen J.P.
      Network meta-analysis of parametric survival curves.
      Uses a multivariate relative treatment effect as an alternative to the synthesis of the trial-specific constant HRs. The hazard functions of the interventions in a trial are modeled using parametric distribution and the difference in the parameters are considered the multidimensional relative treatment effect, which are synthesized (and indirectly compared) across studiesWeibull, Gompertz, log-normal, log-logisticOne-step (trial level–specific relative treatment effects and pooled effects are estimated simultaneously)BayesianApproximation with piecewise constant hazards (discrete hazards) according to a binomial likelihoodMultivariate relative treatment effect parameters regarding scale and shape related factors of the survival distribution/function. These relative treatment effect parameters are used to describe time-varying HRs (or odds ratios in case of log-logistic models)YesYesNo
      Jansen
      • Jansen J.P.
      Network meta-analysis of survival data with fractional polynomials.
      First (Weibull, Gompertz) and second-order fractional polynomials describing the ln-hazards over timeNo
      Jansen and Cope
      • Jansen J.P.
      • Cope S.
      Meta-regression models to address heterogeneity and inconsistency in network meta-analysis of survival outcomes.
      Yes
      Vickers et al
      • Vickers A.D.
      • Winfree K.B.
      • Cuyun Carter G.
      • et al.
      Relative efficacy of interventions in the treatment of second-line non-small cell lung cancer: A systematic review and network meta-analysis.
      Yes - extended to exchangeable treatment-by-covariate-interaction structures
      “Allow relative treatment effects to vary by covariates independently of the other treatments in the network of evidence. The relative treatment effect remains constant for any treatment not specified within a hierarchical exchangeable structure… In addition, where possible, different doses also were included as a hierarchical structure with an overall treatment class effect. Constraints were imposed to ensure that the efficacy increased with dose intensity.” Adapted from Owen et al.17
      Cope et al
      • Cope S.
      • Chan K.
      • Jansen J.
      Multivariate network meta-analysis of survival function parameters.
      For each arm of every RCT in the network, (recreated), IPD are used to estimate alternative survival distributions. Next, for each distribution, its scale and shape parameters are included in a multivariate NMA to obtain time-varying estimates of relative treatment effects between competing interventionsWeibull, Gompertz, log-normal, log-logistic describing the ln-hazards over timeTwo-step (arm-specific survival function parameters are estimated first. Subsequently, these are incorporated in the multivariate NMA)Step 1 – Frequentist;

      Step 2 – Bayesian
      Exact likelihood corresponding to survival distribution selectedNo
      Freeman and Carpenter
      • Freeman S.
      • Carpenter J.
      Bayesian one-step IPD network meta-analysis of time-to-event data using Royston-Parmar models.
      An IPD Royston-Parmar Bayesian NMA model, which provides flexible alternative modeling approach that can accommodate time-dependent effects. The baseline log-cumulative hazards are modeled with restricted cubic splines. HRs are either fixed over time or can be modeled as a function of ln(time)Restricted cubic splines describing the baseline log-cumulative hazard of each trialTwo-step (described as 1-step but requires orthogonalized basis function of study-specific splines as input for NMA)Bayesian framework for NMA but first step in frequentist frameworkGeneral likelihood using “zeros trick” using probability density function of Poisson distribution
      If we wish to implement a likelihood representing the flexible fractional polynomials or cubic splines in WinBUGS, we can use the “zeros” trick17 where a data set comprising entirely of zeros is given a Poisson distribution with its parameter defined equal to the negative log-likelihood (plus a sufficiently large constant); the log-likelihood function corresponding to the fractional polynomial or spline is then written algebraically in the WinBUGS code.
      Constant HRs represented with a single basic parameter by treatment. As an extension, HR can vary over time by adding extra parameters for the interaction between treatment and ln(time)YesYesNo
      Petit et al
      • Petit C.
      • Blanchard P.
      • Pignon J.P.
      • Lueza B.
      Individual patient data network meta-analysis using either restricted mean survival time difference or hazard ratios: is there a difference? A case study on locoregionally advanced nasopharyngeal carcinomas.
      A 2-step analysis to estimate RMST based on (reconstructed) IPD from KMs; then evaluated mean difference in RMST in NMA modelRMS estimated based on (1) trial-specific KM method; (2) AUC of KM + exponential tailTwo-stepFrequentistNormal likelihood for NMA model and exact likelihood corresponding to parametric distribution selected (if extrapolation involved)Difference in RMS (AUC up to specific time point) between treatmentsYesYesNo
      Connock et al
      • Connock M.
      • Armoiry X.
      • Tsertsvadze A.
      • et al.
      Comparative survival benefit of currently licensed second or third line treatments for epidermal growth factor receptor (EGFR) and anaplastic lymphoma kinase (ALK) negative advanced or metastatic non-small cell lung cancer: a systematic review and secondary analysis of trials.
      RMS estimated based on (1) AUC Weibull/gen gamma per arm; (2) mean survival using Weibull; (3) AUC of KM + exponential tailBayesianNoNoNo
      Niglio et al
      • Niglio S.A.
      • Jia R.
      • Ji J.
      • et al.
      Programmed death-1 or programmed death ligand-1 blockade in patients with platinum-resistant metastatic urothelial cancer: a systematic review and meta-analysis.
      RMS estimated based on (1) pseudo-values based on KM; (2) Poisson-gamma frailty modelFrequentistNoNoNo
      AUC indicates area under the curve; HR, hazard ratio; IPD, individual patient data; KM, Kaplan-Meier; NMA, network meta-analysis; PH, proportional hazard; RCT, randomized controlled trial; RMS, restricted mean survival; RMST, restricted mean survival time.
      All methods use time-to-event data based on individual patients either reconstructed from Kaplan-Meier using Guyot algorithm or based on observed individual patient data.
      “Allow relative treatment effects to vary by covariates independently of the other treatments in the network of evidence. The relative treatment effect remains constant for any treatment not specified within a hierarchical exchangeable structure… In addition, where possible, different doses also were included as a hierarchical structure with an overall treatment class effect. Constraints were imposed to ensure that the efficacy increased with dose intensity.” Adapted from Owen et al.
      • Lunn D.
      • Jackson C.
      • Best N.
      • Thomas A.
      • Spiegelhalter D.
      The BUGS Book.
      If we wish to implement a likelihood representing the flexible fractional polynomials or cubic splines in WinBUGS, we can use the “zeros” trick
      • Lunn D.
      • Jackson C.
      • Best N.
      • Thomas A.
      • Spiegelhalter D.
      The BUGS Book.
      where a data set comprising entirely of zeros is given a Poisson distribution with its parameter defined equal to the negative log-likelihood (plus a sufficiently large constant); the log-likelihood function corresponding to the fractional polynomial or spline is then written algebraically in the WinBUGS code.

      One-Step Multivariate NMA Model

      Ouwens et al
      • Ouwens M.J.
      • Philips Z.
      • Jansen J.P.
      Network meta-analysis of parametric survival curves.
      and Jansen
      • Jansen J.P.
      Network meta-analysis of survival data with fractional polynomials.
      proposed a multivariate relative treatment effect measure that describes how the HRs change over time. Parametric survival functions are used to model the hazard functions of the interventions in each trial, and the multivariate relative treatment effect is based on the difference in its parameters, which are pooled and indirectly compared across trials. Using multiple parameters for the relative treatment effects avoids the need for the PH assumption and provides flexibility to fit the data closely.
      Ouwens et al
      • Ouwens M.J.
      • Philips Z.
      • Jansen J.P.
      Network meta-analysis of parametric survival curves.
      proposed these NMA models in the context of Weibull, Gompertz, log-logistic, and log-normal distributions. Jansen and Cope
      • Jansen J.P.
      • Cope S.
      Meta-regression models to address heterogeneity and inconsistency in network meta-analysis of survival outcomes.
      extended this to fractional polynomial (FP) models and to meta-regression analysis. Vickers et al
      • Vickers A.D.
      • Winfree K.B.
      • Cuyun Carter G.
      • et al.
      Relative efficacy of interventions in the treatment of second-line non-small cell lung cancer: A systematic review and network meta-analysis.
      presented a variation by assuming exchangeable covariate interaction effects across treatments (ie, histology, programmed death ligand 1 expression, and mutation status), rather than treatment-specific or constant covariate interaction effects. We label these methods as “1-step” approaches, where trial-specific survival curve parameters and pooled relative treatment effects across studies are estimated simultaneously. These analyses are based on the discrete hazards (or conditional survival probabilities) derived from individual patient data (IPD)
      • Jansen J.P.
      • Cope S.
      Meta-regression models to address heterogeneity and inconsistency in network meta-analysis of survival outcomes.
      or reconstructed from published KM curves.
      • Guyot P.
      • Ades A.
      • Ouwens M.
      • Welton N.
      Enhanced secondary analysis of survival data: reconstructing the data from published Kaplan-Meier survival curves.

      Two-Step Multivariate NMA Model

      With this approach, parametric survival distributions are fitted for each arm of each trial in the first stage. In the second stage, these parameters are synthesized with a multivariate NMA model.
      • Cope S.
      • Chan K.
      • Jansen J.
      Multivariate network meta-analysis of survival function parameters.
      In the first step, an exact likelihood distribution for survival times, for example, Weibull, Gompertz, log-logistic, and log-normal distributions, is used. Chan et al
      • Chan K.
      • Ayers D.
      • Jansen J.
      • Cope S.
      PCN283 extending multivariate network meta-analysis of survival function parameters to fractional polynomials.
      extended this framework to include second-order FP models assuming treatment affects the scale and both shape parameters.

      NMA Model With Cubic Splines for Baseline Hazard

      This NMA method is based on RCS describing the log-cumulative baseline hazard of the trial,
      • Freeman S.
      • Carpenter J.
      Bayesian one-step IPD network meta-analysis of time-to-event data using Royston-Parmar models.
      ,
      • Royston P.
      • Parmar M.K.
      Flexible parameteric proportional-hazards and proportional-odds models for censored survival data, with application to prognostic modelling and estimation of treatment effects.
      which can be estimated more efficiently as an alternative to a Cox-based NMA of IPD. The method focused primarily on constant HRs but can be extended with an interaction between treatment and ln(time) to estimate time-varying HRs, where HR develops as a linear function of ln(time). As such, this model does not use RCS to describe the HRs over time.

      Restricted Mean Survival NMA

      RMST NMAs
      • Petit C.
      • Blanchard P.
      • Pignon J.P.
      • Lueza B.
      Individual patient data network meta-analysis using either restricted mean survival time difference or hazard ratios: is there a difference? A case study on locoregionally advanced nasopharyngeal carcinomas.
      • Connock M.
      • Armoiry X.
      • Tsertsvadze A.
      • et al.
      Comparative survival benefit of currently licensed second or third line treatments for epidermal growth factor receptor (EGFR) and anaplastic lymphoma kinase (ALK) negative advanced or metastatic non-small cell lung cancer: a systematic review and secondary analysis of trials.
      • Niglio S.A.
      • Jia R.
      • Ji J.
      • et al.
      Programmed death-1 or programmed death ligand-1 blockade in patients with platinum-resistant metastatic urothelial cancer: a systematic review and meta-analysis.
      use a 2-step process. First, the differences in RMST between treatments in a trial are estimated based on (reconstructed) IPD. Next, these estimates are synthesized with a standard NMA model, assuming the differences in RMST within each trial are normally distributed. The choice for the specific time point (t∗) is suggested to maximize the use of available data, while minimizing the need for extrapolation.
      • Wei Y.
      • Royston P.
      • Tierney J.F.
      • Parmar M.K.
      Meta-analysis of time-to-event outcomes from randomized trials using restricted mean survival time: application to individual participant data.
      If extrapolation is not required, the difference in the RMST is estimated nonparametrically using the area under the KM curve for each trial, and the delta method is used to estimate the variance of the difference in the RMST.
      • Lamb K.E.
      • Williamson E.J.
      • Coory M.
      • Carlin J.B.
      Bias and precision of measures of survival gain from right-censored data.
      If the survival curves need to be extrapolated, parametric models are used. Alternatively, “pseudo-values” can be used.
      • Wei Y.
      • Royston P.
      • Tierney J.F.
      • Parmar M.K.
      Meta-analysis of time-to-event outcomes from randomized trials using restricted mean survival time: application to individual participant data.
      ,
      • Lueza B.
      • Rotolo F.
      • Bonastre J.
      • Pignon J.P.
      • Michiels S.
      Bias and precision of methods for estimating the difference in restricted mean survival time from an individual patient data meta-analysis.

      Illustrative Example

      Methods

      Evidence base

      Our illustrative example concerns a comparison of immunotherapies (IOs) with tyrosine kinase inhibitors for the first-line treatment of relapsed or stage IV clear cell renal cell carcinoma. The PH assumption of the RCTs, which formed a star-shaped network (Fig. 1), has been questioned by health technology assessment agencies.
      • Amzal B.
      • Fu S.
      • Meng J.
      • Lister J.
      • Karcher H.
      Cabozantinib versus everolimus, nivolumab, axitinib, sorafenib and best supportive care: a network meta-analysis of progression-free survival and overall survival in second line treatment of advanced renal cell carcinoma.
      • Edwards S.J.
      • Wakefield V.
      • Cain P.
      • et al.
      Axitinib, cabozantinib, everolimus, nivolumab, sunitinib and best supportive care in previously treated renal cell carcinoma: a systematic review and economic evaluation.
      • El Rassy E.
      • Aoun F.
      • Sleilaty G.
      • et al.
      Network meta-analysis of second-line treatment in metastatic renal cell carcinoma: efficacy and safety.
      • Elaidi R.
      • Phan L.
      • Borchiellini D.
      • et al.
      Comparative efficacy of first-line immune-based combination therapies in metastatic renal cell carcinoma: a systematic review and network meta-analysis.
      • Hale P.
      • Hahn A.W.
      • Rathi N.
      • Pal S.K.
      • Haaland B.
      • Agarwal N.
      Treatment of metastatic renal cell carcinoma in older patients: a network meta-analysis.
      • Karner C.
      • Kew K.
      • Wakefield V.
      • Masento N.
      • Edwards S.J.
      Targeted therapies for previously treated advanced or metastatic renal cell carcinoma: systematic review and network meta-analysis.
      Nivolumab previously treated advanced renal cell carcinoma. National Institute for Health and Care Excellence (NICE).
      Lenvatinib everolimus previously treated advanced renal cell carcinoma. National Institute for Health and Care Excellence (NICE).
      Nivolumab with ipilimumab untreated advanced renal cell carcinoma. National Institute for Health and Care Excellence (NICE).
      • Porta C.
      • Szczylik C.
      • Casciano R.
      • et al.
      Second-line cabozantinib versus nivolumab in advanced renal cell carcinoma: systematic review and indirect treatment comparison.
      • Tremblay G.
      • McElroy H.J.
      • Westley T.
      • Meier G.
      • Misurski D.
      • Guo M.
      Indirect treatment comparisons including network meta-analysis: lenvatinib plus everolimus for the second-line treatment of advanced/metastatic renal cell carcinoma.
      • Wiecek W.
      • Karcher H.
      Nivolumab versus cabozantinib: comparing overall survival in metastatic renal cell carcinoma.
      Interventions compared with sunitinib (SUN) included nivolumab + ipilimumab (NIVO + IPI, CheckMate 214
      • Motzer R.J.
      • Escudier B.
      • McDermott D.F.
      • et al.
      Survival outcomes and independent response assessment with nivolumab plus ipilimumab versus sunitinib in patients with advanced renal cell carcinoma: 42-month follow-up of a randomized phase 3 clinical trial.
      ,
      • Motzer R.J.
      • Tannir N.M.
      • McDermott D.F.
      • et al.
      Nivolumab plus ipilimumab versus sunitinib in advanced renal-cell carcinoma.
      ), pazopanib (PAZ, COMPARZ
      • Motzer R.J.
      • Hutson T.E.
      • Cella D.
      • et al.
      Pazopanib versus sunitinib in metastatic renal-cell carcinoma.
      ,
      • Motzer R.J.
      • Hutson T.E.
      • McCann L.
      • Deen K.
      • Choueiri T.K.
      Overall survival in renal-cell carcinoma with pazopanib versus sunitinib.
      ), avelumab + axitinib (AVE + AXI, JAVELIN Renal 101
      • Choueiri T.K.
      • Motzer R.J.
      • Rini B.I.
      • et al.
      Updated efficacy results from the JAVELIN Renal 101 trial: first-line avelumab plus axitinib versus sunitinib in patients with advanced renal cell carcinoma.
      ,
      • Motzer R.J.
      • Penkov K.
      • Haanen J.
      • et al.
      Avelumab plus axitinib versus sunitinib for advanced renal-cell carcinoma.
      ), and pembrolizumab + AXI (PEM + AXI, KEYNOTE-426
      • Plimack E.R.
      • Rini B.I.
      • Stus V.
      • et al.
      Pembrolizumab plus axitinib versus sunitinib as first-line therapy for advanced renal cell carcinoma (RCC): updated analysis of KEYNOTE-426.
      • Rini B.I.
      • Plimack E.R.
      • Stus V.
      • et al.
      Pembrolizumab plus axitinib versus sunitinib for advanced renal-cell carcinoma.
      • Powles T.
      • Plimack E.R.
      • Soulières D.
      • et al.
      Pembrolizumab plus axitinib versus sunitinib monotherapy as first-line treatment of advanced renal cell carcinoma (KEYNOTE-426): extended follow-up from a randomised, open-label, phase 3 trial.
      ). The OS and PFS KM curves were digitized and pseudo-IPD were reconstructed
      • Guyot P.
      • Ades A.
      • Ouwens M.
      • Welton N.
      Enhanced secondary analysis of survival data: reconstructing the data from published Kaplan-Meier survival curves.
      (see Appendix Fig. C1 and Tables C1-C5 regarding the review to identify trials, and Appendix Fig. C2 and Table C6 regarding details of the trials in Appendix C in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.017).
      Figure thumbnail gr1
      Figure 1Network of trials for case study: advanced or metastatic first-line renal cell carcinoma among the intent-to-treat population (all risk).
      AVE indicates avelumab; AXI, axitinib; IMDC, International Metastatic Renal Cell Carcinoma Database Consortium; IO, immunotherapy; IPI, ipilimumab; MSKCC, Memorial Sloan Kettering Cancer Center; NIVO, nivolumab; PAZ, pazopanib; PEM, pembrolizumab; SUN, sunitinib; TKI, tyrosine kinase inhibitor.

      Analysis

      All NMA models evaluated were implemented assuming fixed effects rather than random effects, given the small number of studies in the network.

      NMA based on constant HRs

      An NMA assuming constant HRs was performed for comparative purposes.

      One-step multivariate NMA

      In the multivariate NMA using traditional parametric models (ie, Weibull [equivalent to a first-order FP with a log(time) transformation (P1 = 0)], Gompertz [first-order FP without transformation of time (P1 = 1)], log-logistic, and log-normal), relative treatment effects were applied to the scale and shape parameters. For the second-order FPs, we focused on models that were extensions of Weibull or Gompertz (P1 = 0 or 1) models to facilitate interpretation, limiting power transformations of time for this additional shape parameter P2 from −1, −0.5, 0, 0.5, 1 with P2 = 0 defined as ln(time).
      • Jansen J.P.
      Network meta-analysis of survival data with fractional polynomials.
      We explored alternative parameterizations of the relative treatment effects for the second-order FP models with the treatment having an impact on (1) only the scale (ie, constant HR), (2) scale and only 1 shape, and (3) scale and both shape parameters. A binomial likelihood was used for the discrete hazards data.
      • Jansen J.P.
      • Cope S.
      Meta-regression models to address heterogeneity and inconsistency in network meta-analysis of survival outcomes.

      Two-step multivariate NMA

      The 2-step models evaluated were consistent with the 1-step multivariate NMA. However, for second-order FPs, we assumed the treatment had an impact on the scale and both shape parameters. An exact likelihood (ie, Weibull, Gompertz, log-logistic, log-normal, or FP) was used for the trial-specific analysis of the pseudo-IPD in the first step, followed by a multivariate normal likelihood for the NMA model in the second step.

      NMAs with RCS for baseline hazard

      For the NMAs with RCS (RCS NMA), the knots were placed at equally spaced percentiles of the uncensored survival times based on the pseudo-IPD for each trial, with boundary knots at the minimum and maximum values of the uncensored survival times.
      • Royston P.
      • Parmar M.K.
      Flexible parameteric proportional-hazards and proportional-odds models for censored survival data, with application to prognostic modelling and estimation of treatment effects.
      The relative treatment effects were either (1) constant HR over time or (2) time-varying HR based on an interaction between HR and ln(time). The exact likelihood was specified according to the pseudo-IPD assuming RCS.

      Restricted mean survival time NMA

      For the RMST NMA, the area under the curve was defined up to t∗ at 2 or 5 years according to the shortest (JAVELIN) and longest (CheckMate 214) follow-up across trials, respectively. For t∗ = 2 years, RMST was estimated based on KM curves (ie, no extrapolation required). For t∗ = 5 years, the best-fitting RCS function per arm (1, 2, or 3 internal knots) was used. RCS was selected to provide most flexible fit to the observed data. A normal likelihood was used in the second step to combine the differences in RMST.
      • Dias S.
      • Welton N.J.
      • Sutton A.J.
      • Caldwell D.M.
      • Lu G.
      • Ades A.E.
      Evidence synthesis for decision making 4: inconsistency in networks of evidence based on randomized controlled trials.

      Model parameter estimation

      A Bayesian framework was used for all NMA models, with parameters estimated using Markov chain Monte Carlo implemented in the JAGS (v4.1) or Stan software package (RStan v2.21.2). R (v3.6.1) packages were used for trial-specific estimates in the 2-step NMA (flexsurv), the RCS NMA (flexsurvspline), and RMST NMA (survRM2). Additional details regarding the likelihoods, data setup, and priors used for the implementation of the NMA methods for the case study are presented in Appendix D in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.017.

      Model selection by NMA method

      We identified the most appropriate 1 or 2 models for each NMA method using the criteria in Table 2.
      • Ouwens M.J.
      • Philips Z.
      • Jansen J.P.
      Network meta-analysis of parametric survival curves.
      • Jansen J.P.
      Network meta-analysis of survival data with fractional polynomials.
      • Cope S.
      • Chan K.
      • Jansen J.
      Multivariate network meta-analysis of survival function parameters.
      • Freeman S.
      • Carpenter J.
      Bayesian one-step IPD network meta-analysis of time-to-event data using Royston-Parmar models.
      ,
      • Petit C.
      • Blanchard P.
      • Pignon J.P.
      • Lueza B.
      Individual patient data network meta-analysis using either restricted mean survival time difference or hazard ratios: is there a difference? A case study on locoregionally advanced nasopharyngeal carcinomas.
      • Connock M.
      • Armoiry X.
      • Tsertsvadze A.
      • et al.
      Comparative survival benefit of currently licensed second or third line treatments for epidermal growth factor receptor (EGFR) and anaplastic lymphoma kinase (ALK) negative advanced or metastatic non-small cell lung cancer: a systematic review and secondary analysis of trials.
      • Niglio S.A.
      • Jia R.
      • Ji J.
      • et al.
      Programmed death-1 or programmed death ligand-1 blockade in patients with platinum-resistant metastatic urothelial cancer: a systematic review and meta-analysis.
      ,
      • Chan K.
      • Ayers D.
      • Jansen J.
      • Cope S.
      PCN283 extending multivariate network meta-analysis of survival function parameters to fractional polynomials.
      ,
      • Cox D.R.
      Regression models and life-tables.
      • Hosmer D.
      • Lemeshow S.
      • May S.
      Applied Survival Analysis.
      • Schoenfeld D.
      Partial residuals for the proportional hazards regression model.
      • Grambsch P.
      • Therneau T.
      Proportional hazards tests and diagnostics based on weighted residuals.
      First, trial-level diagnostics were used
      • Latimer N.
      NICE DSU technical support document 14: undertaking survival analysis for economic evaluations alongside clinical trials - extrapolation with patient-level data. National Institute for Health and Care Excellence (NICE).
      to assess the development of the hazard over time and whether the PH assumption held for each trial. Next, the goodness of fit of the competing models was assessed. For the 1-step multivariate NMA method, model selection was based on the complete evidence base using the deviance information criterion (DIC).
      • Spiegelhalter D.J.
      • Best N.G.
      • Carlin B.P.
      • Van Der Linde A.
      Bayesian measures of model complexity and fit.
      For the other 3 methods, alternative survival models were fitted to (each arm of) the individual trials and the model of choice was based on the sum of the Akaike information criterion across trials. For the multivariate NMA and the RMST NMA, DICs can be used to compare between fixed- and random-effects models. Nevertheless, given the small number of studies in the network, only a fixed-effects NMA was considered. For the RCS NMA, the DIC was used to decide between constant and time-varying HRs. We also visually compared age-matched general mortality rates with the extrapolated survival and hazards plots up to 5 and 40 years obtained with the alternative models to assess the upper bound of the extrapolations that would be possible.
      Table 2Overview of the NMA model selection process.
      Model selection processOne-step multivariate NMA
      • Ouwens M.J.
      • Philips Z.
      • Jansen J.P.
      Network meta-analysis of parametric survival curves.
      ,
      • Jansen J.P.
      Network meta-analysis of survival data with fractional polynomials.
      Two-step multivariate NMA methods
      • Cope S.
      • Chan K.
      • Jansen J.
      Multivariate network meta-analysis of survival function parameters.
      ,
      • Chan K.
      • Ayers D.
      • Jansen J.
      • Cope S.
      PCN283 extending multivariate network meta-analysis of survival function parameters to fractional polynomials.
      NMA cubic splines for baseline hazard
      • Freeman S.
      • Carpenter J.
      Bayesian one-step IPD network meta-analysis of time-to-event data using Royston-Parmar models.
      RMST NMA
      • Petit C.
      • Blanchard P.
      • Pignon J.P.
      • Lueza B.
      Individual patient data network meta-analysis using either restricted mean survival time difference or hazard ratios: is there a difference? A case study on locoregionally advanced nasopharyngeal carcinomas.
      • Connock M.
      • Armoiry X.
      • Tsertsvadze A.
      • et al.
      Comparative survival benefit of currently licensed second or third line treatments for epidermal growth factor receptor (EGFR) and anaplastic lymphoma kinase (ALK) negative advanced or metastatic non-small cell lung cancer: a systematic review and secondary analysis of trials.
      • Niglio S.A.
      • Jia R.
      • Ji J.
      • et al.
      Programmed death-1 or programmed death ligand-1 blockade in patients with platinum-resistant metastatic urothelial cancer: a systematic review and meta-analysis.
      Competing models consideredTraditional parametric models:

      Log-logistic, Log-normal, Weibull, and Gompertz

      Treatment effect:
      • Scale (intercept) and shape parameter (time-related factor)
      FP models: P1 = 0 or 1; P2 = −1, −0.5, 0, 0.5, or 1

      Treatment effect:
      • Scale only
      • Scale and one of the shape parameters
      • Scale and both shape parameters
      Traditional parametric models:

      Log-logistic, Log-normal, Weibull, Gompertz

      Treatment effect:
      • Scale and shape parameter
      FP models: P1 = 0 or 1; P2 = −1, −0.5, 0, 0.5, or 1

      Treatment effect:
      • Scale and both shape parameters
      Royston-Parmar RCS models:

      1, 2, or 3 internal knots

      All studies with 1 knot

      Study-specific number of knots

      Treatment effect:
      • Constant HR
      • Treatment × ln(time) interaction
      (A) RMST up to 2 years based on KM curves

      (B) RMST up to 5 years using the best-fitting model per arm (RCS with 1, 2, or 3 internal knots)
      Trial-level diagnostics to assess the development of the hazard over time
      (1) PH evaluations and diagnostics (ie, time-dependent covariate test using Cox model,
      • Cox D.R.
      Regression models and life-tables.
      plot scaled Schoenfeld residuals over time
      • Hosmer D.
      • Lemeshow S.
      • May S.
      Applied Survival Analysis.
      ,
      • Schoenfeld D.
      Partial residuals for the proportional hazards regression model.
      and Grambsch and Therneau test
      • Grambsch P.
      • Therneau T.
      Proportional hazards tests and diagnostics based on weighted residuals.
      ); (2) Quantile-quantile plot of times of survival percentiles; (3) Log-cumulative hazard plots vs log(time); and (4) Plot of smoothed empirical hazard vs time
      Select arm or trial-level model

      Not applicable since study-level estimates and indirect comparisons performed simultaneously


      Not applicable since study-level estimates and indirect comparisons performed simultaneously


      Arm-level model:

      AIC per arm and across full network and visual inspection per arm


      Arm-level model: AIC per arm and across full network and visual inspection per arm


      Trial-level models: AIC per trial and visual inspection per trial


      Arm-level models when extrapolation is required for (B): AIC per arm and visual inspection per arm
      NMA models – goodness of fit per method
      DICs across NMA methods cannot be compared because data and likelihoods differ.


      DICs can be compared across parametric models since uses same data and likelihood to inform model
      Given the star-shaped network with only 1 trial per contrast, we were able to use the relative treatment effects as estimated from the NMA model to predict trial-specific hazards and survival (by applying them to trial-specific baseline), which were visually inspected and compared with the observed hazards and survival. Nevertheless, this step is not proposed by authors in the original papers and would not be feasible if there were > 1 study informing each comparison or if there was indirect evidence.


      DICs across FP models can be compared since uses same data and same likelihood to inform model and parameterization of relative treatment effects (PH vs non-PH)
      Given the star-shaped network with only 1 trial per contrast, we were able to use the relative treatment effects as estimated from the NMA model to predict trial-specific hazards and survival (by applying them to trial-specific baseline), which were visually inspected and compared with the observed hazards and survival. Nevertheless, this step is not proposed by authors in the original papers and would not be feasible if there were > 1 study informing each comparison or if there was indirect evidence.


      DICs across parametric models cannot be compared since data (ie, parameters of different distributions) is not the same across models (likelihood is same)
      DICs can be used to compare FE versus RE models versus with or without treatment by covariate interactions.


      DICs across FP models cannot be compared since data (ie, parameters of different distributions) is not the same across models (likelihood is same)
      DICs can be used to compare FE versus RE models versus with or without treatment by covariate interactions.


      DICs across spline models can be compared since the model uses same data and likelihood to inform model and parameterization of relative treatment effects (ie, PH vs non-PH)


      Only 1 RMST model evaluated for extrapolated survival (B); If alternative extrapolations were used to inform RMST, DICs across RMST models could not be compared (data would differ)
      DICs can be used to compare FE versus RE models versus with or without treatment by covariate interactions.
      Green checkmark indicates selection criteria was applicable; Red 'x' indicates the selection criteria was not applicable.
      AIC indicates Akaike information criterion; DIC, deviance information criterion; FE, fixed effect model; FP, fractional polynomials; HR, hazard ratio; KM, Kaplan-Meier; NMA, network meta-analysis; PH, proportional hazards; RCS, restricted cubic splines; RE, random effects model; RMST, restricted mean survival time.
      DICs across NMA methods cannot be compared because data and likelihoods differ.
      Given the star-shaped network with only 1 trial per contrast, we were able to use the relative treatment effects as estimated from the NMA model to predict trial-specific hazards and survival (by applying them to trial-specific baseline), which were visually inspected and compared with the observed hazards and survival. Nevertheless, this step is not proposed by authors in the original papers and would not be feasible if there were > 1 study informing each comparison or if there was indirect evidence.
      DICs can be used to compare FE versus RE models versus with or without treatment by covariate interactions.

      Comparison of NMA methods regarding treatment effects

      The alternative NMA methods were compared based on the selected models regarding the following short-term (5 years, longest trial follow-up) and long-term (40 years, lifetime horizon
      Nivolumab with ipilimumab untreated advanced renal cell carcinoma. National Institute for Health and Care Excellence (NICE).
      ,
      Pembrolizumab with axitinib for untreated advanced renal cell carcinoma. National Institute for Health and Care Excellence (NICE).
      ,
      Avelumab with axitinib for untreated advanced renal cell carcinoma. National Institute for Health and Care Excellence (NICE).
      ) estimates: (1) relative treatment effects for each intervention versus SUN and uncertainty based on the 95% credible intervals, (2) survival proportions by treatment (obtained by multiplying the average hazard function of SUN across the trials in the network by the estimated HRs), and (3) difference in RMST per intervention versus SUN
      • Cope S.
      • Jansen J.P.
      Quantitative summaries of treatment effect estimates obtained with network meta-analysis of survival curves to inform decision-making.
      (rankings estimated based on RMST using the surface under the cumulative ranking curve).
      • Salanti G.
      • Ades A.E.
      • Ioannidis J.P.
      Graphical methods and numerical summaries for presenting results from multiple-treatment meta-analysis: an overview and tutorial.

      Results

      Model selection

      For PFS, a clear violation in PH assumption was identified for CheckMate 214 based on crossing of the hazards (around month 48) and confirmation by both the Grambsch and Therneau and Wald tests. Nevertheless, for OS, no statistically significant violation of the PH assumption was observed, although it was uncertain for KEYNOTE-426 (see Appendix E Table E1 and Figs. E1-E8 for trial-level diagnostics in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.017). Accordingly, we focus our results on PFS in the body of the article but include OS in Appendix F in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.017.
      Selection of the specific NMA models for each of the NMA methods is summarized in Appendix Tables E2 and E3 for PFS and in Tables E4 and E5 for OS in Appendix E in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.017. For both 1- and 2-step multivariate NMA methods, models based on the log-normal and Gompertz distributions were selected among the traditional survival models. When the set of competing models to consider for the 1-step NMA method was expanded to include the more flexible second-order FPs, the models describing ln-hazards over time as a function of ln(time) and 1/time (FP P1 = 0, P2 = −1) with a relative treatment effect on the scale and either the first or second shape parameter were best fitting. For the RCS NMA method, a model with study-specific number of knots and a time-varying HR was selected (number of knots: CheckMate 214, 2; COMPARZ, 2; JAVELIN, 3; KEYNOTE-426, 2). Please note that Akaike information criteria/DICs could not be compared across NMA methods because of differences in data (ie, pseudo-IPD vs discrete hazards derived from pseudo-IPD).

      Comparison of NMA methods regarding relative treatment effects

      Estimated time-varying PFS HRs of the competing interventions versus SUN up to 5 years (longest follow-up in the trials) obtained with different non-PH NMA methods were relatively similar, except for HRs obtained with the multivariate NMA methods assuming Gompertz-distributed survival times (Fig. 2). Applying the time-varying HRs from each NMA method to the average SUN hazard function results in the corresponding PFS curves by treatment (Appendix Fig. F1 and Table F1 in Appendix F in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.017). The difference in RMST per intervention relative to SUN up to 5 years consistently suggested that PEM + AXI and AVE + AXI were the most efficacious treatment combinations, followed closely by NIVO + IPI, although confidence intervals were wide and overlapping for these combinations with all models. PAZ was the least efficacious for all models (Table F2 in Appendix F in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.017).
      Figure thumbnail gr2
      Figure 2NMA models for progression-free survival: HRs versus sunitinib up to 5 years (trial period).
      # indicates number; AXI, axitinib; CrI, credible interval; FP, fractional polynomial; HR, hazard ratio; IPI, ipilimumab; NIVO, nivolumab; NMA, network meta-analysis; PAZ, pazopanib; PEM, pembrolizumab.
      Substantial differences in the time-varying HRs were observed between NMA methods when estimates were fully extrapolated (up to 40 years) (Fig. 3). NIVO + IPI was most efficacious according to the RCS NMA method, multivariate NMA Gompertz-based models, and the multivariate NMA FP-based models with a treatment effect on the scale and first shape parameters (ie, ln[time]). Nevertheless, when the multivariate NMA log-normal-based models and the multivariate NMA FP-based models with a treatment effect on the scale and second shape parameters were used, PEM + AXI and AVE + AXI were most efficacious. The time-varying HRs obtained with the former NMA models resulted in a NIVO + IPI PFS curve that crossed PEM + AXI and AVE + AXI around 40 to 60 months and flattened in the long run. The time-varying HRs obtained with the latter models, which are less flexible, did not result in this crossing of curves or flattening of the NIVO + IPI PFS curve (Fig. F2 in Appendix F in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.017) and were characterized by narrower 95% credible intervals. The difference in RMST per intervention relative to SUN up to 40 years favored NIVO + IPI for some of the models, although credible intervals were wide and overlapping for the IO combinations irrespective of model chosen (Table F2 in Appendix F in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.017).
      Figure thumbnail gr3
      Figure 3NMA models for progression-free survival: progression-free survival up to 40 years (fully extrapolated).
      # indicates number; FP, fractional polynomial; HR, hazard ratio; NMA, network meta-analysis.
      For OS (Figs. F4-F8 and Tables F4 and F5 in Appendix F in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.017), similar findings were observed; as for PFS; short-term estimates (5 years) were relatively consistent across methods, whereas fully extrapolated relative treatment effects diverged. NIVO + IPI was most efficacious when time-varying HRs were used, whereas PEM + AXI was most efficacious when a constant HR was used, followed by either AVE + AXI or PEM + AXI and PAZ or SUN. Estimates of uncertainty (ie, 95% credible intervals) were wider for models that assumed time-varying HRs as opposed to constant HRs.

      Discussion

      Our objective was to summarize and compare published non-PH NMA methods for the analysis of time-to-event outcomes through a case study: 1-step
      • Ouwens M.J.
      • Philips Z.
      • Jansen J.P.
      Network meta-analysis of parametric survival curves.
      ,
      • Jansen J.P.
      Network meta-analysis of survival data with fractional polynomials.
      or 2-step
      • Cope S.
      • Chan K.
      • Jansen J.
      Multivariate network meta-analysis of survival function parameters.
      multivariate NMA models, RCS NMA,
      • Freeman S.
      • Carpenter J.
      Bayesian one-step IPD network meta-analysis of time-to-event data using Royston-Parmar models.
      and RMST NMA.
      • Petit C.
      • Blanchard P.
      • Pignon J.P.
      • Lueza B.
      Individual patient data network meta-analysis using either restricted mean survival time difference or hazard ratios: is there a difference? A case study on locoregionally advanced nasopharyngeal carcinomas.
      In addition to estimating time-varying relative treatment effects with the different NMA methods, we also derived extrapolated PFS and OS for each intervention by applying these relative treatment effects to an average hazard function of SUN across the trials. This is equivalent to the process to incorporate NMA findings into a model-based CEA.

      Findings From the Illustrative Example

      In our case study, PFS hazards increased sharply initially and then decreased gradually over time, except for KEYNOTE-426 with an increase in the tail. Violation of the PH assumption was identified for PFS in CheckMate 214 driven by the comparison of IO combination therapy relative to tyrosine kinase inhibitors. OS hazards were relatively stable over time. For the parametric NMA methods, we need to use a single type of survival distribution applicable to all trials. Therefore, sufficient flexibility is required to capture a diverse set of hazard patterns across trials, especially when drugs of different classes are compared. Given the contrast in patterns of the hazard over time between studies, multivariate NMA models assuming a second-order FP and the RCS NMA models tended to provide improved fit to the observed hazards compared with the traditional parametric models.
      Although FP models provide the most flexibility, they may be sensitive to extreme random fluctuations toward the end of follow-up,
      • Magee L.
      Nonlocal behavior in polynomial regressions.
      potentially negatively affecting the plausibility of extrapolations.
      • Freeman S.
      • Carpenter J.
      Bayesian one-step IPD network meta-analysis of time-to-event data using Royston-Parmar models.
      ,
      • Kearns B.
      • Stevenson M.D.
      • Triantafyllopoulos K.
      • Manca A.
      Generalized linear models for flexible parametric modeling of the hazard function.
      As our case study illustrates, limiting the relative treatment effects to the scale and only 1 of the 2 shape parameters led to less extreme extrapolations. In general, it is important to explore alternative parameterizations and constraints informed by external evidence or expert opinion.
      • Rutherford M.J.
      • Lambert P.C.
      • Sweeting M.J.
      • et al.
      NICE DSU technical support document 21: flexible methods for survival analysis. National Institute for Health and Care Excellence (NICE).
      Time-varying HR estimates obtained with FP or RCS NMA models were more uncertain than those obtained with traditional parametric distributions given the larger number of parameters with the former. Nevertheless, it is important to realize that with the more flexible approaches (ie, FP and RCS) the actual uncertainty in hazard patterns over time is less likely to be underestimated. With simpler models relying on stronger structural assumptions, a certain degree of uncertainty is not reflected in the credible intervals of the HRs.

      Limitations of the Illustrative Example

      Although the included trials were large, the number of trials in our case study was small, so it was not feasible to estimate between-study heterogeneity.
      Additional follow-up for CheckMate 214

      Motzer RJ, Tannir NM, McDermott DF, et al. Conditional survival and 5-year follow-up in CheckMate 214: First-line nivolumab plus ipilimumab (N+I) versus sunitinib (S) in advanced renal cell carcinoma (aRCC). 661P. Presented at: European Society for Medical Oncology, September 16-21, 2021; Virtual Congress.

      was published after we completed the SLR (June 2020). As such, the results should not be overinterpreted from a clinical perspective. To assess the plausibility of our extrapolations, an updated analysis is of interest. There may be value in exploring how well alternative non-PH NMA methods predict results based on different follow-up, generalizing the approach by Klijn et al
      • Klijn S.L.
      • Fenwick E.
      • Kroep S.
      • et al.
      What did time tell us? A comparison and retrospective validation of different survival extrapolation methods for immuno-oncologic therapy in advanced or metastatic renal cell carcinoma.
      to NMAs and focusing on prediction of time-varying HRs.
      Another limitation was that we focused on RCS models to estimate the RMST up to 5 years, rather than exploring all possible models. Although the extrapolated period was minimal for most trials, a more comprehensive approach is advisable.

      Comparison With the Literature

      Freeman et al
      • Freeman S.C.
      • Cooper N.J.
      • Sutton A.J.
      • Crowther M.J.
      • Carpenter J.R.
      • Hawkins N.
      Challenges of modelling approaches for network meta-analysis of time-to-event outcomes in the presence of non-proportional hazards to aid decision making: application to a melanoma network.
      published a case study in melanoma evaluating select NMA methods for time-to-event outcomes. Like our study, they found that different NMA methods can result in different relative treatment effect estimates over time. Nevertheless, our study takes a comprehensive perspective to review and assess published NMA methods. In particular, we (1) performed an SLR to identify methods specifically proposed to avoid reliance on the PH assumption or constant relative treatment effects; (2) presented the alternative methods using a consistent notation in line with Dias et al
      • Dias S.
      • Sutton A.
      • Welton N.
      • Ades A.
      NICE DSU technical support document 6: embedding evidence synthesis in probabilistic cost-effectiveness analysis: software choices. National Institute for Clinical Excellence (NICE).
      to compare and contrast the different approaches; (3) implemented the FP NMA methods using monthly discrete hazards as proposed in the original papers, rather than using longer term intervals; (4) explored frequently overlooked modifications of the second-order FP NMAs to constraint how time-varying relative treatment effects can vary between treatments; (5) compared 1- and 2-step implementation approaches; (6) provided a template to illustrate how to integrate NMA estimates from these methods an Excel-based cost-effectiveness model; and (7) proposed an algorithm for NMA model selection that is generalizable across methods. Freeman et al
      • Freeman S.C.
      • Cooper N.J.
      • Sutton A.J.
      • Crowther M.J.
      • Carpenter J.R.
      • Hawkins N.
      Challenges of modelling approaches for network meta-analysis of time-to-event outcomes in the presence of non-proportional hazards to aid decision making: application to a melanoma network.
      discussed 2 approaches that we did not identify in our SLR: piecewise NMA models and generalized gamma NMA with a treatment effect on only the scale (not published). The first is a valid approach but requires clinically valid decisions regarding the relevant time points, which need to be the same across studies, where the HRs are allowed to jump. The second approach still assumes a single treatment effect and is therefore still constrained in its ability to capture differences in time-varying patterns in relative treatment effects.

      Proposed Model Selection Process for Non-PH NMA Methods

      Given the importance of selecting an appropriate NMA model for the estimation of time-varying relative treatment effects to be used in a model-based CEA, we propose a process for such selection process, as depicted in Figure 4. This first attempt to formalize a general NMA model selection process was informed by the findings of the case study, our general experience with using NMA survival methods, and builds upon the steps proposed by Latimer 2013
      • Latimer N.R.
      Survival analysis for economic evaluations alongside clinical trials--extrapolation with patient-level data: inconsistencies, limitations, and a practical guide.
      for survival analyses to inform CEA in the context of a single RCT. All of the alternative non-PH NMA methods fit this process. Nevertheless, the RMST NMA does not align with the CEA model framework and should arguable not be considered
      • Dias S.
      • Sutton A.
      • Welton N.
      • Ades A.
      NICE DSU technical support document 6: embedding evidence synthesis in probabilistic cost-effectiveness analysis: software choices. National Institute for Clinical Excellence (NICE).
      ,
      Guide to the methods of technology appraisal. National Institute for Clinical Excellence (NICE).
      • Dias S.
      • Welton N.
      • Sutton A.
      • Ades A.E.
      NICE DSU technical support document 5: evidence synthesis in the baseline natural history model. National Institute for Clinical Excellence (NICE).
      • Woods B.
      • Sideris E.
      • Palmer S.
      • Latimer N.
      • Soares M.
      NICE DSU technical support document 19: partitioned survival analysis for decision modelling in health care: a critical review. National Institute for Clinical Excellence (NICE).
      ; RMST NMA is not designed to extrapolate relative treatment effects or facilitate discounting needed for an economic evaluation. We developed an economic model template to illustrate use of results obtained with the other NMA approaches to encourage uptake of these methods.
      Figure thumbnail gr4
      Figure 4Model selection process adapted for NMA.
      AIC indicates Akaike information criterion; BIC, Bayesian information criterion; DIC, deviance information criterion; HR, hazard ratio; NMA, network meta-analysis; OS, overall survival; PFS, progression-free survival; PH, proportional hazard; RCT, randomized controlled trial.
      The 1-step multivariate NMA estimates the trial-specific parameters and pooled estimates across studies simultaneously. Therefore, model selection is based on the full network of evidence, which may imply an overreliance on goodness of fit in the absence of trial-specific extrapolations that can be visually inspected. Methods to improve efficiency of model selection for 1-step NMAs have been proposed by Wiksten et al.
      • Wiksten A.
      • Hawkins N.
      • Piepho H.P.
      • Gsteiger S.
      Nonproportional hazards in network meta-analysis: efficient strategies for model building and analysis.
      One- and 2-step implementations of the multivariate NMA using the same underlying survival functions and choices regarding the parameterization of the treatment effects result in similar relative treatment effect estimates over time. We prefer the 2-step method
      • Cope S.
      • Chan K.
      • Jansen J.
      Multivariate network meta-analysis of survival function parameters.
      given that the fit of alternative models to each trial can be visually inspected to eliminate any distributions that do not yield plausible extrapolations. It also allows evaluating alternative structural assumptions whether treatment affects a single or multiple parameters describing hazard functions at the trial level. This is especially important for second-order FP and RCS models.
      • Royston P.
      • Parmar M.K.
      Flexible parameteric proportional-hazards and proportional-odds models for censored survival data, with application to prognostic modelling and estimation of treatment effects.
      ,
      • Harrell Jr., F.E.
      Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis.
      Furthermore, given that the 2-step approach uses the actual time-to-event likelihood, established packages (flexsurv in R) can be used to overcome barriers identified in the application of non-PH NMA methods.
      • Freeman S.C.
      • Sutton A.J.
      • Cooper N.J.
      Uptake of methodological advances for synthesis of continuous and time-to-event outcomes would maximize use of the evidence base.
      Despite our preference for a 2-step approach, its implementation for RCS based NMA models has not been published (yet). This highlights potential challenges regarding acceptance of the 2-step NMA approach for all potential models describing (cumulative) hazards over time.
      Our case study illustrated the importance of going beyond the goodness-of-fit statistics to assess the plausibility of extrapolations, as emphasized in our proposed NMA model selection algorithm. For PFS there was a clear PH violation in CheckMate 214, which justified exploring non-PH alternative methods. For OS, there was uncertainty regarding the PH assumption in KEYNOTE-426, and therefore, non-PH methods were also explored. Beyond the trial-specific diagnostics, we also evaluated alternative trial-specific survival models using different parametrizations of the relative treatment effects to eliminate approaches that did not fit the data well or result in realistic extrapolations. This trial-level information makes the process of selecting the most appropriate NMA model across the trials more efficient. The case study identified several NMA models (FP and RCS models) for OS based on model fit criteria and visual inspection of extrapolated curves. This illustrates the importance of further considering additional criteria for model selection to make final decisions, including the consistency between extrapolated PFS and OS, as well as the use of external information from observational studies,
      • Ruiz-Morales J.M.
      • Swierkowski M.
      • Wells J.C.
      • et al.
      First-line sunitinib versus pazopanib in metastatic renal cell carcinoma: results from the International Metastatic Renal Cell Carcinoma Database Consortium.
      general mortality, or clinical experts.
      • Latimer N.
      NICE DSU technical support document 14: undertaking survival analysis for economic evaluations alongside clinical trials - extrapolation with patient-level data. National Institute for Health and Care Excellence (NICE).
      ,
      • Rutherford M.J.
      • Lambert P.C.
      • Sweeting M.J.
      • et al.
      NICE DSU technical support document 21: flexible methods for survival analysis. National Institute for Health and Care Excellence (NICE).
      ,
      • Latimer N.R.
      Survival analysis for economic evaluations alongside clinical trials--extrapolation with patient-level data: inconsistencies, limitations, and a practical guide.
      ,
      • Kearns B.
      • Stevens J.
      • Ren S.
      • Brennan A.
      How uncertain is the survival extrapolation? A study of the impact of different parametric survival models on extrapolated uncertainty about hazard functions, lifetime mean survival and cost effectiveness.
      Despite formally summarizing the overall NMA model selection steps we applied with the case study with Figure 4, it is important to highlight that this proposed algorithm is just an initial step. There is significant uncertainty associated with applying such an algorithm with many considerations to be factored in. A formal evaluation of the proposal algorithm would be worthwhile and may result in modifications.

      Conclusion

      When the PH assumption is questionable in a subset of the RCTs, we recommend assessing alternative non-PH NMA methods to estimate relative treatment effects for time-to-event outcomes, especially when these analyses are required for a CEA. We further recommend following a transparent and explicit stepwise considering model fit, external constraints, and clinical validity. Given the inherent uncertainties associated with such a process, sensitivity analysis using findings from more than a single NMA model should be considered.

      Article and Author Information

      Author Contributions: Concept and design: Cope, Chan, Chen, Borrill, May, Malcolm, Kupas, Jansen
      Acquisition of data: Cope, Chan, Chen, May
      Analysis and interpretation of data: Cope, Chan, Campbell, Chen, May, Malcolm, Branchoux, Kupas, Jansen
      Drafting of the manuscript: Cope, Chan, Campbell, Chen, Borrill, May, Branchoux
      Critical revision of the paper for important intellectual content: Cope, Chan, Chen, Borrill, May, Malcolm, Branchoux, Kupas, Jansen
      Statistical analysis: Chan, Campbell
      Obtaining funding: Borrill
      Administrative, technical, or logistic support: Chen
      Supervision: Cope, Jansen
      Conflict of Interest disclosures: Mss Cope and Chen, Mr Chan, and Drs Campbell and Jansen are employees of PRECISIONheor, a consulting firm that received funding from Bristol Myers Squibb for this study. Messrs Borrill and Malcolm and Drs May, Branchoux, and Kupas are employees of and have stock ownership in Bristol Myers Squibb. No other disclosures were reported.
      Funding/Support: This article was supported by Bristol Myers Squibb.
      Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

      Acknowledgment

      The authors acknowledge the helpful feedback from the journal reviewers, Eleanor Paul and Andrea Berardi, for their contribution to the economic model template and Julie Park, Kevin Towle, and Dieter Ayers for their contribution to the literature review to identify relevant NMA methods.

      Supplemental Material

      References

        • Moher D.
        • Hopewell S.
        • Schulz K.F.
        • et al.
        CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials.
        J Clin Epidemiol. 2010; 63: e1-e37
        • Batson S.
        • Greenall G.
        • Hudson P.
        Review of the reporting of survival analyses within randomised controlled trials and the implications for meta-analysis.
        PLoS One. 2016; 11e0154870
        • Royston P.
        • Parmar M.K.
        Restricted mean survival time: an alternative to the hazard ratio for the design and analysis of randomized trials with a time-to-event outcome.
        BMC Med Res Methodol. 2013; 13: 152
        • Schadendorf D.
        • Hodi F.S.
        • Robert C.
        • et al.
        Pooled analysis of long-term survival data from phase II and phase III trials of ipilimumab in unresectable or metastatic melanoma.
        J Clin Oncol. 2015; 33: 1889-1894
        • Li H.
        • Han D.
        • Hou Y.
        • Chen H.
        • Chen Z.
        Statistical inference methods for two crossing survival curves: a comparison of methods.
        PLoS One. 2015; 10e0116774
        • Ouwens M.J.
        • Philips Z.
        • Jansen J.P.
        Network meta-analysis of parametric survival curves.
        Res Synth Methods. 2010; 1: 258-271
        • Jansen J.P.
        Network meta-analysis of survival data with fractional polynomials.
        BMC Med Res Methodol. 2011; 11: 61
        • Cope S.
        • Chan K.
        • Jansen J.
        Multivariate network meta-analysis of survival function parameters.
        Res Synth Methods. 2020; 11: 443-456
        • Freeman S.
        • Carpenter J.
        Bayesian one-step IPD network meta-analysis of time-to-event data using Royston-Parmar models.
        Res Synth Methods. 2017; 8: 451-464
        • Freeman S.C.
        • Sutton A.J.
        • Cooper N.J.
        Uptake of methodological advances for synthesis of continuous and time-to-event outcomes would maximize use of the evidence base.
        J Clin Epidemiol. 2020; 124: 94-105
        • Cope S.
        • Chen J.
        • Ayers D.
        • et al.
        PNS92 a targeted review of indirect treatment comparison synthesis methods that do not rely on the proportional hazard assumption.
        Value Health. 2021; 24S189
        • Jansen J.P.
        • Cope S.
        Meta-regression models to address heterogeneity and inconsistency in network meta-analysis of survival outcomes.
        BMC Med Res Methodol. 2012; 12: 152
        • Vickers A.D.
        • Winfree K.B.
        • Cuyun Carter G.
        • et al.
        Relative efficacy of interventions in the treatment of second-line non-small cell lung cancer: A systematic review and network meta-analysis.
        BMC Cancer. 2019; 19: 353
        • Petit C.
        • Blanchard P.
        • Pignon J.P.
        • Lueza B.
        Individual patient data network meta-analysis using either restricted mean survival time difference or hazard ratios: is there a difference? A case study on locoregionally advanced nasopharyngeal carcinomas.
        Syst Rev. 2019; 8: 96
        • Connock M.
        • Armoiry X.
        • Tsertsvadze A.
        • et al.
        Comparative survival benefit of currently licensed second or third line treatments for epidermal growth factor receptor (EGFR) and anaplastic lymphoma kinase (ALK) negative advanced or metastatic non-small cell lung cancer: a systematic review and secondary analysis of trials.
        BMC Cancer. 2019; 19: 392
        • Niglio S.A.
        • Jia R.
        • Ji J.
        • et al.
        Programmed death-1 or programmed death ligand-1 blockade in patients with platinum-resistant metastatic urothelial cancer: a systematic review and meta-analysis.
        Eur Urol. 2019; 76: 782-789
        • Lunn D.
        • Jackson C.
        • Best N.
        • Thomas A.
        • Spiegelhalter D.
        The BUGS Book.
        Boca Raton, FL: CRC Press, 2013
        • Owen R.
        • Tincello D.
        • Abrams K.
        Network meta-analysis: development of a three-level hierarchical modeling approach incorporating dose-related constraints.
        Value Health. 2015; 18: 116-126
        • Guyot P.
        • Ades A.
        • Ouwens M.
        • Welton N.
        Enhanced secondary analysis of survival data: reconstructing the data from published Kaplan-Meier survival curves.
        BMC Med Res Methodol. 2012; 12: 9
        • Chan K.
        • Ayers D.
        • Jansen J.
        • Cope S.
        PCN283 extending multivariate network meta-analysis of survival function parameters to fractional polynomials.
        Value Health. 2020; 23: S472
        • Royston P.
        • Parmar M.K.
        Flexible parameteric proportional-hazards and proportional-odds models for censored survival data, with application to prognostic modelling and estimation of treatment effects.
        Stat Med. 2002; 21: 2175-2197
        • Wei Y.
        • Royston P.
        • Tierney J.F.
        • Parmar M.K.
        Meta-analysis of time-to-event outcomes from randomized trials using restricted mean survival time: application to individual participant data.
        Stat Med. 2015; 34: 2881-2898
        • Lamb K.E.
        • Williamson E.J.
        • Coory M.
        • Carlin J.B.
        Bias and precision of measures of survival gain from right-censored data.
        Pharm Stat. 2015; 14: 409-417
        • Lueza B.
        • Rotolo F.
        • Bonastre J.
        • Pignon J.P.
        • Michiels S.
        Bias and precision of methods for estimating the difference in restricted mean survival time from an individual patient data meta-analysis.
        BMC Med Res Methodol. 2016; 16: 37
        • Amzal B.
        • Fu S.
        • Meng J.
        • Lister J.
        • Karcher H.
        Cabozantinib versus everolimus, nivolumab, axitinib, sorafenib and best supportive care: a network meta-analysis of progression-free survival and overall survival in second line treatment of advanced renal cell carcinoma.
        PLoS One. 2017; 12e0184423
        • Edwards S.J.
        • Wakefield V.
        • Cain P.
        • et al.
        Axitinib, cabozantinib, everolimus, nivolumab, sunitinib and best supportive care in previously treated renal cell carcinoma: a systematic review and economic evaluation.
        Health Technol Assess. 2018; 22: 1
        • El Rassy E.
        • Aoun F.
        • Sleilaty G.
        • et al.
        Network meta-analysis of second-line treatment in metastatic renal cell carcinoma: efficacy and safety.
        Future Oncol. 2017; 13: 2709-2717
        • Elaidi R.
        • Phan L.
        • Borchiellini D.
        • et al.
        Comparative efficacy of first-line immune-based combination therapies in metastatic renal cell carcinoma: a systematic review and network meta-analysis.
        Cancers. 2020; 12: 1673
        • Hale P.
        • Hahn A.W.
        • Rathi N.
        • Pal S.K.
        • Haaland B.
        • Agarwal N.
        Treatment of metastatic renal cell carcinoma in older patients: a network meta-analysis.
        J Geriatr Oncol. 2019; 10: 149-154
        • Karner C.
        • Kew K.
        • Wakefield V.
        • Masento N.
        • Edwards S.J.
        Targeted therapies for previously treated advanced or metastatic renal cell carcinoma: systematic review and network meta-analysis.
        BMJ Open. 2019; 9e024691
      1. Nivolumab previously treated advanced renal cell carcinoma. National Institute for Health and Care Excellence (NICE).
        https://www.nice.org.uk/guidance/ta417
        Date accessed: February 4, 2021
      2. Lenvatinib everolimus previously treated advanced renal cell carcinoma. National Institute for Health and Care Excellence (NICE).
        https://www.nice.org.uk/guidance/ta498
        Date accessed: February 4, 2021
      3. Nivolumab with ipilimumab untreated advanced renal cell carcinoma. National Institute for Health and Care Excellence (NICE).
        https://www.nice.org.uk/guidance/ta581
        Date accessed: February 4, 2021
        • Porta C.
        • Szczylik C.
        • Casciano R.
        • et al.
        Second-line cabozantinib versus nivolumab in advanced renal cell carcinoma: systematic review and indirect treatment comparison.
        Crit Rev Oncol Hematol. 2019; 139: 143-148
        • Tremblay G.
        • McElroy H.J.
        • Westley T.
        • Meier G.
        • Misurski D.
        • Guo M.
        Indirect treatment comparisons including network meta-analysis: lenvatinib plus everolimus for the second-line treatment of advanced/metastatic renal cell carcinoma.
        PLoS One. 2019; 14e0212899
        • Wiecek W.
        • Karcher H.
        Nivolumab versus cabozantinib: comparing overall survival in metastatic renal cell carcinoma.
        PLoS One. 2016; 11e0155389
        • Motzer R.J.
        • Escudier B.
        • McDermott D.F.
        • et al.
        Survival outcomes and independent response assessment with nivolumab plus ipilimumab versus sunitinib in patients with advanced renal cell carcinoma: 42-month follow-up of a randomized phase 3 clinical trial.
        J Immunother Cancer. 2020; 8e000891
        • Motzer R.J.
        • Tannir N.M.
        • McDermott D.F.
        • et al.
        Nivolumab plus ipilimumab versus sunitinib in advanced renal-cell carcinoma.
        N Engl J Med. 2018; 378: 1277-1290
        • Motzer R.J.
        • Hutson T.E.
        • Cella D.
        • et al.
        Pazopanib versus sunitinib in metastatic renal-cell carcinoma.
        N Engl J Med. 2013; 369: 722-731
        • Motzer R.J.
        • Hutson T.E.
        • McCann L.
        • Deen K.
        • Choueiri T.K.
        Overall survival in renal-cell carcinoma with pazopanib versus sunitinib.
        N Engl J Med. 2014; 370: 1769-1770
        • Choueiri T.K.
        • Motzer R.J.
        • Rini B.I.
        • et al.
        Updated efficacy results from the JAVELIN Renal 101 trial: first-line avelumab plus axitinib versus sunitinib in patients with advanced renal cell carcinoma.
        Ann Oncol. 2020; 31: 1030-1039
        • Motzer R.J.
        • Penkov K.
        • Haanen J.
        • et al.
        Avelumab plus axitinib versus sunitinib for advanced renal-cell carcinoma.
        N Engl J Med. 2019; 380: 1103-1115
        • Plimack E.R.
        • Rini B.I.
        • Stus V.
        • et al.
        Pembrolizumab plus axitinib versus sunitinib as first-line therapy for advanced renal cell carcinoma (RCC): updated analysis of KEYNOTE-426.
        J Clin Oncol. 2020; 38 (5001-5001)
        • Rini B.I.
        • Plimack E.R.
        • Stus V.
        • et al.
        Pembrolizumab plus axitinib versus sunitinib for advanced renal-cell carcinoma.
        N Engl J Med. 2019; 380: 1116-1127
        • Powles T.
        • Plimack E.R.
        • Soulières D.
        • et al.
        Pembrolizumab plus axitinib versus sunitinib monotherapy as first-line treatment of advanced renal cell carcinoma (KEYNOTE-426): extended follow-up from a randomised, open-label, phase 3 trial.
        Lancet Oncol. 2020; 21: 1563-1573
        • Dias S.
        • Welton N.J.
        • Sutton A.J.
        • Caldwell D.M.
        • Lu G.
        • Ades A.E.
        Evidence synthesis for decision making 4: inconsistency in networks of evidence based on randomized controlled trials.
        Med Decis Making. 2013; 33: 641-656
        • Cox D.R.
        Regression models and life-tables.
        J R Stat Soc B. 1972; 34: 187-202
        • Hosmer D.
        • Lemeshow S.
        • May S.
        Applied Survival Analysis.
        2nd ed. John Wiley & Sons, Inc, Hoboken, NJ2008
        • Schoenfeld D.
        Partial residuals for the proportional hazards regression model.
        Biometrika. 1982; 69: 239-241
        • Grambsch P.
        • Therneau T.
        Proportional hazards tests and diagnostics based on weighted residuals.
        Biometrika. 1994; 81: 515-526
        • Latimer N.
        NICE DSU technical support document 14: undertaking survival analysis for economic evaluations alongside clinical trials - extrapolation with patient-level data. National Institute for Health and Care Excellence (NICE).
        • Spiegelhalter D.J.
        • Best N.G.
        • Carlin B.P.
        • Van Der Linde A.
        Bayesian measures of model complexity and fit.
        J R Stat Soc B. 2002; 64: 583-639
      4. National life tables: UK. Office for National Statistics.
      5. Pembrolizumab with axitinib for untreated advanced renal cell carcinoma. National Institute for Health and Care Excellence (NICE).
        https://www.nice.org.uk/guidance/ta650
        Date accessed: February 17, 2022
      6. Avelumab with axitinib for untreated advanced renal cell carcinoma. National Institute for Health and Care Excellence (NICE).
        https://www.nice.org.uk/guidance/ta645
        Date accessed: February 17, 2022
        • Cope S.
        • Jansen J.P.
        Quantitative summaries of treatment effect estimates obtained with network meta-analysis of survival curves to inform decision-making.
        BMC Med Res Methodol. 2013; 13: 147
        • Salanti G.
        • Ades A.E.
        • Ioannidis J.P.
        Graphical methods and numerical summaries for presenting results from multiple-treatment meta-analysis: an overview and tutorial.
        J Clin Epidemiol. 2011; 64: 163-171
        • Magee L.
        Nonlocal behavior in polynomial regressions.
        Am Stat. 1998; 52: 20-22
        • Kearns B.
        • Stevenson M.D.
        • Triantafyllopoulos K.
        • Manca A.
        Generalized linear models for flexible parametric modeling of the hazard function.
        Med Decis Making. 2019; 39: 867-878
        • Rutherford M.J.
        • Lambert P.C.
        • Sweeting M.J.
        • et al.
        NICE DSU technical support document 21: flexible methods for survival analysis. National Institute for Health and Care Excellence (NICE).
        (Accessed February 17, 2022)
      7. Motzer RJ, Tannir NM, McDermott DF, et al. Conditional survival and 5-year follow-up in CheckMate 214: First-line nivolumab plus ipilimumab (N+I) versus sunitinib (S) in advanced renal cell carcinoma (aRCC). 661P. Presented at: European Society for Medical Oncology, September 16-21, 2021; Virtual Congress.

        • Klijn S.L.
        • Fenwick E.
        • Kroep S.
        • et al.
        What did time tell us? A comparison and retrospective validation of different survival extrapolation methods for immuno-oncologic therapy in advanced or metastatic renal cell carcinoma.
        Pharmacoeconomics. 2021; 39: 345-356
        • Freeman S.C.
        • Cooper N.J.
        • Sutton A.J.
        • Crowther M.J.
        • Carpenter J.R.
        • Hawkins N.
        Challenges of modelling approaches for network meta-analysis of time-to-event outcomes in the presence of non-proportional hazards to aid decision making: application to a melanoma network.
        Stat Methods Med Res. 2022; 31: 839-861
        • Dias S.
        • Sutton A.
        • Welton N.
        • Ades A.
        NICE DSU technical support document 6: embedding evidence synthesis in probabilistic cost-effectiveness analysis: software choices. National Institute for Clinical Excellence (NICE).
        (Accessed February 17, 2022)
        • Latimer N.R.
        Survival analysis for economic evaluations alongside clinical trials--extrapolation with patient-level data: inconsistencies, limitations, and a practical guide.
        Med Decis Making. 2013; 33: 743-754
      8. Guide to the methods of technology appraisal. National Institute for Clinical Excellence (NICE).
        (Accessed February 17, 2022)
        • Dias S.
        • Welton N.
        • Sutton A.
        • Ades A.E.
        NICE DSU technical support document 5: evidence synthesis in the baseline natural history model. National Institute for Clinical Excellence (NICE).
        (Accessed February 17, 2022)
        • Woods B.
        • Sideris E.
        • Palmer S.
        • Latimer N.
        • Soares M.
        NICE DSU technical support document 19: partitioned survival analysis for decision modelling in health care: a critical review. National Institute for Clinical Excellence (NICE).
        (Accessed February 17, 2022)
        • Wiksten A.
        • Hawkins N.
        • Piepho H.P.
        • Gsteiger S.
        Nonproportional hazards in network meta-analysis: efficient strategies for model building and analysis.
        Value Health. 2020; 23: 918-927
        • Harrell Jr., F.E.
        Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis.
        Springer, Berlin, Germany2015
        • Ruiz-Morales J.M.
        • Swierkowski M.
        • Wells J.C.
        • et al.
        First-line sunitinib versus pazopanib in metastatic renal cell carcinoma: results from the International Metastatic Renal Cell Carcinoma Database Consortium.
        Eur J Cancer. 2016; 65: 102-108
        • Kearns B.
        • Stevens J.
        • Ren S.
        • Brennan A.
        How uncertain is the survival extrapolation? A study of the impact of different parametric survival models on extrapolated uncertainty about hazard functions, lifetime mean survival and cost effectiveness.
        Pharmacoeconomics. 2020; 38: 193-204