Advertisement

Differences in Evidentiary Requirements Between European Medicines Agency and European Health Technology Assessment of Oncology Drugs—Can Alignment Be Enhanced?

  • Sharon Wolters
    Correspondence
    Correspondence: Sharon Wolters, MSc, Asc Academics B.V., Hereweg 120, 9725 AK, Groningen, The Netherlands.
    Affiliations
    Asc Academics B.V., Groningen, The Netherlands

    Department of Health Sciences, University of Groningen, Groningen, The Netherlands
    Search for articles by this author
  • Frank G.A. Jansman
    Affiliations
    Department of Clinical Pharmacy, Deventer Teaching Hospital, Deventer, The Netherlands

    Unit of Pharmacotherapy, Pharmacoepidemiology and Pharmacoeconomics, Groningen Research Institute of Pharmacy (GRIP), University of Groningen, Groningen, The Netherlands
    Search for articles by this author
  • Maarten J. Postma
    Affiliations
    Department of Health Sciences, University of Groningen, Groningen, The Netherlands

    Department of Economics, Econometrics & Finance, Faculty of Economics and Business, University of Groningen, Groningen, The Netherlands
    Search for articles by this author
Open AccessPublished:June 22, 2022DOI:https://doi.org/10.1016/j.jval.2022.05.006

      Highlights

      • Although the same evidence is used for both the European Medicines Agency oncology assessment as the oncology assessment of health technology assessment bodies, the outcome of the assessment differs owing to different assessment goals.
      • This article highlights the differences between the oncology evidentiary requirements based on the views of representatives working on drug assessments and drug assessment experts from European Medicines Agency and 6 national health technology assessment bodies.
      • Improved alignment in evidentiary requirements is possible. Recommendations for minimizing and optimally managing remaining differences included joint early dialogs, joint relative effectiveness assessments, and the use of managed entry agreements.

      Abstract

      Objectives

      National health technology assessments (HTAs) across Europe show differences in evidentiary requirements from assessments by the European Medicines Agency (EMA), affecting time to patient access for drugs after marketing authorization. This article analyzes the differences between EMA and HTA bodies’ evidentiary requirements for oncology drugs and provides recommendations on potential further alignment to minimize and optimally manage the remaining differences.

      Methods

      Interviews were performed with representatives and drug assessment experts from EMA and HTA bodies to identify evidentiary requirements for several subdomains and collect recommendations for potentially more efficiently addressing differences. A comparative analysis of acceptability of the evidence by EMA and the HTA bodies and for potential further alignment between both authorities was conducted.

      Results

      Acceptability of available evidence was higher for EMA than HTA bodies. HTA bodies and EMA were aligned on evidentiary requirements in most cases. The subdomains showing notable differences concerned the acceptance of limitation of the target population and extrapolation of target populations, progression-free survival and (other) surrogate endpoints as outcomes, cross-over designs, short trial duration, and clinical relevance of the effect size. Recommendations for reducing or optimally managing differences included joint early dialogues, joint relative effectiveness assessments, and the use of managed entry agreements.

      Conclusions

      Differences between assessments of EMA and HTA bodies were identified in important areas of evidentiary requirements. Increased alignment between EMA and HTA bodies is suggested and recommendations for realization are discussed.

      Keywords

      Introduction

      Before a new drug can enter the European market and become accessible to patients, it must be granted marketing authorization (MA). During MA, the drug’s efficacy and safety are assessed.
      • Eichler H.G.
      • Bloechl-Daum B.
      • Abadie E.
      • Barnett D.
      • Konig F.
      • Pearson S.
      Relative efficacy of drugs: an emerging issue between regulatory agencies and third-party payers.
      The main routes for obtaining MA are the centralized route, that is, via the European Medicines Agency (EMA), or the national route.
      After obtaining MA, decisions about pricing and reimbursement need to be made, which are national or regional matters. Health technology assessment (HTA) bodies will assess the drug’s relative effectiveness for the decision on reimbursement.
      Authorisation of medicines. European Medicines Agency.
      These assessments by HTA bodies differ from the assessment for MA. In particular, the goals of MA and HTA are different, which is why the types of evidence required will also diverge. It is the responsibility of the manufacturer to submit data that are aligned with the requirements of the decision maker. Ideally, both assessments should be based on one and the same set of evidence, given that the time needed to generate new data for each process can be time consuming. Due to this difference in goals and specific national requirements, it often happens that new or additional evidence must be provided during the reimbursement process, and the process may be repeated in several countries. Therefore, improving the alignment on evidentiary requirements might be possible and could result in quicker and easier decisions on reimbursement at national level.
      Authorisation of medicines. European Medicines Agency.
      • Wang T.
      • McAuslane N.
      • Liberti L.
      • Leufkens H.
      • Hövels A.
      Building synergy between regulatory and HTA agencies beyond processes and procedures—can we effectively align the evidentiary requirements? A survey of stakeholder perceptions.
      • Ofori-Asenso R.
      • Hallgreen C.E.
      • De Bruin M.L.
      Improving interactions between health technology assessment bodies and regulatory agencies: a systematic review and cross-sectional survey on processes, progress, outcomes, and challenges.
      The emergence of more targeted oncology treatments has brought on regulatory changes in this area, which may affect the assessment.
      • Wilking N.
      • Bucsics A.
      • Sekulovic L.K.
      • et al.
      Achieving equal and timely access to innovative anticancer drugs in the European Union (EU): summary of a multidisciplinary CECOG-driven roundtable discussion with a focus on Eastern and South-Eastern EU countries.
      ,
      • Tafuri G.
      • Pagnini M.
      • Moseley J.
      • et al.
      How aligned are the perspectives of EU regulators and HTA bodies? A comparative analysis of regulatory-HTA parallel scientific advice.
      In addition, oncology drugs with a new active substance are required to obtain MA from EMA.
      Authorisation of medicines. European Medicines Agency.
      In the assessment of targeted oncology drugs, differences in assessments result from both higher-than-average uncertainty in the available data and from differences in the data accepted by EMA and those requested by HTA bodies. Regulators such as EMA appear to be more willing to accept a degree of uncertainty than HTA bodies.
      • Kleijnen S.
      • Lipska I.
      • Alves T.L.
      • et al.
      Relative effectiveness assessments of oncology medicines for pricing and reimbursement decisions in European countries.
      ,
      • Bloem L.T.
      • Vreman R.A.
      • Peeters N.W.L.
      • et al.
      Associations between uncertainties identified by the European Medicines Agency and national decision making on reimbursement by HTA agencies.
      Moreover, drugs following the EMA conditional MA pathway carry even more uncertainty, because of the use of immature data, and therefore are at higher risk of not being reimbursed by national HTA bodies.
      • Lipska I.
      • Hoekman J.
      • McAuslane N.
      • Leufkens H.
      • Hövels A.
      Does conditional approval for new oncology drugs in Europe lead to differences in health technology assessment decisions?.
      Despite the growing awareness of the differences in assessment, no study has yet described what these differences are for oncology drugs. This study, performed in collaboration with the European Federation of Pharmaceutical Industries and Associations, analyzes the differences in the acceptability of evidence (evidentiary requirements) for oncology drugs between EMA and national HTA bodies. Subsequently, recommendations on potential further alignment to minimize and optimally manage these differences according to the authorities will be provided.

      Methods

      A comparative analysis was conducted, in which the evidentiary requirements of EMA were compared with those of HTA bodies. Evidentiary requirements were defined as the minimum level of evidence accepted as convincing. The analysis was conducted in 3 steps:
      • 1.
        Identification and selection of 6 representative European HTA bodies, to capture the differences in assessments between EMA and national European HTA bodies
      • 2.
        Interviews with representatives working on drug assessments and drug assessment experts from EMA and the 6 HTA bodies, with a view to obtaining further details on what evidence is accepted in oncology assessments, and recommendations on potential further alignment to minimize and optimally manage potential differences in evidentiary requirements
      • 3.
        Analyzing the evidentiary requirements on acceptability of evidence and alignment between EMA and HTA bodies, based on analyses of information from steps 1 and 2 earlier
      Accepted classifications of domains and subdomains in evidentiary requirements were used, as well as publicly available oncology-specific assessment guidelines by EMA, as described below.

      Accepted Classifications of Domains and Subdomains

      The Shaping European Early Dialogues (SEED) was selected as a basis for classifying domains and subdomains, which represent the elements of discussion during an oncology assessment.
      • Facey K.
      • Granados A.
      • Meyer F.
      • et al.
      Multi-stakeholder collaboration in generating the best possible knowledge – the SEED experience. Report of HTAi Panel Session. Health Technology Assessment International.
      SEED specifies 6 domains, of which 5 were considered relevant for our study. The 5 domains are population, comparator, endpoints, trial design and data sources, and statistical analyses. The sixth domain of SEED, economic modeling, was excluded, because EMA assessments do not require economic modeling. The 5 domains were divided into 19 subdomains, adapting from the work by Tafuri et al.
      • Tafuri G.
      • Pagnini M.
      • Moseley J.
      • et al.
      How aligned are the perspectives of EU regulators and HTA bodies? A comparative analysis of regulatory-HTA parallel scientific advice.
      The domains and corresponding subdomains are listed in Table 1.
      • Tafuri G.
      • Pagnini M.
      • Moseley J.
      • et al.
      How aligned are the perspectives of EU regulators and HTA bodies? A comparative analysis of regulatory-HTA parallel scientific advice.
      ,
      • Facey K.
      • Granados A.
      • Meyer F.
      • et al.
      Multi-stakeholder collaboration in generating the best possible knowledge – the SEED experience. Report of HTAi Panel Session. Health Technology Assessment International.
      Progression-free survival (PFS) was separated from “other surrogate endpoints,” because PFS is widely used as a primary endpoint in oncology trials, despite the overall and general preference by HTA authorities for evidence on overall survival (OS). PFS is preferred as surrogate parameter for OS over other progression-based endpoints, such as time to progression or time to treatment failure, because only PFS includes death as part of its composite endpoint. An additional benefit of PFS as an endpoint is that it is measured before post progression treatments are initiated, so PFS data are not affected by post protocol agents. Furthermore, PFS is measured earlier and has a higher event frequency than OS. This is particularly important in cancer treatment, where OS is often more difficult to measure on the short term and there is a need to test multiple regimens, take account of ethical considerations, and respond to pressure from patient advocacy groups. Consequently, PFS results may be available sooner with smaller and less costly trials and potentially leading to earlier access for patients to reimbursed novel treatments. Finally, guidelines regarding anticancer treatment are updated continuously based on PFS results from clinical trials, next to evidence on OS. For example, these issues have been described in the article by Miksad.
      • Miksad R.A.
      • Zietemann V.
      • Gothe R.
      • et al.
      Progression-free survival as a surrogate endpoint in advanced breast cancer.
      Table 1Predefined domains and subdomains representing the elements of discussion during an oncology assessment.
      DomainsSubdomains
      Population
      • -
        Target population
      • -
        Use of biomarkers
      • -
        Extrapolation
      Comparator
      • -
        Selected comparator
      • -
        Class effects
      • -
        Indirect comparison
      Endpoints
      • -
        PFS as endpoint
      • -
        QoL/HRQoL and other PROs
      • -
        Other surrogate endpoints
      Trial design and data sources
      • -
        Real-world evidence
      • -
        Network meta-analyses
      • -
        Single-arm trials
      • -
        Novel trial designs
      • -
        Cross-over designs
      • -
        Evidence from small populations
      • -
        Acceptability of short time period
      Statistical analyses
      • -
        Statistical significance
      • -
        Post hoc subgroup analyses
      • -
        Clinical relevance of the effect size
      Note. Based on SEED and Tafuri et al.
      • Tafuri G.
      • Pagnini M.
      • Moseley J.
      • et al.
      How aligned are the perspectives of EU regulators and HTA bodies? A comparative analysis of regulatory-HTA parallel scientific advice.
      ,
      • Facey K.
      • Granados A.
      • Meyer F.
      • et al.
      Multi-stakeholder collaboration in generating the best possible knowledge – the SEED experience. Report of HTAi Panel Session. Health Technology Assessment International.
      HRQoL indicates health-related quality of life; PFS, progression-free survival; PRO, patient-reported outcome; QoL, quality of life; SEED, Shaping European Early Dialogues.

      Publicly Available Oncology-Specific Assessment Guidelines

      To provide guidance on clinical drug development, EMA developed guidelines on evidentiary requirements on what is considered acceptable safety and quality for oncology drugs.
      Evaluation of anticancer medicinal products in man
      European Medicines Agency.
      The first oncology guideline was adopted in 1996 and revised in 2006, whereas the latest version was adopted in 2017. The guideline includes every phase of clinical drug development. Deviation from these guidelines is possible but must be justified. EMA guidelines for assessing oncology drugs were obtained from the EMA website.
      Evaluation of anticancer medicinal products in man
      European Medicines Agency.

      Identification and Selection of HTA Bodies

      To make sure our selection of HTA bodies was representative of the different systems in place in Europe, specific selection criteria were applied. The included HTA bodies were selected on the basis of being representative of a heterogenous set, in terms of geographic location (north, east, south, and west), organization of the healthcare system (national vs regional/local),
      Medicines reimbursement policies in Europe
      WHO.
      • Angelis A.
      • Lange A.
      • Kanavos P.
      Using health technology assessment to assess the value of new medicines: results of a systematic review and expert consultation across eight European countries.
      • Jansen C.
      • Amesz B.
      Every day counts: improving time to patient access to innovative oncology therapies in Europe. EFPIA.
      time between MA and patient access (long, modest, and short, based on the European Federation of Pharmaceutical Industries and Associations Waiting to Access Innovative Therapies indicator),
      EFPIA patients W.A.I.T indicator 2019 survey
      IQVIA.
      and HTA orientation (budget impact, cost-effectiveness, and clinical effectiveness).

      Interviews

      The comparative analysis between HTA bodies and EMA was based on a combination of EMA guidelines and interviews held between August and September 2019. Structured qualitative interviews were conducted with representatives and drug assessment experts from inside and outside of the selected HTA bodies and EMA. An interview guide was designed to elicit the evidentiary requirements for assessment for oncology products and recommendations on potential further alignment to minimize and potentially manage differences in evidentiary requirements according to the authorities. The interview guide contained one interview question per identified subdomain, with exception of “PFS as an endpoint” and “quality of life/health-related quality of life and other patient reported outcomes.” These subdomains were combined into one question. EMA has developed guidelines on evidentiary requirements for oncology drugs. This means that responses to some of the interview questions could be identified before the interview. For those interview questions, the interviewee was asked to confirm that the evidentiary requirements had been correctly described. The interview questions are presented in Appendix 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.05.006.

      Data Analysis

      To facilitate comparison, answers containing the evidentiary requirements were categorized based on similarities between evidentiary requirements (axial coding). Interviewees were asked to validate the categories that emerged from their interview, also in relation to the categories emerging from the other interviewees.
      The data were analyzed following a generic approach on differences and similarities in evidentiary requirements and assessing comparative acceptability of evidence and regulatory alignment between EMA and national HTA bodies.
      • 1.
        To demonstrate the acceptability of the evidence by EMA and the HTA bodies, the evidentiary requirements of EMA and HTA bodies were labeled as “accepted,” “often accepted,” “case dependent,” “often not accepted,” or “not accepted.” The acceptability of the evidence by EMA and by the HTA bodies was quantified by calculating the ratio of evidentiary requirements labeled “accepted” or “often accepted” and the total number of evidentiary requirements, expressed as a percent.
      • 2.
        To demonstrate the alignment of all HTA bodies with EMA, the evidentiary requirements of the 19 subdomains of all HTA bodies were labeled as “aligned,” “often aligned,” “case dependent,” “often not aligned,” or “not aligned.” The degree of alignment with EMA was calculated for the alignment labels, by dividing the number of times an evidentiary requirement occurs in that label by the total number of evidentiary requirements and expressing it as a percent. The alignment with EMA was also calculated for each HTA body and for each subdomain. Obviously, subdomains with the least alignment indicated possible evidence gaps and potential areas where alignment between EMA and HTA bodies could be improved.
      Labeling was performed by 2 authors (S.W. and M.J.P.), and disagreements were solved through consensus. The decisions were based on the summarized answers with the general rule of thumb: When the evidence requirement states that it is accepted, it is labeled as “accepted.” When the evidence requirement states “accepted, if”/“accepted, but,” there is a chance of nonacceptance and therefore is labeled as “often accepted.” When there is even more chance of nonacceptance, an evidence requirement is labeled as “case dependent.” An overview of the above-described analysis of the acceptability of the evidence and alignment between HTA bodies and EMA is presented in Figure 1, with an example of the calculations in Appendix 2 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.05.006.
      Figure thumbnail gr1
      Figure 1Outline of methods used in this research project.
      EMA indicates European Medicines Agency; HTA, health technology assessment; SEED, Shaping European Early Dialogues.
      The recommendations given by interviewees were divided into 4 categories: “recommendations for EMA,” “recommendations for HTA bodies,” “recommendations for EMA and HTA bodies,” and “recommendations for manufacturers”.

      Results

      Based on the selection criteria, the following 6 countries were chosen: Italy, The Netherlands, Poland, Portugal, England (together with Wales), and Sweden. The criteria are presented in Table 2,
      EFPIA patients W.A.I.T indicator 2019 survey
      IQVIA.
      and an overview of the interviewees is presented in Appendix 3 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.05.006.
      Table 2Selection criteria for the 6 representative countries.
      CountryLocationAccess organizationTime between marketing authorization and patient access
      Based on the 34 countries included in the EFPIA W.A.I.T. indicator. Short: The 11 countries included in the EFPIA W.A.I.T. indicator with the least time to patient access. Modest: The 12 countries included in the EFPIA W.A.I.T. indicator with median time to patient access. Long: The 11 countries included in the EFPIA W.A.I.T. indicator with the longest time to patient access.16
      HTA orientation
      ItalySouthRegionalModestClinical effectiveness
      The NetherlandsWestNational and localShortCost-effectiveness
      PolandEastNationalLongBudget impact
      PortugalSouthNationalLongCost-effectiveness
      England (together with Wales)WestNationalShortCost-effectiveness
      SwedenNorthNationalShortCost-effectiveness
      EFPIA indicates European Federation of Pharmaceutical Industries and Associations; HTA, health technology assessment; W.A.I.T., Waiting to Access Innovative Therapies.
      Based on the 34 countries included in the EFPIA W.A.I.T. indicator. Short: The 11 countries included in the EFPIA W.A.I.T. indicator with the least time to patient access. Modest: The 12 countries included in the EFPIA W.A.I.T. indicator with median time to patient access. Long: The 11 countries included in the EFPIA W.A.I.T. indicator with the longest time to patient access.
      EFPIA patients W.A.I.T indicator 2019 survey
      IQVIA.
      Oncology-specific guidelines on the clinical evaluation were available for EMA, but not for national HTA bodies. All interviewees, except the Italian representative, were available for validation of the categories. The assigned categories for Italy have not been validated. The summary of EMA evidentiary requirements is presented in Appendix 4 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.05.006,
      Committee for Medicinal Products for Human Use (CHMP)
      Guideline on the evaluation of anticancer medicinal products in man. European Medicines Agency.
      Committee for Medicinal Products for Human Use (CHMP)
      Need for active control in therapeutic areas where use of placebo is deemed ethical and one or more established medicines are available. European Medicines Agency.
      Points to consider on application with 1. Meta-analyses; 2. One pivotal study. Committee for proprietary medicinal products (CPMP).
      and the summaries of the evidentiary requirements for the respective HTA bodies are presented in Appendix 5 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.05.006.

      Acceptability of the Evidence

      The evidentiary requirements of EMA (n = 18) and the 6 HTA bodies (n = 114) are presented in Table 3. “Target population as authorized by EMA” is not applicable to EMA, meaning it has no evidentiary requirements for this subdomain. As shown by color coding in Table 3, acceptability of the evidence by EMA is 72%, whereas the acceptability of evidence by HTA bodies is 55%.
      Table 3Evidentiary requirements of EMA and the 6 HTA bodies, labeled for the acceptability (symbols) and alignment (column division).
      DomainsSubdomainsEMAHealth technology assessment bodies
      AlignedOften alignedCase dependentOften not alignedNot aligned
      Population
      • Target population as authorized by EMA
      N/A.
      1 (17%)
      Accepted.
      N/A.
      3 (50%)

      Can be restricted to the subgroup that benefits most
      Case dependent.
      2 (33%)

      Restricted to the subgroup that benefits most§
      N/A.
      • Use of biomarkers
      Often accepted.
      6 (100%)

      When validated
      N/A.
      N/A.
      N/A.
      N/A.
      • Extrapolation to other populations
      Accepted.
      1 (17%)
      Accepted.
      1 (17%)
      Often accepted.
      2 (33%)

      Depending on justification (eg, children)
      Case dependent.
      2 (33%)

      Except for rare diseases / age groups
      Often not accepted.
      N/A.
      Comparator
      • Selected comparator
      Best available evidence-based therapeutic option
      Accepted.
      N/A.
      6 (100%)

      Standard of care/drug used in the clinical trial and available in the country
      Often accepted.
      N/A.
      N/A.
      N/A.
      • Class effects
      Accepted for safety, rarely considered for relative efficacy
      Case dependent.
      1 (17%)
      Accepted.
      N/A.
      3 (50%)

      Should be measured separately/ creates uncertainty/accepted for “me-too” drugs
      Case dependent.
      N/A.
      2 (33%)
      Not accepted.
      • Indirect comparisons
      When needed
      Accepted.
      3 (50%)

      When needed/when needed, creates more uncertainty
      Accepted.
      3 (50%)

      When needed and done in accordance with guidelines
      Often accepted.
      N/A.
      N/A.
      N/A.
      Endpoints
      • PFS as endpoint
      Accepted.
      1 (17%)
      Accepted.
      3 (50%)
      Often accepted.
      N/A.
      2 (33%)
      Often not accepted.
      N/A.
      • Other surrogate endpoints
      If there is a quantitative correlation
      Often accepted.
      1 (17%)
      Accepted.
      1 (17%)
      Often accepted.
      2 (33%)
      Case dependent.
      1 (17%)
      Often not accepted.
      1 (17%)
      Not accepted.
      • Absence of QoL data
      QoL is important
      Often not accepted.
      3 (50%)

      QoL is important
      Often not accepted.
      1 (17%)

      QoL is very important
      Not accepted.
      ,
      QoL refers in England (together with Wales) to the health state utility, rather than QoL.
      N/A.
      2 (33%)

      QoL is supportive
      Often accepted.
      N/A.
      Trial design and data sources
      • Real-world evidence
      But not enough (supportive)
      Accepted.
      6 (100%)

      But not enough (supportive)
      Accepted.
      N/A.
      N/A.
      N/A.
      N/A.
      • NMA
      Accepted.
      3 (50%)

      When needed/when needed, creates uncertainty
      Accepted.
      1 (17%)

      When needed and preferably published
      Often accepted.
      2 (33%)

      Could be accepted
      Case dependent.
      N/A.
      N/A.
      • Single-arm trials
      Creates uncertainty
      Accepted.
      3 (50%)

      Creates uncertainty/but treated as less strong evidence
      Accepted.
      2 (33%)

      If evidence is satisfactory/can be accepted, creates uncertainty
      Often accepted.
      N/A.
      1 (17%)
      Often not accepted.
      N/A.
      • Novel trial designs
      If evidence is satisfactory
      Often accepted.
      2 (33%)

      Creates uncertainty/accepted if accepted by EMA
      Accepted.
      3 (50%)

      If methodology is well-described/if controlled/if plausible biological mechanism
      Often accepted.
      1 (17%)
      Case dependent.
      N/A.
      N/A.
      • Cross-over in trials
      Creates uncertainty
      Accepted.
      1 (17%)

      Creates uncertainty
      Accepted.
      2 (33%)

      Can be accepted, creates uncertainty/but creates an interpretation challenge
      Often accepted.
      3 (50%)

      Case dependent/based on the influence on the interpretability of the results
      Case dependent.
      N/A.
      N/A.
      • Evidence from small populations
      Accepted.
      2 (33%)

      Creates uncertainty
      Accepted.
      4 (67%)

      If evidence is satisfactory/if it is the best available evidence/if requirements of GRADE methodology are met
      Often accepted.
      N/A.
      N/A.
      N/A.
      • Short time period
      Justification needed
      Often accepted.
      N/A.
      3 (50%)

      The longer the better/the longer the better, short period creates uncertainty/the longer the better, short period creates uncertainty, a convincing mean OS should be demonstrated
      Often accepted.
      2 (33%)

      Clinically relevant in context of representative for the natural history
      Case dependent.
      1 (17%)

      Hard endpoint should be identified within the time period
      Often not accepted.
      N/A.
      Statistical analysis
      • Absence of statistical significance
      Not accepted.
      4 (67%)
      Not accepted.
      N/A.
      2 (33%)

      Not very decisive (focus on confidence interval)/no hard cut-off point
      Case dependent.
      N/A.
      N/A.
      • Post hoc subgroup analyses
      Often not accepted.
      5 (83%)

      Unless requested by HTA body/unless drug does harm
      Often not accepted.
      1 (17%)
      Not accepted.
      N/A.
      N/A.
      N/A.
      • Clinical relevance of effect size as assessed by EMA
      Case dependent.
      1 (17%)

      Follows EMA
      Accepted.
      N/A.
      4 (67%)

      Own assessment, no threshold
      Case dependent.
      1 (17%)

      Own assessment (threshold 3 months OS)
      Often not accepted.
      N/A.
      Acceptability
      Acceptability was calculated by summing up the subdomains that were labeled as “accepted” or “often accepted,” dividing that by the total number of subdomains, and expressing it as a percent.
      13 (72%)63 (55%)
      Alignment with EMA
      Alignment with EMA was calculated by summing up the subdomains belonging to the alignment label, dividing that by the total number of subdomains, and expressing it as a percent.
      44 (39%)31 (27%)24 (21%)12 (10%)3 (3%)
      Note. Number (percent) in the cells of the evidentiary requirements demonstrate the number (proportion) of HTA bodies with this evidentiary requirement.
      EMA indicates European Medicines Agency; GRADE, Grading of Recommendations, Assessment, Development, and Evaluation; HTA, health technology assessments; N/A, not applicable; NMA, network meta-analysis; OS, overall survival; PFS, progression-free survival; QoL, quality of life.
      N/A.
      Accepted.
      Case dependent.
      § Often not accepted.
      Often accepted.
      Not accepted.
      £ QoL refers in England (together with Wales) to the health state utility, rather than QoL.
      ∗∗ Acceptability was calculated by summing up the subdomains that were labeled as “accepted” or “often accepted,” dividing that by the total number of subdomains, and expressing it as a percent.
      †† Alignment with EMA was calculated by summing up the subdomains belonging to the alignment label, dividing that by the total number of subdomains, and expressing it as a percent.

      Alignment Between EMA and HTA Bodies

      The alignment between EMA and HTA bodies on evidentiary requirements is presented in Table 3. Overall, the HTA bodies are aligned on evidentiary requirements with EMA for 39% of the subdomains, often aligned with EMA for 27%, case dependently aligned for 21%, often not aligned for 10%, and not aligned for 3%. As shown in Table 3, there are 2 subdomains where all countries are aligned with EMA—the use of biomarkers and the use of real-world evidence. The use of biomarkers is accepted when it is validated, and the use of real-world evidence is accepted, but not enough to solely base the assessment on. Real-world evidence is considered supportive data. Although 67% of the HTA bodies have similar acceptability of evidence as EMA on the “clinical relevance of effect size as assessed by EMA” subdomain, they are labeled as case dependent, not as aligned with EMA. The reason for this is that the implications of case dependent can differ for each agency. This is also the case for the subdomain “class effects,” where both EMA and 50% of HTA bodies are labeled case dependent. The frequency of alignment between EMA and each of the HTA bodies is presented in Figure 2.
      Figure thumbnail gr2
      Figure 2Frequency of alignment between EMA and each of the 6 HTA bodies and the overall alignment based on the 19 predefined subdomains.
      AIFA indicates Agenzia italiana del farmaco; AOTMiT, Agencja Oceny Technologii Medycznych i Taryfikacji; EMA, European Medicines Agency; HTA, health technology assessment; INFARMED, National Authority for Medicines and Health Products; NICE, National Institute for Health and Care Excellence; TLV, Dental and Pharmaceutical Benefits Agency; UK, United Kingdom; ZINL, Zorginstituut Nederland.

      Recommendations

      The recommendations for minimizing differences and addressing the remaining differences in evidentiary requirements (so-called evidence gaps) between EMA and HTA bodies can be grouped into recommendations for EMA, for HTA bodies, for both EMA and HTA bodies, and for manufacturers.
      Recommendation for EMA encompassed the following:
      • 1.
        Increasing awareness at EMA about the HTA assessments and their constraints. EMA should be more aware of the challenges HTA bodies are faced with when therapeutic indications are approved for all subgroups of patients through extrapolation.
      Recommendations for HTA bodies were as follows:
      • 1.
        Avoid duplication of EMA processes. Duplication can occur when relative effectiveness and safety play a major role in the assessments by some HTA bodies.
      • 2.
        Support managed entry agreements. The use of managed entry agreements could facilitate the handling of increasing quantities of immature data by HTA bodies owing to conditional MA. The use of immature data results in increased uncertainty in the assessment. A recommendation for HTA bodies dealing with this uncertainty is to support managed entry agreements, well-substantiated pricing, and financing proposals. Managed entry agreements can create flexibility in accepting some uncertainty and could therefore reduce the evidence gaps: it allows countries to compensate the uncertainty.
        • Pauwels K.
        • Huys I.
        • Vogler S.
        • Casteels M.
        • Simoens S.
        Managed entry agreements for oncology drugs: lessons from the European experience to inform the future.
      Recommendations for both EMA and HTA bodies include the following:
      • 1.
        Joint advice by EMA and HTA bodies. Although EMA assesses efficacy and safety, HTA bodies assess relative effectiveness. Assessing different aspects necessitates some divergence in the evidentiary requirements. Nevertheless, because the same data are assessed, EMA and HTA bodies could align on when evidence should be considered sufficient, based on this joint advice.
      • 2.
        More communication and harmonization between EMA and HTA bodies. This recommendation could lead to better alignment between EMA and HTA bodies on the requested data criteria. Examples of criteria on which EMA and HTA could be better aligned include the comparators that should be used in randomized controlled trials, the use of network meta-analyses, or other indirect comparisons.
      The recommendation for manufacturers is as follows:
      • 1.
        Conduct early dialogue with the HTA bodies and EMA. By using early dialogues, alignment on the requested data criteria, for example, with regard to the trial design, patient group (in alignment with the indication), and outcomes to be chosen, can be ensured. To have an effective early dialogue, strong evidence generation plans should be prepared. Manufacturers need to comply with the evidentiary requirements established in the early dialogues.

      Discussion

      This study showed that EMA had a higher degree of acceptability of evidence than HTA bodies, indicating that HTA bodies have different or more stringent evidentiary requirements. Stricter demands for more robust evidence by HTA bodies implicate that new evidence may need to be generated after authorization, prolonging the time to patient access. This highlights the importance for manufacturers to take the evidentiary requirements of HTA bodies into account in an early stage when designing and undertaking clinical trials. Nevertheless, with no oncology-specific guidelines on the clinical evaluation yet available for the 6 HTA bodies, this is more complicated. Development of oncology-specific guidelines on clinical evaluation is recommended.
      The evidentiary requirements by HTA bodies were more often case dependent than EMA. This reflects the struggle of managing uncertainty and makes the outcome of the assessment unpredictable, although it does not necessarily mean the evidence is unlikely to be accepted. With the use of managed entry agreements, uncertainty and unpredictability can be mitigated.
      As expected with differences in MA and HTA goals, there are relevant differences between the assessments of EMA and HTA bodies. Poland, England (together with Wales), and Sweden are most aligned with EMA on most subdomains, whereas the other countries have relevantly lower scores. Although some deviation is inevitable, our results indicate that enhanced alignment is at least feasible for the latter group of countries with lower scores. Given that Poland and Italy have the highest and lowest degree of alignment with EMA, respectively, it is striking that time to patient access is longer for Poland, according to the Waiting to Access Innovative Therapies indicator.
      EFPIA patients W.A.I.T indicator 2019 survey
      IQVIA.
      An explanation for this is that the time to patient access is influenced by more than the alignment on the subdomains, for example, by procedural differences of either the HTA body or the manufacturer. In addition, these results also suggest that there are substantial differences among the 6 HTA bodies, for example, on the use of surrogate endpoints, cross-over in trials, and the clinical relevance of the effect size as assessed by EMA. These differences between HTA bodies and how aligned HTA bodies are with each other should be analyzed and discussed in a future article.
      This analysis suggests that EMA and the HTA bodies are least aligned on the acceptability of target population and the extrapolation to other populations, the use of PFS as an endpoint and the use of surrogate endpoints, cross-over designs, duration of the trial, and the assessment of the clinical relevant effect size. These findings are in line with a former study, in which acceptable primary endpoints and the choice of the use of surrogate endpoints were identified as the main areas where alignment across regulatory and HTA evidentiary requirements could be realized.
      • Wang T.
      • McAuslane N.
      • Liberti L.
      • Leufkens H.
      • Hövels A.
      Building synergy between regulatory and HTA agencies beyond processes and procedures—can we effectively align the evidentiary requirements? A survey of stakeholder perceptions.
      The recommendations proposed by interviewees focused on early joint dialogues, joint relative effectiveness assessments, and the use of managed entry agreements. These recommendations are in line with the mechanisms for improving collaborations identified for medicines: early tripartite dialogues, alignment of evidentiary needs, parallel submissions (reviews), adaptive licensing pathways, and postmarketing data generation.
      • Ofori-Asenso R.
      • Hallgreen C.E.
      • De Bruin M.L.
      Improving interactions between health technology assessment bodies and regulatory agencies: a systematic review and cross-sectional survey on processes, progress, outcomes, and challenges.
      Pilot programs and initiatives related to the recommendations proposed by the interviewees have already demonstrated effectiveness. In 2010, a pilot program offering parallel scientific advice from EMA and HTA bodies showed that, in most cases, evidentiary requirements can be met within one trial design or one development program.
      Report of the pilot on parallel regulatory-health technology assessment scientific advice
      European Medicines Agency.
      The European Network for Health Technology Assessment is an example of an international body that performs joint relative effectiveness assessments with EMA that has replaced the parallel consultation with regulators and HTA bodies (EMA with the European Network for Health Technology Assessment).
      Parallel consultation with regulators and health technology assessment bodies
      European Medicines Agency.
      ,
      EMA and EUnetHTA take stock of their cooperation
      European Medicines Agency.
      Managed entry agreements could help overcome uncertainty created by immature data, as recommended, but could also help overcome uncertainty created by other limitations in the evidence, for example, for studies using a small trial population. An example of an initiative that provides a more flexible and tailor-made framework to allow earlier access whereas managing uncertainty is Accelerated Development of Appropriate Patient Therapies - a Sustainable, Multi-stakeholder Approach from Research to Treatment-outcomes (ADAPT-SMART).
      • Wilking N.
      • Bucsics A.
      • Sekulovic L.K.
      • et al.
      Achieving equal and timely access to innovative anticancer drugs in the European Union (EU): summary of a multidisciplinary CECOG-driven roundtable discussion with a focus on Eastern and South-Eastern EU countries.
      Interviewees in this research project indicated that a high degree of uncertainty may have an impact on HTA assessment. This is in line with a study that researched the effect of uncertainty on HTA decisions.
      • Bloem L.T.
      • Vreman R.A.
      • Peeters N.W.L.
      • et al.
      Associations between uncertainties identified by the European Medicines Agency and national decision making on reimbursement by HTA agencies.
      In that study, high uncertainty was related a 1.9-fold increased risk of negative relative efficacy assessments and 1.6-fold increased risk of negative overall reimbursement recommendations. That study suggested that, with medium level of uncertainty, clinical and economic restrictions, unmet medical need, and price-related aspects could help HTA bodies consider the uncertainty as acceptable. With oncology drugs, there is often a high unmet medical need.
      • Bloem L.T.
      • Vreman R.A.
      • Peeters N.W.L.
      • et al.
      Associations between uncertainties identified by the European Medicines Agency and national decision making on reimbursement by HTA agencies.
      A non–oncology-specific comparative analysis showed that a high level of full agreement was reached between European Union regulators and participating HTA bodies, through the process of parallel scientific advice.
      • Tafuri G.
      • Pagnini M.
      • Moseley J.
      • et al.
      How aligned are the perspectives of EU regulators and HTA bodies? A comparative analysis of regulatory-HTA parallel scientific advice.
      This differs from our analysis, which showed a low level of full alignment between HTA bodies and EMA and a high level of partial alignment. The higher level of full alignment in parallel scientific advice might suggest that the use of joint relative effectiveness assessment advice or joint relative effectiveness assessment consultation could result in a higher level of alignment.
      • Tafuri G.
      • Pagnini M.
      • Moseley J.
      • et al.
      How aligned are the perspectives of EU regulators and HTA bodies? A comparative analysis of regulatory-HTA parallel scientific advice.
      A non–oncology-specific, survey-based study on the synergy between regulatory and HTA agencies presented similar recommendations on evidentiary requirements as we did in our study.
      Authorisation of medicines. European Medicines Agency.
      That study specified one additional recommendation, which was to “align where appropriate and acknowledge national differences,” suggesting that there is no need or place for total alignment. Recommendations should not be limited to obtaining total alignment, but also include ways of optimally managing differences in evidentiary requirements.
      The present article provides an initial overview of the evidence gaps between EMA and European HTA bodies. It is a snapshot—the evidentiary requirements and acceptability of evidence will change over time. Therefore, these changes in acceptability should be monitored, so it can be demonstrated whether the evidence gaps become a smaller or larger problem.
      A limitation of this research is the limited number of selected countries used to represent Europe. It is possible that other differences in evidentiary requirements would have been identified, had more countries been included. Nevertheless, we tried to include countries that are representative for all European national HTA systems. Further research on more European countries is needed to achieve a complete overview of evidentiary requirements in European Union member states.

      Conclusions

      This study identified the differences in evidentiary requirements for the assessment of oncology drugs by EMA and HTA bodies. EMA and the HTA bodies were least aligned on the following subdomains: target population and the extrapolation to other populations, the use of PFS as an endpoint and the use of other surrogate endpoints, cross-over designs, trial duration, and the clinical relevant effect size. Therefore, for these subdomains, alignment could be improved. Recommendations on potential further alignment to minimize and optimally manage the remaining differences include early joint dialogues, joint relative effectiveness assessments, and the use of managed entry agreements.

      Article and Author Information

      Author Contributions: Concept and design: Wolters, Postma
      Acquisition of data: Wolters, Postma
      Analysis and interpretation of data: Wolters, Jansman, Postma
      Drafting of the manuscript: Wolters, Jansman, Postma
      Critical revision of the paper for important intellectual content: Jansman, Postma
      Administrative, technical, or logistic support: Postma
      Supervision: Jansman, Postma
      Conflict of Interest Disclosures: Dr Wolters is employed by Asc Academics and reported that Asc Academics received payment from the European Federation of Pharmaceutical Industries and Associations during the conduct of this study for a project called “Time to Patient Access,” on which this manuscript is based. Asc Academics had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. Dr Postma reported stock ownership in Pharmacoeconomics Advice Groningen and Health-Ecore outside the submitted work; he is also an editor for Value in Health and had no role in the peer-review process for this article. No other disclosures were reported.
      Funding/Support: This work was supported by from European Federation of Pharmaceutical Industries and Associations .
      Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

      Acknowledgment

      Analysis of the data was supported by Christel Jansen, MSc.

      Supplemental Materials

      References

        • Eichler H.G.
        • Bloechl-Daum B.
        • Abadie E.
        • Barnett D.
        • Konig F.
        • Pearson S.
        Relative efficacy of drugs: an emerging issue between regulatory agencies and third-party payers.
        Nat Rev Drug Discov. 2010; 9: 277-291
      1. Authorisation of medicines. European Medicines Agency.
        • Wang T.
        • McAuslane N.
        • Liberti L.
        • Leufkens H.
        • Hövels A.
        Building synergy between regulatory and HTA agencies beyond processes and procedures—can we effectively align the evidentiary requirements? A survey of stakeholder perceptions.
        Value Health. 2018; 21: 707-714
        • Ofori-Asenso R.
        • Hallgreen C.E.
        • De Bruin M.L.
        Improving interactions between health technology assessment bodies and regulatory agencies: a systematic review and cross-sectional survey on processes, progress, outcomes, and challenges.
        Front Med (Lausanne). 2020; 7: 582634
        • Wilking N.
        • Bucsics A.
        • Sekulovic L.K.
        • et al.
        Achieving equal and timely access to innovative anticancer drugs in the European Union (EU): summary of a multidisciplinary CECOG-driven roundtable discussion with a focus on Eastern and South-Eastern EU countries.
        ESMO Open. 2019; 4e000550
        • Tafuri G.
        • Pagnini M.
        • Moseley J.
        • et al.
        How aligned are the perspectives of EU regulators and HTA bodies? A comparative analysis of regulatory-HTA parallel scientific advice.
        Br J Clin Pharmacol. 2016; 82: 965-973
        • Kleijnen S.
        • Lipska I.
        • Alves T.L.
        • et al.
        Relative effectiveness assessments of oncology medicines for pricing and reimbursement decisions in European countries.
        Ann Oncol. 2016; 27: 1768-1775
        • Bloem L.T.
        • Vreman R.A.
        • Peeters N.W.L.
        • et al.
        Associations between uncertainties identified by the European Medicines Agency and national decision making on reimbursement by HTA agencies.
        CTS Clin Transl Sci. 2021; 14: 1566-1577
        • Lipska I.
        • Hoekman J.
        • McAuslane N.
        • Leufkens H.
        • Hövels A.
        Does conditional approval for new oncology drugs in Europe lead to differences in health technology assessment decisions?.
        Clin Pharmacol Ther. 2015; 98: 489-491
        • Facey K.
        • Granados A.
        • Meyer F.
        • et al.
        Multi-stakeholder collaboration in generating the best possible knowledge – the SEED experience. Report of HTAi Panel Session. Health Technology Assessment International.
        • Miksad R.A.
        • Zietemann V.
        • Gothe R.
        • et al.
        Progression-free survival as a surrogate endpoint in advanced breast cancer.
        Int J Technol Assess Health Care. 2008; 24: 371-383
        • Evaluation of anticancer medicinal products in man
        European Medicines Agency.
        • Medicines reimbursement policies in Europe
        WHO.
        • Angelis A.
        • Lange A.
        • Kanavos P.
        Using health technology assessment to assess the value of new medicines: results of a systematic review and expert consultation across eight European countries.
        Eur J Health Econ. 2018; 19: 123-152
        • Jansen C.
        • Amesz B.
        Every day counts: improving time to patient access to innovative oncology therapies in Europe. EFPIA.
        • EFPIA patients W.A.I.T indicator 2019 survey
        IQVIA.
        • Committee for Medicinal Products for Human Use (CHMP)
        Guideline on the evaluation of anticancer medicinal products in man. European Medicines Agency.
        • Committee for Medicinal Products for Human Use (CHMP)
        Need for active control in therapeutic areas where use of placebo is deemed ethical and one or more established medicines are available. European Medicines Agency.
      2. Points to consider on application with 1. Meta-analyses; 2. One pivotal study. Committee for proprietary medicinal products (CPMP).
        • Pauwels K.
        • Huys I.
        • Vogler S.
        • Casteels M.
        • Simoens S.
        Managed entry agreements for oncology drugs: lessons from the European experience to inform the future.
        Front Pharmacol. 2017; 8: 171
        • Report of the pilot on parallel regulatory-health technology assessment scientific advice
        European Medicines Agency.
        • Parallel consultation with regulators and health technology assessment bodies
        European Medicines Agency.
        • EMA and EUnetHTA take stock of their cooperation
        European Medicines Agency.