Advertisement

Improving the Contribution of Regulatory Assessment Reports to Health Technology Assessments—A Collaboration between the European Medicines Agency and the European network for Health Technology Assessment

Open ArchivePublished:June 30, 2014DOI:https://doi.org/10.1016/j.jval.2014.04.006

      Abstract

      In response to a recommendation from the Pharmaceutical Forum, the European Medicines Agency and the European network for Health Technology Assessment initiated a collaboration with the aim to improve the contribution regulatory assessment reports can make to the assessment of relative effectiveness of medicinal products by health technology assessment bodies. This collaboration on improving European Public Assessment Reports (EPARs) started in February 2010 and was performed over 2 years. As a result, the templates for preparing EPARs were revised to better address the needs of heath technology organizations. The better understanding of information needs was a key outcome of the collaboration. To ascertain whether these template changes led to the inclusion of relevant information, a review of a small set of EPARs for recently approved medicinal products was carried out in parallel by both the European network for Health Technology Assessment and the European Medicines Agency. This report provides an account of this project on improving EPARs, which is part of the ongoing dialogue between regulators and health technology assessment bodies on a European level to support policymaker decisions in the future.

      Keywords

      Introduction

      Scientific evaluation of the clinical data produced during drug development can be intended to estimate the benefit/risk ratio of the product (recently in some jurisdictions also referred to as benefit/harm/uncertainty), for the purpose of marketing authorization, or to estimate the effectiveness of the new product as compared with existing therapies, as part of the health technology assessment (HTA) process to support decision making on price and reimbursement.
      The European Medicines Agency (EMA) is responsible for the scientific evaluation of applications for European Union (EU) marketing authorizations for medicinal products in the so-called centralized procedure. Under this procedure, a single marketing-authorization application is submitted to the EMA and once authorization is granted by the European Commission on the basis of a pan-European scientific assessment of quality, efficacy, and safety by EMA committees, a centralized marketing authorization is valid in all EU member states.
      The subsequent evaluation and decision-making process leading to decisions on the pricing and reimbursement of medicinal products lie with the national bodies. HTA is a multidisciplinary process that summarizes information about the medical, social, economic, and ethical issues related to the use of a health technology in a systematic, transparent, unbiased, and robust manner. Its aim is to inform the formulation of safe, effective health policies that are patient focused and seek to achieve best value [
      • Kristensen F.B.
      • Mäkelä M.
      • Allgurin Neikter S.
      • et al.
      European network for Health Technology Assessment, EUnetHTA: planning, development, and implementation of a sustainable European network for Health Technology Assessment.
      ]. Criteria for HTA vary between countries, but generally HTA bodies in Europe use relative effectiveness assessment (REA), a European equivalent to comparative effectiveness research in the United States, as part of the HTA process [

      Kleijnen S, Goettsch W, d’Andon A, et al. Relative effectiveness assessment (REA) of pharmaceuticals—background review, July 2011. Available from: www.eunethta.eu. [Accessed March 6, 2014].

      ].
      The European network for Health Technology Assessment (EUnetHTA) is a network of organizations appointed by EU member states that produce or contribute to HTA to facilitate efficient use of resources available for HTA, to create a sustainable system of HTA knowledge sharing, and to promote good practice in HTA methods and processes [
      • Kristensen F.B.
      • Mäkelä M.
      • Allgurin Neikter S.
      • et al.
      European network for Health Technology Assessment, EUnetHTA: planning, development, and implementation of a sustainable European network for Health Technology Assessment.
      ,

      EUnetHTA. Organisation. Available from: http://www.eunethta.eu/about-us/organisation. [Accessed March 6, 2014].

      ]. Originally started as a project in 2006, EUnetHTA was established to create an effective and sustainable network for HTA across Europe [
      • Kristensen F.B.
      Development of European HTA: from Vision to EUnetHTA.
      ].
      In October 2008, the Pharmaceutical Forum, a high-level ministerial platform for discussion between member states, EU institutions, industry, health care professionals, patients, and insurance funds, agreed on conclusions and recommendations to find relevant solutions to public health considerations regarding pharmaceuticals, while ensuring the competitiveness of the industry and the sustainability of the national health care systems [

      HLPF conclusions and recommendations. Available from: http://ec.europa.eu/enterprise/sectors/healthcare/files/docs/pharmaforum_final_conclusions_en.pdf. [Accessed October 30, 2013].

      ]. One of these recommendations provided a political mandate to initiate collaboration between the EMA and EUnetHTA with the aim to improve the availability and best use of data relevant to HTA. Specifically, the objective of this joint project of regulators and HTA bodies was to “consider how information in the European Public Assessment Report /…/ can further contribute to relative effectiveness assessment” [

      HLPF conclusions and recommendations. Available from: http://ec.europa.eu/enterprise/sectors/healthcare/files/docs/pharmaforum_final_conclusions_en.pdf. [Accessed October 30, 2013].

      ].
      The regulatory assessment reports, which are the basis for central marketing authorizations, are publicly available as so-called European Public Assessment Reports (EPARs). EPARs reflect the scientific conclusions reached by the EMA’s Committee for Medicinal Products for Human Use (CHMP) at the end of the regulatory review process after deletion of commercially confidential information. These are published on the EMA’s Web site for every medicine authorized through the centralized procedure in the EU. Because these contain the CHMP’s judgment on the methodological quality of the studies and the relevance and significance of results, in the perspective of the estimate of the balance between favorable and unfavorable effects for the medicinal product, they may also be used as source documents for REAs from the HTA perspective. In particular, such assessment reports are used in the context of rapid REA of pharmaceuticals [

      EUnetHTA. EUnetHTA rapid REA core model. Available from: www.eunethta.eu. [Accessed October 30, 2013].

      ].
      The collaboration between the EMA and EUnetHTA on EPARs started in February 2010 and was performed over 2 years. It has been subject to a wider consultation within EUnetHTA as part of the Work Package 5 on REA of Pharmaceuticals of EUnetHTA Joint Action 1. This report provides an account of various activities as part of the project to address the recommendation on improving the EPAR.

      Methods

      In the first phase, EUnetHTA provided input on the usefulness of published EPARs and Summary of Product Characteristics (SmPC) in the context of REA of medicinal products based on the initial work performed in 2009 by some partners (which are also EUnetHTA partners) of the Medicine Evaluation Committee (MEDEV), a standing working group of the European Social Health Insurance Forum. This input was reviewed by the EMA and an action plan was established, which was mutually agreed between the EMA and EUnetHTA. Subsequently, the templates and guidance documents for the assessment of initial marketing authorization applications were amended by the EMA to implement the agreed improvements through either specific sections or supportive guidance. The revised templates and guidance documents were presented to the CHMP for adoption and subsequently published on the EMA’s Web site (www.ema.europa.eu) for use by assessors in the centralized procedure (>Home >Regulatory >Human medicines >Pre-authorisation >Templates for assessors).
      The second phase was the critical review of EPARs to establish compliance with revised templates. For this purpose, EPARs of the first 10 products, for which the assessment report has been prepared after the revised templates came into operation, were analyzed for compliance with specific aspects amended in the template. The following EPARs were reviewed: Esbriet (pirfenidone), Gilenya (fingolimod), Halaven (eribulin), Jevtana (cabazitaxel), Pravafenix (fenofibrate/pravastatin), Pumarix (pandemic influenza vaccine [H5N1]), Teysuno (tegafur/gimeracil/oteracil), Trobalt (retigabine), Xeplion (paliperidone), and Xiapex (collagenase clostridium histolyticum). The review took place in second and third quarters of 2011 on the basis of the publicly available EPARs. A questionnaire with 36 questions was developed by the EMA targeting the identified areas for improvement. Two types of answers were attributed depending on the nature of the question: either the option “yes/no/not applicable” (13 questions) if the underlying question was simply whether an item is included or the grading “excellent/good/could be improved/no/not applicable” (23 questions) if a more differentiated review was considered more meaningful. Table 1 indicates for each review item the type of question. Numerical values were assigned against the answers (“1/0” and “3/2/1/0” for binary and graded answers, respectively) to allow for statistical analysis of compliance of the EPARs for each individual question. The review of the 10 EPARs was performed in parallel by the EMA and by EUnetHTA, always involving two reviewers per organization and using the same questionnaire and numerical scales. In case of divergent views between the two reviewers, consensus was formed. One additional question on the benefit-risk assessment was added by HTA organizations at the time of EUnetHTA review. As regards EUnetHTA review, each EPAR was evaluated by two HTA organizations from EUnetHTA. Ten HTA organizations (see Acknowledgment) participated in this exercise coordinated by the French National Authority for Health (HAS). The results were calculated as mean values in terms of the overall compliance rate across all 10 EPARs and reviewers for each of the two organizations (i.e., if the answer for a binary question was “yes” [“1”] in 6 out of 10 EPARs, a 60% overall compliance rate was attributed). For descriptive purpose, the compliance rates are clustered in three distinct categories: more than 80% compliance (mean), 50% to 80% compliance (mean), and less than 50% compliance (mean).
      Table 1Outcome of the review of compliance of EPARs with the template/guidance (N = 10; categorization based on mean values).
      Because a separate reference listing is required only for EPARs with referencing to numerous publications and none of the EPARs subject to the review did fulfill this criterion, the review item “separate reference listing” (planned to be reviewed using a binary question) was not applicable and is therefore not reported in terms of compliance.
      Overall compliance (%) accounted to EUnetHTA reviewReview itemOverall compliance (%) accounted to the EMA review
      <5050–80>80<5050–80>80
      Format and scientific content
      XTable of contents
      Item reviewed using a binary question; i.e., the aspect is included “yes/no.”
      X
      XList of abbreviations
      Item reviewed using a binary question; i.e., the aspect is included “yes/no.”
      X
      XClear referencing of data from publications
      Item reviewed using a binary question; i.e., the aspect is included “yes/no.”
      X
      XStructural formula for chemical substances
      Item reviewed using a binary question; i.e., the aspect is included “yes/no.”
      X
      XStructural characteristics for biologicals
      Item reviewed using a binary question; i.e., the aspect is included “yes/no.”
      X
      XDescription of standard treatments in the EU
      Item reviewed using a binary question; i.e., the aspect is included “yes/no.”
      X
      XDiscussion of compliance with legal requirements
      Item reviewed using a binary question; i.e., the aspect is included “yes/no.”
      X
      XDiscussion of compliance with scientific guidelines (EMA/CHMP)
      Item reviewed using a binary question; i.e., the aspect is included “yes/no.”
      X
      XDiscussion of compliance with scientific advice
      Item reviewed using a binary question; i.e., the aspect is included “yes/no.”
      X
      XDiscussion of outcome of any GCP inspection and its impact on data reliability
      Item reviewed using a binary question; i.e., the aspect is included “yes/no.”
      X
      XDiscussion of key elements of the study design: Patient population
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      XDiscussion of key elements of the study design: Comparators
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      XDiscussion of key elements of the study design: Duration of the study
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      XDiscussion of key elements of the study design: End points and/or composite end point
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      XDisplay of participant flow (graphically or tabular)
      Item reviewed using a binary question; i.e., the aspect is included “yes/no.”
      X
      XSummary of the main efficacy data in the template table
      Item reviewed using a binary question; i.e., the aspect is included “yes/no.”
      X
      XExplanation for reasoning for additional analyses, if requested
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      XExplanation if a subgroup data were considered of particular relevance for the overall assessment of efficacy
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      XJustification for waiver of study or replacement by literature data
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      XHighlighting of shortcomings of the efficacy data including impact on the assessment
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      XReflection of additional input from external experts (SAG, ad-hoc expert group, PDCO), if requested
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      XRationale for deciding that the risk-benefit balance is positive is adequately discussed
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      ,
      Item added by HTA organizations at the time of EUnetHTA review.
      Support for summary of product characteristics (SmPC)
      Only aspects of sections 4.1, 4.2, and 5.1 (Information on approved therapeutic indication) were topics specifically addressed through revisions of template/guidance.
      XSmPC section 4.1: Reflection of approved therapeutic indication including selection of patient population and age range, as applicable
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      XSmPC section 4.2: Substantiation of dose recommendations
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      XSmPC section 4.3: Substantiation of contraindications
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      XSmPC section 4.4: Substantiation of warning/precautions for use
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      XSmPC section 4.5: Substantiation of interaction statements
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      XSmPC section 4.6: Substantiation of use during pregnancy and lactation
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      XSmPC section 4.7: Substantiation of effects on ability to drive and use machines
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      XSmPC section 4.8: Substantiation of adverse drug reaction profile
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      XSmPC section 4.8: Definition of ADRs consistent between SmPC and Assessment Report
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      XSmPC section 4.9: Substantiation of information on overdose
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      XSmPC section 5.1: Information on approved therapeutic indication(s) in line with information in the Assessment Report
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      XSmPC section 5.1: Available data in the pediatric population
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      XSmPC section 5.2: Substantiation of pharmacokinetic properties
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      X
      ADR, adverse drug reaction; CHMP, Committee for Medicinal Products for Human Use; EMA, European Medicines Agency; GCP, good clinical practice; HTA, health technology assessment; EPAR, European Public Assessment Report; EU, European Union; EUnetHTA, European network for Health Technology Assessment; PDCO, Paediatric Committee; SAG, Scientific Advisory Group.
      low asterisk Because a separate reference listing is required only for EPARs with referencing to numerous publications and none of the EPARs subject to the review did fulfill this criterion, the review item “separate reference listing” (planned to be reviewed using a binary question) was not applicable and is therefore not reported in terms of compliance.
      Item reviewed using a binary question; i.e., the aspect is included “yes/no.”
      Item reviewed using a graded question; i.e., the aspect is included “excellent/good/could be improved/no.”
      § Item added by HTA organizations at the time of EUnetHTA review.
      || Only aspects of sections 4.1, 4.2, and 5.1 (Information on approved therapeutic indication) were topics specifically addressed through revisions of template/guidance.

      Results

      As a result of the project, the templates for preparing EPARs were revised to address the comments received. Furthermore, a review of a small set of EPARs was carried out in parallel by both EUnetHTA and the EMA to ascertain whether these template changes led to the inclusion of relevant information. The following sections present these results separately.

      Revision of Templates for Preparing EPARs

      Figure 1 displays the development of revisions to templates for regulatory assessments with the aim to improve the contribution such reports can make to the assessment of relative effectiveness of medicinal products. The initial analysis of a compilation of comments from EUnetHTA/MEDEV identified 34 topic areas for further discussion. These topics areas concerned the format and structure, the scientific content, the evaluation criteria, and other aspects of the assessment report. A detailed discussion of each item revealed opportunities for improvement of template and guidance as well as areas in which adherence to existing template and guidance might need to be improved. Certain items concerned the methodology for conducting an assessment rather than data presentation, such as the acceptability of certain surrogate parameters or the conduct of a wider literature search; these items were excluded from the exercise. Most of the items, which were addressed through revisions to the templates, concerned the scientific content of the assessment report, particularly, the clinical aspects. In most instances, the agreed items are related to a more detailed data presentation or more explicit reasoning of considerations for the regulatory decision making. In addition, specific sections of the SmPC have been identified in which a better substantiation through information and discussion in the assessment report would be beneficial.
      Figure thumbnail gr1
      Fig. 1Development of revisions to templates for regulatory assessments with the aim to improve the contribution such reports can make to the assessment of relative effectiveness of medicinal products. CHMP, Committee for Medicinal Products for Human Use; EMA, European Medicines Agency; EPAR, European Public Assessment Report; EU, European Union; EUnetHTA, European network for Health Technology Assessment; PL, package leaflet; QC, quality control; SAG, Scientific Advisory Group.
      The subsequent revisions concerned various parts of the existing templates and guidance documents. In Figure 1, a tabular overview of the revisions to the templates is provided, identifying updates sections and describing the changes introduced. As an example, for the discussion of key elements of the study design, a new subsection was created in the “Discussion on clinical efficacy” of the assessment report. In this new subsection titled “Design and conduct of clinical studies,” a critical discussion should address among others whether the design of the studies is deemed adequate, the patient population was adequately selected, the comparator considered appropriate, an active comparator was relevant in view of EU-approved treatment options, and the choice of end points and duration of study was adequate considering regulatory guidance. To address the request for better explanation of the reasoning for additional analysis, relevance of a particular subgroup, and the appropriateness to waive studies or replace these against literature data, specific guidance was provided in a new subsection “Efficacy data and additional analysis.”
      A template table for a structured summary of the main efficacy data was developed jointly by the EMA and EUnetHTA and has been newly introduced into the assessment reports. Figure 2 shows this template together with the respective guidance. The objective was to provide a tabulated summary of the efficacy data that were considered most relevant for the regulatory decision making. For each of the main studies, a separate table should be presented detailing the results according to the prespecified analysis as well as any other analysis that is deemed relevant. Because it is foreseen to allow for a plain data presentation, this table should be read in conjunction with the discussion and conclusion on clinical efficacy, as well as the benefit-risk assessment in later parts of the assessment report.
      Figure thumbnail gr2a
      Fig. 2Template for the summary of main efficacy data for the regulatory assessment report with guidance (in italics). ANOVA, analysis of variance; CHMP, Committee for Medicinal Products for Human Use; EudraCT, European Union Drug Regulating Authorities Clinical Trials; ISRCT, International Standard Randomized Controlled Trial; ITT, intention to treat; mITT, modified intention to treat PP, per protocol.
      Figure thumbnail gr2b
      Fig. 2Template for the summary of main efficacy data for the regulatory assessment report with guidance (in italics). ANOVA, analysis of variance; CHMP, Committee for Medicinal Products for Human Use; EudraCT, European Union Drug Regulating Authorities Clinical Trials; ISRCT, International Standard Randomized Controlled Trial; ITT, intention to treat; mITT, modified intention to treat PP, per protocol.

      Parallel Review of EPARs

      The outcome of the review is presented in Table 1. For each review item, the overall compliance with the template across all 10 EPARs is displayed in three categories. This is done separately for the review by EUnetHTA and by the EMA to allow for an overview of the ranking by each of the two organizations as well as a comparison of the results between the two reviews.
      With regard to the format of the EPAR, the two reviews showed that a table of contents was always included whereas individual publications were not always clearly referenced. With regard to the scientific content of the EPAR, a high compliance rate was assigned by both the EMA and EUnetHTA for the presentation of patient flow as graphic or table as well as the newly introduced summary table of main efficacy results and analysis. However, low compliance rates were identified regarding the presentation of discussion on the key elements of the study design including patient populations, comparators, duration of the study, and end points, as also shown in Figure 3.
      Figure thumbnail gr3
      Fig. 3Compliance of EPARs with guidance in templates regarding the presentation of discussion on the key elements of study designs (N = 10). EMA, European Medicines Agency; EPAR, European Public Assessment Report; EUnetHTA, European network for Health Technology Assessment.
      The explanation for reasoning of additional data analyses requested by the CHMP as well as the emphasis on shortcoming of the efficacy data were highlighted by EUnetHTA as critical areas of the clinical efficacy discussion; the compliance according to EunetHTA review was found below 50% on these two items. In addition, explanations when a subgroup analysis is considered of particular relevance, justifications for waiver or replacement of a study by literature data, and reflection of additional input from external expert were also considered of particular importance. These review items were moderately compliant from EUnetHTA perspective, contrary to EMA, which rated these items above 80% compliance, except for the justification for waiver or replacement of a study by literature data (<50%). Regarding the additional item reviewed by EUnetHTA only on the benefit-to-risk assessment, a compliance rate indicating satisfactory adherence to template guidance was documented.
      The support of the SmPC through the assessment reports was considered by EUnetHTA as appropriate for most of the review items (between the range of 50% and 80%); however, none of them was above 80%. Four items were reviewed below 50% of compliance, notably for the substantiation of contraindications. The EMA attributed a compliance rate of 50% to 80% and more than 80% for 10 and 3 of these items, respectively. For 7 of the 13 review items including the sections on therapeutic indication, dose recommendations (particularly deviations from standard dose), and warnings and precautions for use and interactions, the two organizations concluded on 50% to 80% compliance. Specifically for the review items stemming from the initial EUnetHTA/MEDEV comments, namely, the reflection of approved therapeutic indication in section 4.1, the substantiation of dose recommendations in section 4.2, and the information on approved therapeutic indication(s) in section 5.1, both reviews assigned a compliance category of 50% to 80%.

      Discussion

      This joint project between the EMA and EUnetHTA aimed at improving the contribution EPARs can make to the assessment of relative effectiveness by European HTA bodies. EPARs are published regulatory assessment reports supporting the licensing of medicinal products that are centrally authorized in Europe. Given the scope of the centralized procedure, almost all new therapeutics (new molecular entities and biologics) receive their marketing authorization through this licensing route [

      European Medicines Agency. Marketing authorisations for new active substances reviewed in the centralised procedure and the decentralised/mutual recognition procedure, respectively; data on file.

      ].
      Based on observations on the usefulness of EPARs and SmPCs from an HTA perspective, the EMA and EUnetHTA agreed on specific items for improvement of the presentation of data and information in the EPAR. The emphasis was on a better understanding of the regulatory judgment, such as critical discussions of study design as well as specific subgroup analysis. Furthermore, a structured and harmonized presentation of main efficacy data was requested in view of clarity and transparency of the data that is deemed most important from a regulatory perspective. Through changes in the templates and guidance documents for assessment reports, these agreed topics were specifically targeted. The revisions—if adhered to when writing the actual reports—were aimed to ensure that the relevant data and information are provided with the published documents.
      The review of actual EPARs explored how well these revisions are translated into practice. Using the same methodology, both the EMA and EUnetHTA came to the view that the compliance differs across various aspects in the assessment report. Given that the EPAR format and scientific content showed overall high rates of compliance with the revised templates, it can be assumed that the quality of the documents has been improved even if no such assessment of previous EPARs with the same methodology was performed to quantify the difference. An achievement was the introduction of the tabular overview of main efficacy data, which was widely adhered to. Also, more structural aspects in the EPAR were implemented. There remains, however, space for further improvement in the critical discussion of the key elements of the clinical study design, that is, on patient population (including subpopulation and special populations), comparators, duration of the study, end points, and/or composite end-point use. Some of these aspects may be present in the clinical efficacy discussion but are not visible enough or not discussed enough. The substantiation of the SmPC was variable, with some areas requiring special attention mainly for the sections concerning the dose recommendations (particularly deviations from standard dose), contraindications, warnings and precautions, and interactions.
      The most divergent view between the reviews of the two organizations seemed to be regarding the discussion of the shortcomings of efficacy data as well as the identification and explanation of additional analysis requested during the regulatory review. These items are considered as critical areas of the clinical efficacy discussion from EUnetHTA perspective, as well as explanations when a subgroup analysis is considered of particular relevance, justifications for waiver or replacement of a study by literature data, and reflection of additional input from an external expert, which would benefit being more clearly identified in the content of the clinical efficacy discussion and individually discussed. Looking into the responses, however, there might have been different interpretations of the questions between the two reviews. Importantly, aspects such as the request for additional analysis, Good Clinical Practice (GCP) inspections, and external expert consultations are discussed in the EPAR only if they occurred and if they are recorded. EUnetHTA suggestion to the EMA was to increase the granularity in the structure of the clinical efficacy discussion report template to address the main above-mentioned aspects and make the information more visible.
      It was also noted that there are different expectations with regard to the referencing of literature because the regulatory assessment is usually based on study reports rather than published literature. In general, regulatory-specific terminology might have been differently understood in some cases during the review by EUnetHTA; for example, the term scientific guidelines in the regulatory context usually refers to the guidance on the clinical development program to support an application for marketing authorization rather than to treatment recommendations. An overall perception during the review by HTA bodies was that the information was provided in the EPAR but on some occasions was either difficult to locate or not sufficiently discussed.
      A limitation of the compliance review was that it was performed with EPARs that were written immediately after the revised templates came into operation. This meant that the primary assessment reports from assessors, which lead up to the final CHMP assessment report supporting the outcome of the regulatory review and hence the EPAR, have not been prepared following these new standards. Furthermore, the review was limited to 10 EPARs and was performed by the two decision makers without involving a “third” party. Nevertheless, the review highlighted important aspects that should be subject to the quality review process of assessment reports in the future. If subsequently areas are identified that can be improved further in the templates and the guidance documents, these can be implemented through the regular revision processes.

      Conclusions and Perspectives

      With this collaboration the EMA and EUnetHTA responded to a political recommendation to consider how the regulatory assessment report about favorable and unfavorable effects of a medicine can best be used in the assessment of the relative effectiveness of new medicines for HTA purposes in the EU member states. There is growing international experience regarding aspects to be considered to optimize the interface between regulatory approval and reimbursement decision [
      • Tsoi B.
      • Masucci L.
      • Campbell K.
      Harmonization of reimbursement and regulatory approval processes: a systematic review of international experiences.
      ]. The present work was the first joint project of such collaboration between regulators and HTA bodies on a European level. The remit of both parties was respected throughout the project and the focus was on the presentation of data and information without entering into a discussion about potentially divergent views on evidentiary standards. With the improved presentation of data and information in the EPAR, it is envisaged that this regulatory document through harmonized efficacy data presentation will be more useful in the context of rapid REAs by HTA bodies when they inform policymakers and health care decision makers in the future.
      The initiation of the dialogue between the EMA and EUnetHTA on the EPAR has demonstrated the opportunity to engage in discussions about better exchange of data and information. In fact, the parallel review of EPARs has been useful for each of the organizations to not only critically review the end-product “assessment report” using a predefined methodology but also mutually identify areas for future improvement.
      The project has shown the pursuit of one opportunity for regulators to proactively contribute information to the assessment of relative (comparative) effectiveness by HTA bodies, independent of jurisdiction. Beyond this project on EPARs, the EMA and EUnetHTA are continuing to explore other areas of collaboration or exchange of information. These include ways to obtain scientific advice or early dialogues involving regulators and HTAs/payers, exchange on scientific and methodological guidelines, topics in the area of postlicensing data generation, as well as the particularities of orphan medicinal products [

      Minutes from the EMA-EUnetHTA meeting on 14 May 2013, London. Available from: www.ema.europa.eu and www.eunethta.eu. [Accessed October 30, 2013].

      ]. The EMA’s Road Map to 2015 identified the improvement of EPARs in view of their use for HTAs as well as the EMA’s engagement with HTA organizations—regarding drug development and evidential standards as well as postlicensing research and data collection—as major initiatives [

      European Medicines Agency. Road map to 2015—The European Medicines Agency’s contribution to science, medicines and health. Available from: www.ema.europa.eu. [Accessed October 30, 2013].

      ]. Similarly, EUnetHTA has defined collaboration with EMA as a priority [

      EUnetHTA. EUnetHTA Joint Action 2 three-year work plan. Available from: http://www.eunethta.eu/outputs/eunethta-joint-action-2-3-year-work-plan-2013. [Accessed October 30, 2013].

      ]. A joint EMA-EUnetHTA three-year work plan 2013-2015 has recently been agreed [

      EMA-EUnetHTA three-year work plan 2013-2015. Available from: www.ema.europa.eu and www.eunethta.eu. [Accessed March 6, 2014].

      ]. Overall, it is expected that this dialogue between regulators and HTA bodies on a European level will continue to facilitate the bridging between regulatory approval and access to market and will provide important contributions to various health policy topics.

      Acknowledgments

      We thank the following organizations for having performed the compliance review on behalf of EUnetHTA: Andalusian HTA Agency (AETSA), Italian Medicines Agency (AIFA), Catalan Agency for Health Information, Assessment and Quality (CAHIAQ), Center for Medical Technology Policy (CMPT) in Baltimore, Dutch Health Care Insurance Board (CVZ), National Institute for Health and Care Excellence (NICE), Norwegian Knowledge Center for the Health Services (NOKC), French National Authority for Health (HAS), Association of Austrian Social Insurance Institutions (HVB), and University Hospital A. Gemelli, Italy.
      Source of financial support: The EUnetHTA contribution to this article arises from the EUnetHTA Joint Action 1, which has received funding from the European Union, in the framework of the Health Programme.

      References

        • Kristensen F.B.
        • Mäkelä M.
        • Allgurin Neikter S.
        • et al.
        European network for Health Technology Assessment, EUnetHTA: planning, development, and implementation of a sustainable European network for Health Technology Assessment.
        Int J Technol Assess Health Care. 2009; 25: 107-116
      1. Kleijnen S, Goettsch W, d’Andon A, et al. Relative effectiveness assessment (REA) of pharmaceuticals—background review, July 2011. Available from: www.eunethta.eu. [Accessed March 6, 2014].

      2. EUnetHTA. Organisation. Available from: http://www.eunethta.eu/about-us/organisation. [Accessed March 6, 2014].

        • Kristensen F.B.
        Development of European HTA: from Vision to EUnetHTA.
        Michael Q. 2012; 9: 147-156
      3. HLPF conclusions and recommendations. Available from: http://ec.europa.eu/enterprise/sectors/healthcare/files/docs/pharmaforum_final_conclusions_en.pdf. [Accessed October 30, 2013].

      4. EUnetHTA. EUnetHTA rapid REA core model. Available from: www.eunethta.eu. [Accessed October 30, 2013].

      5. European Medicines Agency. Marketing authorisations for new active substances reviewed in the centralised procedure and the decentralised/mutual recognition procedure, respectively; data on file.

        • Tsoi B.
        • Masucci L.
        • Campbell K.
        Harmonization of reimbursement and regulatory approval processes: a systematic review of international experiences.
        Expert Rev Pharmacoecon Outcomes Res. 2013; 13: 497-511
      6. Minutes from the EMA-EUnetHTA meeting on 14 May 2013, London. Available from: www.ema.europa.eu and www.eunethta.eu. [Accessed October 30, 2013].

      7. European Medicines Agency. Road map to 2015—The European Medicines Agency’s contribution to science, medicines and health. Available from: www.ema.europa.eu. [Accessed October 30, 2013].

      8. EUnetHTA. EUnetHTA Joint Action 2 three-year work plan. Available from: http://www.eunethta.eu/outputs/eunethta-joint-action-2-3-year-work-plan-2013. [Accessed October 30, 2013].

      9. EMA-EUnetHTA three-year work plan 2013-2015. Available from: www.ema.europa.eu and www.eunethta.eu. [Accessed March 6, 2014].