Improving the Contribution of Regulatory Assessment Reports to Health Technology Assessments—A Collaboration between the European Medicines Agency and the European network for Health Technology Assessment

Scientific evaluation of the clinical data produced during drug development can be intended to estimate the benefit/risk ratio of the product (recently in some jurisdictions also referred to as benefit/harm/uncertainty), for the purpose of marketing authorization, or to estimate the effectiveness of the new product as compared with existing therapies, as part of the health technology assessment (HTA) process to support decision making on price and reimbursement.

The European Medicines Agency (EMA) is responsible for the scientific evaluation of applications for European Union (EU) marketing authorizations for medicinal products in the so-called centralized procedure. Under this procedure, a single marketing-authorization application is submitted to the EMA and once authorization is granted by the European Commission on the basis of a pan-European scientific assessment of quality, efficacy, and safety by EMA committees, a centralized marketing authorization is valid in all EU member states.

The subsequent evaluation and decision-making process leading to decisions on the pricing and reimbursement of medicinal products lie with the national bodies. HTA is a multidisciplinary process that summarizes information about the medical, social, economic, and ethical issues related to the use of a health technology in a systematic, transparent, unbiased, and robust manner. Its aim is to inform the formulation of safe, effective health policies that are patient focused and seek to achieve best value []. Criteria for HTA vary between countries, but generally HTA bodies in Europe use relative effectiveness assessment (REA), a European equivalent to comparative effectiveness research in the United States, as part of the HTA process [].

The European network for Health Technology Assessment (EUnetHTA) is a network of organizations appointed by EU member states that produce or contribute to HTA to facilitate efficient use of resources available for HTA, to create a sustainable system of HTA knowledge sharing, and to promote good practice in HTA methods and processes [, ]. Originally started as a project in 2006, EUnetHTA was established to create an effective and sustainable network for HTA across Europe [].

In October 2008, the Pharmaceutical Forum, a high-level ministerial platform for discussion between member states, EU institutions, industry, health care professionals, patients, and insurance funds, agreed on conclusions and recommendations to find relevant solutions to public health considerations regarding pharmaceuticals, while ensuring the competitiveness of the industry and the sustainability of the national health care systems []. One of these recommendations provided a political mandate to initiate collaboration between the EMA and EUnetHTA with the aim to improve the availability and best use of data relevant to HTA. Specifically, the objective of this joint project of regulators and HTA bodies was to “consider how information in the European Public Assessment Report /…/ can further contribute to relative effectiveness assessment” [].

The regulatory assessment reports, which are the basis for central marketing authorizations, are publicly available as so-called European Public Assessment Reports (EPARs). EPARs reflect the scientific conclusions reached by the EMA’s Committee for Medicinal Products for Human Use (CHMP) at the end of the regulatory review process after deletion of commercially confidential information. These are published on the EMA’s Web site for every medicine authorized through the centralized procedure in the EU. Because these contain the CHMP’s judgment on the methodological quality of the studies and the relevance and significance of results, in the perspective of the estimate of the balance between favorable and unfavorable effects for the medicinal product, they may also be used as source documents for REAs from the HTA perspective. In particular, such assessment reports are used in the context of rapid REA of pharmaceuticals [].

The collaboration between the EMA and EUnetHTA on EPARs started in February 2010 and was performed over 2 years. It has been subject to a wider consultation within EUnetHTA as part of the Work Package 5 on REA of Pharmaceuticals of EUnetHTA Joint Action 1. This report provides an account of various activities as part of the project to address the recommendation on improving the EPAR.

In the first phase, EUnetHTA provided input on the usefulness of published EPARs and Summary of Product Characteristics (SmPC) in the context of REA of medicinal products based on the initial work performed in 2009 by some partners (which are also EUnetHTA partners) of the Medicine Evaluation Committee (MEDEV), a standing working group of the European Social Health Insurance Forum. This input was reviewed by the EMA and an action plan was established, which was mutually agreed between the EMA and EUnetHTA. Subsequently, the templates and guidance documents for the assessment of initial marketing authorization applications were amended by the EMA to implement the agreed improvements through either specific sections or supportive guidance. The revised templates and guidance documents were presented to the CHMP for adoption and subsequently published on the EMA’s Web site (www.ema.europa.eu) for use by assessors in the centralized procedure (>Home >Regulatory >Human medicines >Pre-authorisation >Templates for assessors).

The second phase was the critical review of EPARs to establish compliance with revised templates. For this purpose, EPARs of the first 10 products, for which the assessment report has been prepared after the revised templates came into operation, were analyzed for compliance with specific aspects amended in the template. The following EPARs were reviewed: Esbriet (pirfenidone), Gilenya (fingolimod), Halaven (eribulin), Jevtana (cabazitaxel), Pravafenix (fenofibrate/pravastatin), Pumarix (pandemic influenza vaccine [H5N1]), Teysuno (tegafur/gimeracil/oteracil), Trobalt (retigabine), Xeplion (paliperidone), and Xiapex (collagenase clostridium histolyticum). The review took place in second and third quarters of 2011 on the basis of the publicly available EPARs. A questionnaire with 36 questions was developed by the EMA targeting the identified areas for improvement. Two types of answers were attributed depending on the nature of the question: either the option “yes/no/not applicable” (13 questions) if the underlying question was simply whether an item is included or the grading “excellent/good/could be improved/no/not applicable” (23 questions) if a more differentiated review was considered more meaningful. Table 1 indicates for each review item the type of question. Numerical values were assigned against the answers (“1/0” and “3/2/1/0” for binary and graded answers, respectively) to allow for statistical analysis of compliance of the EPARs for each individual question. The review of the 10 EPARs was performed in parallel by the EMA and by EUnetHTA, always involving two reviewers per organization and using the same questionnaire and numerical scales. In case of divergent views between the two reviewers, consensus was formed. One additional question on the benefit-risk assessment was added by HTA organizations at the time of EUnetHTA review. As regards EUnetHTA review, each EPAR was evaluated by two HTA organizations from EUnetHTA. Ten HTA organizations (see Acknowledgment) participated in this exercise coordinated by the French National Authority for Health (HAS). The results were calculated as mean values in terms of the overall compliance rate across all 10 EPARs and reviewers for each of the two organizations (i.e., if the answer for a binary question was “yes” [“1”] in 6 out of 10 EPARs, a 60% overall compliance rate was attributed). For descriptive purpose, the compliance rates are clustered in three distinct categories: more than 80% compliance (mean), 50% to 80% compliance (mean), and less than 50% compliance (mean).

As a result of the project, the templates for preparing EPARs were revised to address the comments received. Furthermore, a review of a small set of EPARs was carried out in parallel by both EUnetHTA and the EMA to ascertain whether these template changes led to the inclusion of relevant information. The following sections present these results separately.

This joint project between the EMA and EUnetHTA aimed at improving the contribution EPARs can make to the assessment of relative effectiveness by European HTA bodies. EPARs are published regulatory assessment reports supporting the licensing of medicinal products that are centrally authorized in Europe. Given the scope of the centralized procedure, almost all new therapeutics (new molecular entities and biologics) receive their marketing authorization through this licensing route [].

Based on observations on the usefulness of EPARs and SmPCs from an HTA perspective, the EMA and EUnetHTA agreed on specific items for improvement of the presentation of data and information in the EPAR. The emphasis was on a better understanding of the regulatory judgment, such as critical discussions of study design as well as specific subgroup analysis. Furthermore, a structured and harmonized presentation of main efficacy data was requested in view of clarity and transparency of the data that is deemed most important from a regulatory perspective. Through changes in the templates and guidance documents for assessment reports, these agreed topics were specifically targeted. The revisions—if adhered to when writing the actual reports—were aimed to ensure that the relevant data and information are provided with the published documents.

The review of actual EPARs explored how well these revisions are translated into practice. Using the same methodology, both the EMA and EUnetHTA came to the view that the compliance differs across various aspects in the assessment report. Given that the EPAR format and scientific content showed overall high rates of compliance with the revised templates, it can be assumed that the quality of the documents has been improved even if no such assessment of previous EPARs with the same methodology was performed to quantify the difference. An achievement was the introduction of the tabular overview of main efficacy data, which was widely adhered to. Also, more structural aspects in the EPAR were implemented. There remains, however, space for further improvement in the critical discussion of the key elements of the clinical study design, that is, on patient population (including subpopulation and special populations), comparators, duration of the study, end points, and/or composite end-point use. Some of these aspects may be present in the clinical efficacy discussion but are not visible enough or not discussed enough. The substantiation of the SmPC was variable, with some areas requiring special attention mainly for the sections concerning the dose recommendations (particularly deviations from standard dose), contraindications, warnings and precautions, and interactions.

The most divergent view between the reviews of the two organizations seemed to be regarding the discussion of the shortcomings of efficacy data as well as the identification and explanation of additional analysis requested during the regulatory review. These items are considered as critical areas of the clinical efficacy discussion from EUnetHTA perspective, as well as explanations when a subgroup analysis is considered of particular relevance, justifications for waiver or replacement of a study by literature data, and reflection of additional input from an external expert, which would benefit being more clearly identified in the content of the clinical efficacy discussion and individually discussed. Looking into the responses, however, there might have been different interpretations of the questions between the two reviews. Importantly, aspects such as the request for additional analysis, Good Clinical Practice (GCP) inspections, and external expert consultations are discussed in the EPAR only if they occurred and if they are recorded. EUnetHTA suggestion to the EMA was to increase the granularity in the structure of the clinical efficacy discussion report template to address the main above-mentioned aspects and make the information more visible.

It was also noted that there are different expectations with regard to the referencing of literature because the regulatory assessment is usually based on study reports rather than published literature. In general, regulatory-specific terminology might have been differently understood in some cases during the review by EUnetHTA; for example, the term scientific guidelines in the regulatory context usually refers to the guidance on the clinical development program to support an application for marketing authorization rather than to treatment recommendations. An overall perception during the review by HTA bodies was that the information was provided in the EPAR but on some occasions was either difficult to locate or not sufficiently discussed.

A limitation of the compliance review was that it was performed with EPARs that were written immediately after the revised templates came into operation. This meant that the primary assessment reports from assessors, which lead up to the final CHMP assessment report supporting the outcome of the regulatory review and hence the EPAR, have not been prepared following these new standards. Furthermore, the review was limited to 10 EPARs and was performed by the two decision makers without involving a “third” party. Nevertheless, the review highlighted important aspects that should be subject to the quality review process of assessment reports in the future. If subsequently areas are identified that can be improved further in the templates and the guidance documents, these can be implemented through the regular revision processes.

With this collaboration the EMA and EUnetHTA responded to a political recommendation to consider how the regulatory assessment report about favorable and unfavorable effects of a medicine can best be used in the assessment of the relative effectiveness of new medicines for HTA purposes in the EU member states. There is growing international experience regarding aspects to be considered to optimize the interface between regulatory approval and reimbursement decision []. The present work was the first joint project of such collaboration between regulators and HTA bodies on a European level. The remit of both parties was respected throughout the project and the focus was on the presentation of data and information without entering into a discussion about potentially divergent views on evidentiary standards. With the improved presentation of data and information in the EPAR, it is envisaged that this regulatory document through harmonized efficacy data presentation will be more useful in the context of rapid REAs by HTA bodies when they inform policymakers and health care decision makers in the future.

The initiation of the dialogue between the EMA and EUnetHTA on the EPAR has demonstrated the opportunity to engage in discussions about better exchange of data and information. In fact, the parallel review of EPARs has been useful for each of the organizations to not only critically review the end-product “assessment report” using a predefined methodology but also mutually identify areas for future improvement.

The project has shown the pursuit of one opportunity for regulators to proactively contribute information to the assessment of relative (comparative) effectiveness by HTA bodies, independent of jurisdiction. Beyond this project on EPARs, the EMA and EUnetHTA are continuing to explore other areas of collaboration or exchange of information. These include ways to obtain scientific advice or early dialogues involving regulators and HTAs/payers, exchange on scientific and methodological guidelines, topics in the area of postlicensing data generation, as well as the particularities of orphan medicinal products []. The EMA’s Road Map to 2015 identified the improvement of EPARs in view of their use for HTAs as well as the EMA’s engagement with HTA organizations—regarding drug development and evidential standards as well as postlicensing research and data collection—as major initiatives []. Similarly, EUnetHTA has defined collaboration with EMA as a priority []. A joint EMA-EUnetHTA three-year work plan 2013-2015 has recently been agreed []. Overall, it is expected that this dialogue between regulators and HTA bodies on a European level will continue to facilitate the bridging between regulatory approval and access to market and will provide important contributions to various health policy topics.

We thank the following organizations for having performed the compliance review on behalf of EUnetHTA: Andalusian HTA Agency (AETSA), Italian Medicines Agency (AIFA), Catalan Agency for Health Information, Assessment and Quality (CAHIAQ), Center for Medical Technology Policy (CMPT) in Baltimore, Dutch Health Care Insurance Board (CVZ), National Institute for Health and Care Excellence (NICE), Norwegian Knowledge Center for the Health Services (NOKC), French National Authority for Health (HAS), Association of Austrian Social Insurance Institutions (HVB), and University Hospital A. Gemelli, Italy.

Source of financial support: The EUnetHTA contribution to this article arises from the EUnetHTA Joint Action 1, which has received funding from the European Union, in the framework of the Health Programme.