Advertisement

Policies for Use of Real-World Data in Health Technology Assessment (HTA): A Comparative Study of Six HTA Agencies

Open ArchivePublished:January 27, 2017DOI:https://doi.org/10.1016/j.jval.2016.12.003

      Abstract

      Background

      Randomized controlled trials provide robust data on the efficacy of interventions rather than on effectiveness. Health technology assessment (HTA) agencies worldwide are thus exploring whether real-world data (RWD) may provide alternative sources of data on effectiveness of interventions. Presently, an overview of HTA agencies’ policies for RWD use in relative effectiveness assessments (REA) is lacking.

      Objectives

      To review policies of six European HTA agencies on RWD use in REA of drugs. A literature review and stakeholder interviews were conducted to collect information on RWD policies for six agencies: the Dental and Pharmaceutical Benefits Agency (Sweden), the National Institute for Health and Care Excellence (United Kingdom), the Institute for Quality and Efficiency in Healthcare (Germany), the High Authority for Health (France), the Italian Medicines Agency (Italy), and the National Healthcare Institute (The Netherlands). The following contexts for RWD use in REA of drugs were reviewed: initial reimbursement discussions, pharmacoeconomic analyses, and conditional reimbursement schemes. We identified 13 policy documents and 9 academic publications, and conducted 6 interviews.

      Results

      Policies for RWD use in REA of drugs notably differed across contexts. Moreover, policies differed between HTA agencies. Such variations might discourage the use of RWD for HTA.

      Conclusions

      To facilitate the use of RWD for HTA across Europe, more alignment of policies seems necessary. Recent articles and project proposals of the European network of HTA may provide a starting point to achieve this.

      Keywords

      Introduction

      In light of rising health care costs and the introduction of innovative, yet expensive, pharmaceutical products, health technology assessment (HTA) agencies are seeking robust methods for relative effectiveness assessments (REAs) of drugs in routine clinical practice. The relative effectiveness of an intervention is defined as “[t]he extent to which an intervention does more good than harm, when compared to one or more intervention alternatives for achieving the desired results and when provided under the routine setting of health care practice (i.e. real-world setting)” [

      European Commission. Directive 2001/20/ec of the European Parliament and of the Council. 2015. Available from: http://ec.europa.eu/health/files/eudralex/vol-1/dir_2001_20/dir_2001_20_en.pdf. [Accessed April 24, 2015]

      ].
      Conventionally, data on treatment effects for drugs are collected in the context of randomized controlled trials (RCTs), whereby a selected, homogeneous group of patients is randomly assigned to either the experimental drug or a comparator (e.g., placebo or active comparator) under highly controlled conditions. This study design is ideal to demonstrate the efficacy of a drug, because of its ability to minimize problems with confounding, information bias, and selection bias. Nevertheless, once a drug gains marketing authorization, it is administered to a heterogeneous patient group in routine clinical practice whereby patients present with differing comorbidities, comedications, and genetic profiles. Consequently, it is challenging to extrapolate results from RCTs to drug effects in clinical practice [
      • Eichler H.G.
      • Abadie E.
      • Breckenridge A.
      • et al.
      Bridging the efficacy-effectiveness gap: a regulator’s perspective on addressing variability of drug response.
      ].
      Because of limitations associated with the use of RCT-generated efficacy data to predict the relative effectiveness of drugs, HTA agencies worldwide are currently exploring the possibilities for using real-world data (RWD) to supplement and enrich the evidence for REA of drugs. Examples of national and international collaborations exploring these possibilities include the Patient-Centered Outcomes Research Initiative and the Innovative Medicines Initiative GetReal Consortium (IMI-GetReal). The IMI-GetReal is a 3-year project aiming at investigating policies and methodologies for the collection and use of RWD in drug development and assessment. It combines a broad array of stakeholders across Europe to collaborate on developing a policy framework for RWD use and good practices for its integration in the evidence base.
      In addition, HTA agencies are exploring the use of evidence development strategies that provide effectiveness research data earlier during drug development in the framework of medicine adaptive pathways to patients [
      • Eichler H.G.
      • Oye K.
      • Baird L.G.
      • et al.
      Adaptive licensing: taking the next step in the evolution of drug approval.
      ]. One example, the IMI-ADAPT SMART project, is a 3-year project enabling a platform for multiple-stakeholder discussions on questions relating to the implementation of medicine adaptive pathways to patient activities in the European setting. Moreover, numerous publications have highlighted the growing need for RWD use in HTA decision making to inform clinical effectiveness parameters, natural history of disease, adherence to treatment and health-related quality of life, or information on demand and supply constraints for health economic evaluations in specific settings [
      • Romio S.
      • Sturkenboom M.
      • Corrao G.
      Real-world data from the health decision maker perspective. What are we talking about?.
      ,
      • Garrison L.P.
      • Neumann P.J.
      • Erickson P.
      • et al.
      Using real world data for coverage and payment decisions: the ISPOR Real World Data Task Force Report.
      ,
      • Fung V.
      • Brand R.J.
      • Newhouse J.P.
      • Hsu J.
      Using Medicare data for comparative effectiveness research opportunities and challenges.
      ,

      Annemans L, Aristides M, Kubin M. Real-life data: a growing need. 2015. Available from: http://www.ispor.org/news/articles/oct07/rld.asp. [Accessed April 24, 2015].

      ,

      Makady A. Review of policies and perspectives on real-world data. 2015. Available from: http://tinyurl.com/p2qq9xh. [Accessed April 24, 2015].

      ,
      • Vassall A.
      • Mangham-Jefferies L.
      • Gomez G.B.
      • et al.
      Incorporating demand and supply constraints into economic evaluations in low-income and middle-income countries.
      ].
      Research conducted by the IMI-GetReal identified three contexts within which RWD is currently being used for REA of drugs: as supplementary input for initial REA after market authorization, as input for pharmacoeconomic analyses (PEA), and for the re-assessment of relative effectiveness in conditional reimbursement schemes (CRSs) [

      Makady A. Review of policies and perspectives on real-world data. 2015. Available from: http://tinyurl.com/p2qq9xh. [Accessed April 24, 2015].

      ]. Nevertheless, an overview of the similarities and differences between different HTA agencies’ policies for the use of RWD in the three aforementioned contexts seems to be lacking. Given the recent efforts and growing interest for the harmonization of HTA activities across Europe (e.g., as demonstrated by activities of the European network of HTA [EUnetHTA]), an initial comparison of policies for RWD use by HTA agencies across a number of European jurisdictions may provide a good starting point for further discussions on the harmonization of policies on this topic.
      Therefore, this article aims to review the policies of six HTA agencies in Europe on RWD use in REA of drugs. More specifically, the article considers agencies’ policies regarding RWD accepted or requested as well as policies for the appraisal of RWD in the following three contexts: initial reimbursement discussions (IRDs), PEA, and CRS. It is important to note that this article does not aim to provide a comprehensive overview of RWD policies of HTA agencies in all 29 European jurisdictions but rather aims to present a comparison across several relevant jurisdictions in Europe.

      Methods

      Six European HTA agencies were selected for analysis: the Dental and Pharmaceutical Benefits Agency (Tandvårds- och läkemedelsförmånsverket [TLV], Sweden), the National Institute for Health and Care Excellence (NICE, the United Kingdom), the Institute for Quality and Efficiency in Healthcare (Institut fuer Qualitaet und Wirtschaftlichkeit im Gesundheitswesen [IQWiG], Germany), the High Authority for Health (Haute Autorité de Santé [HAS], France), the Italian Medicines Agency (Agenzia Italiana del farmaco [AIFA], Italy), and the National Healthcare Institute (Zorginstituut Nederland [ZIN], The Netherlands). HTA agencies in France, Germany, Italy, and the United Kingdom were selected because they represent the four largest European jurisdictions (the so-called Big Four)—jurisdictions bearing most influence on European policies on several aspects, including health [

      Organisation for Economic Co-operation and Development. OECD composite leading indicators: reference turning points and component series. 2016. Available from: http://www.oecd.org/std/leading-indicators/oecdcompositeleadingindicatorsreferenceturningpointsandcomponentseries.htm. [Accessed November 2, 2016].

      ,
      • Debaere P.
      EU Coordination in International Institutions: Policy and Process in Gx Forums.
      ,
      • Jorgensen K.E.
      • Laatikainen K.V.
      Routledge Handbook on the European Union and International Institutions: Performance, Policy, Power.
      ]. Meanwhile, HTA agencies in Sweden and the Netherlands were selected because of their pioneering roles, both historically and currently, in cutting-edge European HTA projects, such as the EUnetHTA [

      European network for Health Technology Assessment. EUnetHTA JA3 (2016–2020) lead partners. 2016. Available from: http://eunethta.eu/activities/joint-action-3/jointaction31/eunethta-joint-action-3-2016-2020#involved_organisations. [Accessed November 2, 2016].

      ]. To ensure that all relevant information on agencies’ policies on RWD use in REA of drugs was collected, three methods were used to retrieve information: a review of agencies’ guidelines and policy papers, a review of academic publications by HTA affiliates on RWD use in REA of drugs, and semistructured interviews with representatives from the selected agencies.
      First, the Web sites of the six HTA agencies were searched for guidelines and policy papers in the three contexts: IRD, PEA, and CRS. Documents were included if they were published in English, German, French, or Dutch. Second, a search for academic articles published by agency affiliates relating to RWD use in REA of drugs was conducted in MEDLINE using the PubMed interface (for the search strategy, see the Appendix in Supplemental Materials found at 10.1016/j.jval.2016.12.003). To minimize chances of missing relevant literature, a time span of 10 years was selected. Articles were included if they 1) were published between January 1, 2006, and June 21, 2016 (date of search); 2) explicitly discussed the use of RWD in REA of drugs; 3) were published in English, German, French, or Dutch (Swedish and Italian documents were excluded because the study authors do not master these two languages); and 4) comprised more than an abstract. Articles were excluded if they did not meet all inclusion criteria. Documents retrieved from agency Web sites and PubMed searches were evaluated independently by two authors. Any disagreements regarding inclusion or exclusion of articles were resolved by consensus.
      Third, semistructured interviews were conducted with representatives from the six HTA agencies. Representatives were selectively sampled on the basis of seniority and function, with a preference for senior HTA assessors and research and development senior officers. Information for identifying representatives was retrieved from agency Web sites and/or the authors’ professional network. All representatives were approached by email using a standardized invitation. A standardized questionnaire was sent to all representatives who agreed to participate 2 weeks before the interview to guide discussions (see Appendix Figure i in Supplemental Materials found at 10.1016/j.jval.2016.12.003). To increase the validity of stakeholder views, participants were provided the freedom to invite colleagues they deemed relevant to take part in the interviews. Interviews were conducted, recorded, and subsequently transcribed for further analysis. The sampling of representatives and interview protocols were compared with the consolidated criteria for reporting qualitative studies to ensure good quality [
      • Tong A.
      • Sainsbury P.
      • Craig J.
      Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups.
      ].
      It is important to note that the interviews were conducted as part of a broader review of stakeholder policies and perspectives on RWD [

      Makady A. Review of policies and perspectives on real-world data. 2015. Available from: http://tinyurl.com/p2qq9xh. [Accessed April 24, 2015].

      ]. Therefore, the scope of questions posed in the interviews extended beyond the aims of this research.
      A standardized coding scheme was developed using MaxQDA 11.0 software (Berlin, Germany) to extract data from all compiled documents and transcripts on two aspects: 1) RWD accepted or requested and 2) the appraisal of RWD for REA of drugs within IRD, PEA, and CRS (see Fig. 1). The scheme was developed by iterative assessment of included documents and interview transcripts, in accordance with the directed content analysis approach for qualitative research [
      • Hsieh H.F.
      • Shannon S.E.
      Three approaches to qualitative content analysis.
      ]. Two authors independently performed data abstraction and coding. Any discrepancies were resolved by consensus.
      Fig. 1
      Fig. 1Coding scheme developed to conduct coding analysis. CRS, conditional reimbursement scheme; IRD, initial reimbursement discussion; PEA, pharmacoeconomic analysis; RWD, real-world data.
      The results from the coding analysis of the compiled documents and transcripts reported in this article were subsequently verified with the interviewed representatives of all six agencies to ensure factual correctness.
      For the purpose of this article, we based our definition for RWD on the IMI-GetReal definition:An umbrella term for data regarding the effects of health interventions (e.g. safety, effectiveness, resource use, etc) that are not collected in the context of highly controlled RCTs. Instead, RWD can either be primary research data collected in a manner which reflects how interventions would be used in routine clinical practice or secondary research data derived from routinely collected data. Data collected include, but are not limited to, clinical and economic outcomes, patient-reported outcomes and health-related quality of life. RWD can be obtained from many sources including patient registries, electronic medical records, and claims databases. [

      Makady A. GetReal. Updated glossary of definitions of common terms. 2015. Available from: https://www.imi-getreal.eu/Portals/1/Documents/Publications/D1.3%20GetReal%20Glossary%20-%20Update.pdf. [Accessed June 21, 2016]. [Accessed May 30, 2016].

      ].

      Results

      The search for guidelines and policy papers on RWD use on agency Web sites yielded 13 documents (see Table 1). All six agencies had guidance and policy papers available for IRD, five agencies for PEA, and three agencies for CRS. The number and nature of documents varied per institute. Some agencies (e.g., TLV, HAS, and ZIN) had separate guidelines for IRD and PEA, whereas others (e.g., NICE) combined both in one document.
      Table 1List of policy documents, guidelines, and academic publications retrieved as well as the number of interview participants and transcript reference per agency
      HTA agencyPolicy papers and guidelinesAcademic publicationsNumber of interview participants and transcript reference
      TLVGuide for companies when applying for subsidies and pricing for pharmaceutical products
      • Degli Esposti L.
      • Saragoni S.
      • Sangiorgi D.
      • et al.
      The nationwide Osmed Health-Db database: a tool to support healthcare decision-making and real-world evidence generation.
      1 participant Transcript reference: a
      General guidelines for economic evaluations from the Pharmaceutical Benefits Board (LFNAR 2003:2)
      Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. Allgemeine Methoden. Version 4.2. 2015.
      The Swedish Pharmaceutical Reimbursement System
      Tandvårds- och läkemedelsförmånsverket. General Guidelines for Economic Evaluations from the Pharmaceutical Benefits Board.
      NICEGuide to the methods of technology appraisal 2013
      • Taylor J.
      • Patrick H.
      • Lyratzopoulos G.
      • Campbell B.
      Methodological challenges in evaluating the value of registers.
      Evidence requirements for reimbursements of pharmaceuticals across Europe
      • Jorgensen K.E.
      • Laatikainen K.V.
      Routledge Handbook on the European Union and International Institutions: Performance, Policy, Power.
      3 participants Transcript reference: b
      NICE DSU technical support document 17: The use of observational data to inform the estimates of treatment effectiveness in technology appraisal: Methods for comparative individual patient data
      Tandvårds- och läkemedelsförmånsverket
      Guide for Companies When Applying for Subsidies and Pricing for Pharmaceutical Products.
      Methodological challenges in evaluating the value of registries
      • Tomino C.
      New perspective and new challenges in clinical trial regulation in Italy.
      Evidence informed decision making: The use of “colloquial evidence” at NICE
      • Oyebode O.
      • Garrett Z.
      • George E.
      • et al.
      Evidence requirements for reimbursement of pharmaceuticals across Europe.
      How RWD compensate for scarce evidence in HTA
      • Xoxi E.
      • Tomino T.
      • de Nigro L.
      • Pani L.
      The Italian post-marketing registries.
      How to improve the quality of evidence when new treatments are funded conditional on collecting evidence of effectiveness and safety
      Italian Medicines Agency Research and Development Working Group
      Feasibility and challenges of independent research on drugs: the Italian Medicines Agency (AIFA) experience.
      IQWiGAllgemeine Methoden version 4.2
      • George E.
      How real-world data compensate for scarce evidence in HTA.
      1 participant Transcript reference: c
      HASGeneral method for assessing health technologies
      • Glynn D.
      • Campbell B.
      • Marlow M.
      • Patrick H.
      How to improve the quality of evidence when new treatments are funded conditional on collecting evidence of effectiveness and safety.
      2 participants Transcript reference: d
      Choices in methods for economic evaluation

      Haute Autorité de Santé. General method for assessing health technologies. 2015. Available from: http://www.has-sante.fr/portail/upload/docs/application/pdf/general_method_eval_techno.pdf. [Accessed June 21, 2016].

      Les etudes post-inscription sur les technologies de santé (médicaments, dispositifs médicaux et actes)
      National Institute for Health and Care Excellence
      The Use of Observational Data to Inform Estimates of Treatment Effectiveness in Technology Appraisal: Methods for Comparative Individual Patient Data.
      AIFAEvidence requirements for reimbursements of pharmaceuticals across Europe
      • Jorgensen K.E.
      • Laatikainen K.V.
      Routledge Handbook on the European Union and International Institutions: Performance, Policy, Power.
      2 participants Transcript reference: e
      New perspective and new challenges in clinical trial regulation in Italy

      European network for Health Technology Assessment. EUnetHTA JA3 (2016–2020) lead partners. 2016. Available from: http://eunethta.eu/activities/joint-action-3/jointaction31/eunethta-joint-action-3-2016-2020#involved_organisations. [Accessed November 2, 2016].

      Feasibility and challenges of independent research on drugs: The Italian Medicines Agency (AIFA) experience
      • Hsieh H.F.
      • Shannon S.E.
      Three approaches to qualitative content analysis.
      The Italian postmarketing registries
      • Tong A.
      • Sainsbury P.
      • Craig J.
      Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups.
      The nationwide Osmed Health-Db database: A tool to support health-care decision-making and real-world evidence generation

      Makady A. GetReal. Updated glossary of definitions of common terms. 2015. Available from: https://www.imi-getreal.eu/Portals/1/Documents/Publications/D1.3%20GetReal%20Glossary%20-%20Update.pdf. [Accessed June 21, 2016]. [Accessed May 30, 2016].

      ZINBeoordeling stand van de wetenschap en praktijk
      • Lapeyre-Mestre M.
      • Sapède C.
      • Moore N.
      • et al.
      Études Pharmaco-Épidémiologiques: Quels Niveaux De Preuve Et Comment Les Atteindre?.
      Evidence requirements for reimbursements of pharmaceuticals across Europe
      • Jorgensen K.E.
      • Laatikainen K.V.
      Routledge Handbook on the European Union and International Institutions: Performance, Policy, Power.
      2 participants Transcript reference: f
      Richtlijn voor het uitvoeren van economische evaluaties in de gezondheidszorg
      Zorginstituut Nederland
      Beoordeling stand van de wetenschap en praktijk.
      Leideraad voor Uitkomstenonderzoek
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal 2013.
      Procedure voorwaardelijke toelating geneeskundige zorg 2015
      College voor Zorgverzekeringen. Leideraad voor Uitkomstenonderzoek.
      AIFA, Italian Medicines Agency; HAS, High Authority for Health; HTA, health technology assessment; IQWiG, Institute for Quality and Efficiency in Healthcare; NICE, National Institute for Health and Care Excellence; RWD, real-world data; TLV, Dental and Pharmaceutical Benefits Agency; ZIN, National Healthcare Institute.
      The PubMed search initially yielded 284 hits; 9 were selected for further analysis and 275 were excluded because they did not meet all inclusion criteria (see Fig. 2 for diagram on article selection). Of the 9 selected articles, 1 involved affiliates from several HTA agencies [
      • Oyebode O.
      • Garrett Z.
      • George E.
      • et al.
      Evidence requirements for reimbursement of pharmaceuticals across Europe.
      ], 4 were specific to AIFA affiliates [
      • Tomino C.
      New perspective and new challenges in clinical trial regulation in Italy.
      ,
      • Xoxi E.
      • Tomino T.
      • de Nigro L.
      • Pani L.
      The Italian post-marketing registries.
      ,
      Italian Medicines Agency Research and Development Working Group
      Feasibility and challenges of independent research on drugs: the Italian Medicines Agency (AIFA) experience.
      ,
      • Degli Esposti L.
      • Saragoni S.
      • Sangiorgi D.
      • et al.
      The nationwide Osmed Health-Db database: a tool to support healthcare decision-making and real-world evidence generation.
      ], 3 were specific to NICE affiliates [
      • Taylor J.
      • Patrick H.
      • Lyratzopoulos G.
      • Campbell B.
      Methodological challenges in evaluating the value of registers.
      ,
      • George E.
      How real-world data compensate for scarce evidence in HTA.
      ,
      • Glynn D.
      • Campbell B.
      • Marlow M.
      • Patrick H.
      How to improve the quality of evidence when new treatments are funded conditional on collecting evidence of effectiveness and safety.
      ], and 1 was specific to an HAS affiliate [
      • Lapeyre-Mestre M.
      • Sapède C.
      • Moore N.
      • et al.
      Études Pharmaco-Épidémiologiques: Quels Niveaux De Preuve Et Comment Les Atteindre?.
      ] (see Table 1).
      Fig. 2
      Fig. 2PRISMA diagram of inclusion and exclusion of articles retrieved through the PubMed search. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; REA, relative effectiveness assessment; RWD, real-world data.
      Of the nine agency representatives approached across the six agencies, all agreed to participate (response rate = 100%). For two of the six agencies, one additional colleague was invited by the approached representatives to participate in the interview. Two interviews included one agency participant, three included two agency participants, and one included three agency participants (see Table 1). In total, 22 documents and 6 interview transcripts (labeled as a–f in Table 1) were included in the analysis.

      Initial Reimbursement Discussions

      All HTA agencies accept all available evidence on the drug undergoing REA, which implicitly includes RWD (transcripts a–f) [
      Tandvårds- och läkemedelsförmånsverket
      Guide for Companies When Applying for Subsidies and Pricing for Pharmaceutical Products.
      ,
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal 2013.
      ,
      Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. Allgemeine Methoden. Version 4.2. 2015.
      ,

      Haute Autorité de Santé. General method for assessing health technologies. 2015. Available from: http://www.has-sante.fr/portail/upload/docs/application/pdf/general_method_eval_techno.pdf. [Accessed June 21, 2016].

      ,
      Zorginstituut Nederland
      Beoordeling stand van de wetenschap en praktijk.
      ]. Agencies do not specify which sources of RWD nor which methodologies for RWD collection the applicant should resort to (transcripts a, b, d–f) [
      Tandvårds- och läkemedelsförmånsverket
      Guide for Companies When Applying for Subsidies and Pricing for Pharmaceutical Products.
      ,
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal 2013.
      ,

      Haute Autorité de Santé. General method for assessing health technologies. 2015. Available from: http://www.has-sante.fr/portail/upload/docs/application/pdf/general_method_eval_techno.pdf. [Accessed June 21, 2016].

      ,
      Zorginstituut Nederland
      Beoordeling stand van de wetenschap en praktijk.
      ]. Nevertheless, several do provide suggestions for specific RWD sources as well as preliminary guidance on the suitability of these sources to answering different scientific questions (transcripts b, c, f) [
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal 2013.
      ,
      Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. Allgemeine Methoden. Version 4.2. 2015.
      ,
      National Institute for Health and Care Excellence
      The Use of Observational Data to Inform Estimates of Treatment Effectiveness in Technology Appraisal: Methods for Comparative Individual Patient Data.
      ,
      College voor Zorgverzekeringen. Leideraad voor Uitkomstenonderzoek.
      ].
      Agencies iterate that RWD may be used to demonstrate treatment effects of the assessed drug but only under specific circumstances. For example, RWD may be used in the absence of RCT evidence on drug efficacy (transcripts b and f) [
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal 2013.
      ,
      Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. Allgemeine Methoden. Version 4.2. 2015.
      ,
      Zorginstituut Nederland
      Beoordeling stand van de wetenschap en praktijk.
      ]. In the absence of RCT data on head-to-head comparisons between treatments, RWD may be drawn upon to provide information on estimates of effectiveness to enable indirect treatment comparisons (transcripts b and f) [
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal 2013.
      ,
      Zorginstituut Nederland
      Beoordeling stand van de wetenschap en praktijk.
      ]. Finally, RWD may be used to supplement RCT data on treatment effects if data on specific subpopulations or long-term follow-up are lacking (transcripts b and f) [
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal 2013.
      ,
      Zorginstituut Nederland
      Beoordeling stand van de wetenschap en praktijk.
      ]. In all the aforementioned situations, agencies require an explicit justification why RWD was used and a clear discussion of the biases associated with the RWD used and its consequences on treatment effect estimates (transcripts a–c and e) [
      Tandvårds- och läkemedelsförmånsverket
      Guide for Companies When Applying for Subsidies and Pricing for Pharmaceutical Products.
      ,
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal 2013.
      ,
      Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. Allgemeine Methoden. Version 4.2. 2015.
      ,

      Haute Autorité de Santé. General method for assessing health technologies. 2015. Available from: http://www.has-sante.fr/portail/upload/docs/application/pdf/general_method_eval_techno.pdf. [Accessed June 21, 2016].

      ,
      Zorginstituut Nederland
      Beoordeling stand van de wetenschap en praktijk.
      ].
      Moreover, three agency guidelines iterate that RWD may be used to provide information on aspects other than treatment effect, such as epidemiological data (e.g., incidence and prevalence), resource use data, and cost data [
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal 2013.
      ,
      Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. Allgemeine Methoden. Version 4.2. 2015.
      ,
      Zorginstituut Nederland
      Beoordeling stand van de wetenschap en praktijk.
      ].
      All agencies adopt similar hierarchies of evidence in accordance with principles of evidence-based medicines [
      Tandvårds- och läkemedelsförmånsverket
      Guide for Companies When Applying for Subsidies and Pricing for Pharmaceutical Products.
      ,
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal 2013.
      ,
      Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. Allgemeine Methoden. Version 4.2. 2015.
      ,

      Haute Autorité de Santé. General method for assessing health technologies. 2015. Available from: http://www.has-sante.fr/portail/upload/docs/application/pdf/general_method_eval_techno.pdf. [Accessed June 21, 2016].

      ,
      Zorginstituut Nederland
      Beoordeling stand van de wetenschap en praktijk.
      ]. Adopted hierarchies unanimously place sources of RWD on a lower level of quality and reliability than those of RCTs. Consequently, agencies iterate that RWD may be used to confirm or supplement, rather than substitute, findings on causal treatment effects demonstrated by RCTs (transcripts b–d and f) [
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal 2013.
      ,
      Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. Allgemeine Methoden. Version 4.2. 2015.
      ,

      Haute Autorité de Santé. General method for assessing health technologies. 2015. Available from: http://www.has-sante.fr/portail/upload/docs/application/pdf/general_method_eval_techno.pdf. [Accessed June 21, 2016].

      ,
      Zorginstituut Nederland
      Beoordeling stand van de wetenschap en praktijk.
      ]. Thus, conclusions on treatment effects derived from RWD are generally regarded as more circumspect than RCT-derived conclusions by decision-making committees; examples of quotes to this effect can be found in Table 2. Two agencies, however, explicitly recognize limitations associated with strictly adopting evidence hierarchies in guidelines and state that such hierarchies should not preclude the exclusion of valuable non-RCT evidence from decision making (transcripts b and f) [
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal 2013.
      ,
      Zorginstituut Nederland
      Beoordeling stand van de wetenschap en praktijk.
      ].
      Table 2Examples of interview quotes on RWD use in IRD and CRS
      Context for RWD useTopicQuotation AQuotation B
      IRDAppraisal of RWD vs. RCT data for treatment effect estimates in general“There is this red flag in there. If you use non-randomized and non-controlled evidence, you have to be more careful, more circumspect about the relative treatment effect drawn from those studies. Ideally you should use more than one independent source of such evidence, as a back-up.”b“Of course we accept those data. We are forced by law to accept those data but we don’t have to conclude the benefit from such data.”c
      IRDRWD use to inform treatment effect estimates for orphan diseases“Yes, RWD certainly plays a role in orphan diseases since RCTs are difficult to conduct in that area. In this case, patient registries may be the most ideal source for RWD.”f“… we would then need a registry with a very, very, very, high quality. In terms of having all patients in the registry, no selection criteria and no selection bias. We could imagine that we would only then accept these registry analyses for very rare diseases, but not in general.”c
      CRSUse of RWD generated in CRS for decision making“So, we are used to using that kind of data, though we know the bias and the problems that are related to the robustness of that kind of [RWD] data. For the re-evaluation for the pricing and reimbursement of the product, this kind of data are robust enough for the analysis that we need to do for the reevaluation of pricing and reimbursement of the product.”e“You can’t really rely on it. You can use the RWD as a confirmation of the expectation you have on initial assessment and the data for the first-line population you have, and the data you have had already of the post-hoc subgroup analysis. So it is used as a confirmation of previous conclusions.”f
      CRS, conditional reimbursement scheme; IRD, initial reimbursement discussion; RCT, randomized controlled trial; RWD, real-world data.
      Agencies differ on the acceptability and impact of RWD on decision making in cases in which RCT data are sparse, for example, for orphan diseases; several state that non-RCT data could be resorted to for decision making in these cases (transcripts a, b, and f) [
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal 2013.
      ,
      Zorginstituut Nederland
      Beoordeling stand van de wetenschap en praktijk.
      ], whereas one states that resorting to non-RCT data presents a greater risk to validity of conclusions and should thus be avoided (transcript c) [
      Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. Allgemeine Methoden. Version 4.2. 2015.
      ]. Examples of quotes demonstrating agencies’ disparity of views on this issue can be found in Table 2.
      Table 3 presents a summary of policies on RWD accepted or requested and RWD appraisal in the context of IRD per agency.
      Table 3Summary of policies on RWD accepted or requested and the appraisal of RWD in the context of IRD per agency
      RWD accepted/requestedRWD appraisal
      HTA agencyRWD acceptedRWD to inform treatment effectsRWD to inform other parametersHierarchy of evidence adoptedConclusions on treatment effects on the basis of RWD regarded as circumspectConclusions on treatment effects on the basis of RWD possible in exceptional circumstances (e.g., orphan diseases)
      TLVYesUnder specific circumstancesNot mentionedYes; with regard to evidence for treatment effectsYesYes
      NICEYesUnder specific circumstancesEpidemiological data (e.g., incidence and prevalence), resource use data, and cost dataYes
      However, agency explicitly recognizes limitations associated with strictly adopting evidence hierarchies in guidelines and states that such hierarchies should not preclude the exclusion of valuable non-RCT evidence from decision making.
      ; with regard to evidence for treatment effects
      YesYes
      IQWiGYesUnder specific circumstancesEpidemiological data (e.g., incidence and prevalence) and resource use dataYes; with regard to evidence for treatment effectsYesNo
      HASYesUnder specific circumstancesNot mentionedYes; with regard to evidence for treatment effectsYesNot mentioned
      AIFAYesUnder specific circumstancesNot mentionedYes; with regard to evidence for treatment effectsYesNot mentioned
      ZINYesUnder specific circumstancesEpidemiological data (e.g., incidence and prevalence), resource use data, and cost dataYes
      However, agency explicitly recognizes limitations associated with strictly adopting evidence hierarchies in guidelines and states that such hierarchies should not preclude the exclusion of valuable non-RCT evidence from decision making.
      ; with regard to evidence for treatment effects
      YesYes
      AIFA, Italian Medicines Agency; HAS, High Authority for Health; HTA, health technology assessment; IQWiG, Institute for Quality and Efficiency in Healthcare; IRD, initial reimbursement discussion; NICE, National Institute for Health and Care Excellence; RCT, randomized controlled trial; RWD, real-world data; TLV, Dental and Pharmaceutical Benefits Agency; ZIN, National Healthcare Institute.
      low asterisk However, agency explicitly recognizes limitations associated with strictly adopting evidence hierarchies in guidelines and states that such hierarchies should not preclude the exclusion of valuable non-RCT evidence from decision making.

      Pharmacoeconomic Analyses

      Contrary to the first context, RWD is directly requested by five HTA agencies for various aspects of PEA (the sixth agency does not conduct PEA). More specifically, agencies recommend that epidemiological data (e.g., incidence and prevalence), direct and indirect costs, and resource use in routine practice be collected from national RWD sources (e.g., claims databases, registries, and hospital databases) (transcripts b, e, and f) [
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal 2013.
      ,
      Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. Allgemeine Methoden. Version 4.2. 2015.
      ,
      College voor Zorgverzekeringen. Leideraad voor Uitkomstenonderzoek.
      ,
      Tandvårds- och läkemedelsförmånsverket. General Guidelines for Economic Evaluations from the Pharmaceutical Benefits Board.
      ,
      Haute Autorité de Santé
      Choices in Methods for Economic Evaluation.
      ]. Other aspects of the evaluation, such as adherence to treatment and compliance, can also be collected from RWD sources such as registries, databases, ad hoc studies, or epidemiological surveys [
      Haute Autorité de Santé
      Choices in Methods for Economic Evaluation.
      ].
      Several agencies specify that treatment effects used for modeling relative effectiveness should primarily be based on results from RCTs (transcripts b, d, and f) [
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal 2013.
      ,
      College voor Zorgverzekeringen. Leideraad voor Uitkomstenonderzoek.
      ,
      Haute Autorité de Santé
      Choices in Methods for Economic Evaluation.
      ,
      Zorginstituut Nederland
      Richtlijn voor het uitvoeren van economische evaluaties in de gezondheidszorg.
      ]. Alternatively, RWD may provide complementary evidence on treatment effects (transcripts b, d, and f) [
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal 2013.
      ,
      Zorginstituut Nederland
      Beoordeling stand van de wetenschap en praktijk.
      ,
      Haute Autorité de Santé
      Choices in Methods for Economic Evaluation.
      ], be used to valuate the health effects over time in the form of utilities [
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal 2013.
      ,
      College voor Zorgverzekeringen. Leideraad voor Uitkomstenonderzoek.
      ,
      Zorginstituut Nederland
      Richtlijn voor het uitvoeren van economische evaluaties in de gezondheidszorg.
      ], or provide data on transition probabilities between different disease states in pharmacoeconomic models [
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal 2013.
      ,
      College voor Zorgverzekeringen. Leideraad voor Uitkomstenonderzoek.
      ].
      With regard to RWD appraisal in PEA, the use of RWD for epidemiological data, direct and indirect costs, resource use in routine practice, and adherence to treatment and compliance is largely accepted by HTA agencies. For relative treatment effects, however, the same hierarchies of evidence apply as in the context of IRD, implying that RWD is conventionally placed on a lower quality level (transcripts b, d, and f) [
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal 2013.
      ,
      Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. Allgemeine Methoden. Version 4.2. 2015.
      ,
      College voor Zorgverzekeringen. Leideraad voor Uitkomstenonderzoek.
      ,
      Haute Autorité de Santé
      Choices in Methods for Economic Evaluation.
      ,
      Zorginstituut Nederland
      Richtlijn voor het uitvoeren van economische evaluaties in de gezondheidszorg.
      ]. Therefore, conclusions for relative treatment effects on the basis of RWD are considered as being more circumspect (transcripts b–d and f) [
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal 2013.
      ,
      Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. Allgemeine Methoden. Version 4.2. 2015.
      ,
      College voor Zorgverzekeringen. Leideraad voor Uitkomstenonderzoek.
      ,
      Haute Autorité de Santé
      Choices in Methods for Economic Evaluation.
      ,
      Zorginstituut Nederland
      Richtlijn voor het uitvoeren van economische evaluaties in de gezondheidszorg.
      ].
      Table 4 presents a summary of policies on RWD accepted or requested and RWD appraisal in the context of PEA per agency.
      Table 4Summary of policies on RWD accepted or requested and the appraisal of RWD in the context of PEA per agency
      RWD accepted/requestedRWD appraisal
      HTA agencyRWD recommendedRWD to inform treatment effectsRWD to inform other parametersHierarchy of evidence adoptedConclusions on treatment effects on the basis of RWD regarded as circumspectConclusions on other parameters on the basis of RWD regarded as reliable
      TLVYesUnder specific circumstancesEpidemiological data (e.g., incidence and prevalence), costs (direct and indirect), and resource useYes; with regard to evidence for treatment effectsYesYes
      NICEYesUnder specific circumstancesEpidemiological data (e.g., incidence and prevalence), costs (direct and indirect), and resource useYes
      However, agency explicitly recognizes limitations associated with strictly adopting evidence hierarchies in guidelines and states that such hierarchies should not preclude the exclusion of valuable non-RCT evidence from decision making.
      ; specifically with regard to relative treatment effects
      YesYes
      IQWiGNANANANANANA
      HASYesUnder specific circumstancesEpidemiological data (e.g., incidence and prevalence), costs (direct and indirect), resource use, adherence, and complianceYes; with regard to evidence for treatment effectsYesYes
      AIFAYesUnder specific circumstancesEpidemiological data (e.g., incidence and prevalence), costs (direct and indirect), and resource useYes; with regard to evidence for treatment effectsYesYes
      ZINYesUnder specific circumstancesEpidemiological data (e.g., incidence and prevalence), costs (direct and indirect), and resource useYes
      However, agency explicitly recognizes limitations associated with strictly adopting evidence hierarchies in guidelines and states that such hierarchies should not preclude the exclusion of valuable non-RCT evidence from decision making.
      ; with regard to evidence for treatment effects
      YesYes
      AIFA, Italian Medicines Agency; HAS, High Authority for Health; HTA, health technology assessment; IQWiG, Institute for Quality and Efficiency in Healthcare; NA, not applicable; NICE, National Institute for Health and Care Excellence; PEA, pharmacoeconomic analysis; RCT, randomized controlled trial; RWD, real-world data; TLV, Dental and Pharmaceutical Benefits Agency; ZIN, National Healthcare Institute.
      low asterisk However, agency explicitly recognizes limitations associated with strictly adopting evidence hierarchies in guidelines and states that such hierarchies should not preclude the exclusion of valuable non-RCT evidence from decision making.

      Conditional Reimbursement Schemes

      Three of the six HTA agencies implement CRS (transcripts d–f) [
      • Xoxi E.
      • Tomino T.
      • de Nigro L.
      • Pani L.
      The Italian post-marketing registries.
      ,
      Italian Medicines Agency Research and Development Working Group
      Feasibility and challenges of independent research on drugs: the Italian Medicines Agency (AIFA) experience.
      ,

      Haute Autorité de Santé. Les études post-inscription sur les technologies de santé (médicaments, dispositifs, médicaux et actes). 2015. Available from: http://tinyurl.com/orw8cw2. [Accessed June 21, 2016].

      ,
      Zorginstituut Nederland
      Procedure voorwaardelijke toelating geneeskundige zorg 2015.
      ]. A fourth agency stated briefly that reimbursement can be conditionally offered to allow an applicant time to procure more RWD on long-term effects (transcript a) [
      Tandvårds- och läkemedelsförmånsverket
      The Swedish Pharmaceutical Reimbursement System.
      ]. Meanwhile, a fifth agency recently announced the establishment of a CRS for oncologic drugs (transcript b) [

      National Health Service England. The Cancer Drugs Fund: transition to the new model. Available from: https://www.england.nhs.uk/ourwork/cancer/cdf/#. [Accessed June 21, 2016].

      ]. It, however, remains unclear whether the latter two schemes constitute ones as established as those outlined by the other three agencies (transcripts a and b).
      Only one of the three agencies clearly defined criteria for the selection of candidates for CRS and a procedure to do so (transcript f) [
      College voor Zorgverzekeringen. Leideraad voor Uitkomstenonderzoek.
      ,
      Zorginstituut Nederland
      Procedure voorwaardelijke toelating geneeskundige zorg 2015.
      ].
      The purposes for RWD collection for CRS differed between the three agencies. For the first agency, a product is nominated for conditional reimbursement on two conditions: that it is highly innovative and data on its effectiveness are highly sparse at initial assessment. Therefore, the purpose for data collection is focused primarily on demonstrating effectiveness, with a preference for RCT data and a supplementary role for RWD (transcript f) [
      College voor Zorgverzekeringen. Leideraad voor Uitkomstenonderzoek.
      ,
      Zorginstituut Nederland
      Procedure voorwaardelijke toelating geneeskundige zorg 2015.
      ]. For the second agency, a contract is drawn up between the agency and an applicant to conduct postmarketing studies that aim to answer questions raised during initial assessment. These questions may relate equally to issues of effectiveness and/or cost-effectiveness of the drug in national clinical practice and a preference is made for RWD rather than RCT data (transcript d) [

      Haute Autorité de Santé. Les études post-inscription sur les technologies de santé (médicaments, dispositifs, médicaux et actes). 2015. Available from: http://tinyurl.com/orw8cw2. [Accessed June 21, 2016].

      ]. For the last agency, recommendations to set up postmarketing studies are similarly based on questions raised during initial assessment with a preference for RWD. Nevertheless, the use of study results for the last agency varies; they can be used to inform re-assessment of effectiveness and/or cost-effectiveness in clinical practice, but may also be used for repricing discussions (transcript e) [
      • Xoxi E.
      • Tomino T.
      • de Nigro L.
      • Pani L.
      The Italian post-marketing registries.
      ,
      Italian Medicines Agency Research and Development Working Group
      Feasibility and challenges of independent research on drugs: the Italian Medicines Agency (AIFA) experience.
      ].
      Notwithstanding these principal differences, two agencies follow the same procedure for conditional reimbursement. First, gaps in evidence presented in submissions for IRD are systematically identified by the agencies. Second, the agencies request that the applicant develop a study protocol to collect the RWD needed to inform such gaps, implying that RWD collected for each drug candidate is highly case-specific. Both agencies provide methodological guidance to applicants on which study designs to choose to answer the scientific questions raised during initial assessment. This guidance also includes detailed examples of existing national RWD sources that may be used to answer specific questions [
      College voor Zorgverzekeringen. Leideraad voor Uitkomstenonderzoek.
      ,

      Haute Autorité de Santé. Les études post-inscription sur les technologies de santé (médicaments, dispositifs, médicaux et actes). 2015. Available from: http://tinyurl.com/orw8cw2. [Accessed June 21, 2016].

      ,
      Zorginstituut Nederland
      Procedure voorwaardelijke toelating geneeskundige zorg 2015.
      ]. Third, the applicant’s study protocol(s) are reviewed by independent committees to judge their scientific quality and feasibility. Once relevant adjustments are made to the protocol(s), a contract is drawn up between the agency and the applicant in which the study protocol and the date for submitting additional evidence are specified. Further adaptations to the study protocol by the applicant are possible but only after consultation with the agency [
      College voor Zorgverzekeringen. Leideraad voor Uitkomstenonderzoek.
      ,

      Haute Autorité de Santé. Les études post-inscription sur les technologies de santé (médicaments, dispositifs, médicaux et actes). 2015. Available from: http://tinyurl.com/orw8cw2. [Accessed June 21, 2016].

      ,
      Zorginstituut Nederland
      Procedure voorwaardelijke toelating geneeskundige zorg 2015.
      ]. It is unclear whether the same procedure also applies for CRS implemented by the third agency.
      Unlike the first two agencies, which lay the burden of RWD collection on applicants, the third agency often actively participates in, or initiates its own, product or indication registries (transcripts d–f) [
      • Tomino C.
      New perspective and new challenges in clinical trial regulation in Italy.
      ,
      • Xoxi E.
      • Tomino T.
      • de Nigro L.
      • Pani L.
      The Italian post-marketing registries.
      ,
      Italian Medicines Agency Research and Development Working Group
      Feasibility and challenges of independent research on drugs: the Italian Medicines Agency (AIFA) experience.
      ,

      Haute Autorité de Santé. Les études post-inscription sur les technologies de santé (médicaments, dispositifs, médicaux et actes). 2015. Available from: http://tinyurl.com/orw8cw2. [Accessed June 21, 2016].

      ,
      Zorginstituut Nederland
      Procedure voorwaardelijke toelating geneeskundige zorg 2015.
      ].
      All three agencies require that the studies implemented deliver data of adequate quality and robustness to answer questions identified during initial assessment (transcripts d–f) [
      College voor Zorgverzekeringen. Leideraad voor Uitkomstenonderzoek.
      ,

      Haute Autorité de Santé. Les études post-inscription sur les technologies de santé (médicaments, dispositifs, médicaux et actes). 2015. Available from: http://tinyurl.com/orw8cw2. [Accessed June 21, 2016].

      ,
      Zorginstituut Nederland
      Procedure voorwaardelijke toelating geneeskundige zorg 2015.
      ]. Moreover, two agencies require that the study eventually conducted adhere strictly to the protocol agreed upon by all parties. This is to ensure that the scientific quality and outcomes of the study remain valuable for decision making. If these conditions are met, results generated by the studies would form the basis for decision making during re-assessment (transcripts d–f) [
      College voor Zorgverzekeringen. Leideraad voor Uitkomstenonderzoek.
      ,

      Haute Autorité de Santé. Les études post-inscription sur les technologies de santé (médicaments, dispositifs, médicaux et actes). 2015. Available from: http://tinyurl.com/orw8cw2. [Accessed June 21, 2016].

      ,
      Zorginstituut Nederland
      Procedure voorwaardelijke toelating geneeskundige zorg 2015.
      ]. Nevertheless, quotes from interviews shed light on varying acceptability of results generated from such studies for decision-making practice (see Table 2). Moreover, there was no guidance on the impact of RWD on decision making if conclusions for treatment effects on the basis of RWD contradict those from RCT-based evidence.
      Table 5 presents a summary of policies on RWD use in the context of CRS per agency.
      Table 5Summary of policies on RWD use in the context of CRS per agency
      HTA agencyCRS implemented?CRS aimsCRS procedurePreference for RWDInvolvement in collection of RWD?Preference for RWDImpact of RWD on decision making
      TLVNo
      CRS schemes implemented by the agencies do not constitute schemes as established as those outlined by HAS, AIFA, and ZIN.
      NANANANANANA
      NICENo
      CRS schemes implemented by the agencies do not constitute schemes as established as those outlined by HAS, AIFA, and ZIN.
      NANANANANANA
      IQWiGNoNANANANANANA
      HASYesEffectiveness and/or cost-effectiveness1: Identification of evidence gapYesNoYesConditional on whether data delivered sufficiently address evidence gap highlighted and adherence to agreed-upon study protocol
      2: Consultation on study design
      3: Decision making based on results
      AIFAYesEffectiveness, cost-effectiveness, and/or price re-negotiationsNot mentionedYesYesYesConditional on whether data delivered sufficiently address evidence gap highlighted
      ZINYesEffectiveness1: Identification of evidence gapNo; in first instance RCT data with RWD as supplementaryNoNo; in first instance RCT data with RWD as supplementary evidenceConditional on whether data delivered sufficiently address evidence gap highlighted and adherence to agreed-upon study protocol
      2: Consultation on study design
      3: Decision making based on results
      AIFA, Italian Medicines Agency; CRS, conditional reimbursement scheme; HAS, High Authority for Health; HTA, health technology assessment; IQWiG, Institute for Quality and Efficiency in Healthcare; NICE, National Institute for Health and Care Excellence; RCT, randomized controlled trial; RWD, real-world data; TLV, Dental and Pharmaceutical Benefits Agency; ZIN, National Healthcare Institute.
      low asterisk CRS schemes implemented by the agencies do not constitute schemes as established as those outlined by HAS, AIFA, and ZIN.
      Similarities and differences in policies for RWD accepted or requested and RWD appraisal in IRD, PEA, and CRS are presented in Table 6.
      Table 6Summary of similarities and differences in policies for RWD accepted or requested and RWD appraisal within the three contexts
      ContextRWD accepted/requestedRWD appraisal
      IRDSummary of commonalitiesSummary of commonalities
      • All sources of data are welcomed in submissions. This implies that RWD is also welcome.
      • Treatment effects: RWD can be used to inform on treatment effects when RCT evidence is absent on specific head-to-head comparisons. Biases related to RWD must, however, be explored and documented.
      • Other domains: RWD can be used to provide evidence on epidemiological data, natural history of disease, or resource use data.
      • Agencies do not specify which kind of RWD should be collected nor the methods for collection. Nevertheless, the choice of which RWD and collection methods should be justifiable given the scientific questions at hand.
      • All agencies adopt evidence hierarchies in accordance with evidence-based medicine. Hierarchies consistently rank RWD at a lower quality level than RCT data.
      • Impact of RWD on decision making differs according to contextual factors:
      • Conclusions regarding causal effects that are based on RWD will be regarded as more circumspect.
      • RWD can be used to supplement/confirm RCT-based conclusions on treatment effects.
      • For some agencies, impact of RWD may be higher in cases in which RCTs are difficult to conduct (e.g., rare diseases).
      • There is lack of clarity on RWD impact in the case of conflicting evidence (cf. RCT)
      Summary of differencesSummary of differences
      • One agency recently published a comprehensive list of RWD used in technology appraisals, detailing that comparative IPD, noncomparative IPD, and aggregated data have been used in decision making. In addition, the document included detailed guidance on statistical methods for use of RWD in submissions.
      • No significant differences further.
      • Two agencies explicitly recognize limitations in adhering to strict evidence hierarchies in guidelines by stating that such hierarchies should not preclude the exclusion of valuable non-RCT evidence from decision making. One agency advises against deviating from evidence hierarchies when considering evidence inclusion for decision making.
      • Two agencies stipulate that in cases in which RCT data are sparse (especially orphan diseases), RWD may be the only source of data available and thus could be used for decision making. Contrastingly, one agency stipulates that the circumstance of small patient populations (e.g., orphan diseases) does not necessitate deviance from the principles of evidence hierarchies.
      PEASummary of commonalitiesSummary of commonalities
      • RWD is directly requested by HTA agencies for PEA.
      • Treatment effects: RWD can be used to inform on treatment effects when RCT evidence is absent on specific head-to-head comparisons. Biases related to RWD must, however, be explored and documented.
      • Costs and resource use data: National RWD is the preferred source for costs data (direct and indirect) and resource use data.
      • Other domains: RWD can be used to provide data on quality of life, adherence, epidemiological data, and transition probabilities for models.
      • RWD use to inform parameters other than treatment effects is largely accepted.
      • Hierarchies of evidence adopted by HTA agencies consistently rank RWD at a lower quality level than RCT data.
      • Impact of RWD on decision making differs according to contextual factors:
      • Conclusions regarding causal effects that are based on RWD will be regarded as more circumspect.
      • RWD can be used to supplement/confirm RCT-based conclusions on treatment effects.
      Summary of differencesSummary of differences
      There are no significant differences.There are no significant differences.
      CRSSummary of commonalitiesSummary of commonalities
      • RWD requested in any scheme is case-specific but follows similar processes for two agencies:
      • 1: Identification of evidence gaps during IRD
      • 2: Assessment of study proposal to collect data for scientific quality, feasibility, and relevance
      • 3: Agreement on study protocol and date of collected RWD delivery for re-assessment of relative effectiveness
      • Agencies provide practical guidance for applicants on:
      • Which scientific questions different study designs can and cannot answer.
      • Existing national RWD sources and relevance for providing specific information.
      • The impact of RWD collected rests on the following conditions:
      • That applicants take practical guidance available into consideration when designing the study protocol
      • That the research conducted delivers the answers to evidence gaps identified
      • That the research conducted adheres to the protocol agreed upon by all parties
      Summary of differencesSummary of differences
      • Only three of the six HTA agencies implement CRS.
      • Differing aims of CRS (effectiveness vs. cost-effectiveness vs. repricing discussions) influence the type of data requested in each scheme.
      • The degree of guidance available for applicants varies between the three agencies: two agencies have guidelines to this effect, yet one does not.
      • One agency often actively participates in, or initiates its own, product or indication registries. The remaining two agencies lay the burden of data collection on applicants.
      There are no significant differences.
      CRS, conditional reimbursement scheme; HTA, health technology assessment; IPD, individual patient-level data; IRD, initial reimbursement discussion; PEA, pharmacoeconomic analysis; RCT, randomized controlled trial; RWD, real-world data.

      Discussion

      Policies for RWD accepted or requested and RWD appraisal for REA of drugs adopted by the six agencies differed between the three contexts analyzed. For example, although RWD use for IRD was accepted but not explicitly recommended, its use was recommended by agencies for PEA and CRS. RWD may provide evidence on numerous parameters of REA: (relative) treatment effects, epidemiological data, resource use data, and cost data.
      Policies for RWD accepted or requested and RWD appraisal for REA of drugs differed between the six agencies within the same contexts. An important example relates to RWD use to provide data on treatment effects for IRD in situations in which it may be difficult to conduct RCTs (e.g., orphan diseases). Although some agencies deem this acceptable, others explicitly advise against it. Similarly, policies for CRS differed whereby the aims of the three agencies’ schemes, procedures for conducting CRS, as well as agencies’ involvement in RWD collection in CRS varied.
      Intercontext policy variations may be an issue if the effectiveness and pharmacoeconomic components of HTA dossiers submitted to an agency are examined by two different assessors who subsequently appraise the RWD differently. Another compounding factor presents itself in agencies that offer a possibility for CRS, because the manner with which these different assessors would be required to appraise RWD in the effectiveness and pharmacoeconomic components of a specific dossier will inevitably depend on whether the dossier is submitted as a standard dossier or as a candidate for CRS. Bearing these points in mind, one can argue that standardizing the implementation of policies on RWD use for decision making in practice may be difficult in any single HTA agency.
      Meanwhile, variations between agencies’ policies may present marketing authorization holders (MAHs) with a multitude of challenging questions when developing strategies for evidence generation across the product life cycle [

      Makady A. Review of policies and perspectives on real-world data. 2015. Available from: http://tinyurl.com/p2qq9xh. [Accessed April 24, 2015].

      ,
      • Mestre-Ferrandiz J.
      • Deverka P.
      • Pistollato M.
      • Rosenberg E.
      The Current Drug Development Paradigm: Responding to US and European Demands for Evidence of Comparative Effectiveness and Relative Effectiveness.
      ,

      The Academy of Medical Sciences. Real world evidence: summary of a joint meeting held on 17 September 2015 by the Academy of Medical Sciences and the Association of the British Pharmaceutical Industry. 2016. Available from: https://www.acmedsci.ac.uk/viewFile/56cab22108cf9.pdf. [Accessed July 15, 2016].

      ]. For instance, in the context of CRS, MAHs would need to question whether their product qualifies as a candidate for CRS in the different countries; whether questions raised by the various agencies would overlap or differ; and consider whether one study would suffice to collect the RWD needed for all three agencies.
      Hierarchies of evidence adopted by HTA agencies prominently featured in documents and interview transcripts assessed. Several agencies implement such hierarchies through tools for classification of evidence quality (e.g., The Grading of Recommendations Assessment, Development and Evaluation working group [GRADE]) [
      • Guyatt G.H.
      • Oxman A.D.
      • Vist G.E.
      • et al.
      GRADE Working Group
      GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.
      ]. Although evidence hierarchies have well-established roots in evidence-based medicine, it is debatable whether they are applicable to the concept of RWD use for HTA. Conventionally, hierarchies automatically downgrade all RWD without exploring the subtle differences between the advantages, disadvantages, and relevance of different RWD sources (e.g., patient registries or claims databases). More importantly, such evidence hierarchies do not address the differences in the type of insights provided by RCT data (efficacy data with high internal validity) and different forms of RWD (long-term data on safety and effectiveness from registry data, resource use data from claims databases, or patient-reported outcomes from pragmatic clinical trials). An increasing body of literature also refers to the relevance of using data from pragmatic clinical trials for more generalizable and translatable evidence on real-world outcomes [
      • Geonnotti K.
      • Wang W.
      • Peikes D.
      Using Pragmatic Clinical Trials to Test the Effectiveness of Patient-Centered Medical Home Models in Real-World Settings.
      ,
      • Delitto A.
      Pragmatic clinical trials: implementation opportunity, or just another fad?.
      ,
      • Loudon K.
      • Treweek S.
      • Sullivan F.
      • et al.
      The PRECIS-2 tool: designing trials that are fit for purpose.
      ], yet guidance on this topic was not always reflected in agency guidelines. This can result in excluding valuable evidence in decision making. Furthermore, some agencies may abandon the rigid framework of evidence hierarchies because of pragmatic reasons (e.g., in situations in which RCTs are difficult to conduct or for CRS), and others even provide methodological guidance on such aspects [
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal 2013.
      ,
      National Institute for Health and Care Excellence
      The Use of Observational Data to Inform Estimates of Treatment Effectiveness in Technology Appraisal: Methods for Comparative Individual Patient Data.
      ,
      College voor Zorgverzekeringen. Leideraad voor Uitkomstenonderzoek.
      ,

      Haute Autorité de Santé. Les études post-inscription sur les technologies de santé (médicaments, dispositifs, médicaux et actes). 2015. Available from: http://tinyurl.com/orw8cw2. [Accessed June 21, 2016].

      ]. Therefore, it may be necessary for HTA agencies to consider how implementation of rigid evidence hierarchies could be adapted to enable effective use of RWD in decision-making processes.
      The lack of harmonization of policies for RWD use in REA of drugs may discourage MAHs from collecting or analyzing RWD for HTA purposes [

      Makady A. Review of policies and perspectives on real-world data. 2015. Available from: http://tinyurl.com/p2qq9xh. [Accessed April 24, 2015].

      ,
      • Mestre-Ferrandiz J.
      • Deverka P.
      • Pistollato M.
      • Rosenberg E.
      The Current Drug Development Paradigm: Responding to US and European Demands for Evidence of Comparative Effectiveness and Relative Effectiveness.
      ,

      The Academy of Medical Sciences. Real world evidence: summary of a joint meeting held on 17 September 2015 by the Academy of Medical Sciences and the Association of the British Pharmaceutical Industry. 2016. Available from: https://www.acmedsci.ac.uk/viewFile/56cab22108cf9.pdf. [Accessed July 15, 2016].

      ]. Therefore, it may be useful for HTA agencies in Europe to align policies on RWD and provide guidance on practical aspects of RWD collection and analysis. This is especially important in light of the increasing trend of new (oncology or orphan) drugs granted conditional marketing authorization on the basis of phase II data or surrogate outcomes rather than phase III RCT data [
      • Lipska I.
      • Hoekman J.
      • McAuslane N.
      • et al.
      Does conditional approval for new oncology drugs in Europe lead to differences in health technology assessment decisions?.
      ,
      • Hoekman J.
      • Boon W.P.
      • Bouvy J.C.
      • et al.
      Use of the conditional marketing authorization pathway for oncology medicines in Europe.
      ,
      • Kaaniche A.
      • Troubat A.
      • Sherwood A.
      Does conditional marketing authorisation influence market access in France, England, and Germany?.
      ]. A harmonized set of policies on RWD use for HTA would provide MAHs with the ability to plan alternative evidence generation pathways that rely less on RCTs and more on real-world studies, the latter theoretically yielding outcomes more relevant for HTA purposes [
      • Freemantle N.
      • Strack T.
      Real-world effectiveness of new medicines should be evaluated by appropriately designed clinical trials.
      ,

      Association of the British Pharmaceutical Industry. Demonstrating value with real world data: a practical guide. 2015. Available from: http://www.abpi.org.uk/our-work/library/guidelines/Pages/real-world-data.aspx. [Accessed April 24, 2015].

      ,

      Foltz D, Ferrara L, Volkommer R, Turisco F. Real-World Data Research: A case for action. 2013. Computer Sciences Corporation.

      ,
      • Berger M.L.
      • Lipset C.
      • Gutteridge A.
      • et al.
      Optimizing the leveraging of real-world data to improve the development and use of medicines.
      ]. The EUnetHTA may provide a platform for discussions on aligning RWD policies. The EUnetHTA has recently published position articles on additional (non-RCT) evidence generation for REA and is finalizing proposals for pilot projects that will address some of the aforementioned issues [

      European network for Health Technology Assessment. Position paper on how to best formulate research recommendations for primary research arising from HTA reports. Available from: http://eunethta.eu/sites/5026.fedimbo.belgium.be/files/EUnetHTA%20Position%20Paper%20on%20research%20recommendations.pdf. [Accessed July 15, 2016].

      ,

      European network for Health Technology Assessment. Position paper on how to decide on the appropriate study design for primary research arising from HTA reports. Available from: http://eunethta.eu/sites/5026.fedimbo.belgium.be/files/EUnetHTA%20Position%20Paper%20on%20study%20design_0.pdf. [Accessed July 15, 2016].

      ,

      European network for Health Technology Assessment. Core protocol pilot for additional evidence generation. Available from: http://www.eunethta.eu/sites/5026.fedimbo.belgium.be/files/news-attachments/eunethta_core_protocol_pilot_for_aeg.pdf. [Accessed July 15, 2016].

      ].
      In addition to studying differences in policies for RWD use in REA of drugs between different contexts and agencies, determining whether differences extend to the implementation of these policies in practice is important. When asked if their agency accepts or requests RWD, one HTA representative stated, “Of course we accept those data. We are forced by law to accept those data but we don’t have to conclude the benefit from such data.” This implies that RWD has quite a low impact on decision making in that agency, in contrast to others. When representatives from two of the three agencies implementing CRS were asked about the impact of RWD in decision making at re-assessment, they displayed contradicting views. One stated, “You can’t really rely on them. You can use the RWD as a confirmation of the expectation you have at initial discussions,” whereas the other stated, “For the re-evaluation of pricing and reimbursement of the product, that kind of data are robust enough.” Therefore, the reality of how RWD is used in practice may differ from policies and should be the focus for future research.

      Strengths

      To ensure that all available information on RWD policies was gathered for all six HTA agencies, a mixed-methods approach was used that included a review of agency Web sites, academic literature, and stakeholder interviews. This minimized the probability of important information being excluded from analysis. Moreover, the selection of documents for analysis, data abstraction, and coding was conducted independently by two authors.

      Limitations

      Although six European HTA agencies were included, this does not automatically mean that we provided a representative overview of all European policies on RWD use in REA of drugs. The agencies selected represent only those vested in the Big Four jurisdictions and two agencies with pioneering roles in cutting-edge European HTA initiatives. Nevertheless, considering the novelty of the topic on RWD use in REA of drugs and the impact of the agencies and jurisdictions included, this sample was deemed as relevant for an initial policy analysis on RWD use in REA of drugs in Europe.
      The information available for analysis varied between agencies. Language capabilities of the involved researchers meant that Swedish and Italian documents were excluded from the analysis. As a result, valuable information from documents written by TLV or AIFA may have been overlooked. Moreover, not all agencies published guidelines that specifically focus on the use of RWD in REA. But then, gathering information from several sources through agency Web site searches, the PubMed search, and stakeholder interviews ensured that the impact of excluded information was minimal. Furthermore, TLV published numerous English guidelines on REA [
      Tandvårds- och läkemedelsförmånsverket
      Guide for Companies When Applying for Subsidies and Pricing for Pharmaceutical Products.
      ,
      Tandvårds- och läkemedelsförmånsverket. General Guidelines for Economic Evaluations from the Pharmaceutical Benefits Board.
      ,
      Tandvårds- och läkemedelsförmånsverket
      The Swedish Pharmaceutical Reimbursement System.
      ] and AIFA affiliates published several English academic articles on RWD use in Italian practice [
      • Oyebode O.
      • Garrett Z.
      • George E.
      • et al.
      Evidence requirements for reimbursement of pharmaceuticals across Europe.
      ,
      • Tomino C.
      New perspective and new challenges in clinical trial regulation in Italy.
      ,
      • Xoxi E.
      • Tomino T.
      • de Nigro L.
      • Pani L.
      The Italian post-marketing registries.
      ,
      Italian Medicines Agency Research and Development Working Group
      Feasibility and challenges of independent research on drugs: the Italian Medicines Agency (AIFA) experience.
      ,
      • Degli Esposti L.
      • Saragoni S.
      • Sangiorgi D.
      • et al.
      The nationwide Osmed Health-Db database: a tool to support healthcare decision-making and real-world evidence generation.
      ].
      It can be argued that data gathered during interviews may reflect only the interviewees’ opinion, rather than represent the agencies’ official position. We attempted to account for this through selective sampling of participants, providing all approached participants with the opportunity of inviting colleagues they deemed relevant to the interview and by interviewing more than one person per institute. In addition, information provided during interviews was compared with that from policy documents and academic publications to ensure alignment between data sources.

      Conclusions

      Individual agencies’ policies regarding RWD accepted or requested and appraisal of RWD for REA of drugs vary notably across the three contexts assessed: IRD, PEA, and CRS. In addition, differences are present between each agency’s policies on RWD use for IRD, PEA, and CRS. For example, the manner by which RWD is appraised for decision making varies in any given agency, being largely acceptable for numerous PEA parameters and CRS but not for informing treatment effects for IRD. Moreover, the existence of CRS, as well as the manner of the implementation of RWD use in CRS, is different in the agencies examined.
      The lack of harmonization of policies on RWD use for REA of drugs may present MAHs with a multitude of challenging questions when they consider collecting and using RWD for HTA purposes. As a result, MAHs may be discouraged to use RWD for HTA. Therefore, HTA agencies in Europe may collaborate to align policies on RWD and provide guidance on practical aspects of RWD collection and analysis. Recently published position articles and future project proposals by the EUnetHTA may provide a starting point for discussions and a suitable platform for HTA agencies to achieve this.

      Acknowledgments

      Source of financial support: The work leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant agreement number 115546, resources of which are composed of financial contribution from the European Union’s Seventh Framework Programme (FP7/2007-2013) and European Federation of Pharmaceutical Industries and Associations (EFPIA) companies in kind contribution.

      Supplementary materials

      References

      1. European Commission. Directive 2001/20/ec of the European Parliament and of the Council. 2015. Available from: http://ec.europa.eu/health/files/eudralex/vol-1/dir_2001_20/dir_2001_20_en.pdf. [Accessed April 24, 2015]

        • Eichler H.G.
        • Abadie E.
        • Breckenridge A.
        • et al.
        Bridging the efficacy-effectiveness gap: a regulator’s perspective on addressing variability of drug response.
        Nat Rev Drug Discov. 2011; 10: 495-506
        • Eichler H.G.
        • Oye K.
        • Baird L.G.
        • et al.
        Adaptive licensing: taking the next step in the evolution of drug approval.
        Clin Pharmacol Ther. 2012; 91: 426-437
        • Romio S.
        • Sturkenboom M.
        • Corrao G.
        Real-world data from the health decision maker perspective. What are we talking about?.
        Epidemiol Biostat Public Health. 2013; 10: e8979-1-9
        • Garrison L.P.
        • Neumann P.J.
        • Erickson P.
        • et al.
        Using real world data for coverage and payment decisions: the ISPOR Real World Data Task Force Report.
        Value Health. 2007; 10: 326-335
        • Fung V.
        • Brand R.J.
        • Newhouse J.P.
        • Hsu J.
        Using Medicare data for comparative effectiveness research opportunities and challenges.
        Am J Manag Care. 2011; 17: 488
      2. Annemans L, Aristides M, Kubin M. Real-life data: a growing need. 2015. Available from: http://www.ispor.org/news/articles/oct07/rld.asp. [Accessed April 24, 2015].

      3. Makady A. Review of policies and perspectives on real-world data. 2015. Available from: http://tinyurl.com/p2qq9xh. [Accessed April 24, 2015].

        • Vassall A.
        • Mangham-Jefferies L.
        • Gomez G.B.
        • et al.
        Incorporating demand and supply constraints into economic evaluations in low-income and middle-income countries.
        Health Econ. 2016; 25: 95-115
      4. Organisation for Economic Co-operation and Development. OECD composite leading indicators: reference turning points and component series. 2016. Available from: http://www.oecd.org/std/leading-indicators/oecdcompositeleadingindicatorsreferenceturningpointsandcomponentseries.htm. [Accessed November 2, 2016].

        • Debaere P.
        EU Coordination in International Institutions: Policy and Process in Gx Forums.
        Palgrave Macmillan, New York, NY2015
        • Jorgensen K.E.
        • Laatikainen K.V.
        Routledge Handbook on the European Union and International Institutions: Performance, Policy, Power.
        Routledge, London2012
      5. European network for Health Technology Assessment. EUnetHTA JA3 (2016–2020) lead partners. 2016. Available from: http://eunethta.eu/activities/joint-action-3/jointaction31/eunethta-joint-action-3-2016-2020#involved_organisations. [Accessed November 2, 2016].

        • Tong A.
        • Sainsbury P.
        • Craig J.
        Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups.
        Int J Qual Health Care. 2007; 19: 349-357
        • Hsieh H.F.
        • Shannon S.E.
        Three approaches to qualitative content analysis.
        Qual Health Res. 2005; 15: 1277-1288
      6. Makady A. GetReal. Updated glossary of definitions of common terms. 2015. Available from: https://www.imi-getreal.eu/Portals/1/Documents/Publications/D1.3%20GetReal%20Glossary%20-%20Update.pdf. [Accessed June 21, 2016]. [Accessed May 30, 2016].

        • Oyebode O.
        • Garrett Z.
        • George E.
        • et al.
        Evidence requirements for reimbursement of pharmaceuticals across Europe.
        Int J Technol Assess Health Care. 2015; 31: 59-67
        • Tomino C.
        New perspective and new challenges in clinical trial regulation in Italy.
        Ann Ist Super Sanita. 2011; 47: 19-21
        • Xoxi E.
        • Tomino T.
        • de Nigro L.
        • Pani L.
        The Italian post-marketing registries.
        Pharm Program. 2012; 5: 57-60
        • Italian Medicines Agency Research and Development Working Group
        Feasibility and challenges of independent research on drugs: the Italian Medicines Agency (AIFA) experience.
        Eur J Clin Invest. 2010; 40: 69-86
        • Degli Esposti L.
        • Saragoni S.
        • Sangiorgi D.
        • et al.
        The nationwide Osmed Health-Db database: a tool to support healthcare decision-making and real-world evidence generation.
        Value Health. 2015; 18: A693
        • Taylor J.
        • Patrick H.
        • Lyratzopoulos G.
        • Campbell B.
        Methodological challenges in evaluating the value of registers.
        Int J Technol Assess Health Care. 2014; 30: 28-33
        • George E.
        How real-world data compensate for scarce evidence in HTA.
        Z Evid Fortbild Qual Gesundhwes. 2016; 112: S23-S26
        • Glynn D.
        • Campbell B.
        • Marlow M.
        • Patrick H.
        How to improve the quality of evidence when new treatments are funded conditional on collecting evidence of effectiveness and safety.
        J Health Serv Res Policy. 2016; 21: 71-72
        • Lapeyre-Mestre M.
        • Sapède C.
        • Moore N.
        • et al.
        Études Pharmaco-Épidémiologiques: Quels Niveaux De Preuve Et Comment Les Atteindre?.
        [French] Therapie. 2013; 68: 241-246
        • Tandvårds- och läkemedelsförmånsverket
        Guide for Companies When Applying for Subsidies and Pricing for Pharmaceutical Products.
        Tandvårds- och läkemedelsförmånsverket, Stockholm, Sweden2013
        • National Institute for Health and Care Excellence
        Guide to the Methods of Technology Appraisal 2013.
        National Institute for Health and Care Excellence, London2015
      7. Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. Allgemeine Methoden. Version 4.2. 2015.
      8. Haute Autorité de Santé. General method for assessing health technologies. 2015. Available from: http://www.has-sante.fr/portail/upload/docs/application/pdf/general_method_eval_techno.pdf. [Accessed June 21, 2016].

        • Zorginstituut Nederland
        Beoordeling stand van de wetenschap en praktijk.
        Zorginstituut Nederland, Diemen, The Netherlands2015
        • National Institute for Health and Care Excellence
        The Use of Observational Data to Inform Estimates of Treatment Effectiveness in Technology Appraisal: Methods for Comparative Individual Patient Data.
        National Institute for Health and Care Excellence, London2015
      9. College voor Zorgverzekeringen. Leideraad voor Uitkomstenonderzoek.
      10. Tandvårds- och läkemedelsförmånsverket. General Guidelines for Economic Evaluations from the Pharmaceutical Benefits Board.
        Tandvårds- och Läkemedelsförmånsverket, Stockholm, Sweden2003
        • Haute Autorité de Santé
        Choices in Methods for Economic Evaluation.
        Haute Autorité de Santé, Saint-Denis, France2012
        • Zorginstituut Nederland
        Richtlijn voor het uitvoeren van economische evaluaties in de gezondheidszorg.
        Zorginstituut Nederland, Diemen, The Netherlands2015
      11. Haute Autorité de Santé. Les études post-inscription sur les technologies de santé (médicaments, dispositifs, médicaux et actes). 2015. Available from: http://tinyurl.com/orw8cw2. [Accessed June 21, 2016].

        • Zorginstituut Nederland
        Procedure voorwaardelijke toelating geneeskundige zorg 2015.
        Zorginstituut Nederland, Diemen, The Netherlands2015
        • Tandvårds- och läkemedelsförmånsverket
        The Swedish Pharmaceutical Reimbursement System.
        Tandvårds- och läkemedelsförmånsverket, Stockholm, Sweden2007
      12. National Health Service England. The Cancer Drugs Fund: transition to the new model. Available from: https://www.england.nhs.uk/ourwork/cancer/cdf/#. [Accessed June 21, 2016].

        • Mestre-Ferrandiz J.
        • Deverka P.
        • Pistollato M.
        • Rosenberg E.
        The Current Drug Development Paradigm: Responding to US and European Demands for Evidence of Comparative Effectiveness and Relative Effectiveness.
        Office of Health Economics, London2014
      13. The Academy of Medical Sciences. Real world evidence: summary of a joint meeting held on 17 September 2015 by the Academy of Medical Sciences and the Association of the British Pharmaceutical Industry. 2016. Available from: https://www.acmedsci.ac.uk/viewFile/56cab22108cf9.pdf. [Accessed July 15, 2016].

        • Guyatt G.H.
        • Oxman A.D.
        • Vist G.E.
        • et al.
        • GRADE Working Group
        GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.
        BMJ. 2006; 336: 924-926
        • Geonnotti K.
        • Wang W.
        • Peikes D.
        Using Pragmatic Clinical Trials to Test the Effectiveness of Patient-Centered Medical Home Models in Real-World Settings.
        Agency for Healthcare Research and Quality, Rockville, MD2013
        • Delitto A.
        Pragmatic clinical trials: implementation opportunity, or just another fad?.
        Phys Ther. 2016; 96: 137
        • Loudon K.
        • Treweek S.
        • Sullivan F.
        • et al.
        The PRECIS-2 tool: designing trials that are fit for purpose.
        BMJ. 2015; 350: h2147
        • Lipska I.
        • Hoekman J.
        • McAuslane N.
        • et al.
        Does conditional approval for new oncology drugs in Europe lead to differences in health technology assessment decisions?.
        Clin Pharmacol Ther. 2015; 98: 489-491
        • Hoekman J.
        • Boon W.P.
        • Bouvy J.C.
        • et al.
        Use of the conditional marketing authorization pathway for oncology medicines in Europe.
        Clin Pharmacol Ther. 2015; 98: 534-541
        • Kaaniche A.
        • Troubat A.
        • Sherwood A.
        Does conditional marketing authorisation influence market access in France, England, and Germany?.
        Value Health. 2015; 18: A559
        • Freemantle N.
        • Strack T.
        Real-world effectiveness of new medicines should be evaluated by appropriately designed clinical trials.
        J Clin Epidemiol. 2010; 63: 1053-1058
      14. Association of the British Pharmaceutical Industry. Demonstrating value with real world data: a practical guide. 2015. Available from: http://www.abpi.org.uk/our-work/library/guidelines/Pages/real-world-data.aspx. [Accessed April 24, 2015].

      15. Foltz D, Ferrara L, Volkommer R, Turisco F. Real-World Data Research: A case for action. 2013. Computer Sciences Corporation.

        • Berger M.L.
        • Lipset C.
        • Gutteridge A.
        • et al.
        Optimizing the leveraging of real-world data to improve the development and use of medicines.
        Value Health. 2015; 18: 127-130
      16. European network for Health Technology Assessment. Position paper on how to best formulate research recommendations for primary research arising from HTA reports. Available from: http://eunethta.eu/sites/5026.fedimbo.belgium.be/files/EUnetHTA%20Position%20Paper%20on%20research%20recommendations.pdf. [Accessed July 15, 2016].

      17. European network for Health Technology Assessment. Position paper on how to decide on the appropriate study design for primary research arising from HTA reports. Available from: http://eunethta.eu/sites/5026.fedimbo.belgium.be/files/EUnetHTA%20Position%20Paper%20on%20study%20design_0.pdf. [Accessed July 15, 2016].

      18. European network for Health Technology Assessment. Core protocol pilot for additional evidence generation. Available from: http://www.eunethta.eu/sites/5026.fedimbo.belgium.be/files/news-attachments/eunethta_core_protocol_pilot_for_aeg.pdf. [Accessed July 15, 2016].