Advertisement

Conditional Financing of Drugs in the Netherlands: Past, Present, and Future—Results From Stakeholder Interviews

      Highlights

      • It is long known that managed entry agreements (MEAs) may offer decision makers valuable policy options to share risks associated with the reimbursement of pharmaceutical drugs, whereby uncertainties may be present regarding their clinical effectiveness or cost effectiveness. Nevertheless, little is known on whether such MEAs deliver on their potential benefits in practice. This is partly due to a lack of systematic evaluations of (previous or existing) MEAs in published literature.
      • This article presents a systematic evaluation of stakeholders' perspectives on the conditional financing (CF) of drugs in the Netherlands, one of the first MEAs in Europe. Several aspects are assessed, including the functioning of CF in practice, improvement points for CF, and recommendations for future policy developments. In doing so, the article aims to inform ongoing international discussions on the design and implementation of future MEA schemes.
      • Agreement exists among stakeholders on the shortcomings in the functioning of CF, the positive impact of CF on Dutch healthcare, and improvement points for CF. Despite the belief that CF either did not meet its aims or only partially did so, agreement exists on the need for new policy schemes to address the same aims in the future. The learnings elicited from the Dutch example may provide valuable input for the design of better MEA schemes in the future.

      Abstract

      Background

      Conditional financing (CF) of hospital drugs was implemented in the Netherlands as a form of managed entry agreements between 2006 and 2012. CF was a 4-year process comprising 3 stages: initial health technology assessment of the drug (T = 0), conduct of outcomes research studies, and reassessment of the drug (T = 4).

      Objectives

      To analyze stakeholder experiences in implementing CF in practice.

      Methods

      Public and private stakeholders were approached for participation in stakeholder interviews through standardized email invitations. An interview guide was developed to guide discussions that covered the following topics: perceived aims of CF, functioning of CF, impact of CF, and conclusions and future perspectives. Extensive summaries were generated for each interview and subsequently used for directed content analysis.

      Results

      Thirty stakeholders were interviewed. Differences emerged among the stakeholders on the perceived aims of CF. Conversely, there was some agreement among stakeholders on the shortcomings in the functioning of CF, the positive impact of CF on the Dutch healthcare setting, and improvement points for CF.

      Conclusions

      Despite stakeholders' belief that CF either did not meet its aims or only partially did so, there was agreement on the need for new policy to address the same aims of CF in the future. Nevertheless, stakeholders diverged on whether CF should be improved on the basis of learnings identified and reintroduced into practice or replaced with new policy schemes.

      Keywords

      Introduction

      Provided that healthcare budgets are finite, decision makers face difficult questions regarding the allocation of resources within the healthcare system. According to the Organisation for Economic Co-operation and Development, pharmaceutical drug expenditure accounts for an average of 16.9% of total healthcare expenditures across 31 member countries; in some countries it exceeds 50% of expenditures.
      OECD data: pharmaceutical spending. Organisation for Economic Co-operation and Development.
      Literature also alludes to increased drug expenditure in the future, partly because of an increased trend in the emergence of innovative, yet expensive, drugs.
      • Iyn-Hyang L.
      • Karen B.
      • Catherine H.
      • et al.
      International experience in controlling pharmaceutical expenditure: influencing patients and providers and regulating industry—a systematic review.
      Consequently, policy makers have been attempting to control drug expenditure through various policy instruments (eg, preference systems for generic drugs or co-payment mechanisms).
      • Iyn-Hyang L.
      • Karen B.
      • Catherine H.
      • et al.
      International experience in controlling pharmaceutical expenditure: influencing patients and providers and regulating industry—a systematic review.
      • Godman B.
      • Shrank W.
      • Wettermark B.
      • et al.
      Use of generics: a critical cost containment measure for all healthcare professionals in Europe?.
      One policy instrument comprises managed entry agreements (MEAs). Briefly defined, MEAs are “arrangements between drug manufacturers and payers or providers that ensure access to coverage or reimbursement of a drug or medical technology under specified conditions.”
      • Carlson J.J.
      • Sullivan S.D.
      • Garrison L.P.
      • et al.
      Linking payment to health outcomes: a taxonomy and examination of performance-based reimbursement schemes between healthcare payers and manufacturers.
      Several forms of MEAs exist, each addressing different policy questions. One form, coverage with evidence development (CED) schemes, includes mechanisms to address uncertainties in clinical effectiveness and/or cost effectiveness of drugs through (real-world) evidence generation.
      • Carlson J.J.
      • Sullivan S.D.
      • Garrison L.P.
      • et al.
      Linking payment to health outcomes: a taxonomy and examination of performance-based reimbursement schemes between healthcare payers and manufacturers.
      A notable advantage of CED schemes seems to be their capability to resolve the dilemma between quick patient access to drugs and the collection of additional data to resolve uncertainties in the evidence base. Nevertheless, it remains questionable whether they can deliver on their promises in practice.
      • Garrison Jr., L.P.
      • Carlson J.J.
      • Bajaj P.S.
      • et al.
      Private sector risk-sharing agreements in the United States: trends, barriers, and prospects.
      • Navarria A.
      • Drago V.
      • Gozzo L.
      • et al.
      Do the current performance-based schemes in Italy really work? “Success fee”: a novel measure for cost-containment of drug expenditure.
      In 2005, public outcry in the Netherlands ensued because of unequal access to the then innovative, yet expensive, drug trastuzumab.
      • Klappe-Sabadi G.
      • Jansman F.G.A.
      • Honkoop H.A.
      • et al.
      Regionale verschillen in voorschrijfgedrag trastuzumab opnieuw getoetst.
      Inequality in access led to so-called ZIP code healthcare, whereby patient access varied from 25% in some provinces to 75% in others.
      • Klappe-Sabadi G.
      • Jansman F.G.A.
      • Honkoop H.A.
      • et al.
      Regionale verschillen in voorschrijfgedrag trastuzumab opnieuw getoetst.
      Between 2006 and 2012, the Netherlands Healthcare Authority (Nederlandse Zorgautoriteit [NZa]) devised 2 policy frameworks to facilitate conditional financing (CF) of expensive and orphan drugs in hospitals, respectively, from the national healthcare insurance package (henceforth reimbursement package).
      • Nederlandse Zorgautoriteit
      BELEIDSREGEL BR/CU-2017 Dure Geneesmiddelen.
      The implementation of these frameworks in the form of a CED scheme was subsequently delegated to the National Healthcare Institute (Zorginstituut Nederland [ZIN]; formerly, College voor Zorgverzekeringen), the national health technology assessment (HTA) agency. Drugs qualifying for CF had to meet 3 criteria: have a budget impact higher than €2.5 million per year, have a proven added therapeutic value, and there needed to be uncertainties regarding appropriate use and/or cost effectiveness of the drugs in Dutch clinical practice.
      • Zorginstituut Nederland, mw.dr.G.Ligtenberg
      Voorwaardelijke toelating/financiering van zorg.
      The CF process comprised 3 main stages: initial HTA (T = 0), conduct of outcomes research, and reassessment (T = 4) (see Fig. 1). Various stakeholders were involved at each phase of the process. For example, ZIN was responsible for the assessment of evidence submitted at T = 0 and T = 4 and for providing feedback on outcomes research proposals at T = 0. Meanwhile, the marketing authorization holder was responsible for preparing submissions for T = 0 and T = 4 and submitting an outcomes research study proposal to address uncertainties identified at T = 0. Other stakeholders involved in CF included public policy bodies (eg, NZa), healthcare insurers, medical specialists societies, academic/private hospitals, and patient organizations. Please see Appendix Table 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2018.11.016 for the roles of different stakeholders throughout the CF process.
      Figure thumbnail gr1
      Fig. 1Process chart for CF as implemented by ZIN in the Netherlands. CF indicates conditional financing; ZIN, Zorginstituut Nederland.
      The first T = 0 assessments were published as early as May 2006.
      • Makady A.
      • van Veelen A.
      • de Boer A.
      • et al.
      Implementing managed entry agreement in practice: the Dutch reality check.
      Meanwhile, the last drugs were included for T = 0 assessments in 2012.
      • Makady A.
      • van Veelen A.
      • de Boer A.
      • et al.
      Implementing managed entry agreement in practice: the Dutch reality check.
      Despite being one of the first MEAs implemented in Europe, no policy evaluation of CF has been conducted since the inclusion of the last drugs in 2012. HTA dossiers produced at T = 0 and T = 4 for all CF drugs were recently analyzed to assess procedural, methodological, and decision-making aspects of the scheme.
      • Makady A.
      • van Veelen A.
      • de Boer A.
      • et al.
      Implementing managed entry agreement in practice: the Dutch reality check.
      The present study aimed to evaluate stakeholders' experiences in implementing CF in practice.

      Methods

       Data Collection

      In the first phase, data were collected from public organizations involved in designing and/or implementing policy. These stakeholders were the NZa, the Ministry of Health (Ministerie voor Volksgezondheid, Welzijn en Sport), the Netherlands Organization for Health Research and Development (De Nederlandse organisatie voor gezondheidsonderzoek en zorginnovatie), members of the scientific assessment committee of ZIN (Wetenschappelijk Adviesraad; henceforth, Assessment Committee), members of the Insurance Package Advisory Committee of ZIN (Adviescommissie Pakket; henceforth, Appraisal Committee), senior advisers at ZIN (eg, the secretariat of drug assessors), and pharmacotherapeutic assessors and pharmacoeconomic assessors at ZIN. In the second phase, data were collected from the remaining stakeholders involved in CF, namely, pharmaceutical industry, healthcare insurers, medical specialists societies, academic/private hospitals, and patient organizations.
      The authors used purposeful and snowballing sampling to select stakeholders to approach for participation.
      • Palinkas L.A.
      • Horwitz S.M.
      • Green C.A.
      • et al.
      Purposeful sampling for qualitative data collection and analysis in mixed method implementation research.
      The specific stakeholder representatives approached were sampled on the basis of seniority and function, with a preference for senior representatives with a history of direct involvement in CF. All stakeholder representatives were approached through a standardized email invitation (see Appendix Figure 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2018.11.016). Data saturation was discussed among authors and provided grounds for determining the final number of interviews conducted.
      An interview guide was developed for stakeholder interviews. The guide covered the following topics:
      • Perceived aims of CF (ie, which purpose it served);
      • Perceived functioning of CF (ie, in relation to procedural, methodological, and decision-making aspects; definitions for these aspects correspond to those in the study on HTA dossiers
        • Makady A.
        • van Veelen A.
        • de Boer A.
        • et al.
        Implementing managed entry agreement in practice: the Dutch reality check.
        );
      • Impact of the CF scheme (ie, its positive and negative effects on the Dutch healthcare setting);
      • Conclusions and future perspectives (ie, whether CF achieved its aims, improvement points for CF, and whether CF-like schemes should be stopped, reintroduced, or replaced).
      The interview guide included open- and close-ended questions. See Appendix Figure 2 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2018.11.016 for the interview guide used.
      It is important to note that when the perceived aims of CF provided by interviewees differed from the aims of the CF scheme as described in the introduction, the interviewers subsequently iterated that the aim was 2-fold, specifically to strike a balance between quick patient access to drugs and the promise for additional evidence generation. This was done to avoid any potential influence of differences in perceived aims on the remaining topics of the interview guide.
      A preference was made for face-to-face interviews. If these were infeasible within project timelines, telephone interviews were held. Stakeholders were asked whether interviews could be audio-recorded. Field notes were also taken during the interviews. Two reviewers conducted phase 1 interviews between July 4, 2016, and November 6, 2016, and phase 2 interviews between March 24, 2017, and May 10, 2017.
      On the basis of audio recordings and/or field notes, extensive summaries were made. The summaries were sent to interviewees for a member check and were subsequently edited on the basis of the feedback received and sent to the interviewees for final approval.

       Data Analysis

      Directed content analysis was conducted on the extensive summaries generated using MaxQDA software version 11.0 (VERBI Software GmbH, Berlin, Germany).
      • Hsieh H.F.
      • Shannon S.E.
      Three approaches to qualitative content analysis.
      The empty coding tree was structured to reflect the topics of the aforementioned interview guide. Two of the authors conducted the content analysis for phase 1 and phase 2 interviews in November 2016 and May 2017, respectively. Each author coded half the interview summaries and reviewed the other author's coding for the remaining summaries. Any discrepancies in codes generated were resolved by consensus. Finally, the separate coding trees generated by the analysis of phase 1 and 2 interviews were combined in August 2017.
      Because of the large number of codes generated for open-ended questions for 3 topics (perceived functioning of CF, impact of CF, and conclusions and future perspectives), the authors selected the codes mentioned by at least a quarter of the stakeholders (≥25%) for further descriptive analyses. Illustrative quotes were cited to clarify the meaning of the themes included.
      The methods used for this study were compared with the COnsolidated criteria for REporting Qualitative research 32-item checklist (see Appendix Table 2 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2018.11.016).
      • Tong A.
      • Sainsbury P.
      • Craig J.
      Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups.
      The methods met all necessary criteria as stipulated in this checklist.

      Results

       Study Sample

      Stakeholders approached in phase 1 comprised representatives from non-ZIN public bodies (n = 3), the Assessment Committee (n = 2), the Appraisal Committee (n = 3), senior advisers at ZIN (n = 4), pharmacotherapeutic assessors (n = 4), and pharmacoeconomic assessors (n = 2). Stakeholders approached in phase 2 comprised representatives from pharmaceutical companies (n = 5), healthcare insurers (n = 3), medical specialists societies (n = 3), academic/private hospitals (n = 3), and patient organizations (n = 3). All representatives approached agreed to participate in the interviews (response rate 100%). Eventually, 35 representatives spanning 30 stakeholders were included.
      Thirty interviews were conducted: 14 for phase 1 and 16 for phase 2. Three interviews included 2 or more interviewees. Twenty-five interviews were held face-to-face and 5 over the telephone. Each interview lasted between 60 and 90 minutes. Audio recordings were made for 29 interviews; 1 stakeholder refused to have the interview recorded. For a summary of the study sample, see Table 1. For the full coding tree developed, see Appendix Figure 3A-D in Supplemental Materials found at https://doi.org/10.1016/j.jval.2018.11.016.
      Table 1Summary of the stakeholders interviewed.
      Stakeholder groupStakeholderNumber of intervieweesInterview codeDate of interviewManner of interviewInterview recorded (yes/no)
      External public bodiesNederlandse Zorgautoriteit1PE1July 18, 2016Face-to-faceYes
      Ministerie voor Volksgezondheid, Welzijn en Sport1PE2July 4, 2016Face-to-faceYes
      De Nederlandse organisatie voor gezondheidsonderzoek en zorginnovatie1PE3July 20, 2016Face-to-faceYes
      ZIN, Assessment CommitteeCommittee member1ZW1July 20, 2016TelephoneYes
      Committee member1ZW2July 11, 2016Face-to-faceYes
      ZIN, Appraisal CommitteeCommittee member1ZA1September 7, 2016Face-to-faceYes
      Committee member1ZA2July 12, 2016Face-to-faceYes
      Committee member1ZA3July 14, 2016Face-to-faceYes
      ZIN, senior advisersSenior adviser1ZS1September 6, 2016Face-to-faceYes
      Senior adviser1ZS2September 1, 2016Face-to-faceYes
      Senior adviser1ZS3August 31, 2016Face-to-faceYes
      Senior adviser1ZS4August 30, 2016Face-to-faceYes
      ZIN, drug assessorsPharmacotherapeutic assessors4FG1October 27, 2016Face-to-faceYes
      Pharmacoeconomic assessors2FG2November 16, 2016Face-to-faceYes
      Pharmaceutical industryJanssen Pharmaceuticals BV1PI1April 13, 2017Face-to-faceYes
      Novartis Pharma BV2PI2April 19, 2017TelephoneNo
      Bristol-Myers Squibb Pharmaceuticals BV1PI3April 19, 2017Face-to-faceYes
      Vereniging voor Innovatieve Geneesmiddelen1PI4April 20, 2017Face-to-faceYes
      Healthcare insurersZorgverzekeringen VGZ1HI1May 1, 2017TelephoneYes
      Menzis1HI2April 4, 2017TelephoneYes
      Zorgverzekeraars Nederland1HI3April 24, 2017Face-to-faceYes
      Medical specialists societiesStichting Werkgroep Antibioticabeleid1MS1March 24, 2017Face-to-faceYes
      Stichting Onvologische Samenwerking1MS2May 9, 2017Face-to-faceYes
      Integraal Kankercentrum Nederland1MS3May 10, 2017Face-to-faceYes
      Private/academic hospitalsNederlandse Federatie van Universitair Medische Centra1HO1April 21, 2017Face-to-faceYes
      Het Academisch Medisch Centrum1HO2May 3, 2017Face-to-faceYes
      Nederlandse Vereniging van Ziekenhuizen1HO3April 25, 2017TelephoneYes
      Patient organizationsVereniging Volwassen, Kinderen en Stofwisselingsziekten1PO1April 4, 2017Face-to-faceYes
      Nederlandse Federatie voor Kankerpatientorganisaties1PO2May 8, 2017Face-to-faceYes
      Longfonds1PO3April 3, 2017Face-to-faceYes
      ZIN indicates Zorginstituut Nederland.
      Please note that with specific regard to open-ended questions, the authors discuss only a selection of important themes herein. This is due to the large number of themes identified per topic and word-count considerations. For a full list of themes per topic, see Appendix Table 3, and for illustrative quotes per theme, see Appendix Table 4 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2018.11.016.

       Perceived Aims of CF

      Most stakeholders (55%) indicated that the aim of CF was to strike a balance between quick patient access to drugs and the promise for additional evidence generation. Meanwhile, a few (13%) stakeholders indicated that CF only aimed to promote early access to drugs, and a few others (13%) argued that it was merely a mechanism to control healthcare expenditure. Finally, a last group (19%) believed that CF had other aims. For example, 1 stakeholder (interview code HO1 in Table 1) indicated that CF provided a controlled environment for experimenting with drugs in clinical practice, on the basis of clear agreements on treatment criteria. See Figure 2 for an overview of the perceived aims.
      Figure thumbnail gr2
      Fig. 2Stakeholder views on the perceived aim of CF. CF indicates conditional financing.

       Perceived Functioning of CF

       Procedural aspects

      With regard to procedural aspects of CF, most of the stakeholders (90%) indicated to have doubts toward the envisioned 4-year time frame. For example, stakeholders indicated that for some indications (eg, acute diseases), 4 years may be sufficient to collect meaningful data, whereas for other indications (eg, chronic diseases or orphan diseases), a much longer follow-up would be needed (codes MS3 and PI3). Moreover, stakeholders emphasized the extensive time needed to set up registries for data collection (code PI3).
      Several stakeholders (43%) referred to a design flaw in the CF procedure, namely, the disregard of the relationship between the division of roles and conflicting interests of stakeholders. For example, interviewees indicated that pharmaceutical industry and medical specialists, tasked with financing and implementing outcomes research after T = 0, respectively, may not have been intrinsically inclined to collect robust evidence that could indicate that the drugs are not cost-effective (code ZW2). The interviewees subsequently indicated that if the data would lead to that conclusion, there would have been reason to remove the drug from the reimbursement package. In their opinion, this would result in a loss of revenue for industry, whereas for the medical specialist, it would mean that patients would likely stop receiving their treatment (code ZW2). Therefore, stakeholders mentioned that once reimbursement was granted from the reimbursement package at T = 0, the incentives structure to generate evidence drastically shifted among stakeholders (code PE2). Moreover, stakeholders argued that the financing structure for outcomes studies may have had a negative impact on the independence of research conducted (codes PE3, ZA3, and FG1).
      Another theme mentioned by several stakeholders (27%) was the (lack of) mechanisms embedded in CF for the monitoring of progress. For example, stakeholders indicated during interviews that none of the guidelines specified an interim time point for mid-term reviews of progress in the outcomes research studies (eg, at T = 1 or T = 3); ZIN was also not provided the authority to enforce such mid-term reviews (code ZS4). Stakeholders iterated that the lack of monitoring meant that errors encountered at T = 4 (eg, regarding data collection or analysis) could no longer be retrospectively corrected (code HI2).

       Methodological aspects

      According to several stakeholders (40%), there was no clear methodological guidance and/or consensus with regard to the design of outcomes research studies conducted between T = 0 and T = 4. One interviewee emphasized that methodological guidance on study design by ZIN was finalized only in 2008, 2 years after the start of CF (code ZS4). Meanwhile, other interviewees iterated that at the time of development of CF drugs, there was often limited medical knowledge on the disease areas for which CF drugs were developed. Therefore, consensus on core outcome sets that are relevant to the drugs in question was difficult to reach (codes PI1 and ZS3). According to interviewees, such factors often led to an inflated list of parameters for which data needed to be collected that were, in hindsight, of little relevance to the policy question (codes ZW2, ZS3, ZS4, PI1, PI3, and MS3).
      Furthermore, a third of stakeholders (33%) indicated that the quality of outcomes research studies conducted was generally poor. They referred to recurring problems such as the absence of a control group or that the intervention and control groups were not comparable. In the latter case, patients who did not wish to be treated with the new drug automatically became the control group, leading to potential selection bias (code FG1). Other aspects such as low patient recruitment and fragmented data collection in practice also had an impact on study quality (code ZS2).
      Moreover, a third of stakeholders (33%) emphasized the impact of rapid changes in clinical practice on the relevance of evidence generated through outcomes research studies. Oncology was mentioned by interviewees as a primary example of a disease area where new drugs are introduced at a rapid pace. As a result, drugs that may have been due for investigation in second-line treatment at T = 0 became standard first-line treatments within the duration of the outcomes research study (codes PE3, ZS2, FG1, and HI3). Moreover, different combinations of oncology treatments were introduced after study designs for monotherapies were finalized at T = 0 (code PO2).

       Decision-making aspects

      Half of the stakeholders (50%) stated that external factors had a significant effect on the advice issued by ZIN at T = 4. The main examples whereby political pressure played such a role were alglucosidase-α and agalsidase-α and -β for the treatment of Pompe and Fabry diseases, respectively (codes ZS1, FG2, HI3, and HO2).
      Many stakeholders (43%) expressed the opinion that outcomes research studies conducted as part of CF contributed little to decision making at T = 4. In fact, several indicated that uncertainties were rarely diminished at T = 4, particularly with regard to cost-effectiveness analyses (codes PE2 and FG2). In general, this was the result of the methodological limitations of the studies cited earlier (codes FG2, PI2, and MS3) and/or skepticism regarding the use of real-world evidence (RWE) in decision making (code ZA3).
      Another theme referred to by stakeholders (40%) was the impossibility of removing drugs from the reimbursement package at T = 4, even if ZIN recommended to do so. The associated legal implications were often deemed too large to attempt the feat (code PE1). Another stakeholder referred to the fact that their negotiation power and argumentation to discontinue drug reimbursement was highly compromised at T = 4 (code HI3).

       Impact of the CF Scheme

      Several themes were identified in relation to the positive effects of CF in the Dutch healthcare setting. First, more than half of the stakeholders (53%) stated that CF resulted in cost effectiveness of drugs and the displacement of healthcare because of exorbitant drug expenditures becoming topics of societal debate. In other words, awareness was created among all stakeholders (including the general public) on the sustainability of the healthcare system in light of high drug prices (codes PE2, ZA2, HI1, and MS2).
      Second, a third of stakeholders (33%) were of the opinion that CF delivered valuable experiences from a policy perspective (codes ZS3, ZS4, HI1, and HI2). Two stakeholders asserted that learnings from CF have already been applied for the design of ongoing MEAs (Voorwaardelijke Toelating
      • Zorginstituut Nederland, mw.dr.G.Ligtenberg
      Voorwaardelijke toelating/financiering van zorg.
      ) (codes FG2 and HI1) and for the value-based assessment of drugs that came after the CF scheme (eg, eculizumab and pertuzumab) (code ZS3).
      None of the themes identified with respect to the negative effects of CF met the inclusion criterion (ie, were mentioned by <25% of stakeholders).

       Conclusions and Future Perspectives

      When asked whether CF had achieved its perceived aims, half of the stakeholders (50%) answered “No,” half (50%) answered “Partially,” and none (0%) answered “Yes” (see Fig. 3). For those who answered “Partially” (n = 15), 53% indicated that the goal of early patient access to drugs was met, 20% indicated that the goal of (real-world) evidence generation was met, and 27% indicated other aspects (eg, CF fulfilled its aims for specific drugs [code PO3]).
      Figure thumbnail gr3
      Fig. 3Stakeholder views on CF achieving its aims. CF indicates conditional financing.
      Several themes were identified regarding improvement points for the CF scheme. First, about a third of the stakeholders (37%) emphasized the need for consensus on the aims and relevance of the scheme, as well as the importance of interstakeholder collaboration to achieve these aims (codes ZA1, ZA2, PI1, PI2, HI2, and PO3). Second, a third of the stakeholders (33%) emphasized the need for a framework whereby the underlying incentives structure ensures that different stakeholders take up their responsibilities and be held accountable if such responsibilities are not met. For example, stakeholders indicated that CF included no mechanisms to impose sanctions, a fact that had a great impact on outcomes of the scheme (code ZS3). Moreover, they thought that CF drugs should not have been financed from the reimbursement package, but rather from a temporary funding source (code FG1); knowing that drug availability is temporary, stakeholders responsible for data collection would be better incentivized to do so (codes FG1 and PI4). Furthermore, they indicated that conditions of obligatory inclusion of patients in outcomes research in return for access to the drug should have been considered (codes FG1 and ZA1). In CF, patient inclusion was done on a voluntary basis, leading to many underpowered studies and selection bias (code FG1).
      When asked how to proceed with CF in the future, 37% of stakeholders suggested to replace CF with a scheme that resembles adaptive pathways, a scientific concept for medicine development and data generation whereby an iterative approach to evidence generation is adopted for drugs throughout their lifetime.
      Adaptive pathways. European Medicines Agency.
      • Bouvy J.C.
      • Jonsson P.
      • Longson C.
      • et al.
      Health technology assessment in the context of adaptive pathways for medicines in Europe: challenges and opportunities.
      Meanwhile, 30% suggested to replace CF with other policies such as adaptive pricing or the use of electronic health records (EHRs) for evidence generation. Other stakeholders (27%) suggested to improve CF on the basis of the points mentioned earlier during interviews then subsequently reintroducing it. Finally, a few stakeholders (7%) suggested to stop all forms of CEDs; in their opinion, such schemes do not work in practice (codes ZA3 and HI3). See Figure 4 for views on future perspectives in relation to CF.
      Figure thumbnail gr4
      Fig. 4Stakeholder views on future steps. CF indicates conditional financing; MEA, managed entry agreement.

      Discussion

      This study examined experiences of stakeholders in implementing CF in Dutch practice. Results indicated that stakeholders had different perceptions of the aims of the CF scheme. Moreover, stakeholders highlighted numerous shortcomings in how the CF scheme functioned with regard to procedural, methodological, and decision-making aspects (eg, the 4-year time frame, methodological quality of outcomes research studies, and external political influence on advice at T = 4, respectively). In contrast to this, stakeholders mentioned several positive effects of CF (eg, public discourse on cost effectiveness of drugs and displacement of healthcare). Half of the stakeholders thought that CF had partially achieved its aims, whereas the other half believed it had not. Most of the stakeholders indicated that CF should either be replaced with a new policy or be improved and reintroduced.
      Some of the findings summarized here correspond to those from the first study on HTA dossiers analysis.
      • Makady A.
      • van Veelen A.
      • de Boer A.
      • et al.
      Implementing managed entry agreement in practice: the Dutch reality check.
      For example, stakeholders' critique on the 4-year time frame for CF being too short coincided with findings from dossiers indicating that only 1 CF drug was completed within the envisioned period. Moreover, stakeholders indicated that outcomes research studies were often of low methodological quality and thus of little relevance to decision making. Meanwhile, the dossiers analysis indicated that the studies provided inadequate evidence for almost half of the research questions on cost effectiveness. Finally, stakeholders' emphasis of the impact of external factors on decision making at T = 4 corresponds to findings from the dossiers analysis, indicating that only a couple of CF drugs eventually received a negative reimbursement advice at T = 4.
      Healthcare systems worldwide include a wide array of different stakeholders, each with their differing mandates and a complex network of interactions among them. As a result, MEAs present different trade-offs for each stakeholder in relation to their specific interests. Consequently, from a governance perspective, there is a critical need for clear frameworks that entail stakeholders' roles, responsibilities, incentives, and sanctions.
      • Bevir M.
      Governance: A Very Short Introduction.
      • Duit A.
      • Galaz V.
      Governance and complexity: emerging issues for governance theory.
      To begin with, stakeholders' perceptions of the scheme aims, and thus their own gains, greatly matter; in Germany, similar schemes failed because of clinicians perceiving them as posing limitations on their prescribing choices.
      Opinion on the “cost-sharing initiatives” and “risk-sharing agreements” between pharmaceutical manufacturers and health and hospital. Drug Commission of the German Medical Association.
      Meanwhile in Italy, it still remains nearly impossible to reclaim costs for nonresponder patients from pay-for-performance schemes,
      • Navarria A.
      • Drago V.
      • Gozzo L.
      • et al.
      Do the current performance-based schemes in Italy really work? “Success fee”: a novel measure for cost-containment of drug expenditure.
      possibly because of the absence of mechanisms to impose sanctions on responsible parties. Previous experiences from England also point to problems arising from the absence of “exit strategies.”
      • Faulkner S.D.
      • Lee M.
      • Qin D.
      • et al.
      Pricing and reimbursement experiences and insights in the European Union and the United States: lessons learned to approach adaptive payer pathways.
      Although such concepts on governance may seem quite elementary, their importance cannot be underestimated, provided their recurrence in countries with notably different healthcare system structures.
      In particular, the implementation of CEDs poses additional challenges relating to infrastructure for (real-world) data collection and subsequent data analysis for decision making. The current model for creating ad hoc product or disease registries for separate research questions may be unsustainable because of various reasons, including costs, administrative burden of extra data registration, and data accessibility for research.
      • Garrison Jr., L.P.
      • Carlson J.J.
      • Bajaj P.S.
      • et al.
      Private sector risk-sharing agreements in the United States: trends, barriers, and prospects.
      • Carlson J.J.
      • Gries K.S.
      • Yeung K.
      • et al.
      Current status and trends in performance-based risk-sharing arrangements between healthcare payers and medical product manufacturers.
      • Makady A.
      • Stegenga H.
      • Ciaglia A.
      • et al.
      Practical implications of using real-world evidence (RWE) in comparative effectiveness research: learnings from IMI-GetReal.
      Meanwhile, major investments are needed to establish systems for data collection and analysis, whether through paper-based questionnaires or EHRs.
      • Navarria A.
      • Drago V.
      • Gozzo L.
      • et al.
      Do the current performance-based schemes in Italy really work? “Success fee”: a novel measure for cost-containment of drug expenditure.
      • Faulkner S.D.
      • Lee M.
      • Qin D.
      • et al.
      Pricing and reimbursement experiences and insights in the European Union and the United States: lessons learned to approach adaptive payer pathways.
      In light of stakeholders' comments mentioned earlier on the financing of outcomes research studies, it would be difficult to specify which stakeholders should be responsible for financing the establishment of information technology infrastructures for implementing EHRs. Even with the necessary infrastructures in place, healthcare professionals in clinical practice would need to be trained to use such information technology systems, requiring financial and time investments on their behalf. Provided the high workload experienced by healthcare professionals in general, it may be difficult to commit to such investments. Finally, the availability of data within EHRs does not automatically guarantee access to data for analysis purposes, as illustrated by numerous examples in literature.
      • Makady A.
      • Stegenga H.
      • Ciaglia A.
      • et al.
      Practical implications of using real-world evidence (RWE) in comparative effectiveness research: learnings from IMI-GetReal.
      • van Staa T.P.
      • Goldacre B.
      • Gulliford M.
      • et al.
      Pragmatic randomised trials using routine electronic health records: putting them to the test.
      Clinical Trials Transformation Initiative
      Registry Trials Project Multi-Stakeholder Expert Meeting.
      Another important challenge is the analysis of real-world data (RWD) and interpretation of RWE. The aforementioned findings allude to skepticism among decision makers in basing decisions on RWE. Furthermore, numerous articles refer to the methodological difficulties associated with analyzing RWD and using RWE in decision making.
      • Blommestein H.M.
      • Franken M.G.
      • Uyl-de Groot C.A.
      A practical guide for using registry data to inform decisions about the cost effectiveness of new cancer drugs: lessons learned from the PHAROS registry.
      • Mohseninejad L.
      • van Gils C.
      • Uyl-de Groot C.A.
      • et al.
      Evaluation of patient registries supporting reimbursement decisions: the case of oxaliplatin for treatment of stage III colon cancer.
      • Alemayehu D.
      • Ali MPP Riaz
      • Alvir J.M.
      • et al.
      Examination of data, analytical issues and proposed methods for conducting comparative effectiveness research using “real-world data”.
      From a methodological perspective, many advances have been made in the analysis of RWD, both alone or in combination with randomized controlled trial data.
      • Klungel O.H.
      • Martens E.P.
      • Psaty B.M.
      • et al.
      Methods to assess intended effects of drug treatment in observational studies are reviewed.
      • Garbe E.
      • Suissa S.
      Pharmacoepidemiology.
      • Efthimiou O.
      • Mavridis D.
      • Debray T.
      • et al.
      Combining randomized and non-randomized evidence in network meta-analysis.
      Moreover, recent guidelines issued by the combined efforts of the International Society for Pharmacoeconomics and Outcomes Research and the International Society for Pharmacoepidemiology provide an example of clear guidance on good procedures for the conduct and reporting of real-world studies to increase decision makers' confidence in RWE.
      • Berger M.L.
      • Sox H.
      • Willke R.J.
      • et al.
      Good practices for real-world data studies of treatment and/or comparative effectiveness: recommendations from the joint ISPOR-ISPE Special Task Force on real-world evidence in health care decision making.
      • Wang S.V.
      • Schneeweiss S.
      • Berger M.L.
      • et al.
      Reporting to improve reproducibility and facilitate validity assessment for healthcare database studies V1.0.
      Nevertheless, implementing state-of-the-art methodology for RWD analysis requires extensive training in pharmacoepidemiology and biostatistics, implying yet again the need for considerable investments for the training of personnel conducting the analyses (eg, pharmaceutical industry) or interpreting the results (eg, HTA agencies). As a consequence of factors discussed earlier, decision makers in both public and private stakeholders still have little experience in incorporating RWE in current processes.
      • Makady A.
      • Stegenga H.
      • Ciaglia A.
      • et al.
      Practical implications of using real-world evidence (RWE) in comparative effectiveness research: learnings from IMI-GetReal.
      Despite the challenges, most stakeholders encouraged the development of new CEDs to address dilemmas encountered in decision making on the reimbursement of drugs. Literature also alludes to increasing trends in conditional marketing authorizations issued by regulatory authorities (eg, the European Medicines Agency [EMA]) with relatively larger uncertainties in evidence for HTA (particularly for oncology and orphan drugs)
      • Kleijnen S.
      • Lipska I.
      • Alves T.L.
      • et al.
      Relative effectiveness assessments of oncology medicines for pricing and reimbursement decisions in European countries.
      and increasing trends in MEA use.
      • Garrison Jr., L.P.
      • Carlson J.J.
      • Bajaj P.S.
      • et al.
      Private sector risk-sharing agreements in the United States: trends, barriers, and prospects.
      • Carlson J.J.
      • Gries K.S.
      • Yeung K.
      • et al.
      Current status and trends in performance-based risk-sharing arrangements between healthcare payers and medical product manufacturers.
      Recent EMA initiatives on adaptive pathways and guidelines on postauthorization effectiveness studies
      Adaptive pathways. European Medicines Agency.
      coincide well with HTA agencies' efforts to collaborate on RWE generation and interpretation for decision making in the context of uncertainties. The latter include efforts of the European Network for Health Technology Assessment on additional evidence generation in both pre- and postauthorization stages.
      Work Package 5—Life cycle approach to improve Evidence Generation. European Network for Health Technology Assessment.
      In light of this, one can argue that adaptive pathways and CEDs provide impetus for rigorous collaboration between regulatory and HTA agencies on aspects of RWE generation and use in decision making. In fact, this collaboration has recently been formalized in the EMA-European Network for Health Technology Assessment work plan.
      EMA-EUnetHTA three-year work plan: 2017-2020. European Network for Health Technology Assessment and European Medicines Agency.
      It is our hope that such initiatives will inform the development of more coherent systems for CEDs, whereby the outputs of the regulatory pathway include a clear RWE generation pathway that meets the needs of HTA.
      Bearing in mind the results from this research and additional points discussed earlier, we present 10 key recommendations for the design of improved CED schemes in the future (Table 2). These recommendations pertain to several aspects of CED schemes, including which stakeholders to involve, governance mechanisms within the schemes, and good procedural practice when conducting RWE studies. Although these recommendations are predominantly based on the findings of this study and current literature, we envision ongoing discussions and improvements of these recommendations in the future.
      Table 2Key recommendations for the design of CED schemes.
      No.Recommendations
      1Ensure the involvement of all relevant stakeholders in discussions related to the design of the CED scheme and associated outcomes research study (ie, MAHs, regulatory and HTA agencies, medical specialists, and patient organizations).
      2Tailor the duration of the CED scheme and outcomes research study to the evidentiary needs regarding the nature of the disease (eg, chronic disease vs acute disease), the nature of the intervention (ie, chronic treatment vs acute treatment), and the nature of the surrogate and hard endpoints within the disease indication. A static framework of a specific number of years (eg, 4 y) does not apply to all interventions in all indications.
      3Ensure good procedural practice for the design and conduct of outcomes research studies through a priori publication of the study protocol, the proposed PICO framework for the study, the data analysis methods, and systematic reporting of study findings. Recommendations of the ISPOR-ISPE special task force reports on RWE studies provide comprehensive guidance on this topic.
      4Restrict additional data collection efforts to a minimal set of core outcomes and parameters. This reduces administrative burden for practitioners and increases chances of generating more complete data sets, which are relevant for (HTA) decision making.
      5To avoid issues of selection bias and provided the societal costs of highly expensive (orphan) interventions, make inclusion of patients receiving the intervention in outcomes research studies obligatory.
      6Bear in mind the underlying interests of different stakeholders when allocating responsibilities for patient recruitment, data collection, evidence generation, and data ownership. For example, pharmaceutical industry should not be given the sole responsibility to ensure data collection in clinical practice and evidence generation, particularly in situations where such evidence may lead to disinvestment decisions.
      7Jointly agree to the consequences when stakeholders do not meet responsibilities as outlined in the CED scheme. Accountability in such cases is a critical condition for the success of CEDs. For example, one of the main weaknesses of the Dutch CF framework was the absence of mechanisms to impose sanctions.
      8Embed monitoring mechanisms within the CED scheme, whereby stakeholders can periodically evaluate progress made on the outcomes research study (eg, patient recruitment, data collection, and quality of evidence generated). On the basis of the periodic assessments, joint decisions can be made on the continuation or termination of the CED scheme.
      9Avoid reimbursing interventions that have been allocated to CED schemes from the national healthcare package (or the equivalent thereof in different nations). The reimbursement of interventions in CEDs should come from a temporary (governmental) fund, with a clear emphasis on the temporary nature of reimbursement until the end of the allocated period for the CED. The potential to shift reimbursement of the intervention from the temporary fund to permanent funding would then be conditional on the outcomes of the CED.
      10Avoid coupling the outcomes of CEDs to binary decisions (ie, full reimbursement from the national healthcare package vs full removal from the national healthcare package). Provided the intervention has a proven added clinical benefit (eg, as confirmed by regulatory agencies and initial HTA assessments), other intermediate solutions could include pay-for-performance arrangements, adaptive pricing, and price renegotiations at later stages. In this context, discussions on the extent of proven effectiveness and cost effectiveness of interventions, as well as discussions on pricing of interventions, would be linked to the outcomes of the outcomes research study as agreed to by all stakeholders.
      CED indicates coverage with evidence development; CF, conditional financing; HTA, health technology assessment; ISPOR, International Society for Pharmacoeconomics and Outcomes Research; ISPE, International Society for Pharmacoepidemiology; MAH, marketing authorization holder; PICO, patient, intervention, comparator, and outcome; RWE, real-world evidence.

       Study Limitations

      Although all relevant stakeholder groups were involved, we could not include all individual stakeholders involved with CF in the interviews for this study. Nevertheless, the authors used several sampling methods to ensure that a comprehensive range of stakeholders were included, spanning different stakeholder groups. Moreover, data saturation was discussed among the authors and provided grounds for limiting the number of interviews.
      The threshold implemented to select and include themes from content analysis (≥25% of stakeholders) is not standard. The authors are, however, not aware of the existence of standard thresholds for such criteria in qualitative methods. Moreover, illustrative quotes cited in the Appendix in Supplemental Materials cover additional themes that may not have met the 25% threshold.
      Finally, this study represents a policy evaluation of a national CED. Ideally, the scope of this study would include MEAs implemented in other countries (eg, Italy,
      • Navarria A.
      • Drago V.
      • Gozzo L.
      • et al.
      Do the current performance-based schemes in Italy really work? “Success fee”: a novel measure for cost-containment of drug expenditure.
      France,
      • Makady A.
      • ten Ham R.
      • de Boer A.
      • et al.
      Policies for use of real-world data in health technology assessment (HTA): a comparative study of six HTA agencies.
      Sweden,
      • Makady A.
      • ten Ham R.
      • de Boer A.
      • et al.
      Policies for use of real-world data in health technology assessment (HTA): a comparative study of six HTA agencies.
      the United States,
      • Garrison Jr., L.P.
      • Carlson J.J.
      • Bajaj P.S.
      • et al.
      Private sector risk-sharing agreements in the United States: trends, barriers, and prospects.
      and the United Kingdom
      • Faulkner S.D.
      • Lee M.
      • Qin D.
      • et al.
      Pricing and reimbursement experiences and insights in the European Union and the United States: lessons learned to approach adaptive payer pathways.
      ). Nevertheless, the placement of the authors within Dutch institutions provided extensive access to national stakeholders, thus allowing for a thorough, systematic analysis of CF. Such access may not be equally facilitated in other settings. Provided the complexity of designing and implementing CEDs, we therefore encourage further research on experiences gained in the implementation of MEAs (including CEDs) in other countries to provide complementary learnings for the design of future schemes. Moreover, such studies may shed valuable insights on potential correlations between the successes and failures of MEAs and the structures of healthcare systems in which they are embedded.

      Conclusions

      This study provides insights on stakeholders' experiences in implementing CF in Dutch practice, an example of MEAs (namely, a CED scheme). Results demonstrate differences among the stakeholders on the perceived aims of CF. Conversely, there is some agreement among stakeholders on the shortcomings in the functioning of CF (ie, relating to procedural, methodological, and decision-making aspects), the positive impact of CF on the Dutch healthcare setting, and improvement points for CF. Despite the belief that CF only partially met its aims, if not at all, there is still agreement on the need for either new policy schemes or an improved version of CF to address the same aims in the future.
      This study was conducted with the aim of informing ongoing international discussions on the design and implementation of future MEA schemes. Provided the onslaught of innovative, yet expensive, drugs and increasing trends of MEA use by HTA agencies and payers, further research on experiences gained with other MEAs is critical to inform the design of better schemes in the future.

      Acknowledgments

      We thank all the interviewees for their valuable time and insights contributed to this article.

      Supplemental Materials

      References

      1. OECD data: pharmaceutical spending. Organisation for Economic Co-operation and Development.
        • Iyn-Hyang L.
        • Karen B.
        • Catherine H.
        • et al.
        International experience in controlling pharmaceutical expenditure: influencing patients and providers and regulating industry—a systematic review.
        J Health Serv Res Policy. 2015; 20: 52-59
        • Godman B.
        • Shrank W.
        • Wettermark B.
        • et al.
        Use of generics: a critical cost containment measure for all healthcare professionals in Europe?.
        Pharmaceuticals (Basel). 2010; 3: 2470-2494
        • Carlson J.J.
        • Sullivan S.D.
        • Garrison L.P.
        • et al.
        Linking payment to health outcomes: a taxonomy and examination of performance-based reimbursement schemes between healthcare payers and manufacturers.
        Health Policy. 2010; 96: 179-190
        • Garrison Jr., L.P.
        • Carlson J.J.
        • Bajaj P.S.
        • et al.
        Private sector risk-sharing agreements in the United States: trends, barriers, and prospects.
        Am J Manag Care. 2015; 21: 632-640
        • Navarria A.
        • Drago V.
        • Gozzo L.
        • et al.
        Do the current performance-based schemes in Italy really work? “Success fee”: a novel measure for cost-containment of drug expenditure.
        Value Health. 2015; 18: 131-136
        • Klappe-Sabadi G.
        • Jansman F.G.A.
        • Honkoop H.A.
        • et al.
        Regionale verschillen in voorschrijfgedrag trastuzumab opnieuw getoetst.
        PW Wetenschappelijk Platform. 2008; 2: 154-158
        • Nederlandse Zorgautoriteit
        BELEIDSREGEL BR/CU-2017 Dure Geneesmiddelen.
        2011
        • Zorginstituut Nederland, mw.dr.G.Ligtenberg
        Voorwaardelijke toelating/financiering van zorg.
        2012
        • Makady A.
        • van Veelen A.
        • de Boer A.
        • et al.
        Implementing managed entry agreement in practice: the Dutch reality check.
        Health Policy. 2019; 123: 267-274
        • Palinkas L.A.
        • Horwitz S.M.
        • Green C.A.
        • et al.
        Purposeful sampling for qualitative data collection and analysis in mixed method implementation research.
        Adm Policy Ment Health. 2015; 42: 533-544
        • Hsieh H.F.
        • Shannon S.E.
        Three approaches to qualitative content analysis.
        Qual Health Res. 2005; 15: 1277-1288
        • Tong A.
        • Sainsbury P.
        • Craig J.
        Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups.
        Int J Qual Health Care. 2007; 19: 349-357
      2. Adaptive pathways. European Medicines Agency.
        • Bouvy J.C.
        • Jonsson P.
        • Longson C.
        • et al.
        Health technology assessment in the context of adaptive pathways for medicines in Europe: challenges and opportunities.
        Clin Pharmacol Ther. 2016; 100: 594-597
        • Bevir M.
        Governance: A Very Short Introduction.
        Oxford University Press, Oxford, UK2012
        • Duit A.
        • Galaz V.
        Governance and complexity: emerging issues for governance theory.
        Governance. 2008; 21: 311-335
      3. Opinion on the “cost-sharing initiatives” and “risk-sharing agreements” between pharmaceutical manufacturers and health and hospital. Drug Commission of the German Medical Association.
        • Faulkner S.D.
        • Lee M.
        • Qin D.
        • et al.
        Pricing and reimbursement experiences and insights in the European Union and the United States: lessons learned to approach adaptive payer pathways.
        Clin Pharmacol Ther. 2016; 100: 730-742
        • Carlson J.J.
        • Gries K.S.
        • Yeung K.
        • et al.
        Current status and trends in performance-based risk-sharing arrangements between healthcare payers and medical product manufacturers.
        Appl Health Econ Health Policy. 2014; 12: 231-238
        • Makady A.
        • Stegenga H.
        • Ciaglia A.
        • et al.
        Practical implications of using real-world evidence (RWE) in comparative effectiveness research: learnings from IMI-GetReal.
        J Comp Eff Res. 2017; 6: 485-490
        • van Staa T.P.
        • Goldacre B.
        • Gulliford M.
        • et al.
        Pragmatic randomised trials using routine electronic health records: putting them to the test.
        BMJ. 2012; 344: e55
        • Clinical Trials Transformation Initiative
        Registry Trials Project Multi-Stakeholder Expert Meeting.
        (Summary of the meeting held March 30)2016 (Available from https://www.ctti-clinicaltrials.org/files/registrytrials_03-30-2016_expertmeetingsummary.pdf. Accessed March 15, 2018)
        • Blommestein H.M.
        • Franken M.G.
        • Uyl-de Groot C.A.
        A practical guide for using registry data to inform decisions about the cost effectiveness of new cancer drugs: lessons learned from the PHAROS registry.
        Pharmacoeconomics. 2015; 33: 551-560
        • Mohseninejad L.
        • van Gils C.
        • Uyl-de Groot C.A.
        • et al.
        Evaluation of patient registries supporting reimbursement decisions: the case of oxaliplatin for treatment of stage III colon cancer.
        Value Health. 2015; 18: 84-90
        • Alemayehu D.
        • Ali MPP Riaz
        • Alvir J.M.
        • et al.
        Examination of data, analytical issues and proposed methods for conducting comparative effectiveness research using “real-world data”.
        J Manag Care Pharm. 2011; 17: S3-S37
        • Klungel O.H.
        • Martens E.P.
        • Psaty B.M.
        • et al.
        Methods to assess intended effects of drug treatment in observational studies are reviewed.
        J Clin Epidemiol. 2004; 57: 1223-1231
        • Garbe E.
        • Suissa S.
        Pharmacoepidemiology.
        in: Ahrens W. Pigeot I. Handbook of Epidemiology. Springer, New York, NY2014: 1875-1925
        • Efthimiou O.
        • Mavridis D.
        • Debray T.
        • et al.
        Combining randomized and non-randomized evidence in network meta-analysis.
        Stat Med. 2017; 36: 1210-1226
        • Berger M.L.
        • Sox H.
        • Willke R.J.
        • et al.
        Good practices for real-world data studies of treatment and/or comparative effectiveness: recommendations from the joint ISPOR-ISPE Special Task Force on real-world evidence in health care decision making.
        Pharmacoepidemiol Drug Saf. 2017; 26: 1033-1039
        • Wang S.V.
        • Schneeweiss S.
        • Berger M.L.
        • et al.
        Reporting to improve reproducibility and facilitate validity assessment for healthcare database studies V1.0.
        Pharmacoepidemiol Drug Saf. 2017; 26: 1018-1032
        • Kleijnen S.
        • Lipska I.
        • Alves T.L.
        • et al.
        Relative effectiveness assessments of oncology medicines for pricing and reimbursement decisions in European countries.
        Ann Oncol. 2016; 27: 1768-1775
      4. Work Package 5—Life cycle approach to improve Evidence Generation. European Network for Health Technology Assessment.
        (Accessed January 10, 2018)
      5. EMA-EUnetHTA three-year work plan: 2017-2020. European Network for Health Technology Assessment and European Medicines Agency.
        • Makady A.
        • ten Ham R.
        • de Boer A.
        • et al.
        Policies for use of real-world data in health technology assessment (HTA): a comparative study of six HTA agencies.
        Value Health. 2017; 20: 520-532