Advertisement

Scientific Evidence in Health Technology Assessment Reports: An In-Depth Analysis of European Assessments on High-Risk Medical Devices

Open ArchivePublished:June 20, 2017DOI:https://doi.org/10.1016/j.jval.2017.05.011

      Abstract

      Background

      The aim of this study was to examine the scientific evidence on clinical effectiveness and safety used in health technology assessments (HTAs) of high-risk medical devices (MDs) in Europe.

      Methods

      We applied a systematic approach to identify European institutions involved in HTA and to select reports assessing MDs considered high-risk according to the definition in the new German health care regulation §137h. Reports published between 2010 and 2015 were considered in our subsequent analysis. We used a structured tool based on widely accepted methodologic principles from Drummond’s framework to extract key information on the clinical evidence considered in the reports.

      Results

      Out of 1376 identified reports, 93 were eligible for analysis. All reports based their assessment primarily on direct evidence, in most cases (68%) identified through an independent systematic literature search. In more than half the identified studies considered in the reports, clinical evidence for demonstration of effectiveness and safety was of moderate or low quality. Even when systematic reviews and randomized controlled trials were available for assessment, most studies showed an unclear or high risk of bias.

      Conclusions

      This study confirms that the quality of scientific evidence used in HTA of high-risk MDs is low and therefore the use of evidence needs improvement. The European Commission recently updated the regulation on MDs but mainly focused on the safety of materials and the CE (Conformité Européene [European Conformity]) mark. Our results show that additional changes are necessary, specifically with regard to the marketing authorization process of MDs, with stricter quality requirements based on methodologically robust trials, possibly in combination with other evidence sources.

      Keywords

      Introduction

      According to the definition by the European Union (EU), a medical device (MD) is defined as “any instrument, apparatus, appliance, software, material or other article, whether used alone or in combination, including the software intended by its manufacturer to be used specifically for diagnostic and/or therapeutic purposes and necessary for its proper application, intended by the manufacturer to be used for human beings for the purpose of diagnosis, prevention, monitoring, treatment or alleviation of disease” [

      European Parliament and Council of the European Union. Directive 2007/47/EC of the European Parliament and of the Council of 5 September 2007 amending Council Directive 90/385/EEC on the approximation of the laws of the Member States relating to active implantable medical devices, Council Directive 93/42/EEC concerning medical devices and Directive 98/8/EC concerning the placing of biocidal products on the market. Available from: http://ec.europa.eu/consumers/sectors/medical-devices/files/revision_docs/2007-47-en_en.pdf. [Accessed January 30, 2017].

      ].
      MDs are generally regulated based on three directives referring to active implantable MDs (90/385EEC), MDs (93/42/EEC), and in vitro MDs (98/8/EC) [

      European Commission (EC). Regulatory framework. Available from: http://ec.europa.eu/growth/sectors/medical-devices/regulatory-framework/. [Accessed October 21, 2016].

      ]. Depending on its intended purpose and invasiveness, an MD will be classified as risk class I, IIa, IIb, and III, with class III covering products with the highest risk. For introduction into the European market, MDs need a European Conformity (Conformité Européene [CE]) marking received from an entity that has been accredited by a Member State, a so-called notified body. However, the CE mark does not indicate conformity to a single, predefined standard, nor is it a symbol intended for consumer assurance. It rather acts as a visible sign to let Member State authorities know that the MD is in compliance with the applicable directive(s). Manufacturers must provide evidence that the new device is “substantially equivalent” to a device already on the market. Therefore, obtaining the CE mark does not require a profound demonstration of scientific clinical data relating to effectiveness or safety [
      • Boudard A.
      • Martelli N.
      • Prognon P.
      • Pineau J.
      Clinical studies of innovative medical devices: what level of evidence for hospital-based health technology assessment?.
      ]. Although a subsequent directive (2007/47/EC) as well as a specific guideline (EC MEDDEV 2.7/4) amended the MD 93/42/EEC directive by adding an obligation to generate clinical data for high-risk devices (class III), no detailed information on the requirement of clinical trials was provided [

      European Council. Directive 2007/47/EC of the European Parliament and of the Council. Available from: http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2007:247:0021:0055:en:PDF. [Accessed October 21, 2016].

      ,

      European Commission (EC). Guidelines on clinical investigation: a guide for manufacturers and notified bodies. MEDDEV 2.7/4, December 2010. Available from: http://ec.europa.eu/consumers/sectors/medical-devices/files/meddev/2_7_4_en.pdf. [Accessed October 19, 2016].

      ]. This may have contributed to the lack of robust evidence from high-quality clinical trials (e.g., randomized controlled trials [RCTs]) in the premarket stage of MDs [
      • Rathi V.K.
      • Krumholz H.M.
      • Masoudi F.A.
      • Ross J.S.
      Characteristics of clinical studies conducted over the total product life cycle of high-risk therapeutic medical devices receiving FDA premarket approval in 2010 and 2011.
      ,
      • Hulstaert F.
      • Neyt M.
      • Vinck I.
      • et al.
      Pre-market clinical evaluations of innovative high-risk medical devices in Europe.
      ].
      When it comes to reimbursement decisions in the postmarket stage, policymakers require clinical evidence to demonstrate benefit for patients. One tool to support evidence-informed decision making regarding health technologies, including MDs, is health technology assessment (HTA). HTA is the systematic evaluation of characteristics, effects, and/or impacts of health technology and is conducted by interdisciplinary groups using explicit analytical frameworks that draw from a variety of methods [
      The International Network of Agencies for Health Technology Assessment (INAHTA)
      ]. The results of an HTA are summarized in reports that contain comprehensive information regarding clinical effectiveness and/or cost-effectiveness and may deal with the ethical, legal, and social implications of health technologies for patient health and the health care system [

      EUnetHTA. EUnetHTA HTA adaption toolkit Version 5, October 2011. Available from: http://www.eunethta.eu/sites/5026.fedimbo.belgium.be/files/EUnetHTA_adptation_toolkit_2011%20version%205.pdf. [Accessed August 26, 2016].

      ]. Previous publications have already indicated that HTA agencies face a lack of high-quality clinical evidence when evaluating MDs [
      • Hulstaert F.
      • Neyt M.
      • Vinck I.
      • et al.
      Pre-market clinical evaluations of innovative high-risk medical devices in Europe.
      ,
      • Fuchs S.
      • Olberg B.
      • Panteli D.
      • Perleth M.
      • Busse R.
      HTA of medical devices: challenges and ideas for the future from a European perspective.
      ]. Therefore, the main objective of this study was to investigate this observation in practice by systematically examining the scientific evidence on clinical effectiveness and safety considered in HTA reports of high-risk MDs in Europe. To our knowledge, this is the first attempt to document the issue on a wide scale. Some institutions provide clear recommendations for policymaking. However, examination of the concluding evidence used for the formulation of policy recommendations was not the purpose of this work.

      Materials and Methods

       Selection of Institutions and Composition of the HTA Report Pool

      Our first step was to follow a comprehensive methodology, as described in Fuchs et al. [
      • Fuchs S.
      • Olberg B.
      • Panteli D.
      • Busse R.
      Health technology assessment of medical devices in Europe: processes, practices, and methods.
      ], to identify institutions involved in HTA within European countries. In the second step, we composed a HTA report pool by systematically searching official websites and other online sources (e.g., database of the Centre for Reviews and Dissemination) for publicly available HTA reports published by any of the identified institutions. Reports were included if they focused on an MD (alone or within a procedure), were based on a systematic review methodology, and had been conducted between 2004 and 2015. A more detailed description of the process is given in Fuchs et al. [
      • Fuchs S.
      • Olberg B.
      • Panteli D.
      • et al.
      Testing the plausibility of a taxonomy for medical devices in the logic of HTA.
      ]. All included reports were documented in a Microsoft Excel database, and matching documents were downloaded and archived.
      In our third and final step, we derived our case sample from this database, applying inclusion and exclusion criteria determined a priori. Specifically, we chose a 5-year time frame to better reflect current practices. Moreover, we included MDs considered high risk in accordance with the new German health care regulation (§137h SGB V). The rationale for choosing this definition is that it provides specific selection criteria for high-risk MDs. Consequently, we included high-risk and highly invasive MDs, if they belong to risk class IIb or III or are active implantable devices. Further specifications determined for the definition of high-risk MDs under this new stipulation refer to devices that 1) interact with essential functions of organs or organ systems, especially the heart, the central circulatory system or the CNS; and 2) are assigned to class IIb and transmitting energy or radioactive radiation [
      Gemeinsamer Bundesausschuss (G-BA)
      ]. HTA reports on MDs of risk classes below IIb and/or trials that did not match further specifications determined within the stipulation were excluded. We focused our analysis on scientific evidence considered during the assessment of the clinical review (i.e., clinical effectiveness and/or safety) of an MD. Reports that solely relied on the assessment of other aspects (e.g., costs, without consideration of clinical effectiveness and/or safety) were excluded, as the evaluation of these aspects was not within the scope of our study.
      Furthermore, we included full-text reports published in German, English, Dutch, French, and Spanish. Reports in other languages, for which only an abstract in English was available, were excluded because for our analysis, we could not rely on abstracts alone for the required information.
      A tabular presentation of all relevant inclusion and exclusion criteria used for report selection with respect to the specific case sample is given in Supplementary Table 1.

       Data Abstraction

      For each HTA report, we extracted key information by using a standardized extraction tool. This was developed on the basis of the methodologic principles to be followed when striving for best practices in national HTA programs, as formulated by Drummond et al. [
      • Drummond M.F.
      • Schwartz J.S.
      • Jönsson B.
      • et al.
      Key principles for the improved conduct of health technology assessments for resource allocation decisions.
      ], and already used in previous research [
      • Fuchs S.
      • Olberg B.
      • Panteli D.
      • Perleth M.
      • Busse R.
      HTA of medical devices: challenges and ideas for the future from a European perspective.
      ,
      • Fuchs S.
      • Olberg B.
      • Panteli D.
      • Busse R.
      Health technology assessment of medical devices in Europe: processes, practices, and methods.
      ]. Specific variables for extraction were defined by following these principles, incorporating our team’s knowledge of and experience in HTA report production. As a result, the tool consisted of three parts, addressing 1) general report variables (e.g., type of report, language, year of publication, etc.); 2) assessment variables (e.g., EUnetHTA core model elements [

      EUnetHTA. HTA Core Model Version 3.0. Available from: http://eunethta.eu/sites/5026.fedimbo.belgium.be/files/HTACoreModel3.0.pdf. [Accessed September 29, 2016].

      ]), type of evidence, endpoints, etc.); and 3) variables with respect to decision making (e.g., recommended, not recommended, recommendation with limitations). The full extraction tool is presented in Supplementary Table 2.
      As our primary aim was to assess the individual clinical data considered in each HTA report, we focused on the elements referring to scientific evidence. Specifically, we extracted information regarding the following:
      • Evidence base: This refers to whether the evidence in the HTA reports was primarily based on submissions by the manufacturer, on data identified through an independent systematic literature search, or on both.
      • Type of evidence: We distinguished between “direct” (e.g., head-to-head trials) and “indirect” evidence. Direct evidence from well-conducted RCTs or a meta-analysis of RCTs is seen as providing the most valid estimates regarding the effectiveness of competing health care interventions. However, in some cases, interventions were not directly compared in RCTs. If there is no or insufficient evidence from direct-comparison trials, results of trials with different comparisons can be used to estimate the effects of treatments [
        • Glenny A.M.
        • Altman D.G.
        • Song F.
        • et al.
        Indirect comparisons of competing interventions.
        ].
      • Level of evidence (LoE): We classified clinical studies used in HTA reports according to the LoEs established by the Cochrane Collaboration [

        Cochrane Deutschland. Von der Evidenz zur Empfehlung (Klassifikationssysteme). Available from: http://www.cochrane.de/de/evidenz-empfehlung. [Accessed October 22, 2016].

        ]. These are summarized in Table 1.
        Table 1Level of evidence (LoE) according to the hierarchy by the Cochrane Collaboration

        Cochrane Deutschland. Von der Evidenz zur Empfehlung (Klassifikationssysteme). Available from: http://www.cochrane.de/de/evidenz-empfehlung. [Accessed October 22, 2016].

        LoEStudy designClassification of evidence
        1aEvidence obtained from meta-analysis or systematic review of RCTsHigh
        1bEvidence obtained from at least one RCT
        2aEvidence obtained from at least one well-designed controlled study without randomizationModerate
        2bEvidence obtained from at least one other type of well-designed quasi-experimental study, without randomization
        3Evidence obtained from well-designed non-experimental descriptive studies, such as comparative studies, correlation studies, and case studiesLow
        4Evidence obtained from expert committee reports, or opinions and/or clinical experiences of respected authorities
        RCT, randomized controlled trial.
      • Further considerations on scientific evidence: Whenever possible, we collected the total number of considered studies per HTA report. If HTA reports explicitly evaluated study quality (i.e., risk of bias [RoB]) using a specific tool or approach, such as the Cochrane RoB, Scottish Intercollegiate Guidelines Network (SIGN), or Grading of Recommendations Assessment, Development and Evaluation (GRADE), this information was also considered for analysis. However, an in-depth analysis regarding which RoB assessment tools were used in the HTA reports is outside the scope of this article and will be presented separately. The selection of reports and all extractions were carried out by one researcher and independently checked by another. Discrepancies were discussed and a final data pool was consented. Detailed data sheets for report selection and the extractions are available upon request.

      Results

       Description of the HTA Report Pool

      The total HTA report pool consisted of 1237 reports that evaluated 1376 technologies from 33 European institutions and were published between 2004 and 2015. Among these 1237 reports, 701 assessed high-risk MDs (≥class IIb). After screening for relevance, 93 reports—of which eight reports were updates—produced by 13 institutions from nine countries, fulfilled our inclusion criteria (see the section “Selection of institutions and composition of HTA report pool”) and were considered for analysis (Fig. 1).
      Fig. 1
      Fig. 1Selection process of health technology assessment (HTA) reports for qualitative data analysis.
      Of these 93 assessments, 60% had been conducted in the United Kingdom (UK) (e.g., The National Institute for Health and Care Excellence) or Austria (Ludwig Boltzmann Institute for Health Technology Assessment). Furthermore, 60% of the reports had been published within the last 3 years of the sample period (see Supplementary Figs. 1 and 2). For more details regarding the characteristics of each HTA report, including the corresponding reference, see Supplementary Table 3.
      We coded evaluated indications according to the System Organ Classes by the Medical Dictionary for Regulatory Activities [
      Medical Dictionary for Regulatory Activities (MedDRA)
      ]. Cardiac disorders were the most frequently evaluated (49 reports [53%]), followed by diseases related to the central circulatory system (36 reports [39%]). Eight reports assessed high-risk MDs applied for diseases of the CNS (9%). All evaluated devices were technologies for therapeutic use. In only three reports (3%), the technologies evaluated also served a diagnostic purpose. The most evaluated group of technologies were implantable devices (e.g., cardiac stents) (62 reports [67%]), which mainly belong to risk class III (37 reports [40%]).

       Clinical Evidence in HTA Reports

      With respect to the type of the evidence, our results show that all 93 reports based their evaluations on direct evidence. No report clearly stated the (additional) consideration of indirect evidence for assessment.
      Among the included reports, the scientific evidence was distributed as follows: two-thirds of the assessments (63 reports [68%]) were based on data identified through an independent literature search. Twenty-nine reports (31%) based their assessment on both an independent systematic literature search and additional submissions of clinical data from the manufacturer. Only one assessment report considered evidence solely on the basis of information provided by the manufacturer.
      Our pool of 93 HTA reports included 898 primary studies. When classifying the studies with an LoE of 1a and 1b (high category), studies with an LoE of 2a/2b (moderate category), and LoE class 3/4 (low category) (see Table 1), it was observed that evidence used for MD evaluation consisted mainly of clinical studies ranked moderate to low (551 studies [61%]). Almost half of all studies identified belonged to the lowest evidence category (level 4), primarily represented by case series or reports. Three, five, and seven percent of the studies, referred to studies with a LoE 3, 2b, or 2a, respectively, the majority of which were nonrandomized controlled prospective cohort studies (see Figure 2).
      Fig. 2
      Fig. 2Level of evidence (LoE) identified in health technology assessment (HTA) reports.
      The two highest LoE categories were represented by 347 primary studies (39%), which presented evidence obtained from an RCT or a meta-analysis/systematic review of RCTs. However, as Figure 3 shows, most of the RCTs were of moderate to low quality (61% of RCTs in category 1b and 63% of systematic reviews in category 1a), according to the assessment tools applied in the reports (e.g., Cochrane RoB, SIGN, GRADE, etc.).
      Fig. 3
      Fig. 3Study quality within the category 1a and 1b as documented in included health technology assessment (HTA) reports.
      For 17 (20%) systematic reviews and 73 (28%) RCTs, no information about the quality was given or it was deemed not evaluable because of lack of detailed information. Three reports did not provide information about the number of studies or detailed information about the type of evidence. In six reports, no clinical data for assessment of the clinical effectiveness or safety were identified (see Fig. 3).

      Discussion

      This study systematically analyzed the scientific evidence on postmarket evaluation of MDs by using a tangible basis of existing HTA reports produced by European institutions. The lack of high-quality clinical evidence, which HTA agencies often have to contend with when evaluating high-risk MDs, has been discussed for some time [
      • Hulstaert F.
      • Neyt M.
      • Vinck I.
      • et al.
      Pre-market clinical evaluations of innovative high-risk medical devices in Europe.
      ,
      • Fuchs S.
      • Olberg B.
      • Panteli D.
      • Perleth M.
      • Busse R.
      HTA of medical devices: challenges and ideas for the future from a European perspective.
      ] and was also noticeable in our study.
      Although almost all reports included in this analysis based their evaluation on direct evidence from independent systematic literature searches, good-quality data were scarce. In more than half the reports identified, evidence for the demonstration of the effectiveness and safety came from clinical studies of moderate or low LoE, mainly case series or reports. Additionally, our findings illustrate that even if systematic reviews and RCTs were available for assessment, most of these studies showed an unclear or high RoB according to the specific tools used in their reports. These findings are of great concern because they reflect a tremendous hurdle faced by HTA agencies in their task of making adequate recommendations to health care decision makers. This is particularly problematic with regard to devices associated with significant risks to the patient. The challenge of assessing MDs was also addressed in a recently published retrospective analysis of appraisals of MDs in Austria with regard to other factors (e.g., device risk class, evidence from uncontrolled studies, unmet medical need) that gain importance when making coverage decisions [
      • Kisser A.
      • Tüchler H.
      • Erdös J.
      • Wild C.
      Factors influencing coverage decisions on medical devices: a retrospective analysis of 78 medical device appraisals for the Austrian hospital benefit catalogue 2008–2015.
      ]. Those authors found that high-risk MDs with a low LoE, in particular, have favorable odds of obtaining positive, although restricted, reimbursement decisions. This further underlines the difficulties of decision making with regard to coverage for MDs as a consequence of the gap between market approval requirements and evidence needed for the assessment of clinical risk/benefit that is relevant for patients. This evidence gap is a well-known problem and stands in stark contrast to the very strict requirements in the approval process for pharmaceuticals. In Europe and other geographic jurisdictions, it can be largely attributed to the regulation system for market approval. Although MD investigations must adhere to the principles of good clinical practices (GCPs) laid out in EN ISO 14155 [

      International Organization for Standardization (ISO). ISO 14155:2011: Clinical investigation of medical devices for human subjects—good clinical practice. Available from: http://www.iso.org/iso/catalogue_detail?csnumber=45557. [Accessed October 23, 2016].

      ], there is no legal requirement for valid demonstration of the clinical benefit of a device to obtain CE marking. The main objective of a clinical investigation is to demonstrate the safety and performance (conformity with claims) of an MD. As a consequence, many high-risk devices are granted licensure based on low-quality evidence. Previous research by Rathi et al. [
      • Rathi V.K.
      • Krumholz H.M.
      • Masoudi F.A.
      • Ross J.S.
      Characteristics of clinical studies conducted over the total product life cycle of high-risk therapeutic medical devices receiving FDA premarket approval in 2010 and 2011.
      ] showed that high-risk MDs were cleared for the US market on the basis of only two studies on average: one pivotal study (studies that served as the basis of approval by the US Food and Drug Administration [FDA]) and one nonpivotal study. Nonpivotal studies are typically conducted to assess device feasibility, enrolling a limited number of patients to examine the device’s performance and to guide premarket development and clinical use.
      Following the principle of evidence-based medicine, RCTs are ranked as the gold standard of clinical trials [
      • Sackett D.L.
      • Rosenberg W.M.
      • Gray J.A.
      • et al.
      Evidence based medicine: what it is and what it isn’t.
      ]. This is especially true for pharmaceuticals but is increasingly debated within the evaluation process of MDs. In this debate, some researchers argue that RCTs are not suitable for evaluation of MDs because of alleged methodologic issues (e.g., fundamental differences between pharmaceuticals and MDs, complexity of devices, and impossibility of blinding) [
      German National Associations of Statutory Health Insurance Funds
      Medical devices: the myths and the truth.
      ]. However, successful examples, such as the Stenting and Aggressive Medical Management for Preventing Recurrent Stroke in Intracranial Stenosis (SAMMPRIS) trial showed that high-quality and meaningful RCTs can be conducted for at least some MDs [
      • Derdeyn C.P.
      • Chimowitz M.I.
      • Lynn M.J.
      • et al.
      Aggressive medical treatment with or without stenting in high-risk patients with intracranial artery stenosis (SAMMPRIS): the final results of a randomised trial.
      ]. However, non–RCT-generated data can still be important in the valuation of MDs. A recent report by George et al. [
      • George E.
      How real-world data compensate for scarce evidence in HTA.
      ] set out to explain in what way the National Institute for Health and Care Excellence considers evidence from sources other than RCTs. Specifically, although RCTs remain the preferred and main source of data, the use of non-RCT efficacy data or other clinical evidence is common and necessary for devices (e.g., cochlear implants, insulin pumps, endovascular stents) for which RCTs are, indeed, difficult to conduct or unethical.
      Our findings could not confirm the assumption that high-quality studies will occur in the postmarket approval setting and consequently will be available when it comes to decisions on reimbursement.
      Additionally, even if a comparative RCT follows a product’s launch, the results of the study on relevant outcomes, which are necessary for benefit assessment, may be available only after a considerable amount of time. In such cases, as discussed in the report by Tarricone et al. [
      • Tarricone R.
      • Boscolo P.R.
      • Armeni P.
      What type of clinical evidence is needed to assess medical devices?.
      ], real-world data can be collected and promptly assessed in the meantime to inform policymakers’ decision making. However, such decisions may be of temporary nature, as, for example, in a CED [
      • Miller F.G.
      • Pearson S.D.
      Coverage with evidence development: ethical issues and policy implications.
      ].

       Policy Implications

      We strongly recommend enforcing a requirement of high-quality studies for demonstrating the clinical efficacy and safety of high-risk MDs. Potential methodologic challenges (e.g., blinding) should not preclude RCTs from being carried out (e.g., the SAMMPRIS trial). However, assessors and decision makers often have to consider study designs other than RCTs. Therefore, we suggest the use of guidelines, such as those from EUnetHTA regarding tools or checklists that are suitable for assessing the RoB in nonrandomized study evidence. The EUnetHTA guideline, for example, recommends the recently developed instrument Risk Of Bias In Non-randomised Studies of Interventions (ROBINS-I), previously known as Cochrane Risk Of Bias Assessment Tool for Non-Randomized Studies of Interventions (ACROBAT-NRSI), for adequate evaluation of data quality based on non-RCTs [
      EUnetHTA
      ,
      • Sterne J.A.C.
      • Higgins J.P.T.
      • Reeves B.C.
      on behalf of the development group for ACROBATNRSI
      ]. The preference for this tool mainly stems from its advantages, such as the requirement for an endpoint specific assessment, a summary rating, and the availability of more detailed instruction and documentation guides, compared with other tools (e.g., Risk of Bias Assessment tool for Non-Randomized Studies [RoBANS]). ROBINS-I covers seven domains through which bias might be introduced into an NRSI. The first two domains, confounding and selection of participants into the study, address issues before the start of the interventions that are to be compared (“baseline”). The third domain addresses classification of the interventions themselves. The other four domains address issues after the start of interventions (e.g., biases caused by deviations from intended interventions, missing data, measurement of outcomes, and selection of the reported result) [
      • Sterne J.A.C.
      • Hernán M.A.
      • Reeves B.C.
      • et al.
      ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
      ].
      Similarly, to further enable HTA agencies to conduct adequate assessments, enhancing transparency and endorsing stricter requirements of reporting by using existing guidelines (e.g., CONSORT statement, or the EUnetHTA pilot core project for Additional Evidence Generation) are crucial [

      EUnetHTA. Public consultation on the second draft of: Core protocol Pilot for Additional Evidence Generation (AEG). Available from: http://www.eunethta.eu/news/closed-public-consultation-second-draft-core-protocol-pilot-additional-evidence-generation-aeg. [Accessed October 25, 2016].

      ,
      • Boutron I.
      • Moher D.
      • Altman D.G.
      • et al.
      Extending the CONSORT statement to randomized trials of non-pharmacologic treatment: explanation and elaboration.
      ]. This need was also recognized by the European HTA agencies themselves [
      • Fuchs S.
      • Olberg B.
      • Panteli D.
      • Perleth M.
      • Busse R.
      HTA of medical devices: challenges and ideas for the future from a European perspective.
      ].
      However, actions at the European level are needed to help contain the risks associated with access to technologies without a robust evidence base. Therefore, we welcome the recently introduced new rules of the European Commission (EC) on MD regulation in Europe. These rules address several issues mentioned in our work [

      European Parliament. eMeeting for Committees. Available from: http://www.emeeting.europarl.europa.eu/committees/agenda/201606/ENVI/ENVI%282016%290615_1/sitt-2571599. [Accessed October 23, 2016].

      ]. For example, the proposed changes to establish public access of the European Database on Medical Devices will hopefully lead to greater transparency of the European MD market. This database as a central source of information, as well as the planned postmarket surveillance systems, can be expected to improve the overall availability and regulation of clinical evidence [

      European Commission (EC): Medical devices: deal reached on new EU rules. Available from: http://www.consilium.europa.eu/en/press/press-releases/2016/05/25-medical-devices/. [Accessed October 23, 2016].

      ]. The latter can partially be achieved through postmarket registries, such as the proposed establishment of a “National implants register” in Germany, to obtain (long-term) data on the use, safety, and long-term performance of MDs [

      Medizintechnologie.de. Einführung eines “Nationalen Implantateregisters.” Available from: https://www.medizintechnologie.de/infopool/politik-wirtschaft/2016/einfuehrung-des-nationalen-implantateregisters/. [Accessed October 23, 2016].

      ]. Furthermore, the new regulation announces stricter requirements regarding the surveillance of “Notified Bodies” (e.g., employment of well-qualified staff, scheduled and ad hoc controls). Unfortunately, the new EU regulation still falls short. Stricter requirements with regard to evidence on clinical effectiveness and safety of MDs at the time of market entry are necessary to generate appropriate data that enable new MDs to meet the expectations of policymakers in the context of the reimbursement process. As long as this will not change, restrictions or conditions imposed on new MDs need to include an obligation to improve the evidence base, especially in the case of high-risk MDs. As already indicated above, one approach addressing this particular situation is that of the CED, which includes provisional access to novel medical interventions while generating the evidence needed to determine whether unconditional coverage is warranted. Examples of this practice can be found, for instance, in Germany and France [
      • Olberg B.
      • Perleth M.
      • Busse R.
      The new regulation to investigate potentially beneficial diagnostic and therapeutic methods in Germany: up to international standard?.
      ,
      • Martelli N.
      • van den Brink H.
      • Borget I.
      New French coverage with evidence development for innovative medical devices: improvements and unresolved issues.
      ].
      However, an ideal approach would combine a degree of premarket evaluation with a degree of probable risk and benefit posed by the device while emphasizing rigorous postmarket assessment in conjunction with carefully planned premarket clinical studies. Two current examples, initiated by the FDA, are the planned establishment of the “National Evaluation System for Health Technology” (NEST) program and the “Medical Device Epidemiology Network Initiative (MDEpiNet)”. The NEST program integrates data from clinical registries, electronic health records, and medical billing claims. The MDEpiNet is part of the Epidemiology Research Program at the FDA’s Center for Devices and Radiological Health in collaboration with external partners. Both initiatives aim at gathering more comprehensive evidence of the effectiveness and safety of MDs [
      • Shuren J.
      • Califf R.M.
      Need for a national evaluation system for health technology.
      ,
      Food and Drug Administration (FDA)
      ]. Such new models for evidence generation contain significant potential for reducing the burden of obtaining appropriate evidence across the life cycle of a device.

       Strengths and Limitations

      The major strength of this study is the thorough and systematic approach taken to identify assessment reports from European HTA agencies. To our knowledge, this is the first review of the scientific evidence considered in publicly available HTA reports assessing high-risk MDs used in Europe.
      However, we acknowledge several limitations. Despite the broad set of inclusion criteria, we potentially did not capture all relevant evaluations of the included institutions because of the exclusive reliance on publicly available HTA reports. Although the search for and selection of reports was carried out by two independent reviewers, some reports may have been overlooked. We restricted the selection of HTA reports to those in German, English, French, Spanish, and Dutch, as members of the research team have knowledge of only these languages. Although these comprise 77% of all relevant reports identified, some HTA reports potentially relevant to our analysis may have been excluded. Similarly, some reports in our sample focused exclusively on RCTs, which leaves open the possibility of bias toward this type of study in our findings. Our selection of reports for analysis was based on the definition of high-risk MDs as classified in the new German health care regulation §137h SGB V [
      Gemeinsamer Bundesausschuss (G-BA)
      ]. Based on the stricter specifications given in this stipulation, compared with the European guidelines for classification of MDs, our case sample might restrict the generalizability of our findings in the area of high-risk devices.

      Conclusions

      In the EU countries, MDs are essentially regulated in the same way they have been since the 1990s. This means that high-risk MDs can enter the market and be used in humans without the requirement of evidence from robust clinical studies. As a consequence, scientific evidence prior to market approval of high-risk MDs is often based only on evidence from studies that were methodologically inadequate. Our analysis shows that even in the postmarket approval setting, when key players have to make coverage decisions, the quality of the clinical data considered in assessment reports on high-risk devices is still low and needs to be improved. We recognize the need to enforce stricter requirements for high-quality studies for demonstration of clinical effectiveness and safety, possibly in combination with other evidence sources (e.g., registers). The use of guidelines to adequately deal with data quality based on non-RCTs study designs for MD assessment should become common practice. The EU has revised the rules governing MDs, mainly addressing the safety of materials and increasing the requirements for obtaining CE marks. This is a very important advancement with respect to longstanding and controversial issues of the given MD regulation. Nevertheless, the new regulation still lacks requirements for mandatory high-quality evidence on the effectiveness and safety of MDs for their approval. Innovative evaluation systems at the European or national level (e.g., NEST) that engage all stakeholders could help align interests with regard to device innovation, patient access, and patient safety to drive the development of and more timely access to high-quality, safe, and effective MDs.

      Conflict of Interest

      BO and MP work for the Federal Joint Committee, which is the highest decision-making body of the joint self-government of physicians, dentists, hospitals, and health insurance funds in Germany. One of its tasks is issuing directives determining the benefit basket of statutory health insurance funds.

      Acknowledgements

      This research was funded by the European Union’s Seventh Framework Programme (EU-FP7) and undertaken under the auspices of the ADVANCE_HTA project (Grant number 305983; www.advance-hta.eu). The European Commission had no role in the study design, collection and analysis of data, writing of the report, or submission of the paper for publication.

      Supplementary material

      References

      1. European Parliament and Council of the European Union. Directive 2007/47/EC of the European Parliament and of the Council of 5 September 2007 amending Council Directive 90/385/EEC on the approximation of the laws of the Member States relating to active implantable medical devices, Council Directive 93/42/EEC concerning medical devices and Directive 98/8/EC concerning the placing of biocidal products on the market. Available from: http://ec.europa.eu/consumers/sectors/medical-devices/files/revision_docs/2007-47-en_en.pdf. [Accessed January 30, 2017].

      2. European Commission (EC). Regulatory framework. Available from: http://ec.europa.eu/growth/sectors/medical-devices/regulatory-framework/. [Accessed October 21, 2016].

        • Boudard A.
        • Martelli N.
        • Prognon P.
        • Pineau J.
        Clinical studies of innovative medical devices: what level of evidence for hospital-based health technology assessment?.
        J Eval Clin Pract. 2013; 19: 697-702
      3. European Council. Directive 2007/47/EC of the European Parliament and of the Council. Available from: http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2007:247:0021:0055:en:PDF. [Accessed October 21, 2016].

      4. European Commission (EC). Guidelines on clinical investigation: a guide for manufacturers and notified bodies. MEDDEV 2.7/4, December 2010. Available from: http://ec.europa.eu/consumers/sectors/medical-devices/files/meddev/2_7_4_en.pdf. [Accessed October 19, 2016].

        • Rathi V.K.
        • Krumholz H.M.
        • Masoudi F.A.
        • Ross J.S.
        Characteristics of clinical studies conducted over the total product life cycle of high-risk therapeutic medical devices receiving FDA premarket approval in 2010 and 2011.
        JAMA. 2015; 314: 604-612
        • Hulstaert F.
        • Neyt M.
        • Vinck I.
        • et al.
        Pre-market clinical evaluations of innovative high-risk medical devices in Europe.
        Int J Technol Assess Health Care. 2012; 28: 278-284
        • The International Network of Agencies for Health Technology Assessment (INAHTA)
        What is health technology assessment (HTA)?. 2017; ([Accessed January 16, 2017])
      5. EUnetHTA. EUnetHTA HTA adaption toolkit Version 5, October 2011. Available from: http://www.eunethta.eu/sites/5026.fedimbo.belgium.be/files/EUnetHTA_adptation_toolkit_2011%20version%205.pdf. [Accessed August 26, 2016].

        • Fuchs S.
        • Olberg B.
        • Panteli D.
        • Perleth M.
        • Busse R.
        HTA of medical devices: challenges and ideas for the future from a European perspective.
        Health Policy. 2017; 121 (10.1016/j.healthpol.2016.08.01): 215-229
        • Fuchs S.
        • Olberg B.
        • Panteli D.
        • Busse R.
        Health technology assessment of medical devices in Europe: processes, practices, and methods.
        Int J Technol Assess Health Care. 2016; 32: 246-255
        • Fuchs S.
        • Olberg B.
        • Panteli D.
        • et al.
        Testing the plausibility of a taxonomy for medical devices in the logic of HTA.
        HTAi Oslo. 2015; ([Accessed January 16, 2017])
        • Gemeinsamer Bundesausschuss (G-BA)
        Beschluss des Gemeinsamen Bundesausschusses u¨ber eine A¨nderung der Verfahrensordnung: Verfahren fu¨r Bewertung von Methoden mit Medizinprodukten der hohen Risikoklassen aufgrund § 137h SGB V. 2016; ([Accessed August 9, 2016])
        • Drummond M.F.
        • Schwartz J.S.
        • Jönsson B.
        • et al.
        Key principles for the improved conduct of health technology assessments for resource allocation decisions.
        Int J Technol Assess Health Care. 2008; 24: 244-258
      6. EUnetHTA. HTA Core Model Version 3.0. Available from: http://eunethta.eu/sites/5026.fedimbo.belgium.be/files/HTACoreModel3.0.pdf. [Accessed September 29, 2016].

        • Glenny A.M.
        • Altman D.G.
        • Song F.
        • et al.
        Indirect comparisons of competing interventions.
        Health Technol Assess. 2005; 9 (iii–iv): 1-134
      7. Cochrane Deutschland. Von der Evidenz zur Empfehlung (Klassifikationssysteme). Available from: http://www.cochrane.de/de/evidenz-empfehlung. [Accessed October 22, 2016].

        • Medical Dictionary for Regulatory Activities (MedDRA)
        Introductory Guide MedDRA Version 14.0. 2011; ([Accessed September 29, 2016])
        • Kisser A.
        • Tüchler H.
        • Erdös J.
        • Wild C.
        Factors influencing coverage decisions on medical devices: a retrospective analysis of 78 medical device appraisals for the Austrian hospital benefit catalogue 2008–2015.
        Health Policy. 2016; 120: 903-912
      8. International Organization for Standardization (ISO). ISO 14155:2011: Clinical investigation of medical devices for human subjects—good clinical practice. Available from: http://www.iso.org/iso/catalogue_detail?csnumber=45557. [Accessed October 23, 2016].

        • Sackett D.L.
        • Rosenberg W.M.
        • Gray J.A.
        • et al.
        Evidence based medicine: what it is and what it isn’t.
        BMJ. 1996; 312: 71-72
        • German National Associations of Statutory Health Insurance Funds
        Medical devices: the myths and the truth.
        Common paper of the German National Associations of Statutory Health Insurance Funds. 2013; ([Accessed January 23, 2017])
        • Derdeyn C.P.
        • Chimowitz M.I.
        • Lynn M.J.
        • et al.
        Aggressive medical treatment with or without stenting in high-risk patients with intracranial artery stenosis (SAMMPRIS): the final results of a randomised trial.
        Lancet. 2014; 383: 333-341
        • George E.
        How real-world data compensate for scarce evidence in HTA.
        Z Evid Fortbild Qual Gesundh Wesen. 2016; 112: S23-S26
        • Tarricone R.
        • Boscolo P.R.
        • Armeni P.
        What type of clinical evidence is needed to assess medical devices?.
        Eur Respir Rev. 2016; 25: 259-265
        • Miller F.G.
        • Pearson S.D.
        Coverage with evidence development: ethical issues and policy implications.
        Med Care. 2008; 46: 746-751
        • EUnetHTA
        Publication of a JA2 methodological guideline—internal validity of non-randomised studies (NRS) on interventions. 2015; ([Accessed January 16, 2017])
        • Sterne J.A.C.
        • Higgins J.P.T.
        • Reeves B.C.
        • on behalf of the development group for ACROBATNRSI
        A Cochrane Risk of Bias Assessment Tool: for Non-Randomized Studies of Interventions (ACROBATNRSI), Version 1.0.0. 2014; ([Accessed October 22, 2016])
        • Sterne J.A.C.
        • Hernán M.A.
        • Reeves B.C.
        • et al.
        ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
        BMJ. 2016; 355: i4919
      9. EUnetHTA. Public consultation on the second draft of: Core protocol Pilot for Additional Evidence Generation (AEG). Available from: http://www.eunethta.eu/news/closed-public-consultation-second-draft-core-protocol-pilot-additional-evidence-generation-aeg. [Accessed October 25, 2016].

        • Boutron I.
        • Moher D.
        • Altman D.G.
        • et al.
        Extending the CONSORT statement to randomized trials of non-pharmacologic treatment: explanation and elaboration.
        Ann Intern Med. 2008; 148: 295-309
      10. European Parliament. eMeeting for Committees. Available from: http://www.emeeting.europarl.europa.eu/committees/agenda/201606/ENVI/ENVI%282016%290615_1/sitt-2571599. [Accessed October 23, 2016].

      11. European Commission (EC): Medical devices: deal reached on new EU rules. Available from: http://www.consilium.europa.eu/en/press/press-releases/2016/05/25-medical-devices/. [Accessed October 23, 2016].

      12. Medizintechnologie.de. Einführung eines “Nationalen Implantateregisters.” Available from: https://www.medizintechnologie.de/infopool/politik-wirtschaft/2016/einfuehrung-des-nationalen-implantateregisters/. [Accessed October 23, 2016].

        • Olberg B.
        • Perleth M.
        • Busse R.
        The new regulation to investigate potentially beneficial diagnostic and therapeutic methods in Germany: up to international standard?.
        Health Policy. 2014; 117: 135-145
        • Martelli N.
        • van den Brink H.
        • Borget I.
        New French coverage with evidence development for innovative medical devices: improvements and unresolved issues.
        Value Health. 2016; 19: 17-19
        • Shuren J.
        • Califf R.M.
        Need for a national evaluation system for health technology.
        JAMA. 2016; 316: 1153-1154
        • Food and Drug Administration (FDA)
        Medical Device Epidemiology Network Initiative (MDEpiNet). 2015; ([Accessed January 16, 2017])