Advertisement

Towards Integrated Health Technology Assessment for Improving Decision Making in Selected Countries

Open ArchivePublished:May 12, 2017DOI:https://doi.org/10.1016/j.jval.2017.03.011

      Abstract

      Objectives

      To assess the level of comprehensiveness of health technology assessment (HTA) practices around the globe and to formulate recommendations for enhancing legitimacy and fairness of related decision-making processes.

      Methods

      To identify best practices, we developed an evaluation framework consisting of 13 criteria on the basis of the INTEGRATE-HTA model (integrative perspective on assessing health technologies) and the Accountability for Reasonableness framework (deliberative appraisal process). We examined different HTA systems in middle-income countries (Argentina, Brazil, and Thailand) and high-income countries (Australia, Canada, England, France, Germany, Scotland, and South Korea). For this purpose, desk research and structured interviews with relevant key stakeholders (N = 32) in the selected countries were conducted.

      Results

      HTA systems in Canada, England, and Scotland appear relatively well aligned with our framework, followed by Australia, Germany, and France. Argentina and South Korea are at an early stage, whereas Brazil and Thailand are at an intermediate level. Both desk research and interviews revealed that scoping is often not part of the HTA process. In contrast, providing evidence reports for assessment is well established. Indirect and unintended outcomes are increasingly considered, but there is room for improvement. Monitoring and evaluation of the HTA process is not well established across countries. Finally, adopting transparent and robust processes, including stakeholder consultation, takes time.

      Conclusions

      This study presents a framework for assessing the level of comprehensiveness of the HTA process in a country. On the basis of applying the framework, we formulate recommendations on how the HTA community can move toward a more integrated decision-making process using HTA.

      Keywords

      Introduction

      The tension between rising health care costs and the ability to finance health care for everyone in the society at sustainable costs is well recognized but has gotten more intense, especially after the economic crisis in 2008 [

      Godman B, Oortwijn W, De Waure C, et al. Links between pharmaceutical R&D models and access to affordable medicines. European Union, PE 587.321 EN. IP/A/ENVI/2015-06. 2016. Available from: http://www.europarl.europa.eu/thinktank/en/document.html?reference=IPOL_STU(2016)587321. [Accessed December 2, 2016].

      ]. Health care decision makers increasingly want value for money when assessing the opportunities of health technologies.
      Health technologies include pharmaceuticals, medical devices, procedures, and the organizational and support systems within which health care is delivered. Health technology assessment (HTA) is defined as “a systematic evaluation of the properties and effects of a health technology, addressing the direct and intended effects of this technology, as well as its indirect and unintended consequences” [

      International Network of Agencies for Health Technology Assessment; Health Technology Assessment international. HTA glossary. Available from: http://htaglossary.net/health+technology+assessment+%28HTA%29. [Accessed November 15, 2016].

      ]. Nevertheless, the aspects considered by decision makers regarding reimbursement of a health technology traditionally mainly include the direct and intended effects, such as the level of clinical benefit (compared with the current standard) and incremental cost-effectiveness. It is, however, increasingly argued that indirect and unintended outcomes should also be considered to allow for value-based decisions [
      • Tanios N.
      • Wagner M.
      • Tony M.
      • et al.
      International Task Force on Decision Criteria
      Which criteria are considered in healthcare decisions? Insights from an international survey of policy and clinical decision makers.
      ]. After all, decision makers need assessments that are contextualized, involve a range of stakeholders, take interdependence and interactive aspects into account, and consider varying patient characteristics as well as implementation issues. Current HTA methodologies and decision making informed by HTA only partly respond to these requirements [

      Facey K. Improving the effectiveness and efficiency of evidence production for HTA in the light of current trends in drug and device development, health system funding, regulation and HTA. Background Paper Policy Forum. 2015. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4697336/pdf/S0266462315000355a.pdf. [Accessed April 14, 2017].

      ]. Furthermore, there is a need to improve the quality of decision making. This asks for structured, explicit, and transparent (appraisal) approaches. It has been increasingly acknowledged that it is the decision-making process, and not the robustness of relevant evidence or the formal procedure followed that warrants the legitimacy of reimbursement decisions [
      • Klein R.
      • Day P.
      • Redmayne S.
      Managing Scarcity: Priority Setting and Rationing in the National Health Service.
      ].
      The aim of this study was to examine how HTA practice can better assemble the most appropriate evidence and information and apply them in the most appropriate decision-making framework, enhancing both legitimacy and fairness. We therefore assessed the level of comprehensiveness of HTA practices (compared with traditional HTA) across selected HTA systems around the globe (N = 10), and formulated recommendations to enhance legitimate and fair decisions using HTA. Although there are several publications in which an overview of HTA practices of different countries is given [
      • Neumann P.
      • Drummond M.
      • Jönsson B.
      • et al.
      Are key principles for improved health technology assessment supported and used by health technology assessment organizations?.
      ,
      • Gulácsi L.
      • Rotar A.M.
      • Niewada M.
      • et al.
      Health technology assessment in Poland, the Czech Republic, Hungary, Romania and Bulgaria.
      ], our study adds value to the current knowledge base because we built on a recently developed framework (INTEGRATE-HTA model [

      Wahlster P, Brereton L, Burns J, et al. Guidance on the integrated assessment of complex health technologies—the INTEGRATE-HTA model. 2016. Available from: http://www.integrate-hta.eu/downloads/. [Accessed November 15, 2016]

      ]). Because comprehensive strategies for an integrated assessment of all dimensions of information in HTA were missing, this new approach was developed (see “Development of the Evaluation Framework” section).

      Methods

       Selection of Countries

      We selected 10 HTA systems (Argentina, Australia, Brazil, Canada, France, Germany, South Korea, Thailand, Scotland, and England [United Kingdom]) for the study. We aimed to include countries with a diverse set of HTA systems as well as countries with different levels of economic development (i.e., both middle- and high-income countries) as we have done in a previous study. Australia, Canada, and England are well known for their established HTA systems and are often used as reference countries. Brazil is rapidly developing effective HTA with clear links to the health needs of the country. Nevertheless, HTA processes in Brazil are not yet fully developed, for example, in terms of inclusiveness [
      • Oortwijn W.J.
      • Broos P.
      • Vondeling H.
      • et al.
      Mapping of health technology assessment in selected countries.
      ]. Thailand is considered interesting because it has used an explicit quantitative approach for reimbursement decisions [
      • Adunlin G.
      • Diaby V.
      • Xiao H.
      Application of multicriteria decision analysis in health care: a systematic review and bibliometric analysis.
      ]. Argentina, France, Germany, South Korea, and Scotland were chosen to represent different continents and to reflect both established HTA systems and emerging markets. South Korea is also the first Asian country that introduced economic evaluations for reimbursement decisions, on the basis of countries such as Australia, Canada, and England [
      • Cho E.
      • Park E.C.
      • Kang M.S.
      Pitfalls in reimbursement decisions for oncology drugs in South Korea: need for addressing the ethical dimensions in technology assessment.
      ].

       Development of the Evaluation Framework

      An HTA system is considered to be moving toward a more integrated/comprehensive reimbursement decision-making process when certain elements are covered [

      Wahlster P, Brereton L, Burns J, et al. Guidance on the integrated assessment of complex health technologies—the INTEGRATE-HTA model. 2016. Available from: http://www.integrate-hta.eu/downloads/. [Accessed November 15, 2016]

      ]. First, all relevant stakeholders are given “a voice” throughout the HTA process (from the identification of emerging technologies to scoping, assessment, appraisal, and to the final decision making). This means that HTA should be organized as a learning process [
      • Jansen M.P.
      • Helderman J.K.
      • Boer B.
      • Baltussen R.
      Fair processes for priority setting: putting theory into practice. Comment on “Expanded HTA: enhancing fairness and legitimacy.”.
      ]. Second, all processes, including the assessment aspects and decision criteria, are clearly defined and specified (by guidelines and/or legislation; grounded in evidence or ethical underpinning), become publicly available, and are contestable. Third, the process of deliberation should be transparent [
      • Daniels N.
      • Sabin J.
      Limits to health care: fair procedures, democratic deliberation, and the legitimacy problem for insurers.
      ]. Fourth, the processes need to be coherent with overall decision making (i.e., taking into account the local context) [
      • Klein R.
      • Day P.
      • Redmayne S.
      Managing Scarcity: Priority Setting and Rationing in the National Health Service.
      ].
      To assess the level of comprehensiveness of the HTA practices in the selected countries, we used two theoretical models: the INTEGRATE-HTA model and the Accountability for Reasonableness (A4R) framework developed by Daniels and Sabin [
      • Daniels N.
      • Sabin J.
      Accountability for reasonableness: an update.
      ].
      The INTEGRATE-HTA model was developed as part of a 3-year project funded by the European Commission, under the 7th Framework Programme for Research and Technological Development [

      Wahlster P, Brereton L, Burns J, et al. Guidance on the integrated assessment of complex health technologies—the INTEGRATE-HTA model. 2016. Available from: http://www.integrate-hta.eu/downloads/. [Accessed November 15, 2016]

      ]. The project developed concepts and methods that enable a patient-centered, comprehensive, and integrated assessment of the effectiveness, economic, ethical, sociocultural, and legal issues of health technologies that takes context and implementation into account. The INTEGRATE-HTA model (Fig. 1) enables a coordinated assessment of all these aspects and addresses their interdependencies. This model includes five steps: 1) involve stakeholders to elicit needs, topics, and outcomes; 2) take patient preferences and characteristics, implementation issues, and context into account; 3) assess the evidence regarding effectiveness, economic, ethical, sociocultural, and legal aspects; 4) integrate the evidence in a structured way to respond to the needs of the stakeholders; and 5) feed the evidence into the decision-making process to support the decision makers, that is, select a tool to structure a deliberative discussion (in cooperation with the decision/appraisal committee). This decision support tool may be quantitative (e.g., multicriteria decision analysis) or qualitative (e.g., consensus methods). The INTEGRATE-HTA model makes clear that it is not a matter of simply collecting facts, but a matter of collecting facts that are considered relevant, plausible to stakeholders, and amenable to scientific inquiry. This contributes to a transparent HTA process and a deeper understanding of the value of health technologies [

      Wahlster P, Brereton L, Burns J, et al. Guidance on the integrated assessment of complex health technologies—the INTEGRATE-HTA model. 2016. Available from: http://www.integrate-hta.eu/downloads/. [Accessed November 15, 2016]

      ]. The concepts and methods developed within the INTEGRATE-HTA model have been applied in case studies in England (palliative care) [

      Brereton L, Wahlster P, Lysdahl KB, et al. (on behalf of the INTEGRATE-HTA project team). Integrated assessment of home based palliative care with and without reinforced caregiver support: “a demonstration of INTEGRATE-HTA methodological guidances”—executive summary. Available from: http://www.integrate-hta.eu/downloads/. [Accessed November 15, 2016]

      ] and Canada (dialysis modalities) [

      International Network of Agencies for Health Technology Assessment. INAHTA webinar on INTEGRATE-HTA. 2016. Available from: http://www.inahta.org/2016/09/new-inahta-webinars/. [Accessed November 15, 2016]

      ].
      Fig. 1
      Fig. 1The INTEGRATE-HTA model. HTA, health technology assessment.
      In step 5, the HTA results feed into the decision-making process in a manner that is relevant and meaningful to decision makers, using a fair, deliberative process [
      • Daniels N.
      • Porteny T.
      • Urrutia J.
      Expanded HTA: enhancing fairness and legitimacy.
      ]. In this way, we have operationalized step 5 of the INTEGRATE-HTA model in alignment with the A4R framework. The A4R framework consists of four conditions that must be met to ensure that the resource allocation process is fair and that the actual decisions are based on reasons that are communicated to all relevant stakeholders involved. The first condition is publicity. Resource allocation decisions and the underlying reasons must be transparent and made public. The second condition is relevance. The arguments underlying the actual decisions must be clear and all stakeholders should be given the chance to participate. There is room for providing differing views, and space to consider divergent opinions and preferences. The arguments used should have a scientific basis and should contain reasons and principles that are accepted as relevant. The third condition is revision. There must be a mechanism in place that gives stakeholders the option to appeal against decisions, propose revisions, and receive a reasoned response. This should give decision makers the chance to reconsider their decisions using extra (new) arguments or (new) evidence or societal values. And the fourth condition is enforcement. A process should be in place to ensure that the criteria are met [
      • Daniels N.
      • Sabin J.
      Accountability for reasonableness: an update.
      ].
      On the basis of the INTEGRATE-HTA model as described elsewhere [

      Wahlster P, Brereton L, Burns J, et al. Guidance on the integrated assessment of complex health technologies—the INTEGRATE-HTA model. 2016. Available from: http://www.integrate-hta.eu/downloads/. [Accessed November 15, 2016]

      ], we determined relevant key criteria that reflect the level of comprehensiveness of the HTA process, in addition to traditional criteria often used in HTA (e.g., collecting evidence on safety and clinical effectiveness) [
      • Oortwijn W.J.
      • Broos P.
      • Vondeling H.
      • et al.
      Mapping of health technology assessment in selected countries.
      ]. The development of the INTEGRATE-HTA model is based on the assumption that the (traditional) aspects to be assessed strongly interact with each other, with context and implementation issues and patient characteristics, the degree of uncertainty, and stakeholder values and preferences. Therefore, the key criteria used for this study are related to these aspects. Thereafter, each criterion was translated into judgment criteria (Table 1) to be able to assess the extent to which that specific element is covered in an HTA system. In defining the judgment criteria, we used already validated criteria, such as those used for mapping the level of HTA [
      • Oortwijn W.J.
      • Broos P.
      • Vondeling H.
      • et al.
      Mapping of health technology assessment in selected countries.
      ] and existing checklists, including the International Network of Agencies for Health Technology Assessment (INAHTA) checklist for HTA reports [

      International Network of Agencies for Health Technology Assessment. A checklist for health technology assessment reports. Available from: http://www.inahta.org/wp-content/uploads/2014/04/INAHTA_HTA_Checklist_English.pdf. [Accessed November 15, 2016].

      ].
      Table 1Judgment criteria to assess the level of comprehensiveness of the HTA process.
      PhaseDefinition used
      Assessment phase
      Multiple stakeholders are involved in scoping an HTADefining the objective and research questions of the HTA by a systematic exploration of relevant aspects from multiple perspectives (e.g., patients, informal caregivers, health professionals, and decision makers).
      Context is taken into accountContext is defined as the conditions and circumstances that are relevant to the application of an intervention, for example, setting (e.g., hospital) and sociocultural aspects (knowledge, beliefs, conceptions, customs, institutions, and any other capabilities and habits acquired by a group that may influence uptake).
      Implementation issues are taken into accountImplementation issues refer to the actual delivery of a health technology to policy measures and processes or to funding mechanisms (e.g., tax incentives and reimbursement schemes) that directly concern or indirectly influence the implementation of the health technology.
      Patient-related factors that influence treatment effects are taken into accountPatients may respond differently to treatments in terms of nature and magnitude of a beneficial effect, the time of onset, and adverse outcomes. It is therefore important to identify factors that influence treatment effects to determine which treatments work best for whom; making medicine more personalized; and better valuation of research outcomes.
      Patient preferences with regard to treatment outcomes are taken into accountPatients often have different views on the relative importance of certain treatment outcomes. It is widely acknowledged that understanding patients’ preferences is important for an accurate assessment and appraisal of the impact of a disease on the patient’s quality of life.
      Evidence reports and standardized evidence summaries for each assessment aspect are producedTo provide evidence reports and standardized evidence summaries for each assessment aspect (e.g., report on economics and on ethical aspects).
      Stakeholder consultation is performed to review the evidence reportsRelevant stakeholders are asked to review the assessment results (i.e., evidence reports/summaries) with regard to plausibility. The result is an assessment report that includes a critical evaluation of the available evidence and uncertainty, and an overview of where evidence is missing.
      Appraisal/decision-making phase
      The appraisal/decision-making process is explicitThe criteria
      For example, health outcomes, disease impact, implementation of the intervention, cost-effectiveness, and total costs/budget impact of an intervention [29].
      and methods
      For example, MCDA methods, analytic methods, preference elicitation methods, and consensus methods as described in Wahlster et al. [8].
      used in the process are well described in a publicly available document.
      The appraisal/decision-making process is transparentThe procedures used are well described in a publicly available document and the process is open to the public (e.g., public hearings), and the agenda and notes on the meeting are provided in the public domain.
      The decisions and the underlying reasons are made publicThe final decisions and the underlying reasons are publicly available (e.g., via the Web site of the Ministry of Health and announcement in the official journal).
      Stakeholder involvement is clearly specified and open to the publicThe stakeholders involved (also those in addition to a specific committee for appraisal/HTA decision making) are open to the public (e.g., via the HTA agency Web site and notes on the meetings that are provided in the public domain), and the ways in which the views of the stakeholders are taken into account are well described in a publicly available document.
      Mechanism(s) for appeal, to propose revisions, and to receive a reasoned response are in placeMechanism(s) for appeal, to propose revisions, and to receive a reasoned response are operational and described in a publicly available document.
      Systems are in place to monitor and evaluate the processSystems to monitor and evaluate the process are operational and described in a publicly available document.
      HTA, health technology assessment; MCDA, multicriteria decision analysis.
      low asterisk For example, health outcomes, disease impact, implementation of the intervention, cost-effectiveness, and total costs/budget impact of an intervention
      • Wahlster P.
      • Goetghebeur M.
      • Kriza C.
      • et al.
      Balancing costs and benefits at different stages of medical innovation: a systematic review of multi-criteria decision analysis (MCDA).
      .
      For example, MCDA methods, analytic methods, preference elicitation methods, and consensus methods as described in Wahlster et al.

      Wahlster P, Brereton L, Burns J, et al. Guidance on the integrated assessment of complex health technologies—the INTEGRATE-HTA model. 2016. Available from: http://www.integrate-hta.eu/downloads/. [Accessed November 15, 2016]

      .

       Data Collection

      We performed a (nonexhaustive) literature search to provide an accurate, factual, and current description of the HTA processes in the selected countries. The search built on recent overview studies describing evidence-based approaches for decision making in the selected HTA systems, for example, studies by Oortwijn et al. [
      • Oortwijn W.J.
      • Broos P.
      • Vondeling H.
      • et al.
      Mapping of health technology assessment in selected countries.
      ], Barnieh et al. [
      • Barnieh L.
      • Manns B.
      • Harris A.
      • et al.
      A synthesis of drug reimbursement decision-making processes in organisation for economic co-operation and development countries.
      ], Wildson et al. [
      • Wildson T.
      • Fiz E.
      • Haderi A.
      A comparative analysis of the role and impact of health technology assessment: 2013.
      ], and Panteli et al. [
      • Panteli D.
      • Eckhardt H.
      • Nolting A.
      • et al.
      From market access to patient access: overview of evidence-based approaches for the reimbursement and pricing of pharmaceuticals in 36 European countries.
      ]. In addition, we reviewed the Web sites of the relevant organizations in each country to retrieve the most up-to-date documents that describe the HTA process. HTA organizations were, among others, identified from the INAHTA Web site (www.inahta.org).
      To validate the findings of the literature review and to collect additional information that may not be publicly available, we conducted three to four structured interviews with relevant key stakeholders in each selected country. Stakeholders, including representatives of appraisal committees, decision makers, HTA organizations, industry, clinicians, payers, regulatory agencies, patient representatives, and/or HTA thought leaders (academics), were identified using our own networks, Web sites of HTA organizations (appraisal committees), and the INAHTA Web site. Each invitee received an email containing an invitation letter to participate in the study. Efforts were made to ensure a good response rate and representation across stakeholder categories (within countries), with reminder emails sent 1 week and follow-up calls 2 weeks after the first invitation to stakeholders who did not yet respond. About 1 week before the interview, the respondents received a summary of the interview guide (i.e., topic list). The interview guide was developed to structure and standardize the interviews. Each interview started with a short introduction to the study. Hereafter, questions were asked on the assessment phase and the appraisal phase of the HTA process and were based on the judgment criteria (Table 1). Interviewers specifically asked for recent developments and real-life examples, given their additional value to the literature review. Each interview concluded with the interviewees being asked to indicate whether statements (on the basis of the judgment criteria) on the HTA process applied to their system. Each statement could be answered on a seven-point scale ranging from 1 (strongly disagree) to 7 (strongly agree). Furthermore, interviewees were asked about their views on (other) HTA systems that apply a broad/integrated perspective of the assessment phase and/or use a transparent and explicit approach to achieve fair and legitimate HTA decisions. Each interview lasted approximately 1 hour. Interviews were conducted from June to October 2016.
      The collected data were summarized in country profiles, in which a distinction was made between findings from the literature and findings from the interviews. A common template for drafting the country profile was used, which was based on the criteria for assessing the level of comprehensiveness of the HTA process and a template that was used in a previous study mapping the level of HTA in several countries around the globe [
      • Oortwijn W.J.
      • Broos P.
      • Vondeling H.
      • et al.
      Mapping of health technology assessment in selected countries.
      ]. The country profiles were sent back to the interviewees with the request to review and add to the information provided (see Table 2 for the number of reviews per country).
      Table 2Interview response rates per country.
      CountryNo. of invitations sentNo. of nonrespondersNo. of respondersNo. of confirmationsNo. of rejections
      Reasons for rejection include having no expertise, lack of time, and not willing to participate because of industry funding.
      No. of interviews conductedNo. of reviews
      Interviewee reviewed only the summary (a) or only the interview results (b), or the complete report (c).
      Argentina5233031 (a)
      Australia6243132 (c)
      Brazil21156
      Referred to another person.
      5230
      Canada7254141 (b)
      England10193632 (c)
      France6243131 (c)
      Germany13583533 (c)
      Scotland8173431 (c)
      South Korea8443131 (c)
      Thailand12754141 (c)
      Total96415534223213
      low asterisk Reasons for rejection include having no expertise, lack of time, and not willing to participate because of industry funding.
      Interviewee reviewed only the summary (a) or only the interview results (b), or the complete report (c).
      Referred to another person.

       Analysis

      To map the level of comprehensiveness of the HTA system and to identify (potential) best practices, we used a simple scoring system for each judgment criteria: “yes,” “to some extent,” and “no or very limited.” This scoring system has been used previously in a study to map the level of HTA development at country level in selected countries [
      • Oortwijn W.J.
      • Broos P.
      • Vondeling H.
      • et al.
      Mapping of health technology assessment in selected countries.
      ]. The scores were assigned by one researcher, taking into account the findings of the literature and the interviews, including the interviewees’ scores on the statements. The scores were reviewed by another researcher (i.e., the same researcher reviewed all HTA systems). During a consensus meeting with all the authors, the scores, including the alignment of scores between the countries, were discussed and, if necessary, revised.
      We encountered that, on the basis of the total scoring, no country may be referred to as best practice for its system as a whole (i.e., all judgment criteria should be scored “yes”). Canada may be considered best practice when considering all the criteria related to the assessment phase, whereas Scotland is considered best practice for the appraisal phase. Nevertheless, there are more systems that can be considered best practice when we consider each judgment criterion separately. To illustrate how an HTA system operationalizes the relevant criterion, we provide details on two HTA systems per criterion: Canada (for assessment) or Scotland (for appraisal) and a middle-income country when applicable or a high-income country using the level of documentation available.

      Results

      In total, 55 out of 96 stakeholders responded to the interview invitation. Finally, at least three interviews, with different types of stakeholders, were conducted per HTA system (N = 32 in total). Table 2 provides more details on the response rates.

       Level of Comprehensiveness of the HTA Process across Countries

      The level of comprehensiveness of the HTA process in the selected countries is presented in Table 3. As stated earlier, none of the HTA systems included in this study addresses all selected criteria in both the assessment and appraisal phases. A country may only be a best practice for a (subset of) element(s). With regard to the assessment phase, we consider Canada as best practice, whereas we consider Scotland for the appraisal phase. Canada and Scotland, together with Australia and England, are also mentioned most often as best practice by the stakeholders interviewed. Argentina is at the lowest end of all the HTA systems included in the study. This might be due to the HTA process not being coordinated at the moment because it is awaiting the adoption of a law that constitutes a national HTA agency. Often, Brazil is seen as a best practice example for the Latin American region. The HTA field in Brazil is evolving and regulations for a better HTA process are instigated, but the stakeholders perceive that still a lot has to be learned.
      Table 3Level of comprehensiveness of the HTA process in the selected HTA systems on the basis of literature and interviews with relevant stakeholders.
      ARGAUBRACAFRAGERSCOTKORTHAUKNo. of (+)No. of (+/−)No. of (−)
      Assessment phase
      Scoping++/−+/−++325
      Context+/−++++/−+/−++/−+/−+550
      Implementation issues+/−++++/−++++721
      Patient-related factors+/−++++++/−+622
      Patient preferences+++++/−+/−424
      Evidence reports++++++++/−++910
      Stakeholder consultation+/−+/−++/−++/−+/−+/−+361
      Appraisal/decision-making phase
      The process is explicit++/−+++++/−++721
      The process is transparent+/−++++/−++/−+532
      Underlying reasons are made public+/−++++++/−+/−+622
      Stakeholder involvement is clearly specified++++/−+++/−+622
      Mechanism(s) for appeal are in place++++/−+++/−+622
      Monitoring and evaluation of the process+/−+/−+++/−+/−244
      Analysis
      No. of (+) within an HTA system1951278111411
      No. of (+/−) within an HTA system4321541682
      No. of (−) within an HTA system8160111610
      Ranking10471653982
      Note. The table shows the extent to which the judgment criteria are addressed, per HTA system: “Not or very limited” is indicated by (−), “to some extent” is indicated by (+/−), and “yes” is indicated by (+). The ranking was based on the total number of (+), (+/−), and (−) within an HTA system.
      ARG, Argentina; AU, Australia; BRA, Brazil; CA, Canada; FRA, France; GER, Germany; HTA, health technology assessment; KOR, South Korea; SCOT, Scotland; THA, Thailand; UK, United Kingdom (England).
      To judge the level of comprehensiveness in each HTA system, it is also important to consider to what extent the assessment and appraisal are linked. It is, for example, remarkable that in some systems the assessment and appraisal phases do not seem aligned even though it is the remit of a single organization to have responsibility for (the link between) both phases (e.g., the National Authority for Health [Haute Autorité de Santé] in France and the Pharmaceutical Benefits Advisory Committee in Australia). There may be reasons why such an HTA organization does not feel the intrinsic “responsibility” required for a high level of comprehensiveness. In Australia, for example, the Pharmaceutical Benefits Advisory Committee does not itself perform HTA assessments, but relies on the information provided by manufacturers (dossier-driven) and the evaluation of that information by contracted university-based assessors.

       Criteria Reflecting the Comprehensiveness of the HTA Process

      When looking at the different criteria reflecting the comprehensiveness of the HTA process, we see differences between the HTA systems. With respect to the assessment phase, it is apparent that scoping is not taking place in many HTA systems; it takes place only in Canada, Thailand, and England. Many (well-established) systems do have a scoping phase, but do not (explicitly) involve stakeholders (e.g., Australia). In contrast, evidence assessment is generally well established, that is, most HTA systems provide evidence reports and standardized evidence summaries for each assessment aspect (e.g., report on economics and report on ethical aspects). The other criteria show a more diverse view, which might be because some criteria have not yet gained much attention in the HTA community (e.g., patient heterogeneity). This might be a reason for not having found much literature and/or the diverging stakeholder views. For example, in most HTA systems the views differ with regard to the inclusion of patient-related factors and patient preferences. Regarding the appraisal phase, the monitoring and evaluation of the process is not well established across HTA systems. We only found evidence that both France and Scotland have such a system in place. In Germany, for example, no literature has been found on a formal monitoring and evaluation of the appraisal process, and stakeholders were also not aware of a monitoring process. Also, diverging views within HTA systems were often seen between stakeholders with regard to the involvement of stakeholders in the appraisal phase. For each criterion, details on two countries that are considered best practices are presented in Table 4 (assessment phase) and Table 5 (appraisal phase).
      Table 4Assessment phase: Countries considered as best practice per criterion, description of two examples (boldfaced).
      CriterionBest practicesExamples
      ScopingCA, THA, UKCA: For each assessment of a new pharmaceutical product, patient groups are invited to submit their views with regard to the impact of the disease on patients and their families, experiences with current therapies, and expectations regarding and/or experiences with the product under assessment. The input from patients is sought early in the process to include the views within the assessment protocol and the assessment reports.
      THA: Representatives of relevant stakeholders, such as policymakers, health professionals, academics, patient associations industry, civil society, and lay citizens, may annually suggest topics for assessment. Scoping and prioritization of HTA takes place in the form of a panel including representatives from health professionals, academics, patients, and civil society that make use of criteria, such as severity of the disease and practice variation.
      ContextAU, BRA, CA, SCOT, UKBRA: The IATS includes, besides clinical and economic evidence, the ethical and public health consequences in their evaluation of new pharmaceutical products, services, and technologies.
      CA: Clinical benefit and health economic information is required to establish value for money. Other evidence on equity, public health, and budget impacts may be submitted but is not required. Contextual issues may be addressed in the “other considerations” section of the template. Ethical considerations are often highlighted as part of the review in craving the context of the value of the pharmaceutical product.
      Implementation issuesAU, BRA, CA, GER, SCOT, THA, UKBRA: In Brazil, there is a clear link between HTA and the development of clinical practice guidelines as defined by law.
      CA: The standard reporting template for the clinical and economic evidence includes sections on “the place of therapy” and “other considerations.” With respect to the clinical context, CADTH asks the clinical expert to provide input on whether the pharmaceutical product is expected to fit in the place of therapy on the basis of clinical evidence as well as using expert opinion.
      Patient-related factorsAU, CA, FRA, GER, SCOT, UKAU: The PBAC guideline advises the manufacturer to study patient-related factors on treatment effects. Important effects include dose-response considerations, varying duration of use, pharmaceutical interactions, settings of use, and patients’ baseline characteristics (including risk factors and disease severity).
      CA: In CADTH’s guidelines, patient heterogeneity in relation to treatment effects is mentioned. The guidelines, however, focus on the role of an analyst in a modeling study. This person should uncover heterogeneity in data relating to aspects such as costs, outcomes, and preferences in stratified analysis.
      Patient preferencesAU, CA, GER, SCOTCA: CADTH developed a formal approach for incorporating patients’ perspectives on health outcomes and issues in both the assessment and appraisal phases (both for pharmaceutical products and for nonpharmaceutical interventions). Calls for patient input and the respective deadline are posted on CADTH’s Web site, by CADTH e-alerts and CADTH’s Twitter accounts. A total of 35 business days are provided for preparing and submitting patient input by means of the Patient Input Template. Updates of these templates are used since December 2016. In very few cases, patient group input is not submitted. This may happen, for example, when patients are difficult to reach or have a very short life expectancy. In those cases, CADTH may search for gray literature and/or go to patient groups outside of Canada.
      GER: Opinions and experiences of patients are collected through qualitative research methods. IQWiG consults patients/patient representatives in processing the research questions and in the production of assessment reports. The Federal Joint Committee (Gemeinsame Bundesausschuss), responsible for reimbursement decisions regarding social health insurance in Germany, is a member of the advisory group for PREFER. This project, under the Innovative Medicines Initiative, recently started on when and how to include patient preferences in decision making.
      Evidence reportsARG, AU, BRA, CA, FRA, GER, SCOT, THA, UKAU: Evidence reports for each assessment aspect are produced. For example, the NHMRC Clinical Trials Centre and AHTA produce HTA reports on safety, effectiveness, and cost-effectiveness of new clinical tests and interventions.
      CA: A standard reporting format exists that describes the format for the preparation of reports on economic evaluations by CADTH.
      Stakeholder consultationCA, GER, UKCA: CADTH formally asks feedback from different stakeholder groups (including health care professionals, patients, industry, associations, and other interested parties) on projects and draft reports. In addition, processes are in place for patient groups to provide input on pharmaceutical products that are reviewed through the CDR and the pCODR.
      GER: The report of the benefit assessment is published online, and parties (pharmaceutical companies, federations, and experts) are given the chance to submit statements within a 3-wk time period after publication. The systematic literature search should also include consultation of experts/industry/patient organizations to obtain practice-relevant input. During the development of reports, IQWiG consults patients, patient representatives, as well as national and international experts.
      AHTA, Adelaide Health Technology Assessment; ARG, Argentina; AU, Australia; BRA, Brazil; CA, Canada; CADTH, Canadian Agency for Drugs and Technologies in Health; CDR, Common Drug Review; FRA, France; GER, Germany; HTA, health technology assessment; IATS, Institute for Health Technology Assessment; IQWiG, Institute for Quality and Efficiency in Health Care; NHMRC, National Health and Medical Research Council; PBAC, Pharmaceutical Benefits Advisory Committee; pCODR, pan-Canadian Oncology Drug Review; SCOT, Scotland; THA, Thailand; UK, United Kingdom (England).
      Table 5Appraisal phase: Countries considered as best practice per criterion, description of two examples (boldfaced).
      CriterionBest practicesExamples
      The decision-making process is explicitAU, CA, FRA, GER, SCOT, THA, UKSCOT: The decision-making process is extensively described in a document that is available from the Web site of the SMC. When appraising the evidence, the main criteria used are clinical effectiveness and cost-effectiveness. In some specific cases, other factors may also be taken into consideration when voting for a decision in some specific cases; for example, more economic uncertainty may be accepted in the case of orphan pharmaceutical products.
      THA: Although we did not find a publicly available document on the appraisal phase, the role of the stakeholders involved seems explicit. The evidence (provided by HITAP) and criteria used in decision making are also clear. These criteria include value for money and budget impact, which have been reported to improve transparency and fairness.
      The decision-making process is transparentBRA, CA, FRA, SCOT, UKBRA: Legislation (Law 12.401/2011) includes regulations that aim to make the decision-making process for new technologies increasingly transparent. The law states that the rules of procedures for HTA must include a maximum period and a mandatory public consultation and an optional public hearing as part of the process.
      SCOT: Appraisal meetings are open for the public since May 2014. During these meetings, all eligible SMC members vote individually regarding the decision, taking into account the available evidence and the discussion.
      Underlying reasons are made publicAU, CA, FRA, GER, SCOT, UKAU: All recommendations, reasons, and a summary of evidence considered are made publicly available on a Web site of the Pharmaceutical Benefits Scheme of the Australian government.
      SCOT: The evidence report as well as the final decision are made publicly available on the SMC Web site, 4 wk after the decision is made.
      Stakeholder involvement is clearly specifiedAU, BRA, CA, GER, SCOT, UKBRA: CONITEC, the body that advises the MoH in Brazil with regard to inclusion or exclusion of health technologies, is involving stakeholders in a public and transparent way, as required by law. CONITEC itself represents 13 stakeholders, including different secretariats of the MoH, as well as representatives of state and municipality health secretaries, regulatory agencies, national health council, and medical professionals. They use public consultation to inform the stakeholders and ask for their opinion and criticisms regarding the recommendations. Stakeholders can send their comments and suggestions on the technology and the recommendation (via an electronic form) to CONITEC. Since 2014, contributions can either be technical/scientific or opinions/experiences. Subsequently, the contributions received through public consultation are discussed by CONITEC and are available on the Web site. After the discussion of contributions received by CONITEC, the final (modified) recommendation is sent to the MoH for a final decision. According to CONITEC, there are some cases in which the initial recommendation has changed after public consultation.
      SCOT: SMC consists of around 40 members from different backgrounds, including physicians, health economists, health care managers, and public partners. The public partners present patient group submissions during the SMC meetings. The public partners are also part of the PIN advisory group. This network aims to improve the involvement of patients, carers, and members of the public in the evaluation of new medicines. A representative of the manufacturer of the pharmaceutical product under evaluation may attend the SMC meeting to answer the remaining questions of the committee.
      Mechanism(s) for appeal are in placeAU, CA, FRA, SCOT, KOR, UKSCOT: Manufacturers of a product for which the advice was “not recommended” might resubmit their evidence when there is new evidence available. When there is no new data available, manufacturers might request an independent review.
      KOR: According to the literature, there are several opportunities to appeal in Korea. First, a face-to-face meeting with the HIRA staff. Second, reconsideration could be requested when the submitted pharmaceutical product is rejected, which allows for the inclusion of new evidence. Third, the literature also describes an independent review process, in which an expert panel is responsible for reviewing the aspects of the appeal. Nevertheless, it is apparent from both the literature and the interviews that appeal rarely ever is accepted.
      Monitoring and evaluation of the processFRA, SCOTFRA: According to the literature, there is a re-assessment of pharmaceutical products to be maintained on the list of reimbursed pharmaceutical products. The re-assessment takes place every 5 y for pharmaceutical products listed for admission to community pharmacies and at any time for pharmaceutical products when significant new information becomes available.
      SCOT: After a governmental review on access to new medicines in 2013, several changes have been made to the HTA process in Scotland (e.g., “PACE” and SMC meetings in public). During the summer of 2016, a second review (the so-called Montgomery review) has evaluated how these changes have improved the process, especially focused on patient access to medicines for rare and end-of-life conditions.
      AU, Australia; BRA, Brazil; CA, Canada; CONITEC, National Committee for Technology Incorporation; FRA, France; GER, Germany; HIRA, Health Insurance Review and Assessment service; HITAP, Health Intervention and Technology Assessment Program; HTA, health technology assessment; KOR, South Korea; MoH, Ministry of Health; PACE, Patient and Clinician Engagement; PIN, Public Involvement Network; SCOT, Scotland; SMC, Scottish Medicines Consortium; THA, Thailand; UK, United Kingdom (England).
      It appeared that diverging views were often seen between stakeholders with regard to the involvement of stakeholders in the appraisal phase. The other criteria show a more diverse view, which might be because some criteria have not yet gained much attention in the HTA community and/or related HTA literature (e.g., patient heterogeneity and the inclusion of patient references).

      Conclusions

      Overall, we conclude that, according to the framework that we have developed and used, the HTA processes in Argentina and South Korea are at an early stage. HTA systems in Canada, Scotland, and England are almost aligned with our framework, followed by Australia, Germany, and France. The HTA processes in Brazil and Thailand are in an intermediate position. Thus, countries with a well-established HTA system, that is, Australia, Canada, England, France, Germany, and Scotland, appear to be more aligned with what we perceive as a best practice, compared with countries that have a relatively shorter history in HTA (Argentina, Brazil, South Korea, and Thailand).
      In most high-income countries, the focus lies on improvement of the HTA processes in terms of, for example, broader stakeholder involvement and increasing transparency of the processes. In middle-income countries (Argentina and Brazil), the focus also lies on enhancing capacity building and standardization of methods/development of guidelines. As stated by Kaló et al. [
      • Kaló Z.
      • Bodrogi J.
      • Boncz J.
      • et al.
      Capacity building for HTA implementation in middle-income countries: the case of Hungary.
      ], middle-income countries often do not have a clear (legal) framework for implementation of HTA results in decision-making processes. The results, however, show that even in HTA systems that already have extensive HTA activities and explicit processes in place (most high-income countries), there is a need for continuous review of processes to consolidate HTA efforts and to ensure appropriate use of HTA in health care decision making, both in policy and in practice. This requires strong support from policymakers. Political will is also important for introducing HTA in a country. This especially applies for those countries in which HTA is still in its early phase of institutionalization, that is, Argentina. Furthermore, in low- and middle-income countries it is essential to understand the local context for the development of realistic policy options. In this respect, local capacity is key, but is often limited or lacking. Therefore, sharing expertise and experiences among countries with similar structures and of comparable size is needed to improve the quality and efficiency of evidence generation as well as to support further capacity building in HTA. In this respect, it is noteworthy to mention the work recently undertaken by colleagues involved in the ADVANCE-HTA project. Part of this research project, which was also funded by the European Commission, under the 7th Framework Programme for Research and Technological Development, focused on developing appropriate tools for supporting the implementation of HTA in emerging settings [

      Espín J. HTA in emerging settings—mapping exercise and toolbox. Available from: http://www.advance-hta.eu/PDF/FinalConference/presentations/12-HTA_in_Emerging_Settings_EASP.pdf. [Accessed March 8, 2017]

      ].
      It is important to mention that this study has several limitations. First, it should be noted that the results are solely based on criteria that we selected using the INTEGRATE-HTA model and the A4R framework and which—according to us—reflect a more comprehensive view on the HTA process. These criteria do not cover the aspects of the traditional HTA process; we considered the traditional HTA process to be in place in each HTA system studied. In addition, the information presented is meant to illustrate how an HTA system operationalizes the relevant criterion. Nevertheless, best practices that may work well in some countries might not necessarily work evenly well in other countries. Furthermore, any set of criteria implies that they are equally important, but this is not the case. In reality, some criteria can be more important than others, depending on the context. Second, although the selected HTA systems have developed (explicit or implicit) HTA processes, it is emphasized that reimbursement decisions are made in the context of local health systems that reflect the historical development of the health system and the political processes of that country. It is also known that the health system will influence the role of HTA in a particular country (e.g., specific regulation on HTA is available in some countries, including Germany and Brazil). Also, countries use several approaches to assess potential pricing and reimbursement of health technology. For example, in France and Germany, the perceived level of innovation is subsequently used as a basis for pricing and reimbursement negotiations. In other countries, clinical and economic assessments are combined when appraising potential reimbursement [

      Godman B, Oortwijn W, De Waure C, et al. Links between pharmaceutical R&D models and access to affordable medicines. European Union, PE 587.321 EN. IP/A/ENVI/2015-06. 2016. Available from: http://www.europarl.europa.eu/thinktank/en/document.html?reference=IPOL_STU(2016)587321. [Accessed December 2, 2016].

      ], for example, in Australia, Canada, and England. In addition, the economic profile of a country is influencing resource allocation decisions. From the countries studied, Argentina, Brazil, and Thailand have a middle-income status [

      World Bank. Data World Bank country and lending groups. Available from: https://datahelpdesk.worldbank.org/knowledgebase/articles/906519. [Accessed November 18, 2016]

      ]. Brazil and, to a lesser extent, Argentina have health care problems related to both equity and efficiency, leading to right-to-health litigation cases. Third, although the number of interviews conducted per country is limited to three or four per HTA system, we feel confident to draw general conclusions and formulate recommendations, because we used the most recent literature whenever possible and almost all country profiles (except Brazil) are validated by at least one stakeholder per country.
      On the basis of this study, we make four recommendations targeted to all stakeholders involved in the HTA process, that is, HTA organizations/staff, advisors (academic and government), third-party payers, industry, health care providers, health professionals, patient representatives, and decision makers.
      First, despite increased attention toward the use of HTA, including the views of relevant stakeholders, transparency of processes and methods will not automatically appear as HTA develops. Adopting transparent and robust processes, including stakeholder consultation, takes time. In the long-term a transparent HTA process will increase accountability and predictability for all stakeholders, and support the adequate provision and use of pharmaceuticals, as well as other health technologies. Broad, multidisciplinary, stakeholder involvement will be necessary throughout the process. Engagement of relevant stakeholders should also start from the beginning of the HTA process, but should definitely go beyond topic scoping. It is important to include relevant stakeholders in identifying priorities to address the needs of the health care system, in setting priorities for HTA, the ways that the analyses and their conclusions can best be presented (i.e., evidence reports), and the implications of the HTA for health care policy and practice (appraisal) [

      D. Husereau (on behalf of the HTAi Policy Forum). Changing HTA paradigms. 2016 Policy Forum Background Paper, Health Technology Assessment international, Edmonton, AB, January 8, 2016.

      ,
      • McDonald H.
      • Charles C.
      • Elit L.
      • Gafni A.
      Challenges in striving to simultaneously achieve multiple resource allocation goals: the pan-Canadian Oncology Drug Review (pCDODR) example.
      ]. To this end, it is important to present HTA as a process that not only includes assessment and appraisal but also constitutes a broader process.
      Second, scoping is often not part of the HTA process. It should be emphasized that HTA is not a matter of collecting facts, but a matter of collecting facts that are considered relevant, plausible to stakeholders, and amenable to scientific inquiry [
      • Berglas S.
      • Jutai L.
      • MacKean G.
      • Weeks L.
      Patients’ perspectives can be integrated in health technology assessments: an exploratory analysis of CADTH Common Drug Review.
      ]. Scoping is therefore highly recommended to be used in each HTA system. This is likely to increase the relevance and feasibility of HTA implementation for all relevant stakeholders.
      Third, diversity in patient characteristics and patient preferences influence treatment outcomes. These aspects are increasingly considered in HTA, but there is room for improvement. The HTA should include standardized methods to identify and appraise evidence for clinically important moderators or predictors of treatment effects, and/or how patients differ in their appreciation of various treatment outcomes, to contribute to a deeper understanding of the value of health technologies.
      Fourth, monitoring and evaluation of the HTA process is not well established, even though it is considered to be a key principle of HTA [
      • Drummond M.F.
      • Schwartz J.S.
      • Jönsson B.
      • et al.
      Key principles for the improved conduct of health technology assessments for resource allocation decisions.
      ]. To demonstrate the impact of the HTA and increase publicly legitimate reimbursement decisions, it is important to monitor and review the processes and results at certain intervals (e.g., 3–5 years) to assess their efficiency, consistency, and sustainability over time.

      Acknowledgment

      We thank Marien van Schothorst (ECORYS Nederland B.V., Rotterdam, The Netherlands) for his support in drafting several country profiles.
      Source of financial support: This study was fully funded by a grant from Eli Lilly and Company (i.e., Lilly Corporate Center, Indianapolis, IN). The funding organization provided suggestions for country and interviewee selection, but the final selection was solely determined by the project team. The funding organization did not have any influence on the development of the evaluation framework, data collection, analysis, and reporting.

      References

      1. Godman B, Oortwijn W, De Waure C, et al. Links between pharmaceutical R&D models and access to affordable medicines. European Union, PE 587.321 EN. IP/A/ENVI/2015-06. 2016. Available from: http://www.europarl.europa.eu/thinktank/en/document.html?reference=IPOL_STU(2016)587321. [Accessed December 2, 2016].

      2. International Network of Agencies for Health Technology Assessment; Health Technology Assessment international. HTA glossary. Available from: http://htaglossary.net/health+technology+assessment+%28HTA%29. [Accessed November 15, 2016].

        • Tanios N.
        • Wagner M.
        • Tony M.
        • et al.
        • International Task Force on Decision Criteria
        Which criteria are considered in healthcare decisions? Insights from an international survey of policy and clinical decision makers.
        Int J Technol Assess Health Care. 2013; 29: 456-465
      3. Facey K. Improving the effectiveness and efficiency of evidence production for HTA in the light of current trends in drug and device development, health system funding, regulation and HTA. Background Paper Policy Forum. 2015. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4697336/pdf/S0266462315000355a.pdf. [Accessed April 14, 2017].

        • Klein R.
        • Day P.
        • Redmayne S.
        Managing Scarcity: Priority Setting and Rationing in the National Health Service.
        Open University Press, Buckingham, UK1996
        • Neumann P.
        • Drummond M.
        • Jönsson B.
        • et al.
        Are key principles for improved health technology assessment supported and used by health technology assessment organizations?.
        Int J Technol Assess Health Care. 2010; 26: 71-78
        • Gulácsi L.
        • Rotar A.M.
        • Niewada M.
        • et al.
        Health technology assessment in Poland, the Czech Republic, Hungary, Romania and Bulgaria.
        Eur J Health Econ. 2014; 15: 13
      4. Wahlster P, Brereton L, Burns J, et al. Guidance on the integrated assessment of complex health technologies—the INTEGRATE-HTA model. 2016. Available from: http://www.integrate-hta.eu/downloads/. [Accessed November 15, 2016]

        • Oortwijn W.J.
        • Broos P.
        • Vondeling H.
        • et al.
        Mapping of health technology assessment in selected countries.
        Int J Technol Assess Health Care. 2013; 29: 424-434
        • Adunlin G.
        • Diaby V.
        • Xiao H.
        Application of multicriteria decision analysis in health care: a systematic review and bibliometric analysis.
        Health Expect. 2015; 18: 1894-1905
        • Cho E.
        • Park E.C.
        • Kang M.S.
        Pitfalls in reimbursement decisions for oncology drugs in South Korea: need for addressing the ethical dimensions in technology assessment.
        Asian Pac J Cancer Prev. 2012; 14: 3785-3792
        • Jansen M.P.
        • Helderman J.K.
        • Boer B.
        • Baltussen R.
        Fair processes for priority setting: putting theory into practice. Comment on “Expanded HTA: enhancing fairness and legitimacy.”.
        Int J Health Policy Manag. 2016; 5: 1-5
        • Daniels N.
        • Sabin J.
        Limits to health care: fair procedures, democratic deliberation, and the legitimacy problem for insurers.
        Philos Public Aff. 1997; 26: 303-350
        • Daniels N.
        • Sabin J.
        Accountability for reasonableness: an update.
        BMJ. 2008; 337: a1850
      5. Brereton L, Wahlster P, Lysdahl KB, et al. (on behalf of the INTEGRATE-HTA project team). Integrated assessment of home based palliative care with and without reinforced caregiver support: “a demonstration of INTEGRATE-HTA methodological guidances”—executive summary. Available from: http://www.integrate-hta.eu/downloads/. [Accessed November 15, 2016]

      6. International Network of Agencies for Health Technology Assessment. INAHTA webinar on INTEGRATE-HTA. 2016. Available from: http://www.inahta.org/2016/09/new-inahta-webinars/. [Accessed November 15, 2016]

        • Daniels N.
        • Porteny T.
        • Urrutia J.
        Expanded HTA: enhancing fairness and legitimacy.
        Int J Health Policy Manag. 2016; 5: 1-3
      7. International Network of Agencies for Health Technology Assessment. A checklist for health technology assessment reports. Available from: http://www.inahta.org/wp-content/uploads/2014/04/INAHTA_HTA_Checklist_English.pdf. [Accessed November 15, 2016].

        • Barnieh L.
        • Manns B.
        • Harris A.
        • et al.
        A synthesis of drug reimbursement decision-making processes in organisation for economic co-operation and development countries.
        Value Health. 2014; 17: 98-108
        • Wildson T.
        • Fiz E.
        • Haderi A.
        A comparative analysis of the role and impact of health technology assessment: 2013.
        Final report. 2014; (Available from) ([Accessed December 2, 2016].)
        • Panteli D.
        • Eckhardt H.
        • Nolting A.
        • et al.
        From market access to patient access: overview of evidence-based approaches for the reimbursement and pricing of pharmaceuticals in 36 European countries.
        Health Res Policy Syst. 2015; 13: 39
        • Kaló Z.
        • Bodrogi J.
        • Boncz J.
        • et al.
        Capacity building for HTA implementation in middle-income countries: the case of Hungary.
        Value Health Reg Issues. 2013; 2: 264-266
      8. Espín J. HTA in emerging settings—mapping exercise and toolbox. Available from: http://www.advance-hta.eu/PDF/FinalConference/presentations/12-HTA_in_Emerging_Settings_EASP.pdf. [Accessed March 8, 2017]

      9. World Bank. Data World Bank country and lending groups. Available from: https://datahelpdesk.worldbank.org/knowledgebase/articles/906519. [Accessed November 18, 2016]

      10. D. Husereau (on behalf of the HTAi Policy Forum). Changing HTA paradigms. 2016 Policy Forum Background Paper, Health Technology Assessment international, Edmonton, AB, January 8, 2016.

        • McDonald H.
        • Charles C.
        • Elit L.
        • Gafni A.
        Challenges in striving to simultaneously achieve multiple resource allocation goals: the pan-Canadian Oncology Drug Review (pCDODR) example.
        J Mark Access Health Policy. 2016; 4 (Available at: http://journals.co-action.net/index.php/jmahp/article/view/31463. [Accessed April 14, 2017])
        • Berglas S.
        • Jutai L.
        • MacKean G.
        • Weeks L.
        Patients’ perspectives can be integrated in health technology assessments: an exploratory analysis of CADTH Common Drug Review.
        Res Involvem Engagem. 2016; 2: 21
        • Drummond M.F.
        • Schwartz J.S.
        • Jönsson B.
        • et al.
        Key principles for the improved conduct of health technology assessments for resource allocation decisions.
        Int J Technol Assess Health Care. 2008; 24: 244-258
        • Wahlster P.
        • Goetghebeur M.
        • Kriza C.
        • et al.
        Balancing costs and benefits at different stages of medical innovation: a systematic review of multi-criteria decision analysis (MCDA).
        BMC Health Serv Res. 2015; 15: 262