Advertisement

Identifying the Need for Good Practices in Health Technology Assessment: Summary of the ISPOR HTA Council Working Group Report on Good Practices in HTA

Open ArchivePublished:December 12, 2018DOI:https://doi.org/10.1016/j.jval.2018.08.010

      Abstract

      The systematic use of evidence to inform healthcare decisions, particularly health technology assessment (HTA), has gained increased recognition. HTA has become a standard policy tool for informing decision makers who must manage the entry and use of pharmaceuticals, medical devices, and other technologies (including complex interventions) within health systems, for example, through reimbursement and pricing. Despite increasing attention to HTA activities, there has been no attempt to comprehensively synthesize good practices or emerging good practices to support population-based decision-making in recent years. After the identification of some good practices through the release of the ISPOR Guidelines Index in 2013, the ISPOR HTA Council identified a need to more thoroughly review existing guidance. The purpose of this effort was to create a basis for capacity building, education, and improved consistency in approaches to HTA-informed decision-making. Our findings suggest that although many good practices have been developed in areas of assessment and some other key aspects of defining HTA processes, there are also many areas where good practices are lacking. This includes good practices in defining the organizational aspects of HTA, the use of deliberative processes, and measuring the impact of HTA. The extent to which these good practices are used and applied by HTA bodies is beyond the scope of this report, but may be of interest to future researchers.

      Introduction

      Health technology assessment (HTA) has become a standard policy tool for informing decision makers who must manage the entry and use of pharmaceuticals, medical devices, and other technologies (including complex interventions) within health systems, for example, through reimbursement and pricing. Despite increasing HTA activity, there has been no attempt to comprehensively synthesize good practices or emerging good practices to support population-based decision-making in recent years.
      The purpose of the ISPOR HTA Council Working Group was to provide an up-to-date review of current literature that includes guidance on good practices in the use of evidence to inform population-based healthcare decision-making for pharmaceuticals (drugs and vaccines), medical devices, and other health technologies, that is, HTA. Population-based decisions are those linked to management, administration, and other forms of health system governance and stewardship. The use of evidence to inform individual decisions between patients and clinicians is outside of the scope of this review; nevertheless, the Working Group recognizes that HTA may be used to broadly inform clinical practice decisions through clinical practice guidelines or clinical pathway development and thus have not excluded these from the scope of the article.
      The rationale for identifying good HTA practices in using evidence to inform population-based healthcare decision-making is to provide a basis for capacity building, education, and improved consistency in approaches to HTA-informed decision-making. The primary audiences for this report are those who manage, design, or seek to improve HTA processes, although we hope that it is informative to a wider audience of patients, care providers, payers, academics, and industry stakeholders.
      Given the large scope of this work and to achieve its objectives, the HTA Council Working Group created an overview report with a summary of key references related to good practices in HTA. The overview report outlines where guidance for good practices has been identified and where good practices are still emerging or could not be identified. This report will focus on prioritizing next steps that may be taken by ISPOR and other interested parties and is a summary of the effort. The full report can be found on the ISPOR website (https://www.ispor.org/member-groups/councils-roundtables/health-technology-assessment-council) and as a Supplementary Appendix to this article available at https://doi.org/10.1016/j.jval.2018.08.010.

      Methods

      The Working Group’s approach in developing this report was based on literature review and expert opinion. In this respect it followed a similar approach to that of ISPOR Task Forces.
      ISPOR Good Practices for Outcomes Research. n.d.
      The need for a review of best practices was first identified by the ISPOR HTA Council after a review of the ISPOR Guideline Index for Outcomes Research.
      ISPOR Guidelines Index for Outcomes Research and Use in Health Care Decision-making. Guidelines Index for Outcomes Research and Use in Health Care Decision-making. n.d.
      The council then identified co-chairs who invited members of the Working Group. An outline for the report was then drafted and reviewed by members of the Working Group.
      An early challenge for the Working Group was arriving at consistent conceptual definitions of an HTA process and its associated terminology. In the end, HTA processes were characterized using a combination of concepts derived from healthcare decision-making,
      • Eddy D.M.
      Clinical decision-making: from theory to practice. Anatomy of a decision.
      along with a description of components of an HTA process,
      • Schwarzer R.
      • Siebert U.
      Methods, procedures, and contextual characteristics of health technology assessment and health policy decision-making: comparison of health technology assessment agencies in Germany, United Kingdom, France, and Sweden.
      • Kristensen F.B.
      • Lampe K.
      • Wild C.
      • Cerbo M.
      • Goettsch W.
      • Becla L.
      The HTA Core Model®-10 Years of Developing an International Framework to Share Multidimensional Value Assessment.
      and the structure proposed by the ISPOR Guideline Index for Outcomes Research (Fig. 1).
      ISPOR Guidelines Index for Outcomes Research and Use in Health Care Decision-making. Guidelines Index for Outcomes Research and Use in Health Care Decision-making. n.d.
      The proposed framework assumes that the goal of HTA is to support healthcare decision-making, and it addresses all aspects, including how HTA processes are governed and defined (“Defining the HTA process”); how research information is identified, analyzed, and interpreted (“Assessment”); how these interpretations are applied and weighed to the context of a decision (“Contextualization”); and how this ultimate interpretation and weighting is intended to support healthcare decisions (“Implementation and Monitoring HTA”).
      Figure thumbnail gr1
      Fig. 1Components of HTA within the healthcare decision-making process. HTA indicates health technology assessment.
      Sections of the report identified through the framework were assigned and drafted by individual Working Group members who were encouraged to use comprehensive approaches toward searching for existing descriptions of current practice, guidance for best practice, and to provide expert opinion (preferably based on published reports), identifying issues related to each section assigned. Systematic reviews were typically not conducted by Working Group members, although all authors were encouraged to conduct them or identify systematic reviews in their assigned areas.
      Once drafted, the report was reviewed by all members, revised, and circulated to members of a larger review group (see Acknowledgements); it was then further revised, leading to this final report. In parallel, findings were summarized and presented at open workshops during ISPOR meetings (Boston, MA, USA, and Glasgow, Scotland, UK).

      Findings

      General Findings

      In some areas, we were unable to identify good practices specific to HTA. This included good practices in defining the organizational aspects of HTA, the use of deliberative processes, and measuring the impact of HTA. In some areas, such as guidance for the interpretation of individual studies or bodies of evidence, there was an abundance of available practice guidance that was either discipline or HTA specific.
      A summary of our findings appears in Table 1.
      • Kaufmann D.
      • Kraay A.
      Governance Indicators: Where Are We, Where Should We Be Going? The World Bank; 2007.
      World Health Organization
      The world health report 2000 - Health systems: improving performance. n.d.
      • Greer S.L.
      • Wismar M.
      • Figueras J.
      European Observatory on Health Systems and Policies.
      A Framework for Good Governance in the Pharmaceutical Sector. The Hashemite Kingdom of Jordan. n.d.
      United Nations Development Programme
      Governance for Sustainable Human Development: A UNDP Policy Document.
      • Drummond M.F.
      • Schwartz J.S.
      • Jönsson B.
      • et al.
      Key principles for the improved conduct of health technology assessments for resource allocation decisions.
      • Drummond M.
      • Neumann P.
      • Jönsson B.
      • et al.
      Can we reliably benchmark health technology assessment organizations?.
      • Oortwijn W.
      • Mathijssen J.
      • Banta D.
      The role of health technology assessment on pharmaceutical reimbursement in selected middle-income countries.
      • Oortwijn W.
      • Broos P.
      • Vondeling H.
      • Banta D.
      • Todorova L.
      Mapping of health technology assessment in selected countries.
      • Oortwijn W.
      • Determann D.
      • Schiffers K.
      • Tan S.S.
      • van der Tuin J.
      Towards integrated health technology assessment for improving decision-making in selected countries.
      • Allen N.
      • Liberti L.
      • Walker S.R.
      • Salek S.
      A comparison of reimbursement recommendations by European HTA agencies: is there opportunity for further alignment?.
      • Henshall C.
      • Oortwijn W.
      • Stevens A.
      • Granados A.
      • Banta D.
      Priority setting for health technology assessment. Theoretical considerations and practical approaches. Priority setting Subgroup of the EUR-ASSESS Project.
      • Mobinizadeh M.
      • Raeissi P.
      • Nasiripour A.A.
      • Olyaeemanesh A.
      • Tabibi S.J.
      The health systems’ priority setting criteria for selecting health technologies: a systematic review of the current evidence.
      • Noorani H.Z.
      • Husereau D.R.
      • Boudreau R.
      • Skidmore B.
      Priority setting for health technology assessments: a systematic review of current practical approaches.
      • Specchia M.L.
      • Favale M.
      • Di Nardo F.
      • et al.
      How to choose health technologies to be assessed by HTA? A review of criteria for priority setting.
      • Whitlock E.P.
      • Lopez S.A.
      • Chang S.
      • Helfand M.
      • Eder M.
      • Floyd N.
      AHRQ series paper 3: identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the effective health-care program.
      • Kristensen F.
      • Sigmund H.
      Danish Centre for Health Technology Assessment
      Health Technology Assessment Handbook.
      HTA Core Model® Online Handbook. n.d.
      NICE
      Technology appraisal guidance. NICE guidance. n.d.
      • Ormstad S.S.
      Keeping up to date with information retrieval research: Summarized Research in Information Retrieval (SuRe Info).
      • Higgins J.P.T.
      • Altman D.G.
      • Gøtzsche P.C.
      • et al.
      The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials.
      RoB 2.0 tool—Risk of bias tools. n.d.
      • Sterne J.A.
      • Hernán M.A.
      • Reeves B.C.
      • et al.
      ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
      • O’Neil M.
      • Berkman N.
      • Hartling L.
      • et al.
      Observational Evidence and Strength of Evidence Domains: Case Examples.
      NCBI Bookshelf
      Developing a Protocol for Observational Comparative Effectiveness Research: A User’s Guide. n.d.
      NICE Decision Support Unit
      Observational data TSD. n.d.
      Stratos Initiative
      STRengthening Analytical Thinking for Observational Studies. n.d.
      • Berger M.L.
      • Martin B.C.
      • Husereau D.
      • et al.
      A questionnaire to assess the relevance and credibility of observational studies to inform health care decision-making: an ISPOR-AMCP-NPC Good Practice Task Force report.
      • Walker D.G.
      • Wilson R.F.
      • Sharma R.
      • et al.
      Best Practices for Conducting Economic Evaluations in Health Care: A Systematic Review of Quality Assessment Tools.
      • Drummond M.F.
      • Sculpher M.J.
      • Claxton K.
      • Stoddart G.L.
      • Torrance G.W.
      Methods for the Economic Evaluation of Health Care Programmes.
      • Evers S.
      • Goossens M.
      • de Vet H.
      • van Tulder M.
      • Ament A.
      Criteria list for assessment of methodological quality of economic evaluations: consensus on health economic criteria.
      • Ofman J.J.
      • Sullivan S.D.
      • Neumann P.J.
      • et al.
      Examining the value and quality of health economic analyses: implications of utilizing the QHES.
      • Heintz E.
      • Gerber-Grote A.
      • Ghabri S.
      • Hamers F.F.
      • Rupel V.P.
      • et al.
      Is there a European view on health economic evaluations? Results from a synopsis of methodological guidelines used in the EUnetHTA partner countries.
      • Husereau D.
      • Drummond M.
      • Petrou S.
      • et al.
      Consolidated Health Economic Evaluation Reporting Standards (CHEERS)—Explanation and Elaboration: A Report of the ISPOR Health Economic Evaluations Publication Guidelines Task Force.
      iqwig.de. Methods Paper. n.d.
      Pharmacoeconomic Guidelines Around the World. n.d.
      • Mathes T.
      • Jacobs E.
      • Morfeld J.C.
      • Pieper D.
      Methods of international health technology assessment agencies for economic evaluations—a comparative analysis.
      • Tarn T.
      • Smith M.D.
      Pharmacoeconomic guidelines around the world.
      • Sullivan S.D.
      • Mauskopf J.A.
      • Augustovski F.
      • et al.
      Budget impact analysis-principles of good practice: report of the ISPOR 2012 Budget Impact Analysis Good Practice II Task Force.
      • Curry L.A.
      • Nembhard I.M.
      • Bradley E.H.
      Qualitative and mixed methods provide unique contributions to outcomes research.
      • Vandermause R.
      • Barg F.K.
      • Esmail L.
      • et al.
      Qualitative methods in patient-centered outcomes research.
      • Santiago-Delefosse M.
      • Bruchez C.
      • Gavin A.
      • Stephen S.L.
      • Roux P.
      Complexity of the paradigms present in quality criteria of qualitative research grids.
      • Tong A.
      • Sainsbury P.
      • Craig J.
      Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups.
      • Tong A.
      • Flemming K.
      • McInnes E.
      • Oliver S.
      • Craig J.
      Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ.
      • Moher D.
      • Schulz K.F.
      • Simera I.
      • Altman D.G.
      Guidance for developers of health research reporting guidelines.
      • Hunger T.
      • Schnell-Inderst P.
      • Sahakyan N.
      • Siebert U.
      Using expert opinion in health technology assessment: a guideline review.
      • Lehoux P.
      • Williams-Jones B.
      Mapping the integration of social and ethical issues in health technology assessment.
      • Potter B.K.
      • Avard D.
      • Graham I.D.
      • et al.
      Guidance for considering ethical, legal, and social issues in health technology assessment: application to genetic screening.
      EUnetHTA HTA Adaptation Toolkit. n.d.
      • Hofmann B.
      • Oortwijn W.
      • Bakke Lysdahl K.
      • et al.
      Integrating ethics in health technology assessment: many ways to Rome.
      • Hofmann B.
      • Cleemput I.
      • Bond K.
      • et al.
      Revealing and acknowledging value judgments in health technology assessment.
      • van der Wilt G.J.
      Healthcare technology assessment.
      • Lysdahl K.B.
      • Hofmann B.
      Complex health care interventions: characteristics relevant for ethical analysis in health technology assessment.
      • Saarni S.I.
      • Hofmann B.
      • Lampe K.
      • et al.
      Ethical analysis to improve decision-making on health technologies.
      • Assasi N.
      • Schwartz L.
      • Tarride J.-E.
      • Campbell K.
      • Goeree R.
      Methodological guidance documents for evaluation of ethical considerations in health technology assessment: a systematic review.
      • Rehfuess E.A.
      • Gerhardus A.
      INTEGRATE-HTA
      INTEGRATE-HTA: adopting and implementing an integrated perspective on complex interventions.
      • Wahlster P.
      • Brereton L.
      • Burns J.
      • et al.
      An integrated perspective on the assessment of technologies: INTEGRATE-HTA.
      • Heintz E.
      • Lintamo L.
      • Hultcrantz M.
      • et al.
      Framework for systematic identification of ethical aspects of healthcare technologies: the SBU approach.
      • Mertz M.
      • Kahrass H.
      • Strech D.
      Current state of ethics literature synthesis: a systematic review of reviews.
      • Braunack-Mayer A.J.
      Ethics and health technology assessment: handmaiden and/or critic?.
      • Saarni S.I.
      • Braunack-Mayer A.
      • Hofmann B.
      • van der Wilt G.J.
      Different methods for ethical analysis in health technology assessment: an empirical study.
      • Scott A.M.
      • Hofmann B.
      • Gutiérrez-Ibarluzea I.
      • Bakke Lysdahl K.
      • Sandman L.
      • Bombard Y.
      Q-SEA—a tool for quality assessment of ethics analyses conducted as part of health technology assessments.
      • Bond K.
      • Oremus M.
      • Duthie K.M.
      • Griener G.G.
      Ethics expertise for health technology assessment: a Canadian national survey.
      • Tarricone R.
      • Torbica A.
      • Drummond M.
      MedtecHTA Project Group
      Key recommendations from the MedtecHTA project.
      EUnetHTA
      EUnetHTA Guidelines. n.d.
      • Robinson K.A.
      • Whitlock E.P.
      • O’Neil M.E.
      • et al.
      Integration of Existing Systematic Reviews.
      • Shea B.J.
      • Grimshaw J.M.
      • Wells G.A.
      • et al.
      Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews.
      • Liberati A.
      • Altman D.G.
      • Tetzlaff J.
      • et al.
      The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration.
      • Draborg E.
      • Gyrd-Hansen D.
      • Poulsen P.B.
      • Horder M.
      International comparison of the definition and the practical application of health technology assessment.
      • Sutton A.J.
      • Abrams K.R.
      • Jones D.R.
      • et al.
      Methods for Meta-Analysis in Medical Research. n.d.
      • Jansen J.P.
      • Fleurence R.
      • Devine B.
      • et al.
      Interpreting indirect treatment comparisons and network meta-analysis for health-care decision-making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: Part 1.
      • Jansen J.P.
      • Trikalinos T.
      • Cappelleri J.C.
      • et al.
      Indirect treatment comparison/network meta-analysis study questionnaire to assess relevance and credibility to inform health care decision-making: an ISPOR-AMCP-NPC Good Practice Task Force report.
      • Brignardello-Petersen R.
      • Bonner A.
      • Alexander P.E.
      • et al.
      Advances in the GRADE approach to rate the certainty in estimates from a network meta-analysis.
      • Puhan M.A.
      • Schünemann H.J.
      • Murad M.H.
      • et al.
      A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis.
      • Hutton B.
      • Salanti G.
      • Caldwell D.M.
      • et al.
      The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations.
      • Wong G.
      • Greenhalgh T.
      • Westhorp G.
      • Buckingham J.
      • Pawson R.
      RAMESES publication standards: meta-narrative reviews.
      • Wong G.
      • Greenhalgh T.
      • Westhorp G.
      • Buckingham J.
      • Pawson R.
      RAMESES publication standards: realist syntheses.
      • France E.F.
      • Ring N.
      • Noyes J.
      • et al.
      Protocol-developing meta-ethnography reporting guidelines (eMERGe).
      • O’Cathain A.
      Assessing the Quality of Mixed Methods Research: Toward a Comprehensive Framework.
      • Lewin S.
      • Glenton C.
      • Munthe-Kaas H.
      • et al.
      Using qualitative evidence in decision-making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual).
      • Fearon J.D.
      Deliberation as discussion.
      • Baltussen R.
      • Jansen M.P.M.
      • Bijlmakers L.
      • et al.
      Value assessment frameworks for HTA agencies: the organization of evidence-informed deliberative processes.
      • Culyer A.
      Deliberative Processes in Decisions About Health Care Technologies: Combining Different Types of Evidence, Values, Algorithms and People.
      • Lomas J.
      • Culyer T.
      • Mccutcheon C.
      • Law S.
      • Tetroe J.
      Final Report: Conceptualizing and Combining Evidence for Health System Guidance.
      • Bridges J.F.P.
      • Jones C.
      Patient-based health technology assessment: a vision of the future.
      • Abelson J.
      • Giacomini M.
      • Lehoux P.
      • Gauvin F.-P.
      Bringing “the public” into health technology assessment and coverage policy decisions: from principles to practice.
      • Perfetto E.M.
      • Oehrlein E.M.
      • Boutin M.
      • Reid S.
      • Gascho E.
      Value to whom? The patient voice in the value discussion.
      • Hansen H.P.
      • Draborg E.
      • Kristensen F.B.
      Exploring qualitative research synthesis: the role of patients’ perspectives in health policy design and decision-making.
      • Mühlbacher A.C.
      Patient-centric HTA: different strokes for different folks.
      • Frank L.
      • Forsythe L.
      • Ellis L.
      • et al.
      Conceptual and practical foundations of patient engagement in research at the patient-centered outcomes research institute.
      • Bridges J.F.P.
      • Hauber A.B.
      • Marshall D.
      • et al.
      Conjoint analysis applications in health—a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force.
      • Johnson F.R.
      • Lancsar E.
      • Marshall D.
      • et al.
      Constructing experimental designs for discrete-choice experiments: report of the ISPOR Conjoint Analysis Experimental Design Good Research Practices Task Force.
      • Hauber A.B.
      • González J.M.
      • Groothuis-Oudshoorn C.G.M.
      • et al.
      Statistical methods for the analysis of discrete choice experiments: a report of the ISPOR Conjoint Analysis Good Research Practices Task Force.
      • Facey K.
      • Boivin A.
      • Gracia J.
      • et al.
      Patients’ perspectives in health technology assessment: a route to robust evidence and fair deliberation.
      EUPATI
      Guidance for patient involvement in HTA. 2016.
      • Thokala P.
      • Devlin N.
      • Marsh K.
      • et al.
      Multiple criteria decision analysis for health care decision-making—an introduction: report 1 of the ISPOR MCDA Emerging Good Practices Task Force.
      • Marsh K.
      • Goetghebeur M.
      • Thokala P.
      • Baltussen R.
      Multi-Criteria Decision Analysis to Support Healthcare Decisions.
      • Husereau D.
      • Boucher M.
      • Noorani H.
      Priority setting for health technology assessment at CADTH.
      • Danner M.
      • Hummel J.M.
      • Volz F.
      • et al.
      Integrating patients’ views into health technology assessment: Analytic hierarchy process (AHP) as a method to elicit patient preferences.
      • Castro H.
      • Tringali M.
      • Cleemput I.
      • Devriese S.
      • Leoni O.
      • Lettieri E.
      Advancing MCDA and HTA into Coverage Decision-Making. Multi-Criteria Decision Analysis to Support Healthcare Decisions.
      • Radaelli G.
      • Lettieri E.
      • Masella C.
      • Merlino L.
      • Strada A.
      • Tringali M.
      Implementation of EUnetHTA core Model® in Lombardia: the VTS framework.
      • Golan O.
      • Hansen P.
      Which health technologies should be funded? A prioritization framework based explicitly on value for money.
      • Tony M.
      • Wagner M.
      • Khoury H.
      • et al.
      Bridging health technology assessment (HTA) with multicriteria decision analyses (MCDA): field testing of the EVIDEM framework for coverage decisions by a public payer in Canada.
      • Youngkong S.
      • Baltussen R.
      • Tantivess S.
      • Mohara A.
      • Teerawattananon Y.
      Multicriteria decision analysis for including health interventions in the universal health coverage benefit package in Thailand.
      • Verguet S.
      • Kim J.J.
      • Jamison D.T.
      Extended cost-effectiveness analysis for health policy assessment: a tutorial.
      • McCabe C.
      • Claxton K.
      • Culyer A.J.
      The NICE cost-effectiveness threshold: what it is and what that means.
      • Claxton K.
      • Martin S.
      • Soares M.
      • et al.
      Methods for the estimation of the National Institute for Health and Care Excellence cost-effectiveness threshold.
      • Schwarzer R.
      • Rochau U.
      • Saverno K.
      • et al.
      Systematic overview of cost-effectiveness thresholds in ten countries across four continents.
      • Drummond M.
      • Barbieri M.
      • Cook J.
      • et al.
      Transferability of economic evaluations across jurisdictions: ISPOR Good Research Practices Task Force report.
      • Drummond M.
      • Augustovski F.
      • Kaló Z.
      • et al.
      Challenges faced in transferring economic evaluations to middle income countries.
      • Barbieri M.
      • Drummond M.
      • Willke R.
      • Chancellor J.
      • Jolain B.
      • Towse A.
      Variability of cost-effectiveness estimates for pharmaceuticals in Western Europe: lessons for inferring generalizability.
      • Barbieri M.
      • Drummond M.
      • Rutten F.
      • et al.
      What do international pharmacoeconomic guidelines say about economic data transferability?.
      • Chalon P.X.
      • Kraemer P.
      EUnetHTA information management system: development and lessons learned.
      Danske Regioner—Medicinrådet—In English. n.d.
      National Institute for Health and Care Excellence
      Assessing Resource Impact Methods Guide 2011.
      ICER
      Final Value Assessment Framework for 2017-2019. n.d.
      • Lavis J.N.
      • Robertson D.
      • Woodside J.M.
      • McLeod C.B.
      • Abelson J.
      Knowledge Transfer Study Group. How can research organizations more effectively transfer research knowledge to decision makers?.
      • Lavis J.N.
      • Oxman A.D.
      • Lewin S.
      • Fretheim A.
      SUPPORT Tools for evidence-informed health policymaking (STP).
      Health Systems Evidence. n.d.
      The AMCP format for formulary submissions version 3.0.
      EUnetHTA
      Submission template for pharmaceuticals and submission template for medical devices. n.d.
      • Kaló Z.
      • Gheorghe A.
      • Huic M.
      • Csanádi M.
      • Kristensen F.B.
      HTA implementation roadmap in Central and Eastern European countries.
      • Hailey D.
      • Werkö S.
      • Rosén M.
      • et al.
      Influence of health technology assessment and its measurement.
      • Gerhardus A.
      • Dorendorf E.
      • Røttingen J.-A.
      • Santamera A.S.
      What are the effects of HTA reports on the health system? Evidence from the research literature.
      • Gerhardus A.
      • Dintsios C.-M.
      The impact of HTA reports on health policy: a systematic review.
      • Raftery J.
      • Hanney S.
      • Greenhalgh T.
      • Glover M.
      • Blatch-Jones A.
      Models and applications for measuring the impact of health research: update of a systematic review for the Health Technology Assessment programme.
      • Guthrie S.
      • Hafner M.
      • Bienkowska-Gibbs T.
      • Wooding S.
      Returns on research funded under the NIHR Health Technology Assessment (HTA) Programme: economic analysis and case studies.
      • Jacob R.
      • McGregor M.
      Assessing the impact of health technology assessment.
      • Schumacher I.
      • Zechmeister I.
      Assessing the impact of health technology assessment on the Austrian healthcare system.
      • Zechmeister I.
      • Schumacher I.
      The impact of health technology assessment reports on decision-making in Austria.
      • Callea G.
      • Armeni P.
      • Marsilio M.
      • Jommi C.
      • Tarricone R.
      The impact of HTA and procurement practices on the selection and prices of medical devices.
      Table 1Summary of findings
      HTA PracticeGood practices identifiedExample(s)NotesReferences
      Define the HTA process
       Structure/governance/organizational aspects of HTAFewWHO and World Bank frameworksNot specific to HTA
      • Kaufmann D.
      • Kraay A.
      Governance Indicators: Where Are We, Where Should We Be Going? The World Bank; 2007.
      World Health Organization
      The world health report 2000 - Health systems: improving performance. n.d.
      • Greer S.L.
      • Wismar M.
      • Figueras J.
      European Observatory on Health Systems and Policies.
      A Framework for Good Governance in the Pharmaceutical Sector. The Hashemite Kingdom of Jordan. n.d.
      United Nations Development Programme
      Governance for Sustainable Human Development: A UNDP Policy Document.
       Framework/principles for the HTA processYesVariousSome developed for comparison and benchmarking
      • Drummond M.F.
      • Schwartz J.S.
      • Jönsson B.
      • et al.
      Key principles for the improved conduct of health technology assessments for resource allocation decisions.
      • Drummond M.
      • Neumann P.
      • Jönsson B.
      • et al.
      Can we reliably benchmark health technology assessment organizations?.
      • Oortwijn W.
      • Mathijssen J.
      • Banta D.
      The role of health technology assessment on pharmaceutical reimbursement in selected middle-income countries.
      • Oortwijn W.
      • Broos P.
      • Vondeling H.
      • Banta D.
      • Todorova L.
      Mapping of health technology assessment in selected countries.
      • Oortwijn W.
      • Determann D.
      • Schiffers K.
      • Tan S.S.
      • van der Tuin J.
      Towards integrated health technology assessment for improving decision-making in selected countries.
      • Allen N.
      • Liberti L.
      • Walker S.R.
      • Salek S.
      A comparison of reimbursement recommendations by European HTA agencies: is there opportunity for further alignment?.
       Priority setting processYesEUnetHTA procedure
      • Henshall C.
      • Oortwijn W.
      • Stevens A.
      • Granados A.
      • Banta D.
      Priority setting for health technology assessment. Theoretical considerations and practical approaches. Priority setting Subgroup of the EUR-ASSESS Project.
      • Mobinizadeh M.
      • Raeissi P.
      • Nasiripour A.A.
      • Olyaeemanesh A.
      • Tabibi S.J.
      The health systems’ priority setting criteria for selecting health technologies: a systematic review of the current evidence.
      • Noorani H.Z.
      • Husereau D.R.
      • Boudreau R.
      • Skidmore B.
      Priority setting for health technology assessments: a systematic review of current practical approaches.
      • Specchia M.L.
      • Favale M.
      • Di Nardo F.
      • et al.
      How to choose health technologies to be assessed by HTA? A review of criteria for priority setting.
      • Whitlock E.P.
      • Lopez S.A.
      • Chang S.
      • Helfand M.
      • Eder M.
      • Floyd N.
      AHRQ series paper 3: identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the effective health-care program.
       Framing and scopingYesHTA Core Model, Danish guidelines, NICEAssumed many scoping processes unpublished
      • Kristensen F.
      • Sigmund H.
      Danish Centre for Health Technology Assessment
      Health Technology Assessment Handbook.
      HTA Core Model® Online Handbook. n.d.
      NICE
      Technology appraisal guidance. NICE guidance. n.d.
      Assessment (synthesizing evidence)
       Identifying and interpreting individual studiesYesSummarized Research in Information Retrieval for HTA (SuRe Info)

      Cochrane Risk of Bias Tools

      EuNetHTA Guidance

      ISPOR-AMCP-NPC Good Practice Task Force Questionnaire

      MedTecHTA Recommendations

      HTA Core Model
      Tools for some study types still nascent
      NICE
      Technology appraisal guidance. NICE guidance. n.d.
      • Ormstad S.S.
      Keeping up to date with information retrieval research: Summarized Research in Information Retrieval (SuRe Info).
      • Higgins J.P.T.
      • Altman D.G.
      • Gøtzsche P.C.
      • et al.
      The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials.
      RoB 2.0 tool—Risk of bias tools. n.d.
      • Sterne J.A.
      • Hernán M.A.
      • Reeves B.C.
      • et al.
      ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
      • O’Neil M.
      • Berkman N.
      • Hartling L.
      • et al.
      Observational Evidence and Strength of Evidence Domains: Case Examples.
      NCBI Bookshelf
      Developing a Protocol for Observational Comparative Effectiveness Research: A User’s Guide. n.d.
      NICE Decision Support Unit
      Observational data TSD. n.d.
      Stratos Initiative
      STRengthening Analytical Thinking for Observational Studies. n.d.
      • Berger M.L.
      • Martin B.C.
      • Husereau D.
      • et al.
      A questionnaire to assess the relevance and credibility of observational studies to inform health care decision-making: an ISPOR-AMCP-NPC Good Practice Task Force report.
      • Walker D.G.
      • Wilson R.F.
      • Sharma R.
      • et al.
      Best Practices for Conducting Economic Evaluations in Health Care: A Systematic Review of Quality Assessment Tools.
      • Drummond M.F.
      • Sculpher M.J.
      • Claxton K.
      • Stoddart G.L.
      • Torrance G.W.
      Methods for the Economic Evaluation of Health Care Programmes.
      • Evers S.
      • Goossens M.
      • de Vet H.
      • van Tulder M.
      • Ament A.
      Criteria list for assessment of methodological quality of economic evaluations: consensus on health economic criteria.
      • Ofman J.J.
      • Sullivan S.D.
      • Neumann P.J.
      • et al.
      Examining the value and quality of health economic analyses: implications of utilizing the QHES.
      • Heintz E.
      • Gerber-Grote A.
      • Ghabri S.
      • Hamers F.F.
      • Rupel V.P.
      • et al.
      Is there a European view on health economic evaluations? Results from a synopsis of methodological guidelines used in the EUnetHTA partner countries.
      • Husereau D.
      • Drummond M.
      • Petrou S.
      • et al.
      Consolidated Health Economic Evaluation Reporting Standards (CHEERS)—Explanation and Elaboration: A Report of the ISPOR Health Economic Evaluations Publication Guidelines Task Force.
      iqwig.de. Methods Paper. n.d.
      Pharmacoeconomic Guidelines Around the World. n.d.
      • Mathes T.
      • Jacobs E.
      • Morfeld J.C.
      • Pieper D.
      Methods of international health technology assessment agencies for economic evaluations—a comparative analysis.
      • Tarn T.
      • Smith M.D.
      Pharmacoeconomic guidelines around the world.
      • Sullivan S.D.
      • Mauskopf J.A.
      • Augustovski F.
      • et al.
      Budget impact analysis-principles of good practice: report of the ISPOR 2012 Budget Impact Analysis Good Practice II Task Force.
      • Curry L.A.
      • Nembhard I.M.
      • Bradley E.H.
      Qualitative and mixed methods provide unique contributions to outcomes research.
      • Vandermause R.
      • Barg F.K.
      • Esmail L.
      • et al.
      Qualitative methods in patient-centered outcomes research.
      • Santiago-Delefosse M.
      • Bruchez C.
      • Gavin A.
      • Stephen S.L.
      • Roux P.
      Complexity of the paradigms present in quality criteria of qualitative research grids.
      • Tong A.
      • Sainsbury P.
      • Craig J.
      Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups.
      • Tong A.
      • Flemming K.
      • McInnes E.
      • Oliver S.
      • Craig J.
      Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ.
      • Moher D.
      • Schulz K.F.
      • Simera I.
      • Altman D.G.
      Guidance for developers of health research reporting guidelines.
      • Hunger T.
      • Schnell-Inderst P.
      • Sahakyan N.
      • Siebert U.
      Using expert opinion in health technology assessment: a guideline review.
      • Lehoux P.
      • Williams-Jones B.
      Mapping the integration of social and ethical issues in health technology assessment.
      • Potter B.K.
      • Avard D.
      • Graham I.D.
      • et al.
      Guidance for considering ethical, legal, and social issues in health technology assessment: application to genetic screening.
      EUnetHTA HTA Adaptation Toolkit. n.d.
      • Hofmann B.
      • Oortwijn W.
      • Bakke Lysdahl K.
      • et al.
      Integrating ethics in health technology assessment: many ways to Rome.
      • Hofmann B.
      • Cleemput I.
      • Bond K.
      • et al.
      Revealing and acknowledging value judgments in health technology assessment.
      • van der Wilt G.J.
      Healthcare technology assessment.
      • Lysdahl K.B.
      • Hofmann B.
      Complex health care interventions: characteristics relevant for ethical analysis in health technology assessment.
      • Saarni S.I.
      • Hofmann B.
      • Lampe K.
      • et al.
      Ethical analysis to improve decision-making on health technologies.
      • Assasi N.
      • Schwartz L.
      • Tarride J.-E.
      • Campbell K.
      • Goeree R.
      Methodological guidance documents for evaluation of ethical considerations in health technology assessment: a systematic review.
      • Rehfuess E.A.
      • Gerhardus A.
      INTEGRATE-HTA
      INTEGRATE-HTA: adopting and implementing an integrated perspective on complex interventions.
      • Wahlster P.
      • Brereton L.
      • Burns J.
      • et al.
      An integrated perspective on the assessment of technologies: INTEGRATE-HTA.
      • Heintz E.
      • Lintamo L.
      • Hultcrantz M.
      • et al.
      Framework for systematic identification of ethical aspects of healthcare technologies: the SBU approach.
      • Mertz M.
      • Kahrass H.
      • Strech D.
      Current state of ethics literature synthesis: a systematic review of reviews.
      • Braunack-Mayer A.J.
      Ethics and health technology assessment: handmaiden and/or critic?.
      • Saarni S.I.
      • Braunack-Mayer A.
      • Hofmann B.
      • van der Wilt G.J.
      Different methods for ethical analysis in health technology assessment: an empirical study.
      • Scott A.M.
      • Hofmann B.
      • Gutiérrez-Ibarluzea I.
      • Bakke Lysdahl K.
      • Sandman L.
      • Bombard Y.
      Q-SEA—a tool for quality assessment of ethics analyses conducted as part of health technology assessments.
      • Bond K.
      • Oremus M.
      • Duthie K.M.
      • Griener G.G.
      Ethics expertise for health technology assessment: a Canadian national survey.
      • Tarricone R.
      • Torbica A.
      • Drummond M.
      MedtecHTA Project Group
      Key recommendations from the MedtecHTA project.
      EUnetHTA
      EUnetHTA Guidelines. n.d.
      • Robinson K.A.
      • Whitlock E.P.
      • O’Neil M.E.
      • et al.
      Integration of Existing Systematic Reviews.
       Interpreting bodies of evidenceYesAssessing methodological quality of systematic reviews (AMSTAR) tool

      GRADE-CERQual
      • Robinson K.A.
      • Whitlock E.P.
      • O’Neil M.E.
      • et al.
      Integration of Existing Systematic Reviews.
      • Shea B.J.
      • Grimshaw J.M.
      • Wells G.A.
      • et al.
      Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews.
      • Liberati A.
      • Altman D.G.
      • Tetzlaff J.
      • et al.
      The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration.
      • Draborg E.
      • Gyrd-Hansen D.
      • Poulsen P.B.
      • Horder M.
      International comparison of the definition and the practical application of health technology assessment.
      • Sutton A.J.
      • Abrams K.R.
      • Jones D.R.
      • et al.
      Methods for Meta-Analysis in Medical Research. n.d.
      • Jansen J.P.
      • Fleurence R.
      • Devine B.
      • et al.
      Interpreting indirect treatment comparisons and network meta-analysis for health-care decision-making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: Part 1.
      • Jansen J.P.
      • Trikalinos T.
      • Cappelleri J.C.
      • et al.
      Indirect treatment comparison/network meta-analysis study questionnaire to assess relevance and credibility to inform health care decision-making: an ISPOR-AMCP-NPC Good Practice Task Force report.
      • Brignardello-Petersen R.
      • Bonner A.
      • Alexander P.E.
      • et al.
      Advances in the GRADE approach to rate the certainty in estimates from a network meta-analysis.
      • Puhan M.A.
      • Schünemann H.J.
      • Murad M.H.
      • et al.
      A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis.
      • Hutton B.
      • Salanti G.
      • Caldwell D.M.
      • et al.
      The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations.
      • Wong G.
      • Greenhalgh T.
      • Westhorp G.
      • Buckingham J.
      • Pawson R.
      RAMESES publication standards: meta-narrative reviews.
      • Wong G.
      • Greenhalgh T.
      • Westhorp G.
      • Buckingham J.
      • Pawson R.
      RAMESES publication standards: realist syntheses.
      • France E.F.
      • Ring N.
      • Noyes J.
      • et al.
      Protocol-developing meta-ethnography reporting guidelines (eMERGe).
      • O’Cathain A.
      Assessing the Quality of Mixed Methods Research: Toward a Comprehensive Framework.
      • Lewin S.
      • Glenton C.
      • Munthe-Kaas H.
      • et al.
      Using qualitative evidence in decision-making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual).
      Contextualizing (using evidence)
       Deliberative processesFewOHTAC Deliberative FrameworkFew good practices dedicated to HTA
      • Fearon J.D.
      Deliberation as discussion.
      • Baltussen R.
      • Jansen M.P.M.
      • Bijlmakers L.
      • et al.
      Value assessment frameworks for HTA agencies: the organization of evidence-informed deliberative processes.
      • Culyer A.
      Deliberative Processes in Decisions About Health Care Technologies: Combining Different Types of Evidence, Values, Algorithms and People.
      • Lomas J.
      • Culyer T.
      • Mccutcheon C.
      • Law S.
      • Tetroe J.
      Final Report: Conceptualizing and Combining Evidence for Health System Guidance.
       Patient engagement and patient preferencesYesHTAi Values and Preferences ToolMany approaches
      • Bridges J.F.P.
      • Jones C.
      Patient-based health technology assessment: a vision of the future.
      • Abelson J.
      • Giacomini M.
      • Lehoux P.
      • Gauvin F.-P.
      Bringing “the public” into health technology assessment and coverage policy decisions: from principles to practice.
      • Perfetto E.M.
      • Oehrlein E.M.
      • Boutin M.
      • Reid S.
      • Gascho E.
      Value to whom? The patient voice in the value discussion.
      • Hansen H.P.
      • Draborg E.
      • Kristensen F.B.
      Exploring qualitative research synthesis: the role of patients’ perspectives in health policy design and decision-making.
      • Mühlbacher A.C.
      Patient-centric HTA: different strokes for different folks.
      • Frank L.
      • Forsythe L.
      • Ellis L.
      • et al.
      Conceptual and practical foundations of patient engagement in research at the patient-centered outcomes research institute.
      • Bridges J.F.P.
      • Hauber A.B.
      • Marshall D.
      • et al.
      Conjoint analysis applications in health—a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force.
      • Johnson F.R.
      • Lancsar E.
      • Marshall D.
      • et al.
      Constructing experimental designs for discrete-choice experiments: report of the ISPOR Conjoint Analysis Experimental Design Good Research Practices Task Force.
      • Hauber A.B.
      • González J.M.
      • Groothuis-Oudshoorn C.G.M.
      • et al.
      Statistical methods for the analysis of discrete choice experiments: a report of the ISPOR Conjoint Analysis Good Research Practices Task Force.
      • Facey K.
      • Boivin A.
      • Gracia J.
      • et al.
      Patients’ perspectives in health technology assessment: a route to robust evidence and fair deliberation.
      EUPATI
      Guidance for patient involvement in HTA. 2016.
       Weighted stakeholder preferences and multicriteria decision analysisYesEVIDEM
      • Thokala P.
      • Devlin N.
      • Marsh K.
      • et al.
      Multiple criteria decision analysis for health care decision-making—an introduction: report 1 of the ISPOR MCDA Emerging Good Practices Task Force.
      • Marsh K.
      • Goetghebeur M.
      • Thokala P.
      • Baltussen R.
      Multi-Criteria Decision Analysis to Support Healthcare Decisions.
      • Husereau D.
      • Boucher M.
      • Noorani H.
      Priority setting for health technology assessment at CADTH.
      • Danner M.
      • Hummel J.M.
      • Volz F.
      • et al.
      Integrating patients’ views into health technology assessment: Analytic hierarchy process (AHP) as a method to elicit patient preferences.
      • Castro H.
      • Tringali M.
      • Cleemput I.
      • Devriese S.
      • Leoni O.
      • Lettieri E.
      Advancing MCDA and HTA into Coverage Decision-Making. Multi-Criteria Decision Analysis to Support Healthcare Decisions.
      • Radaelli G.
      • Lettieri E.
      • Masella C.
      • Merlino L.
      • Strada A.
      • Tringali M.
      Implementation of EUnetHTA core Model® in Lombardia: the VTS framework.
      • Golan O.
      • Hansen P.
      Which health technologies should be funded? A prioritization framework based explicitly on value for money.
      • Tony M.
      • Wagner M.
      • Khoury H.
      • et al.
      Bridging health technology assessment (HTA) with multicriteria decision analyses (MCDA): field testing of the EVIDEM framework for coverage decisions by a public payer in Canada.
      • Youngkong S.
      • Baltussen R.
      • Tantivess S.
      • Mohara A.
      • Teerawattananon Y.
      Multicriteria decision analysis for including health interventions in the universal health coverage benefit package in Thailand.
       Use of thresholdsYesUK NICESpecific to certain health systems
      • Verguet S.
      • Kim J.J.
      • Jamison D.T.
      Extended cost-effectiveness analysis for health policy assessment: a tutorial.
      • McCabe C.
      • Claxton K.
      • Culyer A.J.
      The NICE cost-effectiveness threshold: what it is and what that means.
      • Claxton K.
      • Martin S.
      • Soares M.
      • et al.
      Methods for the estimation of the National Institute for Health and Care Excellence cost-effectiveness threshold.
      • Schwarzer R.
      • Rochau U.
      • Saverno K.
      • et al.
      Systematic overview of cost-effectiveness thresholds in ten countries across four continents.
       Interpreting or adapting HTAs from other jurisdictionsYesEUnetHTA adaptation checklist

      ISPOR Good Research Practices Task Force report on transferability of economic evaluations
      Specific guidance for economic evaluation also available
      EUnetHTA HTA Adaptation Toolkit. n.d.
      • Drummond M.
      • Barbieri M.
      • Cook J.
      • et al.
      Transferability of economic evaluations across jurisdictions: ISPOR Good Research Practices Task Force report.
      • Drummond M.
      • Augustovski F.
      • Kaló Z.
      • et al.
      Challenges faced in transferring economic evaluations to middle income countries.
      • Barbieri M.
      • Drummond M.
      • Willke R.
      • Chancellor J.
      • Jolain B.
      • Towse A.
      Variability of cost-effectiveness estimates for pharmaceuticals in Western Europe: lessons for inferring generalizability.
      • Barbieri M.
      • Drummond M.
      • Rutten F.
      • et al.
      What do international pharmacoeconomic guidelines say about economic data transferability?.
      • Chalon P.X.
      • Kraemer P.
      EUnetHTA information management system: development and lessons learned.
       Use of budget impact analysesFewInstitute for Clinical and Economic Review
      Danske Regioner—Medicinrådet—In English. n.d.
      National Institute for Health and Care Excellence
      Assessing Resource Impact Methods Guide 2011.
      ICER
      Final Value Assessment Framework for 2017-2019. n.d.
      Implementing and monitoring HTA
       Implementing HTAYesSUPPORT ToolsDifferent approaches
      • Lavis J.N.
      • Robertson D.
      • Woodside J.M.
      • McLeod C.B.
      • Abelson J.
      Knowledge Transfer Study Group. How can research organizations more effectively transfer research knowledge to decision makers?.
      • Lavis J.N.
      • Oxman A.D.
      • Lewin S.
      • Fretheim A.
      SUPPORT Tools for evidence-informed health policymaking (STP).
      Health Systems Evidence. n.d.
      The AMCP format for formulary submissions version 3.0.
      EUnetHTA
      Submission template for pharmaceuticals and submission template for medical devices. n.d.
      • Kaló Z.
      • Gheorghe A.
      • Huic M.
      • Csanádi M.
      • Kristensen F.B.
      HTA implementation roadmap in Central and Eastern European countries.
       Measuring HTA ImpactFew“Six step” model
      • Hailey D.
      • Werkö S.
      • Rosén M.
      • et al.
      Influence of health technology assessment and its measurement.
      • Gerhardus A.
      • Dorendorf E.
      • Røttingen J.-A.
      • Santamera A.S.
      What are the effects of HTA reports on the health system? Evidence from the research literature.
      • Gerhardus A.
      • Dintsios C.-M.
      The impact of HTA reports on health policy: a systematic review.
      • Raftery J.
      • Hanney S.
      • Greenhalgh T.
      • Glover M.
      • Blatch-Jones A.
      Models and applications for measuring the impact of health research: update of a systematic review for the Health Technology Assessment programme.
      • Guthrie S.
      • Hafner M.
      • Bienkowska-Gibbs T.
      • Wooding S.
      Returns on research funded under the NIHR Health Technology Assessment (HTA) Programme: economic analysis and case studies.
      • Jacob R.
      • McGregor M.
      Assessing the impact of health technology assessment.
      • Schumacher I.
      • Zechmeister I.
      Assessing the impact of health technology assessment on the Austrian healthcare system.
      • Zechmeister I.
      • Schumacher I.
      The impact of health technology assessment reports on decision-making in Austria.
      • Callea G.
      • Armeni P.
      • Marsilio M.
      • Jommi C.
      • Tarricone R.
      The impact of HTA and procurement practices on the selection and prices of medical devices.
      HTA indicates health technology assessment; WHO, World Health Organization.

      Discussion

      Twenty years ago, the EUR-ASSESS Project made it clear that HTA is not defined by a set of methods but by its intent, and given the wide scope of HTA, it should not be viewed as a single discipline or field. Rather, HTA is multidisciplinary and rooted in good practices in evaluation, including sound research methods.
      • Banta D.
      What is technology assessment?.
      Today, HTA still uses a range of approaches intended to inform decision-making and based in research. There is now a more widely shared understanding of the standards that HTA should aim to meet and understanding of the importance of developing, agreeing, and implementing good practices.
      Our findings suggest that many good practices have been developed in areas of assessment and in some aspects of defining HTA processes (priority setting, framing, and scoping principles, as well as in areas of implementation). Few good practices were found related to structure, governance, or organizational aspects of HTA and measuring HTA impact.
      Using these underlying concepts, the challenge for the Working Group was to arrive at consensus regarding the extent to which good practices can be identified and are available. The wide scope of this overview and the approach taken to search and identify relevant guidance, coupled with many approaches not widely publicized and a rapidly growing literature, means that it is possible that some good practices may have been overlooked. The Working Group also acknowledges that regional practices also vary according to resource constraints and health system structures, although this implies there can never be a “one-size-fits-all” approach to HTA. This is, however, not an excuse for applying substandard approaches that may ultimately undermine the intent of HTA.
      HTA, encompassing evidence synthesis, may be viewed as informing evidence-based decision-making—two related but distinct concepts.
      • Teutsch S.M.
      • Berger M.L.
      Evidence synthesis and evidence-based decision-making: related but distinct processes.
      The process of rigorous review and synthesis of scientific evidence focuses on assessing the relative benefits, harms, and costs of healthcare technologies using sound analytic judgments. Evidence-based decision-making, in most cases, explicitly or implicitly incorporates other considerations (eg, affordability, ethical issues, feasibility, and acceptability) that may require mechanisms of contextualization of assessment results, such as deliberative processes, to support them.
      These latter considerations, the discussion of which is sometimes called “appraisal,” can be supported or coordinated by HTA bodies and have recently received heightened attention; their crucial importance in HTA has been recognized. This has led to a fuzzy distinction between the activities of HTA and decision-making, particularly in processes of contextualization, for example, in appraisal and reimbursement committees and the recommendations that come from them. Such recommendations may involve both analytic judgments (such as willingness to include indirect comparison and surrogate endpoints as source of evidence or how quality adjusted life years [QALYs] were derived) and consideration of social values (such as weighing the value of a QALY in the very young or old).
      The ability of decision makers to override recommendations of HTA bodies, based on other considerations and variations in approaches to HTA, makes its role even more difficult to discern, even to experts in the field.
      • Akehurst R.L.
      • Abadie E.
      • Renaudin N.
      • Sarkozy F.
      Variation in health technology assessment and reimbursement processes in Europe.
      This has led to much criticism of HTA in recent years, resulting from the decision-making processes and the extent to which they are transparent and deliberative. Unfortunately, this criticism may result in some spillover and skepticism regarding the assessment process. The future acceptance of HTA may depend on greater clarity regarding the scope of these two processes, largely identified with “assessment” and “contextualization” in this document, and additional measures to enhance the transparency by decision makers regarding the key elements that actually are driving decisions.
      Moving systematic review and synthesis beyond clinical, epidemiological, and economic research into qualitative and quantitative research in patient-, caregiver-, and citizen-generated information (such as perceptions, valuation, and outcomes) is an immediate need in HTA. As part of this effort, there is a need for more research into the structured approaches to deliberative decision making. Such research could potentially support the application of multicriteria decision analysis
      • Garrison L.P.
      • Neumann P.J.
      • Willke R.J.
      • et al.
      A health economics approach to US value assessment frameworks—summary and recommendations of the ISPOR Special Task Force Report [7].
      or other promising methods of integrating social values. This will represent a continuation of the EUR-ASSESS approach as implemented in the HTA Core Model and would help further “populate” the nonclinical domains of the model such as “patient and social” and organizational aspects with good methodologies and more evidence.
      Beyond a clear delineation of the roles of HTA and decision making (as well as scientific judgment and value judgment), HTA bodies may also need to consider what healthcare decisions are best supported by HTA. The move to early dialog and scientific advice on evidence generation to technology developers can be seen as advancement toward more constructive HTA processes, where alignment between patients, payers, regulators, and technology producers is created through shared information requirements and collaborative planning.
      • Husereau D.
      • Henshall C.
      • Sampietro-Colom L.
      • Thomas S.
      Changing health technology assessment paradigms?.
      EUnetHTA
      Early dialogue consolidated procedure. n.d.
      It is also a stepping stone to HTA, considering the costs of innovation, when informing healthcare decision makers. Recognition of the overlapping roles of regulatory and HTA processes is another key area of evolution and development for HTA.
      • Henshall C.
      • Mardhani-Bayne L.
      • Frønsdal K.B.
      • Klemp M.
      Interactions between health technology assessment, coverage, and regulatory processes: emerging issues, goals, and opportunities.
      EUnetHTA
      EMA and EUnetHTA finalise joint work plan for 2017-2020. n.d.
      Efforts by researchers in the disciplines that contribute to HTA will undoubtedly continue to include review of their own good practices and produce guidelines and textbooks that will have immediate relevance for HTA. Taken together, priorities for good practice guidance in HTA, as reflected in this article and the ISPOR Outcomes Research Guidelines Index,
      ISPOR Guidelines Index for Outcomes Research and Use in Health Care Decision-making. Guidelines Index for Outcomes Research and Use in Health Care Decision-making. n.d.
      will likely need to focus on developing good practices in using evidence to support decision-making through monitoring of HTA implementation and its input to various types of decision-making, rather than concentrating the focus of guidance production on HTA research practices (eg, evidence review and synthesis, outcomes research, and health economics), while encouraging and increasingly building on high-quality research guidance from these “contributing” fields of research. With the evolving ISPOR Guidelines Index and this review of current guidance, it may be easier to prioritize where efforts should be put in developing good practices in HTA.

      Acknowledgements

      We are sincerely grateful to the following people, who generously gave their time and shared their expertise by submitting written comments on a draft version of this article: Meindert Boysen, Karen Facey, Ansgar Gerhardus, Clifford Goodman, IQWiG, Zoltán Kaló, Sukyeong Kim, Bryan Luce, Aurelio Mejía, Andrew Mitchell, Wija Oortwijn, Matthias Perleth, Maureen Smith, Brian Solow, Lizzie Thomas, Janet Wale, Richard Willke, Ingrid Zechmeister-Koss, and Kun Zhao. The steady and capable support of ISPOR and its staff, especially Kelly Lenahan, is also genuinely appreciated.

      Supplementary Materials

      References

      1. ISPOR Good Practices for Outcomes Research. n.d.
      2. ISPOR Guidelines Index for Outcomes Research and Use in Health Care Decision-making. Guidelines Index for Outcomes Research and Use in Health Care Decision-making. n.d.
        (Accessed August 1, 2018)
        • Eddy D.M.
        Clinical decision-making: from theory to practice. Anatomy of a decision.
        JAMA. 1990; 263: 441-443
        • Schwarzer R.
        • Siebert U.
        Methods, procedures, and contextual characteristics of health technology assessment and health policy decision-making: comparison of health technology assessment agencies in Germany, United Kingdom, France, and Sweden.
        Int J Technol Assess Health Care. 2009; 25: 305-314
        • Kristensen F.B.
        • Lampe K.
        • Wild C.
        • Cerbo M.
        • Goettsch W.
        • Becla L.
        The HTA Core Model®-10 Years of Developing an International Framework to Share Multidimensional Value Assessment.
        Value Health. 2017; 20: 244-250
        • Kaufmann D.
        • Kraay A.
        Governance Indicators: Where Are We, Where Should We Be Going? The World Bank; 2007.
        (Accessed January 24, 2018.)
        • World Health Organization
        The world health report 2000 - Health systems: improving performance. n.d.
        http://www.who.int/whr/2000/en/
        Date accessed: June 26, 2011
        • Greer S.L.
        • Wismar M.
        • Figueras J.
        European Observatory on Health Systems and Policies.
        in: Strengthening Health System Governance: Better Policies, Stronger Performance. Open University Press, Maidenhead, England2016
      3. A Framework for Good Governance in the Pharmaceutical Sector. The Hashemite Kingdom of Jordan. n.d.
        http://apps.who.int/medicinedocs/en/d/Js17057e/
        Date accessed: September 2, 2017
        • United Nations Development Programme
        Governance for Sustainable Human Development: A UNDP Policy Document.
        United Nations Development Programme, New York1997
        • Drummond M.F.
        • Schwartz J.S.
        • Jönsson B.
        • et al.
        Key principles for the improved conduct of health technology assessments for resource allocation decisions.
        Int J Technol Assess Health Care. 2008; 24 (discussion 362-368): 244-258
        • Drummond M.
        • Neumann P.
        • Jönsson B.
        • et al.
        Can we reliably benchmark health technology assessment organizations?.
        Int J Technol Assess Health Care. 2012; 28: 159-165
        • Oortwijn W.
        • Mathijssen J.
        • Banta D.
        The role of health technology assessment on pharmaceutical reimbursement in selected middle-income countries.
        Health Policy. 2010; 95: 174-184
        • Oortwijn W.
        • Broos P.
        • Vondeling H.
        • Banta D.
        • Todorova L.
        Mapping of health technology assessment in selected countries.
        Int J Technol Assess Health Care. 2013; 29: 424-434
        • Oortwijn W.
        • Determann D.
        • Schiffers K.
        • Tan S.S.
        • van der Tuin J.
        Towards integrated health technology assessment for improving decision-making in selected countries.
        Value Health. 2017; 20: 1121-1130
        • Allen N.
        • Liberti L.
        • Walker S.R.
        • Salek S.
        A comparison of reimbursement recommendations by European HTA agencies: is there opportunity for further alignment?.
        Front Pharmacol. 2017; 8: 384
        • Henshall C.
        • Oortwijn W.
        • Stevens A.
        • Granados A.
        • Banta D.
        Priority setting for health technology assessment. Theoretical considerations and practical approaches. Priority setting Subgroup of the EUR-ASSESS Project.
        Int J Technol Assess Health Care. 1997; 13: 144-185
        • Mobinizadeh M.
        • Raeissi P.
        • Nasiripour A.A.
        • Olyaeemanesh A.
        • Tabibi S.J.
        The health systems’ priority setting criteria for selecting health technologies: a systematic review of the current evidence.
        Med J Islam Repub Iran. 2016; 30: 329
        • Noorani H.Z.
        • Husereau D.R.
        • Boudreau R.
        • Skidmore B.
        Priority setting for health technology assessments: a systematic review of current practical approaches.
        Int J Technol Assess Health Care. 2007; 23: 310-315
        • Specchia M.L.
        • Favale M.
        • Di Nardo F.
        • et al.
        How to choose health technologies to be assessed by HTA? A review of criteria for priority setting.
        Epidemiol Prev. 2015; 39: 39-44
        • Whitlock E.P.
        • Lopez S.A.
        • Chang S.
        • Helfand M.
        • Eder M.
        • Floyd N.
        AHRQ series paper 3: identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the effective health-care program.
        J Clin Epidemiol. 2010; 63: 491-501
        • Kristensen F.
        • Sigmund H.
        • Danish Centre for Health Technology Assessment
        Health Technology Assessment Handbook.
        Danish Centre for Health Technology Assessment, Copenhagen2007
      4. HTA Core Model® Online Handbook. n.d.
        https://www.eunethta.eu/hta-core-model/
        Date accessed: December 8, 2017
        • NICE
        Technology appraisal guidance. NICE guidance. n.d.
        • Ormstad S.S.
        Keeping up to date with information retrieval research: Summarized Research in Information Retrieval (SuRe Info).
        J Eur Assoc Health Inform Libr. 2013; 9: 17-19
        • Higgins J.P.T.
        • Altman D.G.
        • Gøtzsche P.C.
        • et al.
        The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials.
        BMJ. 2011; 343: d5928
      5. RoB 2.0 tool—Risk of bias tools. n.d.
        • Sterne J.A.
        • Hernán M.A.
        • Reeves B.C.
        • et al.
        ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
        BMJ. 2016; 355: i4919
        • O’Neil M.
        • Berkman N.
        • Hartling L.
        • et al.
        Observational Evidence and Strength of Evidence Domains: Case Examples.
        Agency for Healthcare Research and Quality (US), Rockville, MD2014
        • NCBI Bookshelf
        Developing a Protocol for Observational Comparative Effectiveness Research: A User’s Guide. n.d.
        • NICE Decision Support Unit
        Observational data TSD. n.d.
        • Stratos Initiative
        STRengthening Analytical Thinking for Observational Studies. n.d.
        http://www.stratos-initiative.org/
        Date accessed: October 5, 2017
        • Berger M.L.
        • Martin B.C.
        • Husereau D.
        • et al.
        A questionnaire to assess the relevance and credibility of observational studies to inform health care decision-making: an ISPOR-AMCP-NPC Good Practice Task Force report.
        Value Health. 2014; 17: 143-156
        • Walker D.G.
        • Wilson R.F.
        • Sharma R.
        • et al.
        Best Practices for Conducting Economic Evaluations in Health Care: A Systematic Review of Quality Assessment Tools.
        Agency for Healthcare Research and Quality (US), Rockville, MD2012
        • Drummond M.F.
        • Sculpher M.J.
        • Claxton K.
        • Stoddart G.L.
        • Torrance G.W.
        Methods for the Economic Evaluation of Health Care Programmes.
        Oxford University Press, Oxford2015
        • Evers S.
        • Goossens M.
        • de Vet H.
        • van Tulder M.
        • Ament A.
        Criteria list for assessment of methodological quality of economic evaluations: consensus on health economic criteria.
        Int J Technol Assess Health Care. 2005; 21: 240-245
        • Ofman J.J.
        • Sullivan S.D.
        • Neumann P.J.
        • et al.
        Examining the value and quality of health economic analyses: implications of utilizing the QHES.
        J Manag Care Pharm. 2003; 9: 53-61
        • Heintz E.
        • Gerber-Grote A.
        • Ghabri S.
        • Hamers F.F.
        • Rupel V.P.
        • et al.
        Is there a European view on health economic evaluations? Results from a synopsis of methodological guidelines used in the EUnetHTA partner countries.
        Pharmacoeconomics. 2016; 34 (EUnetHTA Joint Action 2, Work Package 7, Subgroup 3): 59-76
        • Husereau D.
        • Drummond M.
        • Petrou S.
        • et al.
        Consolidated Health Economic Evaluation Reporting Standards (CHEERS)—Explanation and Elaboration: A Report of the ISPOR Health Economic Evaluations Publication Guidelines Task Force.
        Value Health. 2013; 16: 231-250
      6. iqwig.de. Methods Paper. n.d.
      7. Pharmacoeconomic Guidelines Around the World. n.d.
        https://tools.ispor.org/peguidelines/
        Date accessed: October 5, 2017
        • Mathes T.
        • Jacobs E.
        • Morfeld J.C.
        • Pieper D.
        Methods of international health technology assessment agencies for economic evaluations—a comparative analysis.
        BMC Health Serv Res. 2013; 13: 371
        • Tarn T.
        • Smith M.D.
        Pharmacoeconomic guidelines around the world.
        ISPOR Connect. 2004; 10: 5-15
        • Sullivan S.D.
        • Mauskopf J.A.
        • Augustovski F.
        • et al.
        Budget impact analysis-principles of good practice: report of the ISPOR 2012 Budget Impact Analysis Good Practice II Task Force.
        Value Health. 2014; 17: 5-14
        • Curry L.A.
        • Nembhard I.M.
        • Bradley E.H.
        Qualitative and mixed methods provide unique contributions to outcomes research.
        Circulation. 2009; 119: 1442-1452
        • Vandermause R.
        • Barg F.K.
        • Esmail L.
        • et al.
        Qualitative methods in patient-centered outcomes research.
        Qual Health Res. 2017; 27: 434-442
        • Santiago-Delefosse M.
        • Bruchez C.
        • Gavin A.
        • Stephen S.L.
        • Roux P.
        Complexity of the paradigms present in quality criteria of qualitative research grids.
        SAGE Open. 2015; 5: 1-13
        • Tong A.
        • Sainsbury P.
        • Craig J.
        Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups.
        Int J Qual Health Care. 2007; 19: 349-357
        • Tong A.
        • Flemming K.
        • McInnes E.
        • Oliver S.
        • Craig J.
        Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ.
        BMC Med Res Methodol. 2012; 12: 181
        • Moher D.
        • Schulz K.F.
        • Simera I.
        • Altman D.G.
        Guidance for developers of health research reporting guidelines.
        PLoS Med. 2010; 7: e1000217
        • Hunger T.
        • Schnell-Inderst P.
        • Sahakyan N.
        • Siebert U.
        Using expert opinion in health technology assessment: a guideline review.
        Int J Technol Assess Health Care. 2016; 32: 131-139
        • Lehoux P.
        • Williams-Jones B.
        Mapping the integration of social and ethical issues in health technology assessment.
        Int J Technol Assess Health Care. 2007; 23: 9-16
        • Potter B.K.
        • Avard D.
        • Graham I.D.
        • et al.
        Guidance for considering ethical, legal, and social issues in health technology assessment: application to genetic screening.
        Int J Technol Assess Health Care. 2008; 24: 412-422
      8. EUnetHTA HTA Adaptation Toolkit. n.d.
        • Hofmann B.
        • Oortwijn W.
        • Bakke Lysdahl K.
        • et al.
        Integrating ethics in health technology assessment: many ways to Rome.
        Int J Technol Assess Health Care. 2015; 31: 131-137
        • Hofmann B.
        • Cleemput I.
        • Bond K.
        • et al.
        Revealing and acknowledging value judgments in health technology assessment.
        Int J Technol Assess Health Care. 2014; 30: 579-586
        • van der Wilt G.J.
        Healthcare technology assessment.
        in: Have H. ten Encyclopedia of Global Bioethics. Springer, New York2015: 1-13
        • Lysdahl K.B.
        • Hofmann B.
        Complex health care interventions: characteristics relevant for ethical analysis in health technology assessment.
        GMS Health Technol Assess. 2016; 12: Doc01
        • Saarni S.I.
        • Hofmann B.
        • Lampe K.
        • et al.
        Ethical analysis to improve decision-making on health technologies.
        Bull World Health Organ. 2008; 86: 617-623
        • Assasi N.
        • Schwartz L.
        • Tarride J.-E.
        • Campbell K.
        • Goeree R.
        Methodological guidance documents for evaluation of ethical considerations in health technology assessment: a systematic review.
        Expert Rev Pharmacoecon Outcomes Res. 2014; 14: 203-220
        • Rehfuess E.A.
        • Gerhardus A.
        • INTEGRATE-HTA
        INTEGRATE-HTA: adopting and implementing an integrated perspective on complex interventions.
        J Public Health (Oxf). 2017; 39: 209-212
        • Wahlster P.
        • Brereton L.
        • Burns J.
        • et al.
        An integrated perspective on the assessment of technologies: INTEGRATE-HTA.
        Int J Technol Assess Health Care. 2017; 33: 1-8
        • Heintz E.
        • Lintamo L.
        • Hultcrantz M.
        • et al.
        Framework for systematic identification of ethical aspects of healthcare technologies: the SBU approach.
        Int J Technol Assess Health Care. 2015; 31: 124-130
        • Mertz M.
        • Kahrass H.
        • Strech D.
        Current state of ethics literature synthesis: a systematic review of reviews.
        BMC Med. 2016; 14: 152
        • Braunack-Mayer A.J.
        Ethics and health technology assessment: handmaiden and/or critic?.
        Int J Technol Assess Health Care. 2006; 22: 307-312
        • Saarni S.I.
        • Braunack-Mayer A.
        • Hofmann B.
        • van der Wilt G.J.
        Different methods for ethical analysis in health technology assessment: an empirical study.
        Int J Technol Assess Health Care. 2011; 27: 305-312
        • Scott A.M.
        • Hofmann B.
        • Gutiérrez-Ibarluzea I.
        • Bakke Lysdahl K.
        • Sandman L.
        • Bombard Y.
        Q-SEA—a tool for quality assessment of ethics analyses conducted as part of health technology assessments.
        GMS Health Technol Assess. 2017; 13: Doc02
        • Bond K.
        • Oremus M.
        • Duthie K.M.
        • Griener G.G.
        Ethics expertise for health technology assessment: a Canadian national survey.
        Int J Technol Assess Health Care. 2014; 30: 131-136
        • Tarricone R.
        • Torbica A.
        • Drummond M.
        • MedtecHTA Project Group
        Key recommendations from the MedtecHTA project.
        Health Econ. 2017; 26: 145-152
        • EUnetHTA
        EUnetHTA Guidelines. n.d.
        http://www.eunethta.eu/eunethta-guidelines
        Date accessed: January 19, 2018
        • Robinson K.A.
        • Whitlock E.P.
        • O’Neil M.E.
        • et al.
        Integration of Existing Systematic Reviews.
        Agency for Healthcare Research and Quality (US), Rockville, MD2014
        • Shea B.J.
        • Grimshaw J.M.
        • Wells G.A.
        • et al.
        Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews.
        BMC Med Res Methodol. 2007; 7: 10
        • Liberati A.
        • Altman D.G.
        • Tetzlaff J.
        • et al.
        The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration.
        BMJ. 2009; 339: b2700
        • Draborg E.
        • Gyrd-Hansen D.
        • Poulsen P.B.
        • Horder M.
        International comparison of the definition and the practical application of health technology assessment.
        Int J Technol Assess Health Care. 2005; 21: 89-95
        • Sutton A.J.
        • Abrams K.R.
        • Jones D.R.
        • et al.
        Methods for Meta-Analysis in Medical Research. n.d.
        • Jansen J.P.
        • Fleurence R.
        • Devine B.
        • et al.
        Interpreting indirect treatment comparisons and network meta-analysis for health-care decision-making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: Part 1.
        Value Health. 2011; 14: 417-428
        • Jansen J.P.
        • Trikalinos T.
        • Cappelleri J.C.
        • et al.
        Indirect treatment comparison/network meta-analysis study questionnaire to assess relevance and credibility to inform health care decision-making: an ISPOR-AMCP-NPC Good Practice Task Force report.
        Value Health. 2014; 17: 157-173
        • Brignardello-Petersen R.
        • Bonner A.
        • Alexander P.E.
        • et al.
        Advances in the GRADE approach to rate the certainty in estimates from a network meta-analysis.
        J Clin Epidemiol. 2018; 93: 36-44
        • Puhan M.A.
        • Schünemann H.J.
        • Murad M.H.
        • et al.
        A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis.
        BMJ. 2014; 349: g5630
        • Hutton B.
        • Salanti G.
        • Caldwell D.M.
        • et al.
        The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations.
        Ann Intern Med. 2015; 162: 777-784
        • Wong G.
        • Greenhalgh T.
        • Westhorp G.
        • Buckingham J.
        • Pawson R.
        RAMESES publication standards: meta-narrative reviews.
        BMC Med. 2013; 11: 20
        • Wong G.
        • Greenhalgh T.
        • Westhorp G.
        • Buckingham J.
        • Pawson R.
        RAMESES publication standards: realist syntheses.
        BMC Med. 2013; 11: 21
        • France E.F.
        • Ring N.
        • Noyes J.
        • et al.
        Protocol-developing meta-ethnography reporting guidelines (eMERGe).
        BMC Med Res Methodol. 2015; 15: 103
        • O’Cathain A.
        Assessing the Quality of Mixed Methods Research: Toward a Comprehensive Framework.
        Sage Inc., Thousand Oaks, CA2010: 531-556
        • Lewin S.
        • Glenton C.
        • Munthe-Kaas H.
        • et al.
        Using qualitative evidence in decision-making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual).
        PLoS Med. 2015; 12: e1001895
        • Fearon J.D.
        Deliberation as discussion.
        in: Elster J. Deliberative Democracy. Cambridge University Press, Cambridge1998: 44-68
        • Baltussen R.
        • Jansen M.P.M.
        • Bijlmakers L.
        • et al.
        Value assessment frameworks for HTA agencies: the organization of evidence-informed deliberative processes.
        Value Health. 2017; 20: 256-260
        • Culyer A.
        Deliberative Processes in Decisions About Health Care Technologies: Combining Different Types of Evidence, Values, Algorithms and People.
        Office of Health Economics, London2009
        • Lomas J.
        • Culyer T.
        • Mccutcheon C.
        • Law S.
        • Tetroe J.
        Final Report: Conceptualizing and Combining Evidence for Health System Guidance.
        Canadian Health Services Research Foundation, Ottawa, ON2005
        • Bridges J.F.P.
        • Jones C.
        Patient-based health technology assessment: a vision of the future.
        Int J Technol Assess Health Care. 2007; 23: 30-35
        • Abelson J.
        • Giacomini M.
        • Lehoux P.
        • Gauvin F.-P.
        Bringing “the public” into health technology assessment and coverage policy decisions: from principles to practice.
        Health Policy. 2007; 82: 37-50
        • NICE
        Patient and public involvement policy. n.d.
        • Perfetto E.M.
        • Oehrlein E.M.
        • Boutin M.
        • Reid S.
        • Gascho E.
        Value to whom? The patient voice in the value discussion.
        Value Health. 2017; 20: 286-291
        • Hansen H.P.
        • Draborg E.
        • Kristensen F.B.
        Exploring qualitative research synthesis: the role of patients’ perspectives in health policy design and decision-making.
        Patient. 2011; 4: 143-152
        • Mühlbacher A.C.
        Patient-centric HTA: different strokes for different folks.
        Expert Rev Pharmacoecon Outcomes Res. 2015; 15: 591-597
        • Frank L.
        • Forsythe L.
        • Ellis L.
        • et al.
        Conceptual and practical foundations of patient engagement in research at the patient-centered outcomes research institute.
        Qual Life Res. 2015; 24: 1033-1041
        • Bridges J.F.P.
        • Hauber A.B.
        • Marshall D.
        • et al.
        Conjoint analysis applications in health—a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force.
        Value Health. 2011; 14: 403-413
        • Johnson F.R.
        • Lancsar E.
        • Marshall D.
        • et al.
        Constructing experimental designs for discrete-choice experiments: report of the ISPOR Conjoint Analysis Experimental Design Good Research Practices Task Force.
        Value Health. 2013; 16: 3-13
        • Hauber A.B.
        • González J.M.
        • Groothuis-Oudshoorn C.G.M.
        • et al.
        Statistical methods for the analysis of discrete choice experiments: a report of the ISPOR Conjoint Analysis Good Research Practices Task Force.
        Value Health. 2016; 19: 300-315
        • Facey K.
        • Boivin A.
        • Gracia J.
        • et al.
        Patients’ perspectives in health technology assessment: a route to robust evidence and fair deliberation.
        Int J Technol Assess Health Care. 2010; 26: 334-340
        • EUPATI
        Guidance for patient involvement in HTA. 2016.
        • Thokala P.
        • Devlin N.
        • Marsh K.
        • et al.
        Multiple criteria decision analysis for health care decision-making—an introduction: report 1 of the ISPOR MCDA Emerging Good Practices Task Force.
        Value Health. 2016; 19: 1-13
        • Marsh K.
        • Goetghebeur M.
        • Thokala P.
        • Baltussen R.
        Multi-Criteria Decision Analysis to Support Healthcare Decisions.
        Springer, New York2017
        • Husereau D.
        • Boucher M.
        • Noorani H.
        Priority setting for health technology assessment at CADTH.
        Int J Technol Assess Health Care. 2010; 26: 341-347
        • Danner M.
        • Hummel J.M.
        • Volz F.
        • et al.
        Integrating patients’ views into health technology assessment: Analytic hierarchy process (AHP) as a method to elicit patient preferences.
        Int J Technol Assess Health Care. 2011; 27: 369-375
        • Castro H.
        • Tringali M.
        • Cleemput I.
        • Devriese S.
        • Leoni O.
        • Lettieri E.
        Advancing MCDA and HTA into Coverage Decision-Making. Multi-Criteria Decision Analysis to Support Healthcare Decisions.
        Springer, Cham, Switzerland2017: 119-146
        • Radaelli G.
        • Lettieri E.
        • Masella C.
        • Merlino L.
        • Strada A.
        • Tringali M.
        Implementation of EUnetHTA core Model® in Lombardia: the VTS framework.
        Int J Technol Assess Health Care. 2014; 30: 105-112
        • Golan O.
        • Hansen P.
        Which health technologies should be funded? A prioritization framework based explicitly on value for money.
        Isr J Health Policy Res. 2012; 1: 44
        • Tony M.
        • Wagner M.
        • Khoury H.
        • et al.
        Bridging health technology assessment (HTA) with multicriteria decision analyses (MCDA): field testing of the EVIDEM framework for coverage decisions by a public payer in Canada.
        BMC Health Serv Res. 2011; 11: 329
        • Youngkong S.
        • Baltussen R.
        • Tantivess S.
        • Mohara A.
        • Teerawattananon Y.
        Multicriteria decision analysis for including health interventions in the universal health coverage benefit package in Thailand.
        Value Health. 2012; 15: 961-970
        • Verguet S.
        • Kim J.J.
        • Jamison D.T.
        Extended cost-effectiveness analysis for health policy assessment: a tutorial.
        Pharmacoeconomics. 2016; 34: 913-923
        • McCabe C.
        • Claxton K.
        • Culyer A.J.
        The NICE cost-effectiveness threshold: what it is and what that means.
        Pharmacoeconomics. 2008; 26: 733-744
        • Claxton K.
        • Martin S.
        • Soares M.
        • et al.
        Methods for the estimation of the National Institute for Health and Care Excellence cost-effectiveness threshold.
        Health Technol Assess. 2015; 19: 1-503, v-vi
        • Schwarzer R.
        • Rochau U.
        • Saverno K.
        • et al.
        Systematic overview of cost-effectiveness thresholds in ten countries across four continents.
        J Comp Eff Res. 2015; 4: 485-504
        • Drummond M.
        • Barbieri M.
        • Cook J.
        • et al.
        Transferability of economic evaluations across jurisdictions: ISPOR Good Research Practices Task Force report.
        Value Health. 2009; 12: 409-418
        • Drummond M.
        • Augustovski F.
        • Kaló Z.
        • et al.
        Challenges faced in transferring economic evaluations to middle income countries.
        Int J Technol Assess Health Care. 2015; 31: 442-448
        • Barbieri M.
        • Drummond M.
        • Willke R.
        • Chancellor J.
        • Jolain B.
        • Towse A.
        Variability of cost-effectiveness estimates for pharmaceuticals in Western Europe: lessons for inferring generalizability.
        Value Health. 2005; 8: 10-23
        • Barbieri M.
        • Drummond M.
        • Rutten F.
        • et al.
        What do international pharmacoeconomic guidelines say about economic data transferability?.
        Value Health. 2010; 13: 1028-1037
        • Chalon P.X.
        • Kraemer P.
        EUnetHTA information management system: development and lessons learned.
        Int J Technol Assess Health Care. 2014; 30: 514-520
      9. Danske Regioner—Medicinrådet—In English. n.d.
        • National Institute for Health and Care Excellence
        Assessing Resource Impact Methods Guide 2011.
        NIHCE, London2011
        • ICER
        Final Value Assessment Framework for 2017-2019. n.d.
        https://icer-review.org/final-vaf-2017-2019/
        Date accessed: January 24, 2018
        • Lavis J.N.
        • Robertson D.
        • Woodside J.M.
        • McLeod C.B.
        • Abelson J.
        Knowledge Transfer Study Group. How can research organizations more effectively transfer research knowledge to decision makers?.
        Milbank Q. 2003; 81 (171-172): 221-248
        • Lavis J.N.
        • Oxman A.D.
        • Lewin S.
        • Fretheim A.
        SUPPORT Tools for evidence-informed health policymaking (STP).
        Health Res Policy Syst. 2009; 7: I1
      10. Health Systems Evidence. n.d.
        https://www.healthsystemsevidence.org/
        Date accessed: March 23, 2017
      11. The AMCP format for formulary submissions version 3.0.
        J Manag Care Pharm. 2010; 16: 1-30
        • EUnetHTA
        Submission template for pharmaceuticals and submission template for medical devices. n.d.
        • Kaló Z.
        • Gheorghe A.
        • Huic M.
        • Csanádi M.
        • Kristensen F.B.
        HTA implementation roadmap in Central and Eastern European countries.
        Health Econ. 2016; 25: 179-192
      12. The Pharmaceutical Price Regulation Scheme. 02/07. Office of Fair Trading, London2007
        • Hailey D.
        • Werkö S.
        • Rosén M.
        • et al.
        Influence of health technology assessment and its measurement.
        Int J Technol Assess Health Care. 2016; 32: 376-384
        • Gerhardus A.
        • Dorendorf E.
        • Røttingen J.-A.
        • Santamera A.S.
        What are the effects of HTA reports on the health system? Evidence from the research literature.
        in: Velasco Garrido M. Kristensen F.B. Palmhøj Nielsen C. Busse R. Health Technology Assessment and Health Policy-Making in Europe: Current status, Challenges and Potential. WHO, Geneva2008: 109-136
        • Gerhardus A.
        • Dintsios C.-M.
        The impact of HTA reports on health policy: a systematic review.
        GMS Health Technol Assess. 2005; 1: Doc02
        • Raftery J.
        • Hanney S.
        • Greenhalgh T.
        • Glover M.
        • Blatch-Jones A.
        Models and applications for measuring the impact of health research: update of a systematic review for the Health Technology Assessment programme.
        Health Technol Assess. 2016; 20: 1-254
        • Guthrie S.
        • Hafner M.
        • Bienkowska-Gibbs T.
        • Wooding S.
        Returns on research funded under the NIHR Health Technology Assessment (HTA) Programme: economic analysis and case studies.
        Rand Health Q. 2016; 5: 5
        • Jacob R.
        • McGregor M.
        Assessing the impact of health technology assessment.
        Int J Technol Assess Health Care. 1997; 13: 68-80
        • Schumacher I.
        • Zechmeister I.
        Assessing the impact of health technology assessment on the Austrian healthcare system.
        Int J Technol Assess Health Care. 2013; 29: 84-91
        • Zechmeister I.
        • Schumacher I.
        The impact of health technology assessment reports on decision-making in Austria.
        Int J Technol Assess Health Care. 2012; 28: 77-84
        • Callea G.
        • Armeni P.
        • Marsilio M.
        • Jommi C.
        • Tarricone R.
        The impact of HTA and procurement practices on the selection and prices of medical devices.
        Soc Sci Med. 2017; 174: 89-95
        • Banta D.
        What is technology assessment?.
        Int J Technol Assess Health Care. 2009; 25: 7-9
        • Teutsch S.M.
        • Berger M.L.
        Evidence synthesis and evidence-based decision-making: related but distinct processes.
        Med Decis Making. 2005; 25: 487-489
        • Akehurst R.L.
        • Abadie E.
        • Renaudin N.
        • Sarkozy F.
        Variation in health technology assessment and reimbursement processes in Europe.
        Value Health. 2017; 20: 67-76
        • Garrison L.P.
        • Neumann P.J.
        • Willke R.J.
        • et al.
        A health economics approach to US value assessment frameworks—summary and recommendations of the ISPOR Special Task Force Report [7].
        Value Health. 2018; 21: 161-165
        • Husereau D.
        • Henshall C.
        • Sampietro-Colom L.
        • Thomas S.
        Changing health technology assessment paradigms?.
        Int J Technol Assess Health Care. 2016; 32: 191-199
        • EUnetHTA
        Early dialogue consolidated procedure. n.d.
        • Henshall C.
        • Mardhani-Bayne L.
        • Frønsdal K.B.
        • Klemp M.
        Interactions between health technology assessment, coverage, and regulatory processes: emerging issues, goals, and opportunities.
        Int J Technol Assess Health Care. 2011; 27: 253-260
        • EUnetHTA
        EMA and EUnetHTA finalise joint work plan for 2017-2020. n.d.
        https://www.eunethta.eu/ema/
        Date accessed: January 19, 2018