Advertisement

What Guidance are Economists Given on How to Present Economic Evaluations for Policymakers? A Systematic Review

      Abstract

      Objectives

      To systematically review health economic guidelines for information on how to present health economic evaluations and consider implications for nontechnical audiences such as policymakers.

      Methods

      Electronic databases and supplementary sources were searched for economic evaluation guidelines. Guidelines were critically appraised. Descriptive characteristics, standard formats, supports for nontechnical audiences, presentation approaches, and common reporting recommendations were extracted. Frequencies were tabulated and trends identified.

      Results

      Thirty-one guidelines were included. Twenty-two guidelines include a standard reporting format with some sample tables and graphs. Common presentation approaches include well-cited tables of data sources, transparent model diagrams and descriptions, disaggregated results, and tabular and graphical displays of sensitivity analyses. Despite most guidelines being funded by policymakers, only five guidelines provided advice on presenting economic evaluations to noneconomists. However, 11 guidelines included a glossary of economic terminology for nontechnical readers. Common concepts that may require further explanation include differences in economic perspectives, appropriateness of time horizons, how economic outcomes such as quality-adjusted life-years relate to their component clinical outcomes, and choice of sensitivity analyses.

      Conclusions

      Health economists have consistent presentation formats and common reporting elements that should be considered when developing user-friendly explanations for general audiences. These overlap with policymakers’ informational needs but may not be sufficient for understanding by nontechnical audiences. Developing presentation formats and tools that incorporate viewpoints of both economists and noneconomists will allow for better application of the results of economic evaluations and enhance the transparency and legitimacy of decision-making processes that are informed by economic evaluations.

      Keywords

      Introduction

      Using cost-effectiveness analyses in health policy and reimbursement decisions has sometimes been questioned, and the policy impact of health economic evaluations is still uncertain after 20 years [
      • Schlander M.
      The use of cost-effectiveness by the National Institute for Health and Clinical Excellence (NICE): no(t yet an) exemplar of a deliberative process.
      ,
      • Drummond M.
      Twenty years of using economic evaluations for drug reimbursement decisions: what has been achieved?.
      ]. Enhancing the transparency and understanding of economic evaluations could increase confidence in and perceived legitimacy of health technology assessment decision-making processes [
      • Daniels N.
      • Sabin J.
      Limits to health care: fair procedures, democratic deliberations and the legitimacy problem for insurers.
      ,
      • Lehoux P.
      • Denis J.L.
      • Tailliez S.
      • et al.
      Dissemination of health technology assessments: identifying the visions guiding an evolving policy innovation in Canada.
      ]. Economic evaluations are not always provided in an accessible format, and more effective presentation of economic evaluations could increase understanding of how economic evaluations inform policy decisions. This would also enable policymakers to better defend decisions that are based, in part, on economic evaluations.
      Health policy decision makers, and those affected by policy decisions such as the tax-paying public, patients, and health care providers, generally do not have the same technical expertise as health economists [
      • Thurston S.J.
      • Craig D.
      • Wilson P.
      • Drummond M.F.
      Increasing decision-makers’ access to economic evaluations: alternative methods of communicating the information.
      ]. However, they are important end users of economic evaluations. Policymakers sometimes have limited training in health economics and may be unclear in how to practically apply the results of economic evaluations to actual decisions if results are not clearly presented [
      • Chaikledkaew U.
      • Lepitakpong C.
      • Teerawattananon Y.
      • et al.
      The current capacity and future development of economic evaluation for policy decision making: a survey among researchers and decision makers in Thailand.
      ,
      • Phillips C.J.
      • Fordham R.
      • March K.
      • et al.
      Exploring the role of economics in prioritization in public health: what do stakeholders think?.
      ].
      Providing simple but robust explanations of economic evaluations to a nontechnical audience is challenging [
      • Choi B.
      • Pang T.
      • Lin V.
      • et al.
      Can scientists and policy makers work together?.
      ]. An appropriate balance of making information understandable while maintaining the completeness and technical accuracy of the economic evaluation must be achieved. Oversimplifying technical information may lose important nuances that are relevant for practical and consistent decision making. Therefore, considering how economists present their work to policymakers and other lay audiences merits further exploration. A number of studies have reported on policymakers’ use of economic evaluations. In general, policymakers want to be able to deconstruct analyses, identify key principles, and understand how results practically apply to specific decisions [
      • Taylor-Robinson D.
      • Milton B.
      • Lloyd Williams F.
      • et al.
      Policy makers’ attitudes to decision support models for coronary heart disease: a qualitative study.
      ,
      • Davies L.
      • Coyle D.
      • Drummond M.
      Current status of economic appraisal of health technology in the European community: report of the network.
      ,
      • Hoffman C.
      • Graf von der Schulenburg J.M.
      The influence of economic evaluation studies on decision making: a European survey.
      ]. It is unclear how this aligns with what economists consider important to present in economic evaluations.
      Elements that economists report on are described in guidelines for conducting economic evaluations [
      • Husereau D.
      • Drummond M.
      • Petrou S.
      • et al.
      Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement.
      ,
      • Canadian Agency for Drugs and Technologies in Health
      Guidelines for the Economic Evaluation of Health Technologies: Canada.
      ]. The general purpose of these guidelines is to increase both the methodological quality of economic evaluations and the transparency of how they are conducted. Although good reporting practices outlined in health economic guidelines are important to follow and can contribute to clarity, they may be insufficient for good communication to nontechnical audiences [
      • Thurston S.J.
      • Craig D.
      • Wilson P.
      • Drummond M.F.
      Increasing decision-makers’ access to economic evaluations: alternative methods of communicating the information.
      ,
      • Galani C.
      • Rutten F.
      Self-reported healthcare decision-makers’ attitudes towards economic evaluations of medical technologies.
      ]. Although adequately reporting economic modeling details is a first essential step that provides a transparent description of the analysis and results to the reader, subsequently arranging this information in presentation formats that assist end users in their understanding and/or application of the information is also an important consideration. Developing tools and alternate presentation formats that enhance the accessibility of economic evaluations may be one approach to increasing their impact and value to clinicians and policymakers. Although standards for transparent health economic reporting may be consistent, regardless of the audience or topic, different presentation formats may be appropriate for different audiences or topics. It is unclear how much guidance economists are given with respect to presentation formats and tailoring presentation of their work to noneconomists such as policymakers.
      A systematic approach to knowledge translation [
      ] and the development of economic tools that would enhance accessibility of health economics and confidence in health policy decision making would be of value. Incorporating guidance from economic guidelines on reporting and presentation standards would lead to knowledge translation tools that can robustly represent complex economic information.
      The objective of this study was to systematically review guidelines to determine what guidance economists are provided on how to present economic evaluations and whether any of this guidance is targeted toward nontechnical end users such as policymakers. Our primary hypothesis was that guidance provided to health economists on presenting economic evaluations is not focused on the needs of nontechnical end users of economic evaluations. We also hypothesized that some elements commonly required for transparent reporting are technical concepts that may require further explanation for nontechnical audiences. Although economic guidelines are developed for multiple audiences including economists in academic settings, industry, or research organizations, our findings will be interpreted in the context of nontechnical audiences who apply the results of economic evaluations to policy decisions.

      Methods

       Systematic Review Protocol

      A systematic review was conducted following Cochrane methodology [

      Cochrane Collaboration. Cochrane Handbook for Systematic Reviews. Available from: www.cochrane.org. [Accessed May 19, 2014].

      ]. Guidelines for conducting and/or reporting on economic evaluations of pharmaceuticals were included that primarily targeted economists and researchers who were providing economic evaluations for policymakers and other end users. Guidelines for assessing the quality of economic evaluations were excluded because they primarily target end users of economic evaluations and not economists who produce economic evaluations. Guidelines were excluded if they addressed only specific components of economic evaluations (e.g., modeling and willingness to pay) and not economic evaluations as a whole; if they addressed affordability (i.e., budget impact) rather than cost effectiveness; or if they advocated a disease-specific or technology-specific (e.g., diagnostic tests and medical devices) approach to economic evaluation. Editorials or opinion articles providing recommendations with no supporting evidence base or methodological process for developing the recommendations were excluded. Guidelines were also excluded if they were not directly linked to a decision-making or reimbursement process; this criterion limited the review to those guidelines most likely to provide approaches to presenting economic evaluations to nontechnical audiences such as policymakers. Finally, if more recent versions of the guidelines were available or if the guideline was developed before 2000, guidelines were excluded. The latter criterion was necessary given the methodological advances in the discipline. No language restrictions were applied. Outcomes of interest were the type and frequency of information on presentation formats and the type and frequency of key reporting recommendations.

       Search Strategy

      Databases searched included Medline (1996 to November 2013), EMBASE (1980 to November 2013), and the Cochrane Database of Systematic Reviews, using a date limit of 2000 but no language restriction. The search terms were “guidelines,” “health economics,” and “economic evaluation.” The gray literature was searched for unpublished studies using the Canadian Agency for Drugs and Technologies in Health Gray Matters checklist as a guide, in addition to other relevant economic and guideline resources, including Web sites for the International Society for Pharmacoeconomics and Outcomes Research (ISPOR), the Enhancing the QUAlity and Transparency Of health Research network, and the Guidelines International Network [
      • Canadian Agency for Drugs and Technologies in Health
      Grey Matters: a Practical Search Tool for Evidence-Based Medicine.
      ,

      International Society for Pharmacoeconomics and Outcomes Research. Pharmacoeconomic (ISPOR) guidelines around the world. Available from: http://www.ispor.org/PEguidelines/index.asp. [Accessed May 19, 2014].

      ,

      Enhancing the QUAlity and Transparency Of health Research (EQUATOR). Available from: http://www.equator-network.org. [Accessed November 24, 2014].

      ,

      Guidelines International Network. Available from: http://www.g-i-n.net. [Accessed November 24, 2014].

      ]. Studies were also selected through hand searching of selected journals, reviewing reference lists of potentially relevant studies, and suggestions from economic experts.

       Study Selection

      Citations were screened for relevance by one review author on the basis of the title and abstract of identified articles. Two review authors independently reviewed the full text of potentially relevant guidelines to assess exclusion or inclusion.

       Critical Appraisal

      Guideline quality was assessed using items identified in the Appraisal of Guidelines for Research & Evaluation (AGREE) II instrument [
      • Brouwers M.C.
      • Kho M.E.
      • Browman G.P.
      • et al.
      AGREE II: advancing guideline development, reporting and evaluation in health care.
      ]. The AGREE II instrument was designed to assess clinical practice guidelines and has six domains: scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, and editorial independence. The overall purpose of the AGREE II instrument is “to provide a framework to: assess the quality of guidelines; provide a methodological strategy for the development of guidelines; and inform what information and how information ought to be reported in guidelines” [
      • Brouwers M.C.
      • Kho M.E.
      • Browman G.P.
      • et al.
      AGREE II: advancing guideline development, reporting and evaluation in health care.
      ]. No instruments, however, currently exist to specifically assess either the quality of methodological guidelines or guidelines for economic evaluations. Therefore, minor modifications to items in the AGREE II instrument were made to apply the instrument to economic guidelines. Principles from a framework for communicating confidence in methodological recommendations for systematic reviews and meta-analyses were also incorporated into the assessment (for more details, see Appendix Table 1 in Supplemental Materials found at doi:10.1016/j.jval.2015.06.007) [

      Trikalinos TA, Dahabreh IJ, Wallace BC, et al. Towards a Framework for Communicating Confidence in Methodological Recommendations for Systematic Reviews and Meta-Analyses. Methods Research Report (Prepared by Tufts Evidence-based Practice Center under contract no. 290-2007-10055-I). AHRQ Publication No. 13-EHC119-EF. Rockville, MD: Agency for Healthcare Research and Quality, 2013. Available from: www.effectivehealthcare.ahrq.gov/reports/final.cfm.

      ]. These principles included consideration of different sources of evidence and approaches and criteria for selecting evidence that may differ in methodological guidelines. Because of these modifications, we did not feel it was appropriate to apply the standard AGREE II scoring system but used items in the instrument as a guide to identify potential limitations.

       Data Extraction

      The following data were extracted from guidelines (for detailed definitions of variables, see Table 2 in Supplemental Materials found at doi:10.1016/j.jval.2015.06.007).
      • 1.
        Descriptive characteristics: geographic location, publication year, author affiliations, guideline purpose, type of guideline, target audience.
      • 2.
        Standard templates: whether a standard reporting template, sample tables, or sample figures were provided,
      • 3.
        Supports for nontechnical end users: if a glossary was provided, whether advice was provided on presenting to noneconomists.
      • 4.
        Presentation formats and approaches: details on how to present data sources, model, base-case results, and sensitivity analyses (deterministic and probabilistic). These four areas were selected as key components of an economic evaluation, and data extraction focused on the format for presenting the information, not just that the information be provided or reported (i.e., focusing on “how” to report not just “what” to report).
      • 5.
        Recommendations for the conduct and transparent reporting of economic evaluations: perspective, target population, subgroup analyses, time horizon, assumptions required, modeling, preferred outcome measures, sensitivity analysis parameters, sensitivity analysis ranges, and methods for conducting sensitivity analyses. These elements were selected from variables in the ISPOR database [

        International Society for Pharmacoeconomics and Outcomes Research. Pharmacoeconomic (ISPOR) guidelines around the world. Available from: http://www.ispor.org/PEguidelines/index.asp. [Accessed May 19, 2014].

        ,
        • Eldessouki R.
        • Smith M.D.
        Health care system information sharing: a step toward better health globally.
        ]. Although some of these elements relate to the methodology and conduct of economic evaluations, they were included because they may influence the overall approach to presenting economic evaluations to nontechnical audiences.
      Detailed guidance on how to conduct an economic evaluation was not extracted (e.g., analytic approaches and distributions to apply) and is not the focus of this systematic review. When available, guideline characteristics from the ISPOR database [

      International Society for Pharmacoeconomics and Outcomes Research. Pharmacoeconomic (ISPOR) guidelines around the world. Available from: http://www.ispor.org/PEguidelines/index.asp. [Accessed May 19, 2014].

      ,
      • Eldessouki R.
      • Smith M.D.
      Health care system information sharing: a step toward better health globally.
      ] were used, and were verified with source documents by one author. A second author verified disagreements. For all remaining data, including the guideline critical appraisal, one author extracted data, which were verified by a second author. A third reviewer resolved disagreements.

       Data Analysis

      The frequency of recommendations on reporting and presenting economic evaluations was tabulated and common trends were noted. More specifically, trends were considered on the basis of year of publication, geographic location, and critical appraisal domains (e.g., stakeholder involvement in guideline development).

      Results

      Of the 6099 citations identified, 32 reports, representing 31 guidelines, were included (see Appendix Figure in Supplemental Materials found at doi:10.1016/j.jval.2015.06.007) [
      • Canadian Agency for Drugs and Technologies in Health
      Guidelines for the Economic Evaluation of Health Technologies: Canada.
      ,
      Institute of Technology Assessment in Health
      Manual for the Development of Economic Evaluations in Health. Bogotá.
      ,
      • Bae S.
      • Lee S.
      • Bae E.
      • Jang S.
      Korean guidelines for pharmacoeconomic evaluation (second and updated version): consensus and compromise.
      ,
      Health Insurance Review and Assessment Service
      Pharmaceutical Economic Evaluation Guidelines.
      ,
      Norwegian Medicines Agency (NOMA)
      Guidelines on How to Conduct Pharmacoeconomic Analyses.
      ,
      Chinese Pharmacoeconomic Evaluation Guide Task Force
      China Guidelines for Pharmacoeconomic Evaluations.
      ,
      Health Information and Quality Authority
      Guidelines for the Economic Evaluation of Health Technologies in Ireland.
      ,
      • Lopez-Bastida J.
      • Oliva J.
      • Antonanzas F.
      • et al.
      Spanish recommendations on economic evaluation of health technologies.
      ,
      • Avksentieva M.V.
      • Antonova N.V.
      • Arutyunov G.P.
      • et al.
      Procedure for Clinical and Economic Evaluation of Drug Lists That are Submitted for Reimbursement Coverage from Public Health Care Budget.
      ,
      Ministry of Health, Pharmaceutical Administration
      Guidelines for the Submission of a Request to Include a Pharmaceutical Product in the National List of Health Services, Version 8.
      ,
      Department of Health
      Regulations Relating to a Transparent Pricing System for Medicines and Scheduled Substances: (Draft Guidelines for Pharmacoeconomic Submissions).
      ,
      Academy of Managed Care Pharmacy (AMCP)
      The AMCP Format for Formulary Submissions Version 3.0.
      ,
      Institute for Quality and Efficiency in Health Care (IQWiG)
      General Methods for the Assessment of the Relation of Benefits to Costs, Version 1.0.
      ,
      Ministry of Social Affairs and Health
      Guidelines for Preparing a Health Economic Evaluation [unofficial translation].
      ,
      Agency for Health Technology Assessment
      Guidelines for Conducting Health Technology Assessment, Version 2.1.
      ,
      • De Mello Vianna C.M.
      • Caetano R.
      Methodological Guidelines: Economic Evaluation of Health Technologies.
      ,
      National Institute for Health and Clinical Excellence (NICE)
      Guide to the Methods of Technology Appraisal.
      ,
      • Cleemput I.
      • Van Wilder P.
      • Vrijens F.
      • et al.
      Guidelines for Pharmacoeconomic Evaluations in Belgium.
      ,
      Ministry of Health
      Guidelines for Economic Evaluation of Health Care Interventions.
      ,
      Pharmaceutical Benefits Advisory Committee (PBAC)
      Guidelines for Preparing Submissions to the Pharmaceutical Benefits Advisory Committee, Version 4.3.
      ,
      • Bertozzi S.M.
      • Mendoza A.V.
      • Garduno A.H.
      • Barajas E.R.
      Guidelines for Economic Appraisal Studies for Updating the Health Sector’s National Formulary in Mexico.
      ,
      • Teerawattananon Y.
      • Chaikledkaew U.
      Thai health technology assessment guideline development.
      ,
      Scottish Medicines Consortium (SMC)
      Guidance to Manufacturers for Completion of New Product Assessment Form, Revised.
      ,
      • Grocott R.
      • Schoeler R.
      • Priest V.
      • et al.
      Prescription for Pharmacoeconomic Analysis, Version 2.
      ,

      Center for Drug Evaluation. Guidelines of Methodological Standards for Pharmacoeconomic Evaluations. Taiwan: Center for Drug Evaluation, 2006.

      ,
      Health Care Insurance Board
      Guidelines for Pharmacoeconomic Research.
      ,
      • Boulenger S.
      • Ulmann P.
      French Guidelines for the Economic Evaluation of Health Care Technologies.
      ,
      • González A.M.G.
      Methodological Guidelines for Health Economic Evaluation.
      ,
      • Edling A.
      • Stenberg A.M.
      General Guidelines for Economic Evaluations from the Pharmaceutical Benefits Board.
      ,
      • Behmane D.
      • Lambot K.
      • Irs A.
      • et al.
      Baltic Guideline for Economic Evaluation of Pharmaceuticals (Pharmacoeconomic Analysis).
      ,
      • Szende A.
      • Mogyorosy Z.
      • Muszbek N.
      • et al.
      Methodological guidelines for conducting economic evaluation of healthcare interventions in Hungary: a Hungarian proposal for methodology standards.
      ,
      • Capri S.
      • Ceci A.
      • Terranova L.
      • et al.
      Guidelines for economic evaluations in Italy: recommendations from the Italian group of pharmacoeconomic studies.
      ]. Approximately half were country-specific pharmacoeconomic evaluation guidelines recognized by decision-making bodies (54%, n = 17 of 31), 22% (n = 7 of 31) were submission guidelines identifying requirements for economic evaluations submitted to decision-making bodies, and 22% (n = 7 of 31) were published recommendations on economic evaluations. Descriptive characteristics of the guidelines are described in Table 1, with additional details provided in Appendix Table 3 in Supplemental Materials found at doi:10.1016/j.jval.2015.06.007.
      Table 1Characteristics of economic evaluation guidelines.
      CountryAuthor affiliationTarget audienceStandard reporting formatSample tables includedSample figures includedGlossary includedPresenting to noneconomists
      Pharmacoeconomic guidelines
      As defined in the ISPOR guideline database [18,20], guidelines were categorized as Pharmacoeconomic Guidelines (country-specific “official” guidelines or policies concerning economic evaluation that are recognized or required by the health care decision-making bodies/entities in this country/region for reimbursement), Submission Guidelines (country-specific “official” guidelines or policies concerning drug submission requirements with an economic evaluation part/section and are required by the health care decision-making bodies/entities in this country/region for reimbursement), or Published Pharmacoeconomic Recommendations (country-specific economic evaluation guidelines or recommendations published by experts in the field but are not “officially” recognized or required by the health care decision-making bodies/entities in this country/region for reimbursement).
       Columbia 2014Decision makers, researchersIndustry, decision makers, researchersYesYes
       South Korea 2013Decision makersIndustry, decision makers, researchersYesYesYes
       Norway 2012Decision makersIndustry, decision makers, researchersNo
       Ireland 2010Decision makersIndustry, decision makers, researchersYesYesYes
       Brazil 2009Decision makers, researchersIndustry, decision makers, researchersYesYesYes
       Germany 2009Decision makersDecision makersYesYes
       Belgium 2008Decision makersIndustry, decision makers, researchersYesYes
       Mexico 2008Researchers, decision makersIndustry, decision makers, researchersYesYesYes
       Slovakia 2008Decision makers, researchersIndustry, decision makers, researchersYes
       New Zealand 2007Decision makersIndustry, decision makers, researchersYesYesYes
       Canada 2006Decision makersDecision makersYesYesYesYes
       Taiwan 2006Researchers, industryIndustry, decision makers, researchersYes
       The Netherlands 2006Decision makers, researchersIndustry, decision makers, researchersYes
       Cuba 2003Decision makers, researchersIndustry, decision makers, researchersYesYes
       France 2003ResearchersIndustry, decision makers, researchersYes
       Sweden 2003Decision makersIndustry, decision makers
       Baltic States 2002Decision makersIndustry, decision makers
      Submission guidelines
      As defined in the ISPOR guideline database [18,20], guidelines were categorized as Pharmacoeconomic Guidelines (country-specific “official” guidelines or policies concerning economic evaluation that are recognized or required by the health care decision-making bodies/entities in this country/region for reimbursement), Submission Guidelines (country-specific “official” guidelines or policies concerning drug submission requirements with an economic evaluation part/section and are required by the health care decision-making bodies/entities in this country/region for reimbursement), or Published Pharmacoeconomic Recommendations (country-specific economic evaluation guidelines or recommendations published by experts in the field but are not “officially” recognized or required by the health care decision-making bodies/entities in this country/region for reimbursement).
       England & Wales 2013Decision makersIndustry, researchersYes
       Israel 2010Decision makersIndustry, decision makers, researchersYesYes
       Finland 2009Decision makersIndustry, researchers
       Australia 2008Decision makersIndustryYesYes
       Thailand 2008Researchers, decision makersIndustry, decision makers, researchersYesYesYes
       Scotland 2007Decision makersIndustry, decision makers
       Poland 2007Decision maker, researchersIndustry, decision makers, researchersYes
      Published pharmacoeconomic recommendations
      As defined in the ISPOR guideline database [18,20], guidelines were categorized as Pharmacoeconomic Guidelines (country-specific “official” guidelines or policies concerning economic evaluation that are recognized or required by the health care decision-making bodies/entities in this country/region for reimbursement), Submission Guidelines (country-specific “official” guidelines or policies concerning drug submission requirements with an economic evaluation part/section and are required by the health care decision-making bodies/entities in this country/region for reimbursement), or Published Pharmacoeconomic Recommendations (country-specific economic evaluation guidelines or recommendations published by experts in the field but are not “officially” recognized or required by the health care decision-making bodies/entities in this country/region for reimbursement).
       South Africa 2013Decision makersIndustry, decision makers, researchersYes
       China 2011Decision maker, researchersIndustry, decision makers, researchers, clinical providersYesYes
       Russia 2010ResearchersIndustry, decision makers, researchersYesYes
       Spain 2009Decision makers, researchersAll: Industry, decision makers, researchers, health providers, patients, citizens
       USA 2009Decision makerDecision makers, researchersYesYesYesYes
       Hungary 2002Decision maker, researchersIndustry, decision makers, health providersYes
       Italy 2001Decision maker, researchers, industryIndustry, decision makers, researchersYes
      ISPOR, International Society for Pharmacoeconomics and Outcomes Research.
      low asterisk As defined in the ISPOR guideline database

      International Society for Pharmacoeconomics and Outcomes Research. Pharmacoeconomic (ISPOR) guidelines around the world. Available from: http://www.ispor.org/PEguidelines/index.asp. [Accessed May 19, 2014].

      ,

      Guidelines International Network. Available from: http://www.g-i-n.net. [Accessed November 24, 2014].

      , guidelines were categorized as Pharmacoeconomic Guidelines (country-specific “official” guidelines or policies concerning economic evaluation that are recognized or required by the health care decision-making bodies/entities in this country/region for reimbursement), Submission Guidelines (country-specific “official” guidelines or policies concerning drug submission requirements with an economic evaluation part/section and are required by the health care decision-making bodies/entities in this country/region for reimbursement), or Published Pharmacoeconomic Recommendations (country-specific economic evaluation guidelines or recommendations published by experts in the field but are not “officially” recognized or required by the health care decision-making bodies/entities in this country/region for reimbursement).

       Critical Appraisal of Guidelines

      Guideline quality was generally low when critically appraised. All guidelines had some limitations related to the rigor of their development and editorial independence. Guidelines varied with respect to the degree of stakeholder involvement and their focus on applicability and implementation. Guidelines from Brazil and Cuba, however, were well developed in both these domains. Common limitations were not conducting a systematic search for evidence and an unclear process for formulating recommendations. Although recommendations were clearly identified in most guidelines, differences in terminology when comparing across guidelines could create confusion and lead to lack of clarity. Decision makers and related organizations frequently provided funding for the guidelines; however, authors’ potential conflicts of interest were not reported in most of the guidelines. More details on the critical appraisal are provided in Appendix Table 4 in Supplemental Materials found at doi:10.1016/j.jval.2015.06.007.

       Standard Templates: Reporting Templates, Sample Figures, and Sample Tables

      Most of the guidelines (71%, n = 22 of 31) provided a standard reporting format for an economic evaluation. The specific content of the report format varied across guidelines but generally addressed major components of the analysis including the design, methods, evidence summary, results, and interpretation. Some guidelines provided sample tables (32%, n = 10 of 31) and figures (16%, n = 5 of 31) for presenting economic evaluations. When all the guidelines were reviewed in detail, however, each provided some guidance on presenting various elements of economic evaluations and is included in the subsequent synthesis on how to present economic evaluations. Formal methods for assessing and designing the reporting structure, format, and content of economic evaluations were not identified in any of the guidelines, and none of the reporting templates specifically targeted policymakers or other nontechnical audiences.

       Supports for Nontechnical End Users: Glossaries and Advice on Presenting to Noneconomists

      Guidelines were most frequently developed by or in collaboration with policymakers (87%, n = 27 of 31) and targeted toward industry (90%, n = 28 of 31), policymakers (90%, n = 28 of 31), or researchers (77%, n = 24 of 31). Advice on presenting economic evaluations to nontechnical audiences was noted in 16% of the guidelines (n = 5 of 31). The American guideline suggested that model calculations should be clearly explained for noneconomists and results disaggregated for formulary committees [
      Academy of Managed Care Pharmacy (AMCP)
      The AMCP Format for Formulary Submissions Version 3.0.
      ]. The Thai guideline suggested that graphical presentations of results would assist general audiences [
      • Teerawattananon Y.
      • Chaikledkaew U.
      Thai health technology assessment guideline development.
      ]. Three guidelines (Brazil, Mexico, and Canada) recommended including an executive summary accessible to noneconomists [
      • Canadian Agency for Drugs and Technologies in Health
      Guidelines for the Economic Evaluation of Health Technologies: Canada.
      ,
      • De Mello Vianna C.M.
      • Caetano R.
      Methodological Guidelines: Economic Evaluation of Health Technologies.
      ,
      • Bertozzi S.M.
      • Mendoza A.V.
      • Garduno A.H.
      • Barajas E.R.
      Guidelines for Economic Appraisal Studies for Updating the Health Sector’s National Formulary in Mexico.
      ] (for more details, see Appendix Table 5 in Supplemental Materials found at doi:10.1016/j.jval.2015.06.007). In addition, 35% of the guidelines (n = 11 of 31) included a glossary of technical terms for those less familiar with health economics. No trends across geographic region or year of publication were identified for guidelines that included advice on presenting to noneconomists or that included a glossary. Four of the 5 guidelines that provided advice on presenting to noneconomists and 10 of the 11 guidelines with glossaries, however, were developed with good stakeholder involvement.

       Recommendations on Presentation Approaches and Formats

      Recommendations on how to present economic evaluations that appeared in at least 20% of the guidelines are provided in Table 2.
      Table 2Presentation approaches recommended in at least 20% of the economic guidelines.
      Recommendations on how to present informationGuidelines (N = 31), n (%)
      Data sources
       Cross-referencing and citations of sources15 (48)
       Detailed table of data and variables12 (39)
       Table of resource use and costs8 (26)
      Model
       Model description transparent enough to justify and/or replicate12 (39)
       Assumptions and/or methods clearly presented10 (32)
       Decision tree diagram9 (29)
       Relevant model diagram and structure presented9 (29)
      Base-case results
       Disaggregated results to increasingly aggregated19 (61)
       Incremental costs and benefits presented19 (61)
       Total costs and benefits presented15 (48)
       Tables of results12 (39)
       Graphical displays requested or encouraged7 (23)
      Sensitivity analyses
       Deterministic
        Tabular format9 (29)
        Tornado diagrams to highlight drivers8 (26)
        Identify influential parameters affecting results8 (26)
       Probabilistic
        Cost-effectiveness acceptability curve11 (35)
        Scatter plot—cost-effectiveness plane10 (32)
      Note. Multiple recommendations could be made in each guideline.

       Presenting data sources

      Sixty-eight percent (n = 21 of 31) of the guidelines provided direction on how to present data sources.
      The most common suggestions for presentation approaches were providing appropriate cross-referencing and citations of sources (48%, n = 15 of 31) and providing detailed tables or a list of included variables and data (39%, n = 12 of 31). Guidance on which variables to include in the tables varied, with some requesting all variables, others requesting only the most important variables, and some indicating only that there should be sufficient transparency to reproduce the analysis. Presenting a list or table including resource use and costs was also sometimes recommended (26%, n = 8 of 31).
      Less frequently recommended approaches for presenting data sources included having copies of original data sources or ensuring their availability (13%, n = 4 of 31), summarizing clinical trial data in tables (10%, n = 3 of 31), and including a quality assessment of the data sources (6%, n = 2 of 31). Three guidelines (10%, n = 3 of 31) also suggested providing more detailed information such as data collection forms or detailed data tables in appendices.

       Presenting economic models

      Sixty-eight percent (n = 21 of 31) of the guidelines provided suggestions on presenting the economic model. Many guidelines (39%, n = 12 of 31) emphasized the need for transparency and clarity in presenting the model description so that the analysis could be replicated and individuals can understand the justifications or why it is different from other existing models. It was noted that the presentation should allow individuals to follow the steps in the modeling and analysis and understand how the data move through the model. Similarly, a number of guidelines emphasized the need for clarity in presenting model assumptions (32%, n = 10 of 31). Figures commonly suggested included decision tree diagrams (29%, n = 9 of 31) or model structure diagrams (29%, n = 9 of 31). For the model structure diagrams, guidelines either requested traditional transition-state Markov model diagrams or did not specify details of the model diagram. Only one guideline specified that a less-detailed schematic diagram of the model be provided for noneconomists [
      Academy of Managed Care Pharmacy (AMCP)
      The AMCP Format for Formulary Submissions Version 3.0.
      ]. Presenting transition probabilities in a matrix was less frequently recommended (10%, n = 3 of 31).

       Presenting base-case results

      Ninety-four percent (n = 29 of 31) of the guidelines provided suggestions on how to present the base-case results. Although presenting incremental changes in results was predominantly recommended (61%, n = 19 of 31), many guidelines recommended that total costs and benefits also be presented (48%, n = 15 of 31) to aid in the interpretation of results. Some guidelines also specifically requested that uncertainty around the incremental cost-effectiveness ratio be presented using either confidence intervals or a range for the incremental cost-effectiveness ratio (13%, n = 4 of 31). Some guidelines (10%, n = 3 of 31) recommend presenting results using natural clinical units before converting to quality-adjusted life-years (QALYs) (e.g., heart attacks, cirrhosis, and adverse events avoided). Some guidelines (10%, n = 3 of 31) also suggested presenting costs and benefits at a per-patient level, not just a population level.
      Many guidelines recommended reporting results in disaggregate as much as possible, leading up to aggregate responses (61%, n = 19 of 31). Examples of disaggregating results included presenting costs and effects separately for each intervention and presenting results for different subpopulations, disease states, model cycles, or health care settings. In addition, some guidelines recommended separately reporting QALYs and life-years when there was a survival benefit (13%, n = 4 of 31). Some guidelines (13%, n = 4 of 31) indicated that results should be presented in a way to allow verification or reproducibility. Two guidelines (6%) also indicated that the limitations and/or quality of the results should be described for transparency purposes.
      Presenting results in tables was frequently recommended (39%, n = 12 of 31). Specific suggestions for tables included summarizing costs, effects, and cost- effectiveness for each therapy or comparing short-term trial-based results with long-term extrapolations. Presenting results using general graphical or visual displays was encouraged (23%, n = 7 of 31). Specific types of graphics included the cost-effectiveness plane (13%, n = 4 of 31), net benefit (6%, n = 2 of 31), changes in outcomes over time when permitted by the model (3%, n = 1 of 31), and willingness to pay (3%, n = 1 of 31).
      Some guidelines also recommended discussing practical issues when presenting results. For example, presenting results that allow one to determine relevance or generalizability to the population was recommended in two guidelines (6%). This could include consideration of disease epidemiology, population characteristics, practice patterns, or patterns of resource use. One guideline (3%) also requested that the magnitude and direction of this impact also be clearly identified. Some guidelines (13%, n = 3 of 31) requested presentation of cost saving and/or opportunity costs when relevant. Additional suggestions related to the relevance and applicability of results included noting implementation factors and the feasibility of adopting on the basis of financial constraints (3%, n = 1 of 31) or the ability to reallocate freed resources for other purposes (3%, n = 1 of 31). Two guidelines (6%) provided suggestions on how to present conclusions, including suggestions for standard wording when presenting cost-effectiveness results [
      Pharmaceutical Benefits Advisory Committee (PBAC)
      Guidelines for Preparing Submissions to the Pharmaceutical Benefits Advisory Committee, Version 4.3.
      ,

      Center for Drug Evaluation. Guidelines of Methodological Standards for Pharmacoeconomic Evaluations. Taiwan: Center for Drug Evaluation, 2006.

      ].

       Presenting deterministic sensitivity analyses

      Sixty-one percent (n = 19 of 31) of the guidelines provided suggestions on how to present the deterministic sensitivity analyses results. A summary of the parameters that should be subjected to sensitivity analyses is described later, in Table 3, as part of guideline reporting requirements.
      Table 3Approaches for conducting and transparently reporting economic evaluations recommended in at least 20% of the guidelines.
      Guideline elementRecommendations on conduct and reportingGuidelines (N = 31), n (%)
      Primary perspectiveHealth care payer19 (61)
      Societal14 (45)
      Target populationApproved indication19 (61)
      Clearly specified target population13 (42)
      Subgroup analysesRequested/permitted but no specific guidance20 (65)
      Time horizonLong enough to cover consequences and benefits19 (61)
      Disease characteristics considered8 (26)
      Assumptions requiredYes21 (68)
      Modeling permittedYes, requires details12 (39)
      Yes, details not specified7 (23)
      Preferred outcome measuresQuality-adjusted life-years26 (84)
      Natural, clinically relevant or patient-relevant units22 (71)
      Life-year gained or mortality20 (65)
      Quality of life11 (35)
      Sensitivity analyses—ParametersParameters with greatest uncertainty18 (58)
      Sensitivity analyses—RangeConfidence intervals8 (26)
      Representing full or credible range of variability8 (26)
      Extreme or best/worst-case values7 (23)
      Sensitivity analyses— MethodsOne-way or univariate21 (68)
      Multiway or multivariate scenario analyses17 (55)
      Probabilistic16 (52)
      Note. Multiple recommendations could be made in each guideline.
      For more details, see ISPOR Guideline Tables

      International Society for Pharmacoeconomics and Outcomes Research. Pharmacoeconomic (ISPOR) guidelines around the world. Available from: http://www.ispor.org/PEguidelines/index.asp. [Accessed May 19, 2014].

      and Appendix Table 6 in Supplemental Materials.
      ISPOR, International Society for Pharmacoeconomics and Outcomes Research.
      The most common recommendation was presenting sensitivity analyses in tables (29%, n = 9 of 31). Specific types of tables included a table comparing sensitivity analyses of broader drug use with the intended drug use, a matrix of variable effects and outcomes that differ between two comparators, and tabulating the sensitivity analyses beside the base-case results. Presenting sensitivity analyses using tornado diagrams to highlight key drivers (26%, n = 8 of 31) and identifying influential parameters that have an impact on results (26%, n = 8 of 31) were also frequently recommended.

       Presenting probabilistic sensitivity analyses

      Forty-two percent (n = 13 of 31) of the guidelines provided suggestions on how to present probabilistic sensitivity analyses. The most frequently recommended formats were the cost-effectiveness acceptability curve (35%, n = 11 of 31) and the scatter plot for the cost-effectiveness plane (32%, n = 10 of 31). Less frequent recommendations included presenting in a tabular format (10%, n = 3 of 31), presenting expected value of information analyses (3%, n = 1 of 31), and identifying the percentage of simulations that are cost saving (3%, n = 1 of 31).

       Recommendations for Conducting and Transparently Reporting Economic Evaluations

      Recommendations for conducting and transparently reporting economic evaluations that appeared in at least 20% of the guidelines are provided in Table 3. Less frequent recommendations are outlined in Appendix Table 6 in Supplemental Materials found at doi:10.1016/j.jval.2015.06.007.
      The perspectives most frequently recommended were that of the health care payer and the societal. Some guidelines suggested presenting multiple perspectives, and in some guidelines, even though one perspective was recommended as the base case, other perspectives were identified as optional.
      The approved indication (61%, n = 19 of 31) or a clearly specified target population (42%, n = 13 of 31) was the most frequently recommended population. Subgroup analyses were requested or permitted (65%, n = 20 of 31), but common guidance across guidelines on which subgroups should be analyzed was not provided.
      Use of a time horizon that could account for all the outcomes and consequences was most frequently recommended (61%, n = 19 of 31). Considering the disease characteristics when selecting the time horizon was also frequently recommended; for example, natural history, chronic versus acute, and long-term effects (26%, n = 8 of 31).
      Assumptions were permitted in most guidelines (68%, n = 21 of 31), and modeling details were also required (61%, n = 19 of 31). Common guidance was not provided for either aspect except to indicate that choices should be justified and transparently described.
      Outcome measures most frequently preferred were QALYs (84%, n = 26 of 31); natural clinical measures (71%, n = 22 of 31; e.g., adverse events avoided and heart attacks) that could be converted to QALYs, life-years, or mortality (65%, n = 20 of 31); and quality of life (35%, n = 11 of 31).
      The most frequent methods of conducting sensitivity analyses were one-way (68%, n = 21 of 31), multiway (55%, n = 17 of 31), and probabilistic (52%, n = 16 of 31). Uncertain parameters were most frequently explored in sensitivity analyses (58%, n = 18 of 31). The parameter values used in sensitivity analyses were most frequently based on confidence intervals (26%, n = 8 of 31), extreme best-case and worst-case scenarios (23%, n = 7 of 31), and those representing a full or credible range of variability (26%, n = 8 of 31).

      Discussion

       Key Findings

      Most guidelines offer some advice on how to present economic evaluations, and many include a standard reporting format with examples of tables and graphs to be included. None of the reporting templates used formal methods for their design or assessment, and none of the reporting templates specifically targeted policymakers or nontechnical audiences. Common ideas across guidelines include presenting well-cited tables of data sources, transparent model descriptions and diagrams, disaggregated results, and tabular and graphical displays of sensitivity analyses. However, despite most guidelines being funded by policymakers for the purpose of developing economic evaluations for policymakers, only five guidelines provided specifics on how to present economic evaluations to noneconomists. Some guidelines included a glossary of economic terminology for nontechnical readers. Those guidelines that provided advice for presenting to noneconomists or that included glossaries were more likely to have good stakeholder involvement in their development. Common elements that are required for transparent reporting of economic evaluations and that may require further explanation for noneconomists include differences between the societal and health care payer perspectives, adequacy of time horizons that cover the period of costs and effects while appropriately considering aspects of the disease, how economic outcomes such as QALYs and life-years relate to their component clinical outcomes such as heart attacks or adverse events, and how sensitivity analyses (methods, parameters, ranges) are chosen to be conducted. Best practices and educational resources for communicating these concepts to nontechnical audiences could be further explored.

       Comparison with Other Literature

      When considering whether this guidance is sufficient for presenting economic evaluations to nontechnical audiences, these findings should be considered in light of 1) what policymakers want from economic evaluations, and 2) frequent limitations in economic models. Finding common ground between how economists present economic evaluations and what policymakers need to understand from an economic evaluation is essential for creating economic evaluations that adequately balance both rigor and accessibility for those who must understand their implications for real-world decisions. A recent ISPOR task force partially attempted to address this gap by developing a questionnaire for decision makers to assess the relevance and credibility of economic modeling studies [
      • Caro J.
      • Eddy D.M.
      • Kan H.
      • et al.
      Questionnaire to assess relevance and credibility of modeling studies for informing health care decision making: an ISPOR-AMCP-NCP Good Practice Task Force report.
      ].
      The recommendations for presentation formats and reporting elements identified in these 31 guidelines sometimes overlap with issues raised by policymakers when applying the results of economic evaluations to policy decisions. Some similarities include ensuring transparency of methods and disaggregation of results. For example, policymakers have indicated that being able to deconstruct the economic analysis [
      • Taylor-Robinson D.
      • Milton B.
      • Lloyd Williams F.
      • et al.
      Policy makers’ attitudes to decision support models for coronary heart disease: a qualitative study.
      ] and clearly outlining model assumptions that contribute to an incremental cost-effectiveness ratio [
      • Davies L.
      • Coyle D.
      • Drummond M.
      Current status of economic appraisal of health technology in the European community: report of the network.
      ] can facilitate their understanding. However, policymakers have also focused on understanding the relevance of economic evaluation results to practical decisions and implementation issues. For example, being aware of opportunity costs and appreciating the difference between theoretical versus real-world cost saving are important considerations for policymakers. However, only a few guidelines emphasized the importance of presenting the cost saving or other factors related to implementation and adoption feasibility. Even when specific suggestions were provided on how to present information to target audiences, these centered on enhancing clarity and transparency, not on identifying relationships to real-world decisions.
      Across guidelines, there were no common recommendations on details to include when reporting on modeling and assumptions. However, these are important concepts when understanding the limitations of economic evaluations. Common limitations identified in submissions of economic evaluations for reimbursement include uncertainty in clinical estimates or flaws in constructing the economic model [
      • Drummond M.
      • Sculpher M.
      Common methodological flaws in economic evaluations.
      ,
      • Kaltenthaler E.
      • Boland A.
      • Carroll C.
      • et al.
      Evidence Review Group approaches to the critical appraisal of manufacturer submissions for the NICE STA process: a mapping study and thematic analysis.
      ,
      • Hill S.R.
      • Mitchell A.S.
      • Henry D.A.
      Problems with the interpretation of pharmacoeconomic analyses: a review of submissions to the Australian Pharmaceutical Benefits Scheme.
      ]. Ensuring that sufficient detail regarding the methods is provided to allow for consistent decision making but doing so without overwhelming nontechnical users requires some consideration. Some guidelines recommended presenting more detailed tables or technical information on modeling in appendices, which could help audiences focus on the most relevant information in the reports. Given established practices in economic evaluations, greater consistency across economic guidelines than what was observed in our present analysis would be desirable. Although consensus documents such as the Consolidated Health Economic Evaluation Reporting Standards statement did not meet the inclusion criteria for our systematic review [
      • Husereau D.
      • Drummond M.
      • Petrou S.
      • et al.
      Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement.
      ], initiatives such as this could offer consistency and a potential structure for guiding the translation of economic evaluations for less technically oriented audiences.

       Strengths and Limitations

      This is the first work to systematically review the guidance economists are given on how to present economic evaluations and consider this in the context of policymakers and other nontechnical audiences. The consistency of recommendations across 31 guidelines suggests that key concepts were captured in this systematic review. Guidelines were focused on the conduct of health economic evaluations; however, other supportive documents or resources may exist in various countries or through international collaborations (e.g., European network for Health Technology Assessment) that provide insights on presenting economic evaluations to nontechnical audiences. However, guidelines are tools that are frequently accessed by economists. To manage the scope of the review, we limited our review to guidelines that are directly linked to decision-making and/or drug reimbursement processes, for which it may be reasonable that presentation approaches targeting noneconomists were addressed.
      Methodological guidelines to critically appraise the quality of economic evaluations do not exist, and a tool for critically appraising clinical practice guidelines was adapted. Therefore, concepts unique to methodological guidelines may have been missed. To supplement the AGREE II approach to assessing clinical practice guidelines, principles from a framework for communicating methodological recommendations for systematic reviews and meta-analyses were incorporated [

      Trikalinos TA, Dahabreh IJ, Wallace BC, et al. Towards a Framework for Communicating Confidence in Methodological Recommendations for Systematic Reviews and Meta-Analyses. Methods Research Report (Prepared by Tufts Evidence-based Practice Center under contract no. 290-2007-10055-I). AHRQ Publication No. 13-EHC119-EF. Rockville, MD: Agency for Healthcare Research and Quality, 2013. Available from: www.effectivehealthcare.ahrq.gov/reports/final.cfm.

      ]. This included allowing for different sources of supporting evidence in guidelines (e.g., simulation studies and methodological case studies) and considering the context in which the guidelines would be applied (e.g., setting, target audience, and outcomes to be optimized by the guidelines). Despite the limitations of adapting these approaches to the current systematic review, the appraisal allowed identification of some key issues in the quality of the guidelines. For example, few guidelines reported that a systematic search for evidence was conducted or identified a rationale for the evidence upon which the guideline recommendations were based. Without this level of transparency, it is unclear whether or how guideline developers considered alternative options when formulating recommendations. In addition, not all guidelines reported authors’ affiliations and/or financial or intellectual conflicts of interest, which may have influenced guideline recommendations. The quality of most guidelines was low around these aspects and should be considered in future if more efforts are devoted to enhancing guidelines on how to present economic evaluations for different end users. Extracting qualitative data from a wide variety of guideline documents requires judgment. However, this task was facilitated by the use of a pre-existing database of guideline characteristics [

      International Society for Pharmacoeconomics and Outcomes Research. Pharmacoeconomic (ISPOR) guidelines around the world. Available from: http://www.ispor.org/PEguidelines/index.asp. [Accessed May 19, 2014].

      ,
      • Eldessouki R.
      • Smith M.D.
      Health care system information sharing: a step toward better health globally.
      ].

       Policy Implications/Interpretation

      Economists have consistent presentation formats and reporting elements that should be considered when providing information to policymakers. However, tailoring information to noneconomists is also important to enhance the accessibility of economic evaluations and their legitimacy in health policy decision making. Although policymakers have influenced how economic evaluations are conducted, less work has been done on how best to present and communicate economic evaluations to noneconomists. Some considerations for enhancing presentation formats are outlined in the following section:
      • 1.
        Presentations of economic evaluations are currently targeted to technical users. Policymakers have reported challenges understanding technical concepts such as QALYs and current graphics such as cost-effectiveness curves [
        • Chen L.C.
        • Ashcroft D.M.
        • Elliott R.A.
        Do economic evaluations have a role in decision-making in medicine management committees? A qualitative study.
        ,
        • Hoffman C.
        • Stoykova B.A.
        • Nixon J.
        • et al.
        Do health care decision makes find economic evaluations useful? The findings of focus group research in UK health authorities.
        ,
        • Williams I.
        • McIver S.
        • Moore D.
        • Bryan S.
        The use of economic evaluations in NHS decision-making: a review and empirical investigation.
        ]. Linking technical concepts to ideas that have practical relevance in a decision-making context and that are already familiar to policymakers could be valuable. In addition, ensuring educational resources and initiatives are in place for policymakers to gain a better understanding of commonly reported technical elements is important.
      • 2.
        Policymakers have indicated that disaggregating results and focusing on the practical relevance of results is important. Although disaggregation is important, on its own, it may be insufficient for enhancing policymaker’s understanding. Following frameworks on how to communicate science to the lay public could guide audiences toward a better understanding these technical concepts (e.g., judicious choice of content, structuring and organizing knowledge, and use of analogies to explain novel concepts and use of narratives) [
        • Baram-Tsabari A.
        • Lewenstein B.V.
        An instrument for assessing scientists’ written skills in public communication of science.
        ].
      • 3.
        Even when reporting guidelines are followed, there will always be complex methodological and modeling decisions that are open to interpretation and their relevance to decision-making context should be considered. For example, limitations in the clinical evidence base and uncertain extrapolations of these data frequently influence the results of economic evaluations [
        • Drummond M.
        • Sculpher M.
        Common methodological flaws in economic evaluations.
        ,
        • Kaltenthaler E.
        • Boland A.
        • Carroll C.
        • et al.
        Evidence Review Group approaches to the critical appraisal of manufacturer submissions for the NICE STA process: a mapping study and thematic analysis.
        ,
        • Hill S.R.
        • Mitchell A.S.
        • Henry D.A.
        Problems with the interpretation of pharmacoeconomic analyses: a review of submissions to the Australian Pharmaceutical Benefits Scheme.
        ]. Ensuring these uncertainties and potential limitations are clearly explained to noneconomists is important for enhancing credibility and consistency in health policy decision making.

      Future Research and Conclusions

      Knowledge translation approaches have frequently been applied to enhancing the understandability and accessibility of clinical trial evidence for policymakers, health care providers, and patients [
      • Chambers D.
      • Wilson P.
      A framework for production of systematic review based briefings to support evidence-informed decision-making.
      ,
      • Akl E.A.
      • Guyatt G.H.
      • Irani J.
      • et al.
      “Might” or “suggest”? No wording approach was clearly superior in conveying the strength of recommendation.
      ,
      • Trevena L.
      • Davey H.
      • Barratt A.
      • et al.
      A systematic review on communicating with patients about evidence.
      ]. Some of these initiatives engage policymakers in developing systematic review or clinical practice guideline summaries to ensure relevance for stakeholders. However, these approaches have not been applied in the field of health economics and could be used to develop technically accurate but more simplified explanations of economic evaluations for general audiences. For example, in addition to providing technical reports and scientific publications of economic evaluations, decision support tools could be developed for complex decisions. These tools could also be used to support educational initiatives or as supplementary resources for nontechnical audiences. With further development of presentation formats that are tailored to noneconomists, individuals will be able to better apply the results of economic evaluations to policy decisions and enhance the transparency and legitimacy of decision-making processes.
      Source of financial support: This research was supported through a grant from the Canadian Institute of Health Research Drug Safety and Effectiveness Network (funding reference no. 116573).

      Supplementary Materials

      References

        • Schlander M.
        The use of cost-effectiveness by the National Institute for Health and Clinical Excellence (NICE): no(t yet an) exemplar of a deliberative process.
        J Med Ethics. 2008; 34: 534-539
        • Drummond M.
        Twenty years of using economic evaluations for drug reimbursement decisions: what has been achieved?.
        J Health Polit Policy Law. 2013; 38: 1081-1102
        • Daniels N.
        • Sabin J.
        Limits to health care: fair procedures, democratic deliberations and the legitimacy problem for insurers.
        Philos Public Aff. 1997; : 303-350
        • Lehoux P.
        • Denis J.L.
        • Tailliez S.
        • et al.
        Dissemination of health technology assessments: identifying the visions guiding an evolving policy innovation in Canada.
        J Health Polit Policy Law. 2005; 30: 603-641
        • Thurston S.J.
        • Craig D.
        • Wilson P.
        • Drummond M.F.
        Increasing decision-makers’ access to economic evaluations: alternative methods of communicating the information.
        Int J Technol Assess Health Care. 2008; 24: 151-157
        • Chaikledkaew U.
        • Lepitakpong C.
        • Teerawattananon Y.
        • et al.
        The current capacity and future development of economic evaluation for policy decision making: a survey among researchers and decision makers in Thailand.
        Value Health. 2009; 12: S31-S35
        • Phillips C.J.
        • Fordham R.
        • March K.
        • et al.
        Exploring the role of economics in prioritization in public health: what do stakeholders think?.
        Eur J Public Health. 2011; 21: 578-584
        • Choi B.
        • Pang T.
        • Lin V.
        • et al.
        Can scientists and policy makers work together?.
        J Epidemiol Community Health. 2005; 59: 632-637
        • Taylor-Robinson D.
        • Milton B.
        • Lloyd Williams F.
        • et al.
        Policy makers’ attitudes to decision support models for coronary heart disease: a qualitative study.
        J Health Serv Res Policy. 2008; 13: 209-214
        • Davies L.
        • Coyle D.
        • Drummond M.
        Current status of economic appraisal of health technology in the European community: report of the network.
        Soc Sci Med. 1994; 38: 1601-1607
        • Hoffman C.
        • Graf von der Schulenburg J.M.
        The influence of economic evaluation studies on decision making: a European survey.
        Health Policy. 2000; 52: 179-192
        • Husereau D.
        • Drummond M.
        • Petrou S.
        • et al.
        Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement.
        BMJ. 2013; 346: f1049
        • Canadian Agency for Drugs and Technologies in Health
        Guidelines for the Economic Evaluation of Health Technologies: Canada.
        3rd ed. Canadian Agency for Drugs and Technologies in Health, Ottawa, Canada2006
        • Galani C.
        • Rutten F.
        Self-reported healthcare decision-makers’ attitudes towards economic evaluations of medical technologies.
        Curr Med Res Opin. 2008; 24: 3049-3058
      1. Straus S. Tetroe J. Graham I.D. Knowledge Translation in Health Care: Moving from Evidence to Practice. 2nd ed. Blackwell Publishing, . ?>Oxford2013
      2. Cochrane Collaboration. Cochrane Handbook for Systematic Reviews. Available from: www.cochrane.org. [Accessed May 19, 2014].

        • Canadian Agency for Drugs and Technologies in Health
        Grey Matters: a Practical Search Tool for Evidence-Based Medicine.
        2nd ed. Canadian Agency for Drugs and Technologies in Health, Ottawa, Canada2013
      3. International Society for Pharmacoeconomics and Outcomes Research. Pharmacoeconomic (ISPOR) guidelines around the world. Available from: http://www.ispor.org/PEguidelines/index.asp. [Accessed May 19, 2014].

      4. Enhancing the QUAlity and Transparency Of health Research (EQUATOR). Available from: http://www.equator-network.org. [Accessed November 24, 2014].

      5. Guidelines International Network. Available from: http://www.g-i-n.net. [Accessed November 24, 2014].

        • Brouwers M.C.
        • Kho M.E.
        • Browman G.P.
        • et al.
        AGREE II: advancing guideline development, reporting and evaluation in health care.
        J Clin Epidemiol. 2010; 63: 1308-1311
      6. Trikalinos TA, Dahabreh IJ, Wallace BC, et al. Towards a Framework for Communicating Confidence in Methodological Recommendations for Systematic Reviews and Meta-Analyses. Methods Research Report (Prepared by Tufts Evidence-based Practice Center under contract no. 290-2007-10055-I). AHRQ Publication No. 13-EHC119-EF. Rockville, MD: Agency for Healthcare Research and Quality, 2013. Available from: www.effectivehealthcare.ahrq.gov/reports/final.cfm.

        • Eldessouki R.
        • Smith M.D.
        Health care system information sharing: a step toward better health globally.
        Value Health Regional. 2012; 1: 118-129
        • Institute of Technology Assessment in Health
        Manual for the Development of Economic Evaluations in Health. Bogotá.
        IETS, Columbia2014
        • Bae S.
        • Lee S.
        • Bae E.
        • Jang S.
        Korean guidelines for pharmacoeconomic evaluation (second and updated version): consensus and compromise.
        Pharmacoeconomics. 2013; 31: 257-267
        • Health Insurance Review and Assessment Service
        Pharmaceutical Economic Evaluation Guidelines.
        Health Insurance Review Agency, South Korea2006
        • Norwegian Medicines Agency (NOMA)
        Guidelines on How to Conduct Pharmacoeconomic Analyses.
        The Norwegian Medicines Agency (NOMA), Norway2012
        • Chinese Pharmacoeconomic Evaluation Guide Task Force
        China Guidelines for Pharmacoeconomic Evaluations.
        Pharmacoeconomics China, China2011
        • Health Information and Quality Authority
        Guidelines for the Economic Evaluation of Health Technologies in Ireland.
        Health Information and Quality Authority, Ireland2010
        • Lopez-Bastida J.
        • Oliva J.
        • Antonanzas F.
        • et al.
        Spanish recommendations on economic evaluation of health technologies.
        Eur J Health Econ. 2010; 11: 513-520
        • Avksentieva M.V.
        • Antonova N.V.
        • Arutyunov G.P.
        • et al.
        Procedure for Clinical and Economic Evaluation of Drug Lists That are Submitted for Reimbursement Coverage from Public Health Care Budget.
        Research Center for Clinical and Economic Evaluation and Pharmacoeconomics of the Russian State Medical University, Russia2010
        • Ministry of Health, Pharmaceutical Administration
        Guidelines for the Submission of a Request to Include a Pharmaceutical Product in the National List of Health Services, Version 8.
        Ministry of Health Pharmaceutical Administration, Israel2010
        • Department of Health
        Regulations Relating to a Transparent Pricing System for Medicines and Scheduled Substances: (Draft Guidelines for Pharmacoeconomic Submissions).
        Department of Health, South Africa2010
        • Academy of Managed Care Pharmacy (AMCP)
        The AMCP Format for Formulary Submissions Version 3.0.
        Academy of Managed Care Pharmacy, United States. Alexandria, VA2009
        • Institute for Quality and Efficiency in Health Care (IQWiG)
        General Methods for the Assessment of the Relation of Benefits to Costs, Version 1.0.
        Institute for Quality and Efficiency in Health Care (IQWiG), Germany2009
        • Ministry of Social Affairs and Health
        Guidelines for Preparing a Health Economic Evaluation [unofficial translation].
        Ministry of Social Affairs and Health, Finland2009
        • Agency for Health Technology Assessment
        Guidelines for Conducting Health Technology Assessment, Version 2.1.
        Agency for Health Technology Assessment, Poland2009
        • De Mello Vianna C.M.
        • Caetano R.
        Methodological Guidelines: Economic Evaluation of Health Technologies.
        Ministry of Health, Brazil2009
        • National Institute for Health and Clinical Excellence (NICE)
        Guide to the Methods of Technology Appraisal.
        National Institute for Health and Clinical Excellence, England & Wales2013
        • Cleemput I.
        • Van Wilder P.
        • Vrijens F.
        • et al.
        Guidelines for Pharmacoeconomic Evaluations in Belgium.
        Federaal Kenniscentrum voor de Gezondheidszorg, Belgium2008
        • Ministry of Health
        Guidelines for Economic Evaluation of Health Care Interventions.
        Ministry of Health, Slovak Republic, Slovakia2008
        • Pharmaceutical Benefits Advisory Committee (PBAC)
        Guidelines for Preparing Submissions to the Pharmaceutical Benefits Advisory Committee, Version 4.3.
        Pharmaceutical Benefits Advisory Committee, Australia2008
        • Bertozzi S.M.
        • Mendoza A.V.
        • Garduno A.H.
        • Barajas E.R.
        Guidelines for Economic Appraisal Studies for Updating the Health Sector’s National Formulary in Mexico.
        National Institute of Public Health, Mexico2008
        • Teerawattananon Y.
        • Chaikledkaew U.
        Thai health technology assessment guideline development.
        J Med Assoc Thai. 2008; 91: S11-S15
        • Scottish Medicines Consortium (SMC)
        Guidance to Manufacturers for Completion of New Product Assessment Form, Revised.
        Scottish Medicines Consortium (SMC), Scotland2007
        • Grocott R.
        • Schoeler R.
        • Priest V.
        • et al.
        Prescription for Pharmacoeconomic Analysis, Version 2.
        Pharmaceutical Management Agency (PHARMAC), New Zealand2007
      7. Center for Drug Evaluation. Guidelines of Methodological Standards for Pharmacoeconomic Evaluations. Taiwan: Center for Drug Evaluation, 2006.

        • Health Care Insurance Board
        Guidelines for Pharmacoeconomic Research.
        The Health Care Insurance Board, The Netherlands2006
        • Boulenger S.
        • Ulmann P.
        French Guidelines for the Economic Evaluation of Health Care Technologies.
        Collège des Économistes de la Santé, France2004
        • González A.M.G.
        Methodological Guidelines for Health Economic Evaluation.
        Ministry of Public Health, Cuba2003
        • Edling A.
        • Stenberg A.M.
        General Guidelines for Economic Evaluations from the Pharmaceutical Benefits Board.
        The Pharmaceutical Benefits Board, Sweden2003
        • Behmane D.
        • Lambot K.
        • Irs A.
        • et al.
        Baltic Guideline for Economic Evaluation of Pharmaceuticals (Pharmacoeconomic Analysis).
        The Baltics: Baltic Health Authorities, 2002
        • Szende A.
        • Mogyorosy Z.
        • Muszbek N.
        • et al.
        Methodological guidelines for conducting economic evaluation of healthcare interventions in Hungary: a Hungarian proposal for methodology standards.
        Eur J Health Econ. 2002; 3: 196-206
        • Capri S.
        • Ceci A.
        • Terranova L.
        • et al.
        Guidelines for economic evaluations in Italy: recommendations from the Italian group of pharmacoeconomic studies.
        Drug Inf J. 2001; 35: 189-201
        • Caro J.
        • Eddy D.M.
        • Kan H.
        • et al.
        Questionnaire to assess relevance and credibility of modeling studies for informing health care decision making: an ISPOR-AMCP-NCP Good Practice Task Force report.
        Value Health. 2014; 17: 174-182
        • Drummond M.
        • Sculpher M.
        Common methodological flaws in economic evaluations.
        Med Care. 2005; 43: 5-14
        • Kaltenthaler E.
        • Boland A.
        • Carroll C.
        • et al.
        Evidence Review Group approaches to the critical appraisal of manufacturer submissions for the NICE STA process: a mapping study and thematic analysis.
        Health Technol Assess. 2011; 15: 1-82
        • Hill S.R.
        • Mitchell A.S.
        • Henry D.A.
        Problems with the interpretation of pharmacoeconomic analyses: a review of submissions to the Australian Pharmaceutical Benefits Scheme.
        JAMA. 2000; 283: 2116-2121
        • Chen L.C.
        • Ashcroft D.M.
        • Elliott R.A.
        Do economic evaluations have a role in decision-making in medicine management committees? A qualitative study.
        Pharm World Sci. 2007; 29: 661-670
        • Hoffman C.
        • Stoykova B.A.
        • Nixon J.
        • et al.
        Do health care decision makes find economic evaluations useful? The findings of focus group research in UK health authorities.
        Value Health. 2002; 5: 71-79
        • Williams I.
        • McIver S.
        • Moore D.
        • Bryan S.
        The use of economic evaluations in NHS decision-making: a review and empirical investigation.
        Health Technol Assess. 2008; 12: 1-175
        • Baram-Tsabari A.
        • Lewenstein B.V.
        An instrument for assessing scientists’ written skills in public communication of science.
        Sci Commun. 2013; 35: 56-85
        • Chambers D.
        • Wilson P.
        A framework for production of systematic review based briefings to support evidence-informed decision-making.
        Syst Rev. 2012; 1: 32
        • Akl E.A.
        • Guyatt G.H.
        • Irani J.
        • et al.
        “Might” or “suggest”? No wording approach was clearly superior in conveying the strength of recommendation.
        J Clin Epidemiol. 2012; 65: 268-275
        • Trevena L.
        • Davey H.
        • Barratt A.
        • et al.
        A systematic review on communicating with patients about evidence.
        J Eval Clin Pract. 2006; 12: 13-23