Advertisement

A Comparison of National Guidelines for Network Meta-Analysis

      Abstract

      Objectives

      Within technology appraisals, it is necessary to compare the complete set of treatments that may be used in the patient group under consideration. Randomized controlled trials are a key source of evidence for these comparisons. The techniques of network meta-analysis allow the networks of trial evidence to be evaluated to obtain estimates of comparative efficacy between sets of treatments. These techniques may be the only source of estimates of comparative effectiveness if trials directly comparing the treatments of interest have not been conducted, and may provide useful additional evidence if both direct and indirect comparisons exist.

      Methods

      We examined both published and draft guidelines from reimbursement and health technology appraisal bodies, and considered their recommendations using appropriate methodology for the conduct of indirect comparisons and the assessments of their validity.

      Results

      Guidelines from 33 countries were reviewed. Of these, guidelines from 9 countries—Australia, Belgium, Canada, France, Germany, Scotland, Spain, South Africa, and the United Kingdom (England and Wales)—included detailed recommendations on the conduct of network meta-analysis. The recommendations were summarized.

      Conclusions

      No two recommendations from the multiple national guidelines are mutually exclusive. It is possible to perform one network meta-analysis for submission to multiple national jurisdictions.

      Keywords

      Introduction

      The development of meaningful clinical treatment guidelines and reimbursement policies entails comparisons of all competing treatment interventions. Some commentators consider systematic reviews of randomized controlled trials (RCTs) to provide the highest level of evidence for evidence-based decision making [
      • Owens D.K.
      • Lohr K.N.
      • Atkins D.
      • et al.
      AHRQ series paper 5: grading the strength of a body of evidence when comparing medical interventions--Agency for Healthcare Research and Quality and the effective health-care program.
      ]. RCTs that simultaneously compare all interventions, however, are rarely available in therapeutic areas with multiple treatment options [
      • Ioannidis J.
      Indirect comparisons: the MESH and MESS of clinical trials.
      ].
      Standard pairwise meta-analyses include studies that compare the same two treatments. A network meta-analysis (NMA) extends the analysis to include a network of pairwise comparisons across a range of different interventions and provides estimates of comparative effectiveness for multiple treatments. NMAs are often performed if direct comparisons are unavailable; however, they can also make valuable contributions to the overall body of evidence even when direct comparisons are available by providing estimates based on a combination of direct and indirect evidence [
      • Caldwell D.M.
      • Ades A.E.
      • Higgins J.P.
      Simultaneous comparison of multiple treatments: combining direct and indirect evidence.
      ,
      • Bucher H.C.
      • Guyatt G.H.
      • Griffith L.E.
      • Walter S.D.
      The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials.
      ,
      • Song F.
      • Altman D.G.
      • Glenny A.M.
      • Deeks J.J.
      Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses.
      ,
      • Lu G.
      • Ades A.E.
      Combination of direct and indirect evidence in mixed treatment comparisons.
      ,
      • Song F.
      • Loke Y.K.
      • Walsh T.
      • et al.
      Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: survey of published systematic reviews.
      ]. National regulatory and reimbursement agencies around the world increasingly regard NMA as a key part of the health care decision-making process. Several countries have released guidelines describing their requirements for such an assessment, or developed review documents highlighting the current best practice to inform organizations preparing submissions.
      There is currently a lack of literature comparing national submission requirements for NMA. The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) has devised an online tool for comparing submission guidelines [

      International Society for Pharmacoeconomics and Outcomes Research. Pharmacoenconomic guidelines around the world. 2012. Available from: http://www.ispor.org/peguidelines/index.asp. [Accessed September 18, 2012].

      ]; however, at present it does not include information comparing the conduct of NMAs. Given the transnational nature of therapeutic interventions, and the need for pharmaceutical manufacturers to apply to multiple national jurisdictions to gain regulatory and market access for their products, there is a clear need for the development of a “super set” of requirements that would facilitate the conduct of NMAs acceptable in multiple jurisdictions. The ability to create a single analysis that is acceptable in multiple jurisdictions has the potential to reduce costs for manufacturers and time-to-market for new interventions.

      Methods

      Identification of Relevant Documents

      The sampling frame for the search of national guidelines compared in this review was the countries listed in the Web-based repository of country-specific pharmacoeconomic guidelines maintained by ISPOR [

      International Society for Pharmacoeconomics and Outcomes Research. Pharmacoenconomic guidelines around the world. 2012. Available from: http://www.ispor.org/peguidelines/index.asp. [Accessed September 18, 2012].

      ]. As of July 22, 2013, this comprised guidelines from 33 countries: Australia, Austria, Baltic states (Latvia, Lithuania, and Estonia), Belgium, Canada, China, Cuba, Denmark, England and Wales, Finland, France, Germany, Hungary, Ireland, Israel, Italy, Mexico, New Zealand, The Netherlands, Norway, Poland, Portugal, the Russian Federation, Scotland, Slovak Republic, South Africa, South Korea, Sweden, Thailand, Taiwan, and the United States.
      The ISPOR repository separates guidelines into three categories: Published Pharmacoeconomic Recommendations (economic evaluation guidelines or recommendations published by experts in the field but not officially recognized or required by health care decision-making bodies); Pharmacoeconomic Guidelines (official guidelines or policies concerning economic evaluation that are recognized or required by health care decision-making bodies); and Submission Guidelines (official guidelines or policies concerning drug submission requirements with an economic evaluation component). Documents from all three categories were considered in this review. In addition, working papers and other methodological reports (including the ISPOR task force report on the conduct of indirect comparisons because this was referenced by a number of guidelines) [
      • Hoaglin D.C.
      • Hawkins N.
      • Jansen J.P.
      • et al.
      Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2.
      ], Web sites, and other listed sources were checked to ensure that the most recent versions of documents were reviewed. To this end, documents in draft were also included in this review. For the purposes of this review, documents were classified as either guidelines or methods reviews.
      Guidelines or methods reviews were screened for references to indirect comparisons or NMA, with documents from 14 of the 33 countries included in the review containing references to the use, conduct, or reporting of NMA. Of these, guidelines from five countries (Ireland, Norway, Poland, Sweden, and the United States) made reference to the potential use of indirect comparisons in technology appraisals but did not provide any further detailed guidance as to their conduct and reporting. For example, the Irish guidelines stated that “In the event of limited head-to-head RCT data, mixed treatment comparisons can be used” [

      An tÙdaràs Um Fhaisnèis agus Càiliocht Slàinte. Guidelines for the economic evaluation of health technologies in Ireland. 2010. Available from: http://www.hiqa.ie/publication/guidelines-economic-evaluation-health-technologies-ireland. [Accessed November 7, 2013].

      ], the United States’ Academy of Managed Care Pharmacy guidance mentions indirect comparisons under the heading of “Other Supporting Evidence” and noted that “Today, network meta-analyses are becoming more relied on and accepted as valid means to compare interventions” [

      AMCP Format Executive Committee and Working Group. The AMCP format for formulary submissions version 3.1. 2012. Available from: http://amcp.org/practice-resources/amcp-format-formulary-submisions.pdf. [Accessed November 7, 2013].

      ], and the Swedish guidelines issued by Tandvårds-och läkemedelsförmånsverkets föreskrifter contained very few requirements, and instead referenced the ISPOR task force report [

      Tandvårds- och läkemedelsförmånsverket. Handbok till Tandvårds- och läkemedelsförmånsverkets föreskrifter (TLVFS 2011:3) om ansökan om pris och subvention för förbrukningsartiklar. 2011. Available from: http://www.tlv.se/Upload/Foretag/Handbok-TLVFS2011-3-subvention-forbrukningsartiklar-sept-2011.pdf. [Accessed June 7, 2013].

      ]. Documents from the remaining nine countries (Australia, Belgium, Canada, England and Wales, France, Germany, Scotland, Spain, and South Africa) provided more detailed guidance, which is summarized in this article. These documents are summarized in Table 1.
      Table 1Summary of national guidelines.
      NationBodyDocument titleYear publishedAuthorsGuideline or method review
      AustraliaPharmaceutical Benefits Advisory Committee (PBAC)Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee (version 4.3)

      Pharmaceutical Benefits Advisory Committee. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee (version 4.3). 2008. Available from: http://www.health.gov.au/internet/main/publishing.nsf/Content/AECB791C29482920CA25724400188EDB/$File/PBAC4.3.2.pdf. [Accessed June 7, 2013].

      2008NSGuideline + methods review
      PBAC Indirect Comparisons Working GroupReport of the Indirect Comparisons Working Group to the Pharmaceutical Benefits Advisory Committee: assessing indirect comparisons

      ICWG. Report of the Indirect Comparisons Working Group (ICWG) to the Pharmaceutical Benefits Advisory Committee (PBAC): Assessing Indirect Comparisons. ICWG. 2013. Available from: http://www.pbs.gov.au/industry/useful-resources/PBAC_feedback_files/ICWG%20Report%20FINAL2.pdf. [Accessed October 29, 2013]

      2008Carlin J, Coory M, Defina J, Eckermann S, Frauman A, Hunt L, McCloud P, McColl G, Sansom L, Viney R, Yuen CGuideline + methods review
      BelgiumFederaal Kenniscentrum voor de Gezondheidszorg Centre fédéral d’expertise des soins de santé (KCE)Guidelines for pharmacoeconomic evaluations in Belgium KCE reports 78C

      Cleemput I, van Wilder P, Vrijens F, et al. Guidelines for pharmacoeconomic evaluations in Belgium: KCE reports 78C. 2008. Available from: https://kce.fgov.be/publication/report/guidelines-for-pharmacoeconomic-evaluations-in-belgium. [Accessed June 7, 2013].

      2008Cleemput I, Van Wilder P, Vrijens P, Huybrechts M, Ramaekers DGuideline
      CanadaCanadian Agency for Drugs and Technology in Health (CADTH)Indirect evidence: indirect treatment comparisons in meta-analysis

      Wells G, Sultan S, Chen L, et al. Indirect evidence: indirect treatment comparisons in meta-analysis. 2009. Available from: http://www.cadth.ca/media/pdf/H0462_itc_tr_e.pdf. [Accessed June 7, 2013].

      2009Wells GA, Sultan SA, Chen L, Khan M, Coyle DMethods review
      CADTHGuidelines for the economic evaluation of health technologies: Canada

      Contandriopolous A, Coyle D, Hailey D, et al. Guidelines for the economic evaluation of health technologies: Canada. 2006. Available from: http://www.cadth.ca/media/pdf/186_EconomicGuidelines_e.pdf. [Accessed June 7, 2013].

      2006Contandriopolous AP, Hailey D, Mamdani M, Coyle D, Jacobs P
      England and WalesNational Institute for Health and Care Excellence (NICE)Guide to the methods of technology appraisal 2013

      National Institute for Health and Care Excellence. Guide to the methods of technology appraisal. 2013. Available from: http://publications.nice.org.uk/guide-to-the-methods-of-technology-appraisal-2013-pmg9. [Accessed June 7, 2013].

      2013NSGuideline
      FranceCollège des Èconomistes de la SantèFrench guidelines for the economic evaluation of health care technologies

      Collège des Èconomistes de la Santè. French guidelines for the economic evaluation of health care technologies. 2004. Available from: http://www.ces-asso.org/docs/France_Guidelines_HE_Evaluation.PDF. [Accessed June 7, 2013].

      2004Boulenger S, Ulmann PGuideline
      Haute Autorité de Santé (HAS)Indirect comparisons— methods and validity

      Cucherat M, Izard V. Indirect comparisons: methods and validity. 2010. Available from: http://www.has-sante.fr/portail/upload/docs/application/pdf/2011-02/summary_report__indirect_comparisons_methods_and_validity_january_2011_2.pdf. [Accessed June 7, 2013].

      2009Cucherat M, Izard VMethods review
      GermanyGemeinsamer BundesausschussDossier for the Benefit Assessment pursuant to Section 35a of the German Social Code Book Five – Module 4

      Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

      2011NSGuideline
      ScotlandScottish Medicines Consortium (SMC)Guidance to manufacturers for completion of New Product Assessment Form (NPAF)

      Scottish Medicines Consortium. Guidance to manufacturers for completion of New Product Assessment Form (NPAF). 2013 . Available from: http://www.scottishmedicines.org.uk/files/submissionprocess/Guidance_on_NPAF_V4_0_-_August_2013.doc. [Accessed August 27, 2013].

      2013NSGuideline
      South AfricaDepartment of Health: Directorate PEEThe guidelines for pharmacoeconomic evaluations of medicine and scheduled substances

      South Africa Department of Health. The guidelines for pharmacoeconomic evaluations of medicines and scheduled substances. 2013. Available from: http://www.info.gov.za/view/DownloadFileAction?id=183162. [Accessed June 7, 2013].

      2013NSGuideline
      SpainGENESIS Group of the Spanish Society of Hospital Pharmacy (SEFH)A check-list for critical appraisal of indirect comparisons (A. Ortega, M. Fraga, E. Alegre, et al., personal communication, 2013)2013Ortega A, Fraga MD, Alegre E, Puigventos F, Porta MA, Ventayol PMethods review
      ISPORInternational Society for Pharmacoeconomics and Outcomes ResearchConducting indirect treatment comparison and network meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2
      • Hoaglin D.C.
      • Hawkins N.
      • Jansen J.P.
      • et al.
      Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2.
      2011Hoaglin DC, Hawkins N, Jansen JP, Scott DA, Itzler R, Cappelleri JC, Boersman C, Thompson D, Larholt HM, Diaz M, Barrett AMethods review
      ISPORInternational Society for Pharmacoeconomics and Outcomes ResearchIndirect treatment comparison/network meta-analysis study questionnaire to assess relevance and credibility to inform healthcare decision making: an ISPOR-AMCP-NPC Good Practice Task Force Report
      • Jansen J.P.
      • Trikalinos T.
      • Cappelleri J.C.
      • et al.
      Indirect treatment comparison/network meta-analysis study questionnaire to assess relevance and credibility to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report.
      2014Jansen JP, Trikalinos T, Cappelleri JC, Andes S, Daw J, Andes S, Eldessouki R, Salanti GMethods review
      IrelandAn tÙdaràs Um Fhaisnèis agus Càiliocht SlàinteGuidelines for the Economic Evaluation of Health Technologies in Ireland.

      An tÙdaràs Um Fhaisnèis agus Càiliocht Slàinte. Guidelines for the Economic Evaluation of Health Technologies in Ireland. HIQA 2010. Available from: http://www.hiqa.ie/publication/guidelines-economic-evaluation-health-technologies-ireland. [Accessed November 7, 2013]

      2010NSGuideline
      NorwayStatens legemiddelverkGuidelines on how to conduct pharmacoeconomic analyses

      Guidelines on how to conduct pharmacoeconomic analyses. Statens legemiddelverket. 2012. Available from: http://www.legemiddelverket.no/English/price_and_reimbursement/application_for_reimursement/Lists/PageAttachments/default/NO/Pharmacoeconomic%20guidelines%20-%20Norway.pdf. [Accessed November 7, 2013]

      2012NSGuideline
      PolandAgencja Oceny Technologii MedyczncyhGuidelines for conducting health technology assessment (HTA)

      Agencja Oceny Technologii Medyczncyh. Guidelines for Conducting Health Technology Assessment (HTA). AOTM. 2009. Available from: http://www.aotm.gov.pl/assets/files/wytyczne_hta/2009/Guidelines_HTA_eng_MS_29062009.pdf. [Accessed November 7, 2013]

      2009Agencja Oceny Technologii MedyczncyhGuideline
      SwedenTandvårds- och läkemedelsförmånsverketHandbok till Tandvårds- och läkemedelsförmansverkets föreskrifter (TLVFS 2011:3) om ansökan om pris och subvention för förbrukningsartiklar

      Tandvårds- och läkemedelsförmånsverket. Handbok till Tandvårds- och läkemedelsförmånsverkets föreskrifter (TLVFS 2011:3) om ansökan om pris och subvention för förbrukningsartiklar. 2011. Available from: http://www.tlv.se/Upload/Foretag/Handbok-TLVFS2011-3-subvention-forbrukningsartiklar-sept-2011.pdf. [Accessed June 7, 2013].

      NSGuideline
      United StatesAcademy of Managed Care Pharmacy (AMCP)The AMCP format for formulary submissions version 3.1

      AMCP Format Executive Committee and Working Group. The AMCP format for formulary submissions version 3.1. 2012. Available from: http://amcp.org/practice-resources/amcp-format-formulary-submisions.pdf. [Accessed November 7, 2013].

      2012AMCP Format Executive Committee and Working GroupGuideline
      ISPOR, International Society for Pharmacoeconomics and Outcomes Research; NS, not specified.

      Comparison of National Guidelines

      The national guidelines were initially reviewed, and checklists were developed to summarize their recommendations. These checklists were completed for each of the guidelines by two separate reviewers. A final review of the guidelines was conducted and any additional items required were added to the checklists. Finally, the checklists were compared across reviewers and any discrepancies were resolved.

      Results

      The recommendations made in the guidelines are described under the following headings: clinical trial search, selection of databases, study selection, bias assessment, and conduct of NMA. Each heading comprises a number of potential recommendations. For each recommendation, we have noted whether it is referred to in the corresponding national guideline; we make no distinction between a “recommendation” and a “requirement.”

      Clinical Trial Search

      The first step in carrying out an NMA is to identify the clinical trials that may potentially form the network of comparisons. Table 2 details recommendations regarding the design, conduct, and reporting of the trial search. These recommendations can be divided into four categories: 1) Definition of search time frame; this allows regulators to assess whether the time frame is adequate; 2) Predefinition of search parameters; typically the population(s), intervention(s), comparator(s), outcome(s), study design approach to reporting studies [

      Centre for Reviews and Dissemination. Systematic Reviews—CRD’s Guidance for Undertaking Reviews in Health Care. York: University of York, 2009.

      ]. This improves transparency and increases confidence in the study findings; 3) Clear description of search conduct; most of the national guidelines require that the search strategy be presented in full with all the terms and relationships documented, and many guidelines require a flow diagram with “n” returns at each step; and 4) Manually checking reference lists in identified articles to increase the sensitivity of the search.
      Table 2Recommendations regarding the design, conduct, and reporting of trial searches.
      CountryReport date(s) search conductedSpecify search date spanSpecify PICO criteriaJustify restrictions, e.g., language and years searchedDescribe search terms and relationshipPresent strategy in blocks by indication, intervention, study type, etc.Perform supplementary searches/manual checkingPresent PRISMA-type diagram showing study dispositionConduct search in accordance with CRD systematic review procedures
      Australia
      Belgium
      Canada
      England and Wales
      France
      The French methods review document contains details of a search strategy used for that review, but not for reviews in general.
      Germany
      Scotland
      South Africa
      Spain
      ISPOR Task Force Best Practice Guide
      ISPOR NMA Assessment Questionnaire
      CRD, Centre for Reviews and Dissemination; NMA, network meta-analysis: PICO, Patient-Intervention-Comparator-Outcome; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.
      low asterisk The French methods review document contains details of a search strategy used for that review, but not for reviews in general.
      There is an overall focus on the transparency and repeatability of the search. Canada and England and Wales require that the search complies with best-practice guidelines issued by the Centre for Reviews and Dissemination [

      Contandriopolous A, Coyle D, Hailey D, et al. Guidelines for the economic evaluation of health technologies: Canada. 2006. Available from: http://www.cadth.ca/media/pdf/186_EconomicGuidelines_e.pdf. [Accessed June 7, 2013].

      ,

      National Institute for Health and Care Excellence. Guide to the methods of technology appraisal. 2013. Available from: http://publications.nice.org.uk/guide-to-the-methods-of-technology-appraisal-2013-pmg9. [Accessed June 7, 2013].

      ]. Germany requires that keywords, MeSH identifiers, and other terms used to search electronic databases be grouped into related blocks in the presentation of the search strategy [

      Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

      ].

      Selection of Databases

      Most of the national pharmacoeconomic guidelines specify which databases should be searched. Table 3 lists the various databases listed in the national guidelines. MEDLINE, EMBASE, and the Cochrane (CENTRAL) databases form a core specified by almost all the national guidelines. Outside of this core there is variation, with some jurisdictions requiring that the search be conducted in databases with a local focus and others requiring more emphasis on clinical trial databases. Four of the nine national guidelines require that the search be conducted in an international registry of clinical trials, either clinicaltrials.gov or the International Clinical Trials Registry Platform, but typically both [

      Contandriopolous A, Coyle D, Hailey D, et al. Guidelines for the economic evaluation of health technologies: Canada. 2006. Available from: http://www.cadth.ca/media/pdf/186_EconomicGuidelines_e.pdf. [Accessed June 7, 2013].

      ,

      National Institute for Health and Care Excellence. Guide to the methods of technology appraisal. 2013. Available from: http://publications.nice.org.uk/guide-to-the-methods-of-technology-appraisal-2013-pmg9. [Accessed June 7, 2013].

      ,

      Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

      ,

      Pharmaceutical Benefits Advisory Committee. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee (version 4.3). 2008. Available from: http://www.health.gov.au/internet/main/publishing.nsf/Content/AECB791C29482920CA25724400188EDB/$File/PBAC4.3.2.pdf. [Accessed June 7, 2013].

      ]. The German guideline references the industry-maintained clinicalstudyresults.org database, which has been closed since the publication of the German guidelines [

      Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

      ]. The national guideline document issued by the Australian Pharmaceutical Benefits Advisory Committee requires that the Australia New Zealand Clinical Trials Registry form part of the search strategy [

      Pharmaceutical Benefits Advisory Committee. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee (version 4.3). 2008. Available from: http://www.health.gov.au/internet/main/publishing.nsf/Content/AECB791C29482920CA25724400188EDB/$File/PBAC4.3.2.pdf. [Accessed June 7, 2013].

      ]. The Australia New Zealand Clinical Trials Registry forms part of the International Clinical Trials Registry Platform search portal that is required by other national guidelines; however, the Pharmaceutical Benefits Advisory Committee specifically differentiates between the two. German and Australian guidelines require that company-specific databases be searched and results presented, although the national guidelines contain no indication how the transparency and repeatability of such a search would be enforced [

      Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

      ,

      Pharmaceutical Benefits Advisory Committee. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee (version 4.3). 2008. Available from: http://www.health.gov.au/internet/main/publishing.nsf/Content/AECB791C29482920CA25724400188EDB/$File/PBAC4.3.2.pdf. [Accessed June 7, 2013].

      ].
      Table 3Specified databases.
      CountryMEDLINEEmbaseCochrane (CENTRAL)Cochrane Database of Systematic ReviewsClinicaltrials.govClinicalstudyresults.org
      Database decommissioned Q4 2011.
      ICTRP search portalNHS CRDANZCTRSubject-specific databases or registers
      For example, PsycINFO (psychology and psychiatry), MANTIS (osteopathy and chiropractic), CINAHL (nursing and allied health).
      Manufacturers’ internal databases
      Australia
      Belgium
      Canada
      England and Wales
      France
      Germany
      Scotland
      South Africa
      Spain
      ISPOR Task Force Best Practice Guide
      ISPOR NMA Assessment Questionnaire
      ANZCTR, Australia New Zealand Clinical Trials Registry (formerly the Australia Clinical Trials Registry); CENTRAL, Cochrane Central Register of Controlled Trials; CRD, Centre for Reviews and Dissemination; EED, Economic Evaluation Database; HEED, Health Economic Evaluation Database; ICTRP, International Clinical Trials Registry Platform; ISPOR, International Society for Pharmacoeconomics and Outcomes Research; NHS, National Health Service; NMA; network meta-analysis.
      low asterisk Database decommissioned Q4 2011.
      For example, PsycINFO (psychology and psychiatry), MANTIS (osteopathy and chiropractic), CINAHL (nursing and allied health).
      The French, Scottish, and Spanish guideline documents do not contain recommendations or requirements regarding the search strategy to be implemented or databases searched [

      Cucherat M, Izard V. Indirect comparisons: methods and validity. 2010. Available from: http://www.has-sante.fr/portail/upload/docs/application/pdf/2011-02/summary_report__indirect_comparisons_methods_and_validity_january_2011_2.pdf. [Accessed June 7, 2013].

      ,

      Collège des Èconomistes de la Santè. French guidelines for the economic evaluation of health care technologies. 2004. Available from: http://www.ces-asso.org/docs/France_Guidelines_HE_Evaluation.PDF. [Accessed June 7, 2013].

      ,

      Scottish Medicines Consortium. Guidance to manufacturers for completion of New Product Assessment Form (NPAF). 2013 . Available from: http://www.scottishmedicines.org.uk/files/submissionprocess/Guidance_on_NPAF_V4_0_-_August_2013.doc. [Accessed August 27, 2013].

      ] (A. Ortega, M. Fraga, E. Alegre, et al., personal communication, 2013). The French methods review document does provide details of the search strategy used in the review document itself, but not for identifying trials as part of an NMA.

      Study Selection

      Following the completion of the search, it is necessary to determine which studies should be included in the NMA. The requirements for the study selection process are listed in Table 4. In many cases, they are less rigorous than the methods recommended by either the Centre for Reviews and Dissemination (CRD) and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses [

      Centre for Reviews and Dissemination. Systematic Reviews—CRD’s Guidance for Undertaking Reviews in Health Care. York: University of York, 2009.

      ,
      • Moher D.
      • Liberati A.
      • Tetzlaff J.
      • Altman D.G.
      Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
      ], with less than half of all national guidelines stipulating explicitly that two reviewers should carry out the selection process.
      Table 4Study selection.
      CountrySelection processTrials include
      Predefined inclusion/exclusion criteria processJustified selection criteriaSelected by two independent reviewersMaintained log of trials deemed ineligibleTrials form one connected network of interventionsRandomized allocation
      Criteria for randomized controlled trials (RCTs).
      Separate arms
      Criteria for randomized controlled trials (RCTs).
      Demographic homogeneity between trialsSimilar prognostic severity
      Australia
      Belgium
      Canada
      England and Wales
      France
      Germany
      Scotland
      South Africa
      Spain
      ISPOR Task Force Best Practice Guide
      ISPOR NMA Assessment Questionnaire
      ISPOR, International Society for Pharmacoeconomics and Outcomes Research; NMA, network meta-analysis.
      low asterisk Criteria for randomized controlled trials (RCTs).
      Almost all national guidelines specify that inclusion and exclusion criteria must be defined beforehand. This is crucial not only to inform the initial search but also to ensure that selection bias is reduced and all appropriately designed studies are included regardless of direction or magnitude. Aside from this consistently applied criterion, there are few proscriptions to the study selection process; and apart from the Australian guidelines, there are few requirements as to the design of the studies themselves. The guidelines from the Pharmaceutical Benefits Advisory Committee require clinical trials included in the meta-analysis to have randomized allocation to separate arms, which are themselves criteria for an RCT [

      Pharmaceutical Benefits Advisory Committee. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee (version 4.3). 2008. Available from: http://www.health.gov.au/internet/main/publishing.nsf/Content/AECB791C29482920CA25724400188EDB/$File/PBAC4.3.2.pdf. [Accessed June 7, 2013].

      ].

      Bias Assessment

      Estimates of treatment effect obtained from NMAs may be biased by the effects of heterogeneity between studies. Table 5 presents the recommended steps taken to assess the potential for bias within NMAs. Almost all national guidelines require an assessment of the adequacy of blinding and that the results of the analysis of the intent-to-treat population are used in the analysis. Also, an assessment of variance between trial protocol and standard practice, and comparison of rates of dropout between the arms of the study, is widely required.
      Table 5Bias assessment.
      Source of biasIdentification and selection of studiesConduct and reporting of clinical trialsNetwork meta-analysis
      National guidelineAssess level of bias within included studiesHomogeneity of prognostic severityDescribe the design and methodology according to CONSORT guidelinesAssessment of variance between trial protocol and standard practiceAssess adequacy of blindingComparison of dropout ratesImplementation of ITTAssessment of difference in baseline risk and placebo responseDescribe time horizonInvestigator conflicts of interest reportedReport of subgroup analysisTreatment effect modifiers identified before comparing study resultsAssessment of publication bias and/or funnel plot
      Australia
      Belgium
      Canada
      England and Wales
      France
      Germany
      Scotland
      South Africa
      Spain
      ISPOR Task Force Best Practice Guide
      ISPOR NMA Assessment Questionnaire
      CONSORT, Consolidated Standards of Reporting Trials; ISPOR, International Society for Pharmacoeconomics and Outcomes Research; ITT, intent to treat; NMA, network meta-analysis.
      The two French guideline documents do not contain requirements for this aspect of NMA beyond a recommendation that publication bias can be reduced by accessing a clinical trials registry and including all trials carried out [

      Cucherat M, Izard V. Indirect comparisons: methods and validity. 2010. Available from: http://www.has-sante.fr/portail/upload/docs/application/pdf/2011-02/summary_report__indirect_comparisons_methods_and_validity_january_2011_2.pdf. [Accessed June 7, 2013].

      ,

      Collège des Èconomistes de la Santè. French guidelines for the economic evaluation of health care technologies. 2004. Available from: http://www.ces-asso.org/docs/France_Guidelines_HE_Evaluation.PDF. [Accessed June 7, 2013].

      ]. German and South African guidelines include unique requirements such as describing included studies according to Consolidated Standards of Reporting Trials guidelines [
      • Schulz K.F.
      • Altman D.G.
      • Moher D.
      CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials.
      ], or noting the time horizon of included studies, respectively [

      Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

      ,

      South Africa Department of Health. The guidelines for pharmacoeconomic evaluations of medicines and scheduled substances. 2013. Available from: http://www.info.gov.za/view/DownloadFileAction?id=183162. [Accessed June 7, 2013].

      ]. Australia and Scotland both require demographic homogeneity between trials [

      Pharmaceutical Benefits Advisory Committee. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee (version 4.3). 2008. Available from: http://www.health.gov.au/internet/main/publishing.nsf/Content/AECB791C29482920CA25724400188EDB/$File/PBAC4.3.2.pdf. [Accessed June 7, 2013].

      ,

      Scottish Medicines Consortium. Guidance to manufacturers for completion of New Product Assessment Form (NPAF). 2013 . Available from: http://www.scottishmedicines.org.uk/files/submissionprocess/Guidance_on_NPAF_V4_0_-_August_2013.doc. [Accessed August 27, 2013].

      ]. In addition, German regulators require that the level of bias in studies be assessed, with the stipulation that studies assessed as being biased must not be excluded [

      Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

      ]. The ISPOR NMA assessment questionnaire and the National Institute for Health and Care Excellence Guide to Technology Appraisal are the only documents that recommend that treatment effect modifiers be identified before comparing the results of individual studies [
      • Ioannidis J.
      Indirect comparisons: the MESH and MESS of clinical trials.
      ,

      National Institute for Health and Care Excellence. Guide to the methods of technology appraisal. 2013. Available from: http://publications.nice.org.uk/guide-to-the-methods-of-technology-appraisal-2013-pmg9. [Accessed June 7, 2013].

      ].

      Conduct of NMA

      Table 6a, Table 6b is a breakdown of the statistical techniques, processes, and recommended scales for data required by the national guidelines. It is important to note the relative consistency in the scales preferred. All national guidelines call for a core set of scales for binary or time-to-event data comprising relative risk, relative risk difference, hazard ratio, odds ratio, or a combination thereof. The Spanish guidelines do not contain any recommendations as to the type of scales to be used for data. France, Scotland, and Spain recommend meta-regression to be performed. Where continuous data are reported, the national guidelines are split four to three on whether to use median difference or weighted mean difference, respectively, with South Africa and Spain having no specific requirements. Even though naive indirect comparisons, where the results of individual arms of different trials are compared as though they were from the same trial, produce evidence equivalent only to observational studies, over half the countries considered it necessary to explicitly prohibit their conduct [

      National Institute for Health and Care Excellence. Guide to the methods of technology appraisal. 2013. Available from: http://publications.nice.org.uk/guide-to-the-methods-of-technology-appraisal-2013-pmg9. [Accessed June 7, 2013].

      ,

      Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

      ,

      Collège des Èconomistes de la Santè. French guidelines for the economic evaluation of health care technologies. 2004. Available from: http://www.ces-asso.org/docs/France_Guidelines_HE_Evaluation.PDF. [Accessed June 7, 2013].

      ,
      • Jansen J.P.
      • Trikalinos T.
      • Cappelleri J.C.
      • et al.
      Indirect treatment comparison/network meta-analysis study questionnaire to assess relevance and credibility to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force Report.
      ] (A. Ortega, M. Fraga, E. Alegre, et al., personal communication, 2013).
      Table 6aConduct of indirect comparisons.
      CountryConduct of indirect comparison
      Descriptionofpatient&treatmentcharacteristicsDescribe&justifystatisticalmethod;BayesianorFrequentistIBayesian,describepreviousdistribution,sensitivitytoprevious,andassessconvergencePresentindividualstudyresultsand/orforestplotGraphicallysummarizerankprobabilitiesDescriptionofrelative-effectestimateIncluderationalefor,anddescriptionof,sensitivityanalysesDescriptionofdifferentfindingswithsensitivity/scenarioanalysisMeta-regressionPoissondistributionNaivecomparisonsofpointestimatesoractivearmsareprohibitedIncludeanalysisofthehypothesisofconsistencyAssesstheresultsforeachcommonreferenceacrosstrialsforanyimportantdifferencesPerformconsistencycheckbetweendirectandindirectevidenceAssesshomogeneityofdirectcomparisonsAssessheterogeneitywithCochraneQ,I2RandomeffectsIncludecodeandspecifysoftwarepackageIncludediagramofnetworkstructure
      Australia
      Belgium
      Canada
      England and Wales
      France
      Germany
      Scotland
      South Africa
      Spain
      ISPOR Task Force Best Practice Guide
      ISPOR NMA Assessment Questionnaire
      Table 6bPresentation of results.
      CountryPresentation of results
      Mediandifference(continuous)Weightedmeandifference(continuous)RelativeriskHazardratioOddsratioRelativeriskdifferenceAbsoluteriskreductions
      Australia
      Belgium
      Canada
      England and Wales
      Mentioned explicitly in the 2008 NICE guide to the methods of technology appraisal [31].
      Mentioned explicitly in the 2008 NICE guide to the methods of technology appraisal [31].
      Mentioned explicitly in the 2008 NICE guide to the methods of technology appraisal [31].
      Mentioned explicitly in the 2008 NICE guide to the methods of technology appraisal [31].
      France
      Germany
      Scotland
      South Africa
      Spain
      ISPOR Task Force Best Practice Guide
      ISPOR NMA Assessment Questionnaire
      HR, hazard ratio; ISPOR, International Society for Pharmacoeconomics and Outcomes Research; NICE, National Institute for Health and Care Excellence; NMA, network meta-analysis; OR, odds ratio; RR, relative risk.
      low asterisk Mentioned explicitly in the 2008 NICE guide to the methods of technology appraisal [31].
      None of the national guidelines stipulated what form sensitivity analyses should take, but seven of the nine national jurisdictions require descriptions and rationale for sensitivity analyses, with five countries subsequently requiring a description of the different findings and three needing a relative effect estimate. France, Germany, Scotland, Spain, and Australia require an assessment of the heterogeneity of direct comparisons, with Germany, Scotland, Spain, and Australia also requiring that a Cochrane Q test be performed [

      Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

      ,

      Pharmaceutical Benefits Advisory Committee. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee (version 4.3). 2008. Available from: http://www.health.gov.au/internet/main/publishing.nsf/Content/AECB791C29482920CA25724400188EDB/$File/PBAC4.3.2.pdf. [Accessed June 7, 2013].

      ,

      Collège des Èconomistes de la Santè. French guidelines for the economic evaluation of health care technologies. 2004. Available from: http://www.ces-asso.org/docs/France_Guidelines_HE_Evaluation.PDF. [Accessed June 7, 2013].

      ]. Most of the countries call for a consistency check between direct and indirect evidence, strongly suggesting that indirect evidence should be considered as an adjunct to direct evidence. None of the national guidelines indicated that differences between direct and indirect evidence would render indirect evidence inadmissible.
      Very few aspects of the conduct of the analysis were required only by a single country. Perhaps surprisingly, only the German and Scottish regulators require a network diagram showing the links between the studies. Germany is also the only country to explicitly request the inclusion of any code and the software package used [

      Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

      ].

      Discussion

      For the first time, multiple national guideline documents have been compared with each other and the differing stipulations on the conduct of indirect comparisons extracted. The technique of indirect NMA is recent in the field of health technology assessment. This combination of relative novelty and the precedent of individual national regulatory bodies having jurisdiction over their respective markets means that there are currently no transnational guidelines for the compilation of indirect evidence. ISPOR has published multiple best-practice documents to inform practitioners and regulators [
      • Hoaglin D.C.
      • Hawkins N.
      • Jansen J.P.
      • et al.
      Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2.
      ,
      • Jansen J.P.
      • Fleurence R.
      • Devine B.
      • et al.
      Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 1.
      ,
      • Jansen J.P.
      • Naci H.
      Is network meta-analysis as valid as standard pairwise meta-analysis? It all depends on the distribution of effect modifiers.
      ]. Although these documents were comprehensive when their collective recommendations were combined, they did not capture every nuance of the process of NMA required by every national regulatory body. By using the aggregated recommendations from the tables herein, it becomes possible to create a super set of recommendations that will satisfy the regulatory bodies considered.
      There is an overall focus on the transparency and repeatability of the search process. Certain jurisdictions such as South Africa will conduct a search of their own to mirror that of the submission and will suspend the application if any relevant studies are found to have been omitted [

      South Africa Department of Health. The guidelines for pharmacoeconomic evaluations of medicines and scheduled substances. 2013. Available from: http://www.info.gov.za/view/DownloadFileAction?id=183162. [Accessed June 7, 2013].

      ]. All national guidelines reviewed that consider the systematic review component of NMA require that a search be conducted in MEDLINE, EMBASE, and the Cochrane (CENTRAL) databases, with certain regulatory bodies requiring that additional databases searched be nationally or regionally specific.
      The task of complying with a “one-size-fits-all” super set of requirements is made more onerous by having to perform tasks required by only a single country, although only a handful of these instances were noted. For example, only Germany requires that the design and methodology of the studies be described according to Consolidated Standards of Reporting Trials guidelines; however, this becomes a requirement for the super set. Although the Australian guidelines appear to place proscriptions on trial selection, the requirement that trials included in the meta-analysis have randomized allocation to separate arms is reflective of standard RCT practice. The requirement for demographic homogeneity between trial populations is reflected in the national guidelines of Belgium, Canada, England and Wales, Germany, Scotland, and Spain, which all require that there be homogeneity of prognostic severity between trials as part of their bias assessment.
      It is notable that there is an overall lack of explicit requirements regarding an assessment of the distribution of treatment effect modifiers across studies included in the NMA. An imbalance in the effect modifiers between the direct comparisons suggests that the transitivity assumption may have been violated. Transitivity in this context means that if treatment A is ranked above treatment B in terms of efficacy, and B is ranked above treatment C, then A must be ranked above C [

      Wells G, Sultan S, Chen L, et al. Indirect evidence: indirect treatment comparisons in meta-analysis. 2009. Available from: http://www.cadth.ca/media/pdf/H0462_itc_tr_e.pdf. [Accessed June 7, 2013].

      ]. Despite the lack of explicit requirements, some guidelines do address the problem implicitly; the National Institute for Health and Care Excellence guidelines require that potential treatment effect modifiers be identified before analysis, and the Belgian guidelines state that treatment effect may be influenced by factors that vary across trials as an example of unacceptable confounding bias [
      • Jansen J.P.
      • Trikalinos T.
      • Cappelleri J.C.
      • et al.
      Indirect treatment comparison/network meta-analysis study questionnaire to assess relevance and credibility to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force Report.
      ]. In addition, six of the national guidelines require that an assessment of the homogeneity of direct comparisons be performed, effectively assessing the potential for treatment effect modifiers to bias results [

      National Institute for Health and Care Excellence. Guide to the methods of technology appraisal. 2013. Available from: http://publications.nice.org.uk/guide-to-the-methods-of-technology-appraisal-2013-pmg9. [Accessed June 7, 2013].

      ,

      Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

      ,

      Scottish Medicines Consortium. Guidance to manufacturers for completion of New Product Assessment Form (NPAF). 2013 . Available from: http://www.scottishmedicines.org.uk/files/submissionprocess/Guidance_on_NPAF_V4_0_-_August_2013.doc. [Accessed August 27, 2013].

      ,

      South Africa Department of Health. The guidelines for pharmacoeconomic evaluations of medicines and scheduled substances. 2013. Available from: http://www.info.gov.za/view/DownloadFileAction?id=183162. [Accessed June 7, 2013].

      ,

      National Institute for Health and Care Excellence. Guides to the methods of technology appraisal. 2008. Available from: http://www.nice.org.uk/aboutnice/howwework/devnicetech/GuideToMethods;TechnologyAppraisal2008.jsp?domedia=1&mid=B52851A3-19B9-E0B5-D48284D172BD8459. [Accessed June 7, 2013].

      ] (A. Ortega, M. Fraga, E. Alegre, et al., personal communication, 2013); however, only England and Wales require that such modifiers be identified in advance [

      National Institute for Health and Care Excellence. Guide to the methods of technology appraisal. 2013. Available from: http://publications.nice.org.uk/guide-to-the-methods-of-technology-appraisal-2013-pmg9. [Accessed June 7, 2013].

      ].
      The strengths of our approach include the process of double extraction by two reviewers with final checking by a third reviewer, the multistage iterative process of extraction, and the validation steps leading to consensus building.
      A critical limitation is the lack of a searchable database for national guidelines. This review has gathered documents from multiple sources including the increasingly comprehensive ISPOR repository, but the difficulty in searching for and accessing guideline documents means that it cannot be certain that all relevant documents are represented herein.
      The limitations of the original documents were a lack of a shared vocabulary of technical terms that occasionally resulted in overlap of requirement categories. Despite locating a guideline document from the French College Des Economistes de la Santé and a French-based method review [

      Cucherat M, Izard V. Indirect comparisons: methods and validity. 2010. Available from: http://www.has-sante.fr/portail/upload/docs/application/pdf/2011-02/summary_report__indirect_comparisons_methods_and_validity_january_2011_2.pdf. [Accessed June 7, 2013].

      ,

      Collège des Èconomistes de la Santè. French guidelines for the economic evaluation of health care technologies. 2004. Available from: http://www.ces-asso.org/docs/France_Guidelines_HE_Evaluation.PDF. [Accessed June 7, 2013].

      ], neither document contained specific information on study search and selection or bias assessment, nor did they indicate that there were no requirements, making it difficult to guarantee compliance with French submission requirements. Second, the fact that the German guidelines include the now-defunct clinicalstudyresults.org database highlights the difficulties inherent in maintaining living guidelines to a rapidly changing field in the Internet age [

      Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

      ]. Potential weaknesses in our analysis come from the availability of national guidelines. Only nine stipulate the conduct of NMA in sufficient detail to make comparison viable, and all these are from either European or Commonwealth countries.
      Further work should focus on the requirements from substantial existing markets not covered, such as Japan and South Korea, and emerging markets such as Brazil and India.
      For the first time, guidelines for the use of indirect evidence from multiple national jurisdictions have been reviewed and the requirements compiled. The aggregate requirements do not include requirements that are mutually prohibitive. Subsequently, it is now possible to perform one NMA for submission to multiple national jurisdictions.
      Source of financial support: The authors have no other financial relationships to disclose.

      References

        • Owens D.K.
        • Lohr K.N.
        • Atkins D.
        • et al.
        AHRQ series paper 5: grading the strength of a body of evidence when comparing medical interventions--Agency for Healthcare Research and Quality and the effective health-care program.
        J Clin Epidemiol. 2010; 63: 513-523
        • Ioannidis J.
        Indirect comparisons: the MESH and MESS of clinical trials.
        Lancet. 2006; 368: 1470-1472
        • Caldwell D.M.
        • Ades A.E.
        • Higgins J.P.
        Simultaneous comparison of multiple treatments: combining direct and indirect evidence.
        BMJ. 2005; 331: 897-900
        • Bucher H.C.
        • Guyatt G.H.
        • Griffith L.E.
        • Walter S.D.
        The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials.
        J Clin Epidemiol. 1997; 50: 683-691
        • Song F.
        • Altman D.G.
        • Glenny A.M.
        • Deeks J.J.
        Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses.
        BMJ. 2003; 326: 472
        • Lu G.
        • Ades A.E.
        Combination of direct and indirect evidence in mixed treatment comparisons.
        Stat Med. 2004; 23: 3105-3124
        • Song F.
        • Loke Y.K.
        • Walsh T.
        • et al.
        Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: survey of published systematic reviews.
        BMJ. 2009; 338: b1147
      1. International Society for Pharmacoeconomics and Outcomes Research. Pharmacoenconomic guidelines around the world. 2012. Available from: http://www.ispor.org/peguidelines/index.asp. [Accessed September 18, 2012].

        • Hoaglin D.C.
        • Hawkins N.
        • Jansen J.P.
        • et al.
        Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2.
        Value Health. 2011; 14: 429-437
      2. An tÙdaràs Um Fhaisnèis agus Càiliocht Slàinte. Guidelines for the economic evaluation of health technologies in Ireland. 2010. Available from: http://www.hiqa.ie/publication/guidelines-economic-evaluation-health-technologies-ireland. [Accessed November 7, 2013].

      3. AMCP Format Executive Committee and Working Group. The AMCP format for formulary submissions version 3.1. 2012. Available from: http://amcp.org/practice-resources/amcp-format-formulary-submisions.pdf. [Accessed November 7, 2013].

      4. Tandvårds- och läkemedelsförmånsverket. Handbok till Tandvårds- och läkemedelsförmånsverkets föreskrifter (TLVFS 2011:3) om ansökan om pris och subvention för förbrukningsartiklar. 2011. Available from: http://www.tlv.se/Upload/Foretag/Handbok-TLVFS2011-3-subvention-forbrukningsartiklar-sept-2011.pdf. [Accessed June 7, 2013].

      5. Centre for Reviews and Dissemination. Systematic Reviews—CRD’s Guidance for Undertaking Reviews in Health Care. York: University of York, 2009.

      6. Contandriopolous A, Coyle D, Hailey D, et al. Guidelines for the economic evaluation of health technologies: Canada. 2006. Available from: http://www.cadth.ca/media/pdf/186_EconomicGuidelines_e.pdf. [Accessed June 7, 2013].

      7. National Institute for Health and Care Excellence. Guide to the methods of technology appraisal. 2013. Available from: http://publications.nice.org.uk/guide-to-the-methods-of-technology-appraisal-2013-pmg9. [Accessed June 7, 2013].

      8. Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

      9. Pharmaceutical Benefits Advisory Committee. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee (version 4.3). 2008. Available from: http://www.health.gov.au/internet/main/publishing.nsf/Content/AECB791C29482920CA25724400188EDB/$File/PBAC4.3.2.pdf. [Accessed June 7, 2013].

      10. Cucherat M, Izard V. Indirect comparisons: methods and validity. 2010. Available from: http://www.has-sante.fr/portail/upload/docs/application/pdf/2011-02/summary_report__indirect_comparisons_methods_and_validity_january_2011_2.pdf. [Accessed June 7, 2013].

      11. Collège des Èconomistes de la Santè. French guidelines for the economic evaluation of health care technologies. 2004. Available from: http://www.ces-asso.org/docs/France_Guidelines_HE_Evaluation.PDF. [Accessed June 7, 2013].

      12. Scottish Medicines Consortium. Guidance to manufacturers for completion of New Product Assessment Form (NPAF). 2013 . Available from: http://www.scottishmedicines.org.uk/files/submissionprocess/Guidance_on_NPAF_V4_0_-_August_2013.doc. [Accessed August 27, 2013].

        • Moher D.
        • Liberati A.
        • Tetzlaff J.
        • Altman D.G.
        Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
        Ann Intern Med. 2009; 151: W64
        • Schulz K.F.
        • Altman D.G.
        • Moher D.
        CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials.
        BMJ. 2010; 340: c332
      13. South Africa Department of Health. The guidelines for pharmacoeconomic evaluations of medicines and scheduled substances. 2013. Available from: http://www.info.gov.za/view/DownloadFileAction?id=183162. [Accessed June 7, 2013].

      14. Cleemput I, van Wilder P, Vrijens F, et al. Guidelines for pharmacoeconomic evaluations in Belgium: KCE reports 78C. 2008. Available from: https://kce.fgov.be/publication/report/guidelines-for-pharmacoeconomic-evaluations-in-belgium. [Accessed June 7, 2013].

        • Jansen J.P.
        • Trikalinos T.
        • Cappelleri J.C.
        • et al.
        Indirect treatment comparison/network meta-analysis study questionnaire to assess relevance and credibility to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force Report.
        Value Health. 2014; 17: 157-173
        • Jansen J.P.
        • Fleurence R.
        • Devine B.
        • et al.
        Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 1.
        Value Health. 2011; 14: 417-428
        • Jansen J.P.
        • Naci H.
        Is network meta-analysis as valid as standard pairwise meta-analysis? It all depends on the distribution of effect modifiers.
        BMC Med. 2013; 11: 159
      15. Wells G, Sultan S, Chen L, et al. Indirect evidence: indirect treatment comparisons in meta-analysis. 2009. Available from: http://www.cadth.ca/media/pdf/H0462_itc_tr_e.pdf. [Accessed June 7, 2013].

      16. National Institute for Health and Care Excellence. Guides to the methods of technology appraisal. 2008. Available from: http://www.nice.org.uk/aboutnice/howwework/devnicetech/GuideToMethods;TechnologyAppraisal2008.jsp?domedia=1&mid=B52851A3-19B9-E0B5-D48284D172BD8459. [Accessed June 7, 2013].

      17. ICWG. Report of the Indirect Comparisons Working Group (ICWG) to the Pharmaceutical Benefits Advisory Committee (PBAC): Assessing Indirect Comparisons. ICWG. 2013. Available from: http://www.pbs.gov.au/industry/useful-resources/PBAC_feedback_files/ICWG%20Report%20FINAL2.pdf. [Accessed October 29, 2013]

        • Jansen J.P.
        • Trikalinos T.
        • Cappelleri J.C.
        • et al.
        Indirect treatment comparison/network meta-analysis study questionnaire to assess relevance and credibility to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report.
        Value Health. 2014; 17: 157-173
      18. An tÙdaràs Um Fhaisnèis agus Càiliocht Slàinte. Guidelines for the Economic Evaluation of Health Technologies in Ireland. HIQA 2010. Available from: http://www.hiqa.ie/publication/guidelines-economic-evaluation-health-technologies-ireland. [Accessed November 7, 2013]

      19. Guidelines on how to conduct pharmacoeconomic analyses. Statens legemiddelverket. 2012. Available from: http://www.legemiddelverket.no/English/price_and_reimbursement/application_for_reimursement/Lists/PageAttachments/default/NO/Pharmacoeconomic%20guidelines%20-%20Norway.pdf. [Accessed November 7, 2013]

      20. Agencja Oceny Technologii Medyczncyh. Guidelines for Conducting Health Technology Assessment (HTA). AOTM. 2009. Available from: http://www.aotm.gov.pl/assets/files/wytyczne_hta/2009/Guidelines_HTA_eng_MS_29062009.pdf. [Accessed November 7, 2013]