A Comparison of National Guidelines for Network Meta-Analysis

National Institute for Health and Care Excellence. Guide to the methods of technology appraisal. 2013. Available from: http://publications.nice.org.uk/guide-to-the-methods-of-technology-appraisal-2013-pmg9. [Accessed June 7, 2013].

Google Scholar

]. Germany requires that keywords, MeSH identifiers, and other terms used to search electronic databases be grouped into related blocks in the presentation of the search strategy [

Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

Google Scholar

].

Selection of Databases

Most of the national pharmacoeconomic guidelines specify which databases should be searched. Table 3 lists the various databases listed in the national guidelines. MEDLINE, EMBASE, and the Cochrane (CENTRAL) databases form a core specified by almost all the national guidelines. Outside of this core there is variation, with some jurisdictions requiring that the search be conducted in databases with a local focus and others requiring more emphasis on clinical trial databases. Four of the nine national guidelines require that the search be conducted in an international registry of clinical trials, either clinicaltrials.gov or the International Clinical Trials Registry Platform, but typically both [

14

Contandriopolous A, Coyle D, Hailey D, et al. Guidelines for the economic evaluation of health technologies: Canada. 2006. Available from: http://www.cadth.ca/media/pdf/186_EconomicGuidelines_e.pdf. [Accessed June 7, 2013].

Google Scholar

,

National Institute for Health and Care Excellence. Guide to the methods of technology appraisal. 2013. Available from: http://publications.nice.org.uk/guide-to-the-methods-of-technology-appraisal-2013-pmg9. [Accessed June 7, 2013].

Google Scholar

,

Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

Google Scholar

,

Pharmaceutical Benefits Advisory Committee. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee (version 4.3). 2008. Available from: http://www.health.gov.au/internet/main/publishing.nsf/Content/AECB791C29482920CA25724400188EDB/$File/PBAC4.3.2.pdf. [Accessed June 7, 2013].

Google Scholar

]. The German guideline references the industry-maintained clinicalstudyresults.org database, which has been closed since the publication of the German guidelines [

Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

Google Scholar

]. The national guideline document issued by the Australian Pharmaceutical Benefits Advisory Committee requires that the Australia New Zealand Clinical Trials Registry form part of the search strategy [

[17]

Pharmaceutical Benefits Advisory Committee. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee (version 4.3). 2008. Available from: http://www.health.gov.au/internet/main/publishing.nsf/Content/AECB791C29482920CA25724400188EDB/$File/PBAC4.3.2.pdf. [Accessed June 7, 2013].

Google Scholar

]. The Australia New Zealand Clinical Trials Registry forms part of the International Clinical Trials Registry Platform search portal that is required by other national guidelines; however, the Pharmaceutical Benefits Advisory Committee specifically differentiates between the two. German and Australian guidelines require that company-specific databases be searched and results presented, although the national guidelines contain no indication how the transparency and repeatability of such a search would be enforced [

Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

Google Scholar

,

Pharmaceutical Benefits Advisory Committee. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee (version 4.3). 2008. Available from: http://www.health.gov.au/internet/main/publishing.nsf/Content/AECB791C29482920CA25724400188EDB/$File/PBAC4.3.2.pdf. [Accessed June 7, 2013].

Google Scholar

].

Table 3Specified databases.

Country	MEDLINE	Embase	Cochrane (CENTRAL)	Cochrane Database of Systematic Reviews	Clinicaltrials.gov	Clinicalstudyresults.org ^* Database decommissioned Q4 2011.	ICTRP search portal	NHS CRD	ANZCTR	Subject-specific databases or registers ^† For example, PsycINFO (psychology and psychiatry), MANTIS (osteopathy and chiropractic), CINAHL (nursing and allied health).	Manufacturers’ internal databases
Australia	•	•	•		•				•		•
Belgium	•	•	•	•				•
Canada	•	•	•		•		•			•
England and Wales	•	•	•		•		•			•
France
Germany	•	•	•		•	•	•			•	•
Scotland
South Africa	•	•	•								•
Spain
ISPOR Task Force Best Practice Guide	•	•	•		•		•			•
ISPOR NMA Assessment Questionnaire	•	•	•		•

ANZCTR, Australia New Zealand Clinical Trials Registry (formerly the Australia Clinical Trials Registry); CENTRAL, Cochrane Central Register of Controlled Trials; CRD, Centre for Reviews and Dissemination; EED, Economic Evaluation Database; HEED, Health Economic Evaluation Database; ICTRP, International Clinical Trials Registry Platform; ISPOR, International Society for Pharmacoeconomics and Outcomes Research; NHS, National Health Service; NMA; network meta-analysis.

Database decommissioned Q4 2011.

† For example, PsycINFO (psychology and psychiatry), MANTIS (osteopathy and chiropractic), CINAHL (nursing and allied health).

Open table in a new tab

The French, Scottish, and Spanish guideline documents do not contain recommendations or requirements regarding the search strategy to be implemented or databases searched [

18

Cucherat M, Izard V. Indirect comparisons: methods and validity. 2010. Available from: http://www.has-sante.fr/portail/upload/docs/application/pdf/2011-02/summary_report__indirect_comparisons_methods_and_validity_january_2011_2.pdf. [Accessed June 7, 2013].

Google Scholar

,

Collège des Èconomistes de la Santè. French guidelines for the economic evaluation of health care technologies. 2004. Available from: http://www.ces-asso.org/docs/France_Guidelines_HE_Evaluation.PDF. [Accessed June 7, 2013].

Google Scholar

,

20

Scottish Medicines Consortium. Guidance to manufacturers for completion of New Product Assessment Form (NPAF). 2013 . Available from: http://www.scottishmedicines.org.uk/files/submissionprocess/Guidance_on_NPAF_V4_0_-_August_2013.doc. [Accessed August 27, 2013].

Google Scholar

] (A. Ortega, M. Fraga, E. Alegre, et al., personal communication, 2013). The French methods review document does provide details of the search strategy used in the review document itself, but not for identifying trials as part of an NMA.

Study Selection

Following the completion of the search, it is necessary to determine which studies should be included in the NMA. The requirements for the study selection process are listed in Table 4. In many cases, they are less rigorous than the methods recommended by either the Centre for Reviews and Dissemination (CRD) and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses [

13

Centre for Reviews and Dissemination. Systematic Reviews—CRD’s Guidance for Undertaking Reviews in Health Care. York: University of York, 2009.

Google Scholar

,

21

Moher D.
Liberati A.
Tetzlaff J.
Altman D.G.

Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.

Google Scholar

], with less than half of all national guidelines stipulating explicitly that two reviewers should carry out the selection process.

Table 4Study selection.

Country	Selection process					Trials include
Country	Predefined inclusion/exclusion criteria process	Justified selection criteria	Selected by two independent reviewers	Maintained log of trials deemed ineligible	Trials form one connected network of interventions	Randomized allocation * Criteria for randomized controlled trials (RCTs).	Separate arms * Criteria for randomized controlled trials (RCTs).	Demographic homogeneity between trials	Similar prognostic severity
Australia	•	•				•	•	•
Belgium	•	•
Canada	•		•						•
England and Wales	•		•	•					•
France	•
Germany	•		•
Scotland	•	•		•				•	•
South Africa				•		•
Spain	•	•		•				•	•
ISPOR Task Force Best Practice Guide	•		•						•
ISPOR NMA Assessment Questionnaire	•				•	•	•	•

ISPOR, International Society for Pharmacoeconomics and Outcomes Research; NMA, network meta-analysis.

Criteria for randomized controlled trials (RCTs).

Open table in a new tab

Almost all national guidelines specify that inclusion and exclusion criteria must be defined beforehand. This is crucial not only to inform the initial search but also to ensure that selection bias is reduced and all appropriately designed studies are included regardless of direction or magnitude. Aside from this consistently applied criterion, there are few proscriptions to the study selection process; and apart from the Australian guidelines, there are few requirements as to the design of the studies themselves. The guidelines from the Pharmaceutical Benefits Advisory Committee require clinical trials included in the meta-analysis to have randomized allocation to separate arms, which are themselves criteria for an RCT [

[17]

Pharmaceutical Benefits Advisory Committee. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee (version 4.3). 2008. Available from: http://www.health.gov.au/internet/main/publishing.nsf/Content/AECB791C29482920CA25724400188EDB/$File/PBAC4.3.2.pdf. [Accessed June 7, 2013].

Google Scholar

].

Bias Assessment

Estimates of treatment effect obtained from NMAs may be biased by the effects of heterogeneity between studies. Table 5 presents the recommended steps taken to assess the potential for bias within NMAs. Almost all national guidelines require an assessment of the adequacy of blinding and that the results of the analysis of the intent-to-treat population are used in the analysis. Also, an assessment of variance between trial protocol and standard practice, and comparison of rates of dropout between the arms of the study, is widely required.

Table 5Bias assessment.

Source of bias	Identification and selection of studies		Conduct and reporting of clinical trials								Network meta-analysis
National guideline	Assess level of bias within included studies	Homogeneity of prognostic severity	Describe the design and methodology according to CONSORT guidelines	Assessment of variance between trial protocol and standard practice	Assess adequacy of blinding	Comparison of dropout rates		Implementation of ITT	Assessment of difference in baseline risk and placebo response	Describe time horizon	Investigator conflicts of interest reported	Report of subgroup analysis	Treatment effect modifiers identified before comparing study results	Assessment of publication bias and/or funnel plot
Australia					•	•	•	•
Belgium		•		•	•	•	•				•
Canada		•		•	•	•	•							•
England and Wales		•		•	•	•	•					•	•
France
Germany	•	•	•		•		•				•
Scotland		•		•							•
South Africa					•	•	•	•	•
Spain		•					•				•
ISPOR Task Force Best Practice Guide				•	•		•	•			•		•
ISPOR NMA Assessment Questionnaire	•				•	•				•		•	•

CONSORT, Consolidated Standards of Reporting Trials; ISPOR, International Society for Pharmacoeconomics and Outcomes Research; ITT, intent to treat; NMA, network meta-analysis.

Open table in a new tab

The two French guideline documents do not contain requirements for this aspect of NMA beyond a recommendation that publication bias can be reduced by accessing a clinical trials registry and including all trials carried out [

18

Cucherat M, Izard V. Indirect comparisons: methods and validity. 2010. Available from: http://www.has-sante.fr/portail/upload/docs/application/pdf/2011-02/summary_report__indirect_comparisons_methods_and_validity_january_2011_2.pdf. [Accessed June 7, 2013].

Google Scholar

,

Collège des Èconomistes de la Santè. French guidelines for the economic evaluation of health care technologies. 2004. Available from: http://www.ces-asso.org/docs/France_Guidelines_HE_Evaluation.PDF. [Accessed June 7, 2013].

Google Scholar

]. German and South African guidelines include unique requirements such as describing included studies according to Consolidated Standards of Reporting Trials guidelines [

[22]

Schulz K.F.
Altman D.G.
Moher D.

CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials.

], or noting the time horizon of included studies, respectively [

Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

Google Scholar

,

23

South Africa Department of Health. The guidelines for pharmacoeconomic evaluations of medicines and scheduled substances. 2013. Available from: http://www.info.gov.za/view/DownloadFileAction?id=183162. [Accessed June 7, 2013].

Google Scholar

]. Australia and Scotland both require demographic homogeneity between trials [

Pharmaceutical Benefits Advisory Committee. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee (version 4.3). 2008. Available from: http://www.health.gov.au/internet/main/publishing.nsf/Content/AECB791C29482920CA25724400188EDB/$File/PBAC4.3.2.pdf. [Accessed June 7, 2013].

Google Scholar

,

20

Scottish Medicines Consortium. Guidance to manufacturers for completion of New Product Assessment Form (NPAF). 2013 . Available from: http://www.scottishmedicines.org.uk/files/submissionprocess/Guidance_on_NPAF_V4_0_-_August_2013.doc. [Accessed August 27, 2013].

Google Scholar

]. In addition, German regulators require that the level of bias in studies be assessed, with the stipulation that studies assessed as being biased must not be excluded [

Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

Google Scholar

]. The ISPOR NMA assessment questionnaire and the National Institute for Health and Care Excellence Guide to Technology Appraisal are the only documents that recommend that treatment effect modifiers be identified before comparing the results of individual studies [

2

Ioannidis J.

Indirect comparisons: the MESH and MESS of clinical trials.

,

National Institute for Health and Care Excellence. Guide to the methods of technology appraisal. 2013. Available from: http://publications.nice.org.uk/guide-to-the-methods-of-technology-appraisal-2013-pmg9. [Accessed June 7, 2013].

Google Scholar

].

Conduct of NMA

Table 6a, Table 6b is a breakdown of the statistical techniques, processes, and recommended scales for data required by the national guidelines. It is important to note the relative consistency in the scales preferred. All national guidelines call for a core set of scales for binary or time-to-event data comprising relative risk, relative risk difference, hazard ratio, odds ratio, or a combination thereof. The Spanish guidelines do not contain any recommendations as to the type of scales to be used for data. France, Scotland, and Spain recommend meta-regression to be performed. Where continuous data are reported, the national guidelines are split four to three on whether to use median difference or weighted mean difference, respectively, with South Africa and Spain having no specific requirements. Even though naive indirect comparisons, where the results of individual arms of different trials are compared as though they were from the same trial, produce evidence equivalent only to observational studies, over half the countries considered it necessary to explicitly prohibit their conduct [

National Institute for Health and Care Excellence. Guide to the methods of technology appraisal. 2013. Available from: http://publications.nice.org.uk/guide-to-the-methods-of-technology-appraisal-2013-pmg9. [Accessed June 7, 2013].

Google Scholar

,

Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

Google Scholar

,

Collège des Èconomistes de la Santè. French guidelines for the economic evaluation of health care technologies. 2004. Available from: http://www.ces-asso.org/docs/France_Guidelines_HE_Evaluation.PDF. [Accessed June 7, 2013].

Google Scholar

,

25

Jansen J.P.
Trikalinos T.
Cappelleri J.C.
et al.

Indirect treatment comparison/network meta-analysis study questionnaire to assess relevance and credibility to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force Report.

] (A. Ortega, M. Fraga, E. Alegre, et al., personal communication, 2013).

Table 6aConduct of indirect comparisons.

Country	Conduct of indirect comparison
	$Description of patient & treatment characteristics$	$Describe & justify statistical method; Bayesian or Frequentist$	$I Bayesian, describe previous distribution,$ $sensitivity to previous, and assess convergence$	$Present individual study results and / or forest plot$	$Graphically summarize rank probabilities$	$Description of relative-effect estimate$	$Include rationale for, and description of, sensitivity analyses$	$Description of different findings with sensitivity / scenario analysis$	$Meta-regression$	$Poisson distribution$	$Naive comparisons of point estimates or active arms are prohibited$	$Include analysis of the hypothesis of consistency$	$Assess the results for each common reference across trials$ $for any important differences$	$Perform consistency check between direct and indirect evidence$	$Assess homogeneity of direct comparisons$	$Assess heterogeneity with Cochrane Q, I^{2}$	$Random effects$	$Include code and specify software package$	$Include diagram of network structure$
Australia				•						•			•			•	•
Belgium							•				•
Canada	•	•	•			•	•	•						•
England and Wales	•			•		•	•	•			•		•	•	•		•
France		•	•						•		•	•		•	•		•
Germany	•	•	•	•			•	•			•			•	•	•	•	•	•
Scotland	•			•			•	•	•					•	•	•	•		•
South Africa	•	•		•			•						•		•		•
Spain			•			•	•	•	•		•			•	•	•
ISPOR Task Force Best Practice Guide	•	•	•	•		•	•	•						•			•
ISPOR NMA Assessment Questionnaire	•	•	•	•	•	•			•		•	•		•	•		•		•

Open table in a new tab

Table 6bPresentation of results.

Country	Presentation of results
	$Median difference (continuous)$	$Weighted mean difference (continuous)$	$Relative risk$	$Hazard ratio$	$Odds ratio$	$Relative risk difference$	$Absolute risk reductions$
Australia		•	•	•	•	•
Belgium	•			•	•
Canada	•		•	•	•	•
England and Wales	•		• ^⁎ Mentioned explicitly in the 2008 NICE guide to the methods of technology appraisal [31].	• ^⁎ Mentioned explicitly in the 2008 NICE guide to the methods of technology appraisal [31].	• ^⁎ Mentioned explicitly in the 2008 NICE guide to the methods of technology appraisal [31].	• ^⁎ Mentioned explicitly in the 2008 NICE guide to the methods of technology appraisal [31].
France		•	•	•	•	•
Germany	•				•
Scotland		•	•
South Africa			•	•	•	•	•
Spain
ISPOR Task Force Best Practice Guide			•	•	•	•
ISPOR NMA Assessment Questionnaire

HR, hazard ratio; ISPOR, International Society for Pharmacoeconomics and Outcomes Research; NICE, National Institute for Health and Care Excellence; NMA, network meta-analysis; OR, odds ratio; RR, relative risk.

Mentioned explicitly in the 2008 NICE guide to the methods of technology appraisal [31].

Open table in a new tab

None of the national guidelines stipulated what form sensitivity analyses should take, but seven of the nine national jurisdictions require descriptions and rationale for sensitivity analyses, with five countries subsequently requiring a description of the different findings and three needing a relative effect estimate. France, Germany, Scotland, Spain, and Australia require an assessment of the heterogeneity of direct comparisons, with Germany, Scotland, Spain, and Australia also requiring that a Cochrane Q test be performed [

Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

Google Scholar

,

Pharmaceutical Benefits Advisory Committee. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee (version 4.3). 2008. Available from: http://www.health.gov.au/internet/main/publishing.nsf/Content/AECB791C29482920CA25724400188EDB/$File/PBAC4.3.2.pdf. [Accessed June 7, 2013].

Google Scholar

,

Collège des Èconomistes de la Santè. French guidelines for the economic evaluation of health care technologies. 2004. Available from: http://www.ces-asso.org/docs/France_Guidelines_HE_Evaluation.PDF. [Accessed June 7, 2013].

Google Scholar

]. Most of the countries call for a consistency check between direct and indirect evidence, strongly suggesting that indirect evidence should be considered as an adjunct to direct evidence. None of the national guidelines indicated that differences between direct and indirect evidence would render indirect evidence inadmissible.

Very few aspects of the conduct of the analysis were required only by a single country. Perhaps surprisingly, only the German and Scottish regulators require a network diagram showing the links between the studies. Germany is also the only country to explicitly request the inclusion of any code and the software package used [

Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

Google Scholar

].

Discussion

For the first time, multiple national guideline documents have been compared with each other and the differing stipulations on the conduct of indirect comparisons extracted. The technique of indirect NMA is recent in the field of health technology assessment. This combination of relative novelty and the precedent of individual national regulatory bodies having jurisdiction over their respective markets means that there are currently no transnational guidelines for the compilation of indirect evidence. ISPOR has published multiple best-practice documents to inform practitioners and regulators [

9

Hoaglin D.C.
Hawkins N.
Jansen J.P.
et al.

Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2.

,

26

Jansen J.P.
Fleurence R.
Devine B.
et al.

Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 1.

,

27

Jansen J.P.
Naci H.

Is network meta-analysis as valid as standard pairwise meta-analysis? It all depends on the distribution of effect modifiers.

]. Although these documents were comprehensive when their collective recommendations were combined, they did not capture every nuance of the process of NMA required by every national regulatory body. By using the aggregated recommendations from the tables herein, it becomes possible to create a super set of recommendations that will satisfy the regulatory bodies considered.

There is an overall focus on the transparency and repeatability of the search process. Certain jurisdictions such as South Africa will conduct a search of their own to mirror that of the submission and will suspend the application if any relevant studies are found to have been omitted [

[23]

South Africa Department of Health. The guidelines for pharmacoeconomic evaluations of medicines and scheduled substances. 2013. Available from: http://www.info.gov.za/view/DownloadFileAction?id=183162. [Accessed June 7, 2013].

Google Scholar

]. All national guidelines reviewed that consider the systematic review component of NMA require that a search be conducted in MEDLINE, EMBASE, and the Cochrane (CENTRAL) databases, with certain regulatory bodies requiring that additional databases searched be nationally or regionally specific.

The task of complying with a “one-size-fits-all” super set of requirements is made more onerous by having to perform tasks required by only a single country, although only a handful of these instances were noted. For example, only Germany requires that the design and methodology of the studies be described according to Consolidated Standards of Reporting Trials guidelines; however, this becomes a requirement for the super set. Although the Australian guidelines appear to place proscriptions on trial selection, the requirement that trials included in the meta-analysis have randomized allocation to separate arms is reflective of standard RCT practice. The requirement for demographic homogeneity between trial populations is reflected in the national guidelines of Belgium, Canada, England and Wales, Germany, Scotland, and Spain, which all require that there be homogeneity of prognostic severity between trials as part of their bias assessment.

It is notable that there is an overall lack of explicit requirements regarding an assessment of the distribution of treatment effect modifiers across studies included in the NMA. An imbalance in the effect modifiers between the direct comparisons suggests that the transitivity assumption may have been violated. Transitivity in this context means that if treatment A is ranked above treatment B in terms of efficacy, and B is ranked above treatment C, then A must be ranked above C [

[28]

Wells G, Sultan S, Chen L, et al. Indirect evidence: indirect treatment comparisons in meta-analysis. 2009. Available from: http://www.cadth.ca/media/pdf/H0462_itc_tr_e.pdf. [Accessed June 7, 2013].

Google Scholar

]. Despite the lack of explicit requirements, some guidelines do address the problem implicitly; the National Institute for Health and Care Excellence guidelines require that potential treatment effect modifiers be identified before analysis, and the Belgian guidelines state that treatment effect may be influenced by factors that vary across trials as an example of unacceptable confounding bias [

[25]

Jansen J.P.
Trikalinos T.
Cappelleri J.C.
et al.

Indirect treatment comparison/network meta-analysis study questionnaire to assess relevance and credibility to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force Report.

]. In addition, six of the national guidelines require that an assessment of the homogeneity of direct comparisons be performed, effectively assessing the potential for treatment effect modifiers to bias results [

National Institute for Health and Care Excellence. Guide to the methods of technology appraisal. 2013. Available from: http://publications.nice.org.uk/guide-to-the-methods-of-technology-appraisal-2013-pmg9. [Accessed June 7, 2013].

Google Scholar

,

Gemeinsamer Bundesausschuss. Dossier for the Benefit Assesment pursuant to Section 35a of the German Social Code Book V - Module 4. 2011. Available from: http://www.g-ba.de/downloads/17-98-2998/II-6_Dossiervorlage_Modul4.pdf. [Accessed June 24, 2013].

Google Scholar

,

20

Scottish Medicines Consortium. Guidance to manufacturers for completion of New Product Assessment Form (NPAF). 2013 . Available from: http://www.scottishmedicines.org.uk/files/submissionprocess/Guidance_on_NPAF_V4_0_-_August_2013.doc. [Accessed August 27, 2013].

Google Scholar

,

23

South Africa Department of Health. The guidelines for pharmacoeconomic evaluations of medicines and scheduled substances. 2013. Available from: http://www.info.gov.za/view/DownloadFileAction?id=183162. [Accessed June 7, 2013].

Google Scholar

,

29

National Institute for Health and Care Excellence. Guides to the methods of technology appraisal. 2008. Available from: http://www.nice.org.uk/aboutnice/howwework/devnicetech/GuideToMethods;TechnologyAppraisal2008.jsp?domedia=1&mid=B52851A3-19B9-E0B5-D48284D172BD8459. [Accessed June 7, 2013].

Google Scholar

] (A. Ortega, M. Fraga, E. Alegre, et al., personal communication, 2013); however, only England and Wales require that such modifiers be identified in advance [

[15]

National Institute for Health and Care Excellence. Guide to the methods of technology appraisal. 2013. Available from: http://publications.nice.org.uk/guide-to-the-methods-of-technology-appraisal-2013-pmg9. [Accessed June 7, 2013].

Google Scholar

].

The strengths of our approach include the process of double extraction by two reviewers with final checking by a third reviewer, the multistage iterative process of extraction, and the validation steps leading to consensus building.

A critical limitation is the lack of a searchable database for national guidelines. This review has gathered documents from multiple sources including the increasingly comprehensive ISPOR repository, but the difficulty in searching for and accessing guideline documents means that it cannot be certain that all relevant documents are represented herein.

The limitations of the original documents were a lack of a shared vocabulary of technical terms that occasionally resulted in overlap of requirement categories. Despite locating a guideline document from the French College Des Economistes de la Santé and a French-based method review [

18

Cucherat M, Izard V. Indirect comparisons: methods and validity. 2010. Available from: http://www.has-sante.fr/portail/upload/docs/application/pdf/2011-02/summary_report__indirect_comparisons_methods_and_validity_january_2011_2.pdf. [Accessed June 7, 2013].

Google Scholar

,

Collège des Èconomistes de la Santè. French guidelines for the economic evaluation of health care technologies. 2004. Available from: http://www.ces-asso.org/docs/France_Guidelines_HE_Evaluation.PDF. [Accessed June 7, 2013].

Google Scholar

], neither document contained specific information on study search and selection or bias assessment, nor did they indicate that there were no requirements, making it difficult to guarantee compliance with French submission requirements. Second, the fact that the German guidelines include the now-defunct clinicalstudyresults.org database highlights the difficulties inherent in maintaining living guidelines to a rapidly changing field in the Internet age [