Advertisement

Critical Appraisal of Systematic Reviews With Costs and Cost-Effectiveness Outcomes: An ISPOR Good Practices Task Force Report

      Introduction

      A systematic review (SR) can provide rigorous and complete evidence to support decision makers who consider both the effectiveness and cost-effectiveness of health interventions. A dramatic increase in published health economic (HE) studies, more specifically cost and cost-effectiveness studies, has resulted in the consequent proliferation of systematic reviews with cost and cost-effectiveness outcomes (SR-CCEO).
      • Luhnen M.
      • Prediger B.
      • Neugebauer E.A.M.
      • Mathes T.
      Systematic reviews of health economic evaluations: a structured analysis of characteristics and methods applied.
      ,
      • Mandrik O.
      • Ekwunife O.I.
      • Meheus F.
      • et al.
      Systematic reviews as a "lens of evidence": Determinants of cost-effectiveness of breast cancer screening.
      First, such reviews help to indentify strenghts and weaknesses in HE studies, modelling methodologies, and data for modelling inputs. Second, SR-CCEOs may be informative for decisionmakers in resource allocation decisions for health interventions, especially in countries with limited capacity for health technology assessment (HTA). For the purpose of this article, cost studies are defined as studies analyzing the costs of healthcare interventions, including cost descriptions and cost-of-illness (economic burden of disease) studies. By cost-effectivenessstudies we mean full economic evaluations, including cost-minimization, cost-effectiveness analysis, cost-utility analysis, cost-benefit analysis, and cost-consequence analysis. Sometimes cost studies might be based on an explicit comparison of alternatives.
      However, it is challenging to appropriately interpret SR-CCEOs owing to their heterogeneity in applied methods and reporting, and furthermore, owing to variability in clinical and health settings in the original studies they include. Methodologic guidance and checklists that improve the quality of SRs on clinical evidence or decrease risk of bias in their interpretation or synthesis
      • Shea B.J.
      • Reeves B.C.
      • Wells G.
      • et al.
      AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both.
      • Oxman A.D.
      • Guyatt G.H.
      Validation of an index of the quality of review articles.
      • Whiting P.
      • Savovic J.
      • Higgins J.P.
      • et al.
      ROBIS: a new tool to assess risk of bias in systematic reviews was developed.
      • Moher D.
      • Liberati A.
      • Tetzlaff J.
      • Altman D.G.
      Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
      have limited applicability for SR-CCEOs. There is little specific methodologic guidance for SR-CCEOs.
      • Thielen F.W.
      • Van Mastrigt G.
      • Burgers L.T.
      • et al.
      How to prepare a systematic review of economic evaluations for clinical practice guidelines: database selection and search strategy development (part 2/3).
      • van Mastrigt G.A.
      • Hiligsmann M.
      • Arts J.J.
      • et al.
      How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: a five-step approach (part 1/3).
      • Wijnen B.
      • Van Mastrigt G.
      • Redekop W.K.
      • Majoie H.
      • De Kinderen R.
      • Evers S.
      How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: data extraction, risk of bias, and transferability (part 3/3).
      Systematic reviews
      CRD’s guidance for undertaking reviews in health care.
      • Mathes T.
      • Walgenbach M.
      • Antoine S.L.
      • Pieper D.
      • Eikermann M.
      Methods for systematic reviews of health economic evaluations: a systematic review, comparison, and synthesis of method literature.
      Although Chapter 20 of the Cochrane Handbook for Systematic Reviews of Interventions of the Cochrane Collaboration
      and 3 articles related to informing clinical practice guidelines
      • Thielen F.W.
      • Van Mastrigt G.
      • Burgers L.T.
      • et al.
      How to prepare a systematic review of economic evaluations for clinical practice guidelines: database selection and search strategy development (part 2/3).
      • van Mastrigt G.A.
      • Hiligsmann M.
      • Arts J.J.
      • et al.
      How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: a five-step approach (part 1/3).
      • Wijnen B.
      • Van Mastrigt G.
      • Redekop W.K.
      • Majoie H.
      • De Kinderen R.
      • Evers S.
      How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: data extraction, risk of bias, and transferability (part 3/3).
      provide guidance, their recommendations do not focus on evaluating the quality of conduct or the risk of bias in SR-CCEOs. A critical analysis of guidelines on conducting and reporting SR-CCEOs identified multiple disagreements in these recommendations, suggesting that a standardized approach to conducting SR-CCEOs is needed.
      • Jacobsen E.
      • Boyers D.
      • Avenell A.
      Challenges of Systematic Reviews of Economic Evaluations: A Review of Recent Reviews and an Obesity Case Study.
      Making universal recommendations for SR-CCEOs is difficult because they differ in several important aspects, in particular, with regard to their search and inclusion criteria, such as the types of studies included (trial or model-based, cost, or cost-effectiveness), or in reporting solely economic characteristics or economic data alongside clinical outcomes. They also have different objectives (eg, to assess variability in outcomes and synthesize the findings) to identify the evidence gaps, or to assess the methods used.
      Overall, SR-CCEO reliability and usefulness will improve with good practice guidance for SR-CCEOs with different objectives. Thus, ISPOR (The Professional Society for Health Economics and Outcomes Research) established a global, multistakeholder, multidisciplinary expert task force to address this need (Appendix 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.01.002).
      Although general recommendations on conducting SR-CCEOs are provided, the main goal is guidance on critical appraisal of SR-CCEOs regarding their quality and risk of bias. This report, which includes the ISPOR Criteria for Cost(-Effectiveness) Review Outcomes (CiCERO) Checklist, will assist researchers, producers of health technologies, and evidence users (decision makers/commissioners). The task force categorized the recommendations according to the 6 stages of conducting an SR-CCEO (Table 1).
      Table 1Overview of major quality and risk of bias criteria for systematic reviews of cost and cost-effectiveness outcomes
      NTitle of the stageTopics covered
      Stage 1Planning and developmentClear objective
      Predefined and available protocol
      Protocol deviations
      Stage 2Search for evidenceUpdate or novel systematic review
      Comprehensive or rapid review
      Choice for database(s)
      No. of databases
      Comprehensiveness and reproducibility
      Use of supplementary materials
      Use of grey literature
      Stage 3Study selection and eligibilityProcess of study selection
      Eligibility criteria used
      Stage 4Critical appraisal of included studiesTools to appraise the included studies
      Process of appraisal
      Stage 5Data extraction and synthesisProcess of data-extraction
      Assessment of heterogeneity
      Methods of synthesis
      Assessment of publication bias
      Stage 6Presentation and reportingReporting of included studies
      Reporting of the synthesis

      Stage 1. Planning and Development

      Each SR should be based on a comprehensive predefined protocol. It is a preferred practice to make the protocol of SR publicly available to prevent duplication of ongoing reviews, increase reproducibility of the research, and to avoid selective reporting. This can be achieved by registering the protocol with either immediate or delayed open access, (International prospective register of systematic reviews PROSPERO, the Centre for Open Science, or another independent online database), or by publishing it. Any deviations from this protocol should be included in the final report or publication. Independent of protocol availability, each review should have clearly stated objectives consistent with its reported results and conclusions, such as to synthesize the outcomes or to assess the methods.
      It is routine practice to develop eligibility criteria around the PICO (population, intervention, comparator, and outcome) mnemonic in clinical reviews
      • Richardson W.S.
      • Wilson M.C.
      • Nishikawa J.
      • Hayward R.S.
      The well-built clinical question: a key to evidence-based decisions.
      or reviews of full economic evaluations.
      • van Mastrigt G.A.
      • Hiligsmann M.
      • Arts J.J.
      • et al.
      How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: a five-step approach (part 1/3).
      However, PICO or its derivatives are not fully applicable for methodologic (eg, reviews appraising the design of economic models) or cost reviews (eg, cost of illness) in which the “comparator” or “intervention” component may be absent.
      Depending on the objectives of the SR-CCEO, its design can be focused on:
      • model-based studies: for example, reviews assessing quality of models and reviews of studies using a life-time time horizon;
      • empirical health economic studies: for example, reviews assessing treatment costs and reviews of cost-effectiveness studies using a short time horizon (The task force uses the term empirical studies for single study-based economic evaluations, such as randomized and nonrandomized trial-based economic evaluations and also observational studies [single arm, multiple arm, real-world data] that are used as a basis for cost-effectiveness analyses, often called piggy-back studies. Empirical studies are contrasted with modeling studies, explicitly synthesizing data using various sources);
      • or both, for example, reviews with broad perspectives and multiple time horizons.
      Because SR-CCEOs are often used to inform decision makers, additional framing definitions are essential: time horizon and study perspective. These elements define which methods should be used for the literature search and synthesis.

      Stage 2. Search for Evidence

      A review cannot be considered systematic if it is based on evidence identified through a nontargeted, unsound, incomplete, or nonreproducible search.
      • Levay P.
      • Craven J.
      Systematic Searching. Practical Ideas for Improving Results.
      The quality of the search depends on the experience of the person or group who developed the search.
      • Dickersin K.
      • Scherer R.
      • Lefebvre C.
      Identifying relevant studies for systematic reviews.
      ,
      • Rethlefsen M.L.
      • Farrell A.M.
      • Osterhaus Trzasko L.C.
      • Brigham T.J.
      Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews.
      Approaches to improving the quality of the search include involving information specialists or library scientists in search strategy development and using the peer-review electronic search strategies guidelines.
      • Rethlefsen M.L.
      • Farrell A.M.
      • Osterhaus Trzasko L.C.
      • Brigham T.J.
      Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews.
      ,
      • McGowan J.
      • Sampson M.
      • Salzwedel D.M.
      • Cogo E.
      • Foerster V.
      • Lefebvre C.
      PRESS peer review of electronic search strategies: 2015 guideline statement.
      If a SR-CCEO is performed to update existing reviews, reusing the same search strategies may be appropriate. However, the quality of the initial search strategy should be re-evaluated. If a review uses search strategies from existing reviews to answer amended research questions, reviewers need to ensure that the adaptations in the objectives are reflected in the search strategy.
      Conducting a SR is time-consuming. For clinical reviews it takes an average of 17 months from the registered project start to the publication date.
      • Ganann R.
      • Ciliska D.
      • Thomas H.
      Expediting systematic reviews: methods and implications of rapid reviews.
      ,
      • Borah R.
      • Brown A.W.
      • Capers P.L.
      • Kaiser K.A.
      Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry.
      We expect that SR-CCEOs will have similar timelines: adding search words related to costs to the search line used in a clinical SR will result in less hits, but a more complicated complementary search for grey literature will often be needed.
      Cochrane requires the search date to be within 12 months of the publication date.
      This requirement is appropriate for SR-CCEOs summarizing outcomes. Therefore, a SR-CCEO should be conducted in the shortest time possible that does not compromise quality and comprehensiveness or should be updated before publication. Approaches that can decrease the review’s time requirement include narrowing the SR-CCEO’s objective or setting search restrictions if it is feasible and defensible. However, the task force believes that time duration may be less crucial for methodologic than other reviews, given their objectives.

      Selection of Literature Databases

      Which sources to include in the systematic search should be justified primarily by the review’s objectives, and it is unlikely that searching a single database will identify all relevant literature.
      • Betran A.P.
      • Say L.
      • Gulmezoglu A.M.
      • Allen T.
      • Hampson L.
      Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality.
      There are different viewpoints on the best databases to search.
      • Thielen F.W.
      • Van Mastrigt G.
      • Burgers L.T.
      • et al.
      How to prepare a systematic review of economic evaluations for clinical practice guidelines: database selection and search strategy development (part 2/3).
      ,
      • Wood H.
      • Arber M.
      • Glanville J.M.
      Systematic reviews of economic evaluations: how extensive are their searches?.
      ,
      • Arber M.
      • Glanville J.
      • Isojarvi J.
      • Baragula E.
      • Edwards M.
      • Shaw A.
      • Wood H.
      Which databases should be used to identify studies for systematic reviews of economic evaluations?.
      However, an empirical study concluded that a search in Embase, HTA-journal database, MEDLINE/PubMed, and Scopus enabled identification of almost all the references in a SR-CCEO.
      • Arber M.
      • Glanville J.
      • Isojarvi J.
      • Baragula E.
      • Edwards M.
      • Shaw A.
      • Wood H.
      Which databases should be used to identify studies for systematic reviews of economic evaluations?.
      To minimize the risk of missing relevant studies, we recommend starting with the most commonly used international databases for cost and cost-effectiveness studies. A review of cost-effectiveness reviews (ie, an umbrella review) showed that the most commonly used resources (in order) were: MEDLINE, NHS EED (updated up to 2015), checking reference lists, Embase, and HTA report databases
      • Wood H.
      • Arber M.
      • Glanville J.M.
      Systematic reviews of economic evaluations: how extensive are their searches?.
      (see Appendix B in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.01.002 for databases reflecting specific health topics and for SR-CCEOs with a regional focus).
      Including multiple databases will likely identify more relevant studies, but it comes at the cost of additional records that need screening.
      • Hopewell S.
      • McDonald S.
      • Clarke M.
      • Egger M.
      Grey literature in meta-analyses of randomized trials of health care interventions.
      ,
      • Halladay C.W.
      • Trikalinos T.A.
      • Schmid I.T.
      • Schmid C.H.
      • Dahabreh I.J.
      Using data sources beyond PubMed has a modest impact on the results of systematic reviews of therapeutic interventions.
      Although we recommend searching at least 3 databases, if the reviewers chose not to, their decision should be well-justified and confirmed with evidence.

      Developing and Reporting a Search Strategy

      The search strategy should be comprehensive enough to identify all relevant literature and reproducible, therefore, described in detail. Existing search filters can be used to identify cost and cost-effectiveness studies.
      Filters to Identify Economic Evaluations
      ISSG Search Filter Resource. The InterTASC Information Specialists SubGroup.
      ,
      • Wood H.
      • Arber M.
      • Glanville J.M.
      Systematic reviews of economic evaluations: how extensive are their searches?.
      In addition, recommendations on search term and filter selection (including Boolean operators), as well as considerations on sensitivity to specificity trade-offs and SR-CCEO objectives, are useful.
      • Thielen F.W.
      • Van Mastrigt G.
      • Burgers L.T.
      • et al.
      How to prepare a systematic review of economic evaluations for clinical practice guidelines: database selection and search strategy development (part 2/3).
      ,
      Review authors should consider whether applying restrictions in the search (date of publication, study design, publication format, language, age of the subjects) might limit identification of all relevant literature. For example, if the review searched both clinical and cost-effectiveness studies and limits the search to RCTs, it misses possibly relevant model-based research.
      Reviewers should consider that empirical studies measuring both clinical and cost outcomes are likely to report clinical and cost/cost-effectiveness results in separate publications. Therefore, for reviews with both clinical and economic studies, separate searching for articles reporting on either outcome may be preferable to increase the search results’ comprehensiveness.

      Supplemental Searches

      Even comprehensive search strategies may miss relevant studies, as approximately 4% of included studies were missed by database searches.
      • Arber M.
      • Glanville J.
      • Isojarvi J.
      • Baragula E.
      • Edwards M.
      • Shaw A.
      • Wood H.
      Which databases should be used to identify studies for systematic reviews of economic evaluations?.
      In addition to database searches, other strategies to identify published literature include “snowballing” techniques (searching the bibliographies of all included studies), personal knowledge of existing studies, citation tracking, or by contacting experts in the field.
      • Greenhalgh T.
      • Peacock R.
      Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources.
      This means that the process of identifying relevant literature should include supplemental searches
      Canberra National Health and Medical Research Council
      How to review the evidence: systematic identification and review of the scientific literature. Handbook series on preparing clinical practice guidelines.
      using at least 1-step back citation tracking of included studies.

      Searching for Grey Literature

      Searching grey literature is challenging because the results are dependent on when the search is conducted, and therefore, potentially nonreproducible. Grey literature refers to research that is either unpublished or has been published outside of the traditional commercial or academic publishing and distribution channels. Examples of grey literature include government reports, policy statements, and issues articles. However, grey literature may be particularly important to SR-CCEOs as one way to address publication bias. Thus, if a search of grey literature is not performed, it should be clearly justified.
      We recommend including grey literature and to follow recommendations on grey literature searches.
      The Canadian Agency for Drugs and Technologies in Health
      Grey Matters: a practical tool for searching health-related grey literature.
      A supplementary search on HTA is especially important for SR-CCEOs because relevant reports may not be in HTA databases (see sources in Appendix B, sections 2 and 3 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.01.002). Furthermore, the authors may want to explore platforms that collect and aggregate grey literature regarding specific topics, such as the Program for Monitoring Emerging Diseases (ProMED) of the International Society for Infectious Diseases (https://promedmail.org/about-promed/).
      As a general rule, we do not recommend that abstracts of conference proceedings be included in a search, even if technically possible. Scientific conference abstracts in SR-CCEOs could increase the risk of bias because it has been shown that more than half of such abstracts ultimately fail to publish their results after peer review in full,
      • Shemilt I.
      • Khan N.
      • Park S.
      • Thomas J.
      Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews.
      and other abstracts (eg, the Society of Medical Decision Making, Health Technology Assessment international conference abstracts) are not indexed in international databases. Nevertheless, reviewers may include them if they make a solid argument for inclusion, for instance, to identify such abstracts for further follow-up for full-text publications.
      Social networks (a social media website or other application sharing information) may become additional sources of both clinical and economic data for SR-CCEOs. Although unknown, the risk of bias from these sources seems obvious. Reviewers should not apply information derived from such networks without first evaluating the risk of bias.

      Stage 3. Study Selection and Eligibility

      The study selection process includes screening of titles, abstracts, and full-text publications. Methods for study selection should promote transparency and minimize bias. The transparency in a SR-CCEO can be achieved by following SR reporting guidelines, such as the PRISMA statement.
      • Moher D.
      • Liberati A.
      • Tetzlaff J.
      • Altman D.G.
      Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
      There is unlikely to be a “one-size-fits-all” approach, so when evaluating a SR-CCEO, it is important to evaluate how the methodologic approach may contribute to risk of bias.
      For a SR-CCEO using the methods or outcomes from previously published reviews, the risk of bias increases when the previous reviews’ data analysis steps are applied. For example, the risk of bias would be higher if not only the search results are applied, but also full-text inclusion, due to the uncertainty in reliability of each of the literature selection and analysis steps.

      Process of Study Selection

      There are a number of tools and methodologic recommendations on study selection in clinical SRs that are relevant for a SR-CCEO. For example, AMSTAR-2 (A MeaSurement Tool to Assess systematic Reviews) appraises the quality of conduct around study selection,
      • Shea B.J.
      • Reeves B.C.
      • Wells G.
      • et al.
      AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both.
      and Robson et al (2018) summarizes the key conclusions of a SR related to study selection methods.
      • Robson R.C.
      • Pham B.
      • Hwee J.
      • et al.
      Few studies exist examining methods for selecting studies, abstracting data, and appraising quality in a systematic review.
      The common recommendation to minimize the risk of excluding a relevant study or including an irrelevant study, is to perform each step of the study selection process, ideally independently, in duplicate, with conflicts resolved through discussion or by a third party while a combination of both is to be preferred.
      One approach to address the risk of bias in literature selection if resources are limited, is to be more liberal in reviewing titles or abstracts for inclusion by a single reviewer and then at the full-text review stage, ensure that there is duplicate reviewing and stringent criteria application. This should mitigate any issues with a single reviewer and balance the risk of overinclusion (which comes with more research costs) with the risk of excluding relevant citations.
      • Shemilt I.
      • Khan N.
      • Park S.
      • Thomas J.
      Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews.
      ,
      • Robson R.C.
      • Pham B.
      • Hwee J.
      • et al.
      Few studies exist examining methods for selecting studies, abstracting data, and appraising quality in a systematic review.
      Another strategy is using tools with machine learning capabilities (eg, Abstrackr, DistillerSR, SWIFT-Active Screener, and RobotAnalyst). In particular, these tools can be used to duplicate the manual selection. Although machine-learning tools decrease screening time, the risk of bias in using such tools is currently uncertain. The available evidence is limited, and their performance is highly varied.
      • Gates A.
      • Guitard S.
      • Pillay J.
      • et al.
      AHRQ Methods for Effective Health Care. Performance and Usability of Machine Learning for Screening in Systematic Reviews: A Comparative Evaluation of Three Tools.
      • Gartlehner G.
      • Wagner G.
      • Lux L.
      • et al.
      Assessing the accuracy of machine-assisted abstract screening with DistillerAI: a user study.
      • Howard B.E.
      • Phillips J.
      • Tandon A.
      • et al.
      SWIFT-Active Screener: accelerated document screening through active learning and integrated recall estimation.
      If nonvalidated artificial intelligence tools are used, their literature screening accuracy should be tested on a sample and their use should be clearly reported.

      Restrictions in Eligibility Criteria

      It is difficult to characterize how the use of greater restrictions in study selection relates to the relevance and bias of a review’s outcomes because such restrictions can increase or decrease these measures. For example, in clinical reviews, restricting the inclusion criteria to RCTs may increase the risk of bias with respect to adverse event rates (underestimation), but decrease the risk of bias in estimates of effectiveness.
      For SR-CCEOs, there are a variety of relevant restrictions that might be considered beyond study design. The combination of these restrictions represents trade-offs between internal validity and broader generalizability (Box 1). Furthermore, restrictions on study perspective and cost methodologies (how and which costs are included in the analyses) may increase or decrease bias relative to the review’s intended purpose.
      Study selection restrictions in eligibility criteria that represent trade-offs between internal validity and generalizability
      • Restriction by publication date: If only including the last X years, the reviewer may actually increase generalizability to current and future years due to changes in research methods, standard of care, or other parameters.
      • Restriction by country/region: Restrictions by country/region are frequently motivated by healthcare system or cost comparability, increasing internal validity for making statements about those settings (conditional on equally high quality of studies). However, this limits generalizability to those countries/regions included and perhaps to very similar country/regions.
      • Restriction by language: This restriction can increase validity, but limit generalizability (eg, restricting to English-language publications while searching for US studies), or bias the outcomes (eg, restricting to English-language publications in studies with a global perspective). The challenge of including studies published in many languages is that the reviewer needs to be able to read/translate/interpret the text in each language, which may not be feasible. Although in some circumstances, Google Translate or other tools can help to automate translations,
        • Robson R.C.
        • Pham B.
        • Hwee J.
        • et al.
        Few studies exist examining methods for selecting studies, abstracting data, and appraising quality in a systematic review.
        the accuracy of these translations should be verified to avoid biases in interpretation.
      Our experience suggests that applying restriction criteria during the search or when screening titles and abstracts is efficient. However, sometimes, full-text reading is unavoidable. If evidence quality is used as an exclusion criterion, another approach to assess the risk of bias would be to apply a scenario analysis where excluded sources are included to see whether that changes the conclusion.

      Stage 4. Critical Appraisal of Included Studies

      HTA bodies demand transparency and sound methods in original cost and cost-effectiveness studies to apply them in appraisals. Logically, to reduce flaws in synthesizing the evidence, a SR-CCEO should include a methodologic quality assessment of included studies.
      Although assessing the quality of included studies, reviewers should provide a qualitative description and a critique of the evidence base. Reviewers should be explicit about: (1) the existence of and the type of biases that may exist in each study (eg, quality, quality of reporting, and sponsorship in the study), and (2) whether and how estimates were adjusted for transferability and with what assumptions. To increase the consistency in assessment of the methodologic quality of each included study, one of the standard checklists (see below) is justified and should be used over self-designed evaluation approaches.
      Appropriate methodologic quality assessment for various kinds of cost and cost-effectiveness publications depends on the type of research conducted (eg, a trial-based study may need to focus more on consideration of population generalizability). Thus, assessment of quality in an empirical cost or cost-effectiveness study should not be handled in the same way as the assessment of a model.
      There are a number of checklists developed to assess methodologic quality or quality of reporting in included cost and cost-effectiveness studies.
      • Wijnen B.
      • Van Mastrigt G.
      • Redekop W.K.
      • Majoie H.
      • De Kinderen R.
      • Evers S.
      How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: data extraction, risk of bias, and transferability (part 3/3).
      The most commonly used are:
      • British Medical Journal checklist
        • Drummond M.F.
        • Jefferson T.O.
        Guidelines for authors and peer reviewers of economic submissions to the BMJ. The BMJ Economic Evaluation Working Party.
        ;
      • Phillips checklist for model-based studies
        • Philips Z.
        • Ginnelly L.
        • Sculpher M.
        • et al.
        Review of guidelines for good practice in decision-analytic modelling in health technology assessment.
        ;
      • Quality of Cost-Effectiveness Studies checklist for model-based evaluations
        • Chiou C.F.
        • Hay J.W.
        • Wallace J.F.
        • et al.
        Development and validation of a grading system for the quality of cost-effectiveness studies.
        ;
      • Consensus on Health Economic Criteria (CHEC) for trial-based studies
        • Evers S.
        • Goossens M.
        • de Vet H.
        • van Tulder M.
        • Ament A.
        Criteria list for assessment of methodological quality of economic evaluations: consensus on Health Economic Criteria.
        ;
      • Consolidated Health Economic Evaluation Reporting Standards (CHEERS)
        • Husereau D.
        • Drummond M.
        • Petrou S.
        • et al.
        Consolidated Health Economic Evaluation Reporting Standards (CHEERS): explanation and elaboration: a report of the ISPOR Health Economic Evaluation Publication Guidelines Good Reporting Practices Task Force.
        ;
      • Bias in Economic Evaluation (ECOBIAS) Checklist for trial- and model-based studies
        • Adarkwah C.C.
        • van Gils P.F.
        • Hiligsmann M.
        • Evers S.M.
        Risk of bias in model-based economic evaluations: the ECOBIAS checklist.
        ;
      • Second Panel on Cost-Effectiveness checklist
        • Sanders G.D.
        • Neumann P.J.
        • Basu A.
        • et al.
        Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine.
        ;
      • TRansparent Uncertainty ASsessmenT (TRUST) Tool for systematically identifying, assessing, and reporting uncertainties in decision models
        • Grimm S.E.
        • Pouwels X.
        • Ramaekers B.L.T.
        • et al.
        Development and Validation of the TRansparent Uncertainty ASsessmenT (TRUST) Tool for assessing uncertainties in health economic decision models.
        ;
      • Questionnaire to assess the relevance and credibility of a modeling studies
        • Jaime Caro J.
        • Eddy D.M.
        • Kan H.
        • et al.
        Questionnaire to assess relevance and credibility of modeling studies for informing health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report.
      Most of these tools are comparable in their coverage of key design characteristics. However, they differ in the extent to which they are suitable for empirical or model-based studies or whether their specific focus is on the quality of methods or on reporting. TRUST deviates in this respect; it is focused on identifying, assessing, and reporting uncertainty.
      • Grimm S.E.
      • Pouwels X.
      • Ramaekers B.L.T.
      • et al.
      Development and Validation of the TRansparent Uncertainty ASsessmenT (TRUST) Tool for assessing uncertainties in health economic decision models.
      In addition, the reviews of modeling studies will benefit from assessment of data source quality in the models.
      • Cooper N.
      • Coyle D.
      • Abrams K.
      • Mugford M.
      • Sutton A.
      Use of evidence in decision models: an appraisal of health technology assessments in the UK since 1997.
      The selection of the right methodologic quality instrument will be a trade-off between the research question and objectives of the SR-CCEO, the available research capacity, the thoroughness of the evaluation of quality, and the requirements of the project funder or the target journal (if any). A comparative assessment of the checklists is reported by Wijnen et al (2016).
      • Wijnen B.
      • Van Mastrigt G.
      • Redekop W.K.
      • Majoie H.
      • De Kinderen R.
      • Evers S.
      How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: data extraction, risk of bias, and transferability (part 3/3).
      No single checklist can be recommended, but a clear motivation must be given for use in the SR-CCEO. To minimize systematic and nonsystematic errors, at least, 2 reviewers should assess the quality of studies included in a SR-CCEO independently.

      Stage 5. Data Extraction and Synthesis

      Performing Data Extraction

      The same data extraction standards and expectations that apply to SRs of clinical effectiveness should be applied to SR-CCEOs. Data extraction by a single reviewer results in more errors on average than does duplicated data extraction with the observed relative difference in accuracy of 21.7%.
      • Buscemi N.
      • Hartling L.
      • Vandermeer B.
      • Tjosvold L.
      • Klassen T.P.
      Single data extraction generated more errors than double data extraction in systematic reviews.
      Although duplicated extraction is preferred from the accuracy viewpoint, there is a trade-off between the accuracy and efforts required,
      • Shemilt I.
      • Khan N.
      • Park S.
      • Thomas J.
      Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews.
      especially because a SR-CCEO generally involves extracting a broad range of target outcomes (ie, clinical, cost, and cost-effectiveness outcomes), as well as data related to methodology. If an independent duplicated extraction is not possible, reviewers may consider performing a verification of study characteristics and extracting outcome data independently in duplicate.
      • Higgins J.
      • Lasserson T.
      • Chandler J.
      • et al.
      Methodological Expectations of Cochrane Intervention Reviews (MECIR) standards for the conduct and reporting of new Cochrane Intervention Reviews, reporting of protocols and the planning, conduct and reporting of updates.

      Performing Data Synthesis

      Considerations for synthesizing data depend on the purpose of the review (eg, synthesizing the outcomes or reporting methodologic issues).
      • Bell C.M.
      • Urbach D.R.
      • Ray J.G.
      • et al.
      Bias in published cost effectiveness studies: systematic review.
      There is no consensus on the best way to synthesize economic evidence. Possible approaches include structured narrative synthesis (using descriptive methods instead of statistical approaches),
      graphical synthesis (eg, cost-effectiveness diagram, permutation matrix),
      • Clarkson P.
      • Davies L.
      • Jasper R.
      • Loynes N.
      • Challis D.
      A systematic review of the economic evidence for home support interventions in dementia.
      ,
      • Gilbody S.
      • Bower P.
      • Whitty P.
      Costs and consequences of enhanced primary care for depression: systematic review of randomised economic evaluations.
      hierarchical matrix,
      • Nixon J.
      • Khan K.S.
      • Kleijnen J.
      Summarising economic evaluations in systematic reviews: a new approach.
      or quantitative synthesis/meta-analysis (see “Meta-analysis in SR-CCEO”).
      • Rotter T.
      • Kinsman L.
      • James E.
      • et al.
      The effects of clinical pathways on professional practice, patient outcomes, length of stay, and hospital costs: Cochrane systematic review and meta-analysis.
      ,
      • de Morton N.A.
      • Keating J.L.
      • Jeffs K.
      Exercise for acutely hospitalised older medical patients.
      The stated order reflects the most applicable synthesis approach (ie, the approach that can be used under any circumstances to the least used synthesis based on lack of applicability).
      One of the main challenges in choosing the “best” synthesis method for a particular SR-CCEO is matching the approach to synthesis to the review’s scope and the observed variability among the studies it identifies. This variability can be methodologic, clinical, or health setting (administrative- or jurisdiction-related). It is especially challenging to make a single recommendation on a synthesis approach because SR-CCEOs themselves have broadly different scopes. Some reviews comment on the implication of the cost and cost-effectiveness studies for a broad range of jurisdictions, and others comment on the implication for a much narrower range (eg, HTA for a single government).
      A premise to enable assessment of the synthesis’s adequacy is a clearly defined objective that includes the intended audience (jurisdiction or health setting). Guiding questions should be used to assess clinical, health setting, and methodologic compatibility (diversity or variability that cannot be measured statistically). These questions should be informed by tools for assessing transferability and applicability,
      • Barbieri M.
      • Drummond M.
      • Rutten F.
      • et al.
      What do international pharmacoeconomic guidelines say about economic data transferability?.
      ,
      • Drummond M.
      • Barbieri M.
      • Cook J.
      • et al.
      Transferability of economic evaluations across jurisdictions: ISPOR Good Research Practices Task Force report.
      for instance using a decision chart for assessing the transferability of cost and cost-effectiveness results between countries.
      • Welte R.
      • Feenstra T.
      • Jager H.
      • Leidl R.
      A decision chart for assessing and improving the transferability of economic evaluation results between countries.
      Generally, results from modeling studies and empirical studies should be synthesized separately. Cost and cost-effectiveness studies based on trials or observational study designs, as well as probabilistic and deterministic analyses, should be synthesized separately, too. In addition, incorporating the results of sensitivity analyses should be considered.
      • Ruiz-Negron N.
      • Menon J.
      • King J.B.
      • Ma J.
      • Bellows B.K.
      Cost-effectiveness of treatment options for neuropathic pain: a systematic review.
      ,
      • Squires H.
      • Poku E.
      • Bermejo I.
      • et al.
      A systematic review and economic evaluation of adalimumab and dexamethasone for treating non-infectious intermediate uveitis, posterior uveitis or panuveitis in adults.
      When synthesizing numeric values, articles will likely be excluded based on missing information necessary for judging eligibility, applicability, homogeneity, etc. For example, missing demographic characteristics of the population analyzed may make it impossible to determine if the study applies to an age group that the SR-CCEO focuses on. This should be properly documented. Ideally, sensitivity analysis should be done with and without the questionable sources.
      In a SR-CCEO that summarizes cost or cost-effectiveness outcomes, all cost data should be converted into the same currency. In addition, it should be expressed in the same year (ie, inflation-adjusted), using the standard inflator for the country on which the analysis is focused, before the results are synthesized either narratively or quantitatively.
      In the assessment of costs heterogeneity, the methodologic, clinical, and setting compatibility should be considered where, in particular, the latter 2 will have their impact on resource use. For instance, the choice of conversion approach for costs would depend on settings’ comparability,
      • Barbieri M.
      • Drummond M.
      • Rutten F.
      • et al.
      What do international pharmacoeconomic guidelines say about economic data transferability?.
      ,
      • Drummond M.
      • Barbieri M.
      • Cook J.
      • et al.
      Transferability of economic evaluations across jurisdictions: ISPOR Good Research Practices Task Force report.
      with purchasing power parity used to compare costs in heterogeneous settings. Although standardization of costs should be undertaken for the synthesis, the original costs reported in the study should also be presented in the SR-CCEO as with all relevant original data, as valuation methods may differ.
      • Mattingly 2nd, T.J.
      • Levy J.F.
      • Slejko J.F.
      • Onwudiwe N.C.
      • Perfetto E.M.
      Estimating drug costs: how do manufacturer net prices compare with other common US price references?.
      SR-CCEOs that assess the methodology of included cost and cost-effectiveness studies have an exceptionally wide set of methodologic questions on which they may focus.
      • Padmasawitri T.I.A.
      • Frederix G.W.
      • Alisjahbana B.
      • Klungel O.
      • Hovels A.M.
      Disparities in model-based cost-effectiveness analyses of tuberculosis diagnosis: a systematic review.
      Hence, for such reviews, it is likely that only the broad criteria on narrative synthesis are applicable, unless the review is based on a narrow objective of only including studies that are comparable.

      Exploring Heterogeneity in Data

      Figure 1 illustrates that the “right” approach for summarizing cost and cost-effectiveness outcomes depends on the degree of clinical and methodologic compatibility in the studies included. When studies are not comparable, narrative synthesis/comparison will be more appropriate. Although not all of the differences in reported values can be explained, we strongly encourage the reviewers to attempt to do so by analyzing characteristics of the studies and their impact on outcomes. Some factors, such as quality of reporting and conflicts of interest in the studies, can be direct indicators of risks of bias and may contribute to heterogeneity in outcomes. It is more challenging, although, to assess how methodologic differences in the studies contribute to heterogeneity in outcomes.
      Figure thumbnail gr1
      Figure 1Flowchart illustrating the method to determine data-synthesis in systematic reviews aiming to summarize cost and cost-effectiveness outcomes.
      NMB indicates net monetary benefit; WTP, willingness to pay.
      Only in the case where the SR-CCEO’s objective is very narrowly focused is it feasible to explore associations between modeling methods and costs or cost-effectiveness outcomes using meta-regression analysis.
      • Burgers L.T.
      • van de Wetering F.T.
      • Severens J.L.
      • Redekop W.K.
      Using meta-regression analyses in addition to conventional systematic review methods to examine the variation in cost-effectiveness results: a case study.
      If methodologic factors that can potentially explain differences between the studies’ outcomes are identified, they should be reported.

      Meta-analysis

      Only studies considered compatible with regard to clinical and health settings (eg, PICO,) and study methodology (ie, time horizon and study perspective) may be considered for synthesis. If a SR-CCEO pools outcomes in one common metric compatibility of different health settings (or jurisdictions) should be carefully assessed (Fig 1). Usually, a very high degree of incompatibility will imply that pooling such results is not appropriate.
      Therefore, a single quantitative synthesis may only be used in narrowly focused reviews with approaches to synthesis based on a distribution of outcomes rather than a single “true” outcome (eg, random-effects models).
      ,
      • DerSimonian R.
      • Laird N.
      Meta-analysis in clinical trials.
      A SR-CCEO with a broad scope should report the results for compatible subgroups that are consciously selected, ideally based on predefined criteria (eg, results for high-income Asian countries).
      It is the task force’s opinion that the costs reported in various cost and cost-effectiveness studies are typically (although not always) more heterogeneous than effects (by heterogeneity we mean statistically measured variability). Therefore, Figure 1 suggests a hierarchical approach in exploring data compatibility/homogeneity and pooling the data. This means each next level is possible on the condition that all of the previous levels have been achieved. In this way, homogeneity can be assessed in a similar manner as in clinical reviews.
      ,
      • DerSimonian R.
      • Laird N.
      Meta-analysis in clinical trials.
      Data that can be pooled:
      • for cost-effectiveness studies, the average and incremental effectiveness when there is sufficient homogeneity, as well as clinical and methodologic comparability (the common effectiveness outcomes in cost-effectiveness studies, for example, quality adjusted life-years [QALYs] or life-years gained),
      • for costing studies, the average costs when there is methodologic and health setting comparability,
      • for cost-effectiveness studies, the average and incremental costs when there is methodologic and health setting comparability,
      • for cost-effectiveness studies, the net benefit (either net monetary benefit or net health benefit) when homogeneity and comparability is achieved in all above levels and willingness to pay threshold homogeneity is observed (or when the disaggregated costs and benefits can be combined using a common willingness to pay threshold).
      To address incomparability among studies, a sensitivity (subgroup) analysis can be used in a SR-CCEO, similar to the clinical reviews.

      Publication Bias

      Publication bias exists if the outcomes of a cost or cost-effectiveness study influence the publication decision. Bias in cost-effectiveness studies exists when published incremental cost-effectiveness ratios (ICERs) cluster around a proposed threshold, and it is likely to relate to the origin of the sponsorship.
      • Bell C.M.
      • Urbach D.R.
      • Ray J.G.
      • et al.
      Bias in published cost effectiveness studies: systematic review.
      Publication bias in SR-CCEO can be related to multiple reasons including:
      • failure to submit (sponsored) cost-effectiveness studies that have unfavorable results (an indicator of publication bias of this type can be a relationship in study sponsorship and reported incremental cost-effectiveness of technologies);
      • priority setting by target journals publishing cost and cost-effectiveness studies; for example, preference to publish methodologic research, innovative evaluations (typically conducted for high-income settings), and to avoid model adaptations.
      Assessment of publication bias may not be straightforward in SR-CCEOs. Researchers are advised to follow the task force’s recommendations in Box 2. However, none of the proposed assessment methods is perfect and we encourage the development of new approaches.
      Task Force recommendations to assess publication bias in systematic reviews with cost and cost effectiveness outcomes (SR-CCEOs)
      • • Search relevant grey literature (see the grey literature subsection).
      • • Search for conference proceedings with published abstracts that did not lead to peer-reviewed publications. (Note: Abstracts should not be searched for inclusion. However, they can be useful to assess possible publication bias.)
      • • Analyze conflicts of interest (sponsorship) reported in included studies.
      • • Analyze any differences in studies’ outcomes by sponsorship and publication status (ie, differences between grey literature and published reports).
      • • Assess and explore the direction and magnitude of cost and effect differences in publications, for instance, placing the effectiveness results from cost-effectiveness analyses in the context of existing reviews of clinical effectiveness.
      • • Analyze the values and interpretations of reported sensitivity analyses (or their lack).
      • • Benchmark the approaches to exploring the publication bias applied in the clinical reviews, such as looking for the trials’ protocols and exploring funnel plot asymmetry (if the SR-CCEO includes empirical studies).

      Stage 6. Presentation and Reporting

      To optimize usefulness, it is important that the review reports, in sufficient detail, study characteristics and specific outcomes (at a minimum). More standardized reporting of SR-CCEOs will improve comparability between reviews and may influence future reporting in primary studies of cost and cost-effectiveness analyses.
      For SR-CCEOs, the outcomes of interest (eg, total costs, life years, QALYs), as well as methodologic aspects (eg, study perspective, health state valuation, type of costs, costs valuation), should be reported for each included study. Both cost and health outcomes should be presented separately for each strategy, within each study. Whether it is relevant to report one “base case” result or a range of results will depend on each specific research question posed in each separate SR-CCEO.
      • Husereau D.
      • Drummond M.
      • Petrou S.
      • et al.
      Consolidated Health Economic Evaluation Reporting Standards (CHEERS): explanation and elaboration: a report of the ISPOR Health Economic Evaluation Publication Guidelines Good Reporting Practices Task Force.
      Economic outcomes and information regarding included studies (eg, the characteristics of patient populations and the methodologic choices adopted in each included study), should be reported in summary tables. Box 3 presents the common elements in existing checklists assessing methodologic quality or quality of reporting in cost or cost-effectiveness studies (the minimum reporting requirements).
      • Drummond M.F.
      • Jefferson T.O.
      Guidelines for authors and peer reviewers of economic submissions to the BMJ. The BMJ Economic Evaluation Working Party.
      ,
      • Philips Z.
      • Ginnelly L.
      • Sculpher M.
      • et al.
      Review of guidelines for good practice in decision-analytic modelling in health technology assessment.
      ,
      • Evers S.
      • Goossens M.
      • de Vet H.
      • van Tulder M.
      • Ament A.
      Criteria list for assessment of methodological quality of economic evaluations: consensus on Health Economic Criteria.
      • Husereau D.
      • Drummond M.
      • Petrou S.
      • et al.
      Consolidated Health Economic Evaluation Reporting Standards (CHEERS): explanation and elaboration: a report of the ISPOR Health Economic Evaluation Publication Guidelines Good Reporting Practices Task Force.
      • Adarkwah C.C.
      • van Gils P.F.
      • Hiligsmann M.
      • Evers S.M.
      Risk of bias in model-based economic evaluations: the ECOBIAS checklist.
      Other elements that researchers may choose to report will depend on the review’s objectives, the analyzed interventions, and can, for instance, include ethical or equity considerations as might have been reported in the studies included, and heterogeneity (subpopulation analysis). The reviewers should acknowledge the process behind the outcomes of interest choices (eg, whether expert opinion was involved).
      Common elements in the existing checklists assessing cost or cost-effectiveness studies
      Tabled 1
      1.Countries (setting of the study)
      2.Population of analysis (population characteristics)
      3.Audience and study perspective
      4.Time horizon and discounting
      5.Adjustment of inflation
      6.Interventions compared
      7.Method(s) for valuation of cost outcomes
      8.Method(s) for valuation of effectiveness and utility outcomes
      9.Compliance/adherence with intervention (eg, screening uptake)
      10.Decision analytic modeling or calculation approach
      11.Health outcomes (eg, gained life years, number of deaths avoided, QALYs)
      12.Uncertainty (eg, deterministic and probabilistic sensitivity analyses, scenario’s, subgroup analyses)
      13.Conflicts of interest and sources of funding
      14.Software (including open-source software)
      QALYs indicates quality of life-years.
      A SR-CCEO that focuses on decision analytic models should also report the:
      • model type and characteristics (eg, clinical pathways, health states, cycle length, transition possibilities, half-cycle correction applied);
      • model validation (eg, face validity, cross-validation against other models, internal and external validity);
      • components of uncertainty analysis extracted and reported separately for probabilistic and deterministic sensitivity analyses, and scenario/subgroup analyses.
      In some cases, there will be more aspects that are relevant to include (eg, disease-specific modeling choices).
      • Chhatwal J.
      • He T.
      • Lopez-Olivo M.A.
      Systematic review of modelling approaches for the cost effectiveness of hepatitis C treatment with direct-acting antivirals.
      If a SR-CCEO includes studies performed without modeling, the specific reporting should include study type (eg, RCT or cohort, method[s] of cost calculation, and eg, regression or descriptive; questionnaires, expert opinion, and control [or stratification] variables).
      A compromise should be found between both the reporting of outcomes in summary tables and their narrative description, especially for items of interest. Although a word limit demanded by peer-reviewed journals can restrict reporting, all the relevant information that cannot be included in the main paper should be presented in online appendices, supplementary materials, or study protocols.

      CiCERO Checklist

      Based on the considerations discussed earlier, the task force developed the ISPOR CiCERO Checklist, a tool to assess the quality of reporting, conduct, and risk of bias in SR-CCEOs. Using CiCERO leads to an overview of the quality and risk of bias in an SR-CCEO (without resulting in a single score). The general conclusion is dependent on the SR-CCEO’s objectives and the data extracted. Assessing the quality and risk of bias will identify the review’s critical weaknesses and give the user a feeling of overall confidence in the results of the SR-CCEO.
      CiCERO includes 13 signaling questions to consider when evaluating the quality of reporting, conduct, and risk of bias in SR-CCEOs (Appendices C, D, and E for the PDF version and Appendix F in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.01.002 for the Excel version). There are 3 versions of the CiCERO checklist for: (1) reviews of cost and cost-effectiveness studies, (2) reviews that summarize methods of cost and costs-effectiveness studies, and (3) SR-CCEOs that use the AMSTAR-2 instrument to assess quality in included studies.
      The process of developing and validating CiCERO is reported in the Box 4. CiCERO’s development was based on current SR-CCEO knowledge and experience. Because this is a rapidly developing research area, it is expected that the task force will update CiCERO and the report’s recommendations in 5 to 7 years.
      Developing the ISPOR CiCERO Checklist
      The ISPOR CrIteria for Cost (-Effectiveness) Review Outcomes (CiCERO) Checklist is based on the ISPOR Critical Appraisal of Systematic Reviews with Cost and Cost-Effectiveness Outcomes Good Practices Task Force Report. CiCERO has a series of questions to consider when evaluating the risk of bias in reviews reporting cost or cost-effectiveness outcomes or reviews reporting the methods of these studies.
      CiCERO was based on combining aspects of existing instruments, such as the Cochrane Handbook for Systematic Reviews of Interventions, AMSTAR-2,
      • Shea B.J.
      • Reeves B.C.
      • Wells G.
      • et al.
      AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both.
      and ROBIS
      • Whiting P.
      • Savovic J.
      • Higgins J.P.
      • et al.
      ROBIS: a new tool to assess risk of bias in systematic reviews was developed.
      plus the deliberation of international experts, task force members representing different stakeholder perspectives from academics to technology assessors and geographies around the world.
      To produce a final checklist, we used a 2-stage validation approach to improve the readability and inter-rater agreement in use of the checklist:
      • (1) By the task force members (8 reviewers, 8 reviews, 2 raters per publication).
      • (2) By members of the ISPOR student network group (minimum a relevant MSc-level) experienced in assessing cost and cost-effectiveness outcomes publications and SR-CCEOs (36 reviewers, 27 reviews, 2-4 raters per publication).
      The task force members piloted the initial instrument then it was adapted and used by the larger panel of students. Each reviewer assessed the risk of bias in the reviews independently. The reasons for disagreements were analyzed resulting in amendments that provided details and clarifications of the checklist. We tested CiCERO on reviews with different objectives: (1) reviews of cost studies, (2) reviews of cost-effectiveness studies, and (3) reviews that summarize methods of cost and costs-effectiveness studies.
      Selection of reviews for validation was based on manuscript diversity in terms of clinical areas, geographical focus, objectives (methodologic vs synthesis), and outcomes (costs or cost-effectiveness). Comments received from the validation groups and the disagreement rates for each question were analyzed to optimize understanding and interpretation of the final version of the checklist.
      Finally, the task force report and checklist underwent 3 formal rounds of review to ensure that the good practice recommendations and checklist meet the high-quality consensus-developed standards of ISPOR Good Practices Task Force Reports.
      ∗There is a shorter version of the CiCERO Checklist for reviews that summarize methods of cost and cost-effectiveness studies and a specific version for SR-CCEOs that are using AMSTAR-2.
      For more details on task force development, see Appendix A in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.01.002 or Criteria and Process for Initiating and Developing an ISPOR Good Practices Task Force.

      Limitations of the Task Force Recommendations and the ISPOR CiCERO Checklist

      Although these recommendations were developed to evaluate the quality of conduct, reporting and risk of bias of SR-CCEOs, they may be used for conducting a rapid review. A poorly conducted systematic review, may not perform as well as a properly conducted, transparently reported rapid review.
      • Tricco A.C.
      • Antony J.
      • Zarin W.
      • et al.
      A scoping review of rapid review methods.
      So far, limited information is available on biases related to social networks as a data source and artificial intelligence in screening and evaluating the literature. Thus, based on more empirical evidence, these topics should be detailed in future discussions regarding quality and risk of bias of SR-CCEO.

      Conclusions

      As the number of SR-CCEOs continues to increase, standardizing the preparation, reporting, and interpretation of their findings is of crucial and growing importance. Such standardization is required to make effective use of this evidence base to support healthcare decision-making. This report describes good practice recommendations, organized in 6 stages, for critically appraising quality and risk of bias in SR- CCEOs. As such, it provides guidance to reviewers on how to minimize the risk of bias, as well as improve the quality of methods and reporting for conducting a SR-CCEO. In this way, SR-CCEOs can provide valuable evidence to healthcare decision makers.
      Author Contributions: Concept and design: Mandrik, Severens, Goldhaber-Fiebert
      Acquisition of data: Mandrik, Severens, Bardach, Ghabri, Hamel, Mathes, Vale, Wisløff, Goldhaber-Fiebert
      Analysis and interpretation of data: Mandrik, Severens, Bardach, Ghabri, Hamel, Mathes, Vale, Wisløff, Goldhaber-Fiebert
      Drafting of the manuscript: Mandrik, Severens, Bardach, Ghabri, Hamel, Mathes, Vale, Wisløff, Goldhaber-Fiebert
      Critical revision of the paper for important intellectual content: Mandrik, Severens, Bardach, Ghabri, Hamel, Mathes, Vale, Wisløff, Goldhaber-Fiebert
      Administrative, technical, or logistic support: Mandrik, Severens, Goldhaber-Fiebert
      Supervision: Severens, Goldhaber-Fiebert
      Other (co-chairs of the task force): Mandrik, Severens, Goldhaber-Fiebert
      Conflict of Interest Disclosures: Dr Severens is an editor of Value in Health and had no role in the peer review process of this article. Dr Vale reported being Chair of the Campbell and Cochrane Economic Methods Group until January 2020. Dr Wisloff reported receiving personal fees from MSD outside the submitted work. No other disclosures were reported.
      Funding/Support: The authors received no financial support for this research.
      Acknowledgment: The co-authors are grateful for contributions from Cristina Masseria (Pfizer), Mark Clowes, and Ruth Wong (the University of Sheffield). In addition, many thanks to Elizabeth Molsen-David at ISPOR for her continuous support from start to finish and her excellent editing.
      The co-authors gratefully acknowledge the especially useful comments provided by Chris Bartlett, Carolina Castañeda, Matt Chenoweth, Michael Drummond, Joost Enzing, Gunhild Hagen, Louise Hartley, Lisa Hess, Manthan Janodia, Rositsa Koleva-Koralova, Kelly Lenahan, Ghislaine van Mastrigt, Joey Mattingly, Celine Pribil, Chao Song, Gian Luca Di Tanna, and Yee Vern Yong.
      The task force is very grateful to the following ISPOR student and new professional members who contributed their time to pilot the CiCERO Checklist: Naveel Atif, Paul Carty, Ziyan Chen, Jinal Chheda, Trupti Dhumal, Jagadeswara Rao Earla, Diána Elmer, Leonie Heron, Fidelia Ida, Chibuzo Iloabuchi, Praneetha Jain, Ki Jin Jeun, Aishwarya Kulkarni, Kelly Lenahan, Ee Vien Low, Rosa Lucchetta, Bhargavi Mahadik, Yves Mbous, Sreeranjani Sundaresan Menon, Abhishek Nair, Haritha S. Nath, Nga Nguyen, Mona Nili, Chinyere Okoh, Apurva Pande, Jose Paredes, Jayeshkumar Patel, Uzma Pathan, Hellen Pham, Sivaraj Raman, Ajaree Rayanakorn, Aditi Shah, Ashna Talwar, Yuzhi Tang, Nirma Vadlamudi, Gintare Valentelyte, Felix Wolfrum, and Yan Wu.
      The co-authors also thank the following ISPOR reviewers for their time and insight on the report: Mariana Adwan, Sinaa Alageel, Reem AL-Mutairi, Naveel Atif, Reshmi Bhageerathy, Zafer Caliskan, Nancy El Hoyek, Ben Forrest, Gunhilde Hagen, Vida Hamidi, April Hebert, Fidelia Ida, Min-Hua Jen, Kirandeep Kaur, Usah Kreaden, Long Le, Yanli Li, Ee Vien Low, Rosa Lucchetta, Abdulrahman Mansir, Mansi Mathur, Sabina Nduaguba, Nga Nguyen, Sivaraj Raman, Anna Ringborg, Saeed Taheri, Bernice Tsoi, Gintare Valentelyte, Qingqing Xu, and Ana Yankovsky.

      References

        • Luhnen M.
        • Prediger B.
        • Neugebauer E.A.M.
        • Mathes T.
        Systematic reviews of health economic evaluations: a structured analysis of characteristics and methods applied.
        Res Synth Methods. 2019; 10: 195-206
        • Mandrik O.
        • Ekwunife O.I.
        • Meheus F.
        • et al.
        Systematic reviews as a "lens of evidence": Determinants of cost-effectiveness of breast cancer screening.
        Cancer Med. 2019; 8: 7846-7858
        • Shea B.J.
        • Reeves B.C.
        • Wells G.
        • et al.
        AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both.
        Bmj. 2017; 358: j4008
        • Oxman A.D.
        • Guyatt G.H.
        Validation of an index of the quality of review articles.
        J Clin Epidemiol. 1991; 44: 1271-1278
        • Whiting P.
        • Savovic J.
        • Higgins J.P.
        • et al.
        ROBIS: a new tool to assess risk of bias in systematic reviews was developed.
        J Clin Epidemiol. 2016; 69: 225-234
        • Moher D.
        • Liberati A.
        • Tetzlaff J.
        • Altman D.G.
        Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
        PLoS Med. 2009; 6e1000097
        • Thielen F.W.
        • Van Mastrigt G.
        • Burgers L.T.
        • et al.
        How to prepare a systematic review of economic evaluations for clinical practice guidelines: database selection and search strategy development (part 2/3).
        Expert Rev Pharmacoecon Outcomes Res. 2016; 16: 705-721
        • van Mastrigt G.A.
        • Hiligsmann M.
        • Arts J.J.
        • et al.
        How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: a five-step approach (part 1/3).
        Expert Rev Pharmacoecon Outcomes Res. 2016; 16: 689-704
        • Wijnen B.
        • Van Mastrigt G.
        • Redekop W.K.
        • Majoie H.
        • De Kinderen R.
        • Evers S.
        How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: data extraction, risk of bias, and transferability (part 3/3).
        Expert Rev Pharmacoecon Outcomes Res. 2016; 16: 723-732
        • Systematic reviews
        CRD’s guidance for undertaking reviews in health care.
        Center for Reviews and Dissemination of York, 2009
        • Mathes T.
        • Walgenbach M.
        • Antoine S.L.
        • Pieper D.
        • Eikermann M.
        Methods for systematic reviews of health economic evaluations: a systematic review, comparison, and synthesis of method literature.
        Med Decis Making. 2014; 34: 826-840
      1. Higgins J.P.T. Green S. Cochrane Handbook for Systematic Reviews of Interventions. Version 6.0 [updated July 2019]. The Cochrane Collaboration, 2019
        • Jacobsen E.
        • Boyers D.
        • Avenell A.
        Challenges of Systematic Reviews of Economic Evaluations: A Review of Recent Reviews and an Obesity Case Study.
        Pharmacoeconomics. 2020; 38: 259-267
        • Richardson W.S.
        • Wilson M.C.
        • Nishikawa J.
        • Hayward R.S.
        The well-built clinical question: a key to evidence-based decisions.
        ACP J Club. 1995; 123: A12-13
        • Levay P.
        • Craven J.
        Systematic Searching. Practical Ideas for Improving Results.
        Facet Publishing, London2019
        • Dickersin K.
        • Scherer R.
        • Lefebvre C.
        Identifying relevant studies for systematic reviews.
        BMJ. 1994; 309: 1286-1291
        • Rethlefsen M.L.
        • Farrell A.M.
        • Osterhaus Trzasko L.C.
        • Brigham T.J.
        Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews.
        J Clin Epidemiol. 2015; 68: 617-626
        • McGowan J.
        • Sampson M.
        • Salzwedel D.M.
        • Cogo E.
        • Foerster V.
        • Lefebvre C.
        PRESS peer review of electronic search strategies: 2015 guideline statement.
        J Clin Epidemiol. 2016; 75: 40-46
        • Ganann R.
        • Ciliska D.
        • Thomas H.
        Expediting systematic reviews: methods and implications of rapid reviews.
        Implement Sci. 2010; 5: 56
        • Borah R.
        • Brown A.W.
        • Capers P.L.
        • Kaiser K.A.
        Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry.
        BMJ Open. 2017; 7e012545
        • Betran A.P.
        • Say L.
        • Gulmezoglu A.M.
        • Allen T.
        • Hampson L.
        Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality.
        BMC Med Res Methodol. 2005; 5: 6
        • Wood H.
        • Arber M.
        • Glanville J.M.
        Systematic reviews of economic evaluations: how extensive are their searches?.
        Int J Technol Assess Health Care. 2017; 33: 25-31
        • Arber M.
        • Glanville J.
        • Isojarvi J.
        • Baragula E.
        • Edwards M.
        • Shaw A.
        • Wood H.
        Which databases should be used to identify studies for systematic reviews of economic evaluations?.
        Int J Technol Assess Health Care. 2018; 34: 547-554
        • Hopewell S.
        • McDonald S.
        • Clarke M.
        • Egger M.
        Grey literature in meta-analyses of randomized trials of health care interventions.
        Cochrane Database Syst Rev. 2007; 18Mr000010
        • Halladay C.W.
        • Trikalinos T.A.
        • Schmid I.T.
        • Schmid C.H.
        • Dahabreh I.J.
        Using data sources beyond PubMed has a modest impact on the results of systematic reviews of therapeutic interventions.
        J Clin Epidemiol. 2015; 68: 1076-1084
        • Filters to Identify Economic Evaluations
        ISSG Search Filter Resource. The InterTASC Information Specialists SubGroup.
        • Greenhalgh T.
        • Peacock R.
        Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources.
        BMJ. 2005; 331: 1064-1065
        • Canberra National Health and Medical Research Council
        How to review the evidence: systematic identification and review of the scientific literature. Handbook series on preparing clinical practice guidelines.
        (Published 1999)
        • The Canadian Agency for Drugs and Technologies in Health
        Grey Matters: a practical tool for searching health-related grey literature.
        • Shemilt I.
        • Khan N.
        • Park S.
        • Thomas J.
        Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews.
        Syst Rev. 2016; 5: 140
        • Robson R.C.
        • Pham B.
        • Hwee J.
        • et al.
        Few studies exist examining methods for selecting studies, abstracting data, and appraising quality in a systematic review.
        J Clin Epidemiol. 2019; 106: 121-135
        • Gates A.
        • Guitard S.
        • Pillay J.
        • et al.
        AHRQ Methods for Effective Health Care. Performance and Usability of Machine Learning for Screening in Systematic Reviews: A Comparative Evaluation of Three Tools.
        Agency for Healthcare Research and Quality (US), Rockville (MD)2019
        • Gartlehner G.
        • Wagner G.
        • Lux L.
        • et al.
        Assessing the accuracy of machine-assisted abstract screening with DistillerAI: a user study.
        Syst Rev. 2019; 8: 277
        • Howard B.E.
        • Phillips J.
        • Tandon A.
        • et al.
        SWIFT-Active Screener: accelerated document screening through active learning and integrated recall estimation.
        Environment International. 2020; 138105623
        • Drummond M.F.
        • Jefferson T.O.
        Guidelines for authors and peer reviewers of economic submissions to the BMJ. The BMJ Economic Evaluation Working Party.
        BMJ. 1996; 313: 275-283
        • Philips Z.
        • Ginnelly L.
        • Sculpher M.
        • et al.
        Review of guidelines for good practice in decision-analytic modelling in health technology assessment.
        Health Technol Assess. 2004; 8 (iii-iv, ix-xi, 1-158)
        • Chiou C.F.
        • Hay J.W.
        • Wallace J.F.
        • et al.
        Development and validation of a grading system for the quality of cost-effectiveness studies.
        Med Care. 2003; 41: 32-44
        • Evers S.
        • Goossens M.
        • de Vet H.
        • van Tulder M.
        • Ament A.
        Criteria list for assessment of methodological quality of economic evaluations: consensus on Health Economic Criteria.
        Int J Technol Assess Health Care. 2005; 21: 240-245
        • Husereau D.
        • Drummond M.
        • Petrou S.
        • et al.
        Consolidated Health Economic Evaluation Reporting Standards (CHEERS): explanation and elaboration: a report of the ISPOR Health Economic Evaluation Publication Guidelines Good Reporting Practices Task Force.
        Value Health. 2013; 16: 231-250
        • Adarkwah C.C.
        • van Gils P.F.
        • Hiligsmann M.
        • Evers S.M.
        Risk of bias in model-based economic evaluations: the ECOBIAS checklist.
        Expert Rev Pharmacoecon Outcomes Res. 2016; 16: 513-523
        • Sanders G.D.
        • Neumann P.J.
        • Basu A.
        • et al.
        Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine.
        JAMA. 2016; 316: 1093-1103
        • Grimm S.E.
        • Pouwels X.
        • Ramaekers B.L.T.
        • et al.
        Development and Validation of the TRansparent Uncertainty ASsessmenT (TRUST) Tool for assessing uncertainties in health economic decision models.
        Pharmacoeconomics. 2020; 38: 205-216
        • Jaime Caro J.
        • Eddy D.M.
        • Kan H.
        • et al.
        Questionnaire to assess relevance and credibility of modeling studies for informing health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report.
        Value Health. 2014; 17: 174-182
        • Cooper N.
        • Coyle D.
        • Abrams K.
        • Mugford M.
        • Sutton A.
        Use of evidence in decision models: an appraisal of health technology assessments in the UK since 1997.
        J Health Serv Res Policy. 2005; 10: 245-250
        • Buscemi N.
        • Hartling L.
        • Vandermeer B.
        • Tjosvold L.
        • Klassen T.P.
        Single data extraction generated more errors than double data extraction in systematic reviews.
        J Clin Epidemiol. 2006; 59: 697-703
        • Higgins J.
        • Lasserson T.
        • Chandler J.
        • et al.
        Methodological Expectations of Cochrane Intervention Reviews (MECIR) standards for the conduct and reporting of new Cochrane Intervention Reviews, reporting of protocols and the planning, conduct and reporting of updates.
        Cochrane Society. 2019;
        • Bell C.M.
        • Urbach D.R.
        • Ray J.G.
        • et al.
        Bias in published cost effectiveness studies: systematic review.
        BMJ. 2006; 332: 699-703
        • Clarkson P.
        • Davies L.
        • Jasper R.
        • Loynes N.
        • Challis D.
        A systematic review of the economic evidence for home support interventions in dementia.
        Value Health. 2017; 20: 1198-1209
        • Gilbody S.
        • Bower P.
        • Whitty P.
        Costs and consequences of enhanced primary care for depression: systematic review of randomised economic evaluations.
        Br J Psychiatry. 2006; 189: 297-308
        • Nixon J.
        • Khan K.S.
        • Kleijnen J.
        Summarising economic evaluations in systematic reviews: a new approach.
        BMJ. 2001; 322: 1596-1598
        • Rotter T.
        • Kinsman L.
        • James E.
        • et al.
        The effects of clinical pathways on professional practice, patient outcomes, length of stay, and hospital costs: Cochrane systematic review and meta-analysis.
        Eval Health Prof. 2012; 35: 3-27
        • de Morton N.A.
        • Keating J.L.
        • Jeffs K.
        Exercise for acutely hospitalised older medical patients.
        Cochrane Database Syst Rev. 2007; Cd005955
        • Barbieri M.
        • Drummond M.
        • Rutten F.
        • et al.
        What do international pharmacoeconomic guidelines say about economic data transferability?.
        Value Health. 2010; 13: 1028-1037
        • Drummond M.
        • Barbieri M.
        • Cook J.
        • et al.
        Transferability of economic evaluations across jurisdictions: ISPOR Good Research Practices Task Force report.
        Value Health. 2009; 12: 409-418
        • Welte R.
        • Feenstra T.
        • Jager H.
        • Leidl R.
        A decision chart for assessing and improving the transferability of economic evaluation results between countries.
        Pharmacoeconomics. 2004; 22: 857-876
        • Ruiz-Negron N.
        • Menon J.
        • King J.B.
        • Ma J.
        • Bellows B.K.
        Cost-effectiveness of treatment options for neuropathic pain: a systematic review.
        Pharmacoeconomics. 2019; 37: 669-688
        • Squires H.
        • Poku E.
        • Bermejo I.
        • et al.
        A systematic review and economic evaluation of adalimumab and dexamethasone for treating non-infectious intermediate uveitis, posterior uveitis or panuveitis in adults.
        Health Technol Assess. 2017; 21: 1-170
        • Mattingly 2nd, T.J.
        • Levy J.F.
        • Slejko J.F.
        • Onwudiwe N.C.
        • Perfetto E.M.
        Estimating drug costs: how do manufacturer net prices compare with other common US price references?.
        Pharmacoeconomics. 2018; 36: 1093-1099
        • Padmasawitri T.I.A.
        • Frederix G.W.
        • Alisjahbana B.
        • Klungel O.
        • Hovels A.M.
        Disparities in model-based cost-effectiveness analyses of tuberculosis diagnosis: a systematic review.
        PloS One. 2018; 13e0193293
        • Burgers L.T.
        • van de Wetering F.T.
        • Severens J.L.
        • Redekop W.K.
        Using meta-regression analyses in addition to conventional systematic review methods to examine the variation in cost-effectiveness results: a case study.
        BMC Health Serv Res. 2016; 16: 23
        • DerSimonian R.
        • Laird N.
        Meta-analysis in clinical trials.
        Controlled Clinical Trials. 1986; 7: 177-188
        • Chhatwal J.
        • He T.
        • Lopez-Olivo M.A.
        Systematic review of modelling approaches for the cost effectiveness of hepatitis C treatment with direct-acting antivirals.
        Pharmacoeconomics. 2016; 34: 551-567
        • Tricco A.C.
        • Antony J.
        • Zarin W.
        • et al.
        A scoping review of rapid review methods.
        BMC Med. 2015; 13: 224