Advertisement

Use of Real-World Evidence in US Payer Coverage Decision-Making for Next-Generation Sequencing–Based Tests: Challenges, Opportunities, and Potential Solutions

  • Patricia A. Deverka
    Correspondence
    Address correspondence to: Patricia A. Deverka, MD, MS, Geisinger National Precision Health, 6101 Executive Blvd, Suite 110, North Bethesda, MD 20852, USA.
    Affiliations
    Deverka Consulting, LLC, Apex, NC, USA
    Search for articles by this author
  • Michael P. Douglas
    Affiliations
    Center for Translational and Policy Research on Personalized Medicine, Department of Clinical Pharmacy, University of California at San Francisco, San Francisco, CA, USA
    Search for articles by this author
  • Kathryn A. Phillips
    Affiliations
    Center for Translational and Policy Research on Personalized Medicine, Department of Clinical Pharmacy, University of California at San Francisco, San Francisco, CA, USA

    Philip R. Lee Institute for Health Policy, University of California, San Francisco, San Francisco, CA, USA

    Helen Diller Family Comprehensive Cancer, University of California at San Francisco, San Francisco, CA, USA
    Search for articles by this author
Open ArchivePublished:March 26, 2020DOI:https://doi.org/10.1016/j.jval.2020.02.001

      Highlights

      • Despite emphasis by researchers and other experts on the potential usefulness of real-world evidence (RWE) for informing coverage decision making, payers have been relatively slower to use RWE in test evaluations. Newer technologies such as next-generation sequencing (NGS) may uniquely benefit from RWE applications, but challenges to payer adoption must be addressed.
      • Although previous studies have evaluated payer uptake of RWE, the focus has been on the context of pharmaceuticals, which have a different regulatory regime than diagnostic tests. We believe this is the first article to directly examine the use of RWE for NGS and what new approaches are needed.
      • The findings and literature review–informed potential solutions should provide insights for test developers pursuing coverage for new NGS tests and, over time, prove useful to public and private payers for coverage decision making,

      Abstract

      Objectives

      Given the potential of real-world evidence (RWE) to inform understanding of the risk–benefit profile of next-generation sequencing (NGS)–based testing, we undertook a study to describe the current landscape of whether and how payers use RWE as part of their coverage decision making and potential solutions for overcoming barriers.

      Methods

      We performed a scoping literature review of existing RWE evidentiary frameworks for evaluating new technologies and identified barriers to clinical integration and evidence gaps for NGS. We synthesized findings as potential solutions for improving the relevance and utility of RWE for payer decision-making.

      Results

      Payers require evidence of clinical utility to inform coverage decisions, yet we found a relatively small number of published RWE studies, and these are predominately focused on oncology, pharmacogenomics, and perinatal/pediatric testing. We identified 3 categories of innovation that may help address the current undersupply of RWE studies for NGS: (1) increasing use of RWE to inform outcomes-based contracting for new technologies, (2) precision medicine initiatives that integrate clinical and genomic data and enable data sharing, and (3) Food and Drug Administration reforms to encourage the use of RWE. Potential solutions include development of data and evidence review standards, payer engagement in RWE study design, use of incentives and partnerships to lower the barriers to RWE generation, education of payers and providers concerning the use of RWE and NGS, and frameworks for conducting outcomes-based contracting for NGS.

      Conclusions

      We provide numerous suggestions to overcome the data, methodologic, infrastructure, and policy challenges constraining greater integration of RWE in assessments of NGS.

      Keywords

      Introduction

      Next-generation sequencing (NGS)–based tests (multigene panels, whole-exome sequencing, whole-genome sequencing) have started to transform the clinical approach to prenatal testing, cancer treatment, diagnosis of rare disorders, and predisposition testing for chronic diseases.
      • Dervan A.P.
      • Deverka P.A.
      • Trosman J.R.
      • et al.
      Payer decision making for next-generation sequencing-based genetic tests: insights from cell-free DNA prenatal screening.
      • Freedman A.N.
      • Klabunde C.N.
      • Wiant K.
      • et al.
      Use of next-generation sequencing tests to guide cancer treatment: results from a nationally representative survey of oncologists in the United States.
      • Splinter K.
      • Adams D.R.
      • Bacino C.A.
      • et al.
      Effect of genetic diagnosis on patients with previously undiagnosed disease.
      • Kalayinia S.
      • Goodarzynejad H.
      • Maleki M.
      • et al.
      Next generation sequencing applications for cardiovascular disease.
      Nevetheless, 2 interrelated factors—inadequate evidence base and lack of coverage by public and private payers—are issues that must be addressed before NGS becomes part of routine care. Payer coverage policies often highlight evidence deficiencies in the clinical validity of the test (result is clinically meaningful) or the clinical utility of the test (result is clinically useful) as reasons for denying coverage. Clinical utility is the evidentiary standard used by most payers when evaluating tests for coverage decision making, a standard that has been affirmed for Medicare by the courts in their determination that the Centers for Medicare and Medicaid services may consider health outcomes and patient management when deciding whether to cover a diagnostic test.
      • Burwell Kort v.
      Civil No. 1:14-cv-01519 (APM).
      From a regulatory perspective in the United States, NGS tests are governed by the Food and Drug Administration’s (FDA’s) Center for Devices and Radiological Health’s evidence requirements for in vitro diagnostics (IVDs), which focuses on the demonstration of analytic validity (technical efficacy of the test) and clinical validity. The absence of regulatory requirements for premarket evidence of clinical utility and complexity of NGS (tests examine multiple genes and produce multiple results, each with distinct clinical implications) makes conventional randomized controlled trials (RCTs) more challenging and expensive to conduct. Also, the clinical actionability of NGS results can evolve over time as new information regarding the relationship between genetic variants and disease risk or drug response becomes available. Thus, observational data may be particularly useful for informing estimates of the effectiveness of testing in different disorders, patient subgroups, and clinical settings.
      Although payers may prefer direct evidence of clinical utility from randomized trials of the impact of NGS tests on provider behavior and patient health outcomes, they frequently must rely on indirect evidence when addressing coverage determinations. In the context of IVDs, indirect evidence is obtained by extrapolating from robust clinical validity studies and treatment outcome studies, constructing a chain of evidence linking test results to patient health outcomes.
      Medical Device Innovation Consortium
      Developing clinical evidence for regulatory and coverage assessments in in vitro diagnostics (IVDs).
      Use of real-world data (RWD) from registries, surveys, and observational studies is a practical alternative for building indirect evidence of the clinical impact of NGS tests, also referred to as real-world evidence (RWE; Box 1).
      Real world Data (RWD): data relating to patient health states and/or delivery of healthcare routinely collected from a variety of sources, including electronic health records, claims and billing data, product and disease registries and data gathered through personal devices and health applications.
      • Jarow J.P.
      • LaVange L.
      • Woodcock J.
      Multidimensional evidence generation and FDA regulatory decision making: defining and using “real-world” data.
      Real world evidence (RWE): the analysis of RWD in a study designed with a high degree of pragmatism, regardless of study type.
      • Jarow J.P.
      • LaVange L.
      • Woodcock J.
      Multidimensional evidence generation and FDA regulatory decision making: defining and using “real-world” data.
      Importantly, the FDA has recognized that RWE can be used as valid scientific evidence to support regulatory claims for new devices. For example, RWE can be used to support expanded indications, conduct postmarketing surveillance, as a control group, and as evidence to identify, demonstrate, or support the clinical validity of a biomarker.
      Use of real-world evidence to support regulatory decision-making for medical devices: guidance for industry and food and drug administration staff.
      Although decision making for regulatory approval of a new test is distinct from payer coverage determinations regarding the test, improvements in the quality of RWD and usefulness of RWE can be seen as “a rising tide that lifts all boats,” such that there should be a downstream benefit to payer decision making.
      • O’Neill T.
      • Miksad R.
      • Miller D.
      • et al.
      ISPOR, the FDA, and the evolving regulatory science of medical device products.
      One caveat is that the FDA can choose to exercise enforcement discretion for IVDs, allowing laboratory-developed tests to enter the market without FDA approval. Nevertheless, there are ongoing efforts to modernize federal oversight of laboratory-developed tests based on a risk-based approach in addition to recent evidence that the FDA is stepping up its regulation of pharmacogenetic testing (PGx) in particular.
      • Ray T.
      FDA stepping up actions against PGx testing, forcing some labs to stop reporting drug information.
      Our objective was to describe the challenges and opportunities for payers, researchers, and test developers to capitalize on the growing availability and applicability of RWE for coverage decisions and develop potential solutions to support greater use of RWE in the context of NGS.
      Previous studies
      • Garrison Jr., L.P.
      • Neumann P.J.
      • Erickson P.
      • et al.
      Using real-world data for coverage and payment decisions: the ISPOR Real-World Data Task Force report.
      • Hampson G.
      • Towse A.
      • Dreitlein W.B.
      • et al.
      Real world evidence for coverage decisions: opportunities and challenges.
      • Hurwitz J.T.
      • Brown M.
      • Graff J.S.
      • et al.
      Is real-world evidence used in P&T monographs and therapeutic class reviews?.
      • Malone D.C.
      • Brown M.
      • Hurwitz J.T.
      • et al.
      Real-world evidence: useful in the real world of US Payer decision making? How? When? And what studies?.
      focused on pharmaceuticals have outlined the relevance of RWE for payers; however, there is a similar and growing interest regarding how RWE may be used to demonstrate the clinical utility of IVDs, a critical evidentiary threshold for payers. The analytic framework for evaluating the clinical utility of IVDs is well established
      • Fryback D.G.
      • Thornbury J.R.
      The efficacy of diagnostic imaging.
      ,
      Methods Guide for Medical Test Reviews. AHRQ Publication No. 12-EC017.
      and makes clear that tests must first have demonstrated analytic and clinical validity before clinical utility can be considered. Once the relationship between test use in a defined patient group changes in provider and patient behavior, and health outcomes are established, RWD from observational studies are frequently used to provide indirect evidence of clinical utility. RWD is also essential in the development of decision-analytic models to evaluate the cost-effectiveness of NGS tests,
      Rational Integration of Clinical Sequencing (RISE), Vanderbilt University Medical Center.
      information that is often relevant for private payer coverage determinations. RWD collection may also occur as part of a coverage with evidence development study or as part of outcomes-based contract evaluations.
      • Garrison Jr., L.P.
      • Neumann P.J.
      • Erickson P.
      • et al.
      Using real-world data for coverage and payment decisions: the ISPOR Real-World Data Task Force report.
      ,
      • Carlson J.J.
      • Chen S.
      • Garrison Jr., L.P.
      Performance-based risk-sharing arrangements: an updated international review.
      The former refers to a situation in which a payer provides provisional coverage of certain items or services conditional on further collection of population-level evidence from a prespecified study
      Centers for Medicare and Medicaid Services
      Coverage with evidence development.
      and the latter to contracts intended to lessen the financial risks to payers for expensive treatments by measuring the actual value delivered to patients.
      • Carlson J.J.
      • Chen S.
      • Garrison Jr., L.P.
      Performance-based risk-sharing arrangements: an updated international review.
      ,
      • Mahendraratnam N.
      • Sorenson C.
      • Richardson E.
      • et al.
      Value-based arrangements may be more prevalent than assumed.
      Nevetheless, whether and how payers use RWE for NGS coverage decision making is not well understood, and diagnostics present unique methodological challenges for the use of RWE.
      In particular, both evidence developers and decision makers have underscored the need to adapt evidentiary frameworks and the use of RWE for NGS-based tests in clinical areas such as oncology, an area of intensive activity for NGS.
      • Conley R.B.
      • Dickson D.
      • Zenklusen J.C.
      • et al.
      Core clinical data elements for cancer genomic repositories: a multi-stakeholder consensus.
      ,
      Green Park Collaborative
      Initial medical policy and model coverage guidelines for clinical next generation sequencing in oncology: report and recommendations.
      There is a similar need to understand the current landscape of how RWD is being used to elucidate the risk–benefit profile of NGS in a broader range of clinical contexts and what factors can accelerate greater uptake of RWE by payers. We briefly describe approaches that have been used to support the use of RWE in general by payers, then focus on identifying trends in the literature, policy environment, and data ecosystem that could affect the use of RWE for coverage decision making for NGS specifically. This information fills a critical gap in the RWE literature and forms the basis of potential solutions for enabling the evidence-based use of NGS by payers over time. The development of potential solutions that can be endorsed and applied by NGS stakeholders is the focus of this study.

      Methods

      Scoping Review

      We undertook a scoping review as opposed to a systematic review to rapidly map key concepts underlying this research topic, identify evidence gaps, and synthesize knowledge within policy and practice contexts.
      • Anderson S.
      • Allen P.
      • Peckham S.
      • et al.
      Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services.
      ,
      • Arksey H.
      • O’Malley L.
      Scoping studies: towards a methodological framework.
      Scoping reviews are particularly useful for examining the extent of research activity in a particular area and determining the value of undertaking a full systematic review in the future. We limited our review to the United States because of significant variability internationally in how healthcare is financed and new technologies are evaluated for approval and payment. For this review, PubMed was used to search peer-reviewed, scientific literature from January 2013 to November 2019 in an attempt to answer 3 research questions: (1) How has RWE been used to support public and private payer decision making in the United States? (2) What are specific examples of RWE uses related to NGS? and (3) How has RWE been used to support decision making by stakeholders that influence payers? (eg, FDA, clinical guideline developers, health technology assessment). The search strategy and inclusion and exclusion criteria are described in Appendix 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.02.001.
      To locate gray literature, we searched Google and GenomeWeb websites as of November 2019 using the following keywords: “Real World Evidence AND Coverage,” “Real World Evidence AND Reimbursement,” “Real World Evidence AND NGS,” “Real World Evidence AND payer,” and “real world evidence” as exact phrases. We also searched Google for gray literature on oncology data-sharing initiatives referenced in articles (see Appendix 1 in the Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.02.001) using the terms “initiative name” and “coverage or reimbursement,” “clinical utility,” “performance-based risk sharing arrangement,” and “payers.” In addition, we reviewed the press releases on company websites known to be sponsors of RWE initiatives (see Appendix 1 in the Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.02.001) to identify additional gray literature. Data abstraction forms were created in Microsoft Excel and used to summarize information from the included peer-reviewed and gray literature articles (see Appendix 1 in the Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.02.001). Two of the authors (P.A.D. and M.P.D.) extracted data independently from the selected documents; the findings were reviewed jointly with discrepancies resolved by consensus.
      Finally, we hand searched the National Human Genome Research Institute’s list of accomplishments in genomic medicine since 2011
      National Human Genome Research Institute
      Accomplishments in genomic medicine.
      and the peer-reviewed and gray literature reference lists of included articles. Search results from scientific (title and abstract) and gray literature were independently screened by 2 authors according to inclusion and exclusion criteria (see Appendix 1 in the Supplemental Material found at https://doi.org/10.1016/j.jval.2020.02.001). Discrepancies were resolved by consensus and review by a third author. Data from final versions of the abstraction forms were synthesized into themes organized by research question and evidence gaps highlighted.

      Results

      Literature Review

      The PubMed search yielded a total of 426 articles, whereas the various gray literature searches produced 183 hits. From the total of 609 articles or hits, we excluded 529 articles based on the following reasons (see Fig. 1): title/abstract review (509), international studies (22), and background studies (19). Hand searching of manuscript reference lists added an additional 53 articles for a total of 69 peer-reviewed studies and 43 gray literature articles included for analysis.
      Figure thumbnail gr1
      Figure 1Flow diagram of included and excluded articles and sources.
      Many articles touted the potential uses of RWE by payers; however, the vast majority of the published literature focused on drugs, not diagnostics.
      • Garrison Jr., L.P.
      • Neumann P.J.
      • Erickson P.
      • et al.
      Using real-world data for coverage and payment decisions: the ISPOR Real-World Data Task Force report.
      ,
      • Hampson G.
      • Towse A.
      • Dreitlein W.B.
      • et al.
      Real world evidence for coverage decisions: opportunities and challenges.
      ,
      • Pearson S.D.
      • Dreitlein W.B.
      • Towse A.
      • et al.
      A framework to guide the optimal development and use of real-world evidence for drug coverage and formulary decisions.
      Green Park Collaborative
      RWE decoder framework: a practical tool for assessing relevance and rigor of real world evidence.
      • Makady A.
      • Ham R.T.
      • de Boer A.
      • et al.
      Policies for use of real-world data in health technology assessment (HTA): a comparative study of six HTA agencies.
      When researchers critically evaluated how payers used RWE for drug decision-making, RWE was infrequently cited in pharmacy and therapeutics materials, even among therapeutic class reviews, in which RWE is more readily available and studies are of high quality.
      • Hurwitz J.T.
      • Brown M.
      • Graff J.S.
      • et al.
      Is real-world evidence used in P&T monographs and therapeutic class reviews?.
      One reason why RWE has not had a bigger impact is that payers lack confidence in the rigor of study designs and the validity of study conclusions (Table 1). This concern by payers exists despite numerous guidance documents published by researchers and other stakeholders regarding good research practices for conducting and reporting RWE studies.
      ENCePP checklist for study protocols (revision 3).
      US Food and Drug Administration
      Good pharmacovigilance practices and pharmacoepidemiologic assessment.
      • Berger M.L.
      • Mamdani M.
      • Atkins D.
      • et al.
      Good research practices for comparative effectiveness research: defining, reporting and interpreting nonrandomized studies of treatment effects using secondary data sources: the ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report—part I.
      • Cox E.
      • Martin B.C.
      • Van Staa T.
      • et al.
      Good research practices for comparative effectiveness research: approaches to mitigate bias and confounding in the design of nonrandomized studies of treatment effects using secondary data sources: the International Society for Pharmacoeconomics and Outcomes Research Good Research Practices for Retrospective Database Analysis Task Force Report—part II.
      • Johnson M.L.
      • Crown W.
      • Martin B.C.
      • et al.
      Good research practices for comparative effectiveness research: analytic methods to improve causal inference from nonrandomized studies of treatment effects using secondary data sources: the ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report—part III.
      • Dreyer N.A.
      • Bryant A.
      • Velentgas P.
      The GRACE checklist: a validated assessment tool for high quality observational studies of comparative effectiveness.
      Patient Centered Outcomes Research Insitute (PCORI)
      methodology standards.
      Table 1RWE limitations.
      Payer concerns about RWE studies
      • Uncertainty regarding internal validity
        • Lack of familiarity with study methods (eg, regression, propensity score matching, etc)
      • Lack of institutionalized methods of obtaining RWD
        • Inaccurate recording of health events
        • Missing data
        • Lack of interoperable data, including genomic data
      • Lack of transparency regarding use of methods and reporting of results
      • Lack of study timeliness for coverage decision making
      • Lack of resources to locate and review observational studies
      • Privacy of patient data/HIPAA compliance
        • Recurring breaches of privacy may discourage individuals from allowing their data to be used for research purposes
      HIPAA indicates Health Insurance Portability and Accountability Act; RWD, real-world data; RWE, real-world evidence.
      The situation becomes more complicated with respect to NGS because researchers attempt to capture probabilistic information as binary results, false-positives/-negatives are not addressed, testing and reporting standards are in flux, data silos prevent data sharing, reclassification of variants is common,
      • Liu P.
      • Meng L.
      • Normand E.A.
      • et al.
      Reanalysis of clinical exome sequencing data.
      and clinical outcomes data are lacking.
      • Dickson D.J.
      • Pfeifer J.D.
      Real-world data in the molecular era: finding the reality in the real world.
      ,
      • Rosenman M.B.
      • Decker B.
      • Levy K.D.
      • et al.
      Lessons learned when introducing pharmacogenomic panel testing into clinical practice.
      A major hurdle for clinical laboratories is deciding which genes have sufficient evidence to support use in clinical care. A method to evaluate the strength of evidence for a gene’s role in a given disease has been developed by ClinGen,
      • Strande N.T.
      • Riggs E.R.
      • Buchanan A.H.
      • et al.
      Evaluating the clinical validity of gene-disease associations: an evidence-based framework developed by the clinical genome resource.
      which oversees the only FDA-recognized public variant database to support clinical validity claims for genetic tests.
      US Food and Drug Adminstration
      Use of public human genetic variant databases to support clinical validity for genetic and genomic-based in vitro diagnostics.
      We identified several RWE studies that evaluated the use of NGS to guide oncology therapy in clinical practice
      • Velcheti V.
      • Patwardhan P.D.
      • Liu F.X.
      • et al.
      Real-world PD-L1 testing and distribution of PD-L1 tumor expression by immunohistochemistry assay type among patients with metastatic non-small cell lung cancer in the United States.
      and the impact of NGS tumor profiling on the health outcomes
      • Presley C.J.
      • Tang D.
      • Soulos P.R.
      • et al.
      Association of broad-based genomic sequencing with survival among patients with advanced non-small cell lung cancer in the community oncology setting.
      ,
      • Singal G.
      • Miller P.G.
      • Agarwala V.
      • et al.
      Association of patient characteristics and tumor genomics with clinical outcomes among patients with non-small cell lung cancer using a clinicogenomic database.
      and economic outcomes
      • Haslem D.S.
      • Chakravarty I.
      • Fulde G.
      • et al.
      Precision oncology in advanced cancer patients improves overall survival with lower weekly healthcare costs.
      • Steuten L.
      • Goulart B.
      • Meropol N.J.
      • Pritchard D.
      • Ramsey S.D.
      Cost-effectiveness of multigene panel sequencing for patients with advanced non-small-cell lung cancer.
      • Signorovitch J.
      • Zhou Z.
      • Ryan J.
      • et al.
      Budget impact analysis of comprehensive genomic profiling in patients with advanced non-small cell lung cancer.
      of cancer patients. Although tumor profiling using NGS has been gaining coverage if the test has received FDA approval
      CMS and commercial insures extend coverage of Oncomine Dx Target Test to more than 160 million U.S. lives.
      or there are clinical guidelines supporting test use,
      • Lu C.Y.
      • Loomer S.
      • Ceccarelli R.
      • et al.
      Insurance coverage policies for pharmacogenomic and multi-gene testing for cancer.
      there is widespread recognition of the need for RWE to drive new frameworks for payer coverage policy development.
      • Trosman J.R.
      • Weldon C.B.
      • Gradishar W.J.
      • et al.
      From the past to the present: insurer coverage frameworks for next-generation tumor sequencing.
      These frameworks include coverage with evidence development and real-world performance-based risk-sharing arrangements (also referred to as “outcomes-based contracts”). There is also a growing body of studies evaluating the clinical utility of NGS for inherited conditions such as hereditary breast and ovarian cancer, familial hypercholesterolemia (FH), and Lynch syndrome (increased risk for colorectal, endometrial, and ovarian cancers), which have resulted in the Centers for Disease Control and Prevention’s assigning these conditions “tier 1” status.
      • Dotson W.D.
      • Douglas M.P.
      • Kolor K.
      • et al.
      Prioritizing genomic applications for action by level of evidence: a horizon-scanning method.
      This designation refers to conditions for which there is some reasonable evidence supporting implementation such as a clinical guideline based on systematic reviews, Centers for Medicare & Medicaid Services (CMS) coverage of testing, or FDA labeling.
      Centers for Disease Control and Prevention
      Introducing the CDC tier-classified database.
      Most public and private payers cover testing for hereditary breast and ovarian cancer and Lynch syndrome
      American Society of Clinical Oncology
      Genetic testing coverage & reimbursement.
      ; however, coverage for FH is highly variable despite recognition by professional associations of the clinical utility of genetic testing. One of the reasons cited for this coverage gap is the lack of cost-effectiveness data supporting FH testing,
      • Hendricks-Sturrup R.M.
      • Lu C.Y.
      Understanding implementation challenges to genetic testing for familial hypercholesterolemia in the United States.
      which requires RWD to populate economic models.
      Pharmacogenomics previously focused on the analysis of variants in a single gene to predict drug response; however, there has recently been much more emphasis on the analysis of multiple pharmacogenes using NGS panels, ideally in a preemptive fashion before the prescription of any targeted drug.
      • Bielinski S.J.
      • Olson J.E.
      • Pathak J.
      • et al.
      Preemptive genotyping for personalized medicine: design of the right drug, right dose, right time-using genomic data to individualize treatment protocol.
      In this scenario, pharmacogenomic results are stored in the electronic health record along with a clinical decision support (CDS) system that alerts the prescriber when an affected drug is prescribed for a patient with variant genetics. The clinical and economic impacts of implementing a preemptive PGx strategy for antiplatelet agents, statins, and warfarin has been demonstrated through modeling and application to a health system cohort.
      • Shi Y.
      • Graves J.A.
      • Garbett S.P.
      • et al.
      A decision-theoretic approach to panel-based, preemptive genotyping.
      The improvements were only modest and are consistent with the observation that payers do not reimburse for preemptive PGx panel testing. The strongest evidence from RWE concerns the relationship between variants in CYP2C19 and antiplatelet therapy such as clopidogrel,
      • Martin J.
      • Williams A.K.
      • Klein M.D.
      • et al.
      Frequency and clinical outcomes of CYP2C19 genotype-guided escalation and de-escalation of antiplatelet therapy in a real-world clinical setting.
      HLA B and carbamazepine,
      • Volpi S.
      • Bult C.J.
      • Chisholm R.L.
      • et al.
      Research directions in the clinical implementation of pharmacogenomics: an overview of US programs and projects.
      and TMTP genotypes and thiopurines,
      • Manzi S.F.
      • Fusaro V.A.
      • Chadwick L.
      • et al.
      Creating a scalable clinical pharmacogenomics service with automated interpretation and medical record result integration: experience from a pediatric tertiary care facility.
      findings further confirmed by a review of 44 economic evaluations of a PGx-informed strategy.
      • Verbelen M.
      • Weale M.E.
      • Lewis C.M.
      Cost-effectiveness of pharmacogenetic-guided treatment: are we there yet?.
      Nevertheless, lack of reimbursement of PGx testing remains a major barrier for the field and appears less related to lack of RWE and more related to implementation barriers, for example, the inability to represent genomic data in the electronic health record and the need to update CDS recommendations to reflect changes in variant interpretation.
      • Roden D.M.
      • McLeod H.L.
      • Relling M.V.
      • et al.
      Pharmacogenomics.
      Some of the most definitive work using RWE has examined the clinical utility and cost-effectiveness of NGS in critically ill infants and pediatric patients with suspected monogenic disorders.
      • Saunders C.J.
      • Miller N.A.
      • Soden S.E.
      • et al.
      Rapid whole-genome sequencing for genetic disease diagnosis in neonatal intensive care units.
      • Monroe G.R.
      • Frederix G.W.
      • Savelberg S.M.
      • et al.
      Effectiveness of whole-exome sequencing and costs of the traditional diagnostic trajectory in children with intellectual disability.
      • Lazaridis K.N.
      • Schahl K.A.
      • Cousin M.A.
      • et al.
      Outcome of whole exome sequencing for diagnostic odyssey cases of an individualized medicine clinic: the Mayo Clinic experience.
      • Clark M.M.
      • Stark Z.
      • Farnaes L.
      • et al.
      Meta-analysis of the diagnostic and clinical utility of genome and exome sequencing and chromosomal microarray in children with suspected genetic diseases.
      • Clark M.M.
      • Hildreth A.
      • Batalov S.
      • et al.
      Diagnosis of genetic diseases in seriously ill children by rapid whole-genome sequencing and automated phenotyping and interpretation.
      Correspondingly, a recent qualitative study of payer decision-making revealed that 71% of payers representing 170 million insured lives cover pediatric exome sequencing, primarily because there are available interventions or to end the diagnostic odyssey.
      • Trosman J.R.
      • Weldon C.B.
      • Slavotinek A.
      • et al.
      Perspectives of US private payers on insurance coverage for pediatric and prenatal exome sequencing: results of a study from the Program in Prenatal and Pediatric Genomic Sequencing (P3EGS).
      In the specific context of neurodevelopmental disorders in pediatric patients, a study of private payer coverage policies of whole-exome sequencing demonstrated a trend toward more favorable coverage decisions over time, correlated with a larger evidence base, including RWE studies.
      • Douglas M.P.
      • Parker S.L.
      • Trosman J.R.
      • et al.
      Private payer coverage policies for exome sequencing (ES) in pediatric patients: trends over time and analysis of evidence cited.
      Another study demonstrated that payers relied on modeled evidence of clinical utility when affirmatively covering noninvasive prenatal testing.
      • Dervan A.P.
      • Deverka P.A.
      • Trosman J.R.
      • et al.
      Payer decision making for next-generation sequencing-based genetic tests: insights from cell-free DNA prenatal screening.
      Finally a review of 55 coverage policies for NGS tests compared with coverage policies for other interventions such as drugs or diagnostic imaging revealed that most clinical studies cited as supporting evidence relied on RCTs; however, some policies did cite health technology assessments and cost-effectiveness assessments, which presumably included RWE.
      • Chambers J.D.
      • Saret C.J.
      • Anderson J.E.
      • et al.
      Examining evidence in U.S. payer coverage policies for multi-gene panels and sequencing tests.
      Notably, NGS tests had a weaker evidence base than other technologies, confirming the perception that there is an undersupply of clinical utility data for this type of intervention.
      There are also studies documenting the growing importance of RWD in the setting of outcomes-based contracting (OBC) for new technologies, primarily concerning drugs.
      • Carlson J.J.
      • Chen S.
      • Garrison Jr., L.P.
      Performance-based risk-sharing arrangements: an updated international review.
      ,
      • Mahendraratnam N.
      • Sorenson C.
      • Richardson E.
      • et al.
      Value-based arrangements may be more prevalent than assumed.
      ,
      • Duhig A.M.
      • Saha S.
      • Smith S.
      • et al.
      The current status of outcomes-based contracting for manufacturers and payers: an AMCP membership survey.
      In addition, there are incentives in bundled care and value-based payment reimbursements mandated by federal legislation
      Centers for Medicare and Medicaid Services
      Medicare Access and CHIP Reauthorization Act of 2015 (MACRA).
      that are based on outcomes best assessed using RWD. This emphasis on value-based payments is a critical factor reshaping how RWE is used for payer decision-making. The limitation is that the results of OBCs are rarely disclosed publicly. The only publicly available description of an OBC related to NGS is the example of Illumina working with Harvard Pilgrim to offer noninvasive prenatal testing to women at average risk while also committing to third-party evaluation of the impact of risk sharing on clinical outcomes and costs and publication of the results. The investigators demonstrated an increase in noninvasive prenatal testing use, modest increases in total testing and diagnostic expenditures, and a decrease in invasive procedures compared with the baseline year when testing was covered only for high-risk pregnancies.
      • McQueen R.
      • Schroeder B.
      • Wright G.
      • et al.
      Evaluating coverage expansion for NIPT through a perfomrance-based risk sharing agreement.
      Numerous data-sharing initiatives have been organized to overcome the data limitations inhibiting the assessment of the real-world impact of NGS. To date, many of these initiatives are specific to oncology, but there are a growing number of population and public health–focused efforts (Table 2).
      • Presley C.J.
      • Tang D.
      • Soulos P.R.
      • et al.
      Association of broad-based genomic sequencing with survival among patients with advanced non-small cell lung cancer in the community oncology setting.
      ,
      Illumina Media Relations
      Illumina and Harvard Pilgrim Partner on value-based contract.
      • Farnaes L.
      • Hildreth A.
      • Sweeney N.M.
      • et al.
      Rapid whole-genome sequencing decreases infant morbidity and cost of hospitalization.
      • Schwartz M.L.B.
      • McCormick C.Z.
      • Lazzeri A.L.
      • et al.
      A model for genome-first care: returning secondary genomic findings to participants and their healthcare providers in a large research cohort.
      • Carey D.J.
      • Fetterolf S.N.
      • Davis F.D.
      • et al.
      The Geisinger MyCode community health initiative: an electronic health record-linked biobank for precision medicine research.
      Renown Institute for Health Innovation
      Healthy Nevada Project.
      Philanthropy News Digest
      Sanford Health receives $25 million for precision medicine initiative.
      • Ray T.
      Precision lung cancer drugs on pricing evaluation group’s agenda.
      • Pecora A.L.
      • Norden A.D.
      • Hervey J.
      • et al.
      Development of a precise, clinically relevant, digital classification schema for cancer.
      The White House
      Fact sheet: President Obama’s precision medicine initiative.
      Foundation Medicine
      Foundation Medicine Launches Precision Medicine Exchange Consortium™ (PMEC) to advance the integration of molecular information in clinical oncology and accelerate adoption of precision care.
      M2GEN.
      VITAL Innovation.
      SetYale School of Medicine
      Digging deep into data.
      McKesson
      iKnowMed: Oncology practice EHR system.
      American Association for Cancer Research
      American Association for Cancer Research’s Genomics Evidence Neoplasia Information Exchange (GENIE).
      Health Resources and Services Administration
      National Coordinating Center for the Regional Genetics Network.
      Michigan Department of Health and Human Services
      Syapse Learning Health Network
      These information networks have enabled greater use of RWD for various types of NGS evaluations, including studies of clinical and economic impact that have the potential to be useful to payers.
      • Presley C.J.
      • Tang D.
      • Soulos P.R.
      • et al.
      Association of broad-based genomic sequencing with survival among patients with advanced non-small cell lung cancer in the community oncology setting.
      ,
      • Haslem D.S.
      • Chakravarty I.
      • Fulde G.
      • et al.
      Precision oncology in advanced cancer patients improves overall survival with lower weekly healthcare costs.
      ,
      • Steuten L.
      • Goulart B.
      • Meropol N.J.
      • Pritchard D.
      • Ramsey S.D.
      Cost-effectiveness of multigene panel sequencing for patients with advanced non-small-cell lung cancer.
      ,
      • Farnaes L.
      • Hildreth A.
      • Sweeney N.M.
      • et al.
      Rapid whole-genome sequencing decreases infant morbidity and cost of hospitalization.
      ,
      • Schwartz M.L.B.
      • McCormick C.Z.
      • Lazzeri A.L.
      • et al.
      A model for genome-first care: returning secondary genomic findings to participants and their healthcare providers in a large research cohort.
      Nevertheless, we were unable to find publicly available evidence that these study results have informed positive or negative coverage decisions to date, perhaps because of their relatively recent formation. Similarly, there are no published examples of coverage with evidence development for NGS, although this strategy is frequently described as a valuable option for payers and manufacturers when there is uncertainty regarding clinical utility.
      Table 2Illustrative US-based RWD networks including NGS.
      Cancer relatedPopulation based
      • ASCO CancerLinq
      • Flatiron Clinico-Genomic Database
      • Foundation Medicine Precision Medicine Exchange Consortium
      • ORIEN M2 Gen
      • Syapse Learning Health Network
      • Information Exchange and Data Transformation (INFORMED)
      • Tempus
      • iKnowMed
      • Cota Healthcare and Horizon BCBS of New Jersey
      • American Association for Cancer Research’s Genomics Evidence Neoplasia Information Exchange (GENIE)
      • All of Us Research Program
        NGS testing currently in planning phase.
      • Million Veterans Program
      • MyCode (Geisinger)
      • Healthy Nevada Project (Renown Institute)
      • Electronic Medical Records and Genomics Network (NHGRI)
      • Healthcare Resources and Services Administration (HRSA) Regional Genetics Network
      • Michigan DHHS Public Health Genomics
      • Alabama Genomic Health Initiative (AGHI)
      NGS indicates next-generation sequencing; RWD, real-world data.
      NGS testing currently in planning phase.
      A residual data barrier to developing RWE for NGS is the fact that genomic data are not represented in a structured format in the electronic health record (EHR), and test billing codes are too nonspecific to be able to rely solely on claims data.
      • Phillips K.A.
      • Deverka P.A.
      • Hooker G.W.
      • et al.
      Genetic test availability and spending: where are we now? Where are we going?.
      ,
      • Denny J.C.
      Chapter 13: Mining electronic health records in the genomics era.
      To facilitate the use of RWD and overcome the need for manual curation, artificial intelligence–based methods such as natural language processing, machine learning, and deep learning are increasingly being applied to process and analyze unstructured data from the EHR and patient-generated data.
      • Boussios C.
      Gliklich R Artificial intelligence for real-world evidence.
      Critical to the success of these methods will be transparency regarding data sources and analytic methods so that payers will understand and trust the results. In addition, systematic analyses of NGS implementation efforts such as the IGNITE (Implementing Genomics in Pratice) network demonstrate that sustainability drivers include infrastructure (EHR, CDS, laboratories, manufacturers, community), evidence of clinical effectiveness, economic measures, workforce and workflow impact, provider and patient education, regulatory/legal updates specific to NGS, and stakeholder engagement in research. After a priority-setting exercise, the top 3 sustainability constructs were provider education, availability of genomic-focused CDS/EHR tools, and reimbursement.
      • Levy K.D.
      • Blake K.
      • Fletcher-Hoppe C.
      • et al.
      Opportunities to implement a sustainable genomic medicine program: lessons learned from the IGNITE Network.
      Addressing these constructs is necessary to ensure the quality and availability of RWD and RWE.
      Finally, there are several policy developments that are also endorsing greater use of RWE by stakeholders that influence payers. In 2017, the Center for Devices and Radiologic Health and the Center for Biologics Evaluation and Research at the FDA issued a guidance document describing the use of RWE to support regulatory decision making for medical devices.
      Use of real-world evidence to support regulatory decision-making for medical devices: guidance for industry and food and drug administration staff.
      The goal is to incentivize the creation of a system for characterizing, aggregating, and analyzing data from all uses of medical devices so that innovative and accurate tests are made available to patients as efficiently as possible. Related efforts include FDA guidance on the use of public human genetic variant databases to support claims of clinical validity for genomic-based IVD and considerations for the development of evidence of analytic validity for NGS-based IVDs for suspected germline diseases.
      • O’Neill T.
      • Miksad R.
      • Miller D.
      • et al.
      ISPOR, the FDA, and the evolving regulatory science of medical device products.
      ,
      US Food and Drug Adminstration
      Use of public human genetic variant databases to support clinical validity for genetic and genomic-based in vitro diagnostics.
      ,
      U. S. Food and Drug Administration
      Considerations for design, development, and analytical validation of next generation sequencing (NGS)—based in vitro diagnostics (IVDs) intended to aid in the diagnosis of suspected germline diseases.
      The FDA has also helped to establish the Medical Device Innovation Consortium (MDIC) in 2012, a public-private partnership focused solely on advancing medical device regulatory science. The MDIC engages a wide variety of stakeholders, including representatives of the FDA, National Institutes of Health, CMS, industry, nonprofits, and patient organizations to improve medical technology development and review processes. The group recently convened FDA, industry, and payer representatives to build a framework to help IVD manufacturers develop credible evidence of analytical and clinical validity and clinical utility. The section on clinical utility specifically describes the use of RWD as a source of evidence that can be used to support regulatory and reimbursement decision-making.
      Medical Device Innovation Consortium
      Developing clinical evidence for regulatory and coverage assessments in in vitro diagnostics (IVDs).
      The FDA has also provided funding to establish the The National Evaluation System for health Technology Coordinating Center (https://nestcc.org/about/faqs/), a coordinating center for a voluntary network of data partners that act as a national evaluation system for devices. This network emphasizes the use of RWE in the evaluation of its test cases, which include an IVD test panel for lung cancer focused on developing evidence of clinical utility.
      • Fleurence R.L.
      • Shuren J.
      Advances in the use of real-world evidence for medical devices: an update from the National Evaluation System for Health Technology.
      ,
      NEST Coordinating Center
      Test-Cases.
      Although all of these efforts are primarily focused on improving regulatory and clinical decision making about device access and safety, the relevance to payer decision making is that RWE studies that meet robust methodological standards in the regulatory science arena should also prove useful to payers in as much as they include information about clinical utility.
      Patient advocacy groups,
      National Health Council
      Patient perspectives on real-world evidence: a roundtable to gather views, needs, and recommendations.
      professional societies,
      • Reinke T.
      Real-world evidence faces some real-world challenges.
      and nonprofit health and research agencies
      • Forsythe L.P.
      • Szydlowski V.
      • Murad M.H.
      • et al.
      A systematic review of approaches for engaging patients for research on rare diseases.
      ,
      • Ray T.
      BCBS “Evidence Street” group enabling engagement with MDx players.
      have all become engaged in developing more comprehensive and accurate RWD to support decision makers, including patients and families. The expectation is that over time, patients will become key drivers of use of RWE by payers, particularly as there is greater availability of patient-generated data as part of clinical care.
      The scoping review revealed a number of evidence gaps for use of RWE in coverage decision-making for NGS. First, most studies are in oncology, PGx, and diagnosis of suspected genetic disorders in the perinatal or pediatric period, rather than the full spectrum of clinical genomic applications. Second, a wide variety of methods are used to conduct RWE studies, and we could find no examples in which investigators cited adherence to methodologic guidance documents or best practices for RWE studies. Third, published analyses of payer coverage policies do not distinguish RWE from RCT data when evaluating the relationship between clinical studies and coverage determinations. Finally, it is difficult to describe how RWE specifically can inform payer decision making based on publicly available data without an in-depth review of coverage policies and their evolution over time.

      Potential Solutions

      We developed potential solutions (Table 3) for overcoming the challenges limiting payer uptake of RWE in the setting of NGS based on our team’s experience and findings from the scoping review. The suggestions spanned a spectrum, from the importance of following RWE methodological best practices, to building transparency and relevance of study methods, to collaborating with existing expert groups addressing issues of data standardization, data sharing, and assessment of value.
      Table 3Potential solutions for advancing use of RWE.
      Area of focusChallengesOpportunitiesPotential solutions
      • I.
        Relevance and rigor of RWE for payers
      • A.
        Payers view many RWE studies as lacking relevance and timeliness for coverage or utilization management decisions.
      • a.
        Payers are willing to engage with test developers and researchers in study planning activities but need to know how to discern clinically valid tests among avalanche of new NGS tests.
      • 1.
        Require test developers to show evidence of analytic and clinical validity, as well as clinical flowchart of how new NGS test is hypothesized to affect health outcomes.
      • 2.
        Develop transparent engagement processes to ensure payer information needs are reflected in study design decisions.
      • B.
        RWE study methods not perceived as sufficiently rigorous and transparent by payers.
      • b.
        Multiple publicly available best practices/recommendations for designing, conducting, and reporting RWE studies.
      • c.
        Several groups have published RWE evaluation tools specifically for payers.
      • 3.
        Convene multistakeholder panels to tailor existing best practices and evaluation tools to NGS.
      • II.
        Incentives for RWE development
      • A.
        Limited number of studies outside of oncology, PGx, and perinatal/pediatrics examples of RWE directly affecting coverage decisions for NGS.
      • a.
        Numerous groups building data infrastructure to support RWE studies, particularly in oncology but also in population-based data networks and learning healthcare systems.
      • 4.
        Encourage industry, federal, and nonprofit funding to conduct and publish RWE studies to demonstrate the relative benefits and harms of NGS testing for patient and consumer subgroups in existing data networks.
      • B.
        RWE to demonstrate clinical utility of NGS is undersupplied by the market.
      • b.
        FDA is supporting the use of RWE to make regulatory decisions for in vitro diagnostics.
      • c.
        Federal efforts to require data standardization and interoperability such as HITECH and value-based payments such as MACRA
        Health Information Technology for Economic and Clinical Health, Medicare Access and CHIP Reauthorization Act of 2015.
        should also facilitate access to RWD.
      • d.
        There are a growing number of learning healthcare systems focused on genomics and publicly subsidized data networks that can reduce research study costs.
      • 5.
        Support FDA efforts such as MDIC by providing use cases to demonstrate how RWE can inform regulatory and payer decision making for NGS.
      • 6.
        Include genomic data in meaningful use requirements.
      • 7.
        Conduct RWE studies in networks such as Intermountain, Geisinger, Innovation Health, Sanford, University of Vermont, and PCORnet.
      • III.
        Educational needs regarding both RWE and NGS
      • A.
        Some payers lack experience and expertise to use RWE study results in their local context.
      • B.
        Some payers lack knowledge on how to evaluate clinical utility of NGS tests.
      • C.
        Some clinicians lack knowledge about when to order tests and how to interpret results.
      • a.
        Investments in RWE methods training and professional hiring are growing rapidly in both public and private sectors.
      • b.
        Industry, nonprofit, and governmental organizations are investing in genomics literacy and NGS education.
      • 8.
        Provide training in observational study methods and pragmatic clinical trials for payers and other stakeholders on NGS.
      • 9.
        Encourage both just-in-time training based on adult learning principles and in-person training at professional and industry meetings.
      • IV.
        Standards for testing and reporting NGS data
      • A.
        Results of NGS tests are complex, and laboratory procedures for test validation and reporting lack transparency.
      • B.
        The same test performed by different labs can produce different results.
      • a.
        Federal and private sector groups (eg, professional societies such as CAP, AMP, ACMG
        College of American Pathology, Association for Molecular Pathology, American College of Genetics and Genomics.
        ) are addressing these problems.
      • b.
        Data sharing is required for NIH grants including genomics and promoted by nonprofits such as Global Alliance for Genomics and Health.
      • 10.
        Labs that follow published guidelines or consensus recommendation statements for NGS assay and bioinformatics pipeline validation could receive higher reimbursement rates to incentivize data sharing.
      • 11.
        Support ClinGen efforts to define the clinical relevance of genes and variants for use in precision medicine and research.
      • 12.
        Support federal efforts to standardize procedures for NGS reanalysis and variant reinterpretation and require clinical labs to have policies and protocols in place to support reporting any reclassifications that may affect clinical management.
      • V.
        Standards for genomic data representation in EHR
      • A.
        Genomic data integration in the EHR is not a top priority for major vendors, yet solutions require their leadership.
      • B.
        Genetic data are not aggregated and stored in a structured manner accessible to researchers and clinicians; EHRs lack comprehensive clinical data and patient-generated data.
      • a.
        Innovative NGS companies and federally funded networks such as eMERGE are developing standardized protocols for genomic data integration across multiple EHR systems.
      • b.
        Nonprofits have developed data exchange standards such as FHIR (Fast Healthcare Interoperability Resources) that better support CDS and unify how genomic variant data are accessed.
      • c.
        Artificial intelligence (ie, natural language processing, machine learning, deep learning)–based methods reduce the need for manual curation and enable integration with other digital data sources.
      • 13.
        Evaluate clinic-genomic interoperable applications using an implementation science framework.
      • 14.
        Recognize that most payers have limited understanding of these methods and will require educational efforts to trust and apply RWE study results developed with cutting-edge methods.
      • VI.
        Standards for NGS evidence review by payers
      • A.
        Payer evidence requirements for NGS coverage decision making not clearly communicated to or understood by test developers.
      • a.
        FDA-CMS parallel review process and MolDx program (CED) is an opportunity to develop necessary evidence for federal payers.
      • b.
        Stakeholders worked through nonprofits such as Center for Medical Technology Policy to recommend coverage for NGS tumor profiling based on number of genes.
      • 15.
        Encourage public and private payer support of coverage with evidence development for tests with promising evidence of clinical utility.
      • 16.
        Convene a multistakeholder group including payers to define methodological requirements for demonstration of clinical utility for specific clinical contexts (eg, PGx, rare disease, cancer, etc).
      • VII.
        Partnerships
      • A.
        No single stakeholder has the resources to develop clinical utility data independently.
      • a.
        Numerous examples of both public and private NGS implementation and data-sharing partnerships, including learning healthcare systems partnering with industry.
      • 17.
        Engage stakeholders in research question priority setting and study design.
      • 18.
        Conduct clinical utility studies in these settings and publish results.
      • VIII.
        Role of RWD in support of OBCs for NGS
      • A.
        OBC implementation is limited by data barriers and lack of outcome measures.
      • B.
        Most OBCs are not publicly disclosed, so there is no shared learning.
      • a.
        Payers are demonstrating increased receptivity to, and use of, OBCs for new and/or expensive interventions.
      • b.
        Precompetitive collaboration to develop an OBC implementation framework has been identified as a critical unmet need.
      • c.
        Companies are beginning to share results of NGS-related OBCs.
      • 19.
        Support risk-based market access and inform evidence claims for NGS by developing an NGS-specific framework for evaluating and implementing OBCs.
      • 20.
        Work with stakeholders to develop NGS-specific outcome measures.
      • IX.
        Narrow definition of how value of NGS is measured
      • A.
        RWE necessary for value-based frameworks and payment models, but the patient and other perspectives (personal utility, reduction in uncertainty) are often missing.
      • a.
        Numerous multistakeholder groups have developed approaches for customizing value assessments for payers, patients, and other decision makers.
      • b.
        Growing evidence that patients may become key drivers of RWE use by payers.
      • 21.
        Build on work by ISPOR
        International Society for Pharmacoeconomics and Outcomes Research.
        and other non-profits to demonstrate how NGS aligns with expanded definition of value.
      • 22.
        Track developments in value-based payment models as critical factors influencing how RWE is being used by payers.
      • 23.
        Promote inclusion of patient-reported outcomes in value assessments.
      CDS indicates clinical decision support; CMS, Centers for Medicare & Medicaid Services; EHR, electronic health record; FDA, US Food and Drug Administration; MDIC, Medical Device Innovation Consortium; NGS, next-generation sequencing; OBC, outcomes-based contracting; PGx, pharmacogenetic testing; RWD, real-world data; RWE, real-world evidence.
      Health Information Technology for Economic and Clinical Health, Medicare Access and CHIP Reauthorization Act of 2015.
      College of American Pathology, Association for Molecular Pathology, American College of Genetics and Genomics.
      International Society for Pharmacoeconomics and Outcomes Research.
      The first 3 solutions address the need to develop processes to promote the relevance and rigor of RWE for payers. Payers must be able to trust study findings and understand how to apply the results in coverage decision-making. Efforts at payer engagement and tailoring existing RWE best practices and evaluation tools to NGS are best pursued as part of multistakeholder initiatives that are already working to advance use of RWE with payers generally. The next 4 potential solutions (4-7) target the lack of incentives for test developers to conduct RWE studies and undersupply of published research. There is an increasing number of curated data networks and integrated healthcare delivery systems focused on genomics that can facilitate robust RWE studies and initiatives supported by the federal government to encourage greater use of RWE in policy decisions, including coverage. Progress could be accelerated if stakeholders advocated for including genomic data in meaningful use requirements. The next 2 solutions (8 and 9) tackle the widespread problem that payers lack the requisite skills to review and apply RWE studies in their local context but through the lens of NGS-related decisions.
      There is widespread recognition that a lack of standards is hampering development of clinical utility; these include standards for testing and reporting NGS data, representing clinic-genomic data in the HER and payer requirements for NGS evidence review. There are 7 potential solutions (10-16) that address these interrelated gaps; however, they vary in the level of effort that will be required to advance these tactics. Until these issues are solved in a scalable way, it will be very difficult for researchers and test developers to capitalize on RWD sources. These solutions also address gaps in capturing the necessary clinical, digital, and patient-reported data to conduct RWE studies by focusing on artificial intelligence–based methods to reduce the need for manual curation.
      The need for public–private partnerships is addressed by the next 2 solutions (17 and 18), as pooling of infrastructure resources, patient populations, and expertise will be required if the clinical utility of NGS will be demonstrated for genomic conditions, many of which are relatively rare. The next 2 potential solutions (19 and 20) take on the need to overcome barriers to using RWD to support OBCs for NGS, recognizing that although all of the obstacles are not specific to NGS, the remedies should be targeted to how OBCs can be structured and evaluated for NGS specifically. Finally, the last 3 solutions (21-23) focus on the importance of including the patient and other relevant perspectives in value-based frameworks focused on NGS. It is also important to note that for most of these proposed solutions, the policy and data environments are in flux; therefore, each effort will need to be evaluated and recalibrated in response to changing trends.

      Discussion

      To our knowledge, this is the first article to directly examine payer use of RWE for NGS and what new approaches are needed. Although the topic of RWE as a valuable source of information for clinical, regulatory, and payer decision makers has been addressed by numerous authors and policy makers,
      • Jarow J.P.
      • LaVange L.
      • Woodcock J.
      Multidimensional evidence generation and FDA regulatory decision making: defining and using “real-world” data.
      ,
      Use of real-world evidence to support regulatory decision-making for medical devices: guidance for industry and food and drug administration staff.
      ,
      • Garrison Jr., L.P.
      • Neumann P.J.
      • Erickson P.
      • et al.
      Using real-world data for coverage and payment decisions: the ISPOR Real-World Data Task Force report.
      • Hampson G.
      • Towse A.
      • Dreitlein W.B.
      • et al.
      Real world evidence for coverage decisions: opportunities and challenges.
      • Hurwitz J.T.
      • Brown M.
      • Graff J.S.
      • et al.
      Is real-world evidence used in P&T monographs and therapeutic class reviews?.
      • Malone D.C.
      • Brown M.
      • Hurwitz J.T.
      • et al.
      Real-world evidence: useful in the real world of US Payer decision making? How? When? And what studies?.
      ,
      • Makady A.
      • Ham R.T.
      • de Boer A.
      • et al.
      Policies for use of real-world data in health technology assessment (HTA): a comparative study of six HTA agencies.
      ,
      • Dickson D.J.
      • Pfeifer J.D.
      Real-world data in the molecular era: finding the reality in the real world.
      ,
      National Health Council
      Patient perspectives on real-world evidence: a roundtable to gather views, needs, and recommendations.
      ,
      • Reinke T.
      Real-world evidence faces some real-world challenges.
      ,
      Center for Medicaid and Medicare Services
      CMS finalizes coverage of next generation sequencing tests, ensuring enhanced access for cancer patients.
      Guardant Health
      Guardant Health announces Medicare coverage for the Guardant360 assay in non-small cell lung cancer.
      • Ray T.
      Myriad pushing ahead with payors on GeneSight as data from large randomized study is published.
      Genomic Health
      Multiple Oncotype DX study presentations at the 2018 San Antonio Breast Cancer Symposium reinforce real-world value of the Oncotype DX Breast Recurrence Score® test in patients regardless of age or race.
      • MacLean E.
      • Cisar L.
      • Mehle K.
      • et al.
      Real-world axitinib use in the United States: a retrospective study using linked datasets.
      • Hess L.M.
      • Kern D.M.
      • Carter G.C.
      • et al.
      Real-world treatment sequences and outcomes among patients with non-small cell lung cancer (RESOUNDS) in the United States: study protocol.
      • Heng D.Y.
      • Signorovitch J.
      • Swallow E.
      • et al.
      Comparative effectiveness of second-line targeted therapies for metastatic renal cell carcinoma: a systematic review and meta-analysis of real-world observational studies.
      • Ostrovsky L.
      Implications of real-world data and pharmacoeconomics for managed care.
      we explored whether and how RWE could play a similar role for NGS specifically in the context of coverage decisions. In the scoping review, we identified numerous examples of how RWE was likely used in oncology, PGx, and pediatric or perinatal settings to bridge the existing evidence gaps for payers, as there is concurrent evidence of increasingly positive coverage decisions for NGS tests in these clinical contexts. These applications included developing empiric evidence of current test and treatment use patterns, assessing real-world cost implications, demonstrating the incremental value of testing additional genes when compared with standard-of-care single-gene tests, supplementing RCT data, creating efficiencies and greater certainty in coverage policy development, and enabling OBC.
      Nevertheless, it appears that payers still primarily rely on clinical guidelines and RCTs as opposed to RWE as the type of evidence cited as justification for their decisions. The reasons for this disconnect are multifactorial, including challenges related to RWD quality and comprehensiveness, difficulties representing genomic data in a standardized manner in the EHR, and lack of payer engagement in study development, resulting in results that are not relevant for coverage decision-making. Payers also lack familiarity with RWE study methods and continue to prefer RCT data over observational data. They also have concerns about the lack of transparency regarding data sources and analytic methods, resulting in a lack of trust, particularly regarding studies conducted by industry. Generally, payers are not sure how to use RWE within their current coverage processes and do not fully understand all the relevant questions that RWE could answer. Nevertheless, there is a confluence of policy, technology, and data infrastructure trends that are likely to enable greater use of RWE for NGS, for example, greater receptivity to RWE by the FDA, artificial intelligence–based methods to facilitate data analyses, and a proliferation of data networks built and curated to support RWD studies involving the use of NGS in people with and without known disease.
      The data and infrastructure-related barriers to conducting RWE studies for NGS are being addressed by healthcare providers, federal research funders, and private companies. Currently, there is a growing number of learning healthcare systems focused on implementing and evaluating population-based NGS and numerous precision medicine programs focused on oncology.
      • Presley C.J.
      • Tang D.
      • Soulos P.R.
      • et al.
      Association of broad-based genomic sequencing with survival among patients with advanced non-small cell lung cancer in the community oncology setting.
      ,
      Illumina Media Relations
      Illumina and Harvard Pilgrim Partner on value-based contract.
      • Farnaes L.
      • Hildreth A.
      • Sweeney N.M.
      • et al.
      Rapid whole-genome sequencing decreases infant morbidity and cost of hospitalization.
      • Schwartz M.L.B.
      • McCormick C.Z.
      • Lazzeri A.L.
      • et al.
      A model for genome-first care: returning secondary genomic findings to participants and their healthcare providers in a large research cohort.
      • Carey D.J.
      • Fetterolf S.N.
      • Davis F.D.
      • et al.
      The Geisinger MyCode community health initiative: an electronic health record-linked biobank for precision medicine research.
      Renown Institute for Health Innovation
      Healthy Nevada Project.
      Philanthropy News Digest
      Sanford Health receives $25 million for precision medicine initiative.
      • Ray T.
      Precision lung cancer drugs on pricing evaluation group’s agenda.
      • Pecora A.L.
      • Norden A.D.
      • Hervey J.
      • et al.
      Development of a precise, clinically relevant, digital classification schema for cancer.
      Each of these organizations has a vested interest in developing the data infrastructure to support collection of high-quality RWD and to publicize the results of RWE studies. By engaging payers in the design of these studies, there is a real opportunity to provide RWE that can inform coverage decision making in an efficient and timely manner. Nevertheless, to ensure that these studies have the desired impact, researchers must follow methodologic best practices for conducting and reporting RWD studies. Fortunately, there are many existing consensus statements from expert groups to guide researcher efforts and reassure payers that the study results are valid. What is missing is agreement about which guidelines are best to follow for NGS. Similarly, existing tools for evaluating the quality and applicability of RWE need to be tailored to NGS for payers to more consistently evaluate these studies. Both efforts will require multistakeholder groups to discuss options, explicitly address conflicts of interest, and agree on a path forward for using RWE to inform payers and other decision makers about the clinical utility of NGS. Although conflicts of interest are ubiquitous, agreeing how these conflicts will be identified and managed is possible, as demonstrated by groups such as the MDIC, a public–private partnership working to promote patient access to innovative medical technologies such as NGS.
      Lack of standards is a major stumbling block for using RWE to evaluate the clinical utility of NGS. This refers not only to a lack of standards for analyzing and reporting NGS results, and a lack of standards for representing genomic data in the EHR, but also to a lack of standards for payer assessment of clinical utility data for NGS. Although individual labs or health systems may attempt to reduce variation in their NGS testing practices or NGS-related CDS tools, the definitive solutions require multistakeholder groups to ensure interoperability and efficiency as patients move from one clinical setting to another. Similarly, multistakeholder groups need to define the methodological standards for developing clinical utility evidence for NGS that will address payer requirements for coverage decision-making. Although this has been successfully done for molecular diagnostics in oncology,
      • Deverka P.
      • Messner D.A.
      • McCormack R.
      • et al.
      Generating and evaluating evidence of the clinical utility of molecular diagnostic tests in oncology.
      RWE has the potential to address the lack of evidence of clinical utility of NGS, provided that payers agree on how RWE can be used for coverage determinations. Although affirmative coverage decisions are necessary, they are not sufficient to ensure clinical uptake of NGS unless there is simultaneous attention to NGS implementation requirements.
      There are also reimbursement-focused enablers that will require access to robust RWD, such as the growing use of OBCs between manufacturers and payers to ensure realization of promised benefits of new interventions. Nevertheless, there are significant barriers to implementing OBCs, and many preliminary discussions fail to result in actual contracts. These challenges include difficulties obtaining accurate data, lack of outcome measures, concerns regarding patient data privacy, and costs of data collection. There is an opportunity for an honest broker to convene payers and manufacturers to develop a framework to assess the suitability/desirability of OBCs for NGS and also guide the negotiation and implementation processes to increase the likelihood of success. Greater transparency regarding the results of OBCs is desirable, but the proprietary nature of contract terms makes this goal unlikely.
      Nonprofit groups such as the Personalized Medicine Coalition, the Innovation and Value Initiative, and patient advocacy groups are highlighting the importance of putting the patient perspective at the center of any assessment of the value of new technologies such as NGS. Development of patient-centered outcome measures would advance these efforts and address the lack of standard outcome measures limiting RWE studies and OBCs. Because each group is engaging relevant stakeholders to develop suggestions or consensus statements for a path forward, we recommend following a similar approach for NGS-specific adaptation or evaluating the feasibility of joint efforts.
      There are several limitations to this study. There is no Medical Subject Headings term for either RWD or RWE
      • Mullins C.D.
      Evolving use of real-world evidence for devices: good for patients, good for policy makers.
      ; therefore, we may have missed relevant articles. We made several efforts to overcome this limitation by searching reference lists and asking experts to share seminal articles. We did not examine specific payer coverage policies, so our conclusions regarding the impact of RWE on payer decision making is likely an underestimate of the true impact. We also limited our evaluation to US payers and data networks. We are aware that there are comparable payer information needs and data initiatives in other countries but determined these to be out of scope for our literature and policy analyses. Future studies could characterize the RWE landscape in other countries and determine whether our recommendations could be generalized.
      This study assessed the opportunities and challenges surrounding use of RWE by payers to inform decision making regarding NGS and found a growing number of published RWE studies, particularly in oncology, PGx, and perinatal or pediatric genomic testing, that have played either a direct or supportive role given the state of published coverage analyses. We also identified many examples of policy and data network enablers designed to address the current undersupply of RWE for NGS. For RWE to become a game changer for payers, there must be multifaceted efforts to raise awareness of data and methods advances and publicly available examples of robust studies that are relevant for payer decision making. We present 20 potential solutions that, if pursued collaboratively with stakeholders over time, represent promising steps toward ensuring greater and more effective use of RWE by payers.

      Acknowledgments

      We gratefully acknowledge comments and input on earlier drafts from the Global Economics and Evaluation of Clinical Sequencing Working Group (GEECS).
      Source of financial support: This study was funded by an unrestricted consulting agreement with Illumina (no number) and grants from the National Human Genome Research Institute (U01 HG009599) and from the National Cancer Institute (R01 CA221870).

      Supplemental Material

      References

        • Dervan A.P.
        • Deverka P.A.
        • Trosman J.R.
        • et al.
        Payer decision making for next-generation sequencing-based genetic tests: insights from cell-free DNA prenatal screening.
        Genet Med. 2017; 19: 559-567
        • Freedman A.N.
        • Klabunde C.N.
        • Wiant K.
        • et al.
        Use of next-generation sequencing tests to guide cancer treatment: results from a nationally representative survey of oncologists in the United States.
        JCO Precis Oncol. 2018; : 1-13
        • Splinter K.
        • Adams D.R.
        • Bacino C.A.
        • et al.
        Effect of genetic diagnosis on patients with previously undiagnosed disease.
        N Engl J Med. 2018; 379: 2131-2139
        • Kalayinia S.
        • Goodarzynejad H.
        • Maleki M.
        • et al.
        Next generation sequencing applications for cardiovascular disease.
        Ann Med. 2018; 50: 91-109
        • Burwell Kort v.
        Civil No. 1:14-cv-01519 (APM).
        United States District Court for the District of Columbia, Washington, DC2016
        • Medical Device Innovation Consortium
        Developing clinical evidence for regulatory and coverage assessments in in vitro diagnostics (IVDs).
        • Jarow J.P.
        • LaVange L.
        • Woodcock J.
        Multidimensional evidence generation and FDA regulatory decision making: defining and using “real-world” data.
        JAMA. 2017; 318: 703-704
      1. Use of real-world evidence to support regulatory decision-making for medical devices: guidance for industry and food and drug administration staff.
        https://www.fda.gov/media/99447/download
        Date accessed: May 29, 2019
        • O’Neill T.
        • Miksad R.
        • Miller D.
        • et al.
        ISPOR, the FDA, and the evolving regulatory science of medical device products.
        Value Health. 2019; 22: 754-761
      2. What are in vitro diagnostic tests, and how are they regulated?.
        • Ray T.
        FDA stepping up actions against PGx testing, forcing some labs to stop reporting drug information.
        • Garrison Jr., L.P.
        • Neumann P.J.
        • Erickson P.
        • et al.
        Using real-world data for coverage and payment decisions: the ISPOR Real-World Data Task Force report.
        Value Health. 2007; 10: 326-335
        • Hampson G.
        • Towse A.
        • Dreitlein W.B.
        • et al.
        Real world evidence for coverage decisions: opportunities and challenges.
        • Hurwitz J.T.
        • Brown M.
        • Graff J.S.
        • et al.
        Is real-world evidence used in P&T monographs and therapeutic class reviews?.
        J Manag Care Spec Pharm. 2017; 23: 613-620
        • Malone D.C.
        • Brown M.
        • Hurwitz J.T.
        • et al.
        Real-world evidence: useful in the real world of US Payer decision making? How? When? And what studies?.
        Value Health. 2018; 21: 326-333
        • Fryback D.G.
        • Thornbury J.R.
        The efficacy of diagnostic imaging.
        Med Decis Making. 1991; 11: 88-94
      3. Methods Guide for Medical Test Reviews. AHRQ Publication No. 12-EC017.
      4. Rational Integration of Clinical Sequencing (RISE), Vanderbilt University Medical Center.
        http://grantome.com/grant/NIH/R01-HG009694-01
        Date accessed: November 19, 2019
        • Carlson J.J.
        • Chen S.
        • Garrison Jr., L.P.
        Performance-based risk-sharing arrangements: an updated international review.
        Pharmacoeconomics. 2017; 35: 1063-1072
        • Centers for Medicare and Medicaid Services
        Coverage with evidence development.
        • Mahendraratnam N.
        • Sorenson C.
        • Richardson E.
        • et al.
        Value-based arrangements may be more prevalent than assumed.
        Am J Manag Care. 2019; 25: 70-76
      5. Shore C. Gee A.W. Kahn B. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. The National Academies Press, Washington, DC2019
        • Conley R.B.
        • Dickson D.
        • Zenklusen J.C.
        • et al.
        Core clinical data elements for cancer genomic repositories: a multi-stakeholder consensus.
        Cell. 2017; 171: 982-986
        • Green Park Collaborative
        Initial medical policy and model coverage guidelines for clinical next generation sequencing in oncology: report and recommendations.
        • Anderson S.
        • Allen P.
        • Peckham S.
        • et al.
        Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services.
        Health Res Policy Syst. 2008; 6: 7
        • Arksey H.
        • O’Malley L.
        Scoping studies: towards a methodological framework.
        Int J Soc Res Methodol. 2008; 8: 19-32
        • National Human Genome Research Institute
        Accomplishments in genomic medicine.
        • Pearson S.D.
        • Dreitlein W.B.
        • Towse A.
        • et al.
        A framework to guide the optimal development and use of real-world evidence for drug coverage and formulary decisions.
        J Comp Eff Res. 2018; 7: 1145-1152
        • Green Park Collaborative
        RWE decoder framework: a practical tool for assessing relevance and rigor of real world evidence.
        • Makady A.
        • Ham R.T.
        • de Boer A.
        • et al.
        Policies for use of real-world data in health technology assessment (HTA): a comparative study of six HTA agencies.
        Value Health. 2017; 20: 520-532
      6. Velentgas P. Dreyer N.A. Nourjah P. Developing a Protocol for Observational Comparative Effectiveness Research: A User’s Guide. Agency for Healthcare Research and Quality, Rockville, MD2013
      7. ENCePP checklist for study protocols (revision 3).
        • US Food and Drug Administration
        Good pharmacovigilance practices and pharmacoepidemiologic assessment.
        • Berger M.L.
        • Mamdani M.
        • Atkins D.
        • et al.
        Good research practices for comparative effectiveness research: defining, reporting and interpreting nonrandomized studies of treatment effects using secondary data sources: the ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report—part I.
        Value Health. 2009; 12: 1044-1052
        • Cox E.
        • Martin B.C.
        • Van Staa T.
        • et al.
        Good research practices for comparative effectiveness research: approaches to mitigate bias and confounding in the design of nonrandomized studies of treatment effects using secondary data sources: the International Society for Pharmacoeconomics and Outcomes Research Good Research Practices for Retrospective Database Analysis Task Force Report—part II.
        Value Health. 2009; 12: 1053-1061
        • Johnson M.L.
        • Crown W.
        • Martin B.C.
        • et al.
        Good research practices for comparative effectiveness research: analytic methods to improve causal inference from nonrandomized studies of treatment effects using secondary data sources: the ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report—part III.
        Value Health. 2009; 12: 1062-1073
        • Dreyer N.A.
        • Bryant A.
        • Velentgas P.
        The GRACE checklist: a validated assessment tool for high quality observational studies of comparative effectiveness.
        J Manag Care Spec Pharm. 2016; 22: 1107-1113
        • Patient Centered Outcomes Research Insitute (PCORI)
        methodology standards.
        • Liu P.
        • Meng L.
        • Normand E.A.
        • et al.
        Reanalysis of clinical exome sequencing data.
        N Engl J Med. 2019; 380: 2478-2480
        • Dickson D.J.
        • Pfeifer J.D.
        Real-world data in the molecular era: finding the reality in the real world.
        Clin Pharmacol Ther. 2016; 99: 186-197
        • Rosenman M.B.
        • Decker B.
        • Levy K.D.
        • et al.
        Lessons learned when introducing pharmacogenomic panel testing into clinical practice.
        Value Health. 2017; 20: 54-59
        • Strande N.T.
        • Riggs E.R.
        • Buchanan A.H.
        • et al.
        Evaluating the clinical validity of gene-disease associations: an evidence-based framework developed by the clinical genome resource.
        Am J Hum Genet. 2017; 100: 895-906
        • US Food and Drug Adminstration
        Use of public human genetic variant databases to support clinical validity for genetic and genomic-based in vitro diagnostics.
        https://www.fda.gov/media/99200/download
        Date accessed: May 29, 2019
        • Velcheti V.
        • Patwardhan P.D.
        • Liu F.X.
        • et al.
        Real-world PD-L1 testing and distribution of PD-L1 tumor expression by immunohistochemistry assay type among patients with metastatic non-small cell lung cancer in the United States.
        PLoS One. 2018; 13: e0206370
        • Presley C.J.
        • Tang D.
        • Soulos P.R.
        • et al.
        Association of broad-based genomic sequencing with survival among patients with advanced non-small cell lung cancer in the community oncology setting.
        JAMA. 2018; 320: 469-477
        • Singal G.
        • Miller P.G.
        • Agarwala V.
        • et al.
        Association of patient characteristics and tumor genomics with clinical outcomes among patients with non-small cell lung cancer using a clinicogenomic database.
        JAMA. 2019; 321: 1391-1399
        • Haslem D.S.
        • Chakravarty I.
        • Fulde G.
        • et al.
        Precision oncology in advanced cancer patients improves overall survival with lower weekly healthcare costs.
        Oncotarget. 2018; 9: 12316-12322
        • Steuten L.
        • Goulart B.
        • Meropol N.J.
        • Pritchard D.
        • Ramsey S.D.
        Cost-effectiveness of multigene panel sequencing for patients with advanced non-small-cell lung cancer.
        JCO Clin Cancer Inform. 2019; 3: 1-10
        • Signorovitch J.
        • Zhou Z.
        • Ryan J.
        • et al.
        Budget impact analysis of comprehensive genomic profiling in patients with advanced non-small cell lung cancer.
        J Med Econ. 2019; 22: 140-150
      8. CMS and commercial insures extend coverage of Oncomine Dx Target Test to more than 160 million U.S. lives.
        • Lu C.Y.
        • Loomer S.
        • Ceccarelli R.
        • et al.
        Insurance coverage policies for pharmacogenomic and multi-gene testing for cancer.
        J Pers Med. 2018; 8
        • Trosman J.R.
        • Weldon C.B.
        • Gradishar W.J.
        • et al.
        From the past to the present: insurer coverage frameworks for next-generation tumor sequencing.
        Value Health. 2018; 21: 1062-1068
        • Dotson W.D.
        • Douglas M.P.
        • Kolor K.
        • et al.
        Prioritizing genomic applications for action by level of evidence: a horizon-scanning method.
        Clin Pharmacol Ther. 2014; 95: 394-402
        • Centers for Disease Control and Prevention
        Introducing the CDC tier-classified database.
        • American Society of Clinical Oncology
        Genetic testing coverage & reimbursement.
        • Hendricks-Sturrup R.M.
        • Lu C.Y.
        Understanding implementation challenges to genetic testing for familial hypercholesterolemia in the United States.
        J Pers Med. 2019; 9: 9
        • Bielinski S.J.
        • Olson J.E.
        • Pathak J.
        • et al.
        Preemptive genotyping for personalized medicine: design of the right drug, right dose, right time-using genomic data to individualize treatment protocol.
        Mayo Clin Proc. 2014; 89: 25-33
        • Shi Y.
        • Graves J.A.
        • Garbett S.P.
        • et al.
        A decision-theoretic approach to panel-based, preemptive genotyping.
        MDM Policy Pract. 2019; 42381468319864337
        • Martin J.
        • Williams A.K.
        • Klein M.D.
        • et al.
        Frequency and clinical outcomes of CYP2C19 genotype-guided escalation and de-escalation of antiplatelet therapy in a real-world clinical setting.
        Genet Med. 2020; 22: 160-169
        • Volpi S.
        • Bult C.J.
        • Chisholm R.L.
        • et al.
        Research directions in the clinical implementation of pharmacogenomics: an overview of US programs and projects.
        Clin Pharmacol Ther. 2018; 103: 778-786
        • Manzi S.F.
        • Fusaro V.A.
        • Chadwick L.
        • et al.
        Creating a scalable clinical pharmacogenomics service with automated interpretation and medical record result integration: experience from a pediatric tertiary care facility.
        J Am Med Inform Assoc. 2017; 24: 74-80
        • Verbelen M.
        • Weale M.E.
        • Lewis C.M.
        Cost-effectiveness of pharmacogenetic-guided treatment: are we there yet?.
        Pharmacogenomics J. 2017; 17: 395-402
        • Roden D.M.
        • McLeod H.L.
        • Relling M.V.
        • et al.
        Pharmacogenomics.
        Lancet. 2019; 394: 521-532
        • Saunders C.J.
        • Miller N.A.
        • Soden S.E.
        • et al.
        Rapid whole-genome sequencing for genetic disease diagnosis in neonatal intensive care units.
        Sci Transl Med. 2012; 4: 154ra35
        • Monroe G.R.
        • Frederix G.W.
        • Savelberg S.M.
        • et al.
        Effectiveness of whole-exome sequencing and costs of the traditional diagnostic trajectory in children with intellectual disability.
        Genet Med. 2016; 18: 949-956
        • Lazaridis K.N.
        • Schahl K.A.
        • Cousin M.A.
        • et al.
        Outcome of whole exome sequencing for diagnostic odyssey cases of an individualized medicine clinic: the Mayo Clinic experience.
        Mayo Clin Proc. 2016; 91: 297-307
        • Clark M.M.
        • Stark Z.
        • Farnaes L.
        • et al.
        Meta-analysis of the diagnostic and clinical utility of genome and exome sequencing and chromosomal microarray in children with suspected genetic diseases.
        NPJ Genom Med. 2018; 3: 16
        • Clark M.M.
        • Hildreth A.
        • Batalov S.
        • et al.
        Diagnosis of genetic diseases in seriously ill children by rapid whole-genome sequencing and automated phenotyping and interpretation.
        Sci Transl Med. 2019; 11
        • Trosman J.R.
        • Weldon C.B.
        • Slavotinek A.
        • et al.
        Perspectives of US private payers on insurance coverage for pediatric and prenatal exome sequencing: results of a study from the Program in Prenatal and Pediatric Genomic Sequencing (P3EGS).
        Genet Med. 2020; 22: 283-291
        • Douglas M.P.
        • Parker S.L.
        • Trosman J.R.
        • et al.
        Private payer coverage policies for exome sequencing (ES) in pediatric patients: trends over time and analysis of evidence cited.
        Genet Med. 2019; 21: 152-160
        • Chambers J.D.
        • Saret C.J.
        • Anderson J.E.
        • et al.
        Examining evidence in U.S. payer coverage policies for multi-gene panels and sequencing tests.
        Int J Technol Assess Health Care. 2017; 33: 534-540
        • Duhig A.M.
        • Saha S.
        • Smith S.
        • et al.
        The current status of outcomes-based contracting for manufacturers and payers: an AMCP membership survey.
        J Manag Care Spec Pharm. 2018; 24: 410-415
        • Centers for Medicare and Medicaid Services
        Medicare Access and CHIP Reauthorization Act of 2015 (MACRA).
        • McQueen R.
        • Schroeder B.
        • Wright G.
        • et al.
        Evaluating coverage expansion for NIPT through a perfomrance-based risk sharing agreement.
        ISPOR Europe Copenhagen, Denmark2019
        • Illumina Media Relations
        Illumina and Harvard Pilgrim Partner on value-based contract.
        • Farnaes L.
        • Hildreth A.
        • Sweeney N.M.
        • et al.
        Rapid whole-genome sequencing decreases infant morbidity and cost of hospitalization.
        NPJ Genom Med. 2018; 3: 10
        • Schwartz M.L.B.
        • McCormick C.Z.
        • Lazzeri A.L.
        • et al.
        A model for genome-first care: returning secondary genomic findings to participants and their healthcare providers in a large research cohort.
        Am J Hum Genet. 2018; 103: 328-337
        • Carey D.J.
        • Fetterolf S.N.
        • Davis F.D.
        • et al.
        The Geisinger MyCode community health initiative: an electronic health record-linked biobank for precision medicine research.
        Genet Med. 2016; 18: 906-913
        • Renown Institute for Health Innovation
        Healthy Nevada Project.
        https://healthynv.org/about/
        Date accessed: May 29, 2019
        • Philanthropy News Digest
        Sanford Health receives $25 million for precision medicine initiative.
        • Ray T.
        Precision lung cancer drugs on pricing evaluation group’s agenda.
        • Pecora A.L.
        • Norden A.D.
        • Hervey J.
        • et al.
        Development of a precise, clinically relevant, digital classification schema for cancer.
        JCO Clin Cancer Inform. 2018; 2: 1-10
        • Tempus Collaborations
        (Accessed May 2019, 2019)
        • The White House
        Fact sheet: President Obama’s precision medicine initiative.
        • CancerLinQ LLC
        Collaborating with the American College of Medical Genetics and Genomics.
        (Accessed May 2019, 2019)
        • Foundation Medicine
        Foundation Medicine Launches Precision Medicine Exchange Consortium™ (PMEC) to advance the integration of molecular information in clinical oncology and accelerate adoption of precision care.
        (Accessed May 2019)
      9. M2GEN.
        https://m2gen.com/
        Date accessed: May 29, 2019
      10. Information Exchange and Data Transformation (INFORMED).
      11. VITAL Innovation.
        https://vitalinnovation.com/
        Date accessed: May 29, 2019
        • SetYale School of Medicine
        Digging deep into data.
        • McKesson
        iKnowMed: Oncology practice EHR system.
        • American Association for Cancer Research
        American Association for Cancer Research’s Genomics Evidence Neoplasia Information Exchange (GENIE).
        (Accessed November, 2019)
        • Health Resources and Services Administration
        National Coordinating Center for the Regional Genetics Network.
        https://nccrcg.org/
        Date accessed: November 20, 2019
        • Michigan Department of Health and Human Services
        • Alabama Genomic Health Initiative (AGHI)
        • Syapse Learning Health Network
        • Phillips K.A.
        • Deverka P.A.
        • Hooker G.W.
        • et al.
        Genetic test availability and spending: where are we now? Where are we going?.
        Health Aff (Millwood). 2018; 37: 710-716
        • Denny J.C.
        Chapter 13: Mining electronic health records in the genomics era.
        PLoS Comput Biol. 2012; 8: e1002823
        • Boussios C.
        Gliklich R Artificial intelligence for real-world evidence.
        • Levy K.D.
        • Blake K.
        • Fletcher-Hoppe C.
        • et al.
        Opportunities to implement a sustainable genomic medicine program: lessons learned from the IGNITE Network.
        Genet Med. 2019; 21: 743-747
        • U. S. Food and Drug Administration
        Considerations for design, development, and analytical validation of next generation sequencing (NGS)—based in vitro diagnostics (IVDs) intended to aid in the diagnosis of suspected germline diseases.
        https://www.fda.gov/media/99208/download
        Date accessed: May 29, 2019
        • Fleurence R.L.
        • Shuren J.
        Advances in the use of real-world evidence for medical devices: an update from the National Evaluation System for Health Technology.
        Clin Pharmacol Ther. 2019; 106: 30-33
        • NEST Coordinating Center
        Test-Cases.
        https://nestcc.org/test-cases/
        Date accessed: November 19, 2019
        • National Health Council
        Patient perspectives on real-world evidence: a roundtable to gather views, needs, and recommendations.
        • Reinke T.
        Real-world evidence faces some real-world challenges.
        • Forsythe L.P.
        • Szydlowski V.
        • Murad M.H.
        • et al.
        A systematic review of approaches for engaging patients for research on rare diseases.
        J Gen Intern Med. 2014; 29: S788-S800
        • Ray T.
        BCBS “Evidence Street” group enabling engagement with MDx players.
        • Center for Medicaid and Medicare Services
        CMS finalizes coverage of next generation sequencing tests, ensuring enhanced access for cancer patients.
        • Guardant Health
        Guardant Health announces Medicare coverage for the Guardant360 assay in non-small cell lung cancer.
        • Ray T.
        Myriad pushing ahead with payors on GeneSight as data from large randomized study is published.
        • Genomic Health
        Multiple Oncotype DX study presentations at the 2018 San Antonio Breast Cancer Symposium reinforce real-world value of the Oncotype DX Breast Recurrence Score® test in patients regardless of age or race.
        • MacLean E.
        • Cisar L.
        • Mehle K.
        • et al.
        Real-world axitinib use in the United States: a retrospective study using linked datasets.
        J Manag Care Spec Pharm. 2016; 22: 723-732u
        • Hess L.M.
        • Kern D.M.
        • Carter G.C.
        • et al.
        Real-world treatment sequences and outcomes among patients with non-small cell lung cancer (RESOUNDS) in the United States: study protocol.
        JMIR Res Protoc. 2017; 6: e195
        • Heng D.Y.
        • Signorovitch J.
        • Swallow E.
        • et al.
        Comparative effectiveness of second-line targeted therapies for metastatic renal cell carcinoma: a systematic review and meta-analysis of real-world observational studies.
        PLoS One. 2014; 9: e114264
        • Davis B.
        • Morgan J.
        • Shah S.
        Getting real with real-world evidence.
        • Ostrovsky L.
        Implications of real-world data and pharmacoeconomics for managed care.
        Am Health Drug Benefits. 2016; 9: 151-155
        • Deverka P.
        • Messner D.A.
        • McCormack R.
        • et al.
        Generating and evaluating evidence of the clinical utility of molecular diagnostic tests in oncology.
        Genet Med. 2016; 18: 780-787
        • Mullins C.D.
        Evolving use of real-world evidence for devices: good for patients, good for policy makers.
        Value Health. 2019; 22: 751-753