Advertisement
Themed Section: Assesing the Value of Next-Generation Sequencing| Volume 21, ISSUE 9, P1054-1061, September 2018

Download started.

Ok

Cost Analyses of Genomic Sequencing: Lessons Learned from the MedSeq Project

  • Kurt D. Christensen
    Correspondence
    Address correspondence to: Kurt D. Christensen, Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School, EC Alumnae Building, Suite 301,41 Avenue Louis Pasteur, Boston, MA 02115, USA.
    Affiliations
    Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School, Boston, MA, USA
    Search for articles by this author
  • Kathryn A. Phillips
    Affiliations
    Department of Clinical Pharmacy, Center for Translational and Policy Research on Personalized Medicine (TRANSPERS), University of California San Francisco, San Francisco, CA, USA

    Philip R. Lee Institute for Health Policy and Helen Diller Family Comprehensive Cancer Center, University of California San Francisco, San Francisco, CA, USA
    Search for articles by this author
  • Robert C. Green
    Affiliations
    Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School, Boston, MA, USA

    Broad Institute of MIT and Harvard, Cambridge, MA, USA

    Partners HealthCare Personalized Medicine, Boston, MA, USA
    Search for articles by this author
  • Dmitry Dukhovny
    Affiliations
    Department of Pediatrics, Oregon Health & Science University, Portland, OR, USA
    Search for articles by this author
Open ArchivePublished:August 13, 2018DOI:https://doi.org/10.1016/j.jval.2018.06.013

      Abstract

      Objective

      To summarize lessons learned while analyzing the costs of integrating whole genome sequencing into the care of cardiology and primary care patients in the MedSeq Project by conducting the first randomized controlled trial of whole genome sequencing in general and specialty medicine.

      Methods

      Case study that describes key methodological and data challenges that were encountered or are likely to emerge in future work, describes the pros and cons of approaches considered by the study team, and summarizes the solutions that were implemented.

      Results

      Major methodological challenges included defining whole genome sequencing, structuring an appropriate comparator, measuring downstream costs, and examining clinical outcomes. Discussions about solutions addressed conceptual and practical issues that arose because of definitions and analyses around the cost of genomic sequencing in trial-based studies.

      Conclusions

      The MedSeq Project provides an instructive example of how to conduct a cost analysis of whole genome sequencing that feasibly incorporates best practices while being sensitive to the varied applications and diversity of results it may produce. Findings provide guidance for researchers to consider when conducting or analyzing economic analyses of whole genome sequencing and other next-generation sequencing tests, particularly regarding costs.

      Keywords

      Introduction

      Advancements in next-generation sequencing (NGS) have made it feasible to integrate whole genome sequencing (WGS) into patient care at a population level, and may streamline the practice of medicine [
      • Armstrong K.
      Can genomics bend the cost curve?.
      ]. Currently, genomic testing begins by testing symptomatic patients with panels of genes in which mutations are most likely to explain the disorder. If no causal variants are identified, physicians may order additional tests to examine other candidate genes, a process that can continue until options are exhausted. WGS allows all candidate genes to be examined at once, including regulatory domains and genes that are not typically tested. In addition, WGS information can influence medication choices, inform reproductive decisions, facilitate targeted prevention, and more [
      • Green R.C.
      • Rehm H.L.
      • Kohane I.S.
      Clinical genome sequencing.
      ,
      • McCarthy J.J.
      • McLeod H.L.
      • Ginsburg G.S.
      Genomic medicine: a decade of successes, challenges, and opportunities.
      ]. Moreover, it can be re-queried for diagnostic and treatment purposes as new needs arise. The ability of WGS to provide information with lifelong utility provides a compelling rationale for its use at a population level.
      Nevertheless, many commentators also fear the cost and budgetary implications of integrating WGS into regular medical practice [
      • Hegde M.
      • Bale S.
      • Bayrak-Toydemir P.
      • et al.
      Reporting incidental findings in genomic scale clinical sequencing—a clinical laboratory perspective: a report of the Association for Molecular Pathology.
      ,
      • Prosser L.A.
      • Grosse S.D.
      • Kemper A.R.
      • et al.
      Decision analysis, economic evaluation, and newborn screening: challenges and opportunities.
      ,
      • Douglas M.P.
      • Ladabaum U.
      • Pletcher M.J.
      • et al.
      Economic evidence on identifying clinically actionable findings with whole-genome sequencing: a scoping review.
      ,
      • Khoury M.J.
      • Coates R.J.
      • Fennell M.L.
      • et al.
      Multilevel research and the challenges of implementing genomic medicine.
      ]. It can be many times more expensive than targeted tests and typically has lower sensitivity for identifying certain types of variants than other types of genomic tests [
      • Gonzaga-Jauregui C.
      • Lupski J.R.
      • Gibbs R.A.
      Human genome sequencing in health and disease.
      ]. WGS also tends to identify more variants of uncertain significance that can require additional clinical workup, and WGS can provide secondary findings that are unrelated to the test indications but may motivate follow-up testing and long-term screening.
      To understand the impact of integrating WGS into the everyday care of sick and healthy populations, we conducted the MedSeq Project, the first randomized controlled trial of WGS in cardiology and primary care settings [
      • Vassy J.
      • Lautenbach D.
      • McLaughlin H.
      • et al.
      The MedSeq Project: a randomized trial of integrating whole genome sequencing into clinical medicine.
      ]. In addition to describing the molecular yield and clinical impact of disclosure [
      • Vassy J.L.
      • Christensen K.D.
      • Schonman E.F.
      • et al.
      The impact of whole genome sequencing on the primary care and outcomes of healthy adult patients: a pilot randomized trial.
      ,
      • Cirino A.L.
      • Lakdawala N.K.
      • McDonough B.
      • et al.
      A comparison of whole genome sequencing to multigene panel testing in hypertrophic cardiomyopathy patients.
      ], we used microcosting and gross costing methods to report the short-term costs of integrating WGS into clinical practice, including its impact on short-term health care utilization and other health sector costs [

      Christensen KD, Vassy JL, Phillips KA, et al. Short term costs of integrating whole genome sequencing into primary care and cardiology settings: a pilot randomized trial. Genet Med. In press.

      ]. Findings showed an incremental cost of approximately $5000 to integrate WGS into patient care in 2015, no noticeable impact on downstream health care utilization for a 6-month time horizon, and less than $200 per patient to disclose secondary findings.
      The purpose of this article is to discuss key methodological challenges that arose in that cost analysis because of the unique characteristics of WGS. The lessons we summarize and the solutions we adopted provide practical guidance and points to consider as researchers and policymakers develop and interpret cost analyses of WGS and NGS tests more broadly.

      Methods

      The methods of the MedSeq Project have been described in detail previously, including the rationale and design of the study [
      • Vassy J.
      • Lautenbach D.
      • McLaughlin H.
      • et al.
      The MedSeq Project: a randomized trial of integrating whole genome sequencing into clinical medicine.
      ], the approach to WGS, variant analysis and reporting [
      • McLaughlin H.M.
      • Ceyhan-Birsoy O.
      • Christensen K.D.
      • et al.
      A systematic approach to the reporting of medically relevant findings from whole genome sequencing.
      ,
      • Vassy J.L.
      • McLaughlin H.L.
      • MacRae C.A.
      • et al.
      A one-page summary report of genome sequencing for the healthy adult.
      ], and the rationale and design of the cost analyses [
      • Christensen K.D.
      • Dukhovny D.
      • Siebert U.
      • et al.
      Assessing the costs and cost-effectiveness of genomic sequencing.
      ]. Key terms that are used in this case report are summarized in Table 1. Briefly, the MedSeq Project was a set of parallel randomized pilot trials to examine two archetypal scenarios for integrating WGS into clinical care. The first, disease-specific genomic medicine, used WGS to identify molecular causes for disease in patients with family histories or symptoms suggestive of a genetic disorder. To examine this scenario, we enrolled cardiologists and patients with diagnoses of hypertrophic or dilated cardiomyopathy. The second scenario, general genomic medicine, used WGS to screen for genetic disorders to enhance disease prevention and to improve medical and personal decision making. To examine this scenario, we enrolled primary care physicians and ostensibly healthy patients.
      Table 1Key terms used in this case report
      Whole genome sequencing (WGS)A laboratory process that is used to determine nearly all of the approximately 3 billion nucleotides of an individual’s complete DNA sequence, including noncoding sequence. Here, we include bioinformatics analyses to identify health-relevant information, and reporting of these findings to health care providers and their patients.
      VariantAn alteration in the most common DNA nucleotide sequence. The term variant can be used to describe an alteration that may be benign, pathogenic, or of unknown significance.
      CoverageThe number of times a nucleotide is read during sequencing.
      Singleton testingA genetic testing strategy that examines the DNA of a patient alone.
      Trio testingA genetic testing strategy that examines the DNA a patient along with the DNA of parent, usually to identify variants that are present in a sick patient that are absent in healthy parents.
      DeletionA type of genetic change that involves the absence of a segment of DNA. It may be as small as a single base but can vary significantly in size.
      InsertionA type of genetic change that involves the addition of a segment of DNA that can be as small as a single base.
      TranslocationA type of chromosomal abnormality in which a chromosome breaks and a portion of it reattaches to a different chromosomal location.
      Sanger sequencingA low-throughput method used to determine a portion of a patient’s nucleotide sequence. This method is well-validated, and has high sensitivity and specificity for identifying variants.
      Structural variantA type of large genetic change (i.e., approximately 1000 base pairs or larger in size). This change can include an inversion (a segment of a chromosome that breaks off and reattaches in the reverse direction), a translocation, an insertion, or a deletion.
      Single-nucleotide polymorphismA type of variant present in at least 1% of the population where a single nucleotide in the genome sequence is altered.
      Definitions were adapted from the NCI Dictionary of Genetics Terms

      National Cancer Institute. NCI Dictionary of Genetics Terms. Available at: www.cancer.gov/publications/dictionaries/genetics-dictionary [Accessed April 19, 2018].

      and from published literature
      • Freeman J.L.
      • Perry G.H.
      • Feuk L.
      • et al.
      Copy number variation: new insights in genome diversity.
      . Terms are presented in the order which they appear in the case report.
      After consenting to the study and completing a baseline survey, patient participants were randomized to meet with their providers and review health information that included or omitted WGS. Participants were then followed for 6 months. Data relevant to the cost analyses were collected from surveys of providers and patients, medical records and administrative data.
      Key challenges that are summarized here were identified by consensus of the investigators who led the cost analyses. We focused on decisions that had a large impact on our analyses and would be applicable to future cost analyses of WGS and other NGS tests. We also highlight issues for which recent developments may change future analyses.

      Results

      We identified three key challenges in conducting cost analyses of WGS: defining the test, developing appropriate comparators, and assessing downstream costs. We additionally describe challenges to collecting data about clinical outcomes.

      Challenge 1: Defining Whole Genome Sequencing

      The first challenge we addressed was to define how we would implement WGS. Decisions about whether to conduct singleton testing or test multiple family members, what sequencing system (“platform”) to use, and the minimum coverage that WGS should achieve can have a large impact on costs and molecular yields [
      • Green R.C.
      • Rehm H.L.
      • Kohane I.S.
      Clinical genome sequencing.
      ,
      • Gonzaga-Jauregui C.
      • Lupski J.R.
      • Gibbs R.A.
      Human genome sequencing in health and disease.
      ]. Professional groups such as the American College of Medical Genetics and Genomics (ACMG), the Association for Molecular Pathology, and the College of Medical Pathologists have been developing standards for WGS [
      • Rehm H.L.
      • Bale S.J.
      • Bayrak-Toydemir P.
      • et al.
      ACMG clinical laboratory standards for next-generation sequencing.
      ,
      • Richards S.
      • Aziz N.
      • Bale S.
      • et al.
      Standards and guidelines for the interpretation of sequence variants: a joint consensus recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology.
      ,
      • Aziz N.
      • Zhao Q.
      • Bry L.
      • et al.
      College of American Pathologists’ laboratory standards for next-generation sequencing clinical tests.
      ], and the optimal approach depends on the purpose of testing, time frame for results, patient characteristics, and more. Even when a consensus approach exists, many aspects of WGS still vary from setting to setting. Here, we focus on decisions about conducting WGS that had a significant impact on costs and molecular yields in the MedSeq Project but might be made differently in future work. These decisions are summarized in Table 2.
      Table 2Clinical and cost implications of different WGS approaches
      ConsiderationMedSeq approachAlternativesClinical implicationsCosts of alternatives (relative to MedSeq)
      Sequencing approachSingletonTrioAbility to identify de novo variants, interpretation of reported variants

      Partners HealthCare. Exome and Genome Sequencing FAQ. Available at: personalizedmedicine.partners.org/laboratory-for-molecular-medicine/faq/exome-genome-sequencing.aspx [Accessed March 1, 2018].

      Sequencing platformIllumina HiSeq 2000NumerousTurnaround time and error rates
      • Levy S.E.
      • Myers R.M.
      Advancements in next-generation sequencing.
      ,
      • Xuan J.
      • Yu Y.
      • Qing T.
      • et al.
      Next-generation sequencing in the clinic: promises and challenges.
      1/4× to 2×, depending on platform and usage
      • Xuan J.
      • Yu Y.
      • Qing T.
      • et al.
      Next-generation sequencing in the clinic: promises and challenges.
      ,
      • Plothner M.
      • Frank M.
      • von der Schulenburg J.G.
      Cost analysis of whole genome sequencing in German clinical practice.
      Mean coverage30×100×+Turnaround time and error rates
      • Rehm H.L.
      • Bale S.J.
      • Bayrak-Toydemir P.
      • et al.
      ACMG clinical laboratory standards for next-generation sequencing.
      • van Nimwegen K.J.
      • van Soest R.A.
      • Veltman J.A.
      • et al.
      Is the $1000 Genome as near as we think? A cost analysis of next-generation sequencing.
      Confirmation approachSanger sequencingNo confirmationError ratesSavings of $250–$625

      Christensen KD, Vassy JL, Phillips KA, et al. Short term costs of integrating whole genome sequencing into primary care and cardiology settings: a pilot randomized trial. Genet Med. In press.

      ,
      • Strom S.P.
      • Lee H.
      • Das K.
      • et al.
      Assessing the necessity of confirmatory testing for exome-sequencing results in a clinical molecular diagnostic laboratory.
      Types of WGS findings reportedMonogenic disease risks in 4600+ genes, carrier status, PGx, cardio-metabolic risks, blood type/antigen predictionsPrimary findings only, different combinations of secondary findings, fewer genesPrimary and secondary prevention, medical decision makingUp to $200/patient

      Christensen KD, Vassy JL, Phillips KA, et al. Short term costs of integrating whole genome sequencing into primary care and cardiology settings: a pilot randomized trial. Genet Med. In press.

      Classifications of secondary findings reportedPathogenic, likely pathogenic, and VUS: favor pathogenicPathogenic only, pathogenic + likely pathogenicClinical validity
      • Richards S.
      • Aziz N.
      • Bale S.
      • et al.
      Standards and guidelines for the interpretation of sequence variants: a joint consensus recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology.
      Up to $200/patient

      Christensen KD, Vassy JL, Phillips KA, et al. Short term costs of integrating whole genome sequencing into primary care and cardiology settings: a pilot randomized trial. Genet Med. In press.

      Only considerations that would affect costs by $200 are presented.
      WGS, whole genome sequencing.
      Many decisions about how to conduct WGS in the MedSeq Project were influenced by practical considerations in addition to conceptual ones. We used a singleton testing approach rather than trio testing to maximize the number of different families we could provide WGS, and because identification of de novo variants—which is optimized with trio sequencing—would have limited utility in our cohort of healthy primary care patients. Sequencing was conducted using the Illumina HiSeq 2000 platform because in 2012, when the MedSeq Project launched, it was one of few platforms with well-established quality metrics. Newer sequencing platforms have emerged, however, with varying advantages and disadvantages with regard to speed, accuracy calling specific types of variants (e.g., substitutions vs. insertions), and costs [
      • Levy S.E.
      • Myers R.M.
      Advancements in next-generation sequencing.
      ,
      • Xuan J.
      • Yu Y.
      • Qing T.
      • et al.
      Next-generation sequencing in the clinic: promises and challenges.
      ,
      • van Nimwegen K.J.
      • van Soest R.A.
      • Veltman J.A.
      • et al.
      Is the $1000 Genome as near as we think? A cost analysis of next-generation sequencing.
      ,
      • Plothner M.
      • Frank M.
      • von der Schulenburg J.G.
      Cost analysis of whole genome sequencing in German clinical practice.
      ]. We opted for at least 30× mean coverage to conform to ACMG standards for WGS, recognizing that greater coverage would improve our ability to identify mosaicisms (i.e., mutations that are present in only a fraction of cells) and variants such as deletions, insertions, and translocations [
      • Rehm H.L.
      • Bale S.J.
      • Bayrak-Toydemir P.
      • et al.
      ACMG clinical laboratory standards for next-generation sequencing.
      ,
      • Fang H.
      • Wu Y.
      • Narzisi G.
      • et al.
      Reducing INDEL calling errors in whole genome and exome sequencing data.
      ,
      • Nielsen R.
      • Paul J.S.
      • Albrechtsen A.
      • et al.
      Genotype and SNP calling from next-generation sequencing data.
      ], although at greater expense.
      Another important consideration that will affect future cost analyses of WGS is whether and how to confirm sequencing findings. Current standards are to confirm variants with additional testing before reporting [
      • Rehm H.L.
      • Bale S.J.
      • Bayrak-Toydemir P.
      • et al.
      ACMG clinical laboratory standards for next-generation sequencing.
      ,
      • Richards S.
      • Aziz N.
      • Bale S.
      • et al.
      Standards and guidelines for the interpretation of sequence variants: a joint consensus recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology.
      ,
      • Aziz N.
      • Zhao Q.
      • Bry L.
      • et al.
      College of American Pathologists’ laboratory standards for next-generation sequencing clinical tests.
      ]. In the MedSeq Project—as is typical in many current clinical testing protocols—we confirmed results using Sanger sequencing, adding over $600 to the average per-patient costs of WGS [

      Christensen KD, Vassy JL, Phillips KA, et al. Short term costs of integrating whole genome sequencing into primary care and cardiology settings: a pilot randomized trial. Genet Med. In press.

      ]. Nevertheless, standards may change in the future given growing evidence about the high accuracy of NGS to detect nucleotide substitutions, although insertions, deletions, and larger structural variants are likely to remain problematic [
      • Baudhuin L.M.
      • Lagerstedt S.A.
      • Klee E.W.
      • et al.
      Confirming variants in next-generation sequencing panel testing by Sanger sequencing.
      ].
      A final consideration about the conduct of WGS is the scope of information that will be reported. Existing guidelines recommend that laboratories query at least 59 genes for known or expected pathogenic variants in actionable conditions whenever sequencing is initiated, regardless of clinical indication [
      • Kalia S.S.
      • Adelman K.
      • Bale S.J.
      • et al.
      Recommendations for reporting of secondary findings in clinical exome and genome sequencing, 2016 update (ACMG SF v2.0): a policy statement of the American College of Medical Genetics and Genomics.
      ,
      • Green R.C.
      • Berg J.S.
      • Grody W.W.
      • et al.
      ACMG recommendations for reporting of incidental findings in clinical exome and genome sequencing.
      ]. Many laboratories will also offer to provide other secondary findings in additional genes, carrier status for autosomal recessive conditions or pharmacogenomic results that could influence drug metabolism and side effects. In the MedSeq Project, WGS reports also included findings classified as pathogenic, likely pathogenic, or of uncertain significance where evidence favored pathogenicity in any of more than 4600 genes associated with monogenic diseases. In addition to the above, we reported risk predictions for eight cardiometabolic traits [
      • Kong S.W.
      • Lee I.H.
      • Leshchiner I.
      • et al.
      Summarizing polygenic risks for complex diseases in a clinical whole-genome report.
      ], and blood group and antigen predictions [
      • Lane W.J.
      • Westhoff C.M.
      • Uy J.M.
      • et al.
      Comprehensive red blood cell and platelet antigen prediction from whole genome sequencing: proof of principle.
      ].
      To account for alternative approaches to conducting WGS in the MedSeq Project and to provide insight about the future, we conducted sensitivity analyses that considered WGS costs as low as $500 and as high as $10,000 (approximately 10% to 200% of the costs, per our analyses, of $5225). In addition, we conducted scenario analyses in which we examined different reporting criteria, such as omitting specific types of results (e.g., carrier status, risk predictions for cardiometabolic traits) or reporting only secondary findings classified as pathogenic. We were able to conduct these scenario analyses by microcosting laboratory and clinical tasks and by having physicians link the follow-up services they ordered to specific WGS findings.

      Challenge 2: Developing an Appropriate Comparator

      At the earliest stages of the MedSeq Project, the study team extensively discussed what intervention to provide to patients randomized to the control arm. Many of these discussions centered on conceptual questions about how to characterize the study’s use of WGS. As a diagnostic tool, we used WGS to identify molecular causes for cardiology patients’ cardiomyopathy diagnoses. At the same time, we used WGS as a screening tool for monogenic disease risks and carrier status, information about cardiometabolic traits, pharmacogenomic information, and red blood cell and platelet antigen might inform targeted prevention. The combination of WGS results with potential benefits for diagnostic, screening, prevention, and decision-making purposes made developing an appropriate comparator challenging.
      Some of the options that the study team considered are summarized in Table 3. We briefly considered comparing WGS against no intervention, given the lack of comparators with similar capabilities and study goals that were not focused on specific clinical outcomes. Nevertheless, a design with an extra clinical encounter to disclosure WGS results had the potential to bias downstream health care utilization and costs in ways that were unrelated to WGS. We also discussed using a wellness visit where physicians would screen for disease and review preventive health recommendations [
      • Colburn J.L.
      • Nothelle S.
      The Medicare Annual Wellness Visit.
      ], but felt that an intervention with a greater focus on genetic disorders would be a better comparator to WGS. A third strategy we considered was providing a genomics-focused comparator using panel testing or by profiling single-nucleotide polymorphisms (SNP). For example, an expansive SNP analysis, similar to those used previously by direct-to-consumer genetic testing companies and studies like the Multiplex Initiative, could provide similar types of information as our WGS analyses, including estimates of disease risk, carrier status, and pharmacogenomic information [
      • McBride C.M.
      • Alford S.H.
      • Reid R.J.
      • et al.
      Characteristics of users of online personalized genomic risk assessments: implications for physician-patient interactions.
      ,
      National Research Council
      Institute of Medicine Roundtable on Translating Genomic-Based Research for Health.
      ]. Nevertheless, these approaches have raised questions about the predictive power of the underlying algorithms and the potential that results may misinform medical decisions [
      • Kalf R.R.J.
      • Bakker R.
      • Janssens A.C.J.W.
      Predictive ability of direct-to-consumer pharmacogenetic testing: when is lack of evidence really lack of evidence?.
      ,
      • Kalf R.R.J.
      • Mihaescu R.
      • Kundu S.
      • et al.
      Variations in predicted risks in personal genome testing for common complex diseases.
      ].
      Table 3Interventions considered for the control arm in the MedSeq Project
      ComparatorProsCons
      No interventionControl arm would be unbiased by an “artificial” interventionMay inflate costs in the WGS arm in ways that were unrelated to genomics by introducing an extra clinic encounter
      Well care visitBalances number of clinical encounters in randomization arms with interventions focused on screening and preventionComparator would lack a genomics focus
      Panel-based genomic testingProvides insight about the incremental benefits and costs of WGS compared to other genomic testing approachesComparator would represent standard of care only among symptomatic patients
      Family history reviewStandard of care, yet frequently neglected; can identify potential genetic disordersCardiology patients already had a thorough FH review, FH reporting often biased
      Advantages and disadvantages focus on the implications for analyzing costs and clinical benefits.
      FH, family history; WGS, whole genome sequencing.
      Ultimately, we developed a comparator that focused on reviewing patients’ family histories of disease. Family history review is frequently neglected, despite being standard of care, and it can identify patterns of disease suggestive of an inherited genetic risk factor [
      • Pyeritz R.E.
      The family history: the first genetic test, and still useful after all those years?.
      ]. Patient participants completed a modified version of the Surgeon General’s “My Family Health Portrait” tool [
      • Facio F.M.
      • Feero W.G.
      • Linn A.
      • et al.
      Validation of My Family Health Portrait for six common heritable conditions.
      ] at enrollment, and physicians reviewed findings from these reports with patients in both randomization arms during disclosure sessions. Furthermore, cardiology patients had to have prior or concurrent cardiomyopathy panel tests as a condition of enrollment, and cardiologists reviewed findings from panel testing in both randomization arms during MedSeq Project disclosure sessions. By implementing cohort-specific interventions based on a genomics-focused standard of care, we were able to create comparators and assess the incremental costs of WGS relative to an idealized standard of care, even if it might not mimic typical clinical practice.

      Challenge 3: Assessing Postdisclosure Costs

      There are substantial concerns about the potential impact of WGS on downstream health sector costs [
      • Phillips K.A.
      • Pletcher M.J.
      • Ladabaum U.
      Is the “$1000 genome” really $1000? Understanding the full benefits and costs of genomic sequencing.
      ,
      • Caulfield T.
      • Evans J.
      • McGuire A.
      • et al.
      Reflections on the cost of “low-cost” whole genome sequencing: framing the health policy debate.
      ]. Assessing the health care services and associated costs that WGS may generate represented a major challenge in the MedSeq Project cost analyses, given the diversity of conditions and information that we disclosed to patients.
      WGS increases the likelihood of identifying variants of uncertain significance that can prompt follow-up testing to verify their clinical importance [
      • Phillips K.A.
      • Pletcher M.J.
      • Ladabaum U.
      Is the “$1000 genome” really $1000? Understanding the full benefits and costs of genomic sequencing.
      ]. Also, additional clinical workup or ongoing screening may be motivated by secondary findings. Only 1% to 3% of people are thought to have disease-causing variants in any of the 59 genes recommended by the ACMG for secondary findings disclosure [
      • Green R.C.
      • Goddard K.A.B.
      • Jarvik G.P.
      • et al.
      Clinical Sequencing Exploratory Research Consortium: accelerating evidence-based practice of genomic medicine.
      ,
      • Dorschner Michael O.
      • Amendola Laura M.
      • Turner Emily H.
      • et al.
      Actionable, pathogenic incidental findings in 1,000 participants exomes.
      ,
      • Natarajan P.
      • Gold N.B.
      • Bick A.G.
      • et al.
      Aggregate penetrance of genomic variants for actionable disorders in European and African Americans.
      ]; but the percentage can be much higher under other reporting criteria. The expansive approach implemented in the MedSeq Project identified monogenic disease risks unrelated to test indications in 16% of cardiology patients and 26% of primary care patients [
      • Vassy J.L.
      • Christensen K.D.
      • Schonman E.F.
      • et al.
      The impact of whole genome sequencing on the primary care and outcomes of healthy adult patients: a pilot randomized trial.
      ,

      Christensen KD, Vassy JL, Phillips KA, et al. Short term costs of integrating whole genome sequencing into primary care and cardiology settings: a pilot randomized trial. Genet Med. In press.

      ]. In addition, nearly all sequenced patients were identified with carrier status for at least one autosomal recessive conditions, and all sequenced patients received pharmacogenomic information, cardiometabolic risk predictions, and blood group/antigen predictions by design [
      • Vassy J.
      • Lautenbach D.
      • McLaughlin H.
      • et al.
      The MedSeq Project: a randomized trial of integrating whole genome sequencing into clinical medicine.
      ]. Recommendations for cost analyses in clinical trials are to collect utilization data about “relevant health care services,” regardless of why they were ordered [

      Glick HA. Economic Evaluation in Clinical Trials (2nd ed.). New York: Oxford University Press, 2014.

      ,
      • Ramsey S.D.
      • Willke R.J.
      • Glick H.
      • et al.
      Cost-effectiveness analysis alongside clinical trials II—an ISPOR Good Research Practices Task Force report.
      ], but the diversity of WGS information provided in the MedSeq Project precluded our ability to use disease-specific health care utilization instruments.
      Our solution was to implement multiple strategies, as summarized in Table 4. First, we asked physicians to complete checklists after disclosure sessions where they documented recommendations for follow-up care that were prompted by family history or WGS findings. Admittedly, this strategy captured only those services that were initiated by a physician shortly after WGS and/or family history disclosure, and missed services that may have been ordered by specialists after referrals. It may have also missed services initiated by participants rather than physicians. Nevertheless, our approach allowed us to report findings about “immediately attributable” costs with acceptable precision.
      Table 4Approaches to assessing the downstream impact of family history reviews and WGS on health care utilization in the MedSeq Project
      StrategyAdvantagesDisadvantagesNotes
      Document physician recommendations during disclosure sessionsClearly identifies services that were initiated as a result of MedSeq Project disclosuresOnly identifies services initiated during disclosure services, cannot account for potential savings.Implemented, with follow-through confirmed through review of medical records
      Identify services by reviewing medical recordsEnsures services occurred, and were often accompanied by billing codesTime-intensive, difficult to link to study disclosure sessions, misses out-of-system servicesImplemented to identify all services, with no attempt to link to disclosure
      Survey patients about follow-up health care servicesEasy to analyze, identifies care that may have occurred outside the Partners HealthCare systemSubject to reporting biases, challenging for patients to completeImplemented, but not linked to disclosure
      Identify potential follow-up through expert reviewEnsures family history and WGS reports are interpreted correctlyArtificialImplemented with a focus on monogenic findings
      WGS, whole genome sequencing.
      Second, we documented all medical services that occurred in the 6 months after disclosure sessions by reviewing participants’ medical records and administrative data, which included billing codes. This time-intensive process missed services that occurred outside the Partners HealthCare system and introduced great variability into cost estimates by including services that were unlikely to be related to MedSeq Project reports. On the other hand, this encompassing approach established the methodological foundation for follow-up studies in which the lack of precision in cost estimates may be overcome by enrolling larger numbers of patients.
      Third, we asked patients to report services they received. We administered survey items about medical testing that were adapted from the Behavioral Risk Factor Surveillance System [
      Centers for Disease Control and Prevention
      Behavioral Risk Factor Surveillance System Survey Questionnaire.
      ] and the Impact of Personal Genomics Study [
      • Carere D.
      • Couper M.
      • Crawford S.
      • et al.
      Design, methods, and participant characteristics of the Impact of Personal Genomics (PGen) Study, a prospective cohort study of direct-to-consumer personal genomic testing customers.
      ], as well as consensus health care utilization measures developed for the Clinical Sequencing Exploratory Research Consortium [
      • Gray S.W.
      • Martins Y.
      • Feuerman L.Z.
      • et al.
      Social and behavioral research in genomic sequencing: approaches from the Clinical Sequencing Exploratory Research Consortium Outcomes and Measures Working Group.
      ], into surveys that patients completed 6 weeks and 6 months after MedSeq Project disclosure sessions. The inclusion of these items provided our study team with insight about services that occurred outside the Partners HealthCare System.
      Finally, we included an approach in which genetic counselors and medical geneticists identified health care services associated with a comprehensive work-up of monogenic disease risks. These services were based on guidelines provided in repositories such as GeneReviews and Online Mendelian Inheritance in Man (OMIM), as well as a review of published literature [
      • Pagon R.A.
      • Tarczy-Hornoch P.
      • Baskin P.K.
      • et al.
      GeneTests-GeneClinics: genetic testing information for a growing audience.
      ,
      • Hamosh A.
      • Scott A.F.
      • Amberger J.
      • et al.
      Online Mendelian Inheritance in Man (OMIM).
      ]. Analyses were included to provide a potential “high side” of costs associated with monogenic conditions, but did not provide insight about services that might be motivated by the information provided in WGS reports about carrier status, drug metabolism, or risk for cardiometabolic conditions.

      Challenge 4: Documenting the Benefits and Harms of Disclosure

      Our published cost analysis provides crucial insight about the short-term impact of WGS, but we recognize that more important questions remain about whether WGS provided benefits that justify any additional spending. Patients outcomes that we considered examining in the MedSeq Project are summarized in Table 5. We previously published monogenic disease risk findings, carrier status findings, pharmacogenomic findings, and cardiometabolic risk predictions descriptively [
      • Vassy J.L.
      • Christensen K.D.
      • Schonman E.F.
      • et al.
      The impact of whole genome sequencing on the primary care and outcomes of healthy adult patients: a pilot randomized trial.
      ,
      • Cirino A.L.
      • Lakdawala N.K.
      • McDonough B.
      • et al.
      A comparison of whole genome sequencing to multigene panel testing in hypertrophic cardiomyopathy patients.
      ,

      Christensen KD, Vassy JL, Phillips KA, et al. Short term costs of integrating whole genome sequencing into primary care and cardiology settings: a pilot randomized trial. Genet Med. In press.

      ]. Nevertheless, metrics that focused on genomic variants did not have an analogue in the comparison arm.
      Table 5Clinical outcomes that were considered in the MedSeq Project
      OutcomeProsCons
      Molecular yieldCommon metric for success of genetic testsFindings have unclear clinical validity; does not indicate changes to or improvements in care; not relevant to family history analyses
      Diagnostic yield (new or revised diagnoses)Applicable to both randomization armsTime to diagnosis may be beyond the study time frame
      Intermediate (e.g., changes in lab scores) and clinical outcomes (e.g., cardiac events)Applicable to both randomization arms, indicate a change in clinical care with likely benefits to healthTypically requires a follow-up clinical encounter that is not mandated in the study
      General health-related quality of lifeApplicable to both randomization arms, facilitates cross-study comparisons, permits estimation of health utilityInsensitive to short-term change
      Metrics that will be examined for future reporting may have greater relevance to both randomization arms. Surveys asked patients whether the information they received led to new diagnoses. We are also using approaches developed in the Electronic Medical Records in Genomics (eMERGE) Network to examine intermediate and clinical outcomes associated with the cardiometabolic risk predictions we reported [
      • Kho A.N.
      • Pacheco J.A.
      • Peissig P.L.
      • et al.
      Electronic medical records for genetic research: results of the eMERGE consortium.
      ]. We will examine whether changes to laboratory scores, such as cholesterol levels, or cardiac events varied by randomization status, although data will only be available on a subset of patients who had clinical encounters and testing after their MedSeq Project disclosure sessions.
      Lastly, we included measured health-related quality of life to inform future economic analyses. We administered the SF-12v2 at disclosure and 6 months after disclosure [
      • Ware J.E.
      • Kosinki M.
      • Keller S.D.
      A 12-item short-form health survey: construction of scales and preliminary tests of reliability and validity.
      ], opting for a generic measure rather than a disease-specific one given the diversity of information and conditions that may be addressed on WGS reports. Moreover, well-established algorithms exist to convert SF-12v2 scores into SF-6D health states and utility values [
      • Brazier J.E.
      • Roberts J.
      The estimation of a preference-based measure of health from the SF-12.
      ]. Six months is likely far too short a time horizon to observe an impact of WGS on quality-adjusted life years, but the data we collected will provide a foundation that can inform long-term follow-up studies.

      Discussion

      Present applications of WGS tend to focus on prenatal, pediatric, oncology, and rare disease contexts, and a growing number of studies have examined the economic impact of genomic sequencing for diagnostic and treatment purposes. The true potential of WGS may be realized in the everyday practice of medicine, however, where it can be additionally used for screening and prevention [
      • Delaney S.K.
      • Hultner M.L.
      • Jacob H.J.
      • et al.
      Toward clinical genomics in everyday medicine: perspectives and recommendations.
      ]. Widespread clinical use of WGS in everyday patient care will only occur if it can provide value. Lessons from the MedSeq Project highlight the unique challenges in assessing the costs of WGS and analyses of other NGS tests.
      The lessons we summarize in this report provide practical guidance not only for researchers who are conducting their own cost analyses, but also for scientists and policymakers who are interpreting the findings from other work. Aggregated estimates of the costs of sequencing, such as those provided by the National Human Genome Research Institute [

      National Human Genome Research Institute. The cost of sequencing a human genome. Available at: www.genome.gov/sequencingcosts [Accessed November 30, 2016].

      ], may be inappropriate if they do not account for differences in testing choices that may be influenced by the clinical context and patient characteristics, as well as regulatory requirements from oversight organizations such as the FDA [
      • Messner D.A.
      • Koay P.
      • Al Naber J.
      • et al.
      Barriers to clinical adoption of next-generation sequencing: a policy Delphi panel’s solutions.
      ]. We were able to develop valid estimates of WGS costs in our study by implementing a work-intensive microcosting approach, but more importantly, researchers of the cost impact of NGS tests will need to be careful to incorporate the full variability of approaches that may be appropriate to their clinical contexts and patient populations.
      Our work also demonstrates how we were able to create a suitable comparator by developing an intervention focused on procedures that are standard of care, but often neglected: family history review. We identified a number of patients with unaddressed family histories of heart disease, dementia, and cancer using this approach, information that motivated providers to make referrals and initiate clinical follow-up [

      Christensen KD, Vassy JL, Phillips KA, et al. Short term costs of integrating whole genome sequencing into primary care and cardiology settings: a pilot randomized trial. Genet Med. In press.

      ]. Nevertheless, it should be noted that numerous approaches exist for collecting and analyzing family history information, which is often inaccurate [
      • Wilson B.J.
      • Qureshi N.
      • Santaguida P.
      • et al.
      Systematic review: family history in risk assessment for common diseases.
      ,
      • Ozanne E.M.
      • O’Connell A.
      • Bouzan C.
      • et al.
      Bias in the reporting of family history: implications for clinical care.
      ]. Tools such as MeTree and Family HealthWare have emerged that not only collect more accurate data, but also generate targeted prevention messages [
      • Orlando L.A.
      • Buchanan A.H.
      • Hahn S.E.
      • et al.
      Development and validation of a primary care-based family health history and decision support program (MeTree).
      ,
      • O’Neill S.M.
      • Rubinstein W.S.
      • Wang C.
      • et al.
      Familial risk for common diseases in primary care: the Family Healthware Impact Trial.
      ]. Also, SNP-based risk predictions have improved since the MedSeq Project launched [
      • Natarajan P.
      • Young R.
      • Stitziel N.O.
      • et al.
      Polygenic risk score identifies subgroup with higher burden of atherosclerosis and greater relative benefit from statin therapy in the primary prevention setting.
      ,
      • Chatterjee N.
      • Shi J.
      • Garcia-Closas M.
      Developing and evaluating polygenic risk prediction models for stratified disease prevention.
      ], and the approach may have greater acceptance in clinical settings in the future. In short, determining appropriate comparators for NGS tests will require researchers to be sensitive to developments emerging in other genomics-related tools, not just advances in NGS technologies.
      Our solution to measuring the short-term downstream costs of WGS was successful. A major benefit to our approach was its flexibility, allowing for sensitivity and scenario analyses that provided insight about different strategies for reporting unanticipated findings. Yet, our approach was time intensive, and it is unclear how well we captured patient-reported services that occurred outside of the Partners HealthCare system. These issues will be only more important to address in the future, as larger studies are conducted. Scalable, more accurate solutions to assessing the downstream cost impact of these tests would track participant expenditures across health systems through initiatives such as an All-Payer Claims Database [

      All-Payer Claims Database Council, National Association of Health Data Organizations, University of New Hampshire. All-Payer Claims Database Council. Available at: www.apcdcouncil.org [Accessed March 20, 2018].

      ].
      Finally, the difficulties our research team faced in measuring clinical outcomes demonstrate the challenges of assessing them when the NGS test being analyzed provides varied health-relevant information. Our principal approach, using a generic measure of health-related quality of life, is an appropriate approach but may lack the sensitivity in a small sample size and short time horizon to detect a clinical impact. In addition to enrolling larger samples into future trials, one promising solution may be the use of technology to widen the scope while tightening the precision of measurements. Computer adaptive tests such as PROMIS have demonstrated great potential to detect changes to general health-related quality of life while minimizing participant burdens [
      • Rose M.
      • Bjorner J.B.
      • Gandek B.
      • et al.
      The PROMIS Physical Function item bank was calibrated to a standardized metric and shown to improve measurement efficiency.
      ], and ongoing efforts work is underway to convert PROMIS scores to health utilities for economic analyses [
      • Hanmer J.
      • Feeny D.
      • Fischhoff B.
      • et al.
      The PROMIS of QALYs.
      ].

      Conclusions

      As a pilot project, the MedSeq Project cost analyses of WGS provide practical guidance about considering the full costs of NGS tests in patient care. The lessons we learned will be particularly relevant given the launch of large population-based initiatives that include sequencing, such as the Million Veteran Program [
      • Gaziano J.M.
      • Concato J.
      • Brophy M.
      • et al.
      Million Veteran Program: a mega-biobank to study genetic influences on health and disease.
      ] and the All of Us Research Program [

      U.S. Department of Health and Human Services. All of Us Research Program. Available at: www.nih.gov/research-training/allofus-research-program [Accessed March 30, 2017].

      ], and the need to understand their cost impact.

      Acknowledgments

      Additional members of the MedSeq Project team are listed in the Appendix.
      Source of financial support: This study was supported by National Institutes of Health grants U01-HG006500, K01-HG009173, and R01-HG007063

      Supplemental Materials

      References

        • Armstrong K.
        Can genomics bend the cost curve?.
        JAMA. 2012; 307: 1031-1032
        • Green R.C.
        • Rehm H.L.
        • Kohane I.S.
        Clinical genome sequencing.
        in: Ginsberg G.S. Willard H.F. Genomic and Personalized Medicine. 2nd ed. Academic Press, San Diego2013
        • McCarthy J.J.
        • McLeod H.L.
        • Ginsburg G.S.
        Genomic medicine: a decade of successes, challenges, and opportunities.
        Sci Transl Med. 2013; 5 (189sr4–89sr4)
        • Hegde M.
        • Bale S.
        • Bayrak-Toydemir P.
        • et al.
        Reporting incidental findings in genomic scale clinical sequencing—a clinical laboratory perspective: a report of the Association for Molecular Pathology.
        J Mol Diagn. 2015; 17: 107-117
        • Prosser L.A.
        • Grosse S.D.
        • Kemper A.R.
        • et al.
        Decision analysis, economic evaluation, and newborn screening: challenges and opportunities.
        Genet Med. 2012; 14: 703-712
        • Douglas M.P.
        • Ladabaum U.
        • Pletcher M.J.
        • et al.
        Economic evidence on identifying clinically actionable findings with whole-genome sequencing: a scoping review.
        Genet Med. 2016; 18: 111-116
        • Khoury M.J.
        • Coates R.J.
        • Fennell M.L.
        • et al.
        Multilevel research and the challenges of implementing genomic medicine.
        J Natl Cancer Inst Monogr. 2012; 2012: 112-120
        • Gonzaga-Jauregui C.
        • Lupski J.R.
        • Gibbs R.A.
        Human genome sequencing in health and disease.
        Annu Rev Med. 2012; 63: 35-61
        • Vassy J.
        • Lautenbach D.
        • McLaughlin H.
        • et al.
        The MedSeq Project: a randomized trial of integrating whole genome sequencing into clinical medicine.
        Trials. 2014; 15: 85
        • Vassy J.L.
        • Christensen K.D.
        • Schonman E.F.
        • et al.
        The impact of whole genome sequencing on the primary care and outcomes of healthy adult patients: a pilot randomized trial.
        Ann Intern Med. 2017; 167: 159-169
        • Cirino A.L.
        • Lakdawala N.K.
        • McDonough B.
        • et al.
        A comparison of whole genome sequencing to multigene panel testing in hypertrophic cardiomyopathy patients.
        Circ Cardiovasc Genet. 2017; 10: e001768
      1. Christensen KD, Vassy JL, Phillips KA, et al. Short term costs of integrating whole genome sequencing into primary care and cardiology settings: a pilot randomized trial. Genet Med. In press.

        • McLaughlin H.M.
        • Ceyhan-Birsoy O.
        • Christensen K.D.
        • et al.
        A systematic approach to the reporting of medically relevant findings from whole genome sequencing.
        BMC Med Genet. 2014; 15: 134
        • Vassy J.L.
        • McLaughlin H.L.
        • MacRae C.A.
        • et al.
        A one-page summary report of genome sequencing for the healthy adult.
        Public Health Genomics. 2015; 18: 123-129
        • Christensen K.D.
        • Dukhovny D.
        • Siebert U.
        • et al.
        Assessing the costs and cost-effectiveness of genomic sequencing.
        J Pers Med. 2015; 5: 470
        • Rehm H.L.
        • Bale S.J.
        • Bayrak-Toydemir P.
        • et al.
        ACMG clinical laboratory standards for next-generation sequencing.
        Genet Med. 2013; 15: 733-747
        • Richards S.
        • Aziz N.
        • Bale S.
        • et al.
        Standards and guidelines for the interpretation of sequence variants: a joint consensus recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology.
        Genet Med. 2015; 17: 405-423
        • Aziz N.
        • Zhao Q.
        • Bry L.
        • et al.
        College of American Pathologists’ laboratory standards for next-generation sequencing clinical tests.
        Arch Pathol Lab Med. 2015; 139: 481-493
        • Levy S.E.
        • Myers R.M.
        Advancements in next-generation sequencing.
        Annu Rev Genomics Hum Genet. 2016; 17: 95-115
        • Xuan J.
        • Yu Y.
        • Qing T.
        • et al.
        Next-generation sequencing in the clinic: promises and challenges.
        Cancer Lett. 2013; 340: 284-295
        • van Nimwegen K.J.
        • van Soest R.A.
        • Veltman J.A.
        • et al.
        Is the $1000 Genome as near as we think? A cost analysis of next-generation sequencing.
        Clin Chem. 2016; 62: 1458-1464
        • Plothner M.
        • Frank M.
        • von der Schulenburg J.G.
        Cost analysis of whole genome sequencing in German clinical practice.
        Eur J Health Econ. 2017; 18: 623-633
        • Fang H.
        • Wu Y.
        • Narzisi G.
        • et al.
        Reducing INDEL calling errors in whole genome and exome sequencing data.
        Genome Med. 2014; 6: 89
        • Nielsen R.
        • Paul J.S.
        • Albrechtsen A.
        • et al.
        Genotype and SNP calling from next-generation sequencing data.
        Nat Rev Genet. 2011; 12: 443-451
        • Baudhuin L.M.
        • Lagerstedt S.A.
        • Klee E.W.
        • et al.
        Confirming variants in next-generation sequencing panel testing by Sanger sequencing.
        J Mol Diagn. 2015; 17: 456-461
        • Kalia S.S.
        • Adelman K.
        • Bale S.J.
        • et al.
        Recommendations for reporting of secondary findings in clinical exome and genome sequencing, 2016 update (ACMG SF v2.0): a policy statement of the American College of Medical Genetics and Genomics.
        Genet Med. 2017; 19: 249-255
        • Green R.C.
        • Berg J.S.
        • Grody W.W.
        • et al.
        ACMG recommendations for reporting of incidental findings in clinical exome and genome sequencing.
        Genet Med. 2013; 15: 565-574
        • Kong S.W.
        • Lee I.H.
        • Leshchiner I.
        • et al.
        Summarizing polygenic risks for complex diseases in a clinical whole-genome report.
        Genet Med. 2015; 17: 536-544
        • Lane W.J.
        • Westhoff C.M.
        • Uy J.M.
        • et al.
        Comprehensive red blood cell and platelet antigen prediction from whole genome sequencing: proof of principle.
        Transfusion. 2016; 56: 743-754
        • Colburn J.L.
        • Nothelle S.
        The Medicare Annual Wellness Visit.
        Clin Geriatr Med. 2018; 34: 1-10
        • McBride C.M.
        • Alford S.H.
        • Reid R.J.
        • et al.
        Characteristics of users of online personalized genomic risk assessments: implications for physician-patient interactions.
        Genet Med. 2009; 11: 582-587
        • National Research Council
        Institute of Medicine Roundtable on Translating Genomic-Based Research for Health.
        Direct-To-Consumer Genetic Testing: Summary of a Workshop. National Academies Press, Washington, DC2010
        • Kalf R.R.J.
        • Bakker R.
        • Janssens A.C.J.W.
        Predictive ability of direct-to-consumer pharmacogenetic testing: when is lack of evidence really lack of evidence?.
        Pharmacogenomics. 2013; 14: 341-344
        • Kalf R.R.J.
        • Mihaescu R.
        • Kundu S.
        • et al.
        Variations in predicted risks in personal genome testing for common complex diseases.
        Genet Med. 2014; 16: 85-91
        • Pyeritz R.E.
        The family history: the first genetic test, and still useful after all those years?.
        Genet Med. 2012; 14: 3-9
        • Facio F.M.
        • Feero W.G.
        • Linn A.
        • et al.
        Validation of My Family Health Portrait for six common heritable conditions.
        Genet Med. 2010; 12: 370-375
        • Phillips K.A.
        • Pletcher M.J.
        • Ladabaum U.
        Is the “$1000 genome” really $1000? Understanding the full benefits and costs of genomic sequencing.
        Technol Health Care. 2015; 23: 373-379
        • Caulfield T.
        • Evans J.
        • McGuire A.
        • et al.
        Reflections on the cost of “low-cost” whole genome sequencing: framing the health policy debate.
        PLoS Biol. 2013; 11: e1001699
        • Green R.C.
        • Goddard K.A.B.
        • Jarvik G.P.
        • et al.
        Clinical Sequencing Exploratory Research Consortium: accelerating evidence-based practice of genomic medicine.
        Am J Hum Genet. 2016; 98: 1051-1066
        • Dorschner Michael O.
        • Amendola Laura M.
        • Turner Emily H.
        • et al.
        Actionable, pathogenic incidental findings in 1,000 participants exomes.
        Am J Hum Genet. 2013; 93: 631-640
        • Natarajan P.
        • Gold N.B.
        • Bick A.G.
        • et al.
        Aggregate penetrance of genomic variants for actionable disorders in European and African Americans.
        Sci Transl Med. 2016; 8: 364ra151
      2. Glick HA. Economic Evaluation in Clinical Trials (2nd ed.). New York: Oxford University Press, 2014.

        • Ramsey S.D.
        • Willke R.J.
        • Glick H.
        • et al.
        Cost-effectiveness analysis alongside clinical trials II—an ISPOR Good Research Practices Task Force report.
        Value Health. 2015; 18: 161-172
        • Centers for Disease Control and Prevention
        Behavioral Risk Factor Surveillance System Survey Questionnaire.
        U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, Atlanta, GA2009
        • Carere D.
        • Couper M.
        • Crawford S.
        • et al.
        Design, methods, and participant characteristics of the Impact of Personal Genomics (PGen) Study, a prospective cohort study of direct-to-consumer personal genomic testing customers.
        Genome Med. 2014; 6: 96
        • Gray S.W.
        • Martins Y.
        • Feuerman L.Z.
        • et al.
        Social and behavioral research in genomic sequencing: approaches from the Clinical Sequencing Exploratory Research Consortium Outcomes and Measures Working Group.
        Genet Med. 2014; 16: 727-735
        • Pagon R.A.
        • Tarczy-Hornoch P.
        • Baskin P.K.
        • et al.
        GeneTests-GeneClinics: genetic testing information for a growing audience.
        Hum Mutat. 2002; 19: 501-509
        • Hamosh A.
        • Scott A.F.
        • Amberger J.
        • et al.
        Online Mendelian Inheritance in Man (OMIM).
        Hum Mutat. 2000; 15: 57-61
        • Kho A.N.
        • Pacheco J.A.
        • Peissig P.L.
        • et al.
        Electronic medical records for genetic research: results of the eMERGE consortium.
        Sci Transl Med. 2011; 3: 79re1
        • Ware J.E.
        • Kosinki M.
        • Keller S.D.
        A 12-item short-form health survey: construction of scales and preliminary tests of reliability and validity.
        Med Care. 1996; 34: 220-233
        • Brazier J.E.
        • Roberts J.
        The estimation of a preference-based measure of health from the SF-12.
        Med Care. 2004; 42: 851-859
        • Delaney S.K.
        • Hultner M.L.
        • Jacob H.J.
        • et al.
        Toward clinical genomics in everyday medicine: perspectives and recommendations.
        Expert Rev Mol Diagn. 2016; 16: 521-532
      3. National Human Genome Research Institute. The cost of sequencing a human genome. Available at: www.genome.gov/sequencingcosts [Accessed November 30, 2016].

        • Messner D.A.
        • Koay P.
        • Al Naber J.
        • et al.
        Barriers to clinical adoption of next-generation sequencing: a policy Delphi panel’s solutions.
        Per Med. 2017; 14: 339-354
        • Wilson B.J.
        • Qureshi N.
        • Santaguida P.
        • et al.
        Systematic review: family history in risk assessment for common diseases.
        Ann Intern Med. 2009; 151: 878-885
        • Ozanne E.M.
        • O’Connell A.
        • Bouzan C.
        • et al.
        Bias in the reporting of family history: implications for clinical care.
        J Genet Couns. 2012; 21: 547-556
        • Orlando L.A.
        • Buchanan A.H.
        • Hahn S.E.
        • et al.
        Development and validation of a primary care-based family health history and decision support program (MeTree).
        N C Med J. 2013; 74: 287-296
        • O’Neill S.M.
        • Rubinstein W.S.
        • Wang C.
        • et al.
        Familial risk for common diseases in primary care: the Family Healthware Impact Trial.
        Am J Prev Med. 2009; 36: 506-514
        • Natarajan P.
        • Young R.
        • Stitziel N.O.
        • et al.
        Polygenic risk score identifies subgroup with higher burden of atherosclerosis and greater relative benefit from statin therapy in the primary prevention setting.
        Circulation. 2017; 135: 2091-2101
        • Chatterjee N.
        • Shi J.
        • Garcia-Closas M.
        Developing and evaluating polygenic risk prediction models for stratified disease prevention.
        Nat Rev Genet. 2016; 17: 392-406
      4. All-Payer Claims Database Council, National Association of Health Data Organizations, University of New Hampshire. All-Payer Claims Database Council. Available at: www.apcdcouncil.org [Accessed March 20, 2018].

        • Rose M.
        • Bjorner J.B.
        • Gandek B.
        • et al.
        The PROMIS Physical Function item bank was calibrated to a standardized metric and shown to improve measurement efficiency.
        J Clin Epidemiol. 2014; 67: 516-526
        • Hanmer J.
        • Feeny D.
        • Fischhoff B.
        • et al.
        The PROMIS of QALYs.
        Health Qual Life Outcomes. 2015; 13: 122
        • Gaziano J.M.
        • Concato J.
        • Brophy M.
        • et al.
        Million Veteran Program: a mega-biobank to study genetic influences on health and disease.
        J Clin Epidemiol. 2016; 70: 214-223
      5. U.S. Department of Health and Human Services. All of Us Research Program. Available at: www.nih.gov/research-training/allofus-research-program [Accessed March 30, 2017].

      6. National Cancer Institute. NCI Dictionary of Genetics Terms. Available at: www.cancer.gov/publications/dictionaries/genetics-dictionary [Accessed April 19, 2018].

        • Freeman J.L.
        • Perry G.H.
        • Feuk L.
        • et al.
        Copy number variation: new insights in genome diversity.
        Genome Res. 2006; 16: 949-961
      7. Partners HealthCare. Exome and Genome Sequencing FAQ. Available at: personalizedmedicine.partners.org/laboratory-for-molecular-medicine/faq/exome-genome-sequencing.aspx [Accessed March 1, 2018].

        • Strom S.P.
        • Lee H.
        • Das K.
        • et al.
        Assessing the necessity of confirmatory testing for exome-sequencing results in a clinical molecular diagnostic laboratory.
        Genet Med. 2014; 16: 510-515