Advertisement

Quantifying Downstream Healthcare Utilization in Studies of Genomic Testing

Open ArchivePublished:March 19, 2020DOI:https://doi.org/10.1016/j.jval.2020.01.017

      Highlights

      • Challenges to quantifying downstream healthcare utilization are common but magnified by genomic testing because of the limited number of patients who have received it to date and the complexity and diversity of information it can provide.
      • We summarize a novel approach for quantifying healthcare utilization related to genomic testing that balances the need to comprehensively define the scope of services that may be influenced by results while avoiding consideration of unrelated services. We also summarize the effectiveness of different approaches for determining how often patients received genomic result-related services.
      • The approach we propose provides methods for researchers to use in analyses of real-world data about the short-term impact of genomic testing on healthcare utilization that more effectively identifies related services and excludes unrelated services that could also be applied in other contexts. The article also provides insight about how effectively healthcare services are identified using different strategies for reviewing medical records, survey data, and family-reported data.

      Abstract

      Objectives

      The challenges of understanding how interventions influence follow-up medical care are magnified during genomic testing because few patients have received it to date and because the scope of information it provides is complex and often unexpected. We tested a novel strategy for quantifying downstream healthcare utilization after genomic testing to more comprehensively and efficiently identify related services. We also evaluated the effectiveness of different methods for collecting these data.

      Methods

      We developed a risk-based approach for a trial of newborn genomic sequencing in which we defined primary conditions based on existing diagnoses and family histories of disease and defined secondary conditions based on unexpected findings. We then created patient-specific lists of services associated with managing primary and secondary conditions. Services were quantified based on medical record reviews, surveys, and telephone check-ins with parents.

      Results

      By focusing on services that genomic testing would most likely influence in the short-term, we reduced the number of services in our analyses by more than 90% compared with analyses of all observed services. We also identified the same services that were ordered in response to unexpected findings as were identified during expert review and by confirming whether recommendations were completed. Data also showed that quantifying healthcare utilization with surveys and telephone check-ins alone would have missed the majority of attributable services.

      Conclusions

      Our risk-based strategy provides an improved approach for assessing the short-term impact of genomic testing and other interventions on healthcare utilization while conforming as much as possible to existing best-practice recommendations.

      Keywords

      Introduction

      Best practices for assessing the value of any intervention are to include how it affects ongoing medical care.
      • Sanders G.D.
      • Neumann P.J.
      • Basu A.
      • et al.
      Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine.
      • Glick H.A.
      Economic Evaluation in Clinical Trials.
      • Caro J.J.
      • Briggs A.H.
      • Siebert U.
      • et al.
      Modeling good research practices—overview: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-1.
      Quantifying downstream healthcare utilization can be difficult, however, especially in the context of genomic testing. Methods that consider all observed services can be uninformative when they require large sample sizes,
      • Diehr P.
      • Yanez D.
      • Ash A.
      • et al.
      Methods for analyzing health care utilization and costs.
      and relatively few patients have received genomic testing to date. Alternative strategies that focus on services to manage specific conditions can miss important procedures when the conditions are complex,
      • Robertson C.
      • Archibald D.
      • Avenell A.
      • et al.
      Systematic reviews of and integrated report on the quantitative, qualitative and economic evidence base for the management of obesity in men.
      ,
      • Nestler-Parr S.
      • Korchagina D.
      • Toumi M.
      • et al.
      Challenges in research and health technology assessment of rare disease technologies: report of the ISPOR Rare Disease Special Interest Group.
      and the scope of information provided by genomic testing can be especially vast and complicated.
      • Phillips K.A.
      • Deverka P.A.
      • Marshall D.A.
      • et al.
      Methodological issues in assessing the economic value of next-generation sequencing tests: many challenges and not enough solutions.
      Moreover, findings are often unanticipated: laboratories frequently disclose secondary findings that are unrelated to test indications, such as monogenic disease risks, carrier status, and pharmacogenomic information.
      • Ackerman S.L.
      • Koenig B.A.
      Understanding variations in secondary findings reporting practices across U.S. genome sequencing laboratories.
      • Ormond K.E.
      • O’Daniel J.M.
      • Kalia S.S.
      Secondary findings: how did we get here, and where are we going?.
      • Delanne J.
      • Nambot S.
      • Chassagne A.
      • et al.
      Secondary findings from whole-exome/genome sequencing evaluating stakeholder perspectives: a review of the literature.
      Thus, best practices for assessing healthcare utilization to track services that may be influenced by the intervention of interest, independent of cause,
      • Glick H.A.
      Economic Evaluation in Clinical Trials.
      ,
      • Ramsey S.D.
      • Willke R.J.
      • Glick H.
      • et al.
      Cost-effectiveness analysis alongside clinical trials II—an ISPOR Good Research Practices Task Force report.
      are difficult to apply.
      The challenges of assessing healthcare utilization downstream of genomic testing and its associated costs have been addressed previously.
      • Reeves P.
      • Doran C.
      • Carey M.
      • et al.
      Costs and cost-effectiveness of targeted, personalized risk information to increase appropriate screening by first-degree relatives of people with colorectal cancer.
      ,
      • Fahr P.
      • Buchanan J.
      • Wordsworth S.
      A review of the challenges of using biomedical big data for economic evaluations of precision medicine.
      In 2018, we addressed this methodological challenge by proposing a solution in which healthcare utilization is assessed through multiple strategies.
      • Christensen K.D.
      • Phillips K.A.
      • Green R.C.
      • et al.
      Cost analyses of genomic sequencing: lessons learned from the MedSeq Project.
      A pilot study implemented these approaches and generated a series of analyses that included (1) all healthcare identified, (2) only services that were immediately attributable to interventions, and (3) all services that could have been ordered during a full genetics workup.
      • Christensen K.D.
      • Phillips K.A.
      • Green R.C.
      • et al.
      Cost analyses of genomic sequencing: lessons learned from the MedSeq Project.
      ,
      • Christensen K.D.
      • Vassy J.L.
      • Phillips K.A.
      • et al.
      Short-term costs of integrating whole-genome sequencing into primary care and cardiology settings: a pilot randomized trial.
      Although this strategy provided unique insight into the impact of genomic testing on healthcare utilization, it also had important conceptual and analytic limitations. Analyses of immediately attributable services, for instance, may have omitted services initiated in response to the initial workup, whereas the overwhelming majority of healthcare utilization included in analyses of all observed services were clearly unrelated to the interventions. The latter approach also generated tremendously large confidence intervals for mean group differences in health sector costs.
      The work presented here addresses the aforementioned concerns through use of a novel approach for identifying downstream healthcare utilization associated with genomic testing. We describe a risk-based approach that narrows the scope of healthcare services we expect genomic testing would influence. We then compared the services we identified using this approach against the services we identified using standard approaches. We also compare the effectiveness of different strategies for identifying and quantifying healthcare utilization. The goal of this article is to provide an alternative approach for researchers who use real-world data to assess the short-term impact of genomic testing on healthcare utilization.

      Methods

      Design of the Case Study

      We developed the approach described here for the BabySeq Project, the first randomized clinical trial of newborn genomic sequencing (nGS).
      • Holm I.A.
      • Agrawal P.B.
      • Ceyhan-Birsoy O.
      • et al.
      The BabySeq project: implementing genomic sequencing in newborns.
      Key terms that are used in this case report are summarized in Table 1. nGS in the BabySeq Project included the use of exome sequencing to screen nearly 1000 genes for variants associated with monogenic childhood-onset conditions and highly actionable adult-onset conditions.
      • Ceyhan-Birsoy O.
      • Machini K.
      • Lebo M.S.
      • et al.
      A curated gene list for reporting results of newborn genomic sequencing.
      nGS also identified carrier status for childhood-onset disease and pharmacogenomic variants for medications that may be prescribed during childhood. Providers had the option to order additional indication-based analyses based on symptoms that suggested a possible genetic disorder.
      Table 1Key terms.
      Definitions were adapted from the National Human Genome Research Institute’s Talking Glossary of Genetic Terms,17 the National Cancer Institute’s Dictionary of Genetics Terms,18 the Genetics Home Reference,19 the World Health Organization’s website,20 and the CSER TOOLKIT.21
      Genomic testingA laboratory process that can identify changes in multiple genes. Genomic testing can include exome and genome sequencing.
      Genome sequencingA laboratory process that determines nearly all of the approximately 3 billion nucleotides of an individual’s complete DNA sequence, including noncoding sequences.
      Exome sequencingA laboratory process that is used to determine the nucleotide sequence of primarily exonic (or protein-coding) regions of an individual’s genome and related sequences, representing approximately 1% of the complete DNA sequence.
      Newborn genomic sequencingA laboratory approach, currently under study, that uses genomic testing to collect and analyze large amounts of DNA sequence data in the newborn period (https://ghr.nlm.nih.gov/primer/newbornscreening/newborngenomicsequencing).
      Monogenic diseaseDisease that result from modifications in a single gene occurring in all cells of the body.
      Carrier statusTo have some diseases, an individual must usually have 2 mutated copies of a gene. Carrier status is positive when an individual has 1 normal copy of a gene and 1 mutated copy of a gene. With rare exceptions, individuals with positive carrier status are not expected to develop the disease.
      PharmacogenomicsPharmacogenomics is a branch of pharmacology concerned with using DNA and amino acid sequence data to inform drug development and testing. An important application of pharmacogenomics is correlating individual genetic variation with drug responses.
      Definitions were adapted from the National Human Genome Research Institute’s Talking Glossary of Genetic Terms,
      National Human Genome Research Institute
      Talking glossary of genetic terms.
      the National Cancer Institute’s Dictionary of Genetics Terms,
      National Cancer Institute
      NCI dictionary of genetics terms.
      the Genetics Home Reference,
      US National Library of Medicine
      What is newborn genomic sequencing?.
      the World Health Organization’s website,
      World Health Organization
      Human genomics in global health.
      and the CSER TOOLKIT.
      Practitioner Education Working Group of the Clinical Sequencing Exploratory Research (CSER) Consortium
      Introduction and overview to genomic test reports.
      A more detailed summary of the BabySeq Project, molecular findings, and parental attitudes at enrollment has been published previously.
      • Holm I.A.
      • Agrawal P.B.
      • Ceyhan-Birsoy O.
      • et al.
      The BabySeq project: implementing genomic sequencing in newborns.
      ,
      • Ceyhan-Birsoy O.
      • Machini K.
      • Lebo M.S.
      • et al.
      A curated gene list for reporting results of newborn genomic sequencing.
      • VanNoy G.E.
      • Genetti C.A.
      • McGuire A.L.
      • et al.
      Challenging the current recommendations for carrier testing in children.
      • Holm I.A.
      • McGuire A.
      • Pereira S.
      • et al.
      Returning a genomic result for an adult-onset condition to the parents of a newborn: insights from the BabySeq Project.
      • Pereira S.
      • Robinson J.O.
      • Gutierrez A.M.
      • et al.
      Perceived benefits, risks, and utility of newborn genomic sequencing in the BabySeq Project.
      • Murry J.B.
      • Machini K.
      • Ceyhan-Birsoy O.
      • et al.
      Reconciling newborn screening and a novel splice variant in BTD associated with partial biotinidase deficiency: a BabySeq Project case report.
      • Genetti C.A.
      • Schwartz T.S.
      • Robinson J.O.
      • et al.
      Parental interest in genomic sequencing of newborns: enrollment experience from the BabySeq Project.
      • Ceyhan-Birsoy O.
      • Murry J.B.
      • Machini K.
      • et al.
      Interpretation of genomic sequencing results in healthy and ill newborns: results from the BabySeq Project.
      The study included healthy newborns born at Brigham and Women’s Hospital (BWH) and newborns in intensive care units (ICUs) at BWH, Boston Children’s Hospital (BCH), and Massachusetts General Hospital (MGH). Newborns were randomized to a modified standard of care (standard newborn screening and an in-depth family history analyses) or standard of care plus nGS. Results were disclosed to parents by a genetic counselor and physician. Reports were uploaded into newborns’ medical records and sent to clinicians involved in the newborns’ care. Primary endpoints included healthcare utilization over a 10-month period after results disclosure. The trial was registered with ClinicalTrials.gov (identifier: NCT02422511) and was approved by the institutional review boards of BWH, BCH, MGH, and the Baylor College of Medicine.

      Defining Relevant Healthcare Services

      An overview of the approach we tested is summarized in Table 2. First, we defined “primary conditions” and “secondary conditions.” Primary conditions are akin to indications in diagnostic testing and are conditions known before randomization for which we would expect nGS to influence care immediately. In the BabySeq Project, these conditions included existing clinical diagnoses and family history reviews that raised the possibility of a genetic disorder. We omitted acute transient conditions that resolved by the time the newborn was discharged from the hospital. We defined secondary conditions as those associated with unexpected monogenic findings and carrier status known to cause disease. To improve the feasibility of implementation, we omitted conditions associated with carrier status that could not manifest as disease. We also omitted conditions associated with pharmacogenomic information because they could not be well defined.
      Table 2Strategy for analyzing the impact of population newborn genomic sequencing (nGS) on downstream healthcare costs.
      StepBabySeq project implementation
      1. Define relevant conditions
       Primary conditionsConditions identified before disclosure for which care may be influenced by nGS
      • Existing diagnoses
      • Family histories suggestive of possible genetic disease
       Secondary conditionsConditions identified during disclosure for which care may be influenced by nGS
      • Unexpected monogenic disease risks
      • Manifesting carrier status
      2. Identify relevant servicesFor each primary and secondary condition, develop patient-specific lists of services that may be used for
      • Diagnosis
      • Screening
      • Treatment
      Services of interest are identified based on review of
      • GeneReviews
        • Pagon R.A.
        • Tarczy-Hornoch P.
        • Baskin P.K.
        • et al.
        GeneTests-GeneClinics: genetic testing information for a growing audience.
      • Online Mendelian Inheritance in Man
        • Hamosh A.
        • Scott A.F.
        • Amberger J.
        • et al.
        Online Mendelian Inheritance in Man (OMIM).
      • National Comprehensive Cancer Network guidelines
        • Winn R.J.
        • Botnick W.
        • Dozier N.
        The NCCN guidelines development program.
      • UpToDate
        • Fox G.N.
        • Moawad N.
        UpToDate: a comprehensive clinical database.
      3. Quantify healthcare utilizationRecord how often the relevant services were received by each patient based on:
      • Medical records reviews
      • Family self-report
        • Surveys
        • Periodic calls with families
        • 10-mo check-in
      • Provider surveys
      Based on the primary and secondary conditions identified, we developed lists for each newborn that included specialist encounters, tests, procedures, prescriptions, and devices that may be ordered to diagnose, screen for, or treat each condition. Services were identified based on condition summaries in GeneReviews,
      • Pagon R.A.
      • Tarczy-Hornoch P.
      • Baskin P.K.
      • et al.
      GeneTests-GeneClinics: genetic testing information for a growing audience.
      Online Mendelian Inheritance in Man,
      • Hamosh A.
      • Scott A.F.
      • Amberger J.
      • et al.
      Online Mendelian Inheritance in Man (OMIM).
      the National Comprehensive Cancer Network,
      • Winn R.J.
      • Botnick W.
      • Dozier N.
      The NCCN guidelines development program.
      and UpToDate.
      • Fox G.N.
      • Moawad N.
      UpToDate: a comprehensive clinical database.
      To blind reviewers of medical records and survey data from each newborn’s randomization status and nGS findings, lists of services omitted the names of the associated conditions. Study personnel then used these patient-specific lists to review the newborns’ medical records and family-reported data to determine how often the services occurred within the trial period. The purpose of distinguishing primary conditions and secondary conditions was to improve the statistical precision of analyses by limiting the scope of considered services to those we may expect nGS to influence in the short-term.

      Approaches to Quantifying Healthcare Utilization

      After developing patient-specific lists of services to analyze, we documented how often the services occurred by reviewing medical records, surveying and interviewing families, and surveying physicians who provided care to the newborn. Approaches are summarized in Table 3. To identify healthcare associated with primary conditions, we reviewed the medical records of the healthcare systems in which the newborns had been enrolled (ie, for newborns enrolled at BWH or MGH, we reviewed each newborn’s medical records at Partners HealthCare; and for newborns enrolled at BCH, we reviewed newborns medical records at BCH). To identify healthcare associated with secondary conditions, we reviewed the medical records at both Partners HealthCare and BCH for all newborns with unexpected monogenic findings and included services that parents reported during follow-up calls from study staff that were conducted periodically postdisclosure.
      Table 3Summary of approaches that were implemented to quantify healthcare utilization.
      ApproachDescription
      Medical records data
       Review of medical recordsInvestigators and study staff reviewed electronic medical records of newborns at their system of enrollment. Medical records at both BCH and Partners HealthCare were reviewed if newborns were identified with an unexpected monogenic disease risk.
      Family self-report
       Phone calls with familiesStudy staff called families of newborns with unexpected monogenic disease risks periodically to ask about healthcare that was received.
       SurveysSurveys administered 3 and 10 mo after disclosure sessions asked parents to report how often their newborn received genetic counseling or specialist visits.
       10-month check-inParents of newborns randomized to nGS were called after 10 mo had passed from disclosure sessions and asked to report follow-up regarding their newborns’ results.
      Provider report
       SurveysProviders of all enrolled newborns were sent surveys asking whether they made referrals or ordered additional tests.
      BCH indicates Boston Children’s Hospital; nGS, newborn genomic sequencing.
      Given the potential for subjects to receive care associated with nGS outside of the Partners HealthCare (BWH and MGH) or BCH systems over the study period, we also collected data from subjects’ families. All families were asked to complete surveys administered 3 months and 10 months after disclosure sessions that asked whether anyone in their families, including the newborn, received genetic counseling outside of the study. Surveys also asked parents to report the number of specialist visits their newborn received. In addition, parents were called after 10 months had passed from their newborns’ results disclosure sessions as part of a 10-month check-in, during which they were asked to report any follow-up their newborns received regarding their results. Parents of newborns identified with unexpected monogenic disease risks were also called by study staff periodically and asked about the services their newborn received.
      As a final strategy for quantifying healthcare utilization, we invited all physicians who provided care to newborns in the BabySeq Project to complete surveys that asked them to describe any referrals or additional tests they initiated based on BabySeq reports.

      Data Analysis

      To provide insight into how much our approach reduced the scope of analyzed services, we descriptively compared the total number of services it identified against the total number of outpatient services observed in medical records data over the 10-month trial period. These analyses were restricted to comparisons of newborns enrolled at Partners HealthCare sites, given challenges in distinguishing inpatient and outpatient care received at BCH.
      To provide insight into the effectiveness of our risk-based approach for identifying services associated with secondary conditions, we descriptively compared the number of services it identified against the number of services identified by expert review of newborns’ medical records and data from periodic follow-up calls. We also compared the number of services identified using our risk-based approach against the number of confirmed services that newborns received in response to recommendations for clinical follow-up issued during results disclosure sessions. Given the heterogeneity of conditions associated with unexpected genomic findings and the potential for families to pursue follow-up outside the systems of enrollment, analyses of healthcare utilization associated with secondary conditions and unexpected monogenic disease risks included medical records data from both Partners HealthCare and BCH for all newborns and services that were identified through family self-report and provider report. Concordance analyses were conducted by calculating Kappa statistics that compared specific approaches. Analyses were conducted using R version 3.6.1.

      Results

      Participant Characteristics and Data Availability

      Results from family history analyses and nGS were disclosed to families of 237 healthy newborns and 61 newborns in the ICU cohort (newborn characteristics are summarized in the Supplementary Materials found at https://doi.org/10.1016/j.jval.2020.01.017). Newborns were 4.6 months old, on average (SD = 55 days), at the time of results disclosure sessions. No genetic variants were identified that conclusively explained existing clinical features, but unexpected findings about monogenic disease risks were identified in 17 of 158 infants (6.6%) randomized to nGS,
      • Ceyhan-Birsoy O.
      • Murry J.B.
      • Machini K.
      • et al.
      Interpretation of genomic sequencing results in healthy and ill newborns: results from the BabySeq Project.
      excluding 1 ill newborn identified with a monogenic disease risk variant who passed away before the results disclosure session.
      Our analyses provide insight about how nonresponse to surveys can compromise the ability to identify healthcare utilization. Thirty-one families (10.4%) reported healthcare utilization at 3 months postdisclosure but not 10 months postdisclosure, and 68 families (22.8%) did not report healthcare utilization data at either time point. Among the 17 families of newborns with unexpected monogenic disease risks, 1 (5.9%) did not complete survey items about healthcare utilization at 10 months postdisclosure only, and 5 others (29.4%) did not complete survey items about healthcare utilization at 3 or 10 months postdisclosure. Data from provider surveys were also limited, with no data available for 167 newborns overall (56.0%) and no data available for 10 newborns with unexpected monogenic disease risks (58.8%).

      Healthcare Services Associated With Primary Conditions

      Analyses of services potentially associated with follow-up care of primary conditions identified 316 services ordered, including 154 services for care of newborns in the nGS arm and 162 services for care of newborns in the control arm (P = .25). Analyses of services received by the 266 newborns enrolled in the Partners HealthCare system showed that restricting analyses to services that may be associated with the care of primary conditions reduced the number of services overall by more than 90%, from 521 total services to 48.

      Healthcare Services Associated With Secondary Conditions

      Among the 17 newborns who were identified with unexpected genomic findings during nGS findings and followed for the trial period, we identified 43 services associated with follow-up care (see Fig. 1). The most services were identified using our proposed risk-based strategy (33 of 43; 76.7%), followed by expert review (31 of 43; 72.1%). All other approaches identified less than 25% of the total number. The concordance between services identified using our risk-based approach and expert review was high (kappa = 0.92; P < .001). The only discrepancy was a hematology encounter for a newborn with a likely pathogenic variant in the GLMN gene that was omitted during expert review. Concordance between our risk-based approach and all other approaches was slight (kappa = 0.22; P = .012) in comparison with recommendations-based reviews and poor (kappa < 0.07, all P > .08) in comparisons with parent surveys, parent interviews, and physician surveys.
      Figure thumbnail gr1
      Figure 1Number of healthcare services associated with follow-up of secondary findings from newborn genomic sequencing using different approaches.
      ∗Parent surveys were omitted because they included no items to assess healthcare utilization other than specialist visits.
      Figure 1 also summarizes differences between the number of follow-up services identified using a risk-based approach and other approaches specific to follow-up visits and other services. The additional visits identified using a risk-based approach rather than a recommendations-based approach were encounters that occurred subsequent to an initial follow-up visit for 4 newborns. No specialist visits were reported at the 10-month check-in, and only 4 of the 17 total specialist visits (23.5%) were observed in both our risk-based medical approach and family surveys. All 3 referrals to specialists reported by providers were for a single patient and could not be corroborated by data from medical record reviews, surveys of families, or interviews of families. Risk-based reviews and expert reviews also identified most of the other types of follow-up services. All services reported by parents at the 10-month check-in were also observed during the risk-based and expert reviews. Physician surveys included recommendations for an echocardiogram and electrocardiogram for the aforementioned newborn in the ICU cohort with the GLMN variant that were not observed in other approaches.

      Discussion

      Even if genomic testing provides clinical benefits, it will need to demonstrate that it provides value. By distinguishing between primary and secondary conditions and defining relevant services on a per-patient basis, we narrowed the scope of services included in our analyses to only those that we would anticipate genomic testing to influence in the short-term and reduced the number of services that were included in our analyses by more than 90%. At the same time, our risk-based approach identified the same services that were ordered in response to unexpected genomic findings as an expert review and by confirming whether recommended services occurred. Our successful proof-of-concept work provides a starting point for other studies of the impact of genomic testing on healthcare utilization, which comprehensively identifies attributable services while improving the precision of statistical analyses.
      Our approach balances best practices and pragmatism. For primary conditions, our strategy is akin to standard analysis of healthcare utilization for indication-based testing. Rather than focusing on services associated with managing a single condition, our approach encompasses an expansive set of conditions that genomic testing may influence over shorter periods of time. In this initial work, we focused on patient diagnoses and family histories of disease suggestive of possible genetic disorders. An alternative approach that could narrow the scope of relevant services further would be to define primary conditions as those with validated gene-disease associations. Another improvement could be to consider services associated with a diagnostic workup and treatment of unexplained phenotypes. Regardless, the approach we currently tested would improve the precision of statistical analyses over traditional approaches while conforming as much as possible to best practice recommendations.
      • Ramsey S.D.
      • Willke R.J.
      • Glick H.
      • et al.
      Cost-effectiveness analysis alongside clinical trials II—an ISPOR Good Research Practices Task Force report.
      The improved methods we have proposed for short-term analyses may help address the needs of policy makers and payers who are making decisions about genomic testing today.
      Our work also provides an improved strategy for assessing healthcare associated with secondary conditions and unexpected genomic findings. Our strategy was effective not only for confirming whether recommendations made at disclosure were completed but also for identifying services ordered after the initial workup. The concordance between the services identified through our risk-based approach and the services identified by confirming recommendations and expert review was not surprising given that the informational resources we used to determine relevant services are frequently used by genetic counselors and medical geneticists.
      • Christensen K.D.
      • Bernhardt B.A.
      • Jarvik G.P.
      • et al.
      Anticipated responses of early adopter genetic specialists and nongenetic specialists to unsolicited genomic secondary findings.
      Investigators may interpret the concordance in 2 ways. On one hand, our findings suggest that expert review of services can be as effective at identifying attributable services as our risk-based approach but requires less effort to implement. On the other hand, our risk-based approach identified the same services as the expert review, although it was executed by study team members with limited expertise in genomics. Moreover, developing lists of relevant services for managing secondary conditions and unanticipated findings provides an approach for assessing healthcare utilization that is potentially more scalable. The genes that are frequently queried for unexpected findings, such as the incidental findings list developed by ACMG for secondary findings disclosure
      • Green R.C.
      • Berg J.S.
      • Grody W.W.
      • et al.
      ACMG recommendations for reporting of incidental findings in clinical exome and genome sequencing.
      ,
      • Kalia S.S.
      • Adelman K.
      • Bale S.J.
      • et al.
      Recommendations for reporting of secondary findings in clinical exome and genome sequencing, 2016 update (ACMG SF v2.0): a policy statement of the American College of Medical Genetics and Genomics.
      or Geisinger Health System’s MyCode list,
      • Carey D.J.
      • Fetterolf S.N.
      • Davis F.D.
      • et al.
      The Geisinger MyCode community health initiative: an electronic health record-linked biobank for precision medicine research.
      typically have major overlap. Developing consensus lists of services that include associated diagnostic and billing codes could help standardize analyses across studies and facilitate automated data abstraction from electronic medical records. These strategies are integral to cost-of-illness studies that use encounter-based approaches
      • Rosen A.B.
      • Cutler D.M.
      Challenges in building disease-based national health accounts.
      or the development of condition-specific resource use questionnaires.
      • Leggett L.E.
      • Khadaroo R.G.
      • Holroyd-Leduc J.
      • et al.
      Measuring resource utilization: a systematic review of validated self-reported questionnaires.
      One approach to developing these lists of procedures and diagnostic codes may be to capitalize on National Institutes of Health–based efforts such as ClinVar
      • Landrum M.J.
      • Lee J.M.
      • Riley G.R.
      • et al.
      ClinVar: public archive of relationships among sequence variation and human phenotype.
      and ClinGen.
      • Rehm H.L.
      • Berg J.S.
      • Brooks L.D.
      • et al.
      ClinGen—the Clinical Genome Resource.
      These efforts have already honed collaborative processes for developing standards and for disseminating data to researchers.
      In addition to providing a strategy for reducing the number of services to consider in analyses, our work provides insight into the effectiveness of different strategies for determining how often services occurred. The data we summarize demonstrate that limiting the approach to confirming whether families followed through on clinical recommendations can miss many of the follow-up services that occurred. Similarly, relying on surveys administered to families may miss many specialty visits, although the surveys we administered did identify visits that medical record reviews and periodic phone calls did not. It is likely that structuring our surveys to query parents about specific provider visits helped families recall some visits that may have occurred at medical systems for which we had no access to medical records.
      • Bhandari A.
      • Wagner T.
      Self-reported utilization of health care services: improving measurement and accuracy.
      ,
      • Gama H.
      • Correia S.
      • Lunet N.
      Effect of questionnaire structure on recall of drug utilization in a population of university students.
      Surveys of families provided little insight about other types of healthcare utilization because they did not ask about services aside from additional genetic testing (which no parents of newborns with unexpected monogenic disease risks reported). Similarly, few parents of newborns with unexpected monogenic disease risks reported services during their end-of-study phone interview. Given that study staff had been calling these families periodically to collect information about the medical follow-up their newborns were receiving, it is possible that many families considered the 10-month check-in to be redundant because they had already shared relevant information.
      Admittedly, our approach is suited primarily for studies with shorter time horizons. Over longer time horizons, the approach is likely to overstate the impact of unexpected genomic findings on healthcare utilization because the approach would not account for potential reductions in resource use enabled by early identification of genetic risk factors and the initiation of preventive care. Greater standardization about what genomic testing results should be disclosed as unexpected findings could also better allow our approach to be suitable for longer-term analyses because it would allow lists of the most common secondary conditions to be created a priori. Such lists could then be used to assess downstream healthcare utilization in all patients included in analyses, regardless of whether they received genomic testing or whether they were identified with relevant pathogenic variants.
      Limitations to our work also include the post hoc identification of primary and secondary conditions and related services. We classified services as having occurred based on family report although respondent recall is often biased or incomplete,
      • Ansah E.K.
      • Powell-Jackson T.
      Can we trust measures of healthcare utilization from household surveys?.
      ,
      • Hunger M.
      • Schwarzkopf L.
      • Heier M.
      • et al.
      Official statistics and claims data records indicate non-response and recall bias within survey-based estimates of health care utilization in the older population.
      particularly for minor services received over longer time horizons.
      • Mauldin P.D.
      • Guimaraes P.
      • Albin R.L.
      • et al.
      Optimal frequency for measuring health care resource utilization in Parkinson’s disease using participant recall: the FS-TOO resource utilization substudy.
      ,
      • Kjellsson G.
      • Clarke P.
      • Gerdtham U.G.
      Forgetting to remember or remembering to forget: a study of the recall period length in health care survey questions.
      Patient- and provider-reported services were not always verified, and the scope of care received by newborns outside of the BCH and Partners HealthCare systems was unclear. Data were collected from a single trial in metro Boston and may not generalize well to other studies and settings.

      Conclusions

      A risk-based approach that defines primary and secondary conditions and relevant healthcare services for individual patients can provide a more comprehensive assessment of services that are affected by genomic testing in the short-term and omit unrelated services from analyses. Approaches such as these can help address the dearth of evidence about the impact of genomic testing
      • Phillips K.A.
      • Deverka P.A.
      • Marshall D.A.
      • et al.
      Methodological issues in assessing the economic value of next-generation sequencing tests: many challenges and not enough solutions.
      ,
      • Buchanan J.
      • Wordsworth S.
      • Schuh A.
      Issues surrounding the health economic evaluation of genomic technologies.
      • Prosser L.A.
      • Grosse S.D.
      • Kemper A.R.
      • et al.
      Decision analysis, economic evaluation, and newborn screening: challenges and opportunities.
      • Phillips K.A.
      • Trosman J.R.
      • Kelley R.K.
      • et al.
      Genomic sequencing: assessing the health care system, policy, and big-data implications.
      and provide more convincing evidence about its value. Our approach can also be used to assess healthcare utilization for other interventions where current usage is limited and the downstream impact on care may be complicated and unexpected.

      Acknowledgments

      This study was supported by National Institutes of Health grants U19HD077671, K01HG009173, U01HG009599, and R01CA221870 and the American College of Medical Genetics and Genomics Summer Genetics Scholars Program. Members of the BabySeq Project Team also include Pankaj B. Agrawal, Ozge Ceyhan-Birsoy, Shawn Fayer, Leslie A. Frankel, Casie A. Genetti, Amanda M. Gutierrez, Maegan Harden, Ingrid A. Holm, Joel B. Krier, Matthew S. Lebo, Kalotina Machini, Amy L. McGuire, Medha Naik, Tiffany T. Nguyen, Stacey Pereira, Vivek Ramanathan, Heidi L. Rehm, Amy Roberts, Jill O. Robinson, Sergei Roumiantsev, Talia S. Schwartz, Tina K. Truong, Grace E. VanNoy, Susan E. Waisbren, and Timothy W. Yu.

      Supplemental Material

      References

        • Sanders G.D.
        • Neumann P.J.
        • Basu A.
        • et al.
        Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine.
        JAMA. 2016; 316: 1093-1103
        • Glick H.A.
        Economic Evaluation in Clinical Trials.
        2nd ed. Oxford University Press, New York, NY2015
        • Caro J.J.
        • Briggs A.H.
        • Siebert U.
        • et al.
        Modeling good research practices—overview: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-1.
        Med Decis Making. 2012; 32: 667-677
        • Diehr P.
        • Yanez D.
        • Ash A.
        • et al.
        Methods for analyzing health care utilization and costs.
        Annu Rev Public Health. 1999; 20: 125-144
        • Robertson C.
        • Archibald D.
        • Avenell A.
        • et al.
        Systematic reviews of and integrated report on the quantitative, qualitative and economic evidence base for the management of obesity in men.
        Health Technol Assess. 2014; 18 (v-vi, xxiii-xxix): 1-424
        • Nestler-Parr S.
        • Korchagina D.
        • Toumi M.
        • et al.
        Challenges in research and health technology assessment of rare disease technologies: report of the ISPOR Rare Disease Special Interest Group.
        Value Health. 2018; 21: 493-500
        • Phillips K.A.
        • Deverka P.A.
        • Marshall D.A.
        • et al.
        Methodological issues in assessing the economic value of next-generation sequencing tests: many challenges and not enough solutions.
        Value Health. 2018; 21: 1033-1042
        • Ackerman S.L.
        • Koenig B.A.
        Understanding variations in secondary findings reporting practices across U.S. genome sequencing laboratories.
        AJOB Empir Bioeth. 2018; 9: 48-57
        • Ormond K.E.
        • O’Daniel J.M.
        • Kalia S.S.
        Secondary findings: how did we get here, and where are we going?.
        J Genet Couns. 2019; 28: 326-333
        • Delanne J.
        • Nambot S.
        • Chassagne A.
        • et al.
        Secondary findings from whole-exome/genome sequencing evaluating stakeholder perspectives: a review of the literature.
        Eur J Med Genet. 2019; 62: 103529
        • Ramsey S.D.
        • Willke R.J.
        • Glick H.
        • et al.
        Cost-effectiveness analysis alongside clinical trials II—an ISPOR Good Research Practices Task Force report.
        Value Health. 2015; 18: 161-172
        • Reeves P.
        • Doran C.
        • Carey M.
        • et al.
        Costs and cost-effectiveness of targeted, personalized risk information to increase appropriate screening by first-degree relatives of people with colorectal cancer.
        Health Educ Behav. 2019; 46: 798-808
        • Fahr P.
        • Buchanan J.
        • Wordsworth S.
        A review of the challenges of using biomedical big data for economic evaluations of precision medicine.
        Appl Health Econ Health Policy. 2019; 17: 43-452
        • Christensen K.D.
        • Phillips K.A.
        • Green R.C.
        • et al.
        Cost analyses of genomic sequencing: lessons learned from the MedSeq Project.
        Value Health. 2018; 21: 1054-1061
        • Christensen K.D.
        • Vassy J.L.
        • Phillips K.A.
        • et al.
        Short-term costs of integrating whole-genome sequencing into primary care and cardiology settings: a pilot randomized trial.
        Genet Med. 2018; 20: 1544-1553
        • Holm I.A.
        • Agrawal P.B.
        • Ceyhan-Birsoy O.
        • et al.
        The BabySeq project: implementing genomic sequencing in newborns.
        BMC Pediatr. 2018; 18: 225
        • National Human Genome Research Institute
        Talking glossary of genetic terms.
        https://www.genome.gov/genetics-glossary
        Date accessed: November 15, 2019
        • National Cancer Institute
        NCI dictionary of genetics terms.
        • US National Library of Medicine
        What is newborn genomic sequencing?.
        • World Health Organization
        Human genomics in global health.
        • Practitioner Education Working Group of the Clinical Sequencing Exploratory Research (CSER) Consortium
        Introduction and overview to genomic test reports.
        • Ceyhan-Birsoy O.
        • Machini K.
        • Lebo M.S.
        • et al.
        A curated gene list for reporting results of newborn genomic sequencing.
        Genet Med. 2017; 19: 809-818
        • VanNoy G.E.
        • Genetti C.A.
        • McGuire A.L.
        • et al.
        Challenging the current recommendations for carrier testing in children.
        Pediatrics. 2019; 143: S27-S32
        • Holm I.A.
        • McGuire A.
        • Pereira S.
        • et al.
        Returning a genomic result for an adult-onset condition to the parents of a newborn: insights from the BabySeq Project.
        Pediatrics. 2019; 143: S37-S43
        • Pereira S.
        • Robinson J.O.
        • Gutierrez A.M.
        • et al.
        Perceived benefits, risks, and utility of newborn genomic sequencing in the BabySeq Project.
        Pediatrics. 2019; 143: S6-S13
        • Murry J.B.
        • Machini K.
        • Ceyhan-Birsoy O.
        • et al.
        Reconciling newborn screening and a novel splice variant in BTD associated with partial biotinidase deficiency: a BabySeq Project case report.
        Cold Spring Harbor Mol Case Stud. 2018; 4
        • Genetti C.A.
        • Schwartz T.S.
        • Robinson J.O.
        • et al.
        Parental interest in genomic sequencing of newborns: enrollment experience from the BabySeq Project.
        Genet Med. 2019; 21: 622-630
        • Ceyhan-Birsoy O.
        • Murry J.B.
        • Machini K.
        • et al.
        Interpretation of genomic sequencing results in healthy and ill newborns: results from the BabySeq Project.
        Am J Hum Genet. 2019; 104: 76-93
        • Pagon R.A.
        • Tarczy-Hornoch P.
        • Baskin P.K.
        • et al.
        GeneTests-GeneClinics: genetic testing information for a growing audience.
        Hum Mutat. 2002; 19: 501-509
        • Hamosh A.
        • Scott A.F.
        • Amberger J.
        • et al.
        Online Mendelian Inheritance in Man (OMIM).
        Hum Mutat. 2000; 15: 57-61
        • Winn R.J.
        • Botnick W.
        • Dozier N.
        The NCCN guidelines development program.
        Oncology (Williston Park, NY). 1996; 10: 23-28
        • Fox G.N.
        • Moawad N.
        UpToDate: a comprehensive clinical database.
        J Fam Pract. 2003; 52: 706-710
        • Christensen K.D.
        • Bernhardt B.A.
        • Jarvik G.P.
        • et al.
        Anticipated responses of early adopter genetic specialists and nongenetic specialists to unsolicited genomic secondary findings.
        Genet Med. 2018; 20: 1186-1195
        • Green R.C.
        • Berg J.S.
        • Grody W.W.
        • et al.
        ACMG recommendations for reporting of incidental findings in clinical exome and genome sequencing.
        Genet Med. 2013; 15: 565-574
        • Kalia S.S.
        • Adelman K.
        • Bale S.J.
        • et al.
        Recommendations for reporting of secondary findings in clinical exome and genome sequencing, 2016 update (ACMG SF v2.0): a policy statement of the American College of Medical Genetics and Genomics.
        Genet Med. 2017; 19: 249-255
        • Carey D.J.
        • Fetterolf S.N.
        • Davis F.D.
        • et al.
        The Geisinger MyCode community health initiative: an electronic health record-linked biobank for precision medicine research.
        Genet Med. 2016; 18: 906-913
        • Rosen A.B.
        • Cutler D.M.
        Challenges in building disease-based national health accounts.
        Med Care. 2009; 47: S7-S13
        • Leggett L.E.
        • Khadaroo R.G.
        • Holroyd-Leduc J.
        • et al.
        Measuring resource utilization: a systematic review of validated self-reported questionnaires.
        Medicine. 2016; 95: e2759
        • Landrum M.J.
        • Lee J.M.
        • Riley G.R.
        • et al.
        ClinVar: public archive of relationships among sequence variation and human phenotype.
        Nucleic Acids Res. 2014; 42: D980-D985
        • Rehm H.L.
        • Berg J.S.
        • Brooks L.D.
        • et al.
        ClinGen—the Clinical Genome Resource.
        N Engl J Med. 2015; 372: 2235-2242
        • Bhandari A.
        • Wagner T.
        Self-reported utilization of health care services: improving measurement and accuracy.
        Medical Care Res Rev. 2006; 63: 217-235
        • Gama H.
        • Correia S.
        • Lunet N.
        Effect of questionnaire structure on recall of drug utilization in a population of university students.
        BMC Med Res Methodol. 2009; 9: 45
        • Ansah E.K.
        • Powell-Jackson T.
        Can we trust measures of healthcare utilization from household surveys?.
        BMC Public Health. 2013; 13: 853
        • Hunger M.
        • Schwarzkopf L.
        • Heier M.
        • et al.
        Official statistics and claims data records indicate non-response and recall bias within survey-based estimates of health care utilization in the older population.
        BMC Health Serv Res. 2013; 13: 1
        • Mauldin P.D.
        • Guimaraes P.
        • Albin R.L.
        • et al.
        Optimal frequency for measuring health care resource utilization in Parkinson’s disease using participant recall: the FS-TOO resource utilization substudy.
        Clin Ther. 2008; 30: 1553-1557
        • Kjellsson G.
        • Clarke P.
        • Gerdtham U.G.
        Forgetting to remember or remembering to forget: a study of the recall period length in health care survey questions.
        J Health Econ. 2014; 35: 34-46
        • Buchanan J.
        • Wordsworth S.
        • Schuh A.
        Issues surrounding the health economic evaluation of genomic technologies.
        Pharmacogenomics. 2013; 14: 1833-1847
        • Prosser L.A.
        • Grosse S.D.
        • Kemper A.R.
        • et al.
        Decision analysis, economic evaluation, and newborn screening: challenges and opportunities.
        Genet Med. 2013; 14: 703-712
        • Phillips K.A.
        • Trosman J.R.
        • Kelley R.K.
        • et al.
        Genomic sequencing: assessing the health care system, policy, and big-data implications.
        Health Aff (Millwood). 2014; 33: 1246-1253