Advertisement

Lessons Learned When Introducing Pharmacogenomic Panel Testing into Clinical Practice

Open ArchivePublished:November 01, 2016DOI:https://doi.org/10.1016/j.jval.2016.08.727

      Abstract

      Objectives

      Implementing new programs to support precision medicine in clinical settings is a complex endeavor. We describe challenges and potential solutions based on the Indiana GENomics Implementation: an Opportunity for the Underserved (INGenious) program at Eskenazi Health—one of six sites supported by the Implementing GeNomics In pracTicE network grant of the National Institutes of Health/National Human Genome Research Institute. INGenious is an implementation of a panel of genomic tests.

      Methods

      We conducted a descriptive case study of the implementation of this pharmacogenomics program, which has a wide scope (14 genes, 27 medications) and a diverse population (patients who often have multiple chronic illnesses, in a large urban safety-net hospital and its outpatient clinics).

      Challenges

      We placed the clinical pharmacogenomics implementation challenges into six categories: patient education and engagement in care decision making; clinician education and changes in standards of care; integration of technology into electronic health record systems; translational and implementation sciences in real-world clinical environments; regulatory and reimbursement considerations, and challenges in measuring outcomes. A cross-cutting theme was the need for careful attention to workflow. Our clinical setting, a safety-net health care system, presented some distinctive challenges. Patients often had multiple chronic illnesses and sometimes were taking more than one pharmacogenomics-relevant medication. Reaching patients for recruitment or follow-up was another challenge.

      Conclusions

      New, large-scale endeavors in health care are challenging. A description of the challenges that we encountered and the approaches that we adopted to address them may provide insights for those who implement and study innovations in other health care systems.

      Keywords

      Introduction

      Implementing new programs to support precision medicine in clinical settings is a complex endeavor. Strong evidence supports using genetic tests to inform prescribing in some scenarios, but there are many implementation challenges. The US Food and Drug Administration (FDA) has placed genetic testing recommendations and black box warnings on 135 labels [

      US Food and Drug Administration. Table of pharmacogenomic biomarkers in drug labeling. Available from: http://www.fda.gov/Drugs/ScienceResearch/ResearchAreas/Pharmacogenetics/ucm083378.htm. [Accessed April 5, 2016].

      ]. Guidelines are being written regarding gene-drug pairs to inform decisions about switching medications or altering doses [
      • Swen J.J.
      • Nijenhuis M.
      • de Boer A.
      • et al.
      Pharmacogenetics: from bench to byte—an update of guidelines.
      ]. The Clinical Pharmacogenetics Implementation Consortium (CPIC) [
      • Relling M.V.
      • Klein T.E.
      CPIC: Clinical Pharmacogenetics Implementation Consortium of the Pharmacogenomics Research Network.
      ] has published guidelines for 33 medications and is planning for an additional 122 [

      Clinical Pharmacogenetics Implementation Consortium. CPIC™ genes-drugs. Available from: https://cpicpgx.org/genes-drugs/. [Accessed April 5, 2016].

      ]. CPIC helps address one of the barriers to implementation: “lack of clear, curated, peer-reviewed guidelines that translate laboratory test results into actionable prescribing decisions” [
      • Relling M.V.
      • Klein T.E.
      CPIC: Clinical Pharmacogenetics Implementation Consortium of the Pharmacogenomics Research Network.
      ].
      Even when clear guidelines exist, successful implementation requires educating clinicians, making pharmacogenomic information available to clinicians in a timely fashion, and reimbursing providers for pharmacogenomics-related activities [
      • Relling M.V.
      • Klein T.E.
      CPIC: Clinical Pharmacogenetics Implementation Consortium of the Pharmacogenomics Research Network.
      ,
      • Lam Y.W.F.
      Scientific challenges and implementation barriers to translation of pharmacogenomics in clinical practice.
      ,
      • Dunnenberger H.M.
      • Crews K.R.
      • Hoffman J.M.
      • et al.
      Preemptive clinical pharmacogenetics implementation: current programs in five United States medical centers.
      ,
      • Farrugia G.
      • Weinshilboum R.M.
      Challenges in implementing genomic medicine: the Mayo Clinic Center for Individualized Medicine.
      ,
      • Shuldiner A.R.
      • Relling M.V.
      • Hicks J.K.
      • et al.
      for the Pharmacogenomics Research Network Translational Pharmacogenetics Program Group
      The Pharmacogenomics Research Network Translational Pharmacogenetics Program: overcoming challenges of real-world implementation.
      ,
      • Johnson J.A.
      Pharmacogenetics in clinical practice: how far have we come and where are we going?.
      ,
      • Relling M.V.
      • Altman R.B.
      • Goetz M.P.
      • Evans W.E.
      Clinical implementation of pharmacogenomics: overcoming genetic exceptionalism.
      ,
      • Gottesman O.
      • Scott S.A.
      • Ellis S.B.
      • et al.
      The CLIPMERGE PGx Program: clinical implementation of personalized medicine through electronic health records and genomics-pharmacogenomics.
      ,
      • Bielinski S.J.
      • Olson J.E.
      • Pathak J.
      • et al.
      Preemptive genotyping for personalized medicine: design of the right drug, right dose, right time-using genomic data to individualize treatment protocol.
      ,
      • Karnes J.H.
      • Van Driest S.
      • Bowton E.A.
      • et al.
      Using systems approaches to address challenges for clinical implementation of pharmacogenomics.
      ]. In particular, the inconsistent quality and completeness of the data of an electronic health record (EHR) present various challenges including defining phenotype cohorts (with regard to exposures and outcomes) and accounting for patients’ adherence to prescribed medications [
      • Laper S.M.
      • Restrepo N.A.
      • Crawford D.C.
      The challenges in using electronic health records for pharmacogenomics and precision medicine research.
      ]. Nevertheless, many centers are actively working on piloting or implementing pharmacogenomics and are integrating it with EHRs and clinical decision support systems (CDSSs). The Electronic Medical Records and Genomics network is a leading example [
      • Rasmussen-Torvik L.J.
      • Stallings S.C.
      • Gordon A.S.
      • et al.
      Design and anticipated outcomes of the eMERGE-PGx project: a multicenter pilot for preemptive pharmacogenomics in electronic health record systems.
      ]. A survey of 10 sites of this network found that all had been able to incorporate pharmacogenomics into their existing CDSSs, and that delays resulted not from pharmacogenomics per se but rather from more general and typical health information technology (IT) implementation challenges related to staffing levels and communication [
      • Herr T.M.
      • Bielinski S.J.
      • Bottinger E.
      • et al.
      Practical considerations in genomic decision support: the eMERGE experience.
      ].
      The Indiana GENomics Implementation: An Opportunity for the Underserved (INGenious) program at Eskenazi Health is one of six sites supported by the Implementing GeNomics In pracTicE network grant of the National Institutes of Health/National Human Genome Research Institute. INGenious is a testing program for a large panel of genes and medications and seeks to examine its value; it is being integrated, to the extent possible, into an EHR. To address the growing spectrum of guidelines, INGenious configured a custom multigene microarray (for 14 genes and 43 genetic variants) that may allow investigators to evaluate the impact of testing for 27 commonly prescribed pharmacogenetically active medications.
      Unlike most previous pharmacogenomics programs, INGenious is being implemented in a safety-net population with diverse chronic illnesses, and across a broad spectrum of care. The setting is a county-owned urban hospital and its clinic system, with approximately 15,000 hospitalizations and almost 1 million outpatient visits per year. Most patients have publicly financed health care coverage (approximately 40% with Medicaid or Medicare, another 40% with county- or other state-financed care). It is important to study safety-net settings. Here, the introduction of innovative techniques and technologies is often delayed compared with places where employer-based or private coverage predominates. Challenges in patient education, recruitment, and informed consent processes may differ in safety-net settings as well. A recent study in academically affiliated safety-net clinics found that heavy use of EHR computers by clinicians was associated with worse patient satisfaction and less clinician-patient rapport [
      • Ratanawongsa N.
      • Barton J.L.
      • Lyles C.R.
      • et al.
      Association between clinician computer use and communication with patients in safety-net clinics.
      ].
      Although the purpose of this study was to discuss challenges in the clinical implementation of a pharmacogenomics program, an additional aspect of this program’s design was a controlled trial (ClinicalTrials.gov Identifier NCT02297126). We will briefly describe the trial here because it was inextricably intertwined, logistically, with the introduction of the clinical pharmacogenomics program. When a potentially eligible subject is prescribed 1 of the 27 medications, the EHR prompts the clinician (with a pop-up alert) to enroll the patient in the trial. If the clinician agrees, the CDSS randomizes the patient to the control arm or the intervention (genotyping) arm. To help ensure that usual care is provided to subjects in the control arm, those patients are not approached for informed consent. If randomized to the genotyping arm, the patient’s name, location, and incident medication are electronically relayed to a research assistant, who attempts to approach the patient for consent (for genotyping and the trial).
      In this descriptive case study, we describe challenges and the approaches taken to address them during the implementation of the Eskenazi Health pharmacogenomics initiative.

      Methods

      We conducted a descriptive case study of the implementation of a pharmacogenomics program with wide scope (14 genes, 43 variants, and 27 medications) and a diverse population (patients who often have multiple chronic illnesses, in a large urban safety-net hospital and its outpatient clinics). We authors each have a distinct specialty or discipline and reflect varied perspectives including scientific, clinical, economic, medical and molecular genetics, laboratorian, and project management.
      We listed and categorized the program’s challenges (and their potential solutions), in the context of the literature of the implementation of pharmacogenomics (or other types of programs and technologies). The themes that we identified were developed informally at first, on the basis of the regular weekly meetings in which challenges and approaches to addressing them were discussed. We then refined the categorization through iterative discussion. Our charge, a priori, had been to write about challenges in the adoption of new technology. If we consider clinical pharmacogenomics as the new technology, then all the categories of challenges that we identified apply. In contrast, if we consider technology more narrowly (e.g., as EHRs, automated CDSSs, and new hardware and software for laboratory-developed tests [LDTs]), then some of the categories that we identified (e.g., with respect to the education and engagement of patients and clinicians) are not specific to technology but rather are broader challenges, ever-present in the evolution of medical care.

      Challenges

      On the basis of consensus, we placed the clinical pharmacogenomics implementation challenges into six categories: patient education and engagement in care decision making, clinician education and changes in standards of care, integration of technology into EHR systems, translational and implementation sciences in real-world clinical environments, regulatory and reimbursement considerations, and challenges in measuring outcomes.

      Patient Education and Engagement in Care Decision Making

      Educating patients about clinical pharmacogenomics is imperative for both clinical implementation and recruitment into a study. Key steps are to appreciate and to overcome the preconceived ideas that patients have about genetic testing. In this regard, our patient education efforts emphasize how pharmacogenomics data can inform a clinician to make better medication choices. We position the patient’s pharmacogenomics data as a critical tool to ensure that the medication a clinician prescribes is effective and safe. During subject recruitment, we also stress the absolute privacy of a patient’s genetic testing data. We believe that educating patients within this framework of clinical utility and data privacy will be effective. In addition, the research assistants who recruit the patients are available to answer questions that patients may have, and the clinicians who see the patients have received education about our clinical pharmacogenomics program and may serve as an additional source of information.
      On a more operational level, we also recognized that our patients are in our internal medicine and other outpatient clinics for a relatively short period of time, and so any educational efforts must be efficient. We developed bright, colorful, and easy-to-comprehend brochures; we emphasized pictures rather than being narrative in explaining clinical pharmacogenomics and the trial. The brochures are available at check-in (see Supplemental Materials found at doi:10.1016/j.jval.2016.08.727).

      Clinician Education and Changes in Standards of Care

      It was also imperative to identify and to align key clinicians in support of the pharmacogenomics endeavor. Because clinical pharmacogenomics encompasses knowledge from a broad spectrum of medical subspecialties, identifying clinicians with interest and expertise in this field was a challenge. We identified key clinicians on the basis of our awareness of previous research on clinical pharmacology that they had conducted within their subspecialties or within our division of clinical pharmacology. Once identified, these individuals played a critical role in the education of their colleagues and in advocacy for our project. We also successfully engaged the support of higher level leaders (of medical departments and patient care areas) on the hospital campus. Fortunately, hospital and clinic leaders were receptive to our project; they provided us with access and greatly enhanced the reach of our clinician education efforts.
      Clinicians practicing in academic centers range from highly experienced faculty to house staff at various levels of training to medical students. The content of our clinical pharmacogenomics education program had to be suitable for these vastly different learners, and the time allotted for the education programs had to be flexible. In addition, success hinged on engaging the nursing staff—the personnel who administer many of the medications to the patients. We engaged the nursing staff by providing a computer-based clinical pharmacogenomics education program as a component of their required continuing education. The computer-based approach permitted the nurses to fit the program into their schedules. We also presented our clinical pharmacogenomics education program to the large audience at our medical center’s nursing staff meetings.
      Finally, we recognized early on that we needed more than one round of education programs and conferences. We re-engaged clinicians through recurrent clinical pharmacogenomics education programs to sustain the initial momentum and enthusiasm for our project. The recurring programs also ensure that we educate new clinicians and allied health care practitioners.

      Integration of Technology into EHR Systems

      Adding pharmacogenomics reporting to the EHR

      The EHR and CDSS present opportunities and challenges in adding pharmacogenomics test reporting to a patient’s medical record. With 27 drug-gene combinations and 43 variants screened on the open-array platform, a decision we faced was how much to reveal to a patient’s clinician(s) after the enrollment-prompting prescription (the “incident prescription”) and the consequent genotyping: results for the entire genotype or results relevant only to the incident prescription. In the latter approach, the CDSS could convey additional genotype-guided recommendations regarding the other 27 medications at future times—upon initial prescription of any of those medications. Although some of the project investigators argued that a full genotyping report of 43 variants might prove overwhelming to the enrolling clinicians, this concern was counterbalanced by both the liability risk and the questionable feasibility of storing unreleased genotype information for future provision by medical informatics or laboratory personnel. We decided to upload the full genotyping report to the EHR within 7 days of an incident prescription.
      This data transfer is not completely automated. Clinicians still receive the report through their preferred mechanism of fax, email, or US mail. The present genotyping request volume is insufficient to warrant running the array more frequently than weekly. Hence, the “vein to brain” time is delayed upon incident prescriptions. However, upon future first-time prescriptions of other study medications, there is no delay; the CDSS then fires interruptive automated alerts with genotype-guided recommendations immediately.
      To streamline clinicians’ understanding, the genotyping report emphasizes recommendations regarding the incident prescription. Any automated alerts that arise at a later date deal specifically with the additional study medication newly prescribed at that time. Standard language was determined by a committee after reviewing CPIC recommendations and evidence for each genotype and medication. Nuances including 1) drug dosing affected by multiple genes, 2) variable levels of evidence, and 3) the spectrum of genotype effect were synthesized into categorical metabolic phenotypes and were distilled into straightforward recommendations (see Table 1 in Supplemental Materials found at doi:10.1016/j.jval.2016.08.727). Matrices were developed by the project’s clinical investigators to allow the medical informatics team to translate the clinical recommendations (see Table 2 in Supplemental Materials found at doi:10.1016/j.jval.2016.08.727). To avoid alert fatigue, alerts pop up only for atypical dosing recommendations.
      In addition to searchable genotyping reports and automated EHR alerts, actionable genotype results are uploaded to the patient’s EHR problem list to promote greater awareness by clinicians of the genotype data. CDSS automation of the problem list update is planned for a future implementation phase; the additions to the problem list are presently added manually to the EHR by study personnel.

      Using the EHR to identify eligible patients

      One of the first steps in implementing the genotyping (and the study) was to program an automated algorithm to identify the eligible patients in the EHR. This step primarily required prescription data. Most of the 27 study medications have multiple appellations in the EHR (see Table 3 in Supplemental Materials found at doi:10.1016/j.jval.2016.08.727). To identify all the appellations, we combed through the list of all orderable medication names (>3,300) in this EHR. In earlier preparatory work to estimate the volume of pharmacogenomics-relevant medication use, we also reviewed all dispensed medication names (>17,000) [

      Carpenter JS, Rosenman MB, Knisely MR, et al. Prevalence and density of pharmacogenomically actionable medications in a safety net hospital system. SAGE Open Med, 2016 Jan 7;4:2050312115624333

      ]. In these ways we honed the normalization of the different strengths, routes, and combination forms. We checked for both generic and brand names, because this EHR has some of each. Because proton-pump inhibitors are used liberally (in many cases without a clear-cut indication), we sought to limit that cohort to those at higher risk of adverse events related to pharmacogenomic variants; therefore, we required a concomitant International Classification of Diseases, Ninth or Tenth Revision, diagnosis code of interest (e.g., Barrett esophagus, gastric or duodenal ulcers, or Zollinger-Ellison syndrome).
      A related challenge was the inclusion criterion that the patient be a new starter of the medication of interest. We wrote the algorithm to look back 13 months from a prescription for a study medication, to rule out previous prescription of the same medication. We initially used 12 months but found that length to be insufficient. Some patients with no refill in 12 months were obtaining a refill at, or just after, the 1-year mark.
      Because, by design, the implementation was limited to the Eskenazi Health main campus, we needed to use the clinic location fields in the EHR to exclude the outlying clinics, and (for main campus) to notify the research assistants where to find the newly eligible patients with an incident prescription. We sifted through 502 location names or abbreviations (including some that were still in use in the EHR even though the referent building had been decommissioned). In the intervention arm, a research assistant reviews the site designation and weeds out off-campus sites that slip through.

      Enrolling patients, assisted by the EHR and the CDSS

      There are substantial parallel challenges in the enrollment workflow. Clinician workflow is crucial. At first a simple pop-up message within the EHR alerted the clinician that a patient qualified for the study. The clinician was asked to check a box on the computer screen documenting his or her agreement to include the patient in the study. Early on, clinicians signified agreement only 40% of the time. To understand this low rate, two investigators met with clinicians and asked open-ended questions about their concerns. The clinicians said that the alert did not clearly state what additional work, if any, would be required if they entered a patient into the study. They also said that the extra keystrokes that were required to signify agreement took time. They found it easier to exit the screen without completing the process.
      On the basis of the feedback, the alert was revised to state that enrollment would require no additional study-related effort. (The clinicians knew that they might later receive a pharmacogenomics report.) The logistics of the agreement step were also revised, so that the clinician had to uncheck the box if he or she did not want the patient to be enrolled. The clinician “yes” rate immediately increased from 40% to 80% and has remained above 60% since these changes. Early testing of the recruitment technology by the clinicians in their actual workflow, along with early discussions with them about how they would like to participate in the recruitment process, is critical.
      Clinicians and clinic staff may have additional interests that compete with the introduction and evaluation of new endeavors. A clinician may wish to enroll a patient and to receive a genotyping report but may perceive that the time required for a research assistant to obtain informed consent may adversely affect patient flow in the clinic.
      The biggest workflow challenge in the implementation has been trial-related, because the implementation and the trial are intertwined. Among the subjects approved for enrollment by the clinicians and randomized to the intervention arm, most complete their health care visit and leave the clinic before a research assistant can arrive to obtain informed consent and a bio-specimen. As a result, most (90%) of the patients enrolled in the (intention-to-treat) trial have not been approached for genotyping. Approachability has supplanted randomization as the principal factor that separates the genotyped group from the control group. This challenge arises because the genotyping technology intervention was introduced in a study requiring randomization and, in only the genotyping arm, informed consent. If genotyping were being introduced universally across a health system, the logistics challenges in making patients aware of the technology and in obtaining bio-specimens may be fewer.

      Transition to a large vendor-based EHR

      The health system in which this pharmacogenomics program was implemented plans to discontinue its present longstanding EHR system and CDSS later this year—about 18 months after the pharmacogenomics program began. A large vendor-based EHR will be installed. This change will require the project to adapt its approach for identifying eligible patients, obtaining approval from clinicians to approach patients, enrolling patients, presenting clinical results and recommendations to clinicians, and analyzing data. The introduction of a new EHR is a major disruption. A recent study of the transition from a homegrown to a vendor EHR in a leading university health care system found that physicians had substantial concerns about the EHR’s effects on workflow and patient safety for at least 2 years after it was introduced [

      Hanauer DA, Branford GL, Greenberg G, et al. Two-year longitudinal assessment of physicians’ perceptions after replacement of a longstanding homegrown electronic health record: does a J-curve of satisfaction really exist? J Am Med Inform Assoc (published online ahead of print July 3, 2016). 10.1093/jamia/ocw077

      ]. When a vendor EHR has many functions, physicians find themselves under increasing time pressure and stress [
      • Babbott S.
      • Manwell L.B.
      • Brown R.
      • et al.
      Electronic medical records and physician stress in primary care: results from the MEMO Study.
      ].

      Translational and Implementation Sciences in Real-World Clinical Environments

      The pharmacogenomics project afforded funding for robust CDSS implementation. Nevertheless, to overcome clinicians’ concerns as well as lack of experience in pharmacogenomics, we also established a pharmacogenomics consult service [
      • Eadon M.
      • Desta Z.
      • Levy K.
      • et al.
      Implementation of a pharmacogenomics consult service to support the INGenious trial.
      ]. All genotyping reports and CDSS automated alerts contain a pager number for the consult service, should the clinician have questions.
      The decision to provide clinicians with an entire genotype panel can result in significant downstream clinical impact. Clinicians receive additional information they did not request unrelated to the incident prescription. Often this additional information will prove irrelevant; however, many of the patients are on multiple medications, and incidental findings have the potential to affect usage of their existing medications. It has not yet been possible for the medical informatics team to automate the presentation of incidental findings. Furthermore, the responsibility to identify these incidental findings should not be shifted to the clinician who enrolled the patient. The liability associated with such a shift might deter clinicians from agreeing to enroll their patients. For these reasons, the project team built an adjudication system in which all genotypes and medication lists are manually reviewed upon initial genotyping [
      • Eadon M.
      • Desta Z.
      • Levy K.
      • et al.
      Implementation of a pharmacogenomics consult service to support the INGenious trial.
      ], and actionable results are conveyed to an appropriate clinician. Upon any future first-time prescription of a study medication, if the genotyping results mean that a change in medication or in dose is indicated (see Tables 1 and 2 in Supplemental Materials), then the CDSS will fire an alert to the clinician. As a result, recurrent screening for incidental findings is unnecessary.
      An additional limitation in the real-world clinical environment is the accuracy of the medication lists in the EHR. A preliminary analysis of the first 127 enrollees (see Table 4 in Supplemental Materials found at doi:10.1016/j.jval.2016.08.727) suggests that 37% of medication lists include at least one medication that the patient is no longer using. (In addition, the medication lists in the EHR in this health system include nonmedications such as medical devices, supplies, and vaccines). These inaccuracies pose a challenge in that actionable drug-gene warnings potentially could be issued for medications that are not presently being taken by the patient. Such warnings may be significant should the patient receive future prescriptions that have a drug-gene conflict; false alerts, however, contribute to physician alert fatigue and thus could reduce attention to all alerts when received.

      Regulatory and Reimbursement Considerations

      Beyond the mechanics of implementing a pharmacogenomics-based alert system in the Indiana setting described earlier, there are additional challenges to more general implementations that arise in the present regulatory and business environment. Most new or innovative tests, such as pharmacogenomics tests, are developed in clinical laboratories. In October 2014, the FDA released a draft guidance document that sets forth a proposed framework to regulate LDTs [
      US Food and Drug Administration
      ]. The FDA’s proposed framework defines LDTs as a class of in vitro diagnostics and therefore makes them subject to the medical device regulations in the Federal Food Drug & Cosmetic Act. Since the 1970s, the FDA has overseen the safety and effectiveness of devices such as implants, hearing aids, and surgical tools, whereas its requirements regarding new diagnostic tests have applied only to manufacturers that develop, box, and distribute kits to clinical laboratories. But the new guidance on LDTs, if finalized, would force all innovative molecular testing services such as pharmacogenomics into the FDA’s framework. In addition, clinical researchers in studies such as ours would be required to obtain investigational device exemptions (IDEs) before studies of new tests begin. An IDE allows a device to be used in a clinical study to collect safety and effectiveness data [

      US Food and Drug Administration. Device advice: investigational device exemption. Available from: http://www.fda.gov/medicaldevices/deviceregulationandguidance/howtomarketyourdevice/investigationaldeviceexemptionide/default.htm. [Accessed April 6, 2016]

      ]. The IDE requirement would delay research studies and would impose additional costs.
      With regard to reimbursement for clinical pharmacogenomics services, Medicare is the largest payer, and its decisions influence commercial payers as well as state Medicaid programs. Most Medicare coverage determinations are made at a local/regional level by Medicare Administrative Contractors. Medicare is required by law to pay only for items and services that are “reasonable and necessary,” which is interpreted generally as those that improve clinically meaningful health outcomes. This project may add to the evidence base for such outcomes.

      Measuring Outcomes

      The business case for pharmacogenomics testing will hinge upon a reduction in the incidence of costly adverse drug events (ADEs). The business case for other medication safety interventions has a substantial literature in which reduction of the costs of ADEs plays a central role [
      • Seidling H.M.
      • Bates D.W.
      Evaluating the impact of health IT on medication safety.
      ]. A study of the introduction of computerized physician order entry in community hospitals reported a moderate return on investment (via reduction of the costs of ADEs); cost-effectiveness was limited by the fact that the accompanying decision support had not yet been fully locally customized for the hospitals in the study [
      • Zimlichman E.
      • Keohane C.
      • Franz C.
      • et al.
      Return on investment for vendor computerized physician order entry in four community hospitals: the importance of decision support.
      ]. Other studies have modeled reductions in ADEs to demonstrate the cost-effectiveness of innovations such as computerized physician order entry in ambulatory care [
      • Forrester S.H.
      • Hepp Z.
      • Roth J.A.
      Cost-effectiveness of a computerized provider order entry system in improving medication safety ambulatory care.
      ] or bar-coding for medication dispensing in hospital pharmacies [
      • Maviglia S.M.
      • Yoo J.Y.
      • Franz C.
      • et al.
      Cost-benefit analysis of a hospital pharmacy bar code solution.
      ]. As Seidling and Bates [
      • Seidling H.M.
      • Bates D.W.
      Evaluating the impact of health IT on medication safety.
      ] point out, because health IT interventions for medication safety are complex, “their impact depends crucially on the real-world clinical setting and the implementation details and thus, transferability of study results is variable.” This dictum is no less true for pharmacogenomics, a complex intervention involving health IT plus other systemic changes [
      • Lam Y.W.F.
      Scientific challenges and implementation barriers to translation of pharmacogenomics in clinical practice.
      ,
      • Dunnenberger H.M.
      • Crews K.R.
      • Hoffman J.M.
      • et al.
      Preemptive clinical pharmacogenetics implementation: current programs in five United States medical centers.
      ,
      • Farrugia G.
      • Weinshilboum R.M.
      Challenges in implementing genomic medicine: the Mayo Clinic Center for Individualized Medicine.
      ,
      • Shuldiner A.R.
      • Relling M.V.
      • Hicks J.K.
      • et al.
      for the Pharmacogenomics Research Network Translational Pharmacogenetics Program Group
      The Pharmacogenomics Research Network Translational Pharmacogenetics Program: overcoming challenges of real-world implementation.
      ,
      • Johnson J.A.
      Pharmacogenetics in clinical practice: how far have we come and where are we going?.
      ,
      • Relling M.V.
      • Altman R.B.
      • Goetz M.P.
      • Evans W.E.
      Clinical implementation of pharmacogenomics: overcoming genetic exceptionalism.
      ,
      • Gottesman O.
      • Scott S.A.
      • Ellis S.B.
      • et al.
      The CLIPMERGE PGx Program: clinical implementation of personalized medicine through electronic health records and genomics-pharmacogenomics.
      ,
      • Bielinski S.J.
      • Olson J.E.
      • Pathak J.
      • et al.
      Preemptive genotyping for personalized medicine: design of the right drug, right dose, right time-using genomic data to individualize treatment protocol.
      ,
      • Karnes J.H.
      • Van Driest S.
      • Bowton E.A.
      • et al.
      Using systems approaches to address challenges for clinical implementation of pharmacogenomics.
      ,
      • Laper S.M.
      • Restrepo N.A.
      • Crawford D.C.
      The challenges in using electronic health records for pharmacogenomics and precision medicine research.
      ,
      • Rasmussen-Torvik L.J.
      • Stallings S.C.
      • Gordon A.S.
      • et al.
      Design and anticipated outcomes of the eMERGE-PGx project: a multicenter pilot for preemptive pharmacogenomics in electronic health record systems.
      ,
      • Herr T.M.
      • Bielinski S.J.
      • Bottinger E.
      • et al.
      Practical considerations in genomic decision support: the eMERGE experience.
      ].
      In reducing ADEs, the addition of a properly designed pharmacogenomics application to the EHR may play an essential role; the application may provide alerts and recommendations for therapy modifications that are easy for providers to understand without causing alert fatigue. Although economic evaluations have been conducted for a limited number of pharmacogenomic interventions [
      • Towse A.
      • Ossa D.
      • Veenstra D.
      • et al.
      Understanding the economic value of molecular diagnostic tests: case studies and lessons learned.
      ], the economic value of the approach has yet to be broadly established [
      • Snyder S.R.
      • Mittropoulou C.
      • Patrinos G.P.
      • Williams M.S.
      Economic evaluation of pharmacogenomics: a value-based approach to pragmatic decision making in the face of complexity.
      ,
      • Veenstra D.L.
      The value of routine pharmacogenomic screening—are we there yet? A perspective on the costs and benefits of routine screening—shouldn’t everyone have this done?.
      ,
      • Berm E.J.J.
      • Looff Md
      • Wilffert B.
      • et al.
      Economic evaluations of pharmacogenetic and pharmacogenomic screening tests: a systematic review. Second update of the literature.
      ]. Evidence to build such a business case has been limited because previous studies have not captured the full range of economic effects of such testing. Three aspects of our study present particular challenges: 1) the use of a multigene rather than a single-gene intervention, 2) implementation in a population with complex social and medical comorbid conditions, and 3) extraction of relevant ADE and cost data from EHRs. When an intervention involves a panel of potential drug-gene interactions, each associated with disparate potential ADEs, there are challenges in knowing whether particular clinical events were ADEs, especially if one is relying on secondary analysis of EHR data. In this pharmacogenomics program, most of the patients have complex health care needs, may be taking multiple medications, and may have additional, unrelated risk factors for the putative ADEs.
      Use of total health care costs as an outcome measure removes the need to identify causal pathways between genotype and outcomes. It allows an overall comparison of intervention and control groups. At the same time, the variability of health care costs, especially in an underserved population, increases the size of the sample needed to detect an intervention effect. Use of costs as an outcome measure requires access to health care charge and utilization data. Because this pharmacogenomics program was implemented in a health care system that serves a disadvantaged population with few other care options, and because EHR and supplemental administrative charge data are available, we expect to be able to capture costs across the care spectrum, including inpatient and outpatient events.

      Conclusions

      A period of 17 years is often cited as the typical lag time between the development of new evidence and its translation into clinical care [
      • Morris Z.S.
      • Wooding S.
      • Grant J.
      The answer is 17 years, what is the question: understanding time lags in translational research.
      ,
      • Westfall J.
      • Mold J.
      • Fagnan L.
      Practice-based research—“Blue Highways” on the NIH roadmap.
      ]. But implementation challenges are diverse and nuanced, and lag times vary from almost 0 to much longer than 17 years—the frameworks and methods for even measuring and understanding implementation delays are in their infancy [
      • Green L.
      • Ottoson J.
      • García C.
      • Hiatt R.
      Diffusion theory and knowledge dissemination, utilization, and integration in public health.
      ,
      • Waltz T.J.
      • Powell B.J.
      • Chinman M.J.
      • et al.
      Expert Recommendations for Implementing Change (ERIC): protocol for a mixed methods study.
      ,
      • Waltz T.J.
      • Powell B.J.
      • Matthieu M.M.
      • et al.
      Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study.
      ]. At the same time, however, technology (or any new intervention) is sometimes implemented hastily or is applied too widely in practice—without adequate evidence or evaluation [
      • Prasad V.
      • Vandross A.
      • Toomey C.
      • et al.
      A decade of reversal: an analysis of 146 contradicted medical practices.
      ,
      • Katz J.N.
      • Brophy R.H.
      • Chaisson C.E.
      • et al.
      Surgery versus physical therapy for a meniscal tear and osteoarthritis.
      ]. Prasad and Cifu [
      • Prasad V.K.
      • Cifu A.S.
      Ending Medical Reversal: Improving Outcomes, Saving Lives.
      ] estimate that perhaps 40% of what are considered to be beneficial new health care interventions may be proven, someday, not to be worthwhile.
      Controversies regarding implementation are easy to find. For example, there is the debate over whether genetic information adds value in guiding the use of warfarin [
      • Kimmel S.E.
      Warfarin pharmacogenomics: current best evidence.
      ]. At the societal level, health IT on a large scale was expected to foster a new era of efficient, high-quality health care. But in a 2015 letter, 23 professional organizations including the American Medical Association and the American College of Surgeons complained that despite its $30 billion investment, the United States had substantially underestimated the risks that rapid EHR implementation has caused for patient safety [
      American Medical Association
      AMDA—the Society for Post-Acute and Long-Term Care Medicine, American Academy of Allergy, Asthma and Immunology, et al. Letter to Karen B. DeSalvo, National Coordinator for Health Information Technology.
      ].
      It is in these contexts that we offer challenges, potential solutions, and lessons learned regarding the implementation of technology in the Eskenazi Health/Indiana program. As did other programs within the Implementing GeNomics In pracTicE network, we found that the implementation of a pharmacogenomic intervention was facilitated by having a multidisciplinary team guide the project, by integrating pharmacogenomics into the clinical workflow as smoothly as possible, and by supplementing automated CDSS with an opportunity for frontline clinicians to consult physicians with expertise in pharmacogenomics [
      • Weitzel K.W.
      • Elsey A.R.
      • Langaee T.Y.
      • et al.
      Clinical pharmacogenetics implementation: approaches, successes, and challenges.
      ]. However, our clinical setting, a safety-net health care system, presented some unique challenges. For instance, many patients were difficult to locate or to contact, even immediately after a clinic visit, for follow-up. Although most patients have cell phones, we found that many have pay-by-the-minute plans and therefore use the phones sparingly. Willingness to consent to genomic testing may also differ in safety-net versus other settings.
      A recent review suggests that “achieving the goal of [widespread] personalized therapy remains years, if not decades, away” [
      • Lam Y.W.F.
      Scientific challenges and implementation barriers to translation of pharmacogenomics in clinical practice.
      ]. We see the strong potential for pharmacogenomics to improve health and health care and to be cost-effective; we also appreciate the many challenges in bringing pharmacogenomics into wider use, and in evaluating its use. We think that the field would benefit from more discussion of these practical considerations. We hope that our experience will be helpful for others.

      Acknowledgments

      We thank Dr Kathryn Phillips and Michael Douglas for their insightful review of the manuscript. The late David Flockhart, MD, PhD, will always be in our thoughts for his mentorship, his leadership in establishing pharmacogenomics in Indiana (including the U01 project), and his support for the idea of this article.
      Source of financial support: This project was supported by Award No. 5U01HG007762 from the National Human Genome Research Institute. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Human Genome Research Institute or the National Institutes of Health.

      Supplemental Materials

      References

      1. US Food and Drug Administration. Table of pharmacogenomic biomarkers in drug labeling. Available from: http://www.fda.gov/Drugs/ScienceResearch/ResearchAreas/Pharmacogenetics/ucm083378.htm. [Accessed April 5, 2016].

        • Swen J.J.
        • Nijenhuis M.
        • de Boer A.
        • et al.
        Pharmacogenetics: from bench to byte—an update of guidelines.
        Clin Pharmacol Ther. 2011; 89: 662-673
        • Relling M.V.
        • Klein T.E.
        CPIC: Clinical Pharmacogenetics Implementation Consortium of the Pharmacogenomics Research Network.
        Clin Pharmacol Ther. 2011; 89: 464-467
      2. Clinical Pharmacogenetics Implementation Consortium. CPIC™ genes-drugs. Available from: https://cpicpgx.org/genes-drugs/. [Accessed April 5, 2016].

        • Lam Y.W.F.
        Scientific challenges and implementation barriers to translation of pharmacogenomics in clinical practice.
        ISRN Pharmacol. 2013; 2013: 641089
        • Dunnenberger H.M.
        • Crews K.R.
        • Hoffman J.M.
        • et al.
        Preemptive clinical pharmacogenetics implementation: current programs in five United States medical centers.
        Annu Rev Pharmacol Toxicol. 2015; 55: 89-106
        • Farrugia G.
        • Weinshilboum R.M.
        Challenges in implementing genomic medicine: the Mayo Clinic Center for Individualized Medicine.
        Clin Pharmacol Ther. 2013; 94: 204-206
        • Shuldiner A.R.
        • Relling M.V.
        • Hicks J.K.
        • et al.
        • for the Pharmacogenomics Research Network Translational Pharmacogenetics Program Group
        The Pharmacogenomics Research Network Translational Pharmacogenetics Program: overcoming challenges of real-world implementation.
        Clin Pharmacol Ther. 2013; 94: 207-210
        • Johnson J.A.
        Pharmacogenetics in clinical practice: how far have we come and where are we going?.
        Pharmacogenomics. 2013; 14: 835-843
        • Relling M.V.
        • Altman R.B.
        • Goetz M.P.
        • Evans W.E.
        Clinical implementation of pharmacogenomics: overcoming genetic exceptionalism.
        Lancet Oncol. 2010; 11: 507-509
        • Gottesman O.
        • Scott S.A.
        • Ellis S.B.
        • et al.
        The CLIPMERGE PGx Program: clinical implementation of personalized medicine through electronic health records and genomics-pharmacogenomics.
        Clin Pharmacol Ther. 2013; 94: 214-217
        • Bielinski S.J.
        • Olson J.E.
        • Pathak J.
        • et al.
        Preemptive genotyping for personalized medicine: design of the right drug, right dose, right time-using genomic data to individualize treatment protocol.
        Mayo Clin Proc. 2014; 89: 25-33
        • Karnes J.H.
        • Van Driest S.
        • Bowton E.A.
        • et al.
        Using systems approaches to address challenges for clinical implementation of pharmacogenomics.
        Wiley Interdiscip Rev Syst Biol Med. 2014; 6: 125-135
        • Laper S.M.
        • Restrepo N.A.
        • Crawford D.C.
        The challenges in using electronic health records for pharmacogenomics and precision medicine research.
        Pac Symp Biocomput. 2016; 21: 369-380
        • Rasmussen-Torvik L.J.
        • Stallings S.C.
        • Gordon A.S.
        • et al.
        Design and anticipated outcomes of the eMERGE-PGx project: a multicenter pilot for preemptive pharmacogenomics in electronic health record systems.
        Clin Pharmacol Ther. 2014; 96: 7
        • Herr T.M.
        • Bielinski S.J.
        • Bottinger E.
        • et al.
        Practical considerations in genomic decision support: the eMERGE experience.
        J Pathol Inform. 2015; 6: 50
        • Ratanawongsa N.
        • Barton J.L.
        • Lyles C.R.
        • et al.
        Association between clinician computer use and communication with patients in safety-net clinics.
        JAMA Intern Med. 2016; 176: 125-128
      3. Carpenter JS, Rosenman MB, Knisely MR, et al. Prevalence and density of pharmacogenomically actionable medications in a safety net hospital system. SAGE Open Med, 2016 Jan 7;4:2050312115624333

      4. Hanauer DA, Branford GL, Greenberg G, et al. Two-year longitudinal assessment of physicians’ perceptions after replacement of a longstanding homegrown electronic health record: does a J-curve of satisfaction really exist? J Am Med Inform Assoc (published online ahead of print July 3, 2016). 10.1093/jamia/ocw077

        • Babbott S.
        • Manwell L.B.
        • Brown R.
        • et al.
        Electronic medical records and physician stress in primary care: results from the MEMO Study.
        J Am Med Inform Assoc. 2014; 21: e100-e106
        • Eadon M.
        • Desta Z.
        • Levy K.
        • et al.
        Implementation of a pharmacogenomics consult service to support the INGenious trial.
        Clin Pharmacol Ther. 2016; 100: 63-66
        • US Food and Drug Administration
        Draft guidance for industry, Food and Drug Administration Staff, and clinical laboratories: framework for regulatory oversight of laboratory developed tests (LDTs). October 3, 2014; (Available from: http://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm416685.pdf.[Accessed April 6, 2016])
      5. US Food and Drug Administration. Device advice: investigational device exemption. Available from: http://www.fda.gov/medicaldevices/deviceregulationandguidance/howtomarketyourdevice/investigationaldeviceexemptionide/default.htm. [Accessed April 6, 2016]

        • Seidling H.M.
        • Bates D.W.
        Evaluating the impact of health IT on medication safety.
        Stud Health Technol Inform. 2016; 222: 195-205
        • Zimlichman E.
        • Keohane C.
        • Franz C.
        • et al.
        Return on investment for vendor computerized physician order entry in four community hospitals: the importance of decision support.
        Jt Comm J Qual Patient Saf. 2013; 39: 312-318
        • Forrester S.H.
        • Hepp Z.
        • Roth J.A.
        Cost-effectiveness of a computerized provider order entry system in improving medication safety ambulatory care.
        Value Health. 2014; 17: 340-349
        • Maviglia S.M.
        • Yoo J.Y.
        • Franz C.
        • et al.
        Cost-benefit analysis of a hospital pharmacy bar code solution.
        Arch Intern Med. 2007; 167: 788-794
        • Towse A.
        • Ossa D.
        • Veenstra D.
        • et al.
        Understanding the economic value of molecular diagnostic tests: case studies and lessons learned.
        J Pers Med. 2013; 3: 288-305
        • Snyder S.R.
        • Mittropoulou C.
        • Patrinos G.P.
        • Williams M.S.
        Economic evaluation of pharmacogenomics: a value-based approach to pragmatic decision making in the face of complexity.
        Public Health Genomics. 2014; 17: 256-264
        • Veenstra D.L.
        The value of routine pharmacogenomic screening—are we there yet? A perspective on the costs and benefits of routine screening—shouldn’t everyone have this done?.
        Clin Pharmacol Ther. 2016; 99: 164-166
        • Berm E.J.J.
        • Looff Md
        • Wilffert B.
        • et al.
        Economic evaluations of pharmacogenetic and pharmacogenomic screening tests: a systematic review. Second update of the literature.
        PLoS One. 2016; 11: e0146262
        • Morris Z.S.
        • Wooding S.
        • Grant J.
        The answer is 17 years, what is the question: understanding time lags in translational research.
        J R Soc Med. 2011; 104: 510-520
        • Westfall J.
        • Mold J.
        • Fagnan L.
        Practice-based research—“Blue Highways” on the NIH roadmap.
        JAMA. 2007; 297: 403-406
        • Green L.
        • Ottoson J.
        • García C.
        • Hiatt R.
        Diffusion theory and knowledge dissemination, utilization, and integration in public health.
        Annu Rev Public Health. 2009; 30: 151-174
        • Waltz T.J.
        • Powell B.J.
        • Chinman M.J.
        • et al.
        Expert Recommendations for Implementing Change (ERIC): protocol for a mixed methods study.
        Implement Sci. 2015; 10: 21
        • Waltz T.J.
        • Powell B.J.
        • Matthieu M.M.
        • et al.
        Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study.
        Implement Sci. 2015; 10: 109
        • Prasad V.
        • Vandross A.
        • Toomey C.
        • et al.
        A decade of reversal: an analysis of 146 contradicted medical practices.
        Mayo Clin Proc. 2013; 88: 790-798
        • Katz J.N.
        • Brophy R.H.
        • Chaisson C.E.
        • et al.
        Surgery versus physical therapy for a meniscal tear and osteoarthritis.
        N Engl J Med. 2013; 368: 1675-1684
        • Prasad V.K.
        • Cifu A.S.
        Ending Medical Reversal: Improving Outcomes, Saving Lives.
        Johns Hopkins University Press, Baltimore, MD2015
        • Kimmel S.E.
        Warfarin pharmacogenomics: current best evidence.
        J Thromb Haemost. 2015; 13: S266-S271
        • American Medical Association
        AMDA—the Society for Post-Acute and Long-Term Care Medicine, American Academy of Allergy, Asthma and Immunology, et al. Letter to Karen B. DeSalvo, National Coordinator for Health Information Technology.
        US Department of Health and Human Services, January 21, 2015 ([Accessed April 3, 2016])
        • Weitzel K.W.
        • Elsey A.R.
        • Langaee T.Y.
        • et al.
        Clinical pharmacogenetics implementation: approaches, successes, and challenges.
        Am J Med Genet C Semin Med Genet. 2014; 166C: 56-67