If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Address correspondence to: Dean A. Regier, Canadian Centre for Applied Research in Cancer Control, Cancer Control Research, BC Cancer, 675 West 10th Avenue, Vancouver, British Columbia, Canada V5Z 1L3.
Canadian Centre for Applied Research in Cancer Control, Cancer Control Research, BC Cancer, Vancouver, British Columbia, CanadaSchool of Population and Public Health, University of British Columbia, Vancouver, British Columbia, Canada
Next-generation sequencing (NGS) technologies have seen variable adoption in the clinic. This is partly due to a lack of clinical and economic studies, with the latter increasingly challenged to examine patient preferences for health and nonhealth outcomes (e.g., false-positive rate).
To conduct a structured review of studies valuing patients’ preference-based utility for NGS outcomes, to highlight identified methodological challenges, and to consider how studies addressed identified challenges.
We searched MEDLINE (PubMed), Embase (Ovid), and Web of Science for published studies examining outcomes from health care decisions informed by NGS. We focused our search on direct elicitations of preference-based utility. We reviewed included studies and qualitatively grouped and summarized stated challenges and solutions by theme.
Eleven studies were included. Most of them (n = 6) used discrete choice experiments to value utility. We categorized challenges into four themes: 1) valuing the full range of NGS outcomes, 2) accounting for accuracy and uncertainty surrounding effectiveness, 3) allowing for simultaneous multiple and cascading risks, and 4) incorporating downstream consequences. Studies found strong evidence of utility for NGS information, regardless of health improvement. Investigators addressed challenges by simplifying complex choices, by including health outcomes alongside nonhealth outcomes, and by using multiple elicitation techniques.
The breadth and complexity of NGS-derived information makes the technology a unique and challenging application for utility valuation. Failing to account for the utility or disutility of NGS-related nonhealth outcomes may lead to overinvestment or underinvestment in NGS, and so there is a need for research addressing unresolved challenges.
Next-generation sequencing (NGS) is an umbrella term for massively parallel DNA sequencing technologies. The result of the application of NGS is information obtained from simultaneously interrogating multiple genes or the whole genome and their biological inter-relationships. Although NGS shows promise for more accurate patient stratification, the translation of NGS into the clinic has been variable [
]. The variability in uptake has been attributed to a lack of evidence base demonstrating clinical effectiveness, clinical utility, and cost effectiveness.
Health technology assessment (HTA) guidelines typically stipulate that off-the-shelf instruments should inform quality-adjusted life-years (QALYs) when answering questions of cost effectiveness. These instruments might not capture all the benefit-risk trade-offs of NGS health, nonhealth, and process outcomes. Buchanan et al. [
] highlighted that measures informing QALYs do not incorporate preferences for nonhealth outcomes (e.g., false-positive rate) or process outcomes (e.g., time waiting for results). This observation is important in context of the assertion by Marshall et al. [
] that the value of NGS depends on the information that patients receive and the benefits that patients and providers ascribe to NGS information.
Recently, the Second Panel on Cost Effectiveness in Health and Medicine made allowance for an economic evaluation reference case that takes account of nonhealth outcomes. The panel noted that decision makers need a “quantification and valuation of all health and non-health effects of interventions” [
]. In principle, this recommendation supports including preference-based utility in economic evaluation beyond what off-the-shelf instruments usually encapsulate.
The types of outcomes that NGS produces are challenging to value, however. This is because NGS has the potential to uncover a multitude of complex clinically and nonclinically actionable results with far-ranging personal and familial implications. Given the import and complexity of NGS information, our objectives were to 1) conduct a structured review of studies valuing the preference-based utility of NGS health, nonhealth, or process outcomes from consumers’ perspectives; 2) highlight the conceptual and methodological challenges these studies encountered when estimating utility; and 3) consider how the included studies addressed the conceptual and stated challenges.
We conducted a literature search of full-text peer-reviewed articles in MEDLINE (PubMed), Embase (Ovid), and Web of Science. We restricted our search to articles in English published between January 1, 2005, and December 31, 2017. We chose the year 2005 because this was the year that NGS was being implemented in research settings. Our search strategy is outlined in the Appendix in Supplemental Materials found at doi:10.1016/j.jval.2018.06.010. After initial identification, we imported all articles into EndNote X6. Two of the researchers independently evaluated the title and abstract of all publications to identify articles for inclusion. We limited the search to direct elicitation of preference-based utility. We excluded studies that did not estimate stated preferences, did not focus on patient and/or general public perspectives, or did not focus on NGS. We identified stated challenges through authors’ statements on the motivation for estimating utility and in the discussion of study limitations. Using directed content analysis and the study by Marshall et al. [
], we grouped challenges according to categories. Solutions were based on study design and analytic approach, on next steps discussed for research, and on feedback from the working group.
Study Acquisition Flow
Figure 1 presents the flow of the included studies. The PubMed search identified 105 records. Four additional records were identified from searches in Ovid (MEDLINE) and Web of Science, as well as from citations in key articles. After screening titles and abstracts, 82 records were excluded and 27 full-text articles were assessed for eligibility. Of these, 11 studies directly elicited preference-based utility to examine the value of NGS health and nonhealth outcomes. Reasons for exclusion were studies not specifically examining NGS (n = 11) or not focusing on preferences from public or patient perspectives (n = 5).
Clinical context, end points, and perspective
Detailed characteristics of each study are available in the Appendix in Supplemental Materials. The clinical contexts included NGS for prenatal testing, genomic testing to inform cancer interventions, and return of genomic information irrespective of disease. Of the included studies, 36% examined preferences from the general population’s perspective, 46% focused on the perspectives of patients or their families, and 18% examined both perspectives. The studies specified a number of end points, including preference-based utility, predicted uptake, and willingness to pay. These end points were chosen for various reasons. Four studies anticipated that their results would be used as inputs in economic evaluation. Two studies aimed to inform shared decision making, one study aimed to guide policy, two studies sought to inform early-stage technology development and investment, and two studies did not explicitly discuss the reason for preference elicitation.
Methods and approaches to elicit preferences
Figure 2 provides an overview of the applied methods, end points, and their potential uses within economic evaluation. The methods used were discrete choice experiments (DCEs; n = 6), contingent valuation (CV; n = 1), time trade-off (n = 1), as well as a combination of DCE, CV, probability trade-off, and/or ranking exercises (n = 3). Health, nonhealth, and process outcomes were identified through a combination of literature review, focus groups, in-person interviews, pilot testing, and expert opinion (see the Appendix in Supplemental Materials). Two studies did not state how they determined relevant outcomes. Most studies incorporated attributes for health, nonhealth, or process outcomes (n = 9). Attributes pertaining to health-related quality of life were included in four studies and involved likely benefit from treatment, likelihood of treatment side effects, complication rate, or pregnancy-specific outcomes. Health-related attributes described the risk of developing the disease after identifying a variant (n = 8), actionability of the genomic variant (n = 4), severity of the identified disease (n = 4), and/or carrier implications (n = 2). Nonhealth attributes included cost (n = 5), turnaround time (n = 3), type of procedure (n = 3), test reliability (n = 2), level of physician support or shared decision making (n = 2), and genetic test score (n = 1).
Conceptual and Methodological Challenges
The studies highlighted a number of conceptual and methodological challenges for estimating the outcomes from NGS. The challenges relate to a need for valuing the full range of NGS-related health and nonhealth outcomes, capturing how preferences change with accuracy of genomic information, multiple scientific uncertainties, and capturing downstream consequences of NGS-generated information.
Challenge 1: Valuing the range of NGS-related health, nonhealth, and process outcomes
NGS produces genomewide information with the potential for multiple outcomes. It is the breadth and complexity of information that makes NGS a unique case for preference-based valuation. The studies noted that NGS provides information on one or all disease diagnoses, prognoses, treatment responses, and hereditary risks. The types of information returned become complex to communicate when NGS uncovers variants of uncertain significance (VUS) or secondary findings (SFs). The potential outcomes of SFs include uncovering risk for diseases that may or may not have effective treatment options, availability of early screening strategies for family members, or pharmacogenetic information on response to drugs. The continuing debate is whether patients should be given information on diseases that are not treatable [
]. This is because patients may value the information absent of effective treatment and improvement in health status.
How studies conceptually addressed the broad range of outcomes
Researchers in the genetics and health economics literature have labeled the value of nonhealth outcomes as “personal utility,” which relates to the benefit that patients ascribe to the spectrum of information derived from genomic technologies, regardless of its potential to improve health [
]. Articulating the concept in this way was important because it broadened the scope of what the genetics literature considered as beneficial or of value to patients, families, and the health care system. The included articles articulated this concept of personal utility in two ways: 1) as the benefits or harms manifested outside medical contexts, as the value of genomic information, or as the value of knowledge that is separate from clinical effectiveness or 2) as the worth individuals ascribe to the full range of NGS testing outcomes, including health, nonhealth, and process outcomes.
The included studies provided evidence that supports the importance of health and nonhealth outcomes to the valuation of benefit. In all studies, respondents valued informational and process attributes. Regier et al. [
] found that 27% of the general population would want SF information for disorders with severe quality-of-life consequences, irrespective of whether effective medical treatment was available. Marshall et al. [
] determined that parents were interested in the return of highly penetrant nonmedically actionable conditions in children, particularly if manifestations were more severe (e.g., earlier age of onset and greater level of disability). Buchanan et al. [
] found that women preferred to receive chemotherapy for early-stage breast cancer if they trusted their oncologist’s opinion, irrespective of NGS estimates of recurrence risk and likely benefit from chemotherapy.
Challenge 2: NGS testing varies in accuracy and has uncertain effectiveness outcomes
Preference-based research has established the importance of accuracy and clinical utility of diagnostic information [
]. In the context of NGS, it has been noted that the evidence needed to support estimates of accuracy and clinical utility requires significant amounts of data. This is apparent in large-scale sequencing initiatives such as the UK 100,000 Genomes Project and the US Precision Medicine Initiative [
]. On a smaller scale, targetable biomarkers are increasingly sought retrospectively as part of randomized controlled trials not powered to find an effect. Given individual genomic heterogeneity and the requirement of large amounts of data, significant valuation challenges arise because NGS outcomes are subject to imprecision around health outcomes (including uncertainty surrounding clinical utility) and prediction error (accuracy), which can lead to disease misclassification [
] addressed sensitivity through describing the proportion of patients who could be cured by a new medication but did not receive the medication because of inaccurate results. Specificity was the proportion of patients who would not benefit from the new medication but would receive it because of incorrect test results. Buchanan et al. [
] incorporated test reliability into their DCE, defining the attribute in terms of the number of tests that provide an incorrect result. These studies found that respondents could distinguish between attribute levels describing various levels of accuracy. None of the included studies addressed the potential for imprecision around clinical effectiveness, and this remains an important application of future research.
Challenge 3: NGS has simultaneous multiple and cascading uncertainties
NGS information includes a process of patients simultaneously receiving layers of information. The initial decision to undergo genomic testing carries immediate risk-benefit trade-offs, including, for example, the risk of adverse effects associated with a biopsy, test accuracy, and the probability of identifying a pathogenic variant.
The challenge is to represent these multiple risks and their interdependencies, termed “multiple cascading uncertainties” by Marshall et al. [
]. From a choice-theoretic perspective, these streams of sequential information and probabilities can be thought of as endogenous to individuals’ choices. This type of endogeneity has been termed “multiple discreteness” [
]. The nature of these probabilities and their influence on choice are not easily incorporated into standard stated preference methods. In terms of econometric estimation, modeling utility without reference to multiple discreteness may result in inconsistent estimators. If these probabilities are included, appropriately presented, and understood by respondents, the econometric model must take into account the interdependence of the attribute levels and their sequential nature with the possibility of diminishing marginal utility as information is returned at various time points [
] focused solely on the return of SFs by describing penetrance (number of people with a genetic variant who actually get the disease), availability of effective interventions, quality of life, return of carrier status, and cost. They did not take into account the upstream choice of undergoing NGS testing for the primary condition. Instead, they asked respondents to imagine that they had been diagnosed with an unspecified disease and that SFs may be found. Marshall et al. [
] simplified their decision scenario by dividing the process into two parts. First, respondents were asked to complete a CV task aimed at understanding the value of NGS information for a broad set of uncertain outcomes. This was followed by DCE tasks that aimed to understand whether respondents would be willing to act on the information they received. The aim of this approach was to reduce the dimensionality of the problem, with the authors concluding that the DCE method alone is limited when inferring more realistic decision-making situations. None of the studies identified approaches to deal with the endogeneity associated with multiple discreteness.
Challenge 4: Downstream health and nonhealth consequences
Downstream outcomes from NGS are contingent on whether information is returned and acted upon [
]. For a hereditary condition, there are downstream outcomes beyond the patient alone because the health care system may fund NGS testing for family members. Additional testing and the concomitant risk-benefit outcomes discussed in the previous section will depend on the patient choosing to notify family members. The choice of the patient can also generate a negative or positive externality. This is because patients’ decisions can affect future health and nonhealth outcomes of family members. From a health care system perspective, a negative externality can occur in the context of overdiagnosis if the affected individual consumes unnecessary health care, potentially incurring the opportunity cost of displacing health from other patients.
In terms of SF, the downstream benefits of identified conditions with effective treatments can be captured by existing preference-based measures. Findings of VUS are potentially associated with treatable conditions that patients are at risk for in the future, but have the added complexity of uncertainty surrounding whether the variant causes disease. Importantly, a VUS may never be determined as clinically relevant because the numbers of other patients with the same rare variant who both have NGS and who express the phenotype may never be sufficient to establish causality [
], this type of uncertainty arises in scenarios in which individuals cannot know all the information they need to accurately understand the odds of a particular outcome occurring. The implication of Knightian uncertainty is that the odds of an outcome are incalculable and as such both the immediate and the long-term preference-based utilities for the outcome are impossible for an individual to calculate. Taking a Bayesian perspective, however, individuals may still attach probabilities to these unknown unknowns.
How studies have incorporated downstream consequences
] was to first specify a profile of health problems related to a hypothetical gene variant and then ask individuals to state a preference between possible downstream medical interventions. These interventions would effectively reduce the risk of health problems arising from the variant but would also carry possible adverse side effects and require out-of-pocket costs. The individual could also choose to undergo a watchful waiting scenario. They found that respondents valued NGS information most if there was a noninvasive medical intervention available to reduce their risk of developing a health problem. Preferences for undergoing a preventative medical intervention were greatly affected by the probability of adverse side effects. Kupperman et al. [
] applied time trade-off techniques to elicit preferences for potential prenatal testing outcomes, including the identification of a VUS. Scenarios involving pregnancy termination in the context of a VUS or having a baby with a VUS received low utility scores.
We identified 11 studies estimating the preference-based value of NGS outcomes. We found that authors directly elicited preference-based utility for NGS because off-the-shelf instruments do not adequately account for the breadth or types of NGS outcomes that patients or families may value. This hypothesis was empirically supported in the results of the included studies.
The evidence suggests that ignoring the preference-based value for health and nonhealth NGS outcomes in economic evaluation may lead to overinvestment or underinvestment if a system wants to maximize patients’ utility from health care services. That is, health care systems may overinvest if individuals have disutility for aspects of NGS-generated knowledge and underinvest if they place importance on the NGS knowledge [
]. Although such analyses will not replace the use of QALYs as the HTA reference case, we believe demand-side approaches to economic evaluation can be used as additional evidence for decision makers’ consideration [
Most of the included studies used DCEs to directly elicit preferences. Nevertheless, given the few preference studies of NGS and a lack of HTA recommendations regarding using DCE to inform resource allocation, a crucial area of research is to outline recommendations on which stated preference methods are appropriate for supporting economic evaluation of NGS. We expect that these recommendations will also need to consider important equity issues surrounding the use of metrics other than QALYs. This is a critical issue that is beyond the scope of this study.
The included studies outlined several challenges when eliciting the value of the large amounts of complex information obtained by NGS. We found that the solutions to challenges were limited and not comprehensive in approach, and they were broadly related to discussions of concepts on what constitutes benefit, descriptions of risk and accuracy, simplifying complex choice tasks, or sequentially using multiple valuation techniques. Although important first steps, these solutions are not unique and there remains scope for a comprehensive solution to valuing NGS outcomes.
Researchers are facing challenges when estimating the preference-based value of NGS. We conclude that it is the breadth and complexity of information that makes NGS-guided health care a unique valuation case study. Indeed, we identified challenges that are a direct result of the types of information that NGS provides. There remains considerable scope regarding providing comprehensive solutions to these challenges. We highlight the problems of incorporating uncertainty surrounding outcomes, endogeneity of choice, and downstream consequences of NGS testing as areas for needed research.
Dr. Regier was supported by the Canadian Centre for Applied Research in Cancer Control, which receives funding from the Canadian Cancer Society Research Institute grant #2015-019789. The authors would like to thank Dr. Kathryn Phillips for her contributions that led to the improvement of this article. The results and view expressed in this manuscript remain the authors’ own.
Source of financial support: Dr. Marshall reports personal fees from Board Membership: Pfizer; personal fees from Consultancy: Optum Insight; personal fees from Consultancy: Research Triangle Institute; personal fees from Consultancy: Roche; personal fees and non-financial support from Consultancy: Novartis, personal fees and non-financial support from Consultancy: Abbvie; personal fees and non-financial support from Consultancy: Janssen, grants from Grants/grants pending, outside the submitted work; and Received travel support from Illumina to attend a past working group meeting in Boston, MA, USA. Illumina Inc. provided travel fund to discuss the theme issue on precision medicine.