Advertisement

Accounting for Preference Heterogeneity in Discrete-Choice Experiments: An ISPOR Special Interest Group Report

      Highlights

      • There is an increasing interest in accounting for preference heterogeneity in discrete choice experiments, matched with a growing portfolio of analytical methods.
      • Accounting for heterogeneity allows researchers to go beyond understanding the “average” preference and reduces bias in the estimated parameters.
      • Most current studies estimate mixed logit models with either continuous (eg, normal, lognormal) or discrete (ie, latent classes) parameter distributions. Some studies attempt to separate heterogeneity in scale and preferences by using more complex models, despite both forms of heterogeneity being statistically confounded.
      • Health preference researchers are using increasingly complex methods to analyze preference data; nevertheless, our survey suggests there is disagreement among experts and applied researchers on the role, capabilities, and suitability of alternative approaches, indicating a need for discourse, alignment, and guidance.

      Abstract

      Objectives

      Discrete choice experiments (DCEs) are increasingly used to elicit preferences for health and healthcare. Although many applications assume preferences are homogenous, there is a growing portfolio of methods to understand both explained (because of observed factors) and unexplained (latent) heterogeneity. Nevertheless, the selection of analytical methods can be challenging and little guidance is available. This study aimed to determine the state of practice in accounting for preference heterogeneity in the analysis of health-related DCEs, including the views and experiences of health preference researchers and an overview of the tools that are commonly used to elicit preferences.

      Methods

      An online survey was developed and distributed among health preference researchers and nonhealth method experts, and a systematic review of the DCE literature in health was undertaken to explore the analytical methods used and summarize trends.

      Results

      Most respondents (n = 59 of 70, 84%) agreed that accounting for preference heterogeneity provides a richer understanding of the data. Nevertheless, there was disagreement on how to account for heterogeneity; most (n = 60, 85%) stated that more guidance was needed. Notably, the majority (n = 41, 58%) raised concern about the increasing complexity of analytical methods. Of the 342 studies included in the review, half (n = 175, 51%) used a mixed logit with continuous distributions for the parameters, and a third (n = 110, 32%) used a latent class model.

      Conclusions

      Although there is agreement about the importance of accounting for preference heterogeneity, there are noticeable disagreements and concerns about best practices, resulting in a clear need for further analytical guidance.

      Keywords

      Introduction

      Since their introduction, stated preference methods, particularly discrete choice experiments (DCEs), have become widely used as researchers and decision makers increasingly seek to elicit, understand, and predict stakeholders’ preferences for health and health-related care.
      • Craig B.M.
      • Lancsar E.
      • Mühlbacher A.C.
      • Brown D.S.
      • Ostermann J.
      Health preference research: an overview.
      ,
      • Soekhai V.
      • de Bekker-Grob E.W.
      • Ellis A.R.
      • Vass C.M.
      Discrete choice experiments in health economics: past, present and future.
      Health preference research covers a broad range of topics, including assessments of clinical effects (eg, benefit-risk analysis), valuation of health states, estimation of welfare measures, priority setting, policy evaluations, and job preferences of the workforce.
      • Soekhai V.
      • de Bekker-Grob E.W.
      • Ellis A.R.
      • Vass C.M.
      Discrete choice experiments in health economics: past, present and future.
      • Clark M.
      • Determann D.
      • Petrou S.
      • Moro D.
      • de Bekker-Grob E.W.
      Discrete choice experiments in health economics: a review of the literature.
      • Ryan M.
      • Gerard K.
      Using discrete choice experiments to value health care programmes: current practice and future research reflections.
      • de Bekker-Grob E.W.
      • Ryan M.
      • Gerard K.
      Discrete choice experiments in health economics: a review of the literature.
      Preferences are typically quantified as trade-offs that respondents (eg, patients, caregivers, or physicians) are willing to make between aspects of health technologies and health-related policies.
      • Vass C.M.
      • Payne K.
      Using discrete choice experiments to inform the benefit - risk assessment of medicines : are we ready yet?.
      Studies have traditionally focused on summarizing a sample’s preferences, often assuming they are homogenous,
      • Soekhai V.
      • de Bekker-Grob E.W.
      • Ellis A.R.
      • Vass C.M.
      Discrete choice experiments in health economics: past, present and future.
      although preferences may in fact be heterogenous among individuals or groups within a given population. Preference heterogeneity typically means that different individuals are expected to make different decisions in the same choice situations. In multiattribute situations, this implies that different individuals place a different relative importance or value on a set of decision criteria (also known as attributes). Preference heterogeneity may or may not be explained by data collected about individuals (eg, clinical or sociodemographic characteristics).
      • Roudijk B.
      • Donders A.R.T.
      • Stalmeier P.F.M.
      Cultural Values Group
      Cultural values: can they explain differences in health utilities between countries?.
      Interest in accounting for preference heterogeneity has increased in recent years.
      • Soekhai V.
      • de Bekker-Grob E.W.
      • Ellis A.R.
      • Vass C.M.
      Discrete choice experiments in health economics: past, present and future.
      For example, regulators may be interested in understanding if and how patients’ willingness to accept treatment outcomes varies based on their disease history, diagnostic options, personal characteristics, or other patient or disease characteristics.
      • Vass C.M.
      • Payne K.
      Using discrete choice experiments to inform the benefit - risk assessment of medicines : are we ready yet?.
      ,
      • Ho M.P.
      • Gonzalez J.M.
      • Lerner H.P.
      • et al.
      Incorporating patient-preference evidence into regulatory decision making.
      ,
      Patient preference information voluntary submission, review in premarket approval applications, humanitarian device exemption applications and de novo requests, and inclusion in decision summaries and device labeling. Food and Drug Administration.
      In contrast to the individual-level values from approaches such as threshold technique or contingent valuation,
      • Hauber B.
      • Coulter J.
      Using the threshold technique to elicit patient preferences: an introduction to the method and an overview of existing empirical applications.
      ,
      • Smith R.D.
      • Sach T.H.
      Contingent valuation: what needs to be done?.
      DCE responses are typically analyzed at the sample level, such that accounting for preference heterogeneity involves making assumptions about the nature of preference heterogeneity and its correlates. For example, although interactions with observable characteristics, such as gender, sex or age, may explain some heterogeneity in preferences, there may be unexplained (by observable correlates) preference heterogeneity because of latent factors that are difficult to measure, idiosyncratic differences among individuals or groups, or complex relationships between observable characteristics that are not well understood.
      • Louviere J.
      • Street D.
      • Carson R.
      • et al.
      Dissecting the random component of utility.
      As methods to account for both explained and unexplained preference heterogeneity
      • Hess S.
      Chapter 14: Latent class structures: taste heterogeneity and beyond.
      became more frequently used, fundamental issues were identified, including computational challenges (eg, premature convergence)
      • Hole A.R.
      • Yoo HIl
      The use of heuristic optimization algorithms to facilitate maximum simulated likelihood estimation of random parameter logit models.
      • Boeri M.
      • Saure D.
      • Schacht A.
      • Riedl E.
      • Hauber B.
      Modeling heterogeneity in patients’ preferences for psoriasis treatments in a multicountry study: a comparison between random-parameters logit and latent class approaches.
      • Mahieu P.A.
      • Andersson H.
      • Beaumais O.
      • Crastes dit Sourd R.
      • Hess S.
      • Wolff F.-C.
      Stated preferences: a unique database composed of 1657 recent published articles in journals related to agriculture, environment, or health.
      and behavioral aspects (eg, accounting for scale heterogeneity along with other forms of preference heterogeneity, choice certainty).

      Cherchi E, Ortúzar Jde D. Empirical identification in the mixed logit model: analysing the effect of data richness. In: Netw Spat Econ. 2008;8(2-3):109-124.

      • Vij A.
      • Walker J.L.
      Chapter 22: Hybrid choice models: the identification problem.
      • Groothuis-Oudshoorn C.G.M.
      • Flynn T.N.
      • Yoo HIl
      • Magidson J.
      • Oppe M.
      Key issues and potential solutions for understanding healthcare preference heterogeneity free from patient-level scale confounds.
      • Vass C.M.
      • Wright S.
      • Burton M.
      • Payne K.
      Scale heterogeneity in healthcare discrete choice experiments: a primer.
      Various approaches have been put forward to address these issues, although there is limited guidance on their relative strengths and weaknesses when accounting for preference heterogeneity in health-related DCE data.
      StataCorp
      Stata Statistical Software: Release 16.
      • Lagarde M.
      Investigating attribute non-attendance and its consequences in choice experiments with latent class models.
      • Veldwijk J.
      • Essers B.A.
      • Lambooij M.S.
      • Dirksen C.D.
      • Smit H.A.
      • Ardine De Wit G.
      Survival or mortality: does risk attribute framing influence decision-making behavior in a discrete choice experiment?.
      • Page M.J.
      • McKenzie J.E.
      • Bossuyt P.M.
      • et al.
      The PRISMA 2020 statement: an updated guideline for reporting systematic reviews.
      • Deidda M.
      • Meleddu M.
      • Pulina M.
      Potential users’ preferences towards cardiac telemedicine: a discrete choice experiment investigation in Sardinia.
      • Vass C.M.
      • Rigby D.
      • Payne K.
      Investigating the heterogeneity in women’s preferences for breast screening: does the communication of risk matter?.
      • Erdem S.
      • Thompson C.
      Prioritising health service innovation investments using public preferences: a discrete choice experiment.
      • Kløjgaard M.E.
      • Hess S.
      Understanding the formation and influence of attitudes in patients’ treatment choices for lower back pain: testing the benefits of a hybrid choice model approach.
      • Van Puyvelde S.
      • Caers R.
      • Du Bois C.
      • Jegers M.
      Does organizational ownership matter? Objectives of employees in public, nonprofit and for-profit nursing homes.
      • Howard K.
      • Salkeld G.P.
      • Patel M.I.
      • Mann G.J.
      • Pignone M.P.
      Men’s preferences and trade-offs for prostate cancer screening: a discrete choice experiment.
      The number and complexity of available methods to account for heterogeneity and the limited available guidance can lead to difficulties for practitioners and decision makers in making judgments about the suitability of these approaches. Therefore, this study aimed to provide an overview of the different analytical methods that are commonly used to account for preference heterogeneity and to assess views about approaches for modeling preference heterogeneity in DCE data.

      Methods

      A strategic working group (co-lead by M.B. and S.H.) within the ISPOR Health Preference Research Special Interest Group (SIG) was established in early 2020. Membership was open to researchers interested in preference heterogeneity in DCEs who were invited via emails through the ISPOR SIG and word of mouth. The working group first met virtually on February 11, 2020, where 2 workstreams were strategically chosen to generate a comprehensive overview of the current state of play: (1) developing an online survey among health preference researchers and (2) conducting a systematic literature review. The 2 workstreams were implemented in parallel with subsequent meetings to ensure project milestones were achieved (see Appendix A in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.01.012 for a summary of the timeline).

      Online Survey

      A cross-sectional online survey was conducted to understand the perspectives of health preference researchers and nonhealth methods experts about heterogeneity in stated preference data.

      Survey design

      The survey was devised in an iterative process with working group members’ input and in consultation with the wider ISPOR SIG to identify and refine key topics of interest and relevance. A draft of the survey was reviewed by the working group and SIG experts. The survey was composed of 6 parts: The first included respondents’ experiences and expertise in preference assessment and choice modeling and selected demographic information (eg, age, location). The second and third parts asked the respondents about their understanding of preference heterogeneity as a concept and methods to account for preference heterogeneity. The fourth and fifth parts included questions about methodological challenges and the need for additional guidance, respectively. The last part of the survey asked respondents for additional comments about preference heterogeneity. The survey was programmed and hosted online by SurveyEngine and tested by working group members. The final survey (see Appendix B in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.01.012) was submitted, evaluated, and deemed exempt from review by the institutional review board of the Ann & Robert H. Lurie Children's Hospital of Chicago, Chicago, Illinois (institutional review board reference 2020-3730).

      Data collection

      An invitation to participate in the survey, including a link to the online survey, was emailed to health preference researchers from the ISPOR SIG, the International Academy of Health Preference Researchers, and the International Health Economics Association SIG. Recipients were encouraged to forward the invitation to other health preference researchers. Eight academic methods experts from outside the health field were also invited because of their experience in developing methods to account for preference heterogeneity. The survey was accessible between July 1, 2020, and September 19, 2020.

      Analysis of survey data

      Descriptive statistics were used to summarize survey responses, using Microsoft Excel and Stata 16.
      StataCorp
      Stata Statistical Software: Release 16.
      No qualitative analysis was performed, but relevant comments were extracted to contextualize key findings.

      Literature Review

      To better understand the current state of practice, a systematic literature review was conducted to identify published DCEs that investigated preference heterogeneity.

      Inclusion and exclusion criteria

      The primary inclusion criteria were peer-reviewed, empirical studies related to health or healthcare that accounted for preference heterogeneity under random utility theory and used a DCE for eliciting preferences. Studies were included if they were also published in English between January 1, 2000, and March 31, 2020, and their analytical methods accounted for either explained or unexplained preference heterogeneity.
      We excluded conference abstracts, conference papers, working papers, reviews, guidelines, and protocols and stated preference studies not directly related to health or healthcare, such as food, transportation, and pollution. When it was uncertain whether a study was related to health, determination was made based on the publishing journal and its target audience. Studies that accounted for heterogeneity in decision rules (eg, attribute nonattendance, differences in utility function)
      • Lagarde M.
      Investigating attribute non-attendance and its consequences in choice experiments with latent class models.
      ,
      • Veldwijk J.
      • Essers B.A.
      • Lambooij M.S.
      • Dirksen C.D.
      • Smit H.A.
      • Ardine De Wit G.
      Survival or mortality: does risk attribute framing influence decision-making behavior in a discrete choice experiment?.
      were also excluded. To simplify the discussion of analytical methods, we excluded studies with scale-based preference elicitation tasks, such as best-worst scaling (BWS) (case 1, case 2, or case 3), best-best scaling, and rating or ranking based elicitation. Although studies with interaction terms were included, estimation results from different samples were excluded (ie, stratification without any reports of account of explained or unexplained preference heterogeneity).

      Search strategy

      PubMed, OVID (EconLit, MEDLINE, Embase), and the Web of Science were searched on March 31, 2020. The search strategy captured the primary inclusion criteria (DCEs, in health, investigating preference heterogeneity) and used search terms from previous reviews of DCEs (see Appendix C in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.01.012). Interim results from the databases were combined with a citation search to capture DCE studies identified in previous reviews of healthcare DCEs,
      • Soekhai V.
      • de Bekker-Grob E.W.
      • Ellis A.R.
      • Vass C.M.
      Discrete choice experiments in health economics: past, present and future.
      and a request was sent to members of the working group to share articles not identified in the search.

      Screening process

      Initial screening removed duplicates and titles with missing abstracts using a referencing software (EndNote x9.3.3) and hand searching. Abstracts were reviewed by 2 members of the working group. If an abstract was rejected by both reviewers, the article was eliminated from the process; likewise, if it was accepted by both reviewers, the article proceeded to full-text review. Where there was a lack of agreement between reviewers, abstracts were independently arbitrated by a third reviewer. If an abstract could not be rejected with certainty, it was included for full-text curation and review. Full-text screening followed the procedures of abstract screening, but when the full-text could not be rejected with certainty, members of the working group discussed the articles and achieved a consensus-based decision.

      Data extraction tool

      To promote consistency in extraction across members of the working group, a tool was developed through an iterative process of consultation and testing with the working group to ensure systematic extraction of the key study features of interest. The tool was developed by CV using Visual Basic for Applications in Microsoft Excel and is included in Appendix D in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.01.012.
      Extracted data included study characteristics, such as date of publication, primary study objective (eg, preferences for health-related technologies, health outcomes, jobs), and the sample, choice set format, and information on the experimental design. Details on preference heterogeneity including choice model, statistical software, and postestimation calculations (eg, willingness to pay, maximum acceptable risk, relative importance scores). The extracted data were tabulated to describe the state of practice.

      Results

      Results of the Online Survey

      Seventy respondents, including 8 nonhealth methods experts, completed the entire survey. An overview of participants’ characteristics is presented in Table 1. Respondents were almost balanced in gender and location, with approximately half working in Europe (n = 36, 51%). The sample included a diverse group of researchers, with participants from academic institutions (n = 60, 86%), contract research organizations, medical institutions, the pharmaceutical industry, regulatory authorities, and health technology assessment bodies. Survey participants had a wide range of experience with nearly a third (n = 20, 29%) having conducted 4 or fewer studies in the last 10 years and almost half (n = 32, 46%) having conducted >10 studies in the same time period.
      Table 1Online sample characteristics (N = 70).
      Sample characteristicsN = 70
      Gender, n (%)
       Male38 (54)
       Female31 (44)
       Prefer not to say1 (1)
      Primary work region, n (%)
       Europe36 (51)
       North America21 (30)
       Oceania10 (14)
       Asia3 (4)
      Affiliation,
      Numbers sum to >100% as respondents could select multiple responses.
      n (%)
       Academic institution60 (86)
       Consulting/CRO25 (36)
       Nonprofit/charity (eg, patient organization)11 (16)
       Clinics/medical institutions7 (10)
       Pharmaceutical industry (ie, manufacturer)5 (7)
       Regulator (eg, FDA, EMA)4 (6)
       Payer/HTA (eg, NICE, HAS)3 (4)
       None1 (1)
      Field of research,
      Numbers sum to >100% as respondents could select multiple responses.
      n (%)
       Healthcare valuation62 (89)
       Health state valuation31 (44)
       Agricultural economics/natural resource management10 (14)
       Environmental economics15 (21)
       Market research/consumer products14 (20)
       Business and economics8 (11)
       Market research/consumer products14 (20)
       Workforce preferences10 (14)
       Public infrastructure9 (13)
       Transportation economics8 (11)
       Other4 (6)
      Number of DCEs contributed to over last 10 years, n (%)
       02 (3)
       1-29 (13)
       3-49 (13)
       5-1018 (26)
       11-2015 (21)
       21-5010 (14)
       >507 (10)
      Roles on previous DCE projects,
      Numbers sum to >100% as respondents could select multiple responses.
      n (%)
       Principle investigator53 (76)
       Lead analyst44 (63)
       Technical advisor/reviewer33 (47)
       Scientific/technical project assistant24 (34)
       Strategic advisor/reviewer27 (39)
       Project manager22 (31)
       Other4 (6)
       None1 (1)
      Study purpose,
      Numbers sum to >100% as respondents could select multiple responses.
      n (%)
       Policy design or evaluation48 (75)
       Clinical decision making36 (56)
       Regulatory decision making28 (44)
       Product development25 (39)
       Commercialization and/or postmarket support21 (33)
       HTA/reimbursement decisions15 (23)
       Other2 (3)
      CRO indicates contract research organization; DCE, discrete choice experiment; EMA, European Medicines Agency; FDA, Food and Drug Administration; HAS, Haute Autorité de Santé; HTA, health technology assessment; NICE, National Institute for Health and Care Excellence.
      Numbers sum to >100% as respondents could select multiple responses.

      Perspectives on Preference Heterogeneity

      Perspectives on preference heterogeneity from health preference researchers versus methods experts are selectively summarized in Figure 1. Most health preference researchers (n = 53, 86%) agreed or strongly agreed that accounting for heterogeneity provides a richer interpretation of the data. Health preference researchers were also mostly in agreement that not explicitly accounting for preference heterogeneity in the analysis leads to bias in preference elicitation (n = 39, 63% somewhat agreed or strongly agreed). More than half agreed that every study should aim to account for preference heterogeneity (n = 36, 58% somewhat agreed or strongly agreed).
      Figure thumbnail gr1
      Figure 1Agreements with statements on preference heterogeneity.
      DCE indicates discrete choice experiment.
      The nonhealth methods experts tended to unanimously agree or disagree with the statements about preference heterogeneity, with one exception: Half agreed or strongly agreed and the other half disagreed or strongly disagreed with the statement “preference heterogeneity can only be meaningfully accounted for if relevant drivers of heterogeneity are collected alongside the DCE.” All nonhealth method experts either disagreed with or held no opinion on whether increased model complexity causes potential challenges for decision makers. In contrast, health preference researchers were more mixed, as illustrated in this comment by one responder:“As researchers in this field we need to be honest and open about the actual benefit of the increasingly complex models that we're using. Do our conclusions actually change as we begin to use more flexible models and incorporate additional data? Or are we just adding layers of complexity and increasing the time requirements for our analyses? If it is to be truly useful, it's important to ensure that future guidance is pragmatic.”(Health preference researcher, ID 11)

      Experience, Challenges, and the Need for Guidance

      Participants who said they were very or somewhat familiar with the concept of heterogeneity (50 preference researchers and all 6 nonhealth experts) were asked specific questions about analysis needs and challenges (results in Appendix E in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.01.012). Despite some concern about model complexity, 30 (54%) preference researchers and 5 (83%) experts familiar with heterogeneity agreed or strongly agreed that there is a need for accounting for both explained and unexplained heterogeneity. On this concept, one health researcher somewhat disagreed and one expert strongly disagreed.
      Participants familiar with heterogeneity also raised concern about statistical confounding between preference and scale heterogeneity. A total of 22 (44%) health preference researchers and 4 (67%) nonhealth methods experts agreed or strongly agreed that preference heterogeneity and scale heterogeneity can be separated using additional data; 1 expert and 1 health preference researcher strongly disagreed. Of the participants familiar with heterogeneity, only 6 (12%) health researchers and 1 expert agreed that it is feasible to distinguish between scale heterogeneity and preference heterogeneity without using additional data; 17 (34%) preference researchers and 5 (83%) of the 6 experts disagreed or strongly disagreed. One expert noted:“You need to allow for the heterogeneity even if you cannot necessarily know where it comes from. If you don't allow for it, you get bias. So eg a fully flexible mixed logit model allows for preference and scale heterogeneity, thus avoiding bias, but DOES NOT try to disentangle them”(Nonhealth methods expert, ID 6)
      Overall, 60 (86%) respondents, including 5 (83%) nonhealth methods experts, stated that there is a need for guidance on how to account for preference heterogeneity. This was also noted by respondents completing the survey:“…Guidance regarding preference heterogeneity is well overdue... It does feel that methods to account for this are so varied and can seem very complicated.”(Health preference researcher, ID 23)
      When researchers familiar with investigating preference heterogeneity were asked whether more clearly defined standards are needed, most (n = 35, 70%) agreed or strongly agreed, as did half of the nonhealth methods experts.

      Results of the Literature Review

      The initial search of electronic databases and studies included in existing literature reviews revealed 1202 potential titles, later reduced to 975 after removing duplicates. The abstracts were double screened, which further reduced the relevant studies to 435 full-length texts; of those, 379 were included. A final 45 studies were excluded during data extraction, and 8 further articles were identified for inclusion by members of the working group. In total, 342 studies were included in the final review. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses diagram is shown in Figure 2.
      • Page M.J.
      • McKenzie J.E.
      • Bossuyt P.M.
      • et al.
      The PRISMA 2020 statement: an updated guideline for reporting systematic reviews.
      Figure thumbnail gr2
      Figure 2PRISMA diagram.
      • Page M.J.
      • McKenzie J.E.
      • Bossuyt P.M.
      • et al.
      The PRISMA 2020 statement: an updated guideline for reporting systematic reviews.
      PRISMA indicates Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

      Overview of Studies

      Approximately half of the reviewed studies (n = 167, 49%) were published between 2016 and 2020. Overall, the number of studies accounting for preference heterogeneity has increased over time (Fig. 3). Most studies were published in the United States (n = 55, 16%), the United Kingdom (n = 48, 14%), Canada (n = 38, 11%), Australia (n = 38, 11%), and Germany (n = 31, 9%). Like other reviews of health-related DCEs,
      • Soekhai V.
      • de Bekker-Grob E.W.
      • Ellis A.R.
      • Vass C.M.
      Discrete choice experiments in health economics: past, present and future.
      most studies (n = 227, 66%) elicited preferences for health technologies including treatments, devices, or procedures. Other applications included informing health policies (n = 46, 14%) and eliciting preferences for health-related job or education choices (n = 24, 7%). Almost half of the studies (n = 163, 46%) were conducted with a patient sample, more than a quarter (n = 93, 27%) with the general public, and some studies (n = 51, 15%) reported the preferences of healthcare workers (eg, physicians, nurses, pharmacists).
      Figure thumbnail gr3
      Figure 3Included articles by publication year.
      GMNL indicates generalized multinomial logit; HB, hierarchical Bayes; LCA, latent class analysis; RPL, random parameters logit; SALC, scale-adjusted latent class.

      Methodological Approaches

      The models most commonly identified in this review are summarized in Figure 4. Many studies (n = 144, 44%) explained preference heterogeneity using interaction terms (ie, interacting personal characteristics with attributes and/or levels) to capture observed heterogeneity. Conditional logit models (n = 88, 26%) were used frequently with interaction terms and/or to inform subsequent analyses (ie, select starting values, investigate alternative specifications of the utility function). Methods to account for unexplained preference heterogeneity were widely used: just more than half of the studies in the review reported using a mixed logit with continuous distributions (n = 175, 51%) and almost a third conducted latent class analysis (LCA) (n = 110, 32%), specifying a discrete distribution of parameters (mixed logit refers to a group of models that extend the conditional multinomial logit model to control for unexplained preferences heterogeneity by estimating a distribution of preferences around each parameter to account for variation among individuals’ preferences. The health preference literature typically refers to models with a continuous distribution as random parameters logit [RPL] and to models with a discrete distribution as LCA). In addition to the 175 RPL models, 28 studies (8%) used hierarchical Bayes as an alternative to classical maximum (simulated) likelihood estimation. Cluster analysis was reported in 1 study.
      • Deidda M.
      • Meleddu M.
      • Pulina M.
      Potential users’ preferences towards cardiac telemedicine: a discrete choice experiment investigation in Sardinia.
      The review found examples of studies accounting for scale heterogeneity (the heteroskedastic conditional logit, generalized multinomial logit [GMNL], and scale-adjusted LCA are all mixed logit models that make different assumptions about parameter distributions), including the heteroskedastic conditional logit model (n = 12, 4%) and methods to jointly estimate scale and preference heterogeneity, such as the GMNL model (n = 20, 6%) (although the GMNL could be classified as an RPL with continuous distributions, in this review the models are reported separately to align with the reporting of the reviewed literature) and scale-adjusted LCA (n = 2, <1%).
      • Vass C.M.
      • Rigby D.
      • Payne K.
      Investigating the heterogeneity in women’s preferences for breast screening: does the communication of risk matter?.
      ,
      • Erdem S.
      • Thompson C.
      Prioritising health service innovation investments using public preferences: a discrete choice experiment.
      There was only one example of a hybrid choice model.
      • Kløjgaard M.E.
      • Hess S.
      Understanding the formation and influence of attitudes in patients’ treatment choices for lower back pain: testing the benefits of a hybrid choice model approach.
      Figure thumbnail gr4
      Figure 4Models used for accounting for preference heterogeneity (2002-2020). Note: Labeling of models refers to terms reported in the articles by the authors. As such, GMNL, mixed logit/RPL, error components, and hierarchical Bayes refer to mixed logit models assuming a continuous mixing distribution to account for heterogeneity, and latent class analysis and SALC refer to mixed logit models assuming a discrete mixing distribution to account for heterogeneity. Numbers sum to >342 as some studies used multiple models.
      GMNL indicates generalized multinomial logit; RPL, random parameters logit; SALC, scale-adjusted latent class.
      In studies that reported using an RPL (n = 79), Halton draws (n = 69) were most commonly used for numerical integration, although 1 study used modified Latin hypercube sampling,
      • Kløjgaard M.E.
      • Hess S.
      Understanding the formation and influence of attitudes in patients’ treatment choices for lower back pain: testing the benefits of a hybrid choice model approach.
      with a median of 1000 draws (minimum 50, maximum 20 000). Studies that reported using continuous distributions for the random parameters (eg, RPL, GMNL) most commonly specified a normal distribution (n = 110), although there were examples of lognormal (n = 15) and triangular distributions (n = 2).
      • Van Puyvelde S.
      • Caers R.
      • Du Bois C.
      • Jegers M.
      Does organizational ownership matter? Objectives of employees in public, nonprofit and for-profit nursing homes.
      ,
      • Howard K.
      • Salkeld G.P.
      • Patel M.I.
      • Mann G.J.
      • Pignone M.P.
      Men’s preferences and trade-offs for prostate cancer screening: a discrete choice experiment.
      A handful of studies
      • Michaels-Igbokwe C.
      • Lagarde M.
      • Cairns J.
      • Terris-Prestholt F.
      Designing a package of sexual and reproductive health and HIV outreach services to meet the heterogeneous preferences of young people in Malawi: results from a discrete choice experiment.
      • Campbell H.E.
      • Gray A.M.
      • Watson J.
      • et al.
      Preferences for interventions designed to increase cervical screening uptake in non-attending young women: how findings from a discrete choice experiment compare with observed behaviours in a trial.
      • Mohammadi T.
      • Zhang W.
      • Sou J.
      • Langlois S.
      • Munro S.
      • Anis A.H.
      A hierarchical Bayes approach to modeling heterogeneity in discrete choice experiments: an application to public preferences for prenatal screening.
      reported using an RPL model with correlated parameters, but reporting on the assumed covariance matrix of random parameters was generally limited.
      In studies that conducted LCA (n = 110, 32%), the median number of classes was 3 (minimum 2, maximum 6). The most common approaches to selecting classes were the Akaike information criterion (n = 72, 66%), the Bayesian information criterion (n = 53, 48%), and the consistent Akaike information criterion (n = 21, 19%). Overall, the most popular analysis software packages used across the studies were Stata (n = 99, 29%), Nlogit (n = 95, 28%), and Sawtooth (n = 31, 9%). Software that was used less frequently included Biogeme (n = 7, 2%)
      • Boeri M.
      • McMichael A.J.
      • Kane J.P.M.
      • O’Neill F.A.
      • Kee F.
      Physician-specific maximum acceptable risk in personalized medicine: implications for medical decision making.
      • Kjær T.
      • Bech M.
      • Kronborg C.
      • Mørkbak M.R.
      Public preferences for establishing nephrology facilities in Greenland: estimating willingness-to-pay using a discrete choice experiment.
      • Tayyari Dehbarez N.
      • Raun Mørkbak M.
      • Gyrd-Hansen D.
      • Uldbjerg N.
      • Søgaard R.
      Women’s preferences for birthing hospital in Denmark: a discrete choice experiment.
      • McMichael A.J.
      • Boeri M.
      • Rolison J.J.
      • et al.
      The influence of genotype information on psychiatrists’ treatment recommendations: more experienced clinicians know better what to ignore.
      • de Bekker-Grob E.W.
      • Donkers B.
      • Bliemer M.C.J.
      • Veldwijk J.
      • Swait J.D.
      Can healthcare choice be predicted using stated preference data?.
      • de Bekker-Grob E.W.
      • Veldwijk J.
      • Jonker M.
      • et al.
      The impact of vaccination and patient characteristics on influenza vaccination uptake of elderly people: a discrete choice experiment.
      • von Arx L.B.
      • Johnson F.R.
      • Mørkbak M.R.
      • Kjær T.
      Be careful what you ask for: effects of benefit descriptions on diabetes patients’ benefit-risk tradeoff preferences.
      and MATLAB (n = 2, <1%).
      • Mohammadi T.
      • Zhang W.
      • Sou J.
      • Langlois S.
      • Munro S.
      • Anis A.H.
      A hierarchical Bayes approach to modeling heterogeneity in discrete choice experiments: an application to public preferences for prenatal screening.
      ,
      • Bosworth R.
      • Cameron T.A.
      • DeShazo J.R.
      Willingness to pay for public health policies to treat illnesses.
      More detailed tabulations of the extracted data can be found in Appendix F in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.01.012.

      Discussion

      Decision makers, such as the US Food and Drug Administration, have noted the importance of going beyond the “average” patient in their consideration of preference information.
      • Fischer R.
      • Gebben D.
      • Gonzalez J.M.
      • Levitan B.
      • Tarver M.
      • Reed S.
      This research was motivated by an upsurge in methods to account for preference heterogeneity,
      • Hole A.R.
      Mixed logit modelling in Stata - an overview. Stata.
      particularly those accounting for unexplained heterogeneity (RPL and LCA) and a lack of formal guidance.
      This study is the first to elicit health preference researchers’ and nonhealth methods experts’ opinions about preference heterogeneity in DCE data and to systematically describe methods used in the published literature to account for such heterogeneity. Our review confirms the expectation that the use of various analytical methods to account for preference heterogeneity has been increasing over time. Survey responses found that both nonhealth methods experts and health preference researchers think preference heterogeneity should be accounted for to minimize bias and to give a richer, more complete interpretation of the data.
      Although the analytical methods to account for preference heterogeneity have typically originated in the marketing, transport, and environmental literature,
      • Ben-Akiva M.
      • Mcfadden D.
      • Train K.
      • et al.
      Hybrid choice models: progress and challenges.
      ,
      • Hensher D.
      • Greene W.
      The mixed logit model: the state of practice.
      there is evidence that these models are being widely applied in health research. The results of the survey also suggest that health preference researchers are concerned about the growing complexity of some models, whereas nonhealth methods experts do not share these concerns. Such differences in opinion could be because of differences in technical expertise, familiarity with approaches and their interpretation, and inadequate communication about the need for sophisticated methods or reflect differences in DCE study properties (eg, sample sizes) or applications (eg, the need to explain results to lay audiences), across fields. Making models more accessible and potentially providing clear guidance on their use could resolve this disagreement and address the concerns of the health preference researchers.
      Furthermore, multiple survey responses suggested that more clarity is needed among researchers on the confounding between heterogeneity in scale and preferences. Despite this hesitancy and mixed support in the wider choice modeling field, the literature review showed models to account for scale were used in health. Although some publications question the importance of incorporating scale heterogeneity,
      • Hess S.
      • Train K.
      Correlation and scale in mixed logit models.
      ,
      • Hess S.
      • Rose J.M.
      Can scale and coefficient heterogeneity be separated in random coefficients models?.
      other studies suggest that doing so can improve model fit
      • Milte R.
      • Ratcliffe J.
      • Chen G.
      • Lancsar E.
      • Miller M.
      • Crotty M.
      Cognitive overload? An exploration of the potential impact of cognitive functioning in discrete choice experiments with older people in health care.
      or predictive validity.
      • de Bekker-Grob E.W.
      • Donkers B.
      • Bliemer M.C.J.
      • Veldwijk J.
      • Swait J.D.
      Can healthcare choice be predicted using stated preference data?.
      The results of both the survey and the literature review confirm the need for further. A range of guidelines and frameworks have been published to establish quality standards in health preference research (eg, the Medical Device Innovation Consortium developed a framework for considering patient preferences in regulatory decision making and 3 ISPOR Good Practices Task Forces published their recommendations

      Bridges JF, Hauber AB, Marshall D, et al. A Checklist for Conjoint Analysis Applications in Health [ISPOR Conjoing Anal Heal Task Force Rep 2008:1-17].

      • Hauber A.B.
      • González J.M.
      • Groothuis-Oudshoorn C.G.M.
      • et al.
      Statistical methods for the analysis of discrete choice experiments: a report of the ISPOR conjoint analysis good research practices task force.
      • Johnson F.
      • Lancsar E.
      • Marshall D.
      • et al.
      Constructing experimental designs for discrete-choice experiments: report of the ISPOR conjoint analysis experimental design good research practices task.
      Medical Device Innovation Consortium (MDIC) patient centered benefit-risk project report: a framework for incorporating information on patient preferences regarding benefit and risk into regulatory assessments of new medical technology. Medical Device Innovation Consortium (MDIC).
      ). Nevertheless, although these guidance documents emphasize the importance of accounting for preference heterogeneity to avoid bias in preference elicitation or suboptimal policy advice, they offer few insights into the choice, application, interpretation, or reporting of methods. This gap suggests that guidance on accounting for preference heterogeneity needs to be produced by an informed discussion and consensus about analysts’ ability to actively identify and interpret heterogeneity, as opposed to simply accounting for it. Feedback from survey respondents indicates that such guidance needs to be clear, readily applicable, and pragmatic.

      Limitations

      As with many surveys, the data may not be representative of the entire health preference research community. Despite seeking input into the questionnaire design from a group of health preferences researchers, variations in the interpretation of questions are likely to remain.
      Additionally, this study was limited to extracting information that was published in the final article; therefore, additional choice models could have been explored but were not reported in the published research. Similarly, if a study reported on multiple models (eg, RPLs for different subsamples), each report was counted as one application for the purpose of this review. Furthermore, the data extracted in this review were conducted systematically by multiple volunteers. As such, there may be differences in interpretation, understanding, or categorization, which could lead to some inconsistencies in the generated data. To minimize potential for inconsistency, practice extraction exercises were conducted before data extraction.
      Further analyses of the data generated in this study (eg, examining free-text comments in the survey, analyzing response by experience or employment area, conducting a thorough evaluation of publishing trends) were beyond the scope of this article.
      Future research may conduct more in-depth investigation into issues such as common covariates used in subgroup and LCAs or the type of distributions in random parameter models. Nevertheless, the combined results of this survey and review are likely to describe the most pertinent aspects of current practice, and further research is unlikely to change the main findings. Although the review and survey focused on DCEs instead of other elicitation techniques (eg, conjoint analysis, BWS), the findings may be transferrable to other stated preference methods, such as BWS case 3. Despite these limitations, the study has several strengths and makes an important contribution to the field of health preference research by illuminating current practices and research constraints.

      Conclusions

      There is a general consensus among practitioners that accounting for preference heterogeneity in DCE data is important for the interpretation of health preference evidence. Therefore, it is no surprise that more complex analytical methods to account for heterogeneity are increasingly being used in health preference research. Despite this trend, the results of this research suggest that health preference researchers are concerned about the rising complexity of models, particularly surrounding the issue of scale heterogeneity. Their concern was matched with confusion and disagreement about the specific capability of approaches and a desire for defined standards. Overall, the results of this review and survey suggest a need for pragmatic guidance on emerging good practices to assist researchers with choosing the appropriate method, conducting the research, and interpreting and reporting results.

      Article and Author Information

      Author Contributions: Concept and design: Boeri, Heidenreich
      Acquisition of data: Vass, Boeri, Karim, Marshall, Craig, Ho, Mott, Ngorsuraches, Badawy, Mühlbacher, Gonzalez, Heidenreich
      Analysis and interpretation of data: Vass, Boeri, Karim, Marshall, Craig, Ho, Mott, Ngorsuraches, Badawy, Mühlbacher, Gonzalez, Heidenreich
      Drafting of the manuscript: Vass, Boeri, Heidenreich
      Critical revision of the paper for important intellectual content: Vass, Boeri, Karim, Marshall, Craig, Ho, Mott, Ngorsuraches, Badawy, Mühlbacher, Gonzalez, Heidenreich
      Statistical analysis: Vass, Heidenreich
      Administrative, technical, or logistic support: Vass, Boeri, Karim, Ho, Heidenreich
      Supervision: Boeri, Heidenreich
      Other (co-chairs of the strategic working group): Boeri, Heidenreich
      Conflict of Interest Disclosures: Dr Mühlbacher is an editor for Value in Health and had no role in the peer review process. No other disclosures were reported.
      Funding/Support: The authors received no financial support for this research.

      Acknowledgments

      The coauthors are grateful for feedback on the study design and contributions to data collection from Alejandra Duenas, Ali Homayouni, Christy Choi, Daria Putignano, Dave Gebben, Elizabeth Molsen, Karin Groothuis-Oudshoorn, Katherine Payne, Laurie Batchelder, Leslie Wilson, F Reed Johnson, Shelby Reed, Silvia Rinali, Stefania Lopatriello, and Ting-Hsuan Lee. In addition, they thank Elizabeth Molsen-David at ISPOR for her continuous support from start to finish and her excellent editing. The coauthors also thank Alejandra Duenas, Laurie Batchelder, Karin Groothuis-Oudshoorn, Robert Launois, Siu Hing Lo, and Karimi Milad, who provided valuable comments during the manuscript review of the ISPOR Health Preference Research Strategic Interest Group.

      Supplemental Materials

      References

        • Craig B.M.
        • Lancsar E.
        • Mühlbacher A.C.
        • Brown D.S.
        • Ostermann J.
        Health preference research: an overview.
        Patient. 2017; 10: 507-510
        • Soekhai V.
        • de Bekker-Grob E.W.
        • Ellis A.R.
        • Vass C.M.
        Discrete choice experiments in health economics: past, present and future.
        Pharmacoeconomics. 2019; 37: 201-226
        • Clark M.
        • Determann D.
        • Petrou S.
        • Moro D.
        • de Bekker-Grob E.W.
        Discrete choice experiments in health economics: a review of the literature.
        Pharmacoeconomics. 2014; 32: 883-902
        • Ryan M.
        • Gerard K.
        Using discrete choice experiments to value health care programmes: current practice and future research reflections.
        Appl Health Econ Health Policy. 2003; 2: 55-64
        • de Bekker-Grob E.W.
        • Ryan M.
        • Gerard K.
        Discrete choice experiments in health economics: a review of the literature.
        Health Econ. 2012; 21: 145-172
        • Vass C.M.
        • Payne K.
        Using discrete choice experiments to inform the benefit - risk assessment of medicines : are we ready yet?.
        Pharmacoeconomics. 2017; 35: 1-21
        • Roudijk B.
        • Donders A.R.T.
        • Stalmeier P.F.M.
        • Cultural Values Group
        Cultural values: can they explain differences in health utilities between countries?.
        Med Decis Making. 2019; 39: 605-616
        • Ho M.P.
        • Gonzalez J.M.
        • Lerner H.P.
        • et al.
        Incorporating patient-preference evidence into regulatory decision making.
        Surg Endosc Other Interv Tech. 2015; 29: 2984-2993
      1. Patient preference information voluntary submission, review in premarket approval applications, humanitarian device exemption applications and de novo requests, and inclusion in decision summaries and device labeling. Food and Drug Administration.
        • Hauber B.
        • Coulter J.
        Using the threshold technique to elicit patient preferences: an introduction to the method and an overview of existing empirical applications.
        Appl Health Econ Health Policy. 2020; 18: 31-46
        • Smith R.D.
        • Sach T.H.
        Contingent valuation: what needs to be done?.
        Heal Econ Policy Law. 2010; 5: 91-111
        • Louviere J.
        • Street D.
        • Carson R.
        • et al.
        Dissecting the random component of utility.
        Mark Lett. 2002; 13: 177-193
        • Hess S.
        Chapter 14: Latent class structures: taste heterogeneity and beyond.
        in: Hess S. Daly A. Handbook of Choice Modelling. Edward Elgar Publishing, Cheltenham, United Kingdom2014: 311-329
        • Hole A.R.
        • Yoo HIl
        The use of heuristic optimization algorithms to facilitate maximum simulated likelihood estimation of random parameter logit models.
        J R Stat Soc C. 2017; 66: 997-1013
        • Boeri M.
        • Saure D.
        • Schacht A.
        • Riedl E.
        • Hauber B.
        Modeling heterogeneity in patients’ preferences for psoriasis treatments in a multicountry study: a comparison between random-parameters logit and latent class approaches.
        Pharmacoeconomics. 2020; 38: 593-606
        • Mahieu P.A.
        • Andersson H.
        • Beaumais O.
        • Crastes dit Sourd R.
        • Hess S.
        • Wolff F.-C.
        Stated preferences: a unique database composed of 1657 recent published articles in journals related to agriculture, environment, or health.
        Rev Agric Food Environ Stud. 2017; 98: 201-220
      2. Cherchi E, Ortúzar Jde D. Empirical identification in the mixed logit model: analysing the effect of data richness. In: Netw Spat Econ. 2008;8(2-3):109-124.

        • Vij A.
        • Walker J.L.
        Chapter 22: Hybrid choice models: the identification problem.
        in: Hess S. Daly A. Handbook of Choice Modelling. Edward Elgar Publishing, Cheltenham, United Kingdom2014: 519-564
        • Groothuis-Oudshoorn C.G.M.
        • Flynn T.N.
        • Yoo HIl
        • Magidson J.
        • Oppe M.
        Key issues and potential solutions for understanding healthcare preference heterogeneity free from patient-level scale confounds.
        Patient. 2018; 11: 463-466
        • Vass C.M.
        • Wright S.
        • Burton M.
        • Payne K.
        Scale heterogeneity in healthcare discrete choice experiments: a primer.
        Patient. 2018; 11: 167-173
        • StataCorp
        Stata Statistical Software: Release 16.
        StataCorp LP, College Station, TX2019
        • Lagarde M.
        Investigating attribute non-attendance and its consequences in choice experiments with latent class models.
        Health Econ. 2012; 22: 554-567
        • Veldwijk J.
        • Essers B.A.
        • Lambooij M.S.
        • Dirksen C.D.
        • Smit H.A.
        • Ardine De Wit G.
        Survival or mortality: does risk attribute framing influence decision-making behavior in a discrete choice experiment?.
        Value Health. 2016; 19: 202-209
        • Page M.J.
        • McKenzie J.E.
        • Bossuyt P.M.
        • et al.
        The PRISMA 2020 statement: an updated guideline for reporting systematic reviews.
        Syst Rev. 2021; 10: 89
        • Deidda M.
        • Meleddu M.
        • Pulina M.
        Potential users’ preferences towards cardiac telemedicine: a discrete choice experiment investigation in Sardinia.
        Heal Policy Technol. 2018; 7: 125-130
        • Vass C.M.
        • Rigby D.
        • Payne K.
        Investigating the heterogeneity in women’s preferences for breast screening: does the communication of risk matter?.
        Value Heal. 2018; 21: 219-228
        • Erdem S.
        • Thompson C.
        Prioritising health service innovation investments using public preferences: a discrete choice experiment.
        BMC Health Serv Res. 2014; 14: 360
        • Kløjgaard M.E.
        • Hess S.
        Understanding the formation and influence of attitudes in patients’ treatment choices for lower back pain: testing the benefits of a hybrid choice model approach.
        Soc Sci Med. 2014; 114: 138-150
        • Van Puyvelde S.
        • Caers R.
        • Du Bois C.
        • Jegers M.
        Does organizational ownership matter? Objectives of employees in public, nonprofit and for-profit nursing homes.
        Appl Econ. 2015; 47: 2500-2513
        • Howard K.
        • Salkeld G.P.
        • Patel M.I.
        • Mann G.J.
        • Pignone M.P.
        Men’s preferences and trade-offs for prostate cancer screening: a discrete choice experiment.
        Heal Expect. 2015; 18: 3123-3135
        • Michaels-Igbokwe C.
        • Lagarde M.
        • Cairns J.
        • Terris-Prestholt F.
        Designing a package of sexual and reproductive health and HIV outreach services to meet the heterogeneous preferences of young people in Malawi: results from a discrete choice experiment.
        Health Econ Rev. 2015; 5: 9
        • Campbell H.E.
        • Gray A.M.
        • Watson J.
        • et al.
        Preferences for interventions designed to increase cervical screening uptake in non-attending young women: how findings from a discrete choice experiment compare with observed behaviours in a trial.
        Heal Expect. 2020; 23: 202-211
        • Mohammadi T.
        • Zhang W.
        • Sou J.
        • Langlois S.
        • Munro S.
        • Anis A.H.
        A hierarchical Bayes approach to modeling heterogeneity in discrete choice experiments: an application to public preferences for prenatal screening.
        Patient. 2020; 13: 211-223
        • Boeri M.
        • McMichael A.J.
        • Kane J.P.M.
        • O’Neill F.A.
        • Kee F.
        Physician-specific maximum acceptable risk in personalized medicine: implications for medical decision making.
        Med Decis Mak. 2018; 38: 593-600
        • Kjær T.
        • Bech M.
        • Kronborg C.
        • Mørkbak M.R.
        Public preferences for establishing nephrology facilities in Greenland: estimating willingness-to-pay using a discrete choice experiment.
        Eur J Heal Econ. 2013; 14: 739-748
        • Tayyari Dehbarez N.
        • Raun Mørkbak M.
        • Gyrd-Hansen D.
        • Uldbjerg N.
        • Søgaard R.
        Women’s preferences for birthing hospital in Denmark: a discrete choice experiment.
        Patient. 2018; 11: 613-624
        • McMichael A.J.
        • Boeri M.
        • Rolison J.J.
        • et al.
        The influence of genotype information on psychiatrists’ treatment recommendations: more experienced clinicians know better what to ignore.
        Value Heal. 2017; 20: 126-131
        • de Bekker-Grob E.W.
        • Donkers B.
        • Bliemer M.C.J.
        • Veldwijk J.
        • Swait J.D.
        Can healthcare choice be predicted using stated preference data?.
        Soc Sci Med. 2020; 246112736
        • de Bekker-Grob E.W.
        • Veldwijk J.
        • Jonker M.
        • et al.
        The impact of vaccination and patient characteristics on influenza vaccination uptake of elderly people: a discrete choice experiment.
        Vaccine. 2018; 36: 1467-1476
        • von Arx L.B.
        • Johnson F.R.
        • Mørkbak M.R.
        • Kjær T.
        Be careful what you ask for: effects of benefit descriptions on diabetes patients’ benefit-risk tradeoff preferences.
        Value Health. 2017; 20: 670-678
        • Bosworth R.
        • Cameron T.A.
        • DeShazo J.R.
        Willingness to pay for public health policies to treat illnesses.
        J Health Econ. 2015; 39: 74-88
        • Fischer R.
        • Gebben D.
        • Gonzalez J.M.
        • Levitan B.
        • Tarver M.
        • Reed S.
        FDA Summit: Methodologic Issues for PPI Studies.
        2020
        • Hole A.R.
        Mixed logit modelling in Stata - an overview. Stata.
        • Ben-Akiva M.
        • Mcfadden D.
        • Train K.
        • et al.
        Hybrid choice models: progress and challenges.
        Mark Lett. 2002; 133: 163-175
        • Hensher D.
        • Greene W.
        The mixed logit model: the state of practice.
        Transport. 2003; 30: 133-176
        • Hess S.
        • Train K.
        Correlation and scale in mixed logit models.
        J Choice Modell. 2017; 23: 1-8
        • Hess S.
        • Rose J.M.
        Can scale and coefficient heterogeneity be separated in random coefficients models?.
        Transportation (Amst). 2012; 39: 1225-1239
        • Milte R.
        • Ratcliffe J.
        • Chen G.
        • Lancsar E.
        • Miller M.
        • Crotty M.
        Cognitive overload? An exploration of the potential impact of cognitive functioning in discrete choice experiments with older people in health care.
        Value Health. 2014; 17: 655-659
      3. Bridges JF, Hauber AB, Marshall D, et al. A Checklist for Conjoint Analysis Applications in Health [ISPOR Conjoing Anal Heal Task Force Rep 2008:1-17].

        • Hauber A.B.
        • González J.M.
        • Groothuis-Oudshoorn C.G.M.
        • et al.
        Statistical methods for the analysis of discrete choice experiments: a report of the ISPOR conjoint analysis good research practices task force.
        Value Heal. 2016; 19: 300-315
        • Johnson F.
        • Lancsar E.
        • Marshall D.
        • et al.
        Constructing experimental designs for discrete-choice experiments: report of the ISPOR conjoint analysis experimental design good research practices task.
        Value Heal. 2013; 16: 3-13
      4. Medical Device Innovation Consortium (MDIC) patient centered benefit-risk project report: a framework for incorporating information on patient preferences regarding benefit and risk into regulatory assessments of new medical technology. Medical Device Innovation Consortium (MDIC).