Advertisement

Deciding Between SF-6Dv2 Health States: A Think-Aloud Study of Decision-Making Strategies Used in Discrete Choice Experiments

Open AccessPublished:September 02, 2022DOI:https://doi.org/10.1016/j.jval.2022.07.018

      Highlights

      • Discrete choice experiments have been used to derive estimated utility weights for multiattribute health utility measures, such as the EQ-5D and the SF-6D.
      • This study adds to the understanding of the decision-making process used by individuals, including their understanding of the task and the strategies they use in determining their answer, when evaluating and selecting preferred SF-6Dv2 health states.
      • The decision-making strategies used by participants include trading attributes, reinterpreting attributes, and relying on preconceived ideas about attributes; these findings underscore the importance of conducting a think-aloud study to ensure greater clarity when using discrete choice experiments to derive utility weights for the SF-6Dv2.

      Abstract

      Objective

      This study aimed to gain insight into decision-making strategies individuals used when evaluating pairs of SF-6Dv2 health states in discrete choice experiments (DCEs).

      Methods

      This qualitative, cross-sectional, noninterventional study asked participants to use a think-aloud approach to compare SF-6Dv2 health states in DCEs. Thematic analysis focused on comprehension and cognitive strategies used to compare health states and make decisions.

      Results

      Participants (N = 40) used 3 main strategies when completing DCEs: (1) trading, (2) reinterpretation, and (3) relying on previous experience. Trading was the most common strategy, used by everyone at least once, and involved prioritizing key attributes, such as preferring a health state with significant depression but no bodily pain. Reinterpretation was used by 17 participants and involved reconstructing health states by changing underlying assumptions (eg, rationalizing selecting a health state with significant pain because they could take pain medications). Finally, some (n = 13) relied on previous experience when making decisions on some choice tasks. Participants with experience dealing with pain, for instance, prioritized health states with the least impact in this dimension.

      Conclusions

      Qualitatively evaluating the decision-making strategies used in DCEs allows researchers to evaluate whether the tasks and attributes are interpreted accurately. The findings from this study add to the understanding of the generation of SF-6Dv2 health utility weights and the validity of these weights (e.g., reinterpreting health states could undermine the validity of DCEs and utility weights), and the overall usefulness of the SF-6Dv2. The methodology described in this study can and should be carried forth in valuing other health utility measures, not just the SF-6Dv2.

      Keywords

      Introduction

      A discrete choice experiment (DCE) is a survey-based methodology wherein individuals assess attributes of different scenarios and choose between them, indicating their overall preference.
      • Clark M.D.
      • Determann D.
      • Petrou S.
      • Moro D.
      • de Bekker-Grob E.W.
      Discrete choice experiments in health economics: a review of the literature.
      ,
      • Lancsar E.
      • Louviere J.
      Conducting discrete choice experiments to inform healthcare decision making: a user’s guide.
      When focused on health-related quality of life, data from DCEs can be used to derive utility indices for calculating quality-adjusted life-years (QALYs), according to individuals’ preferences for living in different health states.
      • Clark M.D.
      • Determann D.
      • Petrou S.
      • Moro D.
      • de Bekker-Grob E.W.
      Discrete choice experiments in health economics: a review of the literature.
      ,
      • Lancsar E.
      • Louviere J.
      Conducting discrete choice experiments to inform healthcare decision making: a user’s guide.
      DCEs have been used to estimate utility weights for health utility measures, such as the EQ-5D and SF-6D Health Utility Survey (SF-6D).
      • Mulhern B.
      • Norman R.
      • Brazier J.
      Valuing SF-6Dv2 in Australia using an international protocol.
      • Mulhern B.J.
      • Bansback N.
      • Norman R.
      • Brazier J.
      • SF-6Dv2 International Project Group
      Valuing the SF-6Dv2 classification system in the United Kingdom using a discrete-choice experiment with duration.
      • Norman R.
      • Cronin P.
      • Viney R.
      A pilot discrete choice experiment to explore preferences for EQ-5D-5L health states.
      • Norman R.
      • Viney R.
      • Brazier J.
      • et al.
      Valuing SF-6D health states using a discrete choice experiment.
      • Wu J.
      • Xie S.
      • He X.
      • et al.
      Valuation of SF-6Dv2 health states in China using time trade-off and discrete-choice experiment with a duration dimension.
      In DCEs used to value health utility measures, participants are asked to choose among 2 health states, each characterized by a length of survival and a level on each of the health dimensions, or attributes, covered by the measures. Attributes assessed in DCEs need to be leveled (ie, each could be perceived as better or worse than another) and capable of being traded (ie, individuals are willing to swap “worse” attributes for “better” ones).
      • Cheraghi-Sohi S.
      • Bower P.
      • Mead N.
      • McDonald R.
      • Whalley D.
      • Roland M.
      Making sense of patient priorities: applying discrete choice methods in primary care using ‘think aloud’ technique.
      ,
      • Lancsar E.
      • Louviere J.
      Deleting ‘irrational’ responses from discrete choice experiments: a case of investigating or imposing preferences?.
      The derivation of utility weights rests on assuming (1) participants will assume health states are stable over time (eg, moderate pain for 2 years is twice as bad as 1 year), (2) participants will consider the length of survival and the decrement from negative health states along the dimensions of the measure (eg, negative outcomes are traded against positive outcomes), and (3) participants will tend to choose the health state with the largest utility.
      • Lancsar E.
      • Louviere J.
      Conducting discrete choice experiments to inform healthcare decision making: a user’s guide.
      ,
      • Reed Johnson F.
      • Lancsar E.
      • Marshall D.
      • et al.
      Constructing experimental designs for discrete-choice experiments: report of the ISPOR Conjoint Analysis Experimental Design Good Research Practices Task Force.
      • Veldwijk J.
      • Determann D.
      • Lambooij M.S.
      • et al.
      Exploring how individuals complete the choice tasks in a discrete choice experiment: an interview study.
      • Ryan M.
      • Watson V.
      • Entwistle V.
      Rationalising the ‘irrational’: a think aloud study of discrete choice experiment responses.
      These underlying assumptions come with inherent problems. First, if an attribute is not clearly understood, there is a risk the individual will ignore it (ie, attribute nonattendance).
      • Cheraghi-Sohi S.
      • Bower P.
      • Mead N.
      • McDonald R.
      • Whalley D.
      • Roland M.
      Making sense of patient priorities: applying discrete choice methods in primary care using ‘think aloud’ technique.
      ,
      • Veldwijk J.
      • Determann D.
      • Lambooij M.S.
      • et al.
      Exploring how individuals complete the choice tasks in a discrete choice experiment: an interview study.
      • Ryan M.
      • Watson V.
      • Entwistle V.
      Rationalising the ‘irrational’: a think aloud study of discrete choice experiment responses.
      • Waters E.A.
      • Weinstein N.D.
      • Colditz G.A.
      • Emmons K.
      Formats for improving risk communication in medical tradeoff decisions.
      • Kenny P.
      • Hall J.
      • Viney R.
      • Haas M.
      Do participants understand a stated preference health survey? A qualitative approach to assessing validity.
      • Bryan S.
      • Dolan P.
      Discrete choice experiments in health economics. For better or for worse?.
      • Lagarde M.
      Investigating attribute non-attendance and its consequences in choice experiments with latent class models.
      • Miguel F.S.
      • Ryan M.
      • Amaya-Amaya M.
      ‘Irrational’ stated preferences: a quantitative and qualitative investigation.
      • Vass C.
      • Rigby D.
      • Payne K.
      I was trying to do the maths”: exploring the impact of risk communication in discrete choice experiments.
      Next, individuals might be unwilling to trade because they have a dominant preference for a single attribute, and make decisions based solely on that preference (ie, lexicographic preferences).
      • Cheraghi-Sohi S.
      • Bower P.
      • Mead N.
      • McDonald R.
      • Whalley D.
      • Roland M.
      Making sense of patient priorities: applying discrete choice methods in primary care using ‘think aloud’ technique.
      ,
      • Lancsar E.
      • Louviere J.
      Deleting ‘irrational’ responses from discrete choice experiments: a case of investigating or imposing preferences?.
      ,
      • Kenny P.
      • Hall J.
      • Viney R.
      • Haas M.
      Do participants understand a stated preference health survey? A qualitative approach to assessing validity.
      To minimize measurement error, careful consideration of study design and methods is needed to ensure participants interpret DCE tasks and attributes accurately.
      • Kenny P.
      • Hall J.
      • Viney R.
      • Haas M.
      Do participants understand a stated preference health survey? A qualitative approach to assessing validity.
      ,
      • Mulhern B.
      • Bansback N.
      • Brazier J.
      • et al.
      Preparatory study for the revaluation of the EQ-5D tariff: methodology report.
      Despite these potential issues, DCEs are important in the development of health utility measures. They examine patient priorities, informing both healthcare policy and clinician education, and enable survey developers to estimate and select a value set for understanding health preferences.
      • Mulhern B.
      • Norman R.
      • Brazier J.
      Valuing SF-6Dv2 in Australia using an international protocol.
      ,
      • Cheraghi-Sohi S.
      • Bower P.
      • Mead N.
      • McDonald R.
      • Whalley D.
      • Roland M.
      Making sense of patient priorities: applying discrete choice methods in primary care using ‘think aloud’ technique.
      ,
      • Morgan A.
      • Shackley P.
      • Pickin M.
      • Brazier J.
      Quantifying patient preferences for out-of-hours primary care.
      ,
      • Vick S.
      • Scott A.
      Agency in health care. Examining patients’ preferences for attributes of the doctor-patient relationship.
      Previous research identified common strategies used when completing DCEs for estimating utility weights. These strategies align with the general assumptions described above and include trading,
      • Cheraghi-Sohi S.
      • Bower P.
      • Mead N.
      • McDonald R.
      • Whalley D.
      • Roland M.
      Making sense of patient priorities: applying discrete choice methods in primary care using ‘think aloud’ technique.
      ,
      • Ryan M.
      • Watson V.
      • Entwistle V.
      Rationalising the ‘irrational’: a think aloud study of discrete choice experiment responses.
      ,
      • Kenny P.
      • Hall J.
      • Viney R.
      • Haas M.
      Do participants understand a stated preference health survey? A qualitative approach to assessing validity.
      ,
      • Vass C.
      • Rigby D.
      • Payne K.
      I was trying to do the maths”: exploring the impact of risk communication in discrete choice experiments.
      ,
      • Whitty J.A.
      • Walker R.
      • Golenko X.
      • Ratcliffe J.
      A think aloud study comparing the validity and acceptability of discrete choice and best worst scaling methods.
      selecting the least risk/most positive scenario,
      • Veldwijk J.
      • Determann D.
      • Lambooij M.S.
      • et al.
      Exploring how individuals complete the choice tasks in a discrete choice experiment: an interview study.
      ,
      • Kenny P.
      • Hall J.
      • Viney R.
      • Haas M.
      Do participants understand a stated preference health survey? A qualitative approach to assessing validity.
      making substitutions by ignoring/not attending to certain attributes,
      • Veldwijk J.
      • Determann D.
      • Lambooij M.S.
      • et al.
      Exploring how individuals complete the choice tasks in a discrete choice experiment: an interview study.
      ,
      • Kenny P.
      • Hall J.
      • Viney R.
      • Haas M.
      Do participants understand a stated preference health survey? A qualitative approach to assessing validity.
      adding or inferring information not included in the scenario (including inferring causality between attributes),
      • Cheraghi-Sohi S.
      • Bower P.
      • Mead N.
      • McDonald R.
      • Whalley D.
      • Roland M.
      Making sense of patient priorities: applying discrete choice methods in primary care using ‘think aloud’ technique.
      ,
      • Ryan M.
      • Watson V.
      • Entwistle V.
      Rationalising the ‘irrational’: a think aloud study of discrete choice experiment responses.
      ,
      • Kenny P.
      • Hall J.
      • Viney R.
      • Haas M.
      Do participants understand a stated preference health survey? A qualitative approach to assessing validity.
      allowing 1 to 3 key attributes to influence decision making,
      • Cheraghi-Sohi S.
      • Bower P.
      • Mead N.
      • McDonald R.
      • Whalley D.
      • Roland M.
      Making sense of patient priorities: applying discrete choice methods in primary care using ‘think aloud’ technique.
      ,
      • Ryan M.
      • Watson V.
      • Entwistle V.
      Rationalising the ‘irrational’: a think aloud study of discrete choice experiment responses.
      ,
      • Kenny P.
      • Hall J.
      • Viney R.
      • Haas M.
      Do participants understand a stated preference health survey? A qualitative approach to assessing validity.
      changing the key attributes in each scenario,
      • Cheraghi-Sohi S.
      • Bower P.
      • Mead N.
      • McDonald R.
      • Whalley D.
      • Roland M.
      Making sense of patient priorities: applying discrete choice methods in primary care using ‘think aloud’ technique.
      ,
      • Ryan M.
      • Watson V.
      • Entwistle V.
      Rationalising the ‘irrational’: a think aloud study of discrete choice experiment responses.
      ,
      • Kenny P.
      • Hall J.
      • Viney R.
      • Haas M.
      Do participants understand a stated preference health survey? A qualitative approach to assessing validity.
      and considering all attributes.
      • Veldwijk J.
      • Determann D.
      • Lambooij M.S.
      • et al.
      Exploring how individuals complete the choice tasks in a discrete choice experiment: an interview study.
      ,
      • Ryan M.
      • Watson V.
      • Entwistle V.
      Rationalising the ‘irrational’: a think aloud study of discrete choice experiment responses.
      Given the variety of potential completion strategies and issues linked to DCE tasks, it is recommended to pilot test DCEs—including asking individuals to complete them using a think-aloud approach—to a priori identify problems
      • Veldwijk J.
      • Determann D.
      • Lambooij M.S.
      • et al.
      Exploring how individuals complete the choice tasks in a discrete choice experiment: an interview study.
      and provide evidence participants interpret the task and attributes accurately.
      • Kenny P.
      • Hall J.
      • Viney R.
      • Haas M.
      Do participants understand a stated preference health survey? A qualitative approach to assessing validity.
      Cognitive interviewing, a qualitative research method informed by cognitive theory,
      • Drennan J.
      Cognitive interviewing: verbal data in the design and pretesting of questionnaires.
      uses think-aloud and probing techniques to understand how individuals form judgments, make decisions, and answer survey questions.
      • Padilla J.-L.
      • Leighton J.P.
      Cognitive interviewing and think aloud methods.
      It can be used to optimize survey design and implementation and to reduce measurement errors.
      • Collins D.
      Pretesting survey instruments: an overview of cognitive methods.
      The purpose of this study was to use a think-aloud approach to gain insight into the decision-making process used by individuals when evaluating and selecting preferred SF-6Dv2 DCE health states.

      Methods

      Study Design

      This qualitative, cross-sectional, noninterventional study consisted of one-on-one cognitive interviews. The 75-minute audio-recorded interviews were conducted by 1 of 2 experienced and trained qualitative researchers (L.B. and M.L.C.). All interviews were conducted using videoconferencing software, allowing for nationwide participation by a diverse geographic sample. All study materials were approved by one central independent review board (WCG/New England Institutional Review Board Study #1293768).

      Study Population

      The study used purposive sampling to recruit 40 participants from the general population via a third-party recruitment vendor. The final sample size was predetermined based on the number of choice tasks, the complexity of the choice tasks, and the desired sample diversity.
      • Willis G.B.
      Cognitive Interviewing: A Tool for Improving Questionnaire Design.
      ,
      • Patrick D.L.
      • Burke L.B.
      • Gwaltney C.J.
      • et al.
      Content validity--establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO Good Research Practices Task Force report: part 2--assessing respondent understanding.
      Specific quotas were established to ensure a diverse and representative sample in age, sex, presence of chronic health conditions, race/ethnicity, and education (see Appendix 1, Table 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.07.018). All participants were required to be 18 years of age or older, living in the United States, and fluent in English.
      Potential participants completed an online screening questionnaire to assess study inclusion criteria and demographic information. A total of 87 potential participants were screened for eligibility, and recruitment was stopped when all quotas were met. Participants were excluded from the study if they were unwilling or unable to participate in an interview. During screening, participants were also asked how they would describe their current health (poor, fair, good, very good, excellent) and to rate their health satisfaction on a scale from 0 (not at all satisfied) to 10 (completely satisfied).

      Study Procedures

      During each interview, participants completed DCE choice tasks using a think-aloud approach. Participants were asked to read all information in each choice task out loud and to articulate their thoughts—including any points of confusion and how they ultimately decided the meaning of the item in question—as they read the health states and selected which they preferred. Upon completing each choice task, participants responded to semistructured follow-up questions, and spontaneous probing questions if necessary, to help the interviewer better understand their decision-making process and rationale for the choice they made. For each choice task, participants rated on a scale of 0 to 10 how difficult it was to select a preferred health state. After each interview, the audio recording was sent for verbatim transcription and each participant was given an honorarium.
      To minimize burden, participants were divided into 8 groups of 5 by their order of entry into the study. Each group of 5 was asked to complete 1 of 8 sets of 4 choice tasks evaluating and selecting from pairs of SF-6Dv2 health state profiles (health states A and B). Each pair of health state profiles included the following attributes (Fig. 1): years of life left until death, physical function (activity limitations), role function (limitations accomplishing tasks), pain intensity, vitality, social function (limitations in social activities), and mental health (feeling depressed or very nervous). Choice tasks were strategically selected so (1) very unusual health states were avoided by only including health states observed in a large general population data set (N = 75 000),
      • Taylor D.C.
      • Kosinski M.
      • Reilly K.
      • Lindner L.
      Comparison of the burden of Ibs with constipation on health-related quality of life (HRQOL), work productivity, and health care utilization to asthma, migraine, and rheumatoid arthritis in the US, UK, and France.
      (2) comparisons including identical levels for 1 or more attributes were avoided, and (3) the number of different attribute levels being examined was maximized. The health states were scored using UK utility weights
      • Mulhern B.J.
      • Bansback N.
      • Norman R.
      • Brazier J.
      • SF-6Dv2 International Project Group
      Valuing the SF-6Dv2 classification system in the United Kingdom using a discrete-choice experiment with duration.
      and choice tasks were characterized as representing a large or small QALY difference (compared with the median difference of 2.4) and a high or a low average QALY level (compared with the median average level of 1.7). The first-choice task in each set represented a large QALY difference between health states and a high average QALY level, the second-choice task represented a large difference and a low level, the third-choice task represented a small difference and a high level, and the fourth-choice task represented a small difference and a low level. These selection criteria aimed to make the tasks challenging but realistic, starting with easier tasks before proceeding to more challenging tasks.
      Figure thumbnail gr1
      Figure 1Sample DCE exemplifying SF-6Dv2 domains and pairing of health state attributes.
      DCE indicates discrete choice experiment.

      Coding and Analysis

      Coding of interview data began immediately after each interview based solely on interviewer field notes. The interviewers populated a spreadsheet with any notable issues that arose during the interview and recorded choice task preferences.
      Once received, all transcripts were cross-checked against the initial coding spreadsheet to ensure consistency and completeness and then coded using NVivo software (QSR International Pty Ltd, Burlington, MA, 2018). Transcripts were coded to identify overall opinions on the choice tasks, the decision-making approaches and strategies used when evaluating the health states, and any other suggestions or insights. The researchers independently coded the same first 2 transcripts and then met to review their coding and resolve any discrepancies. This meeting allowed for any adjustments to the codebook and code definitions. Once coding was consistent, the remaining transcripts were randomly divided between the 2 coders and coded independently. The coders communicated throughout coding (3 formal meetings) to ensure consistency and address any questions; the study principal investigator (L.B.) reviewed all coding to ensure coding reliability.
      Once transcripts were coded, the 2 researchers used inductive thematic analysis to distill meaning from the data. Thematic analysis is a qualitative method in which researchers identify and interpret common themes or patterns of meaning throughout the data.
      • Kiger M.E.
      • Varpio L.
      Thematic analysis of qualitative data: AMEE guide no. 131.
      ,
      • Castleberry A.
      • Nolen A.
      Thematic analysis of qualitative research data: is it as easy as it sounds?.
      Analysis focused on the cognitive strategies used to compare and select preferred health states. This analysis included an assessment of each participant’s decision-making process, preferential attributes of each scenario, clarity of health descriptions, and the level of difficulty in making decisions.
      Given that each health state was assigned a utility score using the UK weights,
      • Mulhern B.J.
      • Bansback N.
      • Norman R.
      • Brazier J.
      • SF-6Dv2 International Project Group
      Valuing the SF-6Dv2 classification system in the United Kingdom using a discrete-choice experiment with duration.
      the research team queried the coded data to evaluate (1) whether cognitive strategies differed according to utility score (ie, whether different strategies were used for health states with high versus low QALY scores) and (2) whether strategies differed according to large versus small differences between the 2 health states. Finally, the data were organized into sets according to age (18-49 years and 50 years or older) and presence or absence of a chronic health condition and queried to determine whether the cognitive strategies used in decision making differed according to these factors.

      Results

      Participant Demographics

      Notably, 40 individuals participated in this study. Participants were white (n = 26, 65.0%), were female (n = 23, 58.0%), had completed some form of post-high school education (n = 29, 72.5%), and had a chronic health condition (n = 29, 72.5%). Half of the sample was between the ages of 18 and 49 years, and the other half was age 50 years or older; 14 participants (35%) rated their overall health as “very good” or “excellent.” Health satisfaction ratings were wide ranging across the 11-point scale, with an average of 5.8 (lowest and highest observed scores were 1 and 9, respectively). See Table 1 for additional demographic data.
      Table 1Participant characteristics.
      Demographicsn (%)
      Sex
      Female23 (57.5)
      Male17 (42.5)
      AgeFemaleMale
      18-29 years1 (4)0 (0)
      30-39 years9 (39)4 (24)
      40-49 years2 (9)4 (24)
      50-59 years9 (39)6 (35)
      60+ years2 (9)3 (18)
      Race/ethnicity
      White26 (65.0)
      Black or African American10 (25.0)
      Asian3 (7.5)
      Hispanic/Latino or of Spanish origin3 (7.5)
      Education
      High school diploma or GED11 (27.5)
      Some college but no degree5 (12.5)
      Associate’s degree or technical certificate8 (20.0)
      Bachelor’s degree (BA, BS)10 (25.0)
      Graduate degree (MA, MS, PhD, MD, etc)6 (15.0)
      Has chronic health condition
      Yes29 (72.5)
      No11 (27.5)
      Region of residence
      Northeast12 (30.0)
      West7 (17.5)
      Midwest5 (12.5)
      South16 (40)
      Current work status
      Retired4 (10.0)
      On disability or leave of absence2 (5.0)
      Temporarily furloughed1 (2.5)
      Unemployed, but looking for work1 (2.5)
      Employed full time (≥ 40 hours per week)21 (52.5)
      Employed part time (< 40 hours per week)8 (20.0)
      Student (full or part time)5 (12.5)
      Stay-at-home parent or spouse4 (10.0)
      Other: self-employed1 (2.5)
      GED indicates General Educational Development.

      Strategies Used

      Interviewers observed participants understood the choice tasks and easily completed them. This was confirmed by participants’ answers to the questions asked after the think-aloud.
      Participants used 3 main strategies when comparing and selecting preferred health states: (1) trading, (2) reinterpreting a health state, and (3) relying on previous experience or preconceived ideas of an attribute. Participants most often used 1 decision-making strategy per choice task. In some cases, participants used the same strategy for every choice task whereas others switched strategies each time. Although decision making was not a linear process, a general pattern for decision making was found across vignettes. When initially evaluating each choice task, participants were observed easily identifying either a key attribute to prioritize or whether deeper consideration of the health states was needed. Figure 2 illustrates the cognitive strategies participants were observed using in this study. Response strategies not following this pattern included a single participant who added up self-defined positives in each scenario and selected the health state with the highest score.
      Figure thumbnail gr2
      Figure 2DCE cognitive strategies for decision making.
      DCE indicates discrete choice experiment.

      Trading

      Trading was the most common strategy deployed in decision making and was used by every participant at least once. When trading, participants sacrificed 1 or more attributes for another. Participants used hierarchical decision making, prioritizing specific, key attributes over others (Table 2). Participants would compare those key attributes between the 2 health states and ultimately base their decision on this comparison, trading less preferable attributes in favor of the key attributes. Nevertheless, some participants were less focused on key attributes and more generally traded multiple less preferable attributes for preferable ones. Less pain, better mental health, and more years of life were frequently determined to be key attributes in a trade. Only when participants considered the combination of key attributes in description A to have similar values to the combination of those in description B did they consider other health dimensions (vitality; physical, role, and social functioning).
      Table 2Participant examples of strategies used.
      DCE indicates discrete choice experiment.

      Reinterpretation of health state

      Just less than half of participants (n = 17) would, for 1 or more choice tasks, reinterpret or change the underlying assumptions of 1 or both health states (Table 2). In using this strategy, participants refused to believe 1 or more of the attributes presented would remain the same for the duration of time provided or refused to believe attributes could go together. In not accepting the less desirable attributes as static, participants indicated they could change over time given the more positive attributes (eg, not having pain would, over time, improve mental health). Alternatively, in refusing to believe the attributes they were presented with could coexist (eg, severe pain and no social limitations), participants selected the health state in which the attributes seemed more consistent and in line with their expectations. Participants also reinterpreted the health states by inferring outside information that was not presented to them. For example, some rationalized living with high levels of pain because they could take pain medications. One participant interpreted all the health states to mean they had a chronic or fatal condition and made decisions based on that inference.

      Relying on previous experience/preconceived ideas of attribute

      The final strategy participants used when deciding between 2 health states was relying on previous experience or preconceived ideas of an attribute (Table 2). This strategy was used at least once by 13 participants. Participants were more apt to use this strategy for attributes with which they had personal experience, most commonly pain and mental health. Participants with experience with significant pain or episodes of depression or anxiety most often prioritized health states with the least impact in these dimensions, regardless of the severity of the other dimensions. This strategy was also used when participants more broadly considered their nonhealth life situations, including how the severity of some attributes might impact their ability to care for their families. Although the use of the other strategies may also have been influenced by personal experience, this strategy is distinct in that participants explicitly indicated when and why they were taking a specific experience or preconceived notion into account versus focusing in on a key attribute.

      Comparison of high and low scores and large versus small differences in scores

      Trading was the predominant strategy used across all DCEs regardless of utility score or the difference between the utility scores of the 2 health states (see Appendix 2, Table 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.07.018). Although some participants used multiple strategies to make their decisions, this approach did not appear to be related to either the utility score level or the difference between the 2 scores. Subsequently, we examined patterns in strategies used by how difficult participants rated the decision-making process (on a scale of 0-10). In general, participants tended to rate health state comparisons with smaller differences (< 2.4) as more difficult, but this trend did not hold up across all comparisons (see Appendix 2, Table 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.07.018). There was no clear pattern to the difficulty rating when comparing high and low utility scores.

      Age

      Age did not appear to make a difference in the strategies used when deciding between health states (Appendix 2, Table 2 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.07.018). All participants used trading at least once, followed by inferring additional information about a health state or an attribute, and finally relying on previous experience or preconceived ideas (specifically, making decisions based on pain and mental health). Participants aged 18 to 49 years tended to reinterpret health states to a larger degree than those aged 50 years and older (55% vs 35%).

      Chronic conditions

      The decision-making processes used by participants with a chronic condition varied slightly from those without a chronic condition (Appendix 2, Table 2 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.07.018). Although all members of both groups used trading, participants without a chronic condition also made decisions by reinterpreting health states more often than participants with a chronic condition did (63.3% vs 34.5%).

      Interaction between age and chronic conditions

      The data suggest decision-making processes were slightly different among individuals who were younger (18-49 years old) and had a chronic condition compared with younger participants without chronic disease (Appendix 2, Table 2 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.07.018). Younger participants with a chronic condition relied less on reinterpreting health states than those without (40% vs 70%) and relied more on previous experience (40% vs 20%). Given that there was only 1 participant older than the age 50 years without a chronic condition, it is not possible to make any comparison about the decision-making strategies that were used.

      Discussion

      This was a qualitative, cross-sectional, noninterventional study consisting of one-on-one, 75-minute interviews designed to provide insight into the decision-making process used by individuals participating in a DCE study comparing and selecting preferred SF6Dv2 health states. This research aimed to ascertain participants’ understanding of the task they were asked to complete and the strategies they used to do so by asking them to use a think-aloud approach. Data from the interviews were coded and analyzed to distill the decision-making strategies used by participants. Coding and analysis proceeded using an inductive approach, after which the research team identified how the data aligned with previous research.
      Using the think-aloud approach and follow-up questions with participants gave great insight into the strategies they used. The strategies identified in this study were (1) trading for a priority attribute over lower-priority attributes, (2) reinterpreting attributes and inferring additional information about health states, and (3) relying on preconceived ideas about attributes. Although trading was the dominant strategy used, participants were observed using the strategies in concert with one another.
      There was some evidence the choice of strategy was influenced by age group and health status. Older participants and those who live with a chronic condition did not need to infer additional information as frequently as those who were younger or did not have a chronic condition. This may have been because these individuals could more easily relate to attributes of a hypothetical health state given their lived experiences. A future study with a more precise distribution of sample by age would enable more in-depth analyses by chronic condition and age.
      The strategies participants used to complete choice tasks align with previous work examining strategies for evaluating health states, indicating decision-making strategies are common across groups and tasks. Previous studies have used a think-aloud approach to examine the decision-making strategies used when completing choice tasks across a number of content areas: breast cancer screening,
      • Vass C.
      • Rigby D.
      • Payne K.
      I was trying to do the maths”: exploring the impact of risk communication in discrete choice experiments.
      bowel cancer screening,
      • Ryan M.
      • Watson V.
      • Entwistle V.
      Rationalising the ‘irrational’: a think aloud study of discrete choice experiment responses.
      vaccination preferences,
      • Veldwijk J.
      • Determann D.
      • Lambooij M.S.
      • et al.
      Exploring how individuals complete the choice tasks in a discrete choice experiment: an interview study.
      ,
      • Ryan M.
      • Watson V.
      • Entwistle V.
      Rationalising the ‘irrational’: a think aloud study of discrete choice experiment responses.
      primary care preferences,
      • Cheraghi-Sohi S.
      • Bower P.
      • Mead N.
      • McDonald R.
      • Whalley D.
      • Roland M.
      Making sense of patient priorities: applying discrete choice methods in primary care using ‘think aloud’ technique.
      prostate cancer screening,
      • Veldwijk J.
      • Determann D.
      • Lambooij M.S.
      • et al.
      Exploring how individuals complete the choice tasks in a discrete choice experiment: an interview study.
      and preferences for funding new health technology assessments.
      • Whitty J.A.
      • Walker R.
      • Golenko X.
      • Ratcliffe J.
      A think aloud study comparing the validity and acceptability of discrete choice and best worst scaling methods.
      These previous studies indicate, regardless of the topic area or the population, the strategies used to complete choice tasks tend to be the same.
      Asking participants to complete this task using a think-aloud approach confirmed earlier findings that not all attributes are relevant or of high priority to all individuals and that they can and will disregard those low priority attributes when making their decision.
      • Ryan M.
      • Watson V.
      • Entwistle V.
      Rationalising the ‘irrational’: a think aloud study of discrete choice experiment responses.
      In the current study, the key, preferred attributes included less pain, better mental health, and longer duration of survival, whereas lower-priority attributes were better vitality, physical functioning, role functioning, and social functioning.
      In this study, attribute nonattendance was not observed, nor was it described during the think-aloud. This may be because the think-aloud exercise of the choice tasks followed a cognitive debriefing of the SF-6Dv2.
      • Broderick L.
      • Bjorner J.B.
      • Lauher-Charest M.
      • et al.
      Development of the SF-6Dv2 health utility survey: comprehensibility and patient preference.
      This meant participants were familiar with and had a clear understanding of the attributes
      • Clark M.D.
      • Determann D.
      • Petrou S.
      • Moro D.
      • de Bekker-Grob E.W.
      Discrete choice experiments in health economics: a review of the literature.
      and had already been considering how each attribute related to their daily lives, making the evaluation during the choice task more personal and less hypothetical in nature. Although not used exclusively, one participant completed choice tasks based solely on their perception of which scenario in the choice task was more “positive.” This approach has been observed in previous think-aloud studies
      • Whitty J.A.
      • Walker R.
      • Golenko X.
      • Ratcliffe J.
      A think aloud study comparing the validity and acceptability of discrete choice and best worst scaling methods.
      and has potential implications for the underlying assumption of DCE studies, which is that individuals make decisions based on preference. Additional decision-making heuristics were not observed.
      Some participants reinterpreted SF-6Dv2 health states to make them more palatable or to make the decision-making process easier. This aligns with previous studies in which individuals were found to reinterpret the attributes they were evaluating, including inferring causality between them.
      • Cheraghi-Sohi S.
      • Bower P.
      • Mead N.
      • McDonald R.
      • Whalley D.
      • Roland M.
      Making sense of patient priorities: applying discrete choice methods in primary care using ‘think aloud’ technique.
      ,
      • Ryan M.
      • Watson V.
      • Entwistle V.
      Rationalising the ‘irrational’: a think aloud study of discrete choice experiment responses.
      ,
      • Kenny P.
      • Hall J.
      • Viney R.
      • Haas M.
      Do participants understand a stated preference health survey? A qualitative approach to assessing validity.
      Reconstructing health states presented in choice tasks has potential issues for the assumptions underlying DCEs, namely, that they are being interpreted as written. It is advisable to revise the DCE instructions to alert participants that the health states they are comparing are hypothetical and should be considered “as written.”
      • Cheraghi-Sohi S.
      • Bower P.
      • Mead N.
      • McDonald R.
      • Whalley D.
      • Roland M.
      Making sense of patient priorities: applying discrete choice methods in primary care using ‘think aloud’ technique.
      ,
      • Pearce A.
      • Harrison M.
      • Watson V.
      • et al.
      Respondent understanding in discrete choice experiments: a scoping review.
      Asking participants to consider the health states “as written” also serves to focus those who may have previous experience with one of the disease states.
      This study is not without limitations. First, the think-aloud method itself has limitations. The request for verbalization may change the process of selecting health states. It is not always clear comments made when thinking aloud aligned with participants’ processing of the task. To mitigate this issue, a series of semistructured follow-up questions was asked after each DCE task to allow the interviewer to gain a better understanding of the person’s process and rationale for choosing a particular health state. Second, it is preferable to conduct these interviews in person, and although videoconferencing software mitigates this limitation, it did not eliminate it. Third, although all health states used in the DCEs had been reported in practice by at least 1 person from a large general population sample, interpretation of what could be considered an unusual health state is subjective. As described earlier, some found combinations unlikely. This has implications for how participants may have valued those specific choice tasks, thus affecting their overall preference. Fourth, the pairs of SF-6Dv2 health states participants were asked to evaluate and choose between did not include an opt-out option and forced participants to make a selection. This adds a degree of difficulty to interpreting the choices made by participants who did not have a clear preference.
      • Lancsar E.
      • Louviere J.
      Conducting discrete choice experiments to inform healthcare decision making: a user’s guide.
      Finally, participants were presented with fewer choice tasks than they would be asked in a typical DCE study. This may have reduced learning effects, survey fatigue, and decision fatigue.
      This study also had a number of strengths. In addition to participants being familiar with the attributes before completing the think-aloud of the DCE health states, qualitative research strategies confirmed their interpretations of the attributes were correct and provided the study team a clear understanding of the decision-making strategies being used. This qualitative understanding of the decision-making strategies has been highlighted as key to developing and conducting DCEs.
      • Clark M.D.
      • Determann D.
      • Petrou S.
      • Moro D.
      • de Bekker-Grob E.W.
      Discrete choice experiments in health economics: a review of the literature.
      ,
      • Lancsar E.
      • Louviere J.
      Conducting discrete choice experiments to inform healthcare decision making: a user’s guide.
      ,
      • Cheraghi-Sohi S.
      • Bower P.
      • Mead N.
      • McDonald R.
      • Whalley D.
      • Roland M.
      Making sense of patient priorities: applying discrete choice methods in primary care using ‘think aloud’ technique.
      ,
      • Ryan M.
      • Watson V.
      • Entwistle V.
      Rationalising the ‘irrational’: a think aloud study of discrete choice experiment responses.
      ,
      • Kenny P.
      • Hall J.
      • Viney R.
      • Haas M.
      Do participants understand a stated preference health survey? A qualitative approach to assessing validity.
      ,
      • Pearce A.
      • Harrison M.
      • Watson V.
      • et al.
      Respondent understanding in discrete choice experiments: a scoping review.
      ,
      • Vass C.
      • Rigby D.
      • Payne K.
      The role of qualitative research methods in discrete choice experiments.
      Additionally, imposing sampling quotas helps ensure the DCEs are understandable across broad populations.
      A further strength of this study was using a think-aloud approach to gain a deeper understanding of the decision-making strategies used when completing DCEs evaluating pairs of SF-6Dv2 health states. The think-aloud process highlighted the importance of ensuring the instructions for how to complete a DCE task are clear and uncomplicated. To that end, minor revisions to the instructions have been made so that when participants complete the DCE in future studies, they are clear on what is being asked of them.

      Conclusion

      To the best of the authors’ knowledge, this is the first qualitative study to investigate the decision-making strategies implicit in valuing multiattribute health utility measures such as the SF-6Dv2. Although qualitative studies have been conducted to evaluate the content validity of existing health utility measures, including the SF-6Dv2,
      • Broderick L.
      • Bjorner J.B.
      • Lauher-Charest M.
      • et al.
      Development of the SF-6Dv2 health utility survey: comprehensibility and patient preference.
      this study sought to gain qualitative insight into the decision-making strategies that drive the scoring of health utility measures, namely, the SF-6Dv2. The evidence from this study adds to the depth of understanding of the choice tasks for the SF-6Dv2, to the strength of the utility weights to be developed, and to the overall usefulness of the SF-6Dv2. The findings further suggest the methodology described in this study can and should be carried forth in valuing other health utility measures, not just the SF-6Dv2.

      Article and Author Information

      Author Contributions: Concept and design: Broderick, Bjorner, Lauher-Charest, White, Kosinski, Brazier
      Acquisition of data: Broderick, Lauher-Charest
      Analysis and interpretation of data: Broderick, Bjorner, Lauher-Charest, White, Mulhern, Brazier
      Drafting of the manuscript: Broderick, Lauher-Charest, White, Kosinski, Mulhern, Brazier
      Critical revision of the paper for important intellectual content: Broderick, Bjorner, Lauher-Charest, White, Kosinski, Mulhern, Brazier
      Provision of study materials or patients: Broderick, Lauher-Charest
      Obtaining funding: Bjorner
      Administrative, technical, or logistic support: Broderick, Lauher-Charest
      Supervision: Bjorner, White
      Conflict of Interest Disclosures: Dr Mulhern reported receiving grants outside the submitted work. He is also an editor of Value in Health and had no role in the peer-review process of this article. Dr Brazier reported receiving royalties paid to his institution based on the use of the SF-6D. No other disclosures were reported.
      Funding/Support: This work was supported by QualityMetric .
      Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

      References

        • Clark M.D.
        • Determann D.
        • Petrou S.
        • Moro D.
        • de Bekker-Grob E.W.
        Discrete choice experiments in health economics: a review of the literature.
        Pharmacoeconomics. 2014; 32: 883-902
        • Lancsar E.
        • Louviere J.
        Conducting discrete choice experiments to inform healthcare decision making: a user’s guide.
        Pharmacoeconomics. 2008; 26: 661-677
        • Mulhern B.
        • Norman R.
        • Brazier J.
        Valuing SF-6Dv2 in Australia using an international protocol.
        Pharmacoeconomics. 2021; 39: 1151-1162
        • Mulhern B.J.
        • Bansback N.
        • Norman R.
        • Brazier J.
        • SF-6Dv2 International Project Group
        Valuing the SF-6Dv2 classification system in the United Kingdom using a discrete-choice experiment with duration.
        Med Care. 2020; 58: 566-573
        • Norman R.
        • Cronin P.
        • Viney R.
        A pilot discrete choice experiment to explore preferences for EQ-5D-5L health states.
        Appl Health Econ Health Policy. 2013; 11: 287-298
        • Norman R.
        • Viney R.
        • Brazier J.
        • et al.
        Valuing SF-6D health states using a discrete choice experiment.
        Med Decis Making. 2014; 34: 773-786
        • Wu J.
        • Xie S.
        • He X.
        • et al.
        Valuation of SF-6Dv2 health states in China using time trade-off and discrete-choice experiment with a duration dimension.
        Pharmacoeconomics. 2021; 39: 521-535
        • Cheraghi-Sohi S.
        • Bower P.
        • Mead N.
        • McDonald R.
        • Whalley D.
        • Roland M.
        Making sense of patient priorities: applying discrete choice methods in primary care using ‘think aloud’ technique.
        Fam Pract. 2007; 24: 276-282
        • Lancsar E.
        • Louviere J.
        Deleting ‘irrational’ responses from discrete choice experiments: a case of investigating or imposing preferences?.
        Health Econ. 2006; 15: 797-811
        • Reed Johnson F.
        • Lancsar E.
        • Marshall D.
        • et al.
        Constructing experimental designs for discrete-choice experiments: report of the ISPOR Conjoint Analysis Experimental Design Good Research Practices Task Force.
        Value Health. 2013; 16: 3-13
        • Veldwijk J.
        • Determann D.
        • Lambooij M.S.
        • et al.
        Exploring how individuals complete the choice tasks in a discrete choice experiment: an interview study.
        BMC Med Res Methodol. 2016; 16: 45
        • Ryan M.
        • Watson V.
        • Entwistle V.
        Rationalising the ‘irrational’: a think aloud study of discrete choice experiment responses.
        Health Econ. 2009; 18: 321-336
        • Waters E.A.
        • Weinstein N.D.
        • Colditz G.A.
        • Emmons K.
        Formats for improving risk communication in medical tradeoff decisions.
        J Health Commun. 2006; 11: 167-182
        • Kenny P.
        • Hall J.
        • Viney R.
        • Haas M.
        Do participants understand a stated preference health survey? A qualitative approach to assessing validity.
        Int J Technol Assess Health Care. 2003; 19: 664-681
        • Bryan S.
        • Dolan P.
        Discrete choice experiments in health economics. For better or for worse?.
        Eur J Health Econ. 2004; 5: 199-202
        • Lagarde M.
        Investigating attribute non-attendance and its consequences in choice experiments with latent class models.
        Health Econ. 2013; 22: 554-567
        • Miguel F.S.
        • Ryan M.
        • Amaya-Amaya M.
        ‘Irrational’ stated preferences: a quantitative and qualitative investigation.
        Health Econ. 2005; 14: 307-322
        • Vass C.
        • Rigby D.
        • Payne K.
        I was trying to do the maths”: exploring the impact of risk communication in discrete choice experiments.
        Patient. 2019; 12: 113-123
        • Mulhern B.
        • Bansback N.
        • Brazier J.
        • et al.
        Preparatory study for the revaluation of the EQ-5D tariff: methodology report.
        Health Technol Assess. 2014; 18 (vii-xxvi, 1-191)
        • Morgan A.
        • Shackley P.
        • Pickin M.
        • Brazier J.
        Quantifying patient preferences for out-of-hours primary care.
        J Health Serv Res Policy. 2000; 5: 214-218
        • Vick S.
        • Scott A.
        Agency in health care. Examining patients’ preferences for attributes of the doctor-patient relationship.
        J Health Econ. 1998; 17: 587-605
        • Whitty J.A.
        • Walker R.
        • Golenko X.
        • Ratcliffe J.
        A think aloud study comparing the validity and acceptability of discrete choice and best worst scaling methods.
        PLoS One. 2014; 9e90635
        • Drennan J.
        Cognitive interviewing: verbal data in the design and pretesting of questionnaires.
        J Adv Nurs. 2003; 42: 57-63
        • Padilla J.-L.
        • Leighton J.P.
        Cognitive interviewing and think aloud methods.
        in: Understanding and Investigating Response Processes in Validation Research. Vol. 69. Springer International Publishing, Cham, Switzerland2017: 211-228
        • Collins D.
        Pretesting survey instruments: an overview of cognitive methods.
        Qual Life Res. 2003; 12: 229-238
        • Willis G.B.
        Cognitive Interviewing: A Tool for Improving Questionnaire Design.
        Sage Publications, Thousand Oaks, CA2005
        • Patrick D.L.
        • Burke L.B.
        • Gwaltney C.J.
        • et al.
        Content validity--establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO Good Research Practices Task Force report: part 2--assessing respondent understanding.
        Value Health. 2011; 14: 978-988
        • Taylor D.C.
        • Kosinski M.
        • Reilly K.
        • Lindner L.
        Comparison of the burden of Ibs with constipation on health-related quality of life (HRQOL), work productivity, and health care utilization to asthma, migraine, and rheumatoid arthritis in the US, UK, and France.
        Value Health. 2014; 17: A371-A372
        • Kiger M.E.
        • Varpio L.
        Thematic analysis of qualitative data: AMEE guide no. 131.
        Med Teach. 2020; 42: 846-854
        • Castleberry A.
        • Nolen A.
        Thematic analysis of qualitative research data: is it as easy as it sounds?.
        Curr Pharm Teach Learn. 2018; 10: 807-815
        • Broderick L.
        • Bjorner J.B.
        • Lauher-Charest M.
        • et al.
        Development of the SF-6Dv2 health utility survey: comprehensibility and patient preference.
        J Patient Rep Outcomes. 2022; 6: 47
        • Pearce A.
        • Harrison M.
        • Watson V.
        • et al.
        Respondent understanding in discrete choice experiments: a scoping review.
        Patient. 2021; 14: 17-53
        • Vass C.
        • Rigby D.
        • Payne K.
        The role of qualitative research methods in discrete choice experiments.
        Med Decis Making. 2017; 37: 298-313