Advertisement

Suitability of Preference Methods Across the Medical Product Lifecycle: A Multicriteria Decision Analysis

Open AccessPublished:December 08, 2022DOI:https://doi.org/10.1016/j.jval.2022.11.019

      Highlights

      • Given that there is no gold standard method for eliciting preferences, the selection of a patient preference method should be based on decision makers’ needs and performance of methods under consideration to meet those needs. Five commonly used preference elicitation methods were compared: discrete choice experiments, swing weighting, probabilistic threshold technique, and best-worst scale cases 1 and 2.
      • Whether a method estimates relative importance and trade-offs between treatment characteristics was considered important to decision makers across all medical product lifecycle (MPLC) decision points. Generally, obtaining qualitative information was more important during early lifecycle decisions, whereas internal and external validity and identifying preference heterogeneity were more important during later decision points. Methods performed relatively well on the criteria that were most important at the different decision points of the MPLC; nevertheless, the value of individual methods was sensitive to changes in the performance matrix.
      • Although discrete choice experiment is the most applied preference elicitation method, best-worst scale, swing weighting, and probabilistic threshold technique should also be considered to address the needs of decision makers throughout the MPLC as they comply with top-weighted methods criteria according to MPLC decision makers.

      Abstract

      Objectives

      This study aimed to understand the importance of criteria describing methods (eg, duration, costs, validity, and outcomes) according to decision makers for each decision point in the medical product lifecycle (MPLC) and to determine the suitability of a discrete choice experiment, swing weighting, probabilistic threshold technique, and best-worst scale cases 1 and 2 at each decision point in the MPLC.

      Methods

      Applying multicriteria decision analysis, an online survey was sent to MPLC decision makers (ie, industry, regulatory, and health technology assessment representatives). They ranked and weighted 19 methods criteria from an existing performance matrix about their respective decisions across the MPLC. All criteria were given a relative weight based on the ranking and rating in the survey after which an overall suitability score was calculated for each preference elicitation method per decision point. Sensitivity analyses were conducted to reflect uncertainty in the performance matrix.

      Results

      Fifty-nine industry, 29 regulatory, and 5 health technology assessment representatives completed the surveys. Overall, “estimating trade-offs between treatment characteristics” and “estimating weights for treatment characteristics” were highly important criteria throughout all MPLC decision points, whereas other criteria were most important only for specific MPLC stages. Swing weighting and probabilistic threshold technique received significantly higher suitability scores across decision points than other methods. Sensitivity analyses showed substantial impact of uncertainty in the performance matrix.

      Conclusion

      Although discrete choice experiment is the most applied preference elicitation method, other methods should also be considered to address the needs of decision makers. Development of evidence-based guidance documents for designing, conducting, and analyzing such methods could enhance their use.

      Keywords

      Introduction

      Increasingly decision makers look for ways to measure patients’ preferences and include such information in decision making along the medical product lifecycle (MPLC).
      • de Bekker-Grob E.W.
      • Berlin C.
      • Levitan B.
      • et al.
      Giving patients’ preferences a voice in medical treatment life cycle: the PREFER public-private project.
      Including preference information might be apparent for some decisions such as for identifying unmet medical needs and selecting endpoints for randomized controlled trial
      • McLeod C.
      • Norman R.
      • Litton E.
      • Saville B.R.
      • Webb S.
      • Snelling T.L.
      Choosing primary endpoints for clinical trials of health care interventions.
      ,
      • Patalano F.
      • Gutzwiller F.S.
      • Shah B.
      • Kumari C.
      • Cook N.S.
      Gathering structured patient insight to drive the PRO strategy in COPD: patient-centric drug development from theory to practice.
      studies from a patient perspective
      • van Overbeeke E.
      • Vanbinst I.
      • Jimenez-Moreno A.C.
      • Huys I.
      Patient centricity in patient preference studies: the patient perspective.
      or for the purpose of quantitative benefit-risk assessment.
      • Smith M.Y.
      • van Til J.
      • DiSantostefano R.L.
      • Hauber A.B.
      • Marsh K.
      Quantitative benefit-risk assessment: state of the practice within industry.
      Nevertheless, the exact role of patient preferences in other industry decision points, especially regulatory and health technology assessment (HTA)/reimbursement-related decisions, is less clear. Both regulatory and HTA agencies advocate for increased transparency in their decision-making process and a focus on patient-centered decision making.
      • Johnson F.R.
      • Zhou M.
      Patient preferences in regulatory benefit-risk assessments: a US perspective.
      • Muhlbacher A.C.
      • Juhnke C.
      • Beyer A.R.
      • Garner S.
      Patient-focused benefit-risk analysis to inform regulatory decisions: the European Union perspective.
      • Mott D.J.
      Incorporating quantitative patient preference data into healthcare decision making processes: is HTA falling behind?.
      The Food and Drug Administration has issued guidance for the conduct of patient preference studies (PPS),
      Patient preference information – voluntary submission, review in premarket approval applications, humanitarian device exemption applications, and de novo requests, and inclusion in decision summaries and device labeling: guidance for industry, food and drug administration staff, and other stakeholders. U.S. Food and Drug Administration.
      and the European Medicines Agency recently provided a positive qualification opinion and asked for public consultation on a preference elicitation framework.
      Qualification opinion of IMI PREFER
      European Medicines Agency.
      For HTA and reimbursement, the inclusion of preferences in decision making seems more distant.
      • Mott D.J.
      Incorporating quantitative patient preference data into healthcare decision making processes: is HTA falling behind?.
      Current cost-utility analysis frameworks do not allow for easy inclusion of patient preference information and require more structural changes.
      • Dirksen C.D.
      The use of research evidence on patient preferences in health care decision-making: issues, controversies and moving forward.
      ,
      • Huls S.P.I.
      • Whichello C.L.
      • van Exel J.
      • Uyl-de Groot C.A.
      • de Bekker-Grob E.W.
      What is next for patient preferences in health technology assessment? A systematic review of the challenges.
      Nevertheless, initiatives are undertaken; for instance, the National Institute for Health and Care Excellence published their perspective on the use of PPS in HTA
      • Bouvy J.C.
      • Cowie L.
      • Lovett R.
      • Morrison D.
      • Livingstone H.
      • Crabb N.
      Use of patient preference studies in HTA decision making: a NICE perspective.
      ,
      • Cowie L.
      • Bouvy J.C.
      Measuring patient preferences: an exploratory study to determine how patient preferences data could be used in health technology assessment (HTA). MyelomaUK.
      and provided scientific advice on the conduct of PPS.
      • Patalano F.
      • Gutzwiller F.S.
      • Shah B.
      • Kumari C.
      • Cook N.S.
      Gathering structured patient insight to drive the PRO strategy in COPD: patient-centric drug development from theory to practice.
      Nevertheless, the weighting or incorporation of preferences against the standard information (eg, clinical data, cost-effectiveness data) in decision making along the MPLC remains debated. According to previous research, the MPLC, in total, consists of approximately 15 decision points for different decision makers: pharmaceutical industry, regulators, and HTA agency/body.
      • Whichello C.
      • Bywall K.S.
      • Mauer J.
      • et al.
      An overview of critical decision-points in the medical product lifecycle: where to include patient preference information in the decision-making process?.
      Decision makers themselves indicated that most of these decisions could include patient preference information to some extent.
      • Whichello C.
      • Bywall K.S.
      • Mauer J.
      • et al.
      An overview of critical decision-points in the medical product lifecycle: where to include patient preference information in the decision-making process?.
      At the same time, decision makers likely require different types of information with varying depth and focus along the MPLC,
      • Whichello C.
      • Bywall K.S.
      • Mauer J.
      • et al.
      An overview of critical decision-points in the medical product lifecycle: where to include patient preference information in the decision-making process?.
      making it complicated to select one or few suitable preference elicitation methods that fit the needs of all decision makers across all decision points of the MPLC.
      A recent literature review identified 22 preference elicitation methods
      • Soekhai V.
      • Whichello C.
      • Levitan B.
      • et al.
      Methods for exploring and eliciting patient preferences in the medical product lifecycle: a literature review.
      grouped into ranking, rating, indifference methods, and discrete choice methods. Within each category, most commonly used methods in healthcare to elicit preferences of patients were discrete choice experiment (DCE),
      • Soekhai V.
      • de Bekker-Grob E.W.
      • Ellis A.R.
      • Vass C.M.
      Discrete choice experiments in health economics: past, present and future.
      probabilistic threshold technique (PTT),
      • Hauber B.
      • Coulter J.
      Using the threshold technique to elicit patient preferences: an introduction to the method and an overview of existing empirical applications.
      swing weighting (SW),
      • Marsh K.
      • IJerman M.
      • Thokala P.
      • et al.
      Multiple criteria decision analysis for health care decision making--emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force.
      ,
      • Thokala P.
      • Devlin N.
      • Marsh K.
      • et al.
      Multiple criteria decision analysis for health care decision making--an introduction: report 1 of the ISPOR MCDA Emerging Good Practices Task Force.
      best-worst scaling case 1 (BWS1),
      • Flynn T.N.
      • Louviere J.J.
      • Peters T.J.
      • Coast J.
      Best--worst scaling: what it can do for health care research and how to do it.
      and best-worst scaling case 2 (BWS2).
      • Flynn T.N.
      • Louviere J.J.
      • Peters T.J.
      • Coast J.
      Best--worst scaling: what it can do for health care research and how to do it.
      In a first effort to identify methods most suitable for satisfying stakeholders’ needs, Whichello and colleagues
      • Whichello C.
      • Levitan B.
      • Juhaeri J.
      • et al.
      Appraising patient preference methods for decision-making in the medical product lifecycle: an empirical comparison.
      combined a Q-method and analytical hierarchy process to appraise all 22 preference elicitation methods. The relative weight of criteria describing methods (eg, duration, costs, validity, and outcomes) was evaluated for 4 hypothetical MPLC scenarios: 2 variations of early clinical development stages, 1 late phase III scenario, and 1 postmarketing scenario.
      • Whichello C.
      • Levitan B.
      • Juhaeri J.
      • et al.
      Appraising patient preference methods for decision-making in the medical product lifecycle: an empirical comparison.
      Weighting of criteria for methods appraisal was mostly based on response of representatives from industry and academia (1 HTA and 1 regulator responded).
      • Whichello C.
      • Levitan B.
      • Juhaeri J.
      • et al.
      Appraising patient preference methods for decision-making in the medical product lifecycle: an empirical comparison.
      To provide insight into suitability of methods across the full MPLC and thereby facilitate methods selection and systematic implementation of preference elicitation along the MPLC, it remains crucial to understand the importance of methods criteria for each decision maker (ie, industry, regulator, HTA agency/body) at their respective critical decision points in the MPLC.
      • Whichello C.
      • van Overbeeke E.
      • Janssens R.
      • et al.
      Factors and situations affecting the value of patient preference studies: semi-structured interviews in Europe and the US.
      • van Overbeeke E.
      • Janssens R.
      • Whichello C.
      • et al.
      Design, conduct, and use of patient preference studies in the medical product life cycle: a multi-method study.
      • van Overbeeke E.
      • Whichello C.
      • Janssens R.
      • et al.
      Factors and situations influencing the value of patient preference studies along the medical product lifecycle: a literature review.
      Therefore, this study aimed to evaluate the importance of methods criteria to fully appraise the performance of 5 commonly used preference elicitation methods against these methods criteria according to decision makers at different moments along the MPLC.

      Methods

      We used multicriteria decision analysis (MCDA) in this study, which is a methodology for appraising alternatives on individual, often conflicting criteria, and combining them into 1 overall appraisal.
      • Keeney R.L.
      • Raiffa H.
      Decisions With Multiple Objectives: Preferences and Value Trade-Offs.
      Common steps in MCDA are (1) defining the decision problem (including decision makers), (2) selecting criteria, (3) measuring performance, (4) weighting of criteria, (5) aggregating results, (6) sensitivity analyses, and (7) interpretation of results.
      • Thokala P.
      • Devlin N.
      • Marsh K.
      • et al.
      Multiple criteria decision analysis for health care decision making--an introduction: report 1 of the ISPOR MCDA Emerging Good Practices Task Force.
      Step 2 to 6 will be detailed below given that step 1 was outlined in the introduction (ie, to provide insight into suitability of methods across the full MPLC and thereby facilitate methods selection and systematic implementation of preference elicitation along the MPLC) and step 7 will be covered in the Results and Discussion sections of this article.

      Selecting Criteria and Measuring Performance

      Whichello et al
      • Whichello C.
      • Levitan B.
      • Juhaeri J.
      • et al.
      Appraising patient preference methods for decision-making in the medical product lifecycle: an empirical comparison.
      initially identified 35 method criteria as being most important for selecting a qualitative or quantitative patient preference method based on literature reviews and previous studies. These were subsequently restricted to 19 criteria (12 operational and 7 outcomes related criteria) by means of a Q-method experiment among stakeholders (N = 54 being academic, representative from industry or regulatory/HTA agency, physician, patient (representative), or consultant) (Table 1
      • Whichello C.
      • van Overbeeke E.
      • Janssens R.
      • et al.
      Factors and situations affecting the value of patient preference studies: semi-structured interviews in Europe and the US.
      ). Whichello et al
      • Whichello C.
      • Levitan B.
      • Juhaeri J.
      • et al.
      Appraising patient preference methods for decision-making in the medical product lifecycle: an empirical comparison.
      subsequently developed a performance matrix (Table 2
      • Whichello C.
      • Levitan B.
      • Juhaeri J.
      • et al.
      Appraising patient preference methods for decision-making in the medical product lifecycle: an empirical comparison.
      ,
      • Jonker M.
      • de Bekker-Grob E.
      • Veldwijk J.
      • Goossens L.
      • Bour S.
      • Rutten-Van Molken M.
      COVID-19 contact tracing apps: predicted uptake in the Netherlands based on a discrete choice experiment.
      • Jonker M.F.
      • Donkers B.
      • Goossens L.M.A.
      • et al.
      Summarizing patient preferences for the competitive landscape of multiple sclerosis treatment options.
      • Rutten-van Molken M.
      • Karimi M.
      • Leijten F.
      • et al.
      Comparing patients’ and other stakeholders’ preferences for outcomes of integrated care for multimorbidity: a discrete choice experiment in eight European countries.
      • Veldwijk J.
      • Johansson J.V.
      • Donkers B.
      • de Bekker-Grob E.W.
      Mimicking real-life decision making in health: allowing respondents time to think in a discrete choice experiment.
      • Visser L.A.
      • Huls S.P.I.
      • Uyl-de Groot C.A.
      • de Bekker-Grob E.W.
      • Redekop W.K.
      An implantable device to treat multiple sclerosis: a discrete choice experiment on patient preferences in three European countries.
      ) specifying the performance of each method for each criterion was created based on semistructured interviews with preference method experts (N = 17) and a literature review. Further details on the method and development of the performance matrix can be found elsewhere.
      • Whichello C.
      • Levitan B.
      • Juhaeri J.
      • et al.
      Appraising patient preference methods for decision-making in the medical product lifecycle: an empirical comparison.
      Table 1Overview of methods criteria identified by Whichello and colleagues
      • Whichello C.
      • van Overbeeke E.
      • Janssens R.
      • et al.
      Factors and situations affecting the value of patient preference studies: semi-structured interviews in Europe and the US.
      CriteriaShort description
      Operational criteria
      Low cost of the patient preference studyThe patient preference study can be conducted at a relatively low cost.
      Quick sessions with participants (≤ 30 min)Completing the patient preference study requires less than 30 min of the patient.
      Low frequency of sessions (< 2)The number of interactions required with each respondent over the course of the entire data collection period is less than 2.
      Study duration (≤ 6 months)The time needed for preparing the study, collecting data, and conducting analysis is less than 6 months.
      8 or more treatment characteristics can be exploredMeasuring the preferences of patients for 8 or more treatment characteristics.
      Small sample size (≤ 100)The patient preference study can accurately be conducted in a sample of less than 100 patients.
      A low cognitive load on patientsIt is important that participating in the patient preference study does not require a low cognitive load on the patients. The preference study could easily be completed by populations who experience heavy cognitive loads or struggle with cognitive tasks.
      Low complexity of instructionsThe instructions that patients need to read about or listen to before being able to participate in the patient preference study are low in complexity.
      Public acknowledgment by your organization as an acceptable methodYour organization or stakeholder group recognizes the method that is used to measure patient preferences as an acceptable method.
      Easy to add new treatment characteristicsIt is easy to add new treatment characteristics to the patient preference study while it is conducted without rendering all the previous data collected meaningless.
      The patient preference study does not include interaction among participantsPatient can complete the preference study on their own and do not need to interact with other patients.
      Group dynamic with participantsPatients interact with each other during in their participation in the preference study.
      Outcome-related criteria
      The patient preference study results allow for the calculation of risk attitudesWhether the analysis of the results of the patient preference study can be used to calculate risk attitudes, such as risk tolerance vs risk aversion
      Exploring reasons behind a preference in qualitative detailQualitative methods or mixed methods can often be used to find out the “why” behind a particular preference or choice or why a participant has made a trade-off or selected a characteristic of a health intervention as being more important than another.
      Estimating weights for treatment characteristicsWhether a patient preference study can estimate weights and thereby tell you how much each treatment characteristic matters to patients
      Estimating trade-offs between treatment characteristicsWhether the results of the patient preference study can be used to calculate for instance maximum acceptable risk. This type of information tells you both how much each treatment characteristic matters and how much of one characteristic patients are willing to lose to gain on another characteristic.
      Quantifying heterogeneity in preferencesWhether or not the results of the patient preference study allow for the estimation of preference heterogeneity. Some preference heterogeneity may be explained by differences in observable patient characteristics.
      Internal validation methods can be incorporatedWhether or not the method used to measure patients’ preferences allows for the inclusion of internal validation measures. Validity means the extent to which a test or study measures what it claims to measure. Internal validity refers to whether a finding that incorporates a causal relationship between 2 or more variables is sound.
      Establishes external validityWhether or not the method used in the patient preference study is proven to be externally valid. External validity refers to whether the results of a study can be generalized beyond the specific research context in which the study was conducted.
      Table 2Performance matrix for DCE, BWS1, BWS2, SW, and PTT separately.
      CriteriaDCEBWS1BWS2SWPTT
      Low cost of the patient preference study1
      Low costs for DCE as the qualitative work across methods is equally much and specialized software and expertise for DCE is no longer a necessity (free packages such as R offer experimental design and advanced statistical modeling options).
      1
      Indicates uncertainty in whether the method does or does not comply with a criterium as specified by Whichello and colleagues.22
      11
      Indicates uncertainty in whether the method does or does not comply with a criterium as specified by Whichello and colleagues.22
      1
      Indicates uncertainty in whether the method does or does not comply with a criterium as specified by Whichello and colleagues.22
      Quick sessions with participants (≤ 30 min)11111
      Indicates uncertainty in whether the method does or does not comply with a criterium as specified by Whichello and colleagues.22
      Low frequency of sessions (< 2)11111
      Study duration(≤ 6 months)1111
      Indicates uncertainty in whether the method does or does not comply with a criterium as specified by Whichello and colleagues.22
      1
      8 or more treatment characteristics can be explored01
      Indicates uncertainty in whether the method does or does not comply with a criterium as specified by Whichello and colleagues.22
      0
      BWS2 such as DCE is not advised with > 8 attributes owing to complexity of choice tasks.
      10
      Indicates uncertainty in whether the method does or does not comply with a criterium as specified by Whichello and colleagues.22
      Small sample size (≤ 100)01
      Indicates uncertainty in whether the method does or does not comply with a criterium as specified by Whichello and colleagues.22
      0
      Sample size of DCE and BWS2 cannot be > 100 to perform the common practice statical models (conditional logit, MIXL, or LCA).
      11
      A low cognitive load on patients1
      Cognitive load low for all methods in accordance with results of recent publications.27-31
      1
      Indicates uncertainty in whether the method does or does not comply with a criterium as specified by Whichello and colleagues.22
      1
      Cognitive load low for all methods in accordance with results of recent publications.27-31
      1
      Indicates uncertainty in whether the method does or does not comply with a criterium as specified by Whichello and colleagues.22
      1
      Indicates uncertainty in whether the method does or does not comply with a criterium as specified by Whichello and colleagues.22
      Low complexity of instructions00
      Indicates uncertainty in whether the method does or does not comply with a criterium as specified by Whichello and colleagues.22
      000
      Indicates uncertainty in whether the method does or does not comply with a criterium as specified by Whichello and colleagues.22
      Public acknowledgment by your organization as an acceptable method11
      Indicates uncertainty in whether the method does or does not comply with a criterium as specified by Whichello and colleagues.22
      111
      Easy to add new treatment characteristics00011
      The patient preference study does not include interaction among participants11111
      Group dynamic with participants0000
      Indicates uncertainty in whether the method does or does not comply with a criterium as specified by Whichello and colleagues.22
      ,
      Group dynamic for SW unrealistic as also in lab conducted experiment you get individual outcomes.
      0
      Indicates uncertainty in whether the method does or does not comply with a criterium as specified by Whichello and colleagues.22
      The patient preference study results allow for the calculation of risk attitudes1000
      Calculation of risk attitudes not possible in BWS and SW given that attributes are not actively traded against each other such as in DCE and PTT where people can focus on avoiding (all) risks.
      1
      Exploring reasons behind a preference in qualitative detail00000
      Estimating weights for treatment characteristics11111
      Estimating trade-offs between treatment characteristics1000
      Trade-offs between treatment characteristics are not common practice in SW; it can theoretically be done but only with (too) many assumptions.
      1
      Quantifying heterogeneity in preferences11111
      Internal validation methods can be incorporated11111
      Establishes external validity00000
      Note. 1 indicates the method complies to the criteria and 0 means it does not. Further indications for changes compared with the original performance matrix and explanations of reasoning behind all changes have been indicated using symbols and to ∗∗.
      BWS indicates best-worst scale; DCE, discrete choice experiment; LCA, latent class analysis; MIXL, mixed logit model; PTT, probabilistic threshold technique; SW, swing weighting.
      Low costs for DCE as the qualitative work across methods is equally much and specialized software and expertise for DCE is no longer a necessity (free packages such as R offer experimental design and advanced statistical modeling options).
      Indicates uncertainty in whether the method does or does not comply with a criterium as specified by Whichello and colleagues.
      • Whichello C.
      • Levitan B.
      • Juhaeri J.
      • et al.
      Appraising patient preference methods for decision-making in the medical product lifecycle: an empirical comparison.
      BWS2 such as DCE is not advised with > 8 attributes owing to complexity of choice tasks.
      § Sample size of DCE and BWS2 cannot be > 100 to perform the common practice statical models (conditional logit, MIXL, or LCA).
      ǁ Cognitive load low for all methods in accordance with results of recent publications.
      • Jonker M.
      • de Bekker-Grob E.
      • Veldwijk J.
      • Goossens L.
      • Bour S.
      • Rutten-Van Molken M.
      COVID-19 contact tracing apps: predicted uptake in the Netherlands based on a discrete choice experiment.
      • Jonker M.F.
      • Donkers B.
      • Goossens L.M.A.
      • et al.
      Summarizing patient preferences for the competitive landscape of multiple sclerosis treatment options.
      • Rutten-van Molken M.
      • Karimi M.
      • Leijten F.
      • et al.
      Comparing patients’ and other stakeholders’ preferences for outcomes of integrated care for multimorbidity: a discrete choice experiment in eight European countries.
      • Veldwijk J.
      • Johansson J.V.
      • Donkers B.
      • de Bekker-Grob E.W.
      Mimicking real-life decision making in health: allowing respondents time to think in a discrete choice experiment.
      • Visser L.A.
      • Huls S.P.I.
      • Uyl-de Groot C.A.
      • de Bekker-Grob E.W.
      • Redekop W.K.
      An implantable device to treat multiple sclerosis: a discrete choice experiment on patient preferences in three European countries.
      Group dynamic for SW unrealistic as also in lab conducted experiment you get individual outcomes.
      £ Calculation of risk attitudes not possible in BWS and SW given that attributes are not actively traded against each other such as in DCE and PTT where people can focus on avoiding (all) risks.
      ∗∗ Trade-offs between treatment characteristics are not common practice in SW; it can theoretically be done but only with (too) many assumptions.

      Weighting of Criteria

      Three surveys were developed to assess the relative importance, that is, the weights, of the methods criteria for each of the critical decision points in the MPLC in which patient preference information could be considered in addition to the current evidence used for decision making. Furthermore, the surveys were tailored to decision processes of the 3 decision-maker groups in the MPLC in such a way that surveys for industry representative included 6 industry-related decision points (ie, select and prioritize targets and leads, prioritize studies, prioritize assets, optimize and prioritize assets, regulatory submission and launch, manage MPLC, and prioritize opportunities), the survey for regulators contained 1 regulatory decision point (scientific opinion), and the survey for HTA agency/body representative included 1 HTA decision point (appraisal).
      Respondents (recruitment strategies are described below) were invited to participate by sharing the link for the survey and an explanatory letter. The survey started with an explanation of what a PPS constitutes, and after that, 4 background questions were included to get insight in the respondents and their experience with such preference studies. In the next part of the survey, the respondents were asked to rank the method criteria included in the performance matrix (Table 2
      • Whichello C.
      • Levitan B.
      • Juhaeri J.
      • et al.
      Appraising patient preference methods for decision-making in the medical product lifecycle: an empirical comparison.
      ,
      • Jonker M.
      • de Bekker-Grob E.
      • Veldwijk J.
      • Goossens L.
      • Bour S.
      • Rutten-Van Molken M.
      COVID-19 contact tracing apps: predicted uptake in the Netherlands based on a discrete choice experiment.
      • Jonker M.F.
      • Donkers B.
      • Goossens L.M.A.
      • et al.
      Summarizing patient preferences for the competitive landscape of multiple sclerosis treatment options.
      • Rutten-van Molken M.
      • Karimi M.
      • Leijten F.
      • et al.
      Comparing patients’ and other stakeholders’ preferences for outcomes of integrated care for multimorbidity: a discrete choice experiment in eight European countries.
      • Veldwijk J.
      • Johansson J.V.
      • Donkers B.
      • de Bekker-Grob E.W.
      Mimicking real-life decision making in health: allowing respondents time to think in a discrete choice experiment.
      • Visser L.A.
      • Huls S.P.I.
      • Uyl-de Groot C.A.
      • de Bekker-Grob E.W.
      • Redekop W.K.
      An implantable device to treat multiple sclerosis: a discrete choice experiment on patient preferences in three European countries.
      ) from most to least important for each of the decision points that related to their specific decisional framework (for instance, HTA representatives were only asked about the importance of the methods criteria related to appraisal). To avoid ordering bias, the order in which the criteria were presented to respondents was randomized. For the criteria that a respondent ranked in their top 10, the respondents were asked to rate (on a scale of 100) the criterion compared with their top-ranked criterion, the score of which was set to 100. They were specifically not asked to weight all 19 criteria owing to the high cognitive burden of such a task resulting in fatigue and further potential bias induced by such a request.
      • Marsh K.
      • IJerman M.
      • Thokala P.
      • et al.
      Multiple criteria decision analysis for health care decision making--emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force.
      The surveys were constructed by the research team and reviewed by different decision makers (ie, 5 industry representatives, 2 Food and Drug Administration representatives, and 1 Belgium HTA representative [also representing the EUnetHTA]). Thereafter, the surveys were pretested by means of 3 think-aloud interviews (using conveyance sampling) to refine language, relevance, and usability of the survey. After the pretest, changes were made to the surveys related to the explanation of the decision points, what constitutes a preference study and the content/meaning of the criteria. The surveys were developed in Lighthouse Studio 9.7.0. Industry representatives within the Patient Preferences in Benefit-Risk Assessments During the Drug Life Cycle (PREFER) consortium and the Benefit-Risk Assessment, Communication, and Evaluation special interest group were asked to invite industry representatives to complete the survey. When disseminating the survey, it was requested to forward the invitation to colleagues at different departments (eg, regulatory-policy, drug safety, epidemiology, clinical development, health outcomes research, value and access groups). Regulatory representatives were contacted via the European Medicines Agency. HTA agency/body representatives were contacted via the head of the PREFER HTA advisory board. In total, 20 to 40 respondents per group of decision makers were anticipated to result in sufficient data to arrive at meaningful conclusions.
      • Tervonen T.
      • Gelhorn H.
      • Sri Bhashyam S.
      • et al.
      MCDA swing weighting and discrete choice experiments for elicitation of patient benefit-risk preferences: a critical assessment.

      Aggregation

      Based on the ranking position and the points from the rating exercise, all the criteria were given a relative average weight wi for each decision point.
      • Belton V.
      • Stewart T.J.
      Multiple Criteria Decision Analysis: An Integrated Approach.
      ,
      • Marsh K.
      • Goetghebeur M.
      • Thokala P.
      • Baltussen R.
      Multi-Criteria Decision Analysis to Support Healthcare Decisions.
      For each respondent, the criteria with an individual ranking outside the top 10 were given a weight of 0. The weights of the other criteria were calculated by scaling the ratings with the total sum of the points such that the sum of all weights equaled to 100. Next, the individual weights were averaged over all respondents giving the average weight wi for each criterion. Subsequently, for each preference elicitation method, an overall value was calculated per critical decision point along the MPLC based on whether the methods met certain criteria (the scoring, see performance matrix in Table 2
      • Whichello C.
      • Levitan B.
      • Juhaeri J.
      • et al.
      Appraising patient preference methods for decision-making in the medical product lifecycle: an empirical comparison.
      ,
      • Jonker M.
      • de Bekker-Grob E.
      • Veldwijk J.
      • Goossens L.
      • Bour S.
      • Rutten-Van Molken M.
      COVID-19 contact tracing apps: predicted uptake in the Netherlands based on a discrete choice experiment.
      • Jonker M.F.
      • Donkers B.
      • Goossens L.M.A.
      • et al.
      Summarizing patient preferences for the competitive landscape of multiple sclerosis treatment options.
      • Rutten-van Molken M.
      • Karimi M.
      • Leijten F.
      • et al.
      Comparing patients’ and other stakeholders’ preferences for outcomes of integrated care for multimorbidity: a discrete choice experiment in eight European countries.
      • Veldwijk J.
      • Johansson J.V.
      • Donkers B.
      • de Bekker-Grob E.W.
      Mimicking real-life decision making in health: allowing respondents time to think in a discrete choice experiment.
      • Visser L.A.
      • Huls S.P.I.
      • Uyl-de Groot C.A.
      • de Bekker-Grob E.W.
      • Redekop W.K.
      An implantable device to treat multiple sclerosis: a discrete choice experiment on patient preferences in three European countries.
      ) and the points allocated to that criterion (the weighting) for that particular decision point.
      • Belton V.
      • Stewart T.J.
      Multiple Criteria Decision Analysis: An Integrated Approach.
      ,
      • Marsh K.
      • Goetghebeur M.
      • Thokala P.
      • Baltussen R.
      Multi-Criteria Decision Analysis to Support Healthcare Decisions.
      The overall value of the separate methods for each critical decision point was calculated as:
      totalvalue=i=1kwixi


      where xi indicates the scoring of a method on criterium i (0 or 1), wi the average weight of criterium i for a critical decision point, k total number of criteria, and i index of summation.
      • Belton V.
      • Stewart T.J.
      Multiple Criteria Decision Analysis: An Integrated Approach.
      ,
      • Marsh K.
      • Goetghebeur M.
      • Thokala P.
      • Baltussen R.
      Multi-Criteria Decision Analysis to Support Healthcare Decisions.
      The overall value can in principle range between 0 and 100. Bootstrap sampling was used to estimate nonparametric confidence intervals for the overall values per method.
      • Efron B.
      • Tibshirani R.J.
      An Introduction to the Bootstrap.

      Sensitivity Analyses

      Separate sensitivity analyses were conducted to account for the uncertainty in the performance matrix because of (1) a lack of consensus among experts or (2) conflicting evidence from literature and experts (in these cases, final decisions were based on literature).
      • Whichello C.
      • Levitan B.
      • Juhaeri J.
      • et al.
      Appraising patient preference methods for decision-making in the medical product lifecycle: an empirical comparison.
      Conducted sensitivity analyses are listed below including a rational for each of the analyses. Analyses are grouped based on their origin (ie, either based on uncertainty in performance matrix or based on additional insights).
      • 1.
        Analysis based on uncertainties in the original methods performance matrix
        • A.
          Assigning a value of 0 to all criteria for which a value of 1 was uncertain
        • B.
          Assigning a value of 0 to all criteria for which a value of 1 was uncertain and assigning a value of 1 to all criteria for which a value of 0 was assigned with uncertainty
      • 2.
        Analysis based on insights from PREFER case studies and expert consultation within the consortium
        • C.
          Assigning low cognitive load (of method on patient) for all methods (given that recent research reported DCE not to be perceived difficult by respondents
          • Jonker M.
          • de Bekker-Grob E.
          • Veldwijk J.
          • Goossens L.
          • Bour S.
          • Rutten-Van Molken M.
          COVID-19 contact tracing apps: predicted uptake in the Netherlands based on a discrete choice experiment.
          • Jonker M.F.
          • Donkers B.
          • Goossens L.M.A.
          • et al.
          Summarizing patient preferences for the competitive landscape of multiple sclerosis treatment options.
          • Rutten-van Molken M.
          • Karimi M.
          • Leijten F.
          • et al.
          Comparing patients’ and other stakeholders’ preferences for outcomes of integrated care for multimorbidity: a discrete choice experiment in eight European countries.
          • Veldwijk J.
          • Johansson J.V.
          • Donkers B.
          • de Bekker-Grob E.W.
          Mimicking real-life decision making in health: allowing respondents time to think in a discrete choice experiment.
          • Visser L.A.
          • Huls S.P.I.
          • Uyl-de Groot C.A.
          • de Bekker-Grob E.W.
          • Redekop W.K.
          An implantable device to treat multiple sclerosis: a discrete choice experiment on patient preferences in three European countries.
          )
        • D.
          Reassigning methods criteria according to the revised performance matrix in Table 3
          Table 3Overview of respondents’ characteristics stratified by the decision maker.
          Respondents' characteristicsIndustry (n = 59)Regulators (n = 29)HTA/payer (n = 5)
          Country of residence
          Belgium00.0%417.4%240.0%
          France23.6%28.7%00.0%
          Germany1425.5%28.7%120.0%
          Italy00.0%00.0%00.0%
          The Netherlands47.3%28.7%00.0%
          Switzerland35.5%00.0%00.0%
          Sweden00.0%14.3%00.0%
          United Kingdom35.5%00.0%00.0%
          United States2443.6%00.0%120.0%
          Other
          Most reported countries under “other” were Austria, Latvia, Slovakia, Ireland, Greece, Portugal, Poland, Finland, Denmark, and Canada.
          59.1%1252.2%120.0%
          Department of employment
          Regulatory affairs712.5%
          Real-world evidence group1119.6%
          Patient (or) drug safety1730.4%
          Epidemiology or pharmacoepidemiology2442.9%
          Clinical development814.3%
          Medical affairs58.9%
          Health economics and outcomes research916.1%
          Market access47.1%
          Consultancy agency58.9%
          Other: statistics23.6%
          Experience with PPS
          Read PPS4678.0%1655.2%5100%
          Organized/designed or managed PPS2542.4%26.9%360%
          Evaluated PPS2644.1%931.0%360%
          Used PPS in work3050.8%1024.5%240%
          Attended webinars/conferences on PPS4474.6%1137.9%360%
          Other experience1016.9%00.0%120%
          Do not know what a PPS is
          Respondents who indicated not to know what a PPS is were excluded from the survey.
          23.4%413.8%00.0%
          HTA indicates health technology assessment; PPS, patient preference study.
          Most reported countries under “other” were Austria, Latvia, Slovakia, Ireland, Greece, Portugal, Poland, Finland, Denmark, and Canada.
          Respondents who indicated not to know what a PPS is were excluded from the survey.
        • E.
          Reassigning methods criteria according to the revised performance matrix in Table 3 and indicating a 1 for DCE for “establishes external validity” (given that research has shown and is currently conducted on external validity in DCE studies
          • de Bekker-Grob E.W.
          • Swait J.D.
          • Kassahun H.T.
          • et al.
          Are healthcare choices predictable? The impact of discrete choice experiment designs and models.
          • Lambooij M.S.
          • Harmsen I.A.
          • Veldwijk J.
          • et al.
          Consistency between stated and revealed preferences: a discrete choice experiment and a behavioural experiment on vaccination behaviour compared.
          • Salampessy B.H.
          • Veldwijk J.
          • Jantine Schuit A.
          • et al.
          The predictive value of discrete choice experiments in public health: an exploratory application.
          • Quaife M.
          • Terris-Prestholt F.
          • Di Tanna G.L.
          • Vickerman P.
          How well do discrete choice experiments predict health choices? A systematic review and meta-analysis of external validity.
          • de Bekker-Grob E.W.
          • Donkers B.
          • Bliemer M.C.J.
          • Veldwijk J.
          • Swait J.D.
          Can healthcare choice be predicted using stated preference data?.
          )
        • F.
          Reassigning methods criteria according to the revised performance matrix in Table 3 and indicating a 1 for BWS2 for “estimating trade-offs between treatment characteristics” (given that latent class analyses and mixed logit models can be used for the analyses, the estimates resulting from such analysis for BWS2 could be used to calculate secondary outcomes measures such as trade-offs)

      Results

      In total, 59 industry representatives, 29 regulatory representatives, and 5 HTA agency/body representatives completed the survey. Most participants were from the United States and Germany (see Table 3 for a full overview). Industry representatives had an average of 9.9 (SD = 7.3) years of experience in their current position, with a range from 1 to 30 years. Among regulators and HTA representatives, respectively, the average years of experience in their current position were 8.7 years (SD = 5.5; range 2-23 years) and 11 years (SD = 6.1; range 3-16 years). From industry, most self-identified as working in “epidemiology or pharmacoepidemiology” or “patient (or) drug safety” (see Table 3 for full overview). Respondents differed in their familiarity with PPS; although the majority had read PPS, only approximately half of the respondents had used PPS in their work (see Table 3 for full overview).

      Weighting of Methods Criteria

      Methods criteria that were appointed the largest values were reported per decision point of each of the decision makers. When a criterion had a total value of 8 or more (meaning the criterion is 50% more important than the value that would have been calculated if all criteria were equally important), the criterion was marked among the highest weighted criteria. Values and standard deviations of all criteria are listed in Table 4, with the top-weighed criteria being specifically indicated (∗). Please see Table 1 for a full definition of all criteria.
      Table 4Weight (SD) of criteria as appraised by the decision makers stratified per decision point in the MPLC.
      CriteriaIndustry decisionsRegulatory decisionHTA decision
      Select and prioritize targets and leads

      (n = 20)
      Prioritize studies

      (n = 16)
      Prioritize assets (phase 2)

      (n = 21)
      Optimize and prioritize Assets (phase 3)

      (n = 21)
      Regulatory submission and launch

      (n = 24)
      Manage product lifecycle

      (n = 20)
      Scientific opinion

      (n = 29)
      Appraisal

      (n = 5)
      A low cognitive load on patients5.1 (5.1)4.4 (4.9)3.8 (5.0)4.5 (4.9)3.8 (4.8)4.1 (4.8)5.3 (6.2)7.7 (5.2)
      Easy to add new treatment characteristics4.5 (5.5)4.1 (5.6)3.8 (5.3)4.5 (5.3)5.2 (5.1)5.1 (5.5)1.8 (3.3)4.0 (4.7)
      Eight or more treatment characteristics can be explored2.0 (4.3)2.5 (5.1)1.7 (3.6)2.9 (4.6)3.2 (5.2)3.0 (5.1)1.3 (2.6)5.9 (7.2)
      Establishes external validity5.5 (4.9)6.4 (5.0)7.6 (5.2)8.6 (4.8)
      Indicates top-weighted criteria.
      11.4 (2.5)
      Indicates top-weighted criteria.
      11.6 (2.5)
      Indicates top-weighted criteria.
      18.0 (24.1)
      Indicates top-weighted criteria.
      7.4 (5.4)
      Estimating trade-offs between treatment characteristics11.6 (25.0)
      Indicates top-weighted criteria.
      8.6 (5.0)
      Indicates top-weighted criteria.
      10.3 (4.8)
      Indicates top-weighted criteria.
      12.2 (2.4)
      Indicates top-weighted criteria.
      12.5 (2.5)
      Indicates top-weighted criteria.
      12.3 (2.3)
      Indicates top-weighted criteria.
      9.2 (5.6)
      Indicates top-weighted criteria.
      11.3 (15.7)
      Indicates top-weighted criteria.
      Estimating weights for treatment characteristics8.0 (10.1)
      Indicates top-weighted criteria.
      11.0 (12.0)
      Indicates top-weighted criteria.
      12.7 (12.2)
      Indicates top-weighted criteria.
      10.6 (6.2)
      Indicates top-weighted criteria.
      10.4 (5.0)
      Indicates top-weighted criteria.
      10.2 (4.4)
      Indicates top-weighted criteria.
      8.6 (5.3)
      Indicates top-weighted criteria.
      6.2 (7.3)
      Exploring reasons behind a preference in qualitative detail11.3 (8.3)
      Indicates top-weighted criteria.
      13.4 (6.0)
      Indicates top-weighted criteria.
      11.4 (4.3)
      Indicates top-weighted criteria.
      10.2 (5.6)
      Indicates top-weighted criteria.
      5.1 (4.7)5.8 (4.5)7.5 (4.4)14.3 (13.9)
      Indicates top-weighted criteria.
      Group dynamic with participants2.2 (4.6)1.5 (4.6)0.0 (0.0)0.0 (0.0)0.0 (0.0)0.0 (0.0)1.0 (2.0)2.6 (5.2)
      Internal validation methods can be incorporated4.5 (5.2)6.1 (5.5)7.0 (5.0)8.0 (5.0)
      Indicates top-weighted criteria.
      7.9 (5.8)8.2 (5.7)
      Indicates top-weighted criteria.
      8.6 (7.0)
      Indicates top-weighted criteria.
      8.6 (6.1)
      Indicates top-weighted criteria.
      Low complexity of instructions4.7 (4.9)4.5 (4.0)4.7 (4.3)4.1 (4.4)4.3 (4.1)4.2 (4.3)5.1 (5.8)5.2 (6.1)
      Low cost of the patient preference study8.6 (6.6)
      Indicates top-weighted criteria.
      7.1 (6.0)3.6 (5.3)3.0 (3.8)2.7 (3.9)3.0 (3.9)1.4 (3.7)0.0 (0.0)
      Low frequency of sessions (< 2)1.6 (3.5)1.5 (3.0)0.9 (2.5)0.0 (0.0)0.5 (1.9)0.5 (1.8)1.0 (2.6)0.5 (1.0)
      Public acknowledgment by your organization as an acceptable method5.3 (7.0)2.6 (4.1)5.7 (5.4)4.3 (5.1)7.2 (5.7)6.4 (5.5)2.8 (4.1)8.2 (5.8)
      Indicates top-weighted criteria.
      Quantifying heterogeneity in preferences6.1 (4.9)6.4 (5.1)8.5 (4.5)
      Indicates top-weighted criteria.
      9.2 (3.5)
      Indicates top-weighted criteria.
      9.8 (3.4)
      Indicates top-weighted criteria.
      9.7 (3.5)
      Indicates top-weighted criteria.
      10.1 (3.7)
      Indicates top-weighted criteria.
      13.5 (14.2)
      Indicates top-weighted criteria.
      Quick sessions with participants (= 30 min)1.3 (2.7)1.3 (2.7)1.2 (2.9)0.8 (2.6)0.4 (1.5)0.4 (1.5)1.5 (3.0)0.5 (1.1)
      Small sample size (= 100)6.2 (5.7)4.0 (3.8)2.9 (4.0)1.7 (3.0)1.0 (2.5)0.9 (2.5)4.0 (5.7)2.3 (4.5)
      Study duration (= 6 months)5.2 (5.9)6.6 (6.9)6.0 (5.4)8.6 (6.2)
      Indicates top-weighted criteria.
      6.0 (5.2)5.8 (5.5)3.3 (5.1)0.0 (0.0)
      The patient preference study does not include interaction among participants0.4 (1.7)0.7 (2.2)0.0 (0.0)0.0 (0.0)0.8 (2.3)0.7 (2.2)2.6 (4.0)0.0 (0.0)
      The patient preference study results allow for the calculation of risk attitudes5.9 (6.1)7.3 (6.1)8.3 (5.8)
      Indicates top-weighted criteria.
      6.8 (5.9)7.7 (4.9)7.8 (4.7)7.1 (5.1)1.8 (3.6)
      HTA indicates health technology assessment; MPLC, medical product lifecycle.
      Indicates top-weighted criteria.
      Overall, “estimating trade-offs between treatment characteristics” and “estimating weights for treatment characteristics” were important criteria throughout all decision points of the MPLC. “Exploring reasons behind preferences in qualitative detail” seemed most important in the early industry decisions and in HTA/appraisal. “External validity,” “internal validation methods can be incorporated,” and “quantifying heterogeneity in preferences” showed to be more important from clinical development phase 3 and onward to the later stages in the MPLC.

      Aggregation

      Both for BWS1 and BWS2, the total values across decision points were relatively stable implying them to be equally suitable for all decision points. There was more variability across total values of the other methods included (Table 5). Based on the valuation of the methods criteria, DCEs seemed to be most suitable during clinical development and regulatory launch. SW and PTT seemed to be most suitable throughout all industry decision points but total values were lower for regulatory and HTA decision making. When comparing the suitability of the methods across the decision points, SW and PTT were valued significantly better for all decision points than the other methods.
      Table 5Mean (95% confidence interval) score of methods across MPLC decision points.
      Decision pointsBWS1BWS2DCEPTTSW
      Industry decision points
      Select and prioritize targets and leads54.3 (43.3-61.9)49.2 (39.1-56.5)49.9 (40.6-60.5)74.3 (68.9-80.0)78.5 (74.8-82.7)
      Prioritize studies54.2 (50.1-58.3)49.8 (45.0-54.3)52.2 (41.8-63.0)71.7 (64.7-77.7)75.8 (72.2-79.1)
      Prioritize assets (phase 2)53.9 (49.3-59.2)50.1 (45.6-54.7)60.6 (52.5-68.1)74.6 (70.4-78.9)76.3 (72.8-79.9)
      Optimize and prioritize assets (phase 3)53.7 (48.6-58.5)49.2 (42.9-54.4)60.6 (54.8;65.6)74.2 (70.0-78.7)77.1 (73.5-81.1)
      Regulatory submission and launch53.8 (49.8-58.0)49.9 (45.0-54.3)63.2 (56.6-69.7)75.9 (71.4-80.3)79.2 (75.5-83.1)
      Manage product lifecycle and prioritize opportunities53.0 (49.4-57.3)48.9 (44.2-53.6)62.1 (56.1-68.4)75.3 (71.2-79.1)78.3 (74.9-81.7)
      Regulatory decision point
      Scientific opinion50.4 (40.9-57.8)45.0 (35.3-52.5)54.7 (44.5-62.2)67.2 (55.4-75.3)69.4 (57.4-77.1)
      HTA decision point
      Appraisal53.3 (40.3-66.1)45.6 (35.3-55.9)50.6 (31.2-68.4)64.6 (52.4-76.9)73.1 (66.8-79.3)
      BWS indicates best-worst scale; DCE, discrete choice experiment; HTA, health technology assessment; MPLC, medical product lifecycle; PTT, probabilistic threshold technique; SW, swing weighting.
      Dividing the values of methods based on operational versus outcome criteria, all methods tended to score lowest for HTA decision making when looking at operational criteria only. The DCE method in total scored lowest for operational criteria across decision points and in comparison with other methods. When only considering outcome criteria, BWS1 and BWS2 scored relatively lower than the DCE, SW, and PTT methods across all decision points.

      Sensitivity Analyses

      Sensitivity analyses show that the overall value of methods changed substantially from the base case (Appendix in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.019) (Fig. 1A) depending on the scoring in the performance matrix (Appendix in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.019) (Fig. 1B-G). Although the total value of BWS1 and BWS2 remained quite consistent, the total value of the DCE substantially increased in some instances whereas the total value of mainly SW was reduced in several analyses.
      Figure thumbnail gr1
      Figure 1Total value of methods across base case and other scenarios. (A) Base case according to the original performance matrix. (B) Assigning a value of 0 to all criteria for which a value of 1 was uncertain. (C) Assigning a value of 0 to all criteria for which a value of 1 was uncertain and assigning a value of 1 to all criteria for which a value of 0 was assigned with uncertainty. (D) Assigning low cognitive load to all methods. (E) Reassigning methods criteria according to the revised performance matrix in . (F) Reassigning methods criteria according to the revised performance matrix and indicating a value of 1 for DCE for “establishes external validity.” (G) Reassigning methods criteria according to the revised performance matrix and indicating a value of 1 for BWS2 for “estimating trade-offs between treatment characteristics.”
      BWS indicates best-worst scale; DCE, discrete choice experiment; HTA, health technology assessment; PTT, probabilistic threshold technique; SW, swing weighting.

      Discussion

      This study evaluated the importance of methods criteria according to decision makers at different moments along the MPLC to appraise the performance of 5 commonly used preference elicitation methods. Weights were calculated for a total of 19 methods criteria across the MPLC. The top-ranked criteria for all decision makers across all decision points included “whether a method could estimate trade-offs between treatment characteristics” and “estimate weights for treatment characteristics.” “Exploring reasons behind preferences in qualitative detail” seemed most important in the early industry decisions and in HTA/appraisal. External validity, internal validity, and the quantification of preference heterogeneity showed to be more important from clinical development phase 3 and for regulatory and HTA decision makers. Scoring the methods based on these weights across decision points of the MPLC has shown that SW and PTT had significantly higher scores across all MPLC decision points than DCE, BWS1, and BWS2. DCE scored higher for all industry decision points (except for select and prioritize targets and leads) and regulatory decision making. All methods had better scores for industry-related decision points than regulatory and HTA decisions.
      Not all methods criteria were equally important for each decision point according to the decision makers. This was in line with expectations based on a previous interview study regarding what type of information is being used at each decision point
      • Whichello C.
      • Bywall K.S.
      • Mauer J.
      • et al.
      An overview of critical decision-points in the medical product lifecycle: where to include patient preference information in the decision-making process?.
      and concerns and expectations for PPS.
      • Whichello C.
      • van Overbeeke E.
      • Janssens R.
      • et al.
      Factors and situations affecting the value of patient preference studies: semi-structured interviews in Europe and the US.
      ,
      • Janssens R.
      • Huys I.
      • van Overbeeke E.
      • et al.
      Opportunities and challenges for the inclusion of patient preferences in the medical product life cycle: a systematic review.
      ,
      • Janssens R.
      • Russo S.
      • van Overbeeke E.
      • et al.
      Patient preferences in the medical product life cycle: what do stakeholders think? Semi-structured qualitative interviews in Europe and the USA.
      Regulatory decision makers put relatively more weight on external validity of a method, and HTA decision makers put relatively more weight on the ability to explore reasons behind preferences and in qualitative detail than industry decision makers. This likely explains why all preference elicitation methods score relatively lower for HTA and regulatory decision points than industry decision points.
      Methods were appraised using a previously established performance matrix,
      • Whichello C.
      • Levitan B.
      • Juhaeri J.
      • et al.
      Appraising patient preference methods for decision-making in the medical product lifecycle: an empirical comparison.
      but also based on adapted matrices, which showed substantial differences in the overall scoring of methods. Owing to the ongoing advancements in the field of preference elicitation methods (eg, improvements in their [experimental] design, analysis), performance matrices of preference methods should continue to be updated with empirical evidence. Furthermore, there may be value in using a more detailed performance matrix that allows a less strict value function. Although the performance matrix used in the current study is based on a binary value function allowing methods to comply or not with a certain criterion, an alternative (eg, partial) value function might be more appropriate for several criteria. For instance, according to the current matrix, all methods can be used to identify preference heterogeneity. Although subgroup analysis can be conducted on the data retrieved for all methods, only some methods allow for further investigation of heterogeneity even within subgroups by means of more complex modeling strategies (ie, via mixed logit models or latent class analysis
      • Hensher D.
      • Rose J.M.
      • Greene W.H.
      Applied Choice Analysis.
      ). In addition, assessment of external validity is lacking for most methods except for DCE where recent research is showing favorable results.
      • de Bekker-Grob E.W.
      • Swait J.D.
      • Kassahun H.T.
      • et al.
      Are healthcare choices predictable? The impact of discrete choice experiment designs and models.
      • Lambooij M.S.
      • Harmsen I.A.
      • Veldwijk J.
      • et al.
      Consistency between stated and revealed preferences: a discrete choice experiment and a behavioural experiment on vaccination behaviour compared.
      • Salampessy B.H.
      • Veldwijk J.
      • Jantine Schuit A.
      • et al.
      The predictive value of discrete choice experiments in public health: an exploratory application.
      • Quaife M.
      • Terris-Prestholt F.
      • Di Tanna G.L.
      • Vickerman P.
      How well do discrete choice experiments predict health choices? A systematic review and meta-analysis of external validity.
      Although the existing empirical evidence does not fully establish external validity for DCE, it is trending in a favorable direction. The sensitivity analyses that were conducted as part of this study clearly show the impact of small changes in the performance matrix on the overall appraisal of methods.
      Although this study was conducted in an international multidisciplinary team and recruited decision makers across the MPLC to determine the weights of methods criteria for all critical decisions points, this study is subject to some limitations. First, a very limited number of HTA representatives (n = 5) responded to the survey making outcomes of the MCDA related to HTA decisions less reliable. Related to this point, owing to the applied recruitment strategy, overall response rate cannot be reported. Second, a large set of criteria was included in this study; therefore, respondents were asked to rank only their 10 criteria and in some cases axiomatic properties might have been violated.
      • Marsh K.
      • IJerman M.
      • Thokala P.
      • et al.
      Multiple criteria decision analysis for health care decision making--emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force.
      ,
      • Dodgson J.
      • Spackman M.
      • Pearman A.
      • Phillips L.
      Multi-Criteria Analysis: A Manual.
      For example, some methods criteria were showing dependence, which is not accounted for in the MCDA.
      This study showed that methods differed in their suitability across specific decision points of the MPLC. In the healthcare setting, DCEs are most applied for eliciting preferences,
      • Soekhai V.
      • de Bekker-Grob E.W.
      • Ellis A.R.
      • Vass C.M.
      Discrete choice experiments in health economics: past, present and future.
      likely in part owing to the fact that insights into the design and conduct of this method have been published.
      • Hensher D.
      • Rose J.M.
      • Greene W.H.
      Applied Choice Analysis.
      ,
      • Bridges J.F.
      • Hauber A.B.
      • Marshall D.
      • et al.
      Conjoint analysis applications in health--a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force.
      • Hauber A.B.
      • Gonzalez J.M.
      • Groothuis-Oudshoorn C.G.
      • et al.
      Statistical methods for the analysis of discrete choice experiments: a report of the ISPOR Conjoint Analysis Good Research Practices Task Force.
      • Ryan M.
      • Gerard K.
      • Amaya-Amaya M.
      Using Discrete Choice Experiments to Value Health and Health Care. The Economics of Non-Market Goods and Resources.
      Nevertheless, other methods including the PTT, SW, BWS1, BWS2, and the remaining 8 methods that were marked promising by Whichello et al
      • Whichello C.
      • Levitan B.
      • Juhaeri J.
      • et al.
      Appraising patient preference methods for decision-making in the medical product lifecycle: an empirical comparison.
      should be considered when setting up future preference studies given that also those methods comply with the top-weighted methods criteria according to decision makers. Additional research leading up to evidence-based guidance documents for designing, conducting, and analyzing such methods could enhance their use and implementation. Nevertheless, methods appraisal based on performance matrices should never be the sole determinant for method selection in a case study. Other important considerations such as the research question, requested endpoints, and operational aspects of the study should also be taken into account.

      Article and Author Information

      Author Contributions: Concept and design: Veldwijk, de Bekker-Grob, Juhaeri, Tcherny-Lessenot, Pinto, van Overbeeke, Groothuis-Oudshoorn
      Acquisition of data: Veldwijk, Groothuis-Oudshoorn
      Analysis and interpretation of data: Veldwijk, de Bekker-Grob, Juhaeri, Tcherny-Lessenot, Pinto, van Overbeeke, Groothuis-Oudshoorn, DiSantostefano
      Drafting of the manuscript: Veldwijk, Groothuis-Oudshoorn
      Critical revision of the paper for important intellectual content: Veldwijk, de Bekker-Grob, Juhaeri, Tcherny-Lessenot, Pinto, van Overbeeke, Groothuis-Oudshoorn, DiSantostefano
      Statistical analysis: Veldwijk, Groothuis-Oudshoorn
      Provision of study materials or patients: Veldwijk
      Obtaining funding: Veldwijk, de Bekker-Grob, Juhaeri, Tcherny-Lessenot, DiSantostefano
      Administrative, technical, or logistic support: Veldwijk
      Supervision: Groothuis-Oudshoorn
      Conflict of Interest Disclosures: Dr Veldwijk reported receiving grants from IMI, during the conduct of the study. Drs Juhaeri and Tcherny-Lessenot are employees of Sanofi. They receive a salary from Sanofi and own Sanofi shares. Dr Juhaeri also has an investment portfolio that from time to time includes shares of other biopharmaceutical companies. Dr Pinto is an employee of MSD, a subsidiary of Merck & Co, Inc, Kenilworth, NJ. She receives a salary and owns shares of corporate stock. Dr DiSantostefano reported receiving other from Johnson & Johnson, during the conduct of the study, and other from Johnson & Johnson, outside the submitted work. Dr van Overbeeke is an employee of NV/SA and owns Pfizer Inc shares and options based on her employment. No other disclosures were reported. This article and its contents reflect the view of the authors and not the view of PREFER, IMI, the European Union, or the European Federation of Pharmaceutical Industries and Associations.
      Funding/Support: This study formed part of the PREFER project. The PREFER project has received funding from the Innovative Medicines Initiative 2 Joint Undertaking under grant agreement no. 115966. This Joint Undertaking receives support from the European Union’s Horizon 2020 research and innovation program and the European Federation of Pharmaceutical Industries and Associations.
      Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

      Acknowledgment

      The authors thank all consortium members of PREFER who contributed to the review and dissemination of the survey or early drafts of the protocol for this study. A special thanks to Jennifer Viberg Johansson for her assistance in the design and pretesting of the initial surveys; Jürgen Kübler for his guidance in the initiation stages of this study; Bennett Levitan and Irina Cleemput for their help with defining appropriate guidance for regulators and HTA representatives in our surveys; Brett Hauber, Annalisa Rubino, and Nathalie Bere for their assistance in the recruitment stages of this study; and Francesco Pignatti for his assistance with recruitment and his valuable comments when reviewing the manuscript.

      Supplemental Material

      Supplementary data associated with this article can be found in the online version at https://doi.org/10.1016/j.jval.2022.11.019.

      References

        • de Bekker-Grob E.W.
        • Berlin C.
        • Levitan B.
        • et al.
        Giving patients’ preferences a voice in medical treatment life cycle: the PREFER public-private project.
        Patient. 2017; 10: 263-266
        • McLeod C.
        • Norman R.
        • Litton E.
        • Saville B.R.
        • Webb S.
        • Snelling T.L.
        Choosing primary endpoints for clinical trials of health care interventions.
        Contemp Clin Trials Commun. 2019; 16100486
        • Patalano F.
        • Gutzwiller F.S.
        • Shah B.
        • Kumari C.
        • Cook N.S.
        Gathering structured patient insight to drive the PRO strategy in COPD: patient-centric drug development from theory to practice.
        Adv Ther. 2020; 37: 17-26
        • van Overbeeke E.
        • Vanbinst I.
        • Jimenez-Moreno A.C.
        • Huys I.
        Patient centricity in patient preference studies: the patient perspective.
        Front Med (Lausanne). 2020; 7: 93
        • Smith M.Y.
        • van Til J.
        • DiSantostefano R.L.
        • Hauber A.B.
        • Marsh K.
        Quantitative benefit-risk assessment: state of the practice within industry.
        Ther Innov Regul Sci. 2021; 55: 415-425
        • Johnson F.R.
        • Zhou M.
        Patient preferences in regulatory benefit-risk assessments: a US perspective.
        Value Health. 2016; 19: 741-745
        • Muhlbacher A.C.
        • Juhnke C.
        • Beyer A.R.
        • Garner S.
        Patient-focused benefit-risk analysis to inform regulatory decisions: the European Union perspective.
        Value Health. 2016; 19: 734-740
        • Mott D.J.
        Incorporating quantitative patient preference data into healthcare decision making processes: is HTA falling behind?.
        Patient. 2018; 11: 249-252
      1. Patient preference information – voluntary submission, review in premarket approval applications, humanitarian device exemption applications, and de novo requests, and inclusion in decision summaries and device labeling: guidance for industry, food and drug administration staff, and other stakeholders. U.S. Food and Drug Administration.
        • Qualification opinion of IMI PREFER
        European Medicines Agency.
        • Dirksen C.D.
        The use of research evidence on patient preferences in health care decision-making: issues, controversies and moving forward.
        Expert Rev Pharmacoecon Outcomes Res. 2014; 14: 785-794
        • Huls S.P.I.
        • Whichello C.L.
        • van Exel J.
        • Uyl-de Groot C.A.
        • de Bekker-Grob E.W.
        What is next for patient preferences in health technology assessment? A systematic review of the challenges.
        Value Health. 2019; 22: 1318-1328
        • Bouvy J.C.
        • Cowie L.
        • Lovett R.
        • Morrison D.
        • Livingstone H.
        • Crabb N.
        Use of patient preference studies in HTA decision making: a NICE perspective.
        Patient. 2020; 13: 145-149
        • Cowie L.
        • Bouvy J.C.
        Measuring patient preferences: an exploratory study to determine how patient preferences data could be used in health technology assessment (HTA). MyelomaUK.
        • Whichello C.
        • Bywall K.S.
        • Mauer J.
        • et al.
        An overview of critical decision-points in the medical product lifecycle: where to include patient preference information in the decision-making process?.
        Health Policy. 2020; 124: 1325-1332
        • Soekhai V.
        • Whichello C.
        • Levitan B.
        • et al.
        Methods for exploring and eliciting patient preferences in the medical product lifecycle: a literature review.
        Drug Discov Today. 2019; 24: 1324-1331
        • Soekhai V.
        • de Bekker-Grob E.W.
        • Ellis A.R.
        • Vass C.M.
        Discrete choice experiments in health economics: past, present and future.
        Pharmacoeconomics. 2019; 37: 201-226
        • Hauber B.
        • Coulter J.
        Using the threshold technique to elicit patient preferences: an introduction to the method and an overview of existing empirical applications.
        Appl Health Econ Health Policy. 2020; 18: 31-46
        • Marsh K.
        • IJerman M.
        • Thokala P.
        • et al.
        Multiple criteria decision analysis for health care decision making--emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force.
        Value Health. 2016; 19: 125-137
        • Thokala P.
        • Devlin N.
        • Marsh K.
        • et al.
        Multiple criteria decision analysis for health care decision making--an introduction: report 1 of the ISPOR MCDA Emerging Good Practices Task Force.
        Value Health. 2016; 19: 1-13
        • Flynn T.N.
        • Louviere J.J.
        • Peters T.J.
        • Coast J.
        Best--worst scaling: what it can do for health care research and how to do it.
        J Health Econ. 2007; 26: 171-189
        • Whichello C.
        • Levitan B.
        • Juhaeri J.
        • et al.
        Appraising patient preference methods for decision-making in the medical product lifecycle: an empirical comparison.
        BMC Med Inform Decis Mak. 2020; 20: 114
        • Whichello C.
        • van Overbeeke E.
        • Janssens R.
        • et al.
        Factors and situations affecting the value of patient preference studies: semi-structured interviews in Europe and the US.
        Front Pharmacol. 2019; 10: 1009
        • van Overbeeke E.
        • Janssens R.
        • Whichello C.
        • et al.
        Design, conduct, and use of patient preference studies in the medical product life cycle: a multi-method study.
        Front Pharmacol. 2019; 10: 1395
        • van Overbeeke E.
        • Whichello C.
        • Janssens R.
        • et al.
        Factors and situations influencing the value of patient preference studies along the medical product lifecycle: a literature review.
        Drug Discov Today. 2019; 24: 57-68
        • Keeney R.L.
        • Raiffa H.
        Decisions With Multiple Objectives: Preferences and Value Trade-Offs.
        Cambridge University Press, Cambridge, England1993
        • Jonker M.
        • de Bekker-Grob E.
        • Veldwijk J.
        • Goossens L.
        • Bour S.
        • Rutten-Van Molken M.
        COVID-19 contact tracing apps: predicted uptake in the Netherlands based on a discrete choice experiment.
        JMIR MHealth UHealth. 2020; 8e20741
        • Jonker M.F.
        • Donkers B.
        • Goossens L.M.A.
        • et al.
        Summarizing patient preferences for the competitive landscape of multiple sclerosis treatment options.
        Med Decis Making. 2020; 40: 198-211
        • Rutten-van Molken M.
        • Karimi M.
        • Leijten F.
        • et al.
        Comparing patients’ and other stakeholders’ preferences for outcomes of integrated care for multimorbidity: a discrete choice experiment in eight European countries.
        BMJ Open. 2020; 10e037547
        • Veldwijk J.
        • Johansson J.V.
        • Donkers B.
        • de Bekker-Grob E.W.
        Mimicking real-life decision making in health: allowing respondents time to think in a discrete choice experiment.
        Value Health. 2020; 23: 945-952
        • Visser L.A.
        • Huls S.P.I.
        • Uyl-de Groot C.A.
        • de Bekker-Grob E.W.
        • Redekop W.K.
        An implantable device to treat multiple sclerosis: a discrete choice experiment on patient preferences in three European countries.
        J Neurol Sci. 2021; 428117587
        • Tervonen T.
        • Gelhorn H.
        • Sri Bhashyam S.
        • et al.
        MCDA swing weighting and discrete choice experiments for elicitation of patient benefit-risk preferences: a critical assessment.
        Pharmacoepidemiol Drug Saf. 2017; 26: 1483-1491
        • Belton V.
        • Stewart T.J.
        Multiple Criteria Decision Analysis: An Integrated Approach.
        Kluwer Academic Publishers, Boston, MA2002
        • Marsh K.
        • Goetghebeur M.
        • Thokala P.
        • Baltussen R.
        Multi-Criteria Decision Analysis to Support Healthcare Decisions.
        Springer, Cham, Switzerland2017
        • Efron B.
        • Tibshirani R.J.
        An Introduction to the Bootstrap.
        Chapman & Hall/CRC, New York, NY1994
        • de Bekker-Grob E.W.
        • Swait J.D.
        • Kassahun H.T.
        • et al.
        Are healthcare choices predictable? The impact of discrete choice experiment designs and models.
        Value Health. 2019; 22: 1050-1062
        • Lambooij M.S.
        • Harmsen I.A.
        • Veldwijk J.
        • et al.
        Consistency between stated and revealed preferences: a discrete choice experiment and a behavioural experiment on vaccination behaviour compared.
        BMC Med Res Methodol. 2015; 15: 19
        • Salampessy B.H.
        • Veldwijk J.
        • Jantine Schuit A.
        • et al.
        The predictive value of discrete choice experiments in public health: an exploratory application.
        Patient. 2015; 8: 521-529
        • Quaife M.
        • Terris-Prestholt F.
        • Di Tanna G.L.
        • Vickerman P.
        How well do discrete choice experiments predict health choices? A systematic review and meta-analysis of external validity.
        Eur J Health Econ. 2018; 19: 1053-1066
        • de Bekker-Grob E.W.
        • Donkers B.
        • Bliemer M.C.J.
        • Veldwijk J.
        • Swait J.D.
        Can healthcare choice be predicted using stated preference data?.
        Soc Sci Med. 2020; 246112736
        • Janssens R.
        • Huys I.
        • van Overbeeke E.
        • et al.
        Opportunities and challenges for the inclusion of patient preferences in the medical product life cycle: a systematic review.
        BMC Med Inform Decis Mak. 2019; 19: 189
        • Janssens R.
        • Russo S.
        • van Overbeeke E.
        • et al.
        Patient preferences in the medical product life cycle: what do stakeholders think? Semi-structured qualitative interviews in Europe and the USA.
        Patient. 2019; 12: 513-526
        • Hensher D.
        • Rose J.M.
        • Greene W.H.
        Applied Choice Analysis.
        2nd ed. Cambridge University Press, Cambridge, England2015
        • Dodgson J.
        • Spackman M.
        • Pearman A.
        • Phillips L.
        Multi-Criteria Analysis: A Manual.
        Department for Communities and Local Government, London, England2009
        • Bridges J.F.
        • Hauber A.B.
        • Marshall D.
        • et al.
        Conjoint analysis applications in health--a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force.
        Value Health. 2011; 14: 403-413
        • Hauber A.B.
        • Gonzalez J.M.
        • Groothuis-Oudshoorn C.G.
        • et al.
        Statistical methods for the analysis of discrete choice experiments: a report of the ISPOR Conjoint Analysis Good Research Practices Task Force.
        Value Health. 2016; 19: 300-315
        • Ryan M.
        • Gerard K.
        • Amaya-Amaya M.
        Using Discrete Choice Experiments to Value Health and Health Care. The Economics of Non-Market Goods and Resources.
        Springer, Dordrecht, The Netherlands2008