If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Correspondence: Jorien Veldwijk, PhD, Erasmus School of Health Policy & Management, Erasmus University Rotterdam, P.O. Box 1738, 3000 DR Rotterdam, The Netherlands.
Erasmus School of Health Policy & Management, Erasmus University Rotterdam, Rotterdam, The NetherlandsJulius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
Given that there is no gold standard method for eliciting preferences, the selection of a patient preference method should be based on decision makers’ needs and performance of methods under consideration to meet those needs. Five commonly used preference elicitation methods were compared: discrete choice experiments, swing weighting, probabilistic threshold technique, and best-worst scale cases 1 and 2.
•
Whether a method estimates relative importance and trade-offs between treatment characteristics was considered important to decision makers across all medical product lifecycle (MPLC) decision points. Generally, obtaining qualitative information was more important during early lifecycle decisions, whereas internal and external validity and identifying preference heterogeneity were more important during later decision points. Methods performed relatively well on the criteria that were most important at the different decision points of the MPLC; nevertheless, the value of individual methods was sensitive to changes in the performance matrix.
•
Although discrete choice experiment is the most applied preference elicitation method, best-worst scale, swing weighting, and probabilistic threshold technique should also be considered to address the needs of decision makers throughout the MPLC as they comply with top-weighted methods criteria according to MPLC decision makers.
Abstract
Objectives
This study aimed to understand the importance of criteria describing methods (eg, duration, costs, validity, and outcomes) according to decision makers for each decision point in the medical product lifecycle (MPLC) and to determine the suitability of a discrete choice experiment, swing weighting, probabilistic threshold technique, and best-worst scale cases 1 and 2 at each decision point in the MPLC.
Methods
Applying multicriteria decision analysis, an online survey was sent to MPLC decision makers (ie, industry, regulatory, and health technology assessment representatives). They ranked and weighted 19 methods criteria from an existing performance matrix about their respective decisions across the MPLC. All criteria were given a relative weight based on the ranking and rating in the survey after which an overall suitability score was calculated for each preference elicitation method per decision point. Sensitivity analyses were conducted to reflect uncertainty in the performance matrix.
Results
Fifty-nine industry, 29 regulatory, and 5 health technology assessment representatives completed the surveys. Overall, “estimating trade-offs between treatment characteristics” and “estimating weights for treatment characteristics” were highly important criteria throughout all MPLC decision points, whereas other criteria were most important only for specific MPLC stages. Swing weighting and probabilistic threshold technique received significantly higher suitability scores across decision points than other methods. Sensitivity analyses showed substantial impact of uncertainty in the performance matrix.
Conclusion
Although discrete choice experiment is the most applied preference elicitation method, other methods should also be considered to address the needs of decision makers. Development of evidence-based guidance documents for designing, conducting, and analyzing such methods could enhance their use.
Increasingly decision makers look for ways to measure patients’ preferences and include such information in decision making along the medical product lifecycle (MPLC).
Including preference information might be apparent for some decisions such as for identifying unmet medical needs and selecting endpoints for randomized controlled trial
Nevertheless, the exact role of patient preferences in other industry decision points, especially regulatory and health technology assessment (HTA)/reimbursement-related decisions, is less clear. Both regulatory and HTA agencies advocate for increased transparency in their decision-making process and a focus on patient-centered decision making.
Patient preference information – voluntary submission, review in premarket approval applications, humanitarian device exemption applications, and de novo requests, and inclusion in decision summaries and device labeling: guidance for industry, food and drug administration staff, and other stakeholders. U.S. Food and Drug Administration.
and the European Medicines Agency recently provided a positive qualification opinion and asked for public consultation on a preference elicitation framework.
Nevertheless, initiatives are undertaken; for instance, the National Institute for Health and Care Excellence published their perspective on the use of PPS in HTA
Measuring patient preferences: an exploratory study to determine how patient preferences data could be used in health technology assessment (HTA). MyelomaUK.
Nevertheless, the weighting or incorporation of preferences against the standard information (eg, clinical data, cost-effectiveness data) in decision making along the MPLC remains debated. According to previous research, the MPLC, in total, consists of approximately 15 decision points for different decision makers: pharmaceutical industry, regulators, and HTA agency/body.
An overview of critical decision-points in the medical product lifecycle: where to include patient preference information in the decision-making process?.
An overview of critical decision-points in the medical product lifecycle: where to include patient preference information in the decision-making process?.
An overview of critical decision-points in the medical product lifecycle: where to include patient preference information in the decision-making process?.
making it complicated to select one or few suitable preference elicitation methods that fit the needs of all decision makers across all decision points of the MPLC.
A recent literature review identified 22 preference elicitation methods
grouped into ranking, rating, indifference methods, and discrete choice methods. Within each category, most commonly used methods in healthcare to elicit preferences of patients were discrete choice experiment (DCE),
Multiple criteria decision analysis for health care decision making--emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force.
combined a Q-method and analytical hierarchy process to appraise all 22 preference elicitation methods. The relative weight of criteria describing methods (eg, duration, costs, validity, and outcomes) was evaluated for 4 hypothetical MPLC scenarios: 2 variations of early clinical development stages, 1 late phase III scenario, and 1 postmarketing scenario.
Weighting of criteria for methods appraisal was mostly based on response of representatives from industry and academia (1 HTA and 1 regulator responded).
To provide insight into suitability of methods across the full MPLC and thereby facilitate methods selection and systematic implementation of preference elicitation along the MPLC, it remains crucial to understand the importance of methods criteria for each decision maker (ie, industry, regulator, HTA agency/body) at their respective critical decision points in the MPLC.
Therefore, this study aimed to evaluate the importance of methods criteria to fully appraise the performance of 5 commonly used preference elicitation methods against these methods criteria according to decision makers at different moments along the MPLC.
Methods
We used multicriteria decision analysis (MCDA) in this study, which is a methodology for appraising alternatives on individual, often conflicting criteria, and combining them into 1 overall appraisal.
Common steps in MCDA are (1) defining the decision problem (including decision makers), (2) selecting criteria, (3) measuring performance, (4) weighting of criteria, (5) aggregating results, (6) sensitivity analyses, and (7) interpretation of results.
Step 2 to 6 will be detailed below given that step 1 was outlined in the introduction (ie, to provide insight into suitability of methods across the full MPLC and thereby facilitate methods selection and systematic implementation of preference elicitation along the MPLC) and step 7 will be covered in the Results and Discussion sections of this article.
initially identified 35 method criteria as being most important for selecting a qualitative or quantitative patient preference method based on literature reviews and previous studies. These were subsequently restricted to 19 criteria (12 operational and 7 outcomes related criteria) by means of a Q-method experiment among stakeholders (N = 54 being academic, representative from industry or regulatory/HTA agency, physician, patient (representative), or consultant) (Table 1
Comparing patients’ and other stakeholders’ preferences for outcomes of integrated care for multimorbidity: a discrete choice experiment in eight European countries.
) specifying the performance of each method for each criterion was created based on semistructured interviews with preference method experts (N = 17) and a literature review. Further details on the method and development of the performance matrix can be found elsewhere.
The patient preference study can be conducted at a relatively low cost.
Quick sessions with participants (≤ 30 min)
Completing the patient preference study requires less than 30 min of the patient.
Low frequency of sessions (< 2)
The number of interactions required with each respondent over the course of the entire data collection period is less than 2.
Study duration (≤ 6 months)
The time needed for preparing the study, collecting data, and conducting analysis is less than 6 months.
8 or more treatment characteristics can be explored
Measuring the preferences of patients for 8 or more treatment characteristics.
Small sample size (≤ 100)
The patient preference study can accurately be conducted in a sample of less than 100 patients.
A low cognitive load on patients
It is important that participating in the patient preference study does not require a low cognitive load on the patients. The preference study could easily be completed by populations who experience heavy cognitive loads or struggle with cognitive tasks.
Low complexity of instructions
The instructions that patients need to read about or listen to before being able to participate in the patient preference study are low in complexity.
Public acknowledgment by your organization as an acceptable method
Your organization or stakeholder group recognizes the method that is used to measure patient preferences as an acceptable method.
Easy to add new treatment characteristics
It is easy to add new treatment characteristics to the patient preference study while it is conducted without rendering all the previous data collected meaningless.
The patient preference study does not include interaction among participants
Patient can complete the preference study on their own and do not need to interact with other patients.
Group dynamic with participants
Patients interact with each other during in their participation in the preference study.
Outcome-related criteria
The patient preference study results allow for the calculation of risk attitudes
Whether the analysis of the results of the patient preference study can be used to calculate risk attitudes, such as risk tolerance vs risk aversion
Exploring reasons behind a preference in qualitative detail
Qualitative methods or mixed methods can often be used to find out the “why” behind a particular preference or choice or why a participant has made a trade-off or selected a characteristic of a health intervention as being more important than another.
Estimating weights for treatment characteristics
Whether a patient preference study can estimate weights and thereby tell you how much each treatment characteristic matters to patients
Estimating trade-offs between treatment characteristics
Whether the results of the patient preference study can be used to calculate for instance maximum acceptable risk. This type of information tells you both how much each treatment characteristic matters and how much of one characteristic patients are willing to lose to gain on another characteristic.
Quantifying heterogeneity in preferences
Whether or not the results of the patient preference study allow for the estimation of preference heterogeneity. Some preference heterogeneity may be explained by differences in observable patient characteristics.
Internal validation methods can be incorporated
Whether or not the method used to measure patients’ preferences allows for the inclusion of internal validation measures. Validity means the extent to which a test or study measures what it claims to measure. Internal validity refers to whether a finding that incorporates a causal relationship between 2 or more variables is sound.
Establishes external validity
Whether or not the method used in the patient preference study is proven to be externally valid. External validity refers to whether the results of a study can be generalized beyond the specific research context in which the study was conducted.
Low costs for DCE as the qualitative work across methods is equally much and specialized software and expertise for DCE is no longer a necessity (free packages such as R offer experimental design and advanced statistical modeling options).
Calculation of risk attitudes not possible in BWS and SW given that attributes are not actively traded against each other such as in DCE and PTT where people can focus on avoiding (all) risks.
1
Exploring reasons behind a preference in qualitative detail
0
0
0
0
0
Estimating weights for treatment characteristics
1
1
1
1
1
Estimating trade-offs between treatment characteristics
1
0
0
0
Trade-offs between treatment characteristics are not common practice in SW; it can theoretically be done but only with (too) many assumptions.
1
Quantifying heterogeneity in preferences
1
1
1
1
1
Internal validation methods can be incorporated
1
1
1
1
1
Establishes external validity
0
0
0
0
0
Note. 1 indicates the method complies to the criteria and 0 means it does not. Further indications for changes compared with the original performance matrix and explanations of reasoning behind all changes have been indicated using symbols ∗ and ‡ to ∗∗.
∗ Low costs for DCE as the qualitative work across methods is equally much and specialized software and expertise for DCE is no longer a necessity (free packages such as R offer experimental design and advanced statistical modeling options).
† Indicates uncertainty in whether the method does or does not comply with a criterium as specified by Whichello and colleagues.
Comparing patients’ and other stakeholders’ preferences for outcomes of integrated care for multimorbidity: a discrete choice experiment in eight European countries.
¶ Group dynamic for SW unrealistic as also in lab conducted experiment you get individual outcomes.
£ Calculation of risk attitudes not possible in BWS and SW given that attributes are not actively traded against each other such as in DCE and PTT where people can focus on avoiding (all) risks.
∗∗ Trade-offs between treatment characteristics are not common practice in SW; it can theoretically be done but only with (too) many assumptions.
Three surveys were developed to assess the relative importance, that is, the weights, of the methods criteria for each of the critical decision points in the MPLC in which patient preference information could be considered in addition to the current evidence used for decision making. Furthermore, the surveys were tailored to decision processes of the 3 decision-maker groups in the MPLC in such a way that surveys for industry representative included 6 industry-related decision points (ie, select and prioritize targets and leads, prioritize studies, prioritize assets, optimize and prioritize assets, regulatory submission and launch, manage MPLC, and prioritize opportunities), the survey for regulators contained 1 regulatory decision point (scientific opinion), and the survey for HTA agency/body representative included 1 HTA decision point (appraisal).
Respondents (recruitment strategies are described below) were invited to participate by sharing the link for the survey and an explanatory letter. The survey started with an explanation of what a PPS constitutes, and after that, 4 background questions were included to get insight in the respondents and their experience with such preference studies. In the next part of the survey, the respondents were asked to rank the method criteria included in the performance matrix (Table 2
Comparing patients’ and other stakeholders’ preferences for outcomes of integrated care for multimorbidity: a discrete choice experiment in eight European countries.
) from most to least important for each of the decision points that related to their specific decisional framework (for instance, HTA representatives were only asked about the importance of the methods criteria related to appraisal). To avoid ordering bias, the order in which the criteria were presented to respondents was randomized. For the criteria that a respondent ranked in their top 10, the respondents were asked to rate (on a scale of 100) the criterion compared with their top-ranked criterion, the score of which was set to 100. They were specifically not asked to weight all 19 criteria owing to the high cognitive burden of such a task resulting in fatigue and further potential bias induced by such a request.
Multiple criteria decision analysis for health care decision making--emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force.
The surveys were constructed by the research team and reviewed by different decision makers (ie, 5 industry representatives, 2 Food and Drug Administration representatives, and 1 Belgium HTA representative [also representing the EUnetHTA]). Thereafter, the surveys were pretested by means of 3 think-aloud interviews (using conveyance sampling) to refine language, relevance, and usability of the survey. After the pretest, changes were made to the surveys related to the explanation of the decision points, what constitutes a preference study and the content/meaning of the criteria. The surveys were developed in Lighthouse Studio 9.7.0. Industry representatives within the Patient Preferences in Benefit-Risk Assessments During the Drug Life Cycle (PREFER) consortium and the Benefit-Risk Assessment, Communication, and Evaluation special interest group were asked to invite industry representatives to complete the survey. When disseminating the survey, it was requested to forward the invitation to colleagues at different departments (eg, regulatory-policy, drug safety, epidemiology, clinical development, health outcomes research, value and access groups). Regulatory representatives were contacted via the European Medicines Agency. HTA agency/body representatives were contacted via the head of the PREFER HTA advisory board. In total, 20 to 40 respondents per group of decision makers were anticipated to result in sufficient data to arrive at meaningful conclusions.
Based on the ranking position and the points from the rating exercise, all the criteria were given a relative average weight wi for each decision point.
For each respondent, the criteria with an individual ranking outside the top 10 were given a weight of 0. The weights of the other criteria were calculated by scaling the ratings with the total sum of the points such that the sum of all weights equaled to 100. Next, the individual weights were averaged over all respondents giving the average weight wi for each criterion. Subsequently, for each preference elicitation method, an overall value was calculated per critical decision point along the MPLC based on whether the methods met certain criteria (the scoring, see performance matrix in Table 2
Comparing patients’ and other stakeholders’ preferences for outcomes of integrated care for multimorbidity: a discrete choice experiment in eight European countries.
The overall value of the separate methods for each critical decision point was calculated as:
where xi indicates the scoring of a method on criterium i (0 or 1), wi the average weight of criterium i for a critical decision point, k total number of criteria, and i index of summation.
The overall value can in principle range between 0 and 100. Bootstrap sampling was used to estimate nonparametric confidence intervals for the overall values per method.
Separate sensitivity analyses were conducted to account for the uncertainty in the performance matrix because of (1) a lack of consensus among experts or (2) conflicting evidence from literature and experts (in these cases, final decisions were based on literature).
Conducted sensitivity analyses are listed below including a rational for each of the analyses. Analyses are grouped based on their origin (ie, either based on uncertainty in performance matrix or based on additional insights).
1.
Analysis based on uncertainties in the original methods performance matrix
A.
Assigning a value of 0 to all criteria for which a value of 1 was uncertain
B.
Assigning a value of 0 to all criteria for which a value of 1 was uncertain and assigning a value of 1 to all criteria for which a value of 0 was assigned with uncertainty
2.
Analysis based on insights from PREFER case studies and expert consultation within the consortium
C.
Assigning low cognitive load (of method on patient) for all methods (given that recent research reported DCE not to be perceived difficult by respondents
Comparing patients’ and other stakeholders’ preferences for outcomes of integrated care for multimorbidity: a discrete choice experiment in eight European countries.
Reassigning methods criteria according to the revised performance matrix in Table 3 and indicating a 1 for DCE for “establishes external validity” (given that research has shown and is currently conducted on external validity in DCE studies
Reassigning methods criteria according to the revised performance matrix in Table 3 and indicating a 1 for BWS2 for “estimating trade-offs between treatment characteristics” (given that latent class analyses and mixed logit models can be used for the analyses, the estimates resulting from such analysis for BWS2 could be used to calculate secondary outcomes measures such as trade-offs)
Results
In total, 59 industry representatives, 29 regulatory representatives, and 5 HTA agency/body representatives completed the survey. Most participants were from the United States and Germany (see Table 3 for a full overview). Industry representatives had an average of 9.9 (SD = 7.3) years of experience in their current position, with a range from 1 to 30 years. Among regulators and HTA representatives, respectively, the average years of experience in their current position were 8.7 years (SD = 5.5; range 2-23 years) and 11 years (SD = 6.1; range 3-16 years). From industry, most self-identified as working in “epidemiology or pharmacoepidemiology” or “patient (or) drug safety” (see Table 3 for full overview). Respondents differed in their familiarity with PPS; although the majority had read PPS, only approximately half of the respondents had used PPS in their work (see Table 3 for full overview).
Weighting of Methods Criteria
Methods criteria that were appointed the largest values were reported per decision point of each of the decision makers. When a criterion had a total value of 8 or more (meaning the criterion is 50% more important than the value that would have been calculated if all criteria were equally important), the criterion was marked among the highest weighted criteria. Values and standard deviations of all criteria are listed in Table 4, with the top-weighed criteria being specifically indicated (∗). Please see Table 1 for a full definition of all criteria.
Table 4Weight (SD) of criteria as appraised by the decision makers stratified per decision point in the MPLC.
Criteria
Industry decisions
Regulatory decision
HTA decision
Select and prioritize targets and leads (n = 20)
Prioritize studies (n = 16)
Prioritize assets (phase 2) (n = 21)
Optimize and prioritize Assets (phase 3) (n = 21)
Regulatory submission and launch (n = 24)
Manage product lifecycle (n = 20)
Scientific opinion (n = 29)
Appraisal (n = 5)
A low cognitive load on patients
5.1 (5.1)
4.4 (4.9)
3.8 (5.0)
4.5 (4.9)
3.8 (4.8)
4.1 (4.8)
5.3 (6.2)
7.7 (5.2)
Easy to add new treatment characteristics
4.5 (5.5)
4.1 (5.6)
3.8 (5.3)
4.5 (5.3)
5.2 (5.1)
5.1 (5.5)
1.8 (3.3)
4.0 (4.7)
Eight or more treatment characteristics can be explored
Overall, “estimating trade-offs between treatment characteristics” and “estimating weights for treatment characteristics” were important criteria throughout all decision points of the MPLC. “Exploring reasons behind preferences in qualitative detail” seemed most important in the early industry decisions and in HTA/appraisal. “External validity,” “internal validation methods can be incorporated,” and “quantifying heterogeneity in preferences” showed to be more important from clinical development phase 3 and onward to the later stages in the MPLC.
Aggregation
Both for BWS1 and BWS2, the total values across decision points were relatively stable implying them to be equally suitable for all decision points. There was more variability across total values of the other methods included (Table 5). Based on the valuation of the methods criteria, DCEs seemed to be most suitable during clinical development and regulatory launch. SW and PTT seemed to be most suitable throughout all industry decision points but total values were lower for regulatory and HTA decision making. When comparing the suitability of the methods across the decision points, SW and PTT were valued significantly better for all decision points than the other methods.
Table 5Mean (95% confidence interval) score of methods across MPLC decision points.
Decision points
BWS1
BWS2
DCE
PTT
SW
Industry decision points
Select and prioritize targets and leads
54.3 (43.3-61.9)
49.2 (39.1-56.5)
49.9 (40.6-60.5)
74.3 (68.9-80.0)
78.5 (74.8-82.7)
Prioritize studies
54.2 (50.1-58.3)
49.8 (45.0-54.3)
52.2 (41.8-63.0)
71.7 (64.7-77.7)
75.8 (72.2-79.1)
Prioritize assets (phase 2)
53.9 (49.3-59.2)
50.1 (45.6-54.7)
60.6 (52.5-68.1)
74.6 (70.4-78.9)
76.3 (72.8-79.9)
Optimize and prioritize assets (phase 3)
53.7 (48.6-58.5)
49.2 (42.9-54.4)
60.6 (54.8;65.6)
74.2 (70.0-78.7)
77.1 (73.5-81.1)
Regulatory submission and launch
53.8 (49.8-58.0)
49.9 (45.0-54.3)
63.2 (56.6-69.7)
75.9 (71.4-80.3)
79.2 (75.5-83.1)
Manage product lifecycle and prioritize opportunities
Dividing the values of methods based on operational versus outcome criteria, all methods tended to score lowest for HTA decision making when looking at operational criteria only. The DCE method in total scored lowest for operational criteria across decision points and in comparison with other methods. When only considering outcome criteria, BWS1 and BWS2 scored relatively lower than the DCE, SW, and PTT methods across all decision points.
Sensitivity Analyses
Sensitivity analyses show that the overall value of methods changed substantially from the base case (Appendix in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.019) (Fig. 1A) depending on the scoring in the performance matrix (Appendix in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.11.019) (Fig. 1B-G). Although the total value of BWS1 and BWS2 remained quite consistent, the total value of the DCE substantially increased in some instances whereas the total value of mainly SW was reduced in several analyses.
Figure 1Total value of methods across base case and other scenarios. (A) Base case according to the original performance matrix. (B) Assigning a value of 0 to all criteria for which a value of 1 was uncertain. (C) Assigning a value of 0 to all criteria for which a value of 1 was uncertain and assigning a value of 1 to all criteria for which a value of 0 was assigned with uncertainty. (D) Assigning low cognitive load to all methods. (E) Reassigning methods criteria according to the revised performance matrix in Table 3. (F) Reassigning methods criteria according to the revised performance matrix and indicating a value of 1 for DCE for “establishes external validity.” (G) Reassigning methods criteria according to the revised performance matrix and indicating a value of 1 for BWS2 for “estimating trade-offs between treatment characteristics.”
This study evaluated the importance of methods criteria according to decision makers at different moments along the MPLC to appraise the performance of 5 commonly used preference elicitation methods. Weights were calculated for a total of 19 methods criteria across the MPLC. The top-ranked criteria for all decision makers across all decision points included “whether a method could estimate trade-offs between treatment characteristics” and “estimate weights for treatment characteristics.” “Exploring reasons behind preferences in qualitative detail” seemed most important in the early industry decisions and in HTA/appraisal. External validity, internal validity, and the quantification of preference heterogeneity showed to be more important from clinical development phase 3 and for regulatory and HTA decision makers. Scoring the methods based on these weights across decision points of the MPLC has shown that SW and PTT had significantly higher scores across all MPLC decision points than DCE, BWS1, and BWS2. DCE scored higher for all industry decision points (except for select and prioritize targets and leads) and regulatory decision making. All methods had better scores for industry-related decision points than regulatory and HTA decisions.
Not all methods criteria were equally important for each decision point according to the decision makers. This was in line with expectations based on a previous interview study regarding what type of information is being used at each decision point
An overview of critical decision-points in the medical product lifecycle: where to include patient preference information in the decision-making process?.
Regulatory decision makers put relatively more weight on external validity of a method, and HTA decision makers put relatively more weight on the ability to explore reasons behind preferences and in qualitative detail than industry decision makers. This likely explains why all preference elicitation methods score relatively lower for HTA and regulatory decision points than industry decision points.
Methods were appraised using a previously established performance matrix,
but also based on adapted matrices, which showed substantial differences in the overall scoring of methods. Owing to the ongoing advancements in the field of preference elicitation methods (eg, improvements in their [experimental] design, analysis), performance matrices of preference methods should continue to be updated with empirical evidence. Furthermore, there may be value in using a more detailed performance matrix that allows a less strict value function. Although the performance matrix used in the current study is based on a binary value function allowing methods to comply or not with a certain criterion, an alternative (eg, partial) value function might be more appropriate for several criteria. For instance, according to the current matrix, all methods can be used to identify preference heterogeneity. Although subgroup analysis can be conducted on the data retrieved for all methods, only some methods allow for further investigation of heterogeneity even within subgroups by means of more complex modeling strategies (ie, via mixed logit models or latent class analysis
Although the existing empirical evidence does not fully establish external validity for DCE, it is trending in a favorable direction. The sensitivity analyses that were conducted as part of this study clearly show the impact of small changes in the performance matrix on the overall appraisal of methods.
Although this study was conducted in an international multidisciplinary team and recruited decision makers across the MPLC to determine the weights of methods criteria for all critical decisions points, this study is subject to some limitations. First, a very limited number of HTA representatives (n = 5) responded to the survey making outcomes of the MCDA related to HTA decisions less reliable. Related to this point, owing to the applied recruitment strategy, overall response rate cannot be reported. Second, a large set of criteria was included in this study; therefore, respondents were asked to rank only their 10 criteria and in some cases axiomatic properties might have been violated.
Multiple criteria decision analysis for health care decision making--emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force.
For example, some methods criteria were showing dependence, which is not accounted for in the MCDA.
This study showed that methods differed in their suitability across specific decision points of the MPLC. In the healthcare setting, DCEs are most applied for eliciting preferences,
should be considered when setting up future preference studies given that also those methods comply with the top-weighted methods criteria according to decision makers. Additional research leading up to evidence-based guidance documents for designing, conducting, and analyzing such methods could enhance their use and implementation. Nevertheless, methods appraisal based on performance matrices should never be the sole determinant for method selection in a case study. Other important considerations such as the research question, requested endpoints, and operational aspects of the study should also be taken into account.
Article and Author Information
Author Contributions:Concept and design: Veldwijk, de Bekker-Grob, Juhaeri, Tcherny-Lessenot, Pinto, van Overbeeke, Groothuis-Oudshoorn
Acquisition of data: Veldwijk, Groothuis-Oudshoorn
Analysis and interpretation of data: Veldwijk, de Bekker-Grob, Juhaeri, Tcherny-Lessenot, Pinto, van Overbeeke, Groothuis-Oudshoorn, DiSantostefano
Drafting of the manuscript: Veldwijk, Groothuis-Oudshoorn
Critical revision of the paper for important intellectual content: Veldwijk, de Bekker-Grob, Juhaeri, Tcherny-Lessenot, Pinto, van Overbeeke, Groothuis-Oudshoorn, DiSantostefano
Provision of study materials or patients: Veldwijk
Obtaining funding: Veldwijk, de Bekker-Grob, Juhaeri, Tcherny-Lessenot, DiSantostefano
Administrative, technical, or logistic support: Veldwijk
Supervision: Groothuis-Oudshoorn
Conflict of Interest Disclosures: Dr Veldwijk reported receiving grants from IMI, during the conduct of the study. Drs Juhaeri and Tcherny-Lessenot are employees of Sanofi. They receive a salary from Sanofi and own Sanofi shares. Dr Juhaeri also has an investment portfolio that from time to time includes shares of other biopharmaceutical companies. Dr Pinto is an employee of MSD, a subsidiary of Merck & Co, Inc, Kenilworth, NJ. She receives a salary and owns shares of corporate stock. Dr DiSantostefano reported receiving other from Johnson & Johnson, during the conduct of the study, and other from Johnson & Johnson, outside the submitted work. Dr van Overbeeke is an employee of NV/SA and owns Pfizer Inc shares and options based on her employment. No other disclosures were reported. This article and its contents reflect the view of the authors and not the view of PREFER, IMI, the European Union, or the European Federation of Pharmaceutical Industries and Associations.
Funding/Support: This study formed part of the PREFER project. The PREFER project has received funding from the Innovative Medicines Initiative 2 Joint Undertaking under grant agreement no. 115966. This Joint Undertaking receives support from the European Union’s Horizon 2020 research and innovation program and the European Federation of Pharmaceutical Industries and Associations.
Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Acknowledgment
The authors thank all consortium members of PREFER who contributed to the review and dissemination of the survey or early drafts of the protocol for this study. A special thanks to Jennifer Viberg Johansson for her assistance in the design and pretesting of the initial surveys; Jürgen Kübler for his guidance in the initiation stages of this study; Bennett Levitan and Irina Cleemput for their help with defining appropriate guidance for regulators and HTA representatives in our surveys; Brett Hauber, Annalisa Rubino, and Nathalie Bere for their assistance in the recruitment stages of this study; and Francesco Pignatti for his assistance with recruitment and his valuable comments when reviewing the manuscript.
Patient preference information – voluntary submission, review in premarket approval applications, humanitarian device exemption applications, and de novo requests, and inclusion in decision summaries and device labeling: guidance for industry, food and drug administration staff, and other stakeholders. U.S. Food and Drug Administration.
Measuring patient preferences: an exploratory study to determine how patient preferences data could be used in health technology assessment (HTA). MyelomaUK.
An overview of critical decision-points in the medical product lifecycle: where to include patient preference information in the decision-making process?.
Multiple criteria decision analysis for health care decision making--emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force.
Comparing patients’ and other stakeholders’ preferences for outcomes of integrated care for multimorbidity: a discrete choice experiment in eight European countries.
04751-9&id=gr1.jpg){kind=link}