Advertisement

Reporting Quality of Marginal Rates of Substitution in Discrete Choice Experiments That Elicit Patient Preferences

Open ArchivePublished:July 15, 2020DOI:https://doi.org/10.1016/j.jval.2020.04.1831

      Highlights

      • Discrete choice experiments (DCEs) that elicit patient preferences are increasingly being conducted, and their role in decision making is increasing. One of the key outputs of a DCE is the marginal rate of substitution (MRS), but it has been suggested in the literature that reporting of uncertainty around MRSs is inadequate.
      • In this study, a systematic literature review was conducted to determine the frequency with which DCE studies that elicit patient preferences reported MRS without information on the degree of uncertainty. An analysis was also conducted to determine how standard errors (SEs) of MRSs calculated using commonly reported information compare with the actual SEs.
      • The degree of uncertainty around MRS estimates was not reported in over one-third of patient preference DCE studies that reported MRS. The analysis indicated that estimates of SEs calculated using commonly reported information are often significantly biased when covariances (typically not reported) are not included in calculations. Thus, it is crucial that researchers calculate and report SEs of MRSs.

      Abstract

      Background

      Discrete choice experiments (DCEs) are commonly used to elicit patient preferences as marginal rates of substitution (MRSs) between treatment or health service attributes. Because these studies are increasing in importance, it is vital that uncertainty around MRS estimates is reported.

      Objective

      To review recently published DCE studies that elicit patient preferences in relation to MRS reporting and to explore the accuracy of using other reported information to estimate the uncertainty of the MRSs.

      Methods

      A systematic literature review of DCEs conducted with patients between 2014 and July 2019 was performed. The number of studies reporting coefficients, MRSs, standard errors (SEs), and confidence intervals was recorded. If all information was reported, studies were included in an analysis to determine the impact of estimating the SEs of MRSs using coefficients and assuming zero covariance, to determine the impact of this assumption.

      Results

      Two hundred and thirty-two patient DCEs were identified in the review; 34.1% (n = 79) reported 1 or more MRS and, of these, only 62.0% (n = 49) provided an estimate of the uncertainty. Of these studies, 16 contained enough information for inclusion in the analysis, providing 116 datapoints. Actual SEs were smaller than estimated SEs in 75.0% of cases (n = 87), and estimated SEs were within 25% of the actual SE in 59.5% of cases (n = 69).

      Conclusion

      Uncertainty of MRS estimates is unreported in a substantial proportion of recently published DCE studies. Estimating the SE of a MRS by solely using the SEs of the utility coefficients is likely to lead to biased estimates of the precision of patient trade-offs.

      Keywords

      Introduction

      The role of quantitative patient preference information in regulatory and reimbursement decision making has increased in recent years.
      • Johnson F.R.
      • Beusterien K.
      • Özdemir S.
      • Wilson L.
      Giving patients a meaningful voice in United States regulatory decision making: the role for health preference research.
      • Mühlbacher A.C.
      • Johnson F.R.
      Giving patients a meaningful voice in European health technology assessments: the role of health preference research.
      • Mott D.J.
      Incorporating quantitative patient preference data into healthcare decision making processes: is HTA falling behind?.
      • Janssens R.
      • Huys I.
      • van Overbeeke E.
      • et al.
      Opportunities and challenges for the inclusion of patient preferences in the medical product life cycle: a systematic review.
      • van Overbeeke E.
      • Whichello C.
      • Janssens R.
      • et al.
      Factors and situations influencing the value of patient preference studies along the medical product lifecycle: a literature review.
      Numerous methodologies exist to elicit patient preferences,
      • Tervonen T.
      • Gelhorn H.
      • Sri Bhashyam S.
      • et al.
      MCDA swing weighting and discrete choice experiments for elicitation of patient benefit-risk preferences: a critical assessment.
      • Hauber B.
      • Coulter J.
      Using the threshold technique to elicit patient preferences: an introduction to the method and an overview of existing empirical applications.
      • Schmidt K.
      • Babac A.
      • Pauer F.
      • Damm K.
      • von der Schulenburg J.-M.
      Measuring patients’ priorities using the Analytic Hierarchy Process in comparison with Best-Worst-Scaling and rating cards: methodological aspects and ranking tasks.
      • Krucien N.
      • Sicsic J.
      • Ryan M.
      For better or worse? Investigating the validity of best–worst discrete choice experiments in health.
      but by far the most commonly applied quantitative method in practice is the discrete choice experiment (DCE).
      • Soekhai V.
      • Whichello C.
      • Levitan B.
      • et al.
      Methods for exploring and eliciting patient preferences in the medical product lifecycle: a literature review.
      DCE is a stated preference methodology that requires individuals to make hypothetical choices in a series of scenarios.
      • Louviere J.J.
      • Lancsar E.
      Choice experiments in health: the good, the bad, the ugly and toward a brighter future.
      In each scenario, the alternatives are described using a set of attributes with accompanying levels that are varied in the choice scenarios. In health, it is common that the scenarios consist of 2 alternatives representing different treatments, and the attributes and levels describe the treatment characteristics (eg, the associated benefits, risks, and delivery mode).
      • Clark M.D.
      • Determann D.
      • Petrou S.
      • Moro D.
      • de Bekker-Grob E.W.
      Discrete choice experiments in health economics: a review of the literature.
      • de Bekker-Grob E.W.
      • Ryan M.
      • Gerard K.
      Discrete choice experiments in health economics: a review of the literature.
      • Ryan M.
      • Gerard K.
      Using discrete choice experiments to value health care programmes: current practice and future research reflections.
      • Soekhai V.
      • de Bekker-Grob E.W.
      • Ellis A.R.
      • Vass C.M.
      Discrete choice experiments in health economics: past, present and future.
      The output of a DCE provides insightful information on the relative importance of the different attributes.
      • Lancsar E.
      • Louviere J.
      Conducting discrete choice experiments to inform healthcare decision making: a user’s guide.
      • Bridges J.F.P.
      • Hauber A.B.
      • Marshall D.
      • et al.
      Conjoint analysis applications in health—a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force.
      • Hauber A.B.
      • González J.M.
      • Groothuis-Oudshoorn C.G.M.
      • et al.
      Statistical methods for the analysis of discrete choice experiments: a report of the ISPOR Conjoint Analysis Good Research Practices Task Force.
      Nevertheless, arguably the most valuable output from a DCE are marginal rates of substitution (MRSs). MRSs provide information on the rate at which respondents are willing to trade one attribute for preferred levels of another attribute. For example, in a health context, this could be the amount of risk that individuals are willing to accept to achieve a given health benefit (referred to as maximum acceptable risk [MAR]), or the amount of money that individuals are willing to forgo (referred to as willingness to pay [WTP]) to receive their preferred mode of administration.
      Of all the information obtained using a DCE, MRSs are the most likely to be used within other analyses to inform decision making. For example, benefit-risk trade-offs such as MAR could be used in a quantitative benefit-risk analysis to inform regulatory approval decisions,
      • Ho M.P.
      • Gonzalez J.M.
      • Lerner H.P.
      • et al.
      Incorporating patient-preference evidence into regulatory decision making.
      and WTP estimates could be incorporated into economic evaluations to inform reimbursement decisions.
      • Tinelli M.
      • Ryan M.
      • Bond C.
      What, who and when? Incorporating a discrete choice experiment into an economic evaluation.
      It is therefore crucial that any uncertainty around MRS estimates is estimated, reported, and later incorporated into these subsequent analyses. Whilst there have been several comprehensive systematic reviews of DCEs in health, none have gone further than reporting the frequency of different outputs, yet Vass and Payne (2017) noted recently that uncertainty of MRSs is often left unreported.
      • Vass C.M.
      • Payne K.
      Using discrete choice experiments to inform the benefit-risk assessment of medicines: are we ready yet?.
      Therefore, while this problem is known, the extent of the issue is currently unknown in relation to published health-related DCEs.
      This study set out to fill this gap by conducting a systematic literature review of recently published DCE studies in health that elicit patient preferences, to explore the frequency and quality of MRS reporting. Additionally, an analysis was conducted to explore whether other frequently reported information could be used to generate reasonable estimates of the uncertainty around reported MRSs.

      Methods

       Search Strategy

      The search strategy for the literature review built upon existing strategies that have been applied in this area. Specifically, the same keywords, database, and constraints as in the review by Soekhai et al (2019) were used.
      • Soekhai V.
      • de Bekker-Grob E.W.
      • Ellis A.R.
      • Vass C.M.
      Discrete choice experiments in health economics: past, present and future.
      The main difference in our search strategy was that the term patient was added to limit the results to DCEs that elicited patient preferences. We also looked at more recent studies; we searched for studies published between 2014 and July 16, 2019 to ensure that the most recent literature was identified, such that we could comment on recent practice. The full search strategy can be found in Appendix 1 in the Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.04.1831.
      Studies were included in the review if they reported the primary data collection of patient preferences for healthcare, elicited using a DCE. The number of studies reporting coefficients, MRSs, and an estimate of the uncertainty of the MRSs were recorded. Studies that reported the following information were also included in the MRS analysis: MRSs, standard error (SE) or confidence interval (CI) of the reported MRSs, coefficients of individual attributes, and the SEs or CIs of the coefficients. Studies that reported median or simulated MRSs were excluded from the MRS analysis because further information would have been required to replicate the results. Additionally, studies were also excluded from the MRS analysis if the reported MRS were found to be non-reproducible and efforts to contact the authors for clarification were unsuccessful.
      For the included studies, the overall (full sample) results were prioritized in all cases where the MRS was reported for multiple subgroups, and any MRSs relating to other stakeholder groups (ie, not patients) were not extracted. The MRSs were only included if they could be accurately extracted (eg, MRSs reported in figures were not included unless numbers were clearly reported). The MRSs calculated for effects-coded attributes were excluded, because calculating the SE using the delta method requires the covariances of all parameters that define the effects coding for the 2 attributes in question, instead of a single covariance as is the case with dummy-coded attributes (see equation 2 below).

       MRS Analysis

      The typical approach for estimating MRS using DCE output is to take the ratio of the 2 attribute coefficients of interest, as in equation 1.
      • Lancsar E.
      • Louviere J.
      Conducting discrete choice experiments to inform healthcare decision making: a user’s guide.
      MRSxy=βxβy
      (1)


      where MRSxy is the MRS between attributes x and y, and βx and βy are their coefficients in the choice model. The standard error of MRSxy is often computed using the delta method,
      • Bliemer M.C.J.
      • Rose J.M.
      Confidence intervals of willingness-to-pay for random coefficient logit models.
      as in equation 2.
      SE[MRSxy]=Var[βx]+MRSxy2Var[βy]2MRSxyCov[βx,βy]βy
      (2)


      where Var[βx] and Var[βy] are the variances of the coefficients and Cov[βx,βy] is the covariance between the 2 estimates.
      Our a priori hypothesis was that coefficients and their accompanying SEs or CIs would often be reported in published studies, and that covariances would not be reported. If MRSs are not reported, or if they are reported without accompanying SEs or CIs, the only missing information required to calculate MRSs and accompanying SEs (using equations 1 and 2, respectively) in a typical study would be the covariances.
      The objective of the MRS analysis was to determine the potential impact on estimates of SEMRSxy if Cov[βx,βy] was assumed to be equal to zero. If SEs or CIs of MRSs are not reported, this estimate is the closest that a reader could get to the actual SE without contacting the authors of the publication for further data. We compared these estimates with the actual SE and quantified the level of imprecision in our estimate to assess whether the Cov[βx,βy]=0 assumption is reasonable. Specifically, we focused on the ratio of our recalculated SEs assuming Cov[βx,βy]=0 with the actual SEs, as in equation 3, because this allowed us to make comparisons between studies. We refer to this as the magnitude of error and include this for illustrative rather than practical purposes, that is, to illustrate how (in)accurate SEs are when calculated without all the necessary information.
      SE[MRSxy]  Recalculated SE[MRSxy]assumingCov[βx,βy]=0 
      (3)


      Results

       Literature Review

      A total of 484 abstracts were identified in the initial database search. After title and abstract screening, a total of 239 studies were excluded. This left a total of 245 studies for full-text screening. Upon reviewing the full-texts, 2 studies did not contain any primary data collection and 11 did not elicit preferences from patients. Thus, we identified a total of 232 DCE studies that elicited patient preferences overall. Of these studies, 65.1% (n = 151) reported 1 or more coefficients and a smaller proportion (34.1%; n = 79) reported 1 or more MRSs. Of the 79 studies that reported MRSs, 62.0% (n = 49) provided an estimate of the uncertainty, that is, a SE or CI. Of these 49 studies, only 43% (n = 21) reported the method used to generate SEs or CIs. The delta method was the most commonly reported (42.9%; n = 9), closely followed by bootstrapping (38.1%; n = 8). Other reported methods were Krinsky-Robb (9.5%; n = 2) and Markov Chain Monte Carlo (9.5%; n = 2).
      Although 49 studies reported MRSs with an estimate of the uncertainty, only 16 met the criteria for inclusion in the MRS analysis. This was due to a variety of reasons, with the most common being a lack of accompanying coefficients with SEs or CIs (42.4%; n = 14) and the exclusive use of effects coding for relevant attributes (30.3%; n = 10). The overall review process is illustrated in a Preferred Reporting Items for Systematic Reviews and Meta-analyses flow diagram
      • Moher D.
      • Liberati A.
      • Tetzlaff J.
      • Altman D.G.
      PRISMA Group
      Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
      in Appendix 2 in the Supplemental Materials found at https://doi.org/10.1016/j.jval.2020.04.1831.

       MRS Analysis

      Details of the studies included in the MRS analysis can be found in Table 1. The included studies were conducted in a range of countries, with European studies the most common (56.3%; n = 9), and the mean sample size was 201. Willingness to pay was the most common type of MRS (43.8%; n = 7) and, on average, the studies provided 7 data points each. Despite reporting SEs and CIs, most included studies did not report the method used (56.3%; n = 9). Of those that did, the delta method and bootstrapping were the most common (n = 3 each). In total, there were 116 different data points available for the MRS analysis.
      Table 1Characteristics of the studies included in the MRS analysis.
      StudyCountryContextSample size
      Sample size pertaining to the extracted MRS.
      Econometric modelNo. MRSsType(s) of MRSMethod to obtain SEs/CIs
      Hawken et al
      • Hawken N.
      • Torvinen S.
      • Neine M.-E.
      • et al.
      Patient preferences for dry powder inhaler attributes in asthma and chronic obstructive pulmonary disease in France: a discrete choice experiment.
      (2017)
      FRRespiratory201 & 93
      This article had 2 patient samples: asthma patients (n = 201) and chronic obstructive pulmonary disease patients (n = 93).
      Ordered logit16WTPMCMC
      Meads et al
      • Meads D.M.
      • O’Dwyer J.L.
      • Hulme C.T.
      • Chintakayala P.
      • Vinall-Collier K.
      • Bennett M.I.
      Patient preferences for pain management in advanced cancer: results from a discrete choice experiment.
      (2017)
      UKOncology221Mixed logit16WTP/WTWN/R
      Tinelli et al
      • Tinelli M.
      • Nikoloski Z.
      • Kumpunen S.
      • et al.
      Decision-making criteria among European patients: exploring patient preferences for primary care services.
      (2015)
      DE/SI/UKPrimary care services692Conditional logit13WTWN/R
      Goto et al
      • Goto R.
      • Uda A.
      • Hiroi S.
      • Iwasaki K.
      • Takashima K.
      • Kurebayashi J.
      Cost analysis of leuprorelin acetate in Japanese pre-menopausal breast-cancer patients: comparison between 6-month and 3-month depot formulations.
      (2017)
      JPOncology107Conditional logit11WTPN/R
      Goto et al
      • Goto R.
      • Uda A.
      • Hiroi S.
      • Iwasaki K.
      • Takashima K.
      • Oya M.
      Cost analysis of leuprorelin acetate in Japanese prostate cancer patients: comparison between 6-month and 3-month depot formulations.
      (2017)
      JPOncology108Conditional logit11WTPN/R
      Gray et al
      • Gray E.
      • Eden M.
      • Vass C.
      • McAllister M.
      • Louviere J.
      • Payne K.
      Valuing preferences for the process and outcomes of clinical genetics services: a pilot study.
      (2016)
      UKClinical genetics services37Ordered logit7WTPBootstrapping
      Kløjgaard et al
      • Kløjgaard M.E.
      • Manniche C.
      • Pedersen L.B.
      • Bech M.
      • Søgaard R.
      Patient preferences for treatment of low back pain—a discrete choice experiment.
      (2014)
      DKOrthopedics348Conditional logit7WTWN/R
      Zanolini et al
      • Zanolini A.
      • Sikombe K.
      • Sikazwe I.
      • et al.
      Understanding preferences for HIV care and treatment in Zambia: evidence from a discrete choice experiment among patients who have been lost to follow-up.
      (2018)
      ZMHIV280Mixed logit6WTWN/R
      Wong et al
      • Wong S.F.
      • Norman R.
      • Dunning T.L.
      • et al.
      A discrete choice experiment to examine the preferences of patients with cancer and their willingness to pay for different types of health care appointments.
      (2016)
      AUSOncology185Conditional logit5WTPN/R
      Havrilesky et al
      • Havrilesky L.J.
      • Alvarez Secord A.
      • Ehrisman J.A.
      • et al.
      Patient preferences in advanced or recurrent ovarian cancer: patient preferences in ovarian cancer.
      (2014)
      USOncology95Mixed logit5WTTSBootstrapping
      Najafzadeh et al
      • Najafzadeh M.
      • Schneeweiss S.
      • Choudhry N.K.
      • Avorn J.
      • Gagne J.J.
      General population vs. patient preferences in anticoagulant therapy: a discrete choice experiment.
      (2018)
      USCardiovascular284Conditional logit5MARDelta
      Powell et al
      • Powell G.
      • Holmes E.A.F.
      • Plumpton C.O.
      • et al.
      Pharmacogenetic testing prior to carbamazepine treatment of epilepsy: patients’ and physicians’ preferences for testing and service delivery: genetic testing preferences in epilepsy.
      (2015)
      UKEpilepsy82Random effects logit4WTWBootstrapping
      Brooks et al
      • Brooks A.
      • Langer J.
      • Tervonen T.
      • Hemmingsen M.P.
      • Eguchi K.
      • Bacci E.D.
      Patient preferences for GLP-1 receptor agonist treatment of type 2 diabetes mellitus in Japan: a discrete choice experiment.
      (2019)
      JPDiabetes161Mixed logit4MARN/R
      de Bekker-Grob et al
      • de Bekker-Grob E.W.
      • Niers E.J.
      • van Lanschot JJB
      • Steyerberg E.W.
      • Wijnhoven B.P.L.
      Patients’ preferences for surgical management of esophageal cancer: a discrete choice experiment.
      (2015)
      NLOncology97Mixed logit3WTTSDelta
      Sever et al
      • Sever I.
      • Verbič M.
      • Sever E.K.
      Valuing the delivery of dental care: heterogeneity in patients’ preferences and willingness-to-pay for dental care attributes.
      (2017)
      CRDental265Mixed logit2WTPDelta
      van den Brink et al
      • van den Brink M.J.
      • Beelen P.
      • Herman M.C.
      • et al.
      Women’s preferences for the levonorgestrel intrauterine system versus endometrial ablation for heavy menstrual bleeding.
      (2018)
      NLGynecology165Mixed logit1MARN/R
      AUS indicates Australia; CI, confidence interval; CR, Croatia; DE, Germany; DK, Denmark; FR, France; JP, Japan; MAE, minimum acceptable efficacy; MAR, maximum acceptable risk; MCMC, Markov Chain Monte Carlo method; NL, The Netherlands; N/R, not reported; SE, standard error; SI, Slovenia; US, United States; WTP, willingness to pay; WTT, willingness to travel; WTTS, willing to trade survival; WTW, willingness to wait; ZM, Zambia.
      Sample size pertaining to the extracted MRS.
      This article had 2 patient samples: asthma patients (n = 201) and chronic obstructive pulmonary disease patients (n = 93).
      Figure 1 illustrates the magnitude of error in SE[MRSxy] (equation 3). The difference between the actual SEs and recalculated SEs varied substantially and worked in both directions, reflecting the variation in covariances. The average was 92.7%, indicating that the recalculated SEs were typically larger than the actual SEs.
      Figure thumbnail gr1
      Figure 1Magnitude of error when recalculating SEs of MRSs assuming zero covariance.
      Cov indicates covariance; MRS, marginal rate of substitution; SE, standard error.
      The majority of recalculated SEs were within 25% of the actual SE (59.5%; n = 69), indicating that calculating SEs assuming zero covariance may have a modest effect in many cases. For example, in Hawken et al (2017), the authors estimated that asthma patients would be willing to pay €4.49 per month (95% CI: €2.95-6.02) for an inhaler that could be used during episodes of breathing difficulties (relative to one that cannot). We estimated the covariance to be -0.0003 based on reported data; however, when assuming zero covariance, the recalculated SE was 0.65 (compared to 0.79 for the actual SE), meaning that the actual SE was 121.5% the magnitude of the recalculated SE. If the recalculated SE were used to estimate the 95% CI, it would have been €3.21 to €5.77, inaccurately implying that the WTP estimate was slightly more precise.
      In our data set, it was more common for recalculated SEs to be larger than the actual SEs (75.0%; n = 87). In these cases, estimating the SE using reported information would have led to underestimation of precision of the MRS estimate. A substantial proportion of recalculated SEs had a magnitude of error of greater than 25% (40.5%; n = 47). The impact of the zero-covariance assumption is greater in these instances. For example, in Goto et al (2017) it was estimated that breast cancer patients were willing to pay ¥9658 (95% CI: ¥8607-¥10 460) for a decrease in risk of injection site reactions from 10% to 1%. We estimated the covariance to be 0.00001 based on reported data; however, when assuming zero covariance, the recalculated SE was 1304 (compared with 536 for the actual SE), meaning that the actual SE was 41.1% of the magnitude of the recalculated SE. If the recalculated SE were used to estimate the 95% CI, it would have been ¥7101 to ¥12 214, inaccurately implying that the WTP estimate was significantly less precise.

      Discussion

       Implications of the Results

      Regulators such as the US Food and Drug Administration (FDA) have become increasingly interested in benefit-risk assessments that are informed by patient preferences. The FDA’s own guidance on patient preference studies recommends the reporting of uncertainty,
      U.S. Food & Drug Administration
      Patient Preference Information – Voluntary Submission, Review in Premarket Approval Applications, Humanitarian Device Exemption Applications, and De Novo Requests, and Inclusion in Decision Summaries and Device Labeling: Guidance for Industry, Food and Drug Administration Staff, and Other Stakeholders.
      particularly when a parameter may be influential, which may be the case for MRSs such as MAR. Comparatively, the role of patient preference studies in health technology assessment (HTA) is less clear, because there is little in the way of official guidance,
      • Mott D.J.
      Incorporating quantitative patient preference data into healthcare decision making processes: is HTA falling behind?.
      ,
      • Mott D.J.
      • Najafzadeh M.
      Whose preferences should be elicited for use in health-care decision-making? A case study using anticoagulant therapy.
      though there is an increasing interest from agencies such as the National Institute for Health and Care Excellence.
      • Bouvy J.C.
      • Cowie L.
      • Lovett R.
      • Morrison D.
      • Livingstone H.
      • Crabb N.
      Use of patient preference studies in HTA decision making: a NICE perspective.
      The HTA agencies that provide methodological guidance for cost-effectiveness analyses typically recommend that uncertainty is estimated and shared with decision makers.
      CADTH
      Guidelines for the Economic Evaluation of Health Technologies: Canada.
      ,
      National Institute for Health and Care Excellence
      Guide to the Methods of Technology Appraisal.
      Arguably, it follows that uncertainty around patient preference study results would also be of importance in the HTA context.
      The current study identified a total of 232 health-related DCE studies that elicited patient preferences published between 2014 and July 16, 2019. Only one-third of these studies reported 1 or more MRSs despite MRS being one of the most useful outputs of a DCE. Of the studies that reported MRSs, just under two-thirds reported accompanying estimates of uncertainty (62.0%; n = 49). Therefore, based on our review, over one-third of patient preference studies that report MRSs fall short of providing information on uncertainty that would be of real importance to decision makers.

       Improving Reporting Standards

      The reasons behind the inadequate reporting of MRSs are unclear, especially given that published studies are subject to peer review processes. The poor reporting could be due to a lack of statistical proficiency among authors and reviewers, or due to some misconceptions surrounding the calculation of CIs for ratios. Although both could be addressed by the existence and application of good practice guidelines, commonly referenced articles providing guidance for the conduct of DCEs in health fail to provide information on how to calculate SEs or CIs for MRSs
      • Lancsar E.
      • Louviere J.
      Conducting discrete choice experiments to inform healthcare decision making: a user’s guide.
      ,
      • Hauber A.B.
      • González J.M.
      • Groothuis-Oudshoorn C.G.M.
      • et al.
      Statistical methods for the analysis of discrete choice experiments: a report of the ISPOR Conjoint Analysis Good Research Practices Task Force.
      —or even how to calculate MRSs.
      • Hauber A.B.
      • González J.M.
      • Groothuis-Oudshoorn C.G.M.
      • et al.
      Statistical methods for the analysis of discrete choice experiments: a report of the ISPOR Conjoint Analysis Good Research Practices Task Force.
      Regarding misconceptions, Vass and Payne (2017) suggest that some researchers mistakenly believe that the ratio of 2 statistically significant coefficients will always be statistically significant and therefore an estimate of the uncertainty can be left omitted.
      • Vass C.M.
      • Payne K.
      Using discrete choice experiments to inform the benefit-risk assessment of medicines: are we ready yet?.
      We therefore recommend that, at minimum, researchers make use of post-estimation commands to calculate SEs or CIs for MRSs. For example, the delta method can be computed using the nlcom command in Stata
      or the DeltaMethod function in R
      R Core Team
      The R project for statistical computing. R Foundation for Statistical Computing.
      (part of the RcmdrMisc package
      • Fox J.
      • Muenchen R.
      • Putler D.
      RcmdrMisc: R commander miscellaneous functions.
      ). However, as highlighted by the results of this review, alternative methods are sometimes used to estimate CIs. In contrast to the delta method, other alternatives such as Krinsky-Robb, bootstrapping, and Markov Chain Monte Carlo method require simulations.
      • Hole A.R.
      A comparison of approaches to estimating confidence intervals for willingness to pay measures.
      ,
      • Atchadé Y.F.
      Markov Chain Monte Carlo confidence intervals.
      The former can be computed using the user-written wtp command
      • Hole A.
      WTP: Stata Module to Estimate Confidence Intervals for Willingness to Pay Measures. Statistical Software Components S456808.
      in Stata or the mwtp function in R (part of the support.CEs package
      • Aizaki H.
      Basic functions for supporting an implementation of choice experiments in R.
      ). Although in theory some methods are better suited to different circumstances and may provide different results, a comparison by Hole (2007)
      • Hole A.R.
      A comparison of approaches to estimating confidence intervals for willingness to pay measures.
      found that all the methods compared were reasonably accurate and results were similar. Finally, a potentially useful alternative could be to estimate models in WTP-space, rather than the preference-space.
      • Train K.
      • Weeks M.
      Discrete choice models in preference space and willingness-to-pay space.

       Strengths and Limitations

      The main strength of this study is that it sought to provide evidence of a significant reporting issue that has been acknowledged but not thoroughly explored to date. Furthermore, rather than simply stating the frequency of misreporting, we went beyond this to attempt to illustrate the potential implications of leaving SE calculations to be made by readers. Finally, another strength is the systematic literature review, which closely followed the approach taken in major literature reviews of health-related DCEs that have been published in the past.
      This study also has its limitations. Our review focused solely on patient preference studies, rather than including all health-related DCE studies. This approach was taken because of the current regulatory interest in patient preferences to inform treatment marketing authorization decisions, as well as the increasing interest from HTA agencies. We had no a priori hypothesis that uncertainty would be reported differently in studies eliciting preferences of other stakeholders. Regardless, our recommendations hold for any DCE study that reports MRSs. Another limitation was that we had to exclude a significant number of studies (and individual MRSs) from the MRS analysis owing to the use of effects coding for categorical attributes. Although this limited the number of data points, the analysis was intended to simply illustrate that SEs or CIs calculated by readers using the reported information are unlikely to be accurate; additional data points would not have changed this conclusion. Finally, another limitation is that we did not double screen the identified studies during the abstract and title screening, nor the full-text screening. Nevertheless, the primary reviewer (N.C.) consulted with the coauthors (D.M. and T.T.) whenever any queries arose regarding inclusion/exclusion or the data extraction process.

      Conclusion

      Despite being good practice, the uncertainty of MRSs is not reported in a significant proportion of recently published DCE studies. Our analysis indicates that SE estimates of MRSs based on other reported information will often be biased without covariances, which cannot be assumed to be zero. As patient preference studies are becoming increasingly important, it is crucial that all studies reporting MRSs also report the degree of uncertainty around these estimates.

      Acknowledgment

      We are grateful to Sebastian Heidenreich, Katherine Carr, Tara Homer, and participants at the Health Economists' Study Group meeting in Newcastle upon Tyne in January 2020 for feedback on earlier drafts of this paper.

      Supplemental Material

      References

        • Johnson F.R.
        • Beusterien K.
        • Özdemir S.
        • Wilson L.
        Giving patients a meaningful voice in United States regulatory decision making: the role for health preference research.
        Patient. 2017; 10: 523-526
        • Mühlbacher A.C.
        • Johnson F.R.
        Giving patients a meaningful voice in European health technology assessments: the role of health preference research.
        Patient. 2017; 10: 527-530
        • Mott D.J.
        Incorporating quantitative patient preference data into healthcare decision making processes: is HTA falling behind?.
        Patient. 2018; 11: 249-252
        • Janssens R.
        • Huys I.
        • van Overbeeke E.
        • et al.
        Opportunities and challenges for the inclusion of patient preferences in the medical product life cycle: a systematic review.
        BMC Med Inform Decis Mak. 2019; 19: 189
        • van Overbeeke E.
        • Whichello C.
        • Janssens R.
        • et al.
        Factors and situations influencing the value of patient preference studies along the medical product lifecycle: a literature review.
        Drug Discov Today. 2019; 24: 57-68
        • Tervonen T.
        • Gelhorn H.
        • Sri Bhashyam S.
        • et al.
        MCDA swing weighting and discrete choice experiments for elicitation of patient benefit-risk preferences: a critical assessment.
        Pharmacoepidemiol Drug Saf. 2017; 26: 1483-1491
        • Hauber B.
        • Coulter J.
        Using the threshold technique to elicit patient preferences: an introduction to the method and an overview of existing empirical applications.
        Appl Health Econ Health Policy. 2020; 18: 31-46
        • Schmidt K.
        • Babac A.
        • Pauer F.
        • Damm K.
        • von der Schulenburg J.-M.
        Measuring patients’ priorities using the Analytic Hierarchy Process in comparison with Best-Worst-Scaling and rating cards: methodological aspects and ranking tasks.
        Health Econ Rev. 2016; 6: 50
        • Krucien N.
        • Sicsic J.
        • Ryan M.
        For better or worse? Investigating the validity of best–worst discrete choice experiments in health.
        Health Econ. 2019; 28: 572-586
        • Soekhai V.
        • Whichello C.
        • Levitan B.
        • et al.
        Methods for exploring and eliciting patient preferences in the medical product lifecycle: a literature review.
        Drug Discov Today. 2019; 24: 1324-1331
        • Louviere J.J.
        • Lancsar E.
        Choice experiments in health: the good, the bad, the ugly and toward a brighter future.
        Health Econ Policy Law. 2009; 4: 527-546
        • Clark M.D.
        • Determann D.
        • Petrou S.
        • Moro D.
        • de Bekker-Grob E.W.
        Discrete choice experiments in health economics: a review of the literature.
        Pharmacoeconomics. 2014; 32: 883-902
        • de Bekker-Grob E.W.
        • Ryan M.
        • Gerard K.
        Discrete choice experiments in health economics: a review of the literature.
        Health Econ. 2012; 21: 145-172
        • Ryan M.
        • Gerard K.
        Using discrete choice experiments to value health care programmes: current practice and future research reflections.
        Appl Health Econ Health Policy. 2003; 2: 55-64
        • Soekhai V.
        • de Bekker-Grob E.W.
        • Ellis A.R.
        • Vass C.M.
        Discrete choice experiments in health economics: past, present and future.
        Pharmacoeconomics. 2019; 37: 201-226
        • Lancsar E.
        • Louviere J.
        Conducting discrete choice experiments to inform healthcare decision making: a user’s guide.
        Pharmacoeconomics. 2008; 26: 661-677
        • Bridges J.F.P.
        • Hauber A.B.
        • Marshall D.
        • et al.
        Conjoint analysis applications in health—a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force.
        Value Health. 2011; 14: 403-413
        • Hauber A.B.
        • González J.M.
        • Groothuis-Oudshoorn C.G.M.
        • et al.
        Statistical methods for the analysis of discrete choice experiments: a report of the ISPOR Conjoint Analysis Good Research Practices Task Force.
        Value Health. 2016; 19: 300-315
        • Ho M.P.
        • Gonzalez J.M.
        • Lerner H.P.
        • et al.
        Incorporating patient-preference evidence into regulatory decision making.
        Surg Endosc. 2015; 29: 2984-2993
        • Tinelli M.
        • Ryan M.
        • Bond C.
        What, who and when? Incorporating a discrete choice experiment into an economic evaluation.
        Health Econ Rev. 2016; 6: 31
        • Vass C.M.
        • Payne K.
        Using discrete choice experiments to inform the benefit-risk assessment of medicines: are we ready yet?.
        Pharmacoeconomics. 2017; 35: 859-866
        • Bliemer M.C.J.
        • Rose J.M.
        Confidence intervals of willingness-to-pay for random coefficient logit models.
        Transp Res Part B Method. 2013; 58: 199-214
        • Moher D.
        • Liberati A.
        • Tetzlaff J.
        • Altman D.G.
        • PRISMA Group
        Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
        BMJ. 2009; 339: b2535
        • Hawken N.
        • Torvinen S.
        • Neine M.-E.
        • et al.
        Patient preferences for dry powder inhaler attributes in asthma and chronic obstructive pulmonary disease in France: a discrete choice experiment.
        BMC Pulm Med. 2017; 17: 99
        • Meads D.M.
        • O’Dwyer J.L.
        • Hulme C.T.
        • Chintakayala P.
        • Vinall-Collier K.
        • Bennett M.I.
        Patient preferences for pain management in advanced cancer: results from a discrete choice experiment.
        Patient. 2017; 10: 643-651
        • Tinelli M.
        • Nikoloski Z.
        • Kumpunen S.
        • et al.
        Decision-making criteria among European patients: exploring patient preferences for primary care services.
        Eur J Public Health. 2015; 25: 3-9
        • Goto R.
        • Uda A.
        • Hiroi S.
        • Iwasaki K.
        • Takashima K.
        • Kurebayashi J.
        Cost analysis of leuprorelin acetate in Japanese pre-menopausal breast-cancer patients: comparison between 6-month and 3-month depot formulations.
        J Med Econ. 2017; 20: 1163-1169
        • Goto R.
        • Uda A.
        • Hiroi S.
        • Iwasaki K.
        • Takashima K.
        • Oya M.
        Cost analysis of leuprorelin acetate in Japanese prostate cancer patients: comparison between 6-month and 3-month depot formulations.
        J Med Econ. 2017; 20: 1155-1162
        • Gray E.
        • Eden M.
        • Vass C.
        • McAllister M.
        • Louviere J.
        • Payne K.
        Valuing preferences for the process and outcomes of clinical genetics services: a pilot study.
        Patient. 2016; 9: 135-147
        • Kløjgaard M.E.
        • Manniche C.
        • Pedersen L.B.
        • Bech M.
        • Søgaard R.
        Patient preferences for treatment of low back pain—a discrete choice experiment.
        Value Health. 2014; 17: 390-396
        • Zanolini A.
        • Sikombe K.
        • Sikazwe I.
        • et al.
        Understanding preferences for HIV care and treatment in Zambia: evidence from a discrete choice experiment among patients who have been lost to follow-up.
        PLoS Med. 2018; 15e1002636
        • Wong S.F.
        • Norman R.
        • Dunning T.L.
        • et al.
        A discrete choice experiment to examine the preferences of patients with cancer and their willingness to pay for different types of health care appointments.
        J Natl Compr Canc Netw. 2016; 14: 311-319
        • Havrilesky L.J.
        • Alvarez Secord A.
        • Ehrisman J.A.
        • et al.
        Patient preferences in advanced or recurrent ovarian cancer: patient preferences in ovarian cancer.
        Cancer. 2014; 120: 3651-3659
        • Najafzadeh M.
        • Schneeweiss S.
        • Choudhry N.K.
        • Avorn J.
        • Gagne J.J.
        General population vs. patient preferences in anticoagulant therapy: a discrete choice experiment.
        Patient. 2019; 12: 235-246
        • Powell G.
        • Holmes E.A.F.
        • Plumpton C.O.
        • et al.
        Pharmacogenetic testing prior to carbamazepine treatment of epilepsy: patients’ and physicians’ preferences for testing and service delivery: genetic testing preferences in epilepsy.
        Br J Clin Pharmacol. 2015; 80: 1149-1159
        • Brooks A.
        • Langer J.
        • Tervonen T.
        • Hemmingsen M.P.
        • Eguchi K.
        • Bacci E.D.
        Patient preferences for GLP-1 receptor agonist treatment of type 2 diabetes mellitus in Japan: a discrete choice experiment.
        Diabetes Ther. 2019; 10: 735-749
        • de Bekker-Grob E.W.
        • Niers E.J.
        • van Lanschot JJB
        • Steyerberg E.W.
        • Wijnhoven B.P.L.
        Patients’ preferences for surgical management of esophageal cancer: a discrete choice experiment.
        World J Surg. 2015; 39: 2492-2499
        • Sever I.
        • Verbič M.
        • Sever E.K.
        Valuing the delivery of dental care: heterogeneity in patients’ preferences and willingness-to-pay for dental care attributes.
        J Dent. 2018; 69: 93-101
        • van den Brink M.J.
        • Beelen P.
        • Herman M.C.
        • et al.
        Women’s preferences for the levonorgestrel intrauterine system versus endometrial ablation for heavy menstrual bleeding.
        Eur J Obstet Gynecol Reprod Biol. 2018; 228: 143-147
        • U.S. Food & Drug Administration
        Patient Preference Information – Voluntary Submission, Review in Premarket Approval Applications, Humanitarian Device Exemption Applications, and De Novo Requests, and Inclusion in Decision Summaries and Device Labeling: Guidance for Industry, Food and Drug Administration Staff, and Other Stakeholders.
        U.S. Food & Drug Administration, Silver Spring, MD2016
        • Mott D.J.
        • Najafzadeh M.
        Whose preferences should be elicited for use in health-care decision-making? A case study using anticoagulant therapy.
        Expert Rev Pharmacoecon Outcomes Res. 2016; 16: 33-39
        • Bouvy J.C.
        • Cowie L.
        • Lovett R.
        • Morrison D.
        • Livingstone H.
        • Crabb N.
        Use of patient preference studies in HTA decision making: a NICE perspective.
        Patient. 2020; 13: 145-149
        • CADTH
        Guidelines for the Economic Evaluation of Health Technologies: Canada.
        4th ed. CADTH, Ottawa, Canada2017
        • National Institute for Health and Care Excellence
        Guide to the Methods of Technology Appraisal.
        National Institute for Health and Care Excellence, London, England2013
      1. Stata Statistical Software: Release 16. StataCorp LLC, College Station, TX2019
        • R Core Team
        The R project for statistical computing. R Foundation for Statistical Computing.
        • Fox J.
        • Muenchen R.
        • Putler D.
        RcmdrMisc: R commander miscellaneous functions.
        • Hole A.R.
        A comparison of approaches to estimating confidence intervals for willingness to pay measures.
        Health Econ. 2007; 16: 827-840
        • Atchadé Y.F.
        Markov Chain Monte Carlo confidence intervals.
        Bernoulli. 2016; 22: 1808-1838
        • Hole A.
        WTP: Stata Module to Estimate Confidence Intervals for Willingness to Pay Measures. Statistical Software Components S456808.
        Boston College Department of Economics, Boston, MA2007
        • Aizaki H.
        Basic functions for supporting an implementation of choice experiments in R.
        J Stat Softw. 2012; 50: 1-24
        • Train K.
        • Weeks M.
        Discrete choice models in preference space and willingness-to-pay space.
        in: Scarpa R. Alberini A. Applications of Simulation Methods in Environmental and Resource Economics. Springer Netherlands, Dordrecht2005: 1-16