If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Canadian Centre for Applied Research in Cancer Control (ARCC), British Columbia Cancer Agency, Vancouver, WA, USALeslie Diamond Chair in Cancer Survivorship, Simon Fraser University, Vancouver, WA, USA
Health care decisions are complex and involve confronting trade-offs between multiple, often conflicting, objectives. Using structured, explicit approaches to decisions involving multiple criteria can improve the quality of decision making and a set of techniques, known under the collective heading multiple criteria decision analysis (MCDA), are useful for this purpose. MCDA methods are widely used in other sectors, and recently there has been an increase in health care applications. In 2014, ISPOR established an MCDA Emerging Good Practices Task Force. It was charged with establishing a common definition for MCDA in health care decision making and developing good practice guidelines for conducting MCDA to aid health care decision making. This initial ISPOR MCDA task force report provides an introduction to MCDA - it defines MCDA; provides examples of its use in different kinds of decision making in health care (including benefit risk analysis, health technology assessment, resource allocation, portfolio decision analysis, shared patient clinician decision making and prioritizing patients’ access to services); provides an overview of the principal methods of MCDA; and describes the key steps involved. Upon reviewing this report, readers should have a solid overview of MCDA methods and their potential for supporting health care decision making.
In May 2014, the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Health Science Policy Council recommended to the ISPOR Board of Directors that an ISPOR Emerging Good Practices Task Force on multiple criteria decision analysis (MCDA) and its use in health care decision making be established. MCDA comprises a broad set of methodological approaches from operations research now being used increasingly in the health care sector. The task force goal was to provide a foundational report on the topic, an MCDA primer, and then focus on initial recommendations on how best to use MCDA methods to support health care decision making.
The task force leadership group comprises experts in health technology assessment, health care research, modeling, pricing, formulary development, epidemiology, and economics. Task force members were selected to represent a diverse range of perspectives. They work in hospital health systems, health technology assessment agencies, research organizations, academia, and the insurance and pharmaceutical industries. The task force had international representation with members from the United Kingdom, Belgium, Canada, the Netherlands, Sweden, Hungary, and the United States, in addition to reviewers from around the world.
The task force met approximately every four weeks by teleconference to develop detailed outlines and discuss issues and revisions. In addition, task force members met in person at ISPOR International Meetings and European Congresses. The four co-chairs taught an MCDA course at two of these ISPOR meetings and presented their preliminary findings at workshop and forum presentations multiple times. The final reports were presented at the Third Plenary of the ISPOR 18th European Congress in Milan.
Many comments were received during these presentations. Equally, if not more importantly, both reports were submitted for review twice. Nearly 50 ISPOR members knowledgeable on the topic submitted substantive written comments during these review rounds. All comments were considered. These were discussed by the task force on a series of teleconferences and during a 1.5-day task force face-to-face consensus meeting. Comments were addressed as appropriate in subsequent versions of the report. We gratefully acknowledge our reviewers for their contribution to the task force consensus development process and to the quality of these ISPOR MCDA task force reports.
All written comments are published at the ISPOR Web site on the task force’s Web page: http://www.ispor.org/Multi-Criteria-Decision-Analysis-guideline.asp. The task force report and Web page may also be accessed from the ISPOR homepage (www.ispor.org) via the purple Research Tools menu, ISPOR Good Practices for Outcomes Research, heading: Use of Outcomes Research in Health Care Decisions.
Health care decisions are rarely simple with easy answers. Complexity in these decisions is inevitable, whether a high-level decision, such as that made by a budget holder, allocating limited resources across treatments, or at the micro level, such as a patient’s decision on the best treatment alternative. Multiple factors impact these decisions, a number of alternatives exist, and the information available about each alternative is often imperfect. The cognitive burden involved can lead to the use of heuristics—in effect, mental shortcuts, such as adopting a “rule of thumb”—and systematic mistakes [
]. It can be a Herculean effort to assess the alternatives and the relevant evidence to make a good, informed decision.
Decision makers, whether they are individuals or committees, have difficulty processing and systematically evaluating relevant information. This assessment process involves confronting trade-offs between the alternatives under consideration. Each decision maker will need to prioritize what matters most. If more than one individual is involved, the priorities of involved decision makers can, and frequently do, conflict, increasing the difficulty and complexity of the decision-making process. Despite this complexity, decisions are made: even sticking with status quo is itself a decision. Relying on informal processes or judgments can lead to suboptimal decisions.
Without a formal process to evaluate alternatives and priorities, there may be inconsistency, variability, or a lack of predictability on a particular factor’s or criterion’s importance in the decision. The decision makers’ credibility and potentially legitimacy may come into question, and this is especially true for accountability on a decision made by a public body if there is a lack of transparency about how a decision was made. For example, it is argued that health technology assessment (HTA) agencies should aim to be transparent about the decision-making process to ensure fair resource allocation [
The decision-making process can be improved by working with decision makers and stakeholders providing support and structure to the process. Using structured, explicit approaches to decisions involving multiple criteria can improve the quality of decision making and a set of techniques, known under the collective heading multiple criteria decision analysis (MCDA), are useful for this purpose. This set of techniques provides clarity on which criteria are relevant, the importance attached to each, and how to use this information in a framework for assessing the available alternatives. By doing so, they can help increase the consistency, transparency, and legitimacy of decisions.
MCDA comprises a broad set of methodological approaches, originating from operations research, yet with a rich intellectual grounding in other disciplines [
]. The health care sector has been relatively slow to apply MCDA. But as more researchers and practitioners have become aware of the techniques, there has been a sharp increase in its health care application [
]. These differ not just in how MCDA is put into practice but also in terms of the fundamental theories and beliefs underpinning them. The existence of different schools of thought, representing different positions on how MCDA should be performed, makes the choice of MCDA method to use in any given context quite complex. This is made still more difficult by the existence of various commercial and not-for-profit MCDA “toolkits” promoted by their developers. The current literature on MCDA in health care offers little guidance to users on how to choose from the bewildering array of approaches, on the “best” approach for different types of decisions, and what the relevant considerations are. In the absence of guidance on how to implement MCDA techniques in health care, MCDA can be misused and the decision makers misled [
In 2014, ISPOR established an Emerging Good Practices Task Force, charged with the objectives of establishing a common definition for MCDA in health care decision making and developing good practice guidelines for conducting MCDA to aid health care decision making. This initial ISPOR MCDA task force report provides an introduction to MCDA for those unfamiliar with it. It defines MCDA, provides examples of its use in different kinds of decision making in health care, provides an overview of the principal methods of MCDA, and describes the key steps involved. The second task force report builds on this by providing emerging good practice guidelines for selecting the “right” approach to MCDA in each type of decision and how to implement these approaches, and also provides a checklist for those conducting an MCDA. The task force reports do not provide specific recommendations for individual applications (e.g., how MCDA should be used in HTA), and further research is required to thoroughly address the issues relevant to each decision.
] seminal book on MCDA defines it as “an extension of decision theory that covers any decision with multiple objectives. A methodology for appraising alternatives on individual, often conflicting criteria, and combining them into one overall appraisal…” An alternative definition in another influential text by Belton and Stewart [
] defines MCDA as “an umbrella term to describe a collection of formal approaches, which seek to take explicit account of multiple criteria in helping individuals or groups explore decisions that matter.” These definitions are not mutually exclusive, and Keeney and Raiffa’s definition is a subsect of Belton and Stewart’s definition. Keeney and Raiffa’s definition of MCDA necessitates the aggregation of information on criteria into a single expression of value, whereas Belton and Stewart’s definition allows the possibility of MCDA being used to support deliberation without the need for aggregation.
The task force adopted a broad approach, including in our consideration of MCDA methods that help deliberative discussions using explicitly defined criteria, but without quantitative modeling. For example, evidence on each alternative under consideration, in terms of its measured performance on each of the criteria considered to be relevant, can be assembled in a “performance matrix”—an example is the European Medicine Agency’s (EMA’s) effects table for benefit-risk analysis (BRA) [
]. Decision makers can find this “partial” form of MCDA a useful way of summarizing the relevant evidence, to help structure their deliberations about which alternatives are best.
Types of Health Care Decisions Supported by MCDA
Examples of current and potential applications of MCDA in health care decision making are summarized in Table 1 and detailed in this section. This is not an exhaustive list, and MCDA can be useful in many decision contexts. In addition to those mentioned in Table 1, MCDA has been used to develop disease classifications [
The criteria used differ between HTA systems, but might include effectiveness, patient need/burden of disease/severity. (Note: the role of cost, cost- effectiveness, and budget impact as criteria in MCDA is contentious
HTA aims to apply an agreed set of principles to make judgments about reimbursement of new technologies. Arguably, the same principles and criteria should be used across repeated decisions, to ensure consistency and accountability
Portfolio decision analysis (PDA)
Decisions made by life sciences companies, choosing where best to direct R&D efforts. See below for a pharmaceutical company’s experience
Resource allocation decisions made by local budget holders in the English NHS. Decisions made by private insurers about the bundle of services to reimburse. See below for details on English local budget holders’ experience with MCDA
The criteria used vary considerably, but might include effectiveness, meeting unmet need/equity objectives, meeting government targets, etc.
Committee in charge of making the funding decisions
These are “repeated” decisions, inasmuch as there is a single fixed budget, and the criteria used to prioritize any one service should also apply to decisions about other potential services, to ensure consistency and the achievement of allocative efficiency
Shared decision making (SDM)
Decisions made by patients, in discussion with their doctors, about choice of treatments. See below for an example of MCDA being used to assess cancer screening alternatives
For example, effect on life expectancy, quality of life, side effects from treatment, and the process of care
Patients and clinicians
One-off decisions as the relevant risks and benefits will differ from case to case and the patients will have different preferences. The criteria and their importance will therefore differ between decisions
Prioritizing patients’ access to health care
Prioritization of patients for health care services. See below for the use of “points systems” to prioritize patients awaiting elective surgery in New Zealand
Various measures of patient “need” and ability to benefit from treatment
Clinical leaders, patient groups, and other health professionals. Organ procurement organization at international, national, or regional level
These often use “points systems”—algorithmic approaches that apply identical criteria and rules across all cases to ensure fairness. These are repeated decisions coordinated by a central office with no direct relationship with patients or their physicians
DCE, discrete choice experiment; EMA, European Medicines Agency; EVIDEM, Evidence and Value: Impact on Decision Making; G-BA, Der Gemeinsame Bundesausschuss; IQWiG, Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen; MCDA, multiattribute decision analysis; NHS, National Health Service; NICE, National Institute for Health and Care Excellence; PBAC, Pharmaceutical Benefits Advisory Committee; PBMA, Programme Budgeting and Marginal Analysis; R&D, research and development.
These examples in Table 1 demonstrate the diverse range of decision problems and similarly diverse decision makers/organizations that MCDA can support. The criteria and stakeholders used in the MCDA depend on the type of decision problem. The outputs from the MCDA can be used to support different types of decisions, for example, to understand the value of the alternatives (e.g., understanding the value of treatment for purposes of reimbursement), to develop a ranking of alternatives (e.g., treatment alternatives for patients), or to allocate alternatives to a categorical outcome (e.g., “approve,” “deny,” or “coverage with evidence development” recommendations for new technologies). Also, the decisions can be one-off (e.g., patient choosing between treatment alternatives) or repeated (e.g., reimbursement decisions at HTA agencies).
Although various terms have been used to refer to the value judgments made during an MCDA—for instance, priorities, preferences, importance, and values—the terminology “preferences” is adopted in the task force reports. Also, the task force report refers to decision makers as those who make the choice between alternatives and stakeholders as the source of preferences. The task force acknowledges that the term “stakeholders” is used quite broadly in the health economics literature but within the MCDA literature, stakeholders are those providing the preferences and this terminology is used in the task force reports.
EMA’s Benefit-Risk Methodology Project
EMA’s Benefit-Risk Methodology Project developed and tested methods for balancing multiple benefits and risks, which can be used to inform regulatory decisions about medicinal products. The project was organized around five work packages (WPs). WP1 reported on the current practice of benefit-risk assessment in the European Union regulatory network, and WP2 examined the applicability of different methods for BRA.
WP3 performed the field testing of different methods in five European medicine regulatory agencies [
]. The eight-stage PrOACT-URL framework (Problem formulation, Objectives, Alternatives, Consequences, Trade-Offs, Uncertainties, Risk Attitude and Linked Decisions) was combined with MCDA to perform BRA. It highlighted the importance of the Effects Table (in essence, a “performance matrix”) and emphasized the value of the graphical MCDA displays.
WP4 synthesized the results of the earlier WPs to develop recommendations for undertaking BRA [
]. It suggested that a full MCDA model would be most useful for difficult or contentious cases, when the benefit-risk balance is marginal and could tip either way depending on judgements of the clinical relevance of the effects, favourable or unfavourable, and in the case of many conflicting attributes.
It also suggested that postapproval monitoring of the benefit-risk balance of a medicinal product in complex or marginal cases could be supported using quantitative MCDA modeling as the model can be updated with new information to see whether the benefit-risk balance has changed.
WP5 involved a 5-month pilot of the use of the “Effects Table” as a tool to summarize the key benefits and risks and to supplement the benefit-risk section of the Committee for Medicinal Products for Human Use assessment report [
Patient Involvement in HTA: MCDA Pilot by the German Institute for Quality and Efficiency in Health Care
In 2010, the German Institute for Quality and Efficiency in Health Care initiated a study to explore the use of MCDA methods as a means of incorporating patient involvement into its HTA process. This was because although patient involvement is widely acknowledged to be important in HTA and health care decision making, quantitative approaches to ascertain patients’ preferences for treatment end points are not yet established. The project used two MCDA techniques, the analytic hierarchy process (AHP) and discrete choice experiments (DCEs), as preference elicitation methods.
The AHP study included two workshops: one with 12 patients, and one with 7 health care professionals. In the workshops, the participants rated their preferences with respect to the importance of different end points of antidepressant treatment by a pairwise comparison of individual end points. These comparisons were then analyzed to generate the relative importance for each end point [
In the DCE study, patients and health care professionals were asked to choose between two (hypothetical) hepatitis C treatment alternatives that differed in their performance on various treatment characteristics (e.g., outcomes). These choices were analyzed using logistic regression models to estimate the importance of the individual treatment characteristics [
Both studies concluded that these types of techniques can be used to support the HTA process, but also highlight some methodological challenges that need addressing before their full-scale implementation.
Use of MCDA in HTA Decisions for Universal Coverage in Thailand
The National Health Security Office, the institute that manages Thailand’s universal coverage scheme, conducted a collaborative research and development project in the period 2009 to 2010 with two independent research institutes—the Health Intervention and Technology Assessment Program and the International Health Policy Program. The project used MCDA to guide the coverage decisions on including health interventions in the universal coverage scheme health benefit package in Thailand [
The process was carried out in four steps—nomination of interventions for assessment, selection of interventions for assessment using MCDA, technology assessment of interventions, and appraisal of interventions.
A consultation panel was established which worked with a large variety of stakeholders to identify potential interventions and a total of 17 interventions were nominated. The performance of these 17 interventions was assessed on five criteria—size of population affected by disease, effectiveness of health intervention, variation in practice, economic impact on household expenditure, and equity and social implications. After inspection and deliberation, nine interventions were selected for further quantitative assessment. The nine interventions were then assessed in terms of their value for money (incremental cost-effectiveness ratios) and budget impact. Decision makers qualitatively appraised the information on these nine interventions and deliberated to reach consensus on which interventions should be adopted in the package.
The study concluded that MCDA has the potential to contribute to a rational, transparent, and fair priority-setting process.
Implementation of European network for Health Technology Assessment (EUnetHTA) Core Model in Lombardy, Italy
Lombardy Region developed an HTA framework (named Valutazione delle Tecnologie Sanitarie [VTS] [
]. The VTS framework mapped EUnetHTA domains into dimensions that Lombardy Region set up to legitimize the prioritization of technologies and included criteria from the EVIDEM framework to support the systematic appraisal of the assessment report into a final decision.
The VTS framework is applied in a three-step process comprising 1) a prioritization of requests, grounded on a “quick and dirty” assessment limited to VTS dimensions; 2) a full assessment of the prioritized technologies, provided by answering EUnetHTA-based issues; and 3) an appraisal of the assessed technologies, grounded on the analysis of multiple criteria, using the EVIDEM framework.
The appraisal using the EVIDEM framework is based on 15 explicit quantitative criteria and 6 implicit qualitative criteria. The appraisal thus results in a quantitative result—that is, the score for 15 criteria—and 6 qualitative evaluations that are used to substantiate and legitimize the final decision and to communicate it to the public, the industry, the providers, and other relevant stakeholders.
The VTS framework has been used in Lombardy Region since 2011 to decide on the introduction and delisting of health technologies (including diagnostics, devices, interventional procedures, and drugs). The study concluded that the EUnetHTA assessment tools can be combined with multiple criteria appraisal methods to support HTA.
Portfolio Decision Analysis in a Pharmaceutical Company
MCDA modeling helped inform final portfolio decisions by Allergan by prioritizing the projects on the basis of their value for money [
A 2-day workshop was conducted to elicit stakeholders’ preferences for the different criteria and MCDA modeling was used to collapse multiple dimensions of benefit into a single risk-adjusted benefit. The benefit criteria used were net present value (financial value), medical need (extent to which the project will meet unmet medical need), business impact (protecting the existing business), future value (contribution to evolution to a specialty pharmaceutical company), and probability of success (probability that the benefits will be realized). Risk-adjusted benefit for each project was estimated by multiplying the benefits by the probability of realizing them.
The portfolio consisted of 52 projects structured within five areas of interest: front and back of eye, skin care, neurological, and new technologies. Data on the performance of the projects against the value criteria were estimated, and the projects were prioritized on the basis of their “risk-adjusted” benefit-to-cost ratio to maximize the total benefit that can be achieved from a given budget.
The study concluded that the MCDA process helps to increase communication across silos, to develop shared understanding of the portfolio as a whole, and the transparency also makes it easy to brief upwards, and provides an audit trail of the decision-making process.
Local Commissioning—A Local Health Care Planner in the English National Health Service
Isle of Wight Primary Care Trust used MCDA to support the allocation of resources across 21 interventions in five priority areas of respiratory, mental, and children’s health, cardiovascular disease, and cancer [
Key stakeholders engaged in the analysis were clinicians, council representatives, voluntary sector representatives, nurses, public and patients’ representatives, hospital managers, and the ambulance service. An impartial facilitator worked iteratively with key stakeholders to generate a formal, requisite model to assess options on multiple criteria using MCDA and to generate a summary benefit score. Interventions were assessed on three criteria: increased health (reduced mortality and increased quality of life), reduced health inequalities, and operational and political feasibility.
The research team collected data on the performance of these interventions on these criteria, and MCDA modeling was used to estimate the total value of each intervention. The information on value was combined with data on the cost to generate a priority list in which interventions were ranked according to value for money. Sensitivity analysis was used to explore the uncertainties and disagreements among participants.
The proposed investment plan based on the final priority list was approved by the Isle of Wight Primary Care Trust Board. The study concluded that MCDA has the potential to support local health planners in their task of allocating fixed budgets to a wide range of types of health care.
Shared Decision Making—Evaluating Cancer Screening Alternatives
Multiple screening options are available for people at average risk for colorectal cancer, and the US colorectal cancer screening guidelines recommend that screening decisions should reflect individual patient preferences. AHP, an MCDA technique, was used to elicit the decision priorities of people at average risk for colorectal cancer attending four primary care practices (Rochester, New York, Birmingham, and Indianapolis) in the United States [
On the basis of American guideline statements, the researchers identified six criteria: ability to prevent cancer, avoidance of side effects, minimizing false positives, and logistical complexity, further divided into three subcriteria: frequency of testing, preparation required, and method of testing procedure. Participants were asked to indicate the relative importance of two criteria on a scale of 1 to 9, where 1 indicates that the criteria are equally important and 9 indicates that one criterion is extremely important relative to the other. All comparisons were made on an interactive computer program developed in Microsoft Excel, which was then used to calculate the priorities assigned by the participants to the decision criteria and subcriteria.
Patients were also asked several questions about the feasibility of using AHP. A high proportion (92%–93% across the sites in which the study was undertaken) of the 484 participants indicated that it was not hard to understand the criteria; most found it easy to follow the pairwise comparison process (91%) and make the comparisons (85%). The majority (88%) stated that they would be willing to use a similar procedure to help make important health care decisions. Thus, the study concluded that patients are able and willing to perform such a complex AHP analysis and that it was possible to use these techniques to foster patient-centered decision making.
Prioritizing Patients’ Access to Elective Services
New Zealand’s Ministry of Health has worked with 1000Minds, an MCDA tool, to create new points systems based on a consensus of clinical judgments for prioritizing patients for access to elective services in New Zealand [
These points systems were developed using MCDA processes, supported by Internet‐based software, by a working group of clinical leaders for the elective service concerned, in consultation with patient groups and other clinicians.
The points systems consist of criteria for deciding patients’ relative priorities for treatment, in which each criterion is demarcated into two or more categories. Each category is worth a certain number of points intended to reflect both the relative importance of the criterion and its degree of achievement. Each patient is “scored” on the criteria and their corresponding point values summed to get a “total score,” by which patients are ranked (prioritized) relative to each other.
The process was initially applied to coronary artery bypass graft surgery and then successively to other elective services. The points systems for coronary artery bypass graft surgery, hip and knee replacements, cataract surgery, plastic surgery, otorhinolaryngology, and many others have been endorsed by their professional organizations (e.g., New Zealand Orthopaedic Association) and are in use throughout New Zealand.
Adunlin G, Diaby V, Xiao H. Application of multicriteria decision analysis in health care: a systematic review and bibliometric analysis [published online ahead of print October 18, 2014]. Health Expect. http://dx.doi.org/10.1111/hex.12287.
] recently published reviews of MCDA in health care. Most (56%) of the MCDAs reviewed by Marsh et al. were undertaken to support health care investment decisions, such as HTA and national and local coverage decisions. However, MCDAs were also identified supporting authorization (12%) and prescription decisions (22%), and the allocation of health research funding (2%). Adunlin et al. report that the most frequent use of MCDA was in the area of diagnostics and treatment (39%), but again a diversity of application areas was also identified, including formulary management, geographic information systems, HTA, medical automation, organ transplantation, pain management, performance measurement, priority setting, professional practice, public health and policy, resource planning, site selection, and supply chain.
A common finding between these articles is the diversity of decisions for which MCDA techniques are applied and the wide variety of MCDA methods used. The next section describes the MCDA modeling approaches available.
MCDA Modeling Approaches
MCDA approaches can be broadly classified into value measurement models, outranking models, and reference-level models [
Value measurement models involve constructing and comparing numerical scores (overall value) to identify the degree to which one decision alternative is preferred over another. They most frequently involve additive models (sometimes referred to as “weighted-sum” models, or “additive multiattribute value models”), which multiply a numerical score for each alternative on a given criterion by the relative weight for the criterion and then sum these weighted scores to get a “total score” for each alternative.
Outranking methods typically involve making pairwise comparison of alternatives on each criterion, which, in turn, are then combined to obtain a measure of support for each alternative being judged the top-ranked alternative overall. Outranking algorithms include the ELimination and Choice Expressing Reality (ELECTRE) family of methods [
As described in the second section, other MCDA methods not based on formal modeling also exist. At the most rudimentary level, the alternatives’ performance on criteria can simply be reported in a table, known as a “performance matrix” or “consequences table” [
]. This matrix/table can be used, in effect, as an aide-mémoire for decision makers’ deliberations focused on reaching a consensus ranking or choice of the “best” alternative (depending on the nature of the decision being made).
Value measurement approaches are by far the most widely used in health care [
], with outranking and goal programming approaches much less commonly used. However, which MCDA model is most appropriate depends on the objective of the analysis and the nature of decision makers’ preferences. There is often no simple answer to which approach is best suited to which decision problem. Those undertaking MCDA should always provide a clear rationale for opting for one approach over another, and ensure that it is compatible with the decision problem being addressed (see second task force report published in a later issue for further guidance). For example, value measurement approaches should be selected when decision makers consider criteria to be compensatory; that is, an improvement in one criterion can compensate for a worsening in another. Outranking methods may be useful if the goal is to identify a small subset of alternatives that fulfill a minimum requirement from a large set of alternatives (because developing a total value score using weighted-sum models for each alternative is not efficient).
An Overview of Steps in Conducting an MCDA
The focus of the task force reports is on the value measurement approaches because other techniques are rarely used in health care [
]. Although there are many differences in the ways in which these models are used and applied, there are several main elements of the process that are common among these methods. Broadly speaking, any value measurement modeling approach entails defining the decision problem being addressed, selecting the criteria, measuring alternatives’ performance, scoring alternatives and weighting criteria, aggregation, uncertainty analysis, and interpretation of results.
Several comprehensive descriptions of the steps involved in conducting MCDA have been published [
Weistroffer HR, Li Y. Multiple Criteria Decision Analysis software. In: Greco S, Ehrgott M, Figueira JR, eds., Multiple Criteria Decision Analysis: State of the Art Surveys (2nd ed.). Springer, USA, 2014.
]. An overview of the main steps involved in conducting an MCDA is presented in Table 2. It is important to emphasize that the steps can be performed in a different sequence and the process of undertaking an MCDA is iterative, rather than comprising a strictly sequential set of steps. Also, the performance matrix produced after the first three steps can be used to support deliberative decision making (i.e., “partial” MCDA without explicit weighting and scoring).
Table 2Steps in a value measurement MCDA process
Defining the decision problem
Identify objectives, type of decision, alternatives, stakeholders, and output required
Selecting and structuring criteria
Identify criteria relevant for evaluating alternatives
Gather data about the alternatives’ performance on the criteria and summarize this in a “performance matrix”
Elicit stakeholders’ preferences for changes within criteria
Elicit stakeholders’ preferences between criteria
Calculating aggregate scores
Use the alternatives’ scores on the criteria and the weights for the criteria to get “total value” by which the alternatives are ranked
Dealing with uncertainty
Perform uncertainty analysis to understand the level of robustness of the MCDA results
Reporting and examination of findings
Interpret the MCDA outputs, including uncertainty analysis, to support decision making
The purpose of this section is to identify the objectives of each of these eight steps and the methods alternatives available within each step. The second task force report will provide further guidance on selecting and implementing these methods. For illustrative purposes, a simple worked example of scoring, weighting, and aggregation in an additive model is provided in the Appendix.
Defining the Decision Problem
The starting point for any MCDA involves understanding and defining the decision problem and the corresponding decision goal. This also involves identifying the appropriate stakeholders, the alternatives under consideration, and the output required. As seen in Table 1, the stakeholders, depending on the application, may include patients, clinicians, payers, regulators, and the general population. In some applications, stakeholders might be acting on behalf of others (e.g., an HTA committee making decisions in the interests of the general population), whereas in other applications, the stakeholders may be the decision makers themselves (e.g., patients in shared decision making). The types of decision problems can vary (e.g., understanding the value of alternatives or to rank/categorize the alternatives, as seen in Table 1), and can include one-off problems (e.g., patient choosing between treatment alternatives) or repeated problems (e.g., reimbursement decisions at HTA agencies).
Selecting and Structuring Criteria
Once the decision problem has been identified, the next step is to identify and agree on the criteria by which the alternatives will be evaluated. For example, as observed in Table 1, authorization decisions may be informed exclusively by clinical outcomes, whereas prioritization decisions may incorporate a broader set of criteria. Criteria can be identified in a number of ways, from reviews of previous decisions to focus groups and facilitated workshops. The criteria used in an additive model should meet certain requirements such as completeness, nonredundancy, nonoverlap, and preferential independence (see the second task force report for further information on these requirements). Once the criteria have been identified, they can be structured using “value trees” [
], which decompose the overall value into criteria and subcriteria in a visual manner.
Once the criteria are agreed upon, the performance of the alternatives on each of the criteria is determined (e.g., gathering evidence that drug A leads to a mean overall survival of x months and the overall survival with drug B is y months). Data on the alternatives’ performance on each of the criteria can be gathered in various ways, ranging from standard evidence synthesis techniques (e.g., systematic reviews and meta-analysis) to elicitation of expert opinion in the absence of “harder” data. The alternatives’ performance on criteria can be reported in a table, known as a “performance matrix” [
]. As described earlier, this “performance matrix” can be used, in effect, as an aide-mémoire for decision makers’ deliberations without explicit scoring and weighting.
Following the analysis of the alternatives’ performance, stakeholders’ priorities or preferences for changes within criteria (scores) are captured. Scores are often derived by defining rules or functions for converting performance measurements into scores. In this instance, the scores differ from performance measures in two ways. First, scores are often used to translate performance measures using different units for each criterion onto a common scale, for instance, a 0 to 100 scale. Second, scores incorporate priorities or preferences for changes in performance within criterion, converting performance measures into scores such that the same change along the scoring scale (e.g., 10–20 or 60–70) is equally preferred.
Scoring elicitation methods can be broadly defined as “compositional” and “decompositional”; compositional methods look at each criterion separately and build up the overall value, whereas decompositional methods look at the overall value of alternatives as a whole, from which weights and scores for criteria are derived.
Compositional methods generate separate estimates of scores and weights, which are then combined in subsequent steps of the MCDA (step 6). In the Keeney and Raiffa MCDA approach, the scores are generated by tracing out the shape of the “value function” that relates alternatives’ performance on the criterion to their value to decision makers (e.g., using “bisection” and “difference” methods [
]—involve participants ranking two or more real or hypothetical alternatives defined on some or all of the criteria. For DCEs, weights and scores are derived from these rankings simultaneously using regression-based techniques, in the form of coefficients estimating how the preference for, or utility associated with, an alternative varied with changes in performance against each criteria. PAPRIKA uses quantitative methods based on linear programming to also estimate weights and scores simultaneously.
Weighting involves eliciting stakeholders’ preferences between criteria. Weights represent “trade-offs” between criteria and are used to combine the scores on individual criterion into a measure of “total value.” Weighting can be thought of as analogous to setting exchange rates—the scores on different criteria all represent value (e.g., as euros, US dollars, and UK pounds do)—but they are not commensurate and have to be made commensurate by applying weights (i.e., exchange rates).
In the Keeney and Raiffa MCDA approach, the weights are elicited using “swing weighting” by taking into account the ranges of performance relevant to a set of alternatives (i.e., the “swing” in performance). A number of other compositional methods have also been applied to generate weights, including direct rating (visual analogue scale, direct rating, and points allocation) [
Scoring and weighting is usually done by the stakeholders, who at times can also be decision makers. For example, a patient using an MCDA approach as a decision aid in choosing between treatment alternatives is the sole stakeholder/decision maker. In other applications, it is less clear-cut whose scores/weights should be used. For example, in HTA, criteria weights could come from committee members, from patients, or from the general public. The choice about whose preferences are relevant to a given decision problem is a normative one, and the outcome from the decision-making process may be sensitive to which and whose scores/weights are used.
Calculating Aggregate Scores
Value measurement models often use the additive model for computing aggregate results. For the compositional methods and the PAPRIKA (decompositional) method [
], each alternative’s scores on the criteria are multiplied by the weights and these weighted scores are then summed across the criteria to get a “total value” for each alternative. With conjoint analysis or a DCE method, the data on performance of the alternatives are input into the valuation function derived from regression analysis to estimate each alternative’s value (or “utility”) or its probability of being the preferred alternative. Other types of aggregative methods such as multiplicative models [
Uncertainty may affect both the design and evidence feeding into the assessment. All aspects of MCDA such as what criteria are selected, performance against those criteria, and whose views should inform the weighting/scoring of criteria are subject to uncertainty [
]. It is important to understand the impact that this uncertainty has on MCDA results, to evaluate the robustness of the decision outcomes. Parameter uncertainty (e.g., uncertainty in the performance of alternatives) can be addressed using techniques such as probabilistic sensitivity analysis techniques. Structural uncertainty (e.g., choice of criteria) can be addressed using scenario analyses; for example, different sets of criteria can be used to understand whether the results from the MCDA exercises differ. Heterogeneity in preferences among subgroups can be studied by using weights and scores obtained from different stakeholder groups in the MCDA model.
Interpretation and Reporting the Results
MCDA results can be presented in tabular or graphical form for decision makers. Aggregate value scores can be interpreted and used in different ways, that is, to rank the alternatives in order of importance or providing a measure of value for each of the alternatives. Alternatives’ total scores can also be combined with cost data to identify “value for money” of each alternative to allow portfolio or resource allocation decisions. Also, it is worthwhile emphasizing that MCDA is intended to serve as a tool to help decision makers reach a decision—their decision, not the tool’s decision. The decision makers/stakeholders can be presented with the MCDA model to allow them to explore the results for different scenarios. The exceptions are “algorithmic” MCDA frameworks such as points systems for prioritizing elective services [
]. These “algorithmic” frameworks are relatively rare in health care, and are used where the aim of the tools is to minimize the human factor, for either potential positive or negative discrimination, in making decisions.
MCDA can support decision making in health care. It improves transparency and consistency in decisions—and potentially, the accountability of public sector decision makers. It does not replace judgment, but rather identifies, collects and structures the information required by those making judgements to support the deliberative process. This report defines MCDA, provides an overview of the wide range of applications and describes the key steps involved. It will be helpful for those with little knowledge of MCDA. The second MCDA task force report goes into more depth providing guidance on how to choose the MCDA method as well as a good practice guidelines checklist and recommendations. It will be particularly useful for those designing and reviewing MCDA applications in health care.
The individual contribution by Paul Hansen is gratefully acknowledged. Many thanks to Jaime Caro, Mara Airoldi, Alec Morton, Mireille Goetghebeur, Janine van Til and Larry Phillips who contributed to the development of this manuscript. We thank the reviewers who commented during the forums we presented at two ISPOR meetings, and especially those below, who reviewed our drafts and submitted written comments. Their feedback has both improved the manuscript and made it an expert consensus ISPOR report.
Many thanks to Abdallah Abo-Taleb, Maria Agapova, Bhagwan Aggarwal, Aris Angelis, Segolene Ayme, Karl Claxton, Manie de Klerk, Karam Diaby, Jim Dolan, Thomas Ecker, Sonia Garcia Perez, Andreas Gerber-Grote, Salah Ghabri, Jean-Francois Grenier, Karin Groothuis-Oudshoorn, Brett Hauber, Nadine Hillock, Marjan Hummel, Ilya Ivlev, Tianze Jiao, Cheryl Kaltz, Panos Kanavos, Carlo Giacomo Leo, Himadri Malakar, Pierpaolo Mincarone, Axel Muhlbacher, Sandra Nestler-Parr, Theresa Ofili, Monica Olivera, Javier Parrondo Garcia, Oresta Piniazhko, Waranya Rattanavipapong, Brian Reddy, Carina Schey, Mark Sculpher, Sumitra Sri Bhashyam, Yot Teerwattananon, Tommi Tervonen, Fatema Turkistani, Mathieu Uhart and Rik Viergever.
We also would especially like to thank Elizabeth Molsen who project managed this task force, and Eden McConnell and Kelly Lenahan at ISPOR who assisted us through this long process.
Appendix. MCDA Modeling Example
We present here a stylized hypothetical example to demonstrate the Keeney and Raiffa MCDA approach, which involves scoring using “partial value functions” and “swing” weighting. For illustrative purposes, the decision problem is kept quite simple (with three criteria and two alternatives) and described in abstract terms (e.g., “alternative 1”). However, the same principles can be extended to any MCDA problem.
Let us assume that after completing the first three steps of MCDA, the performance matrix generated is as presented in Table A1. Note that aa, bb, and cc are the units used to measure criteria A, B, and C, respectively. Criteria A and C are measured in units where higher performance is better, whereas criterion B is measured in units where lower performance is better (e.g., frequency of adverse events). Organizing the evidence in this kind of performance matrix might in itself help decision makers weigh up multiple criteria, by highlighting the trade-offs that need to be made. For example, alternative 1 outperforms alternative 2 on criteria A and C, but alternative 2 outperforms alternative 1 on criterion B (because lower performance in criterion B is better). MCDA can help aggregate the performance in these criteria, measured in different units, into a single overall value for each alternative. To do so requires us to establish the preferences of decision makers within and between criteria, via scoring and weighting.
Scores on a criterion differ from performance measures on a criterion in two important ways. First, scores are often used to translate performance measures using different units for each criterion onto a common scale. Second, scores incorporate preferences for changes in performance, such that the same change along the scoring scale (e.g., 10–20 or 60–70) is equally preferred.
Scoring in this example is performed on the measurement scale for each criterion: 65–90 aa for criterion A, 5–10 bb for criterion B, and 0–1 cc for criterion C. It is assumed that these scales represent the feasible ranges for each criterion.
When the ranges of each criterion have been identified, partial value functions are developed to specify the relationship between changes along these ranges of performance and their value (i.e., the score that will be input into the MCDA, often defined on a scale of 0–100 points). There are two factors to consider when establishing the partial value functions: 1) whether the relationship between changes along these ranges of performance and the score is linear and 2) whether high performance is better (e.g., treatment effectiveness) or low performance is better (e.g., adverse events). If a linear function defines this relationship, the specification of the partial value function is straightforward, as shown in Fig. A1, Fig. A2. It is assumed that both criteria A and B have a linear partial value function, but higher performance in criterion A is better whereas lower performance in criterion B is better.
If this function is nonlinear, there are a number of methods that can support the elicitation of these functions from stakeholders [
]. For example, a partial value function for the 0–1 cc range of criterion C could be developed by answering the following question: “What level is x so positioned that a change from 0 cc (i.e., worst value with a score of 0 points) to x cc gives you as much value as a change from x cc to 1 cc (i.e., the most desirable value with a score of 100 points)?” This process can be iterated for several points in the scale to understand the shape of the value function. This method to derive partial value functions is known as the “bisection method.”
Figure A3 shows the nonlinear function describing the relationship between partial values scores and the performance on criterion C. It can be observed that there is steep increase in scores until up to performances of 0.3 cc after which the relationship slows down slightly. For example, the increase in the scores as a result of going from 0.55 cc to 0.63 cc is only 5 points, whereas the same increase in the performance earlier (from 0.15 cc to 0.23 cc) produces an increase of 10 points. This suggests that the preference for increases in performance in criterion C depends on the baseline measure of the performance, with a similar increase in performance valued more highly when the baseline performance is low rather than when the baseline performance is high.
The partial value scores on different criteria all represent value, but they have to be made commensurate into a measure of “total value,” which is performed by applying weights. Weighting involves eliciting stakeholders’ preferences between criteria as “trade-offs” and can be thought of as setting exchange rates (e.g., to combine €, $, and £ into a single overall value).
The first step in the swing weighting exercise is to identify and assign 100 points to the criterion with the swing (range of performance) that matters most. This is followed by a pairwise comparison between this criterion and each of the others to determine the relative importance of swings in criteria, and correspondingly allocate the points between 0 and 100. For example, if criterion B was assigned 100 points, we might ask stakeholders: “If a decrease of 10 bb to 5 bb in criterion B is given 100 points, on a scale of 0 to 100 how important is an improvement in the criterion A from 65 aa to 90 aa?” This process is then repeated for all remaining criteria to obtain an estimate of the relative value of the swing on each criterion, as shown in Figure A4.
Let us assume that after completing the weighting exercise, the points allocated for criteria A, B, and C are 60, 100, and 80. These are then normalized (i.e., dividing each criterion’s points by the sum of points) so that the weights add up to 1, resulting in weights of 0.25, 0.42, and 0.33, respectively.
After eliciting the scores and the weights, the aggregation is frequently performed using an additive model. For each alternative, this involves multiplying the scores of the alternative on the criterion with the weight of that criterion and summing across criteria. The scores, weights, and the total “value” of the two alternatives are presented in Table A2.
Table A2Aggregation to estimate the overall value of the alternatives
The total “value” of alternative 1 is 58.25 and that of alternative 2 is 55.55, which suggests that alternative 1 is preferable.
The results from MCDA should not be taken as the “final decision” but rather the MCDA model should be used to explore the uncertainty in the decision problem. The decision makers can be presented with results from analyses exploring different types of uncertainty (e.g., parameter uncertainty, structural uncertainty, and heterogeneity) to support decision making. A simple example of scenario analysis is presented here.
Let us assume that there is uncertainty about the performance of alternative 2 on criterion B and that another source suggests that this performance is 6 bb (instead of 6.5 bb). Using the partial value function for criterion B in Figure A2, it can be estimated that the score of alternative 2 on criterion B is now 80 (instead of 70). Using this score in Table A2 results in an overall value of 59.75, which suggests that alternative 2 is preferable. Threshold analysis can also be performed to determine at which point of performance alternative 2 becomes the preferred alternative.
MCDA is intended to serve as a tool to help decision makers reach a decision—their decision, not the tool’s decision. The decision makers can deliberate on which is the most appropriate evidence (and thus, the most appropriate score and the most appropriate “total value”) before making their final decision.
Priority setting of health interventions: the need for multi-criteria decision analysis.
Adunlin G, Diaby V, Xiao H. Application of multicriteria decision analysis in health care: a systematic review and bibliometric analysis [published online ahead of print October 18, 2014]. Health Expect. http://dx.doi.org/10.1111/hex.12287.
Weistroffer HR, Li Y. Multiple Criteria Decision Analysis software. In: Greco S, Ehrgott M, Figueira JR, eds., Multiple Criteria Decision Analysis: State of the Art Surveys (2nd ed.). Springer, USA, 2014.