Methodology| Volume 23, ISSUE 8, P1040-1048, August 01, 2020

# ICER Value Framework 2020 Update: Recommendations on the Aggregation of Benefits and Contextual Considerations

Open ArchivePublished:August 11, 2020

## Highlights

• Multiple-criteria decision analysis (MCDA) has emerged as a methodology for value-based assessment in healthcare evaluation and health technology assessment (HTA) with a number of conceptual, methodological, and empirical studies in recent years.
• Existing evidence from the cognitive psychology literature on whether or not the use of an algorithm is a necessary step for the combination of multiple benefits and their value trade-offs, or whether it can be left to decision makers’ own capabilities, suggests that the human brain has a restricted “integrator” capacity, focusing on a limited number of effects.
• An evidence-based response is provided to a number of claims by the Institute for Clinical and Economic Review (ICER) in the United States with regards to the use of MCDA methods and stakeholder engagement. These responses relate to aspects of methodological robustness and reliability, complication of use, validation and consensus, and judgment transparency.
• Quantitative decision analysis approaches such as MCDA offer an integration of all relevant benefits and risks for a decision problem, including their value trade-offs, into an overall value function that provides increased scope and depth of the clinically relevant evidence.
• Value preferences can be constructed via decision conferencing, for which hundreds of successful decision analysis applications exist worldwide.

## Abstract

The Institute for Clinical and Economic Review (ICER) in the United States recently published a 2020 update to its value assessment framework. We are commenting on the method by which the benefits of health interventions are integrated, relating to contextual considerations and other factors relevant to an intervention’s value. We start by discussing the theoretical foundations of decision analysis and its extension to multiple criteria decision analysis (MCDA). Then we provide a detailed, evidence-based response to some of the claims made by ICER with regard to the use of MCDA methods and stakeholder engagement. Finally, we provide a number of recommendations on the use of quantitative decision analysis and decision conferencing that could be of relevance to the ICER methodology. Overall, we agree that some of the proposed changes by ICER are moving in the right direction toward improving transparency in the value assessment process, but these changes are probably inadequate. We advocate that more serious attention should be paid to the use of quantitative decision analysis together with decision conferencing for the construction of value preferences via group processes for the integration of an intervention’s various benefit components.

## Introduction

Since 2006, the Institute for Clinical and Economic Review (ICER) in the United States has been evaluating the clinical and economic value of innovative health interventions and has been making pricing recommendations with the goal of achieving “sustainable access to high value care for all patients.”
ICER
A guide to ICER’s methods for health technology assessment.
The methodological framework it has developed focuses on the assessment of interventions’ long-term value for money and short-term affordability.
ICER
A guide to ICER’s methods for health technology assessment.
In summary, the ICER value framework adopts a clear conceptual structure, as part of which an intervention’s long-term value for money is measured through its incremental cost-effectiveness, with deliberative adjustments made possible for the existence of other benefits and contextual considerations via a majority voting process between members of its appraisal committee. More precisely, in terms of other benefits and contextual considerations, the appraisal committee members are asked to first vote individually on the existence or not of any other such benefits or disadvantages and contextual considerations, before being asked to reflect on the voting results as part of the final voting on the intervention's long-term value for money. Complementary to the intervention’s long-term value for money, its short-term affordability is accounted for via the potential budget impact at health system level; if this is in excess of a national threshold (tied to the growth of the overall US economy), it acts as a signal for potential access challenges and possibly as a triggering mechanism for policy discussions and negotiation with the manufacturer.
Similar to the public engagement approach adopted by other healthcare evaluation agencies when seeking to revise their methods, ICER recently invited all interested parties to submit input for the 2020 update of its value assessment framework as part of an open consultation. The fourth point of the consultation invitation focused on possible methods by which intervention benefits could be integrated, effectively relating to contextual considerations and other factors relevant when assessing an intervention’s value, and possible ways of making them more explicit. Past experience and literature suggest that although similar types of evidence are being assessed during healthcare evaluations in different countries, the specific evaluation criteria used and the way they are incorporated vary, with their relative importance remaining largely unknown.
• Angelis A.
• Lange A.
• Kanavos P.
Using health technology assessment to assess the value of new medicines: results of a systematic review and expert consultation across eight European countries.
For example, a number of social value elements relating to the burden of disease, innovation level, socioeconomic impact, and other types of evidence are considered in the majority of cases; however, this usually takes place on an ad-hoc basis and lacks transparency regarding the role and impact of the various evidence components as part of a deliberative decision-making process.
• Angelis A.
• Lange A.
• Kanavos P.
Using health technology assessment to assess the value of new medicines: results of a systematic review and expert consultation across eight European countries.
Such differences in evaluation processes could partly explain the considerable variation observed in drug coverage decisions across countries,
• Clement F.M.
• Harris A.
• Li J.J.
• et al.
Using effectiveness and cost-effectiveness to make drug coverage decisions: a comparison of Britain, Australia, and Canada.
with evidence suggesting the existence of poor to moderate agreement in health technology assessment (HTA) recommendations.
• Nicod E.
• Kanavos P.
Commonalities and differences in HTA outcomes: a comparative analysis of five countries and implications for coverage decisions.
Any significant intercountry variability in the evaluation processes could have implications on efficiency and fairness that could risk the reasonableness and credibility of the decisions
• Daniels N.
• Sabin J.
The ethics of accountability in managed care reform.
,
• Daniels N.
; as such, any inconsistency deserves careful attention. In this article and in response to the ICER public consultation, we propose an improved way to incorporate benefits and other considerations in an effort to inform future updates of the value assessment framework.
A range of different methods exist for the purpose of assessing the value of new health technologies as part of healthcare evaluation, an interdisciplinary scientific field also known as HTA. In current HTA practices, a number of value assessment approaches are used across different jurisdictions, which could be broadly divided into (1) comparative clinical benefit assessments, (2) economic evaluations, and (3) value-based assessments. We perceive the ICER methodology to pertain to the “value-based assessment” group of approaches, having as its mission a desire to serve the needs of decision makers interested in measuring the benefits of interventions that go beyond the clinical value added and cost-effectiveness, aiming to capture other aspects of value as part of a more comprehensive approach. This group of approaches is not yet characterized by a single or specific type of methodology, but most approaches have emerged because of the inadequacy of economic evaluations to capture the multidimensional nature of new medicines’ value in a structured and consistent way. Given that economic evaluation is founded upon the utility-maximizing behavior of a single individual, a uniform cost-effectiveness criterion applied at a heterogeneous population level is unlikely to yield Pareto-optimal resource allocations.
• Garber A.M.
• Phelps C.E.
Economic foundations of cost-effectiveness analysis.
The grounding of cost-effectiveness analysis in von Neumann-Morgenstern’s utility theory depends on a set of restrictive utility assumptions, including that the quality-adjusted life-year (QALY), a measure of health benefit that accounts for both length of life and quality of life, adequately represents preferences; however, this is not always the case.
• Garber A.M.
• Phelps C.E.
Economic foundations of cost-effectiveness analysis.
,
• Pliskin J.
• Separd D.
• Weinstein M.
Utility functions for life years and health status.
This idea has important implications for the incorporation of utilities at population level, such as equity, fairness, and a range of other aspects recently identified by a special International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Task Force Report.
• Lakdawalla D.N.
• Doshi J.A.
• Garrison L.P.
• et al.
Defining elements of value in health care—a health economics approach: an ISPOR Special Task Force Report [3].
Examples of such value-based approach efforts include the tiered incremental cost-effectiveness ratio threshold in England, which is based on additional value concerns relating to end-of-life criteria (that can effectively increase the valuation of a unit of health outcome for terminal illnesses) and ultra-rare conditions,
NICE
Guide to the Methods of Technology Appraisal 2013.
NICE
Developing NICE Guidelines: the Manual.
NICE
Interim Process and Methods of the Highly Specialised Technologies Programme Updated to Reflect 2017 Changes.
or the fluctuating threshold in Sweden, which is based on disease severity or need.
• Angelis A.
• Lange A.
• Kanavos P.
Using health technology assessment to assess the value of new medicines: results of a systematic review and expert consultation across eight European countries.
,
• Willis M.
• Ödegaard K.
A case study of ex ante , value- based price and reimbursement decision- making: TLV and rimonabant in Sweden.
,
• Golan O.
• Hansen P.
• Kaplan G.
• et al.
Health technology prioritization: Which criteria for prioritizing new technologies and what are their relative weights?.
In response to the desire to improve consistency in the consideration of additional benefits and the transparency of their influence on drug coverage decisions, multiple criteria decision analysis (MCDA) has emerged as a potential methodology for value-based assessment in HTA.
• Devlin N.
• Sussex J.
Incorporating multiple criteria in HTA: methods and processes.
• Thokala P.
• Duenas A.
Multiple criteria decision analysis for health technology assessment.
• Kanavos P.
• Angelis A.
Multiple criteria decision analysis for value based assessment of new medical technologies: a conceptual framework. The LSE Health Working Paper Series in Health Policy and Economics.
• Marsh K.
• Lanitis T.
• Neasham D.
• et al.
Assessing the value of healthcare interventions using multi-criteria decision analysis: a review of the literature.
• Angelis A.
• Kanavos P.
Multiple criteria decision analysis (MCDA) for evaluating new medicines in health technology assessment and beyond: the advance value framework.
Besides the generation of guidelines for the conduct of MCDA in HTA,
• Thokala P.
• Devlin N.
• Marsh K.
• et al.
Multiple criteria decision analysis for health care decision making--an introduction: Report 1 of the ISPOR MCDA Emerging Good Practices Task Force.
• Marsh K.
• Ijzerman M.
• Thokala P.
• et al.
Multiple criteria decision analysis for health care decision making-Emerging Good Practices: Report 2 of the ISPOR MCDA Emerging Good Practices Task Force.
a number of empirical studies have been conducted in collaboration with decision makers to test and advance these methods in practice.
• Lettieri E.
• Masella C.
• et al.
Implementation of EUnetHTA core Model® in Lombardia: the VTS framework.
• Tony M.
• Wagner M.
• Khoury H.
• et al.
Bridging health technology assessment (HTA) with multicriteria decision analyses (MCDA): field testing of the EVIDEM framework for coverage decisions by a public payer in Canada.
• Jaramillo H.E.C.
• Goetghebeur M.
• Moreno-Mattar O.
Testing multi-criteria decision analysis for more transparent resource-allocation decision making in Colombia.
• Angelis A.
Evaluating the benefits of new drugs in health technology assessment using multiple criteria decision analysis: a case study on metastatic prostate cancer with the dental and pharmaceuticals benefits agency (TLV) in Sweden.
• Angelis A.
• Linch M.
• Montibeller G.
• et al.
Multiple criteria decision analysis for HTA across four EU member states: piloting the Advance Value Framework.

## Decision Analysis and MCDA

We believe that the potential usefulness of decision analytic approaches for measuring the overall value of health interventions, including aggregating different benefit components, has not been fully appreciated by ICER. Despite evolving from expected utility theory, these approaches represent an alternative way of thinking to economic evaluation. Perhaps the most relevant theoretical framework to value measurement relates to decision theory, with decision analysis, its applied discipline, acting as the practical instrument of analysis.
Howard Raiffa first defined the spirit of decision analysis as “divide and conquer: decompose a complex problem into simpler problems, get one’s thinking straight on these simpler problems, paste these analyses together with logical glue, and come out with a program of action for the complex problem” (page 271).
• Raiffa H.
Decision Analysis: Introductory Lectures on Choices Under Uncertainty.
Different decision analysis approaches exist, which could be broadly divided into qualitative and quantitative types; all approaches contain the definition of objectives and criteria, the identification of alternatives and options, the collection of data and evidence, and the elicitation of consequences and preferences.
• Angelis A.
• Phillips L.D.
Advancing structured decision-making in drug regulation at the FDA and EMA.
Quantitative approaches, however, move beyond this to quantify values (or utilities), trade-offs, and uncertainty, and aggregate all components together using an algorithm, which can be as simple as a weighted average. The extension of decision analysis applications to include decision problems with multiple objectives led to the foundations of MCDA,
• Keeney R.
• Raiffa H.
Decisions with Multiple Objectives: Preferences and Value Trade-Offs.
effectively a fully quantitative decision analysis approach. The development of multiattribute utility theory from a utility framework is also well illustrated elsewhere.
• Ralph L.K.
Multiplicative utility functions.
The process of MCDA in the context of HTA could be summarized in the phases of problem structuring, model building, model assessment, model appraisal, and the development of action plans.
• Angelis A.
• Kanavos P.
Value-based assessment of new medical technologies: towards a robust methodological framework for the application of multiple criteria decision analysis in the context of health technology assessment.
In turn, these phases include the steps of selecting and structuring criteria, measuring the performance of alternative options, scoring the options, weighting the criteria, calculating weighted aggregate scores, dealing with uncertainty, and examining the findings.
• Thokala P.
• Devlin N.
• Marsh K.
• et al.
Multiple criteria decision analysis for health care decision making--an introduction: Report 1 of the ISPOR MCDA Emerging Good Practices Task Force.
Evidence on whether or not the use of an algorithm is a necessary step for the aggregation of values, trade-offs, and uncertainty (eg, as part of MCDA), or whether it can be left to decision makers’ own capabilities have existed in the cognitive psychology literature since the 1950s. In surveying the literature on the construction of complex values, Hastie and Dawes conclude that “the process of looking first within each attribute and then comparing across by some weighting system is superior to that of making global intuitive judgements across attributes.”
• Dawes R.M.
• Hastie R.
Rational Choice in an Uncertain World.
The superiority is due in part to a limitation that Miller had noted earlier, namely that a human brain can at one time only retain 5 to 9 pieces of information.
• Miller G.A.
The magical number seven, plus or minus two: some limits on our capacity for processing information.
In the behavioral economics literature over the last couple of decades, Thaler and Sunstein raised questions about the rationality of human judgments and decisions because of a number of biases and heuristics,
• Thaler R.H.
Nudge: Improving decisions About health, wealth, and happiness.
and, subsequently, Kahneman more thoroughly acknowledged that the human integrator has restricted capacity (or that the human brain lacks such an “integrator” altogether).
• Kahneman D.
Thinking, Fast and Slow.
More recently, Montibeller and Winterfeldt described the focus on a small number of effects because of limited mental capacity as “myopic problem representation,” explaining in detail the 12 cognitive biases and 14 motivational biases that can affect decision makers and experts when making judgments in decision and risk analyses.
• Montibeller G.
• Winterfeldt D.
Cognitive and motivational biases in decision and risk analysis.
Although the biases focus mainly on modeling uncertainty and values, they also include qualitative considerations associated with problem structuring. For all 26 biases, they include guidance about debiasing techniques that can eliminate or reduce them.

## Response to the ICER Value Framework

Although we are not aware of the type of MCDA method(s) considered and tested by ICER, together with its independent committees, for weighting individual elements, we consider that dismissing these methods altogether on the ground that they are not “robust enough to add to reliability of value judgements” (pages 19-20)2 to be exaggerated. A relevant question to ICER’s rejection of MCDA methods would be, “Not robust compared with what?” As outlined below, there is an increasing accumulation of knowledge that the current process is flawed. At what point in time would the testing of other methods be acceptable?
There have been many successful applications of decision analysis methods across a number of areas, including in drug evaluation
• Angelis A.
Evaluating the benefits of new drugs in health technology assessment using multiple criteria decision analysis: a case study on metastatic prostate cancer with the dental and pharmaceuticals benefits agency (TLV) in Sweden.
,
• Nutt D.J.
• King L.A.
• Phillips L.D.
Drug harms in the UK: a multicriteria decision analysis.
• Angelis A.
• Montibeller G.
• Hochhauser D.
• et al.
Multiple criteria decision analysis in the context of health technology assessment: a simulation exercise on metastatic colorectal cancer with multiple stakeholders in the English setting.
• Felli J.C.
• Noel R.A.
• Cavazzoni P.A.
A multiattribute model for evaluating the benefit-risk profiles of treatment alternatives.
• Marsh K.
• Zaiser E.
• Orfanos P.
• et al.
Evaluation of COPD treatments: a multicriteria decision analysis of aclidinium and tiotropium in the United States.
• Moore A.
• Crossley A.
• Ng B.
• et al.
Use of multicriteria decision analysis for assessing the benefit and risk of over-the-counter analgesics.
• Nixon R.
A case study using the PrOACT-URL and BRAT frameworks for structured benefit risk assessment.
and other nonhealth application contexts.
• Keeney R.L.
• von Winterfeldt D.
Managing nuclear waste from power plants.
• Phillips L.D.
The Royal Navy's type 45 story: a case study.
• Gregory R.
Using Stakeholder Values to Make Smarter Environmental Decisions.
• Bana e Costa C.A.
• Antão da Silva P.
• Nunes Correia F.
Multicriteria evaluation of flood control measures: the case of Ribeira do Livramento.
• French S.
• Kelly N.
• Morrey M.
Decision conferencing and the international Chernobyl project.
An insightful MCDA application is SMART Vaccines: Strategic Multi-Attribute Ranking Tool for Vaccines, developed by the Institute of Medicine in collaboration with the National Academy of Engineering.
SMART Vaccines: Strategic Multi-Attribute Ranking Tool for Vaccines. The National Academies Press.
From the initial conceptual demonstration of a prioritization framework to the final empirical application of a software tool for prioritizing new vaccines for development, this is a valuable open-access resource providing interested users with a hands-on experience with the concept and use of quantitative decision analysis modeling.
Another relevant MCDA application topic in the context of guiding the development of new health interventions would be along their clinical phases, using information from target product profiles (TPPs). A TPP, which provides a summary format for a new health intervention in development (eg, drug or vaccine), is described in terms of labeling concepts acting as the goal for the development program and is updated dynamically over time.
FDA
Guidance for industry and review staff target product profile — a strategic development process tool.
For example, the World Health Organization (WHO) TPPs provide sets of product attributes reflecting preferred and minimal product characteristics for specific disease areas, acting as benchmarks for product development by manufacturers.
World Health Organisation
WHO Target Product Profiles (TPPs).
Such information could be used for MCDA model building and assessment to construct value scales and score the performance of options against the attributes of interest, followed by their weighting. Therefore, in such contexts, well-designed MCDA models could act as mechanisms for guiding the development of future innovations in alignment with what decision makers value.
This is not to say that the application of MCDA in drug evaluation or HTA comes without challenges or limitations.
• Morton A.
Treacle and smallpox: two tests for multicriteria decision analysis models in health technology assessment.
• Marsh K.
• Sculpher M.
• Caro J.J.
• et al.
The Use of MCDA in HTA: great potential, but more effort needed.
• Marsh K.
• Thokala P.
• Youngkong S.
• et al.
Incorporating MCDA into HTA: challenges and potential solutions, with a focus on lower income settings.
It might be more constructive, however, to view any robustness concerns of MCDA studies in alignment with their compliance to good methodological practice,
• Phillips L.D.
Best Practice for MCDA in healthcare.
as something which has been shown to be poor in practice,
• Oliveira M.
• Mataloto I.
• Kanavos P.
Multi-criteria decision analysis for health technology assessment: addressing methodological challenges to improve the state of the art.
rather than prematurely dismissing this group of methods altogether, leaving no opportunity for their appropriate development and effective application.
Unavoidably, whether these methods are judged to be “too complicated for reliable use” (page 20) or not, will depend on the knowledge, expertise, and experience of the people facilitating the overall process, especially during the stage of building a model of values by elicitating value preferences and their trade-offs. With regard to the claimed complexity and time required to build a fully quantitative MCDA model, the Benefit-Risk Project of the European Medicines Agency (EMA) created MCDA models in decision conferences for 5 new drugs, then under review by the Committee for Human Medicinal Products (CHMP), and easily created each benefit-risk balance model within just 6 hours.
EMA
Benefit-risk methodology project. Work package 3 report: Field tests.
In terms of their reliability, 3 separate studies modeling the harms of drugs with different groups of experts produced similar results with high correlations proving a high degree of reliability and accuracy.
• van Amsterdam J.
• Nutt D.
• Phillips L.
• et al.
European rating of drug harms.
Another early HTA study with 2 rounds of preference elicitation engaging participants from 3 different countries resulted in virtually identical results.

Angelis A, Thursz M, Ratziu V, et al. Early health technology assessment during nonalcoholic steatohepatitis drug development: a two-round, cross-country, multicriteria decision analysis. Medical Decision Making 940672. (In Press)

In any case, for some relatively easier decision problems the use of qualitative decision analysis (or other) methods may be adequate. Conditions for the choice of quantitative over qualitative approaches could be based on decision importance and analysis complexity, such as, for example, the severity of the disease indication, the unmet clinical need, the number of outcomes being assessed, the type of trade-offs to be valued, or the performance of the treatments.
Similarly, in response to any concerns “that there are no validated or consensus methods to integrate these factors into overall judgements of value,” quantitative decision analysis methods are probably the most validated methods known for carrying out such integration of partial components of value judgments and deriving an overall value function, as evident through the many theoretical and empirical applications in Keeney’s landmark book, Value Focused Thinking.
• Keeney R.
Value Focused Thinking: A Path to Creative Decision Making.
In his book, Keeney explains how the overall value of an option is derived from the extent to which an objective or a number of objectives are judged to add value, and the elicitation of trade-offs that provide a common measure of added value. The implication for any drug evaluation process, including HTA, is that although clinical evidence on drug performance for efficacy, safety, and quality is based on objective evidence, subjective judgments are always needed for a number of context- and evidence-related concerns; for example, the appropriateness of the data for the intended disease indication, the clinical meaningfulness of the data, or the relative clinical relevance of different benefits and risks.
• Angelis A.
• Phillips L.D.
Advancing structured decision-making in drug regulation at the FDA and EMA.
It is in regard to this subjective interpretation of objective data that the use of quantitative decision analysis methods can be of great significance, but also in regard to the overall valuation of evidence, spanning both performance and trade-offs; this is because they can accommodate these aspects in a structured and transparent way instead of leaving them to become randomly incorporated through ad hoc and vague efforts.
Furthermore, although the establishment of cost-effectiveness thresholds, and therefore any “value-based prices,” are associated with a number of theoretical and practical challenges (if not limitations),
• Phillips L.D.
• Bana e Costa C.A.
Transparent prioritisation, budgeting and resource allocation with multi-criteria decision analysis and decision conferencing.
,
• Claxton K.
• Martin S.
• Soares M.
• et al.
Methods for the estimation of the National Institute for Health and care excellence cost-effectiveness threshold.
we are glad to see that one of the ultimate aims of ICER is to “engage all stakeholders in a shared process of learning” to “offer a transparent, reliable approach” (page 23) for the integration of benefits. In this regard, the social psychology literature could be very insightful. In terms of learning processes, it should become clear that preferences do not just “sit in our heads” waiting to be extracted, but they need to become constructed in a process of value measurement as part of which “added value” is always a matter of judgment. Construction of preferences can be facilitated through group elicitation processes and it could be argued that “many heads are better than one,” as it has been illustrated through an experiment on probability distributions obtained from individual versus group consensus.
• Phillips L.D.
Group elicitation of probability distributions: are many heads better than one?.
Because of a number of problems relating to interaction processes and cognitive processing, interacting groups (process techniques) might fail to generate judgments as accurate as those of their most capable members, but a combination of group facilitation with judgment analysis and information technology can significantly improve the performance of a group’s interaction.
• Reagan-Cirincione P.
Improving the accuracy of group judgment: a process intervention combining group facilitation, social judgment analysis, and information technology.
Nevertheless, it should be noted that when making judgments in groups, decision makers and experts are affected by group-level biases.
• Montibeller G.
• von Winterfeldt D.
Individual and group biases in value and uncertainty judgments.
As with the case of individual judgments, group judgments are subject to several biases, with their relative magnitudes depending on factors such as group size, initial individual judgment, magnitude of bias among individuals, and the group-judgment process adopted.
• Kerr N.L.
• Maccoun R.J.
• Kramer G.P.
Bias in judgment: comparing individuals and groups.

## Recommendations on the Use of Quantitative Decision Analysis and Decision Conferencing

The ICER value framework adopts a clear conceptual structure and a well-defined set of benefits, with an incremental cost utility ratio (ie, incremental cost per QALY gained) acting as the key evaluation metric. Until recently, the appraisal committee members followed a deliberative voting process for the aggregation of any “other benefits or disadvantages” and “contextual considerations”: there was an initial voting by each member on the existence of any such benefits and considerations, followed by a reflection of the voting results, before a final voting on the interventions’ long-term value for money (Fig. 1A ). The explicit consideration of well-specified “other benefits or disadvantages” (n = 7) and “contextual considerations” (n = 6) is unavoidably a noteworthy feature for improving the transparency of what influences an intervention’s long-term value for money.
Following the submission and review of public input to the consultation, ICER published a summary of proposed changes to its 2020 value assessment framework update, rejecting the adoption of any formal multicriteria decision analytic approach. Instead, it was proposed to retain a “modified approach to integrating other factors into deliberation and decision-making,” (page 32), using an expanded set of potential other benefits and contextual considerations for which independent evidence appraisal committee members could vote using a 3-level Likert-scale. After noting the proposed changes, ICER published the final 2020 updates, as part of which a revised list of other benefits and contextual considerations was considered (n = 9), using the new 3-level Likert-scale voting format ranging from “lower value” to “intermediate value” to “higher value” (see Fig. 1B).
ICER
2020 Value Assessment Framework: Final Framework.
The goal of this adaptation was to “provide the appraisal committees with a clearer understanding of the end of the spectrum within which they are expected to vote,” having also the intention “to produce a more transparent record of how the appraisal committee feels that these considerations should be applied when integrated with the cost-effectiveness results in making decisions about pricing” (page 33).
ICER
2020 Value Assessment Framework: Final Framework.
Nevertheless, it is not entirely clear how the various value levels will inform decision making; for example, will the lower value levels be used to justify no value increment or to justify value decrement, in the form of unchanged or decreased health-benefit (ie, ex value-based) price benchmarks, respectively? Consider the case of the lower level of “uncertainty in model assumptions” (creating significant risk that the best-case cost-effectiveness estimates are too optimistic), which could be used as a decrement of value. The same might not hold true, however, for the lower value level of “mechanism of action” (being similar compared with other active treatments), which could be used to justify a lack of value increment rather than a decrement.
We agree that the Likert-scale format change is a step in the right direction toward improving the transparency that different factors exert on decision makers’ judgments; but it is still inadequate, because transparency requires access to both the underlying performance data and the model structure, in which case it relates both to valuation of performance and valuation of trade-offs. The very last stage of “human integration” required for aggregating together all these additional components into the intervention’s core cost-utility ratio to derive its long-term value for money lacks transparency and, more importantly, is prone to failure and susceptible to bias, mainly because of the limited human mental capacity to support such complex tasks, as evident in the behavioral and decision science literature.
• Dawes R.M.
• Hastie R.
Rational Choice in an Uncertain World.
,
• Miller G.A.
The magical number seven, plus or minus two: some limits on our capacity for processing information.
,
• Kahneman D.
Thinking, Fast and Slow.
,
• Montibeller G.
• Winterfeldt D.
Cognitive and motivational biases in decision and risk analysis.
,
• Meehl P.E.
Clinical Versus Statistical Prediction: A Theoretical Analysis and a Review of the Evidence.
Incorporating additional factors without a clear conceptual framework for the expression of preferences and judgments can degrade the validity of rankings, as has already been demonstrated elsewhere.
• Montibeller G.
• Patel P.
• Del Rio Vilas V.J.
A critical analysis of multi-criteria models for the prioritisation of health threats.
Among the most important features of quantitative decision analysis approaches and MCDA is the encompassing integration of all relevant benefits for a decision problem, and their value trade-offs, into an overall value function. Recently, the ISPOR Special Task Force on US Value Assessment Frameworks reviewed the use of MCDA for the aggregation of benefits into a single value metric and recommended further research on its use through testing in real-life decision settings because it might provide the best opportunity for improvement.
• Phelps C.E.
• Lakdawalla D.N.
• Basu A.
• et al.
Approaches to aggregation and decision making—a health economics approach: an ISPOR Special Task Force Report [5].
In the public policy domain, HM Treasury in the UK recently updated the content of The Green Book regarding guidelines on the appraisal and evaluation of central government projects and programmes to recommend the use of MCDA for estimating the value of social benefits (and costs), including for the consideration of trade-offs.
Value preferences could be constructed via decision conferencing, defined as “a gathering of key players who wish to resolve important issues facing their organization, assisted by an impartial facilitator, using a model of relevant data and judgements created on-the-spot to assist the group in thinking more clearly about the issues” (page 54).
• Phillips L.D.
• Bana e Costa C.A.
Transparent prioritisation, budgeting and resource allocation with multi-criteria decision analysis and decision conferencing.
Typical stages of decision conference workshops include exploring the issues, structuring and building the model, exploring the model, and agreeing on the way forward, all of which can be in alignment with requisite modeling: a decision model whose form and content are sufficient to solve a particular problem.
• Phillips L.
A theory of requisite decision models.
Among the requirements for constructive decision conferencing processes are ensuring that a diversity of perspectives is represented and that a feeling of “cohesiveness” exists between participants as part of which different opinions are being heard in a trusted manner; group numbers of between 7 and 15 participants have shown to be ideal because they are small enough to allow participants to reach an agreement but sufficiently large to represent all perspectives and interests.
• Phillips L.
• Phillips M.
Faciliated work groups: theory and practice.
Together with appropriate facilitation from a decision analysis expert, this can engage participants in constructive discourse and peer review, thus enabling the group outputs to be better than those of the best individual in the group.
• Reagan-Cirincione P.
Improving the accuracy of group judgment: a process intervention combining group facilitation, social judgment analysis, and information technology.
Hundreds of successful decision analysis applications exist worldwide for which preferences have been constructed using decision conferencing.
• Dodgson J.
• Spackman M.
• Pearman A.
• et al.
Multi-Criteria Analysis: A Manual.
,
• Phillips L.
Decision conferencing.
A recent MCDA pilot study in collaboration with European HTA bodies revealed how the application of quantitative decision analysis in combination with decision conferencing could take place by adopting a clean slate approach without the use of economic evaluations.
• Angelis A.
• Linch M.
• Montibeller G.
• et al.
Multiple criteria decision analysis for HTA across four EU member states: piloting the Advance Value Framework.
A number of evaluation criteria were incorporated in each jurisdiction's value model and decision-maker value preferences were elicited during the decision conferences, including performance scores and criteria weights. Following the estimation of the drugs’ overall benefits, a subsequent consideration of drug costs enabled the demonstration of value for money in the form of estimated “cost per unit of value” ratios. Although findings revealed some differences in value preferences across countries, drug rankings remained consistent. Importantly, the study demonstrated how MCDA could act as a decision support tool for HTA, because of the transparency in the construction of value preferences in a collaborative manner.
In the context of HTA decision making, assuming that an agreement has been reached regarding the level of the cost-effectiveness threshold(s), adjusting the threshold or the interventions’ incremental cost utility ratio to accommodate other benefits and contextual considerations should be possible. For example, as part of an incremental MCDA approach to economic evaluation, a “baseline” threshold could be expanded proportionally with any additional value not captured by the QALY component.
• Phelps C.
Resource allocation in decision support frameworks.
Alternatively, as part of a clean slate approach independent from economic evaluations, assuming the existence of a defined budget (eg, within a particular indication or therapeutic area), a value function could be used with the purchasing costs of the interventions to calculate multidimensional value for money ratios; this could be followed by a multicriteria portfolio decision analysis aiming to maximize benefits given the budget constraint, while allowing for opportunity costs to be naturally incorporated.
• Salo A.
• Keisler J.
• Morton A.
Portfolio Decision Analysis: Improved Methods for Resource Allocation.
In any case, further research would be required to develop and test such new methodological applications.
In a similar fashion, for the context of ICER’s assessments, the selection and aggregation of any relevant “other benefits or disadvantages” and “contextual considerations,” including the assignment of relative weights of importance, could be implemented at the level of the independent evidence appraisal committees (see Fig. 1C). Given ICER’s use of value-based price benchmarks representing acceptable long-term value for money, a possible way to fuse together the MCDA results with the use of a baseline incremental cost-utility ratio could be by expanding the latter proportionally to reflect how much of the model’s total value is accounted for by the non-QALY component.
• Phelps C.
Resource allocation in decision support frameworks.

## ICER’s Past Assessment of Tisagenlecleucel for Pediatric B-cell Acute Lymphoblastic Leukemia

As an illustrative example, the case of tisagenlecleucel’s ICER assessment for the treatment of pediatric B-cell acute lymphoblastic leukemia (ALL) could be considered.
ICER
Chimeric Antigen Receptor T-Cell Therapy for B-Cell Cancers: Effectiveness and Value, Final Evidence Report.
With regard to “other benefits or disadvantages,” the question was first asked: “Does treating patients with tisagenlecleucel offer one or more of the following other benefits?” and then 6 different benefit aspects were specified (Table 1). The 13-member evidence committee unanimously (13 out of 13) voted that the “intervention offers a novel mechanism of action or approach that will allow successful treatment of many patients who have failed other available treatments.” The majority of the evidence committee (9 out of 13) also voted that the “intervention will have a significant impact on improving return to work and/or overall productivity.” Finally, only a minority of the committee voted in favor that the “intervention will significantly reduce caregiver or broader family burden” (4 out of 13), and no member of the committee voted in favor of the existence of any other benefits.
Table 1List of relevant “other benefits or disadvantages” and respective voting results for tisagenlecleucel in the ICER assessment.
 This intervention offers reduced complexity that will significantly improve patient outcomes. 0/13 This intervention will reduce important health disparities across racial, ethnic, gender, socioeconomic, or regional categories. 0/13 This intervention will significantly reduce caregiver or broader family burden. 4/13 This intervention offers a novel mechanism of action or approach that will allow for successful treatment of many patients who have failed other available treatments. 13/13 This intervention will have a significant impact on improving return to work and/or overall productivity. 9/13 There are other important benefits or disadvantages that should have an important role in judgments of the value of this intervention. 0/13
ICER indicates Institute for Clinical and Economic Review.
With regard to “contextual considerations,” the evidence committee was then asked: “Are any of the following contextual considerations important in assessing tisagenlecleucel’s long-term value for money?”, followed by the specification of 6 different contextual aspects (Table 2). The majority of the committee (12 out of 13) voted that the “intervention is intended for the care of individuals with a condition of particularly high severity,” that there is “significant uncertainty about the long-term risk of serious side effects” (9 out of 13), and that there is “significant uncertainty about the magnitude or durability of the long-term benefits” (9 out of 13). A minority of the committee voted that the “intervention is the first to offer any improvement for patients with this condition” (5 out of 13), that “there are additional contextual considerations that should have an important role in judgments of the value of this intervention” (4 out of 13), and that “the intervention is intended for the care of individuals with a condition that represents a particularly high lifetime burden of illness” (1 out of 13).
Table 2List of relevant “contextual considerations” and respective voting results for tisagenlecleucel in the ICER assessment.
 This intervention is intended for the care of individuals with a condition of particularly high severity in terms of impact on length of life and/or quality of life. 12/13 This intervention is intended for the care of individuals with a condition that represents a particularly high lifetime burden of illness. 1/13 This intervention is the first to offer any improvement for patients with this condition. 5/13 Compared with the clofarabine or comparable chemo- or immunotherapy, there is significant uncertainty about the long-term risk of serious side effects of this intervention. 9/13 Compared with the clofarabine or comparable chemo- or immunotherapy, there is significant uncertainty about the magnitude or durability of the long-term benefits of this intervention. 9/13 There are additional contextual considerations that should have an important role in judgments of the value of this intervention. 4/13
After these 2 voting procedures, the last voting question corresponded to tisagenlecleucel’s overall long-term value for money based on the totality of the evidence: “Given the available evidence on comparative effectiveness and incremental cost-effectiveness, and considering other benefits, disadvantages, and contextual considerations, what is the long-term value for money of treatment with tisagenlecleucel versus treatment with clofarabine (Table 3)?” In response to that question, 7 members voted “intermediate,” 3 members voted “high,” and 1 member voted “low,” with 2 members abstaining; panel members that voted “intermediate” or “low” noted that “the high degree of uncertainty regarding long-term benefits and harms” led them to vote for a lower category of value than they would have done otherwise, with the two abstaining members also disclosing the same uncertainty issues responsible for precluding their ability to assess long-term value for money.
Table 3Voting results for tisagenlecleucel’s overall long-term value for money in the ICER assessment.
 Low: 1/13 Intermediate: 7/13 High: 3/13 Abstain: 2/13
Based on the above, it becomes evident that the overall 3-step voting process relating to other benefits, contextual considerations, and overall long-term value for money is associated with various challenges. Initially, the binary voting on the existence of other benefits and contextual considerations in the first 2 stages excludes performance valuation, towards understanding the magnitudes of committee members’ value preferences. This could be partially addressed through the use of the recently introduced Likert-scale but still, the relationship between product performance (in terms of benefits) or condition characteristics (in terms of contextual considerations) and value preferences would be characterized by limited granularity. Beyond that, valuation of trade-offs associated with the relative importance of the various aspects (pertaining to other benefits and contextual considerations against each other, and versus the core cost per QALY metric) would still remain completely unaccounted for. Ultimately, the robustness-determining step of the overall process would be the overly complex integration of all the evidence pieces together in the last stage that also remains completely unfacilitated and which is limited by restricted human mental capacity to carry out such tasks.
Assuming that a threshold range of $100 000 to$150 000 per QALY reflects the maximum acceptable long-term value for money based on which health-benefit (ie, ex value-based) price benchmarks are estimated, $100 000 per QALY could hypothetically be chosen as the baseline threshold for interventions whose value is not associated with any other benefits or contextual considerations. This baseline threshold could then be expanded up to$150 000 per QALY if the intervention fully satisfies all the additional benefit and contextual consideration–related value aspects, or a certain number of them deemed to be sufficient for reaching the maximum threshold (see Fig. 1C). For this to take place, the relative importance of each additional value aspect that is relevant for the particular decision context would need to be elicited as part of each technology’s evidence appraisal, such as, for example, using swing weighting via decision conferencing.
Ultimately, the decision of whether an intervention’s long-term value for money is acceptable and at what price would be fully transparent. This would be reflected based on the intervention’s performance on an explicit set of evaluation criteria but also the relevance and relative importance of this performance for the particular decision context, as judged by the members of the evidence appraisal committee. Nevertheless, the details of such a socio-technical process would require much attention and it should be clear that MCDA, as an approach, does not provide an “off-the-shelf” template. Instead, it must be tailor-made for each field of inquiry, as it has been evident from past experience with EMA and EU HTA bodies, both of which took several years to establish MCDA feasibility tests and methodological developments.

## Conclusion

Overall, we propose that the combination of quantitative decision analysis together with decision conferencing to be considered by ICER for the purpose of integrating together the various benefit components of interventions through the engagement of different stakeholders following group processes for the construction of value preferences. Ideally, any potential dismissal of MCDA methods by decision-makers and HTA institutions, including healthcare evaluation agencies like ICER, should be preceded by adequate research in their development and application; for example, this could take the form of case study work involving specific MCDA techniques and the conduct of participatory processes to arrive at the value of interventions for different contexts. We believe that through further collaborative work and improvement by MCDA practitioners and HTA researchers, quantitative decision analysis methods could act as valuable decision-making tools for ICER and other organizations keen in using value-based assessment approaches for HTA. Furthermore, given their explicit and transparent nature, such methods could serve as a template for future innovation, by guiding the development of new health interventions to what the relevant actors and society value most. We would therefore recommend their meticulous hands-on testing via continuous practical applications, in collaboration with the relevant HTA institutions, decision makers, and stakeholders. ICER’s mantra is known to be “Fair Pricing, Fair Access, Future Innovation”; however, fair pricing relies on accurate, consistent, and transparent assessment, for which the appropriate construction of value preferences and aggregation of benefits is crucial.

## Acknowledgments

The authors are grateful to Dan Ollendorf and 4 anonymous reviewers for their valuable comments that helped to improve the article.

## References

• ICER
A guide to ICER’s methods for health technology assessment.
(Accessed January 27, 2020)
• ICER
Overview of the ICER value assessment framework and update for 2017-2019.
(Accessed September 23, 2019)
• Angelis A.
• Lange A.
• Kanavos P.
Using health technology assessment to assess the value of new medicines: results of a systematic review and expert consultation across eight European countries.
Eur J Health Econ. 2018; 19: 123-152
• Clement F.M.
• Harris A.
• Li J.J.
• et al.
Using effectiveness and cost-effectiveness to make drug coverage decisions: a comparison of Britain, Australia, and Canada.
JAMA. 2009; 302: 1437-1443
• Nicod E.
• Kanavos P.
Commonalities and differences in HTA outcomes: a comparative analysis of five countries and implications for coverage decisions.
Health Policy. 2012; 108: 167-177
• Daniels N.
• Sabin J.
The ethics of accountability in managed care reform.
Health Affairs. 1998; 17: 50-64
• Daniels N.
J Urban Health. 1999; 76: 176-191
• Garber A.M.
• Phelps C.E.
Economic foundations of cost-effectiveness analysis.
J Health Econ. 1997; 16: 1-31
• Pliskin J.
• Separd D.
• Weinstein M.
Utility functions for life years and health status.
Oper Res. 1980; 28: 206-224
• Lakdawalla D.N.
• Doshi J.A.
• Garrison L.P.
• et al.
Defining elements of value in health care—a health economics approach: an ISPOR Special Task Force Report [3].
Value Health. 2018; 21: 131-139
• NICE
Guide to the Methods of Technology Appraisal 2013.
National Institute for Health and Care Excellence, London, UK2013
• NICE
Developing NICE Guidelines: the Manual.
The National Institute for Health and Care Excellence, London, UK2014
• NICE
Interim Process and Methods of the Highly Specialised Technologies Programme Updated to Reflect 2017 Changes.
National Institute for Health and Care Excellence, London, UK2017
• Willis M.
• Ödegaard K.
A case study of ex ante , value- based price and reimbursement decision- making: TLV and rimonabant in Sweden.
Eur J Health Econ. 2010; 11: 195-203
• Golan O.
• Hansen P.
• Kaplan G.
• et al.
Health technology prioritization: Which criteria for prioritizing new technologies and what are their relative weights?.
Health Policy. 2011; 102: 126-135
• Devlin N.
• Sussex J.
Incorporating multiple criteria in HTA: methods and processes.
Office of Health Economics, London, UK2011
• Thokala P.
• Duenas A.
Multiple criteria decision analysis for health technology assessment.
Value Health. 2012; 15: 1172-1181
• Kanavos P.
• Angelis A.
Multiple criteria decision analysis for value based assessment of new medical technologies: a conceptual framework. The LSE Health Working Paper Series in Health Policy and Economics.
London School of Economics and Political Science, London, England2013
• Marsh K.
• Lanitis T.
• Neasham D.
• et al.
Assessing the value of healthcare interventions using multi-criteria decision analysis: a review of the literature.
Pharmacoeconomics. 2014; 32: 345-365
• Angelis A.
• Kanavos P.
Multiple criteria decision analysis (MCDA) for evaluating new medicines in health technology assessment and beyond: the advance value framework.
Soc Sci Med. 2017; 188: 137-156
• Thokala P.
• Devlin N.
• Marsh K.
• et al.
Multiple criteria decision analysis for health care decision making--an introduction: Report 1 of the ISPOR MCDA Emerging Good Practices Task Force.
Value Health. 2016; 19: 1-13
• Marsh K.
• Ijzerman M.
• Thokala P.
• et al.
Multiple criteria decision analysis for health care decision making-Emerging Good Practices: Report 2 of the ISPOR MCDA Emerging Good Practices Task Force.
Value Health. 2016; 19: 125-137
• Lettieri E.
• Masella C.
• et al.
Implementation of EUnetHTA core Model® in Lombardia: the VTS framework.
Int J Technol Assess Health Care. 2014; 30: 105-112
• Tony M.
• Wagner M.
• Khoury H.
• et al.
Bridging health technology assessment (HTA) with multicriteria decision analyses (MCDA): field testing of the EVIDEM framework for coverage decisions by a public payer in Canada.
BMC Health Services Research. 2011; 11: 329
• Jaramillo H.E.C.
• Goetghebeur M.
• Moreno-Mattar O.
Testing multi-criteria decision analysis for more transparent resource-allocation decision making in Colombia.
Int J Technol Assess Health Care. 2016; 32: 307-314
• Angelis A.
Evaluating the benefits of new drugs in health technology assessment using multiple criteria decision analysis: a case study on metastatic prostate cancer with the dental and pharmaceuticals benefits agency (TLV) in Sweden.
3. MDM Policy & Practice, 2018
• Angelis A.
• Linch M.
• Montibeller G.
• et al.
Multiple criteria decision analysis for HTA across four EU member states: piloting the Advance Value Framework.
Soc Sci Med. 2020; 246112595
• Raiffa H.
Decision Analysis: Introductory Lectures on Choices Under Uncertainty.
• Angelis A.
• Phillips L.D.
Advancing structured decision-making in drug regulation at the FDA and EMA.
Br J Clin Pharmacol. 2020; : 1-11
• Keeney R.
• Raiffa H.
Decisions with Multiple Objectives: Preferences and Value Trade-Offs.
1993 ed. Cambridge University Press, Cambridge, MA1976
• Ralph L.K.
Multiplicative utility functions.
Oper Res. 1974; 22: 22-34
• Angelis A.
• Kanavos P.
Value-based assessment of new medical technologies: towards a robust methodological framework for the application of multiple criteria decision analysis in the context of health technology assessment.
Pharmacoeconomics. 2016; 34: 435-446
• Dawes R.M.
• Hastie R.
Rational Choice in an Uncertain World.
Harcourt Brace Jovanovich, San Diego, CA1988
• Miller G.A.
The magical number seven, plus or minus two: some limits on our capacity for processing information.
Psychol Rev. 1956; 63: 81-97
• Thaler R.H.
Nudge: Improving decisions About health, wealth, and happiness.
Yale University Press, New Haven, CT2008
• Kahneman D.
Thinking, Fast and Slow.
Penny Lane; Penguin, London, UK2011
• Montibeller G.
• Winterfeldt D.
Cognitive and motivational biases in decision and risk analysis.
Risk Anal. 2015; 35: 1230-1251
• Nutt D.J.
• King L.A.
• Phillips L.D.
Drug harms in the UK: a multicriteria decision analysis.
Lancet. 2010; 376: 1558-1565
• Angelis A.
• Montibeller G.
• Hochhauser D.
• et al.
Multiple criteria decision analysis in the context of health technology assessment: a simulation exercise on metastatic colorectal cancer with multiple stakeholders in the English setting.
BMC Med Inform Decis Mak. 2017; 17
• Felli J.C.
• Noel R.A.
• Cavazzoni P.A.
A multiattribute model for evaluating the benefit-risk profiles of treatment alternatives.
Med Decis Mak. 2009; 29: 104-115
• Marsh K.
• Zaiser E.
• Orfanos P.
• et al.
Evaluation of COPD treatments: a multicriteria decision analysis of aclidinium and tiotropium in the United States.
Value Health. 2017; 20: 132-140
• Moore A.
• Crossley A.
• Ng B.
• et al.
Use of multicriteria decision analysis for assessing the benefit and risk of over-the-counter analgesics.
J Pharm Pharmacol. 2017; 69
• Nixon R.
A case study using the PrOACT-URL and BRAT frameworks for structured benefit risk assessment.
Biom J. 2016; 58: 8
• Keeney R.L.
• von Winterfeldt D.
Managing nuclear waste from power plants.
Risk Analysis. 1994; 14: 107-130
• Phillips L.D.
The Royal Navy's type 45 story: a case study.
in: Salo A. Keisler J. Morton A. Portfolio Decision Analysis: Improved Methods for Resource Allocation. Springer-Verlag, New York2011
• Gregory R.
Using Stakeholder Values to Make Smarter Environmental Decisions.
Environment. 2000; 42: 34-44
• Bana e Costa C.A.
• Antão da Silva P.
• Nunes Correia F.
Multicriteria evaluation of flood control measures: the case of Ribeira do Livramento.
Water Res Mgmt. 2003; 18: 263-283
• French S.
• Kelly N.
• Morrey M.
Decision conferencing and the international Chernobyl project.
J Radiolog Protection. 1992; 12: 17-28
SMART Vaccines: Strategic Multi-Attribute Ranking Tool for Vaccines. The National Academies Press.
(Accessed December 31, 2019)
• FDA
Guidance for industry and review staff target product profile — a strategic development process tool.
(Accessed July 20, 2020)
• World Health Organisation
WHO Target Product Profiles (TPPs).
WHO, 2020 (https://www.who.int/immunization/research/ppc-tpp/target_product_profiles/en/. Accessed June 11, 2020.)
• Morton A.
Treacle and smallpox: two tests for multicriteria decision analysis models in health technology assessment.
Value Health. 2017; 20: 512-515
• Marsh K.
• Sculpher M.
• Caro J.J.
• et al.
The Use of MCDA in HTA: great potential, but more effort needed.
Value Health. 2018; 21: 394-397
• Marsh K.
• Thokala P.
• Youngkong S.
• et al.
Incorporating MCDA into HTA: challenges and potential solutions, with a focus on lower income settings.
Cost Eff Res Alloc. 2018; 16: 43
• Phillips L.D.
Best Practice for MCDA in healthcare.
in: Marsh K. Goetghebeur M. Thokala P. Multi-Criteria Decision Analysis to Support Healthcare Decisions. Springer, 2017
• Oliveira M.
• Mataloto I.
• Kanavos P.
Multi-criteria decision analysis for health technology assessment: addressing methodological challenges to improve the state of the art.
Eur J Health Econ. 2019; : 1-28
• EMA
Benefit-risk methodology project. Work package 3 report: Field tests.
European Medicines Agency, 2011 (https://www.ema.europa.eu/en/documents/report/benefit-risk-methodology-project-work-package-3-report-field-tests_en.pdf. Accessed February 13, 2019.)
• van Amsterdam J.
• Nutt D.
• Phillips L.
• et al.
European rating of drug harms.
J Psychopharm. 2015; 29: 655-660
1. Angelis A, Thursz M, Ratziu V, et al. Early health technology assessment during nonalcoholic steatohepatitis drug development: a two-round, cross-country, multicriteria decision analysis. Medical Decision Making 940672. (In Press)

• Keeney R.
Value Focused Thinking: A Path to Creative Decision Making.
Harvard University Press, Cambridge, MA1992
• Phillips L.D.
• Bana e Costa C.A.
Transparent prioritisation, budgeting and resource allocation with multi-criteria decision analysis and decision conferencing.
Ann Op Res. 2007; 154: 51-68
• Claxton K.
• Martin S.
• Soares M.
• et al.
Methods for the estimation of the National Institute for Health and care excellence cost-effectiveness threshold.
Health Tech Assess. 2015; 19: 1-503
• Phillips L.D.
Group elicitation of probability distributions: are many heads better than one?.
in: Shanteau J. Mellors B. Schum D. Decision Science and Technology: Reflections on the Contributions of Ward Edwards. Kluwer Academic Publishers, Norwell, MA1999
• Reagan-Cirincione P.
Improving the accuracy of group judgment: a process intervention combining group facilitation, social judgment analysis, and information technology.
Org Behav Human Decis Proc. 1994; 58: 246-270
• Montibeller G.
• von Winterfeldt D.
Individual and group biases in value and uncertainty judgments.
in: Dias L.C. Morton A. Quigley J. Elicitation: The Science and Art of Structuring Judgement. Springer, New York, NY2017
• Kerr N.L.
• Maccoun R.J.
• Kramer G.P.
Bias in judgment: comparing individuals and groups.
Psych Rev. 1996; 103: 687-719
• ICER
2020 Value Assessment Framework: Proposed Changes.
(Accessed September 23, 2019)
• ICER
2020 Value Assessment Framework: Final Framework.
Institute for Clinical and Economic Review, 2020 (https://icer-review.org/material/2020-value-assessment-framework-final-framework/. Accessed February 20, 2020)
• Meehl P.E.
Clinical Versus Statistical Prediction: A Theoretical Analysis and a Review of the Evidence.
University of Minnesota Press, Minneapolis, MN1954
• Montibeller G.
• Patel P.
• Del Rio Vilas V.J.
A critical analysis of multi-criteria models for the prioritisation of health threats.
Eur J Oper Res. 2020; 281: 87-99
• Phelps C.E.
• Lakdawalla D.N.
• Basu A.
• et al.
Approaches to aggregation and decision making—a health economics approach: an ISPOR Special Task Force Report [5].
Value Health. 2018; 21: 146-154
• HM T.
The green book: central government guidance on appraisal and evaluation.
(Accesed October 7, 2019)
• Phillips L.
A theory of requisite decision models.
Acta Psych. 1984; 56: 29-48
• Phillips L.
• Phillips M.
Faciliated work groups: theory and practice.
The J Operation Res Soc. 1993; 44
• Dodgson J.
• Spackman M.
• Pearman A.
• et al.
Multi-Criteria Analysis: A Manual.
2009 ed. Department for Communities and Local Government, London, UK2000
• Phillips L.
Decision conferencing.
in: Edwards W. Miles R. von Winterfeldt D. Advances in Decision Analysis: From Foundations to Applications. Cambridge University Press, Cambridge, MA2007
• Phelps C.