Advertisement

Evaluating Frameworks That Provide Value Measures for Health Care Interventions

      Abstract

      The recent acceleration of scientific discovery has led to greater choices in health care. New technologies, diagnostic tests, and pharmaceuticals have widely varying impact on patients and populations in terms of benefits, toxicities, and costs, stimulating a resurgence of interest in the creation of frameworks intended to measure value in health. Many of these are offered by providers and/or advocacy organizations with expertise and interest in specific diseases (e.g., cancer and heart disease). To help assess the utility of and the potential biases embedded in these frameworks, we created an evaluation taxonomy with seven basic components: 1) define the purpose; 2) detail the conceptual approach, including perspectives, methods for obtaining preferences of decision makers (e.g., patients), and ability to incorporate multiple dimensions of value; 3) discuss inclusions and exclusions of elements included in the framework, and whether the framework assumes clinical intervention or offers alternatives such as palliative care or watchful waiting; 4) evaluate data sources and their scientific validity; 5) assess the intervention’s effect on total costs of treating a defined population; 6) analyze how uncertainty is incorporated; and 7) illuminate possible conflicts of interest among those creating the framework. We apply the taxonomy to four representative value frameworks recently published by professional organizations focused on treatment of cancer and heart disease and on vaccine use. We conclude that each of these efforts has strengths and weaknesses when evaluated using our taxonomy, and suggest pathways to enhance the utility of value-assessing frameworks for policy and clinical decision making.

      Keywords

      Introduction

      The explosion of scientific knowledge, discovery, and technology began a new era of health care, wherein deployment of health care interventions depends on personal, lifestyle, genetic, and molecular disease factors. Although these advances have the potential to improve health, they increase the complexity of analyses to determine value at reasonable and affordable costs [
      • Lawler M.
      • French D.
      • Henderson R.
      • et al.
      Shooting for the moon or flying too near the sun? Crossing the value Rubicon in precision cancer care.
      ,
      • Chandra A.
      • Shafrin J.
      • Dhawan R.
      Utility of cancer value frameworks for patients, payers, and physicians.
      ,
      • Swank M.F.
      • Brennan L.K.
      • Gentry D.
      • Kemner A.L.
      Using frameworks to diagram value in complex policy and environmental interventions to prevent childhood obesity.
      ,
      • Rogowski W.
      • Payne K.
      • Schnell-Inderst P.
      • et al.
      Concepts of “personalization” in personalized medicine: implications for economic evaluation.
      ,
      Committee on Valuing Community-Based, Non-Clinical Prevention Programs; Board on Population Health and Public Health Practice; Institute of Medicine
      An Integrated Framework for Assessing the Value of Community-Based Prevention.
      ,
      • Pronk N.P.
      • Hernandez L.M.
      • Lawrence R.S.
      An integrated framework for assessing the value of community-based prevention: a report of the Institute of Medicine.
      ,
      • Anderson J.L.
      • Heidenreich P.A.
      • Barnett P.G.
      • et al.
      ACC/AHA statement on cost/value methodology in clinical practice guidelines and performance measures: a report of the American College of Cardiology/American Heart Association Task Force on Performance Measures and Task Force on Practice Guidelines.
      ,
      • Porter M.E.
      What is value in health care?.
      ].
      Faced with new health care interventions and escalating costs of care, many professional organizations have developed frameworks to measure and/or rank the value of novel interventions [
      • Swank M.F.
      • Brennan L.K.
      • Gentry D.
      • Kemner A.L.
      Using frameworks to diagram value in complex policy and environmental interventions to prevent childhood obesity.
      ,
      • Anderson J.L.
      • Heidenreich P.A.
      • Barnett P.G.
      • et al.
      ACC/AHA statement on cost/value methodology in clinical practice guidelines and performance measures: a report of the American College of Cardiology/American Heart Association Task Force on Performance Measures and Task Force on Practice Guidelines.
      ,
      • de Souza J.A.
      • Seiwert T.Y.
      A value framework in head and neck cancer care.
      ,
      • Harris R.P.
      • Wilt T.J.
      • Qaseem A.
      A value framework for cancer screening: advice for high-value care from the American College of Physicians.
      ,
      • Thaker N.G.
      • Pugh T.J.
      • Mahmood U.
      • et al.
      Defining the value framework for prostate brachytherapy using patient-centered outcome metrics and time-driven activity-based costing.
      ,
      • Teckie S.
      • McCloskey S.A.
      • Steinberg M.L.
      Value: a framework for radiation oncology.
      ,
      • Yount K.W.
      • Turrentine F.E.
      • Lau C.L.
      • Jones R.S.
      Putting the value framework to work in surgery.
      ,
      • Barnighausen T.
      • Bloom D.E.
      • Cafiero-Fonseca E.T.
      • O’Brien J.C.
      Valuing vaccination.
      ,
      • Mitera G.
      • Earle C.
      • Latosinsky S.
      • et al.
      Choosing wisely Canada cancer list: ten low-value or harmful practices that should be avoided in cancer care.
      ,

      National Comprehensive Cancer Network. NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines®) with NCCN Evidence Blocks. Available from: http://www.nccn.org/evidenceblocks/pdf. [Accessed August 1, 2016].

      ,

      Institute for Clinical and Economic Review. Methods: the ICER approach. Available from: http//:icer-review.org/methodology/icers-methods/. [Accessed August 1, 2016].

      ,
      • Schnipper L.E.
      • Davidson N.E.
      • Wollins D.S.
      • et al.
      American Society of Clinical Oncology statement: a conceptual framework to assess the value of cancer treatment options.
      ,
      • Schnipper L.E.
      • Davidson N.E.
      • Wollins D.S.
      • et al.
      Updating the American Society of Clinical Oncology value framework: revisions and reflections in response to comments received.
      ,
      • Cheung M.C.
      • Sabharwal M.
      • Chambers A.
      • et al.
      Multiple dimensions of value: evaluative frameworks for new cancer therapies.
      ,
      • Hoch J.S.
      • Sabharwal M.
      Informing Canada’s cancer drug funding decisions with scientific evidence and patient perspectives: the Pan-Canadian Oncology Drug Review.
      ,
      • Cherny N.I.
      • Sullivan R.
      • Dafni U.
      • et al.
      A standardised, generic, validated approach to stratify the magnitude of clinical benefit that can be anticipated from anti-cancer therapies: the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS).
      ]. Some of these new frameworks use components of cost-effectiveness analysis (CEA), whereas others create de novo value measures. It remains unclear how such frameworks will guide clinical practice policy, and whether they offer particular advantages over traditional approaches such as CEA.
      First, we summarize a taxonomy of principles—expanding upon cost-effectiveness theory—that provides an approach to assess different value frameworks. Next, we apply this taxonomy to evaluate exemplar value frameworks to illustrate strengths, gaps, and potential concerns. We intend for this synthesis to stimulate discussion about best practices and use of value metrics in policy and clinical decisions. We derived these principles in our synthesis of standard practice for the reporting of CEAs [
      ,
      ,
      • Sanders G.D.
      • Neumann P.J.
      • Basu A.
      • et al.
      Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine.
      ], recent International Society for Pharmacoeconomics and Outcomes Research summaries of multicriteria decision analysis (MCDA) reporting guidelines [
      • Marsh K.
      • IJzerman M.
      • Thokala P.
      • et al.
      Multiple criteria decision analysis for health care decision making—emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force.
      ], standard medical journal requirements for conflict of interest disclosures [

      International Committee of Medical Journal Editors. Standard statistical evaluation tools for medical research. Available from: http://www.icmje.org/recommendations/browse/roles-and-responsibilities/author-responsibilities--conflicts-of-interest.html. [Accessed October 10, 2016].

      ], standard Cochrane review methods [

      Cochrane Training. Cochrane handbook for systematic reviews of interventions. Available from: http://training.cochrane.org/handbook. [Accessed October 10, 2016].

      ], and our own analysis. Although we draw many of these ideas from others, assembling them to evaluate models that estimate the value of health care is novel.

      Value Taxonomy

      Value frameworks can assist in public health prioritization, practice guidelines, formulary or related resource expenditure decisions (e.g., purchase of equipment or coverage of a specific drug), and patient-physician decision making about a specific care choice. Table 1 describes our taxonomy to evaluate these value frameworks. We intend for this to apply to these different purposes, but specific elements may be more or less salient for different uses of the value framework. The following section highlights the implications of these varying purposes when using this taxonomy to evaluate value frameworks.
      Table 1Proposed taxonomy for evaluation of frameworks assessing the value of health care interventions
      CategoryComponent details
      PurposeIs the purpose defined and are the elements of the framework consistent with the purpose?
      Conceptual approach, perspective, and preferencesIs the value structure of the framework clear and transparent? If it requires assumptions, do they have general and clinical face validity?
      What perspective does the framework use (e.g., societal, patient, provider, payer)? Is the perspective made explicit or embedded in the framework?
      Does the framework include standard methods of measuring value, such as cost-effectiveness analysis?
      Does the framework consider multiple attributes of medical interventions such as individual preferences, equity and the distribution of health care benefits and costs, issues involving externalities (such as contagious diseases), and the value of scientific breakthrough?
      If the framework includes multiple dimensions of value, are the weights for each component population-based or expert-provided, and are the methods for eliciting such weights both clearly stated and methodologically sound?
      Does the framework allow individual patient preferences? If so, are they elicited with methods known to be free of bias and to produce reliable results?
      Intervention components and comparatorsDoes the framework include all components and consequences of the intervention, or merely a portion of those (e.g., drug acquisition costs)?
      Does the framework aggregate or disaggregate such things as toxicity or other side effects of intervention?
      Does the framework assume as a baseline that some intervention will be provided, or does it allow for “watchful waiting” or “palliative care” as an option?
      Data sourcesAre clinical and other data derived from expert opinion, population surveys, or other sources? Are the sources made clear, and is the process replicable?
      Economics/costsWhat is the effect of the intervention on the total cost of treating a defined population, including whether inclusion of the intervention will increase, decrease, or leave unchanged the total costs of care for that population?
      Uncertainty and identification of important gapsIs uncertainty in data about costs or effects considered in conclusions about value? Are there gaps in knowledge about aspects of care that could change the value rating?
      Conflicts of interestDoes the sponsoring organization have a financial stake in the process, and if so, is this declared? Does the framework bias the results in favor of the sponsor’s financial position?

      Conceptual Approach, Perspective, and Preferences in Measuring Value

      Value frameworks have evolved, in part, to provide greater standardization and transparency in health care decision making [
      • Madhavan G.
      • Phelps C.E.
      • Colwell R.R.
      • et al.
      In plain view: a transparent systems approach for enhancing health policy decisions.
      ]. Furthermore, formal frameworks can minimize biases of intuitive decision making from issues of framing of the problem, risk analysis, intertemporal trade-offs, and others [
      • Kahneman D.
      Thinking, Fast and Slow.
      ].
      We began our taxonomy in the formal framework of CEA. CEAs most commonly use the societal perspective, but this perspective usually does not correspond to that held by individual participants in health care decisions—patients, providers, payers, public policy decision makers, or owners of resources used to create health care interventions—nor does it consider community well-being or other important policy issues such as the distribution of benefits and costs across various subsets of the population (among other omissions).
      No single perspective can represent the interests of all participants in value-based decisions. For example, physicians generally advise patients using a patient perspective, but may introduce the societal perspective in their role as the steward of societal resources [
      • Schnipper L.E.
      • Bastian A.
      New frameworks to assess value of cancer care: strengths and limitations.
      ] or, in some cases, their own professional organization’s or personal financial situation. Thus, we believe that multiple perspectives should be considered, varying by individual or organization assessing the intervention, consistent with the most recent recommendations for reporting CEAs [
      ,
      • Sanders G.D.
      • Neumann P.J.
      • Basu A.
      • et al.
      Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine.
      ].
      CEA can also be useful in developing value frameworks because it provides data on intermediate events of importance to judging the value of an intervention, such as toxicity rates and overall survival, measured in life-years saved, quality-adjusted life-years, or disability-adjusted life-years [
      ,
      ,
      • Sanders G.D.
      • Neumann P.J.
      • Basu A.
      • et al.
      Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine.
      ,

      World Health Organization. Metrics: disability-adjusted life year (DALY). Available from: http://www.who.int/healthinfo/global_burden_disease/metrics_daly/en/. [Accessed August 1, 2016].

      ,
      • Malone D.C.
      • Berg N.S.
      • Claxton K.
      • et al.
      International Society for Pharmacoeconomics and Outcomes Research comments on the American Society of Clinical Oncology value framework.
      ]. Compared with the existing care standard, the ratio of added costs to added gains in health benefits—the incremental cost-effectiveness ratio—stands as one value criterion. Nevertheless, no consensus exists on cutoff points to determine the “acceptability” of a given incremental cost-effectiveness ratio, and both illness severity and the number of people affected can influence “cutoff point” choices.
      CEA also does not typically capture less tangible factors such as the fear associated with diseases such as HIV, cancer, degenerative neurological conditions, or mental illness, because these do not represent distinct health outcome states for which utility measurements exist. In concept, CEA could incorporate such issues, but in many cases, the data requirements would be impossibly complex. Full CEA evaluation requires population-based utility-level estimates for every state of nature in the model. Capturing such things as fear of the diseases, contagion effects, restrictions on freedom, and modification of religious rites (all arising in response to the Ebola virus) provides examples of issues that CEA cannot meaningfully capture.
      Although grounded in CEA, our taxonomy extends beyond this foundation. CEA does not generally consider many other attributes of decision making that affect the use of health care interventions. For example, CEA cannot address the distribution of health burdens that might fall particularly heavily on disadvantaged populations (e.g., sickle cell anemia in African Americans), or whether some aspects of a medical intervention have important moral or ethical components (e.g., abortion after detection of a fetal genetic mutation with profound consequences).
      Applying CEA for decision making brings another issue to the foreground: a new technology may pass standard “cutoff” requirements but then overwhelm the available budget—the issue of “affordability.” CEA methods are at present not equipped to deal with this issue.
      MCDA tools offer one approach to formally include these “other issues” into the value framework. These tools include such approaches as the analytic hierarchy process [
      • Saaty T.L.
      Decision making with the analytic hierarchy process.
      ] and the multi-attribute utility theory [
      • Edwards W.
      • Barron F.H.
      SMARTS and SMARTER: improved simple methods for multiattribute utility measurement.
      ,
      • Barron F.H.
      • Barrett B.E.
      Decision quality using ranked attribute weights.
      ], both used in recent health care decision processes [
      • Schmidt K.
      • Aumann I.
      • Hollander I.
      • et al.
      Applying the analytic hierarchy process in healthcare research: a systematic literature review and evaluation of reporting.
      ,
      ,
      ,
      ].
      These models bring to the foreground an issue that does not arise in CEA: How does one balance among competing goals in a value model? MCDA tools use various methods to elicit weights attached to each (potentially) competing goal. With a single decision maker, these methods work well in general [
      • Saaty T.L.
      Decision making with the analytic hierarchy process.
      ,
      • Barron F.H.
      • Barrett B.E.
      Decision quality using ranked attribute weights.
      ], but the best methods to combine individual into group preferences remain elusive. If group estimates are not derived from relevant population samples, our framework suggests that analysts should provide good justification and clear logic for using alternative sources.
      Our taxonomy proposes that value frameworks incorporate preferences, examine multiple perspectives, and are transparent. In addition, we suggest that value frameworks should consider distribution and ethical issues and other pertinent factors by using MCDA methods.

      Accounting for All Components of the Health Condition and Intervention

      Value frameworks should consider many aspects of each disease process and all short-term and downstream benefits and harms of interventions. For example, in cancer, including only one cost component, such as drug acquisition costs for a new treatment, would bias the overall assessment by ignoring other costs, such as those of drug administration, supportive care, and other aspects of therapy. In the treatment of HIV, cancer, and other conditions, drugs may have significant toxicity that can affect their value. Screening tests such as mammography can save lives, but also carry with them meaningful rates of false-positive results and overdiagnosis. These issues should enter a full value assessment, not only in health status outcomes and costs but also in emotional burdens created [
      • Nelson H.D.
      • Fu R.
      • Cantor A.
      • et al.
      Effectiveness of breast cancer screening: systematic review and meta-analysis to update the 2009 U.S. Preventive Services Task Force recommendation..
      ,
      • Nelson H.D.
      • Pappas M.
      • Cantor A.
      • et al.
      Harms of breast cancer screening: systematic review to update the 2009 U.S. Preventive Services Task Force recommendation.
      ].
      Some value frameworks treat all harms as equal in consequence to patients or other decision makers. For example, when the intervention is a drug treatment, rates of all side effects might be used in comparing the value of two therapies. This creates biases when the intensities and/or durations of the side effects differ, or when some effects have greater consequence than do others to individual patients (e.g., nausea/vomiting vs. fatigue vs. increased risk of iatrogenic death). In best practice, our taxonomy recommends that individual preferences on these issues inform the overall assessments of alternative therapies, and that when infeasible, population-based estimates serve as an alternative.
      Present value frameworks tend to focus on active therapy (e.g., drugs, surgery, and/or invasive tests). More recently, for certain diseases, such as early, nonaggressive prostate cancers, with invasive and potentially toxic standard-care therapies, alternatives now include no therapy (active surveillance). In the advanced disease setting, palliative interventions are being increasingly considered. If value frameworks do not explicitly consider these less aggressive approaches (i.e., only comparing novel therapies with standard therapy), the resulting decision may be biased toward therapeutic action, possibly conflicting with patient preferences. Therefore, we suggest that value frameworks compare all feasible options for patients, not just those using therapeutic intervention.

      What are the sources of and process for use of data in the framework? Does this process conform to accepted methods for evidence evaluation?

      Several accepted methods and hierarchies evaluate the quality of evidence about health care intervention effects. Randomized controlled trials generally provide stronger evidence than do other methods (e.g., historical comparisons, case control studies, and others), and sample sizes and research design in all studies affect precision and accuracy of parameter estimates [

      Cochrane Training. Cochrane handbook for systematic reviews of interventions. Available from: http://training.cochrane.org/handbook. [Accessed October 10, 2016].

      ].
      In cases in which good-quality, empirically based evidence on key issues in the value framework is not available, expert judgment may provide the only realistic option. Using expert-based estimates creates biases when the experts have a professional or financial stake in choices being evaluated (e.g., profit from sales of a device or drug, or enhanced reimbursement for delivery of a new agent [see later for further discussion]). When possible, valuation models should use experts not in conflict of interest situations and should clearly identify when any such conflicts exist.

      Does the framework include the full economic consequences of all intervention components?

      As in standard CEA, our taxonomy recommends that value frameworks include the full cost of all the components and consequences of an intervention. For example, in frameworks to evaluate drug treatments, economic impacts would include drug costs, provider time and delivery costs, staff time, facilities and equipment overhead and costs, costs of treatment-related side effects and supportive care medications required, patient time and travel costs, and costs of all downstream events until death from the disease or other causes. To the extent that frameworks exclude portions of these costs, they result in biased recommendations.
      Many value frameworks have different perspectives and include different costs. Patient-perspective frameworks, for example, might normally include co-payments made by patients only. Payer-centric frameworks may focus only on the costs and savings to the health care provider. These approaches can lead to suboptimal value-based decisions from a societal perspective.
      Costs also need to be considered in the context of extant budgets. For example, hepatitis C medications have cost-effectiveness ratios within the range of other accepted interventions, but still have high absolute costs that may overwhelm pre-established budgets.
      Projected budget impact has played a key role in some health care system decisions about these medications, mostly appearing as an independent discussion of budget impact in addition to standard CEA [

      Institute for Clinical and Economic Review. Methods: the ICER approach. Available from: http//:icer-review.org/methodology/icers-methods/. [Accessed August 1, 2016].

      ]. We believe that this approach misses a key point: budgets and acceptable levels for cost-effectiveness ratios cannot be considered independently. Introducing a new (and expensive) treatment adds budget pressure, which invariably will raise the value of incremental resources (what economists call the “shadow price”). The proper solution will likely involve re-assessment of the existing budget and re-examination of interventions that cannot compete (in a cost-effectiveness paradigm) with the newly tightened budget situation.

      How does the framework account for uncertainty in effect or cost data?

      Many analytic methods incorporate effects of uncertainty on the conclusions of a value recommendation. A key aspect of characterizing uncertainty relates to identification of gaps in knowledge or evidence about aspects of care that could change the value conclusions. Our taxonomy does not specify a particular method to identify the impact of uncertainty, but rather that uncertainty and gaps in data be explicitly evaluated.

      Conflicts of Interest

      Professional specialty organizations often have the best knowledge about a particular set of health care interventions, and thus are well placed to develop value frameworks, but may also have conflicts of interest in these issues. Value assessments may be subject to bias if financial incentives favor more (and costlier) interventions in certain delivery settings (e.g., fee-for-service vs. managed care or accountable care organizations). These issues extend also to the use of facilities owned in part or whole, directly or indirectly, by decision-making providers.
      Beyond financial incentives, the purpose of specialist organizations is to advocate for professional authority and action, which may further bias these sponsors toward interventions rooted in their specialties (such as chemotherapy, which is rooted in oncology, compared with palliative care, with separate specialty affiliations). Thus, we should consider whether the value framework’s sponsoring organization has a professional stake in the process, and if so, whether the framework potentially biases the results in favor of the sponsor’s position.

      Examples of Recent Frameworks

      Value frameworks have recently emerged from different medical specialty groups representing oncology [
      • de Souza J.A.
      • Seiwert T.Y.
      A value framework in head and neck cancer care.
      ,

      National Comprehensive Cancer Network. NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines®) with NCCN Evidence Blocks. Available from: http://www.nccn.org/evidenceblocks/pdf. [Accessed August 1, 2016].

      ,

      Institute for Clinical and Economic Review. Methods: the ICER approach. Available from: http//:icer-review.org/methodology/icers-methods/. [Accessed August 1, 2016].

      ,
      • Schnipper L.E.
      • Davidson N.E.
      • Wollins D.S.
      • et al.
      American Society of Clinical Oncology statement: a conceptual framework to assess the value of cancer treatment options.
      ,
      • Schnipper L.E.
      • Davidson N.E.
      • Wollins D.S.
      • et al.
      Updating the American Society of Clinical Oncology value framework: revisions and reflections in response to comments received.
      ,
      • Cheung M.C.
      • Sabharwal M.
      • Chambers A.
      • et al.
      Multiple dimensions of value: evaluative frameworks for new cancer therapies.
      ,
      • Hoch J.S.
      • Sabharwal M.
      Informing Canada’s cancer drug funding decisions with scientific evidence and patient perspectives: the Pan-Canadian Oncology Drug Review.
      ,
      • Cherny N.I.
      • Sullivan R.
      • Dafni U.
      • et al.
      A standardised, generic, validated approach to stratify the magnitude of clinical benefit that can be anticipated from anti-cancer therapies: the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS).
      ], cancer screening [
      • Harris R.P.
      • Wilt T.J.
      • Qaseem A.
      A value framework for cancer screening: advice for high-value care from the American College of Physicians.
      ], radiation oncology [
      • Thaker N.G.
      • Pugh T.J.
      • Mahmood U.
      • et al.
      Defining the value framework for prostate brachytherapy using patient-centered outcome metrics and time-driven activity-based costing.
      ,
      • Teckie S.
      • McCloskey S.A.
      • Steinberg M.L.
      Value: a framework for radiation oncology.
      ], cardiology [
      • Anderson J.L.
      • Heidenreich P.A.
      • Barnett P.G.
      • et al.
      ACC/AHA statement on cost/value methodology in clinical practice guidelines and performance measures: a report of the American College of Cardiology/American Heart Association Task Force on Performance Measures and Task Force on Practice Guidelines.
      ], surgery [
      • Yount K.W.
      • Turrentine F.E.
      • Lau C.L.
      • Jones R.S.
      Putting the value framework to work in surgery.
      ], pediatrics [
      • Swank M.F.
      • Brennan L.K.
      • Gentry D.
      • Kemner A.L.
      Using frameworks to diagram value in complex policy and environmental interventions to prevent childhood obesity.
      ], and infectious disease and vaccines [
      • Barnighausen T.
      • Bloom D.E.
      • Cafiero-Fonseca E.T.
      • O’Brien J.C.
      Valuing vaccination.
      ]. To demonstrate the application of our taxonomy, we selected four exemplar value frameworks spanning several specialties (Table 2).
      Table 2Examples of application of proposed taxonomy for evaluation of frameworks assessing the value of health care interventions
      Protocol elementAmerican Society of Clinical Oncology
      • Schnipper L.E.
      • Davidson N.E.
      • Wollins D.S.
      • et al.
      Updating the American Society of Clinical Oncology value framework: revisions and reflections in response to comments received.
      National Comprehensive Cancer Network

      National Comprehensive Cancer Network. NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines®) with NCCN Evidence Blocks. Available from: http://www.nccn.org/evidenceblocks/pdf. [Accessed August 1, 2016].

      American Heart Association-American College of Cardiology
      • Anderson J.L.
      • Heidenreich P.A.
      • Barnett P.G.
      • et al.
      ACC/AHA statement on cost/value methodology in clinical practice guidelines and performance measures: a report of the American College of Cardiology/American Heart Association Task Force on Performance Measures and Task Force on Practice Guidelines.
      European Vaccine Working Group
      • Malone D.C.
      • Berg N.S.
      • Claxton K.
      • et al.
      International Society for Pharmacoeconomics and Outcomes Research comments on the American Society of Clinical Oncology value framework.
      Purpose
       Stated purposePatient-physician drug treatment decisions; separate framework for adjuvant vs. metastatic RxPatient-physician drug treatment decisions for surgery, radiation, and drugsInform practice guidelines and performance measuresPublic health policy decisions
       Framework consistent with purposeYes; will need to develop format for use in practiceYes; included with online treatment guidelinesYesYes
      Approach
       Clear and transparent conceptual frameworkNoNoYesYes
       Do assumptions have empirically supported clinical basis?Bonus points subjectively assignedScoring system subjective/expert opinionNAYes
       Is the perspective made explicitImplies clinical decision-making perspectiveImplies clinical decision-making perspectiveSocietal, with separate analyses with patient perspectives for clinical decisions; consider subgroupsSocietal with long-term population outcomes
       References existing methods of assessing value, allowing comparison across diseasesDe novo method: net health benefit score and costs per drug treatment course. “Bonus points” and toxicity included in net benefits score for a new regimen vs. control therapyDe novo method: evidence block score from 1 to 5 on the basis of subjective assessment of benefitsUses costs per QALY to grade value low to high value using WHO-CHOICE framework; includes uncertain value categoryUses dynamic agent-based models and costs per QALY
      Points for benefit are assigned on the basis of categories of median survivalDoes not allow comparisons across multiple available regimensValue categories can be based on societal cutoff points or a percent of GDPMay consider single or multiple vaccines
      Does not allow comparisons across multiple available regimens
       Considers multidecision attributes (preferences, equity, distribution, externalities)Preferences included in metastatic framework only, but based on expert designation of “bonus points” for symptom palliation, and quality of life, longer life with metastasesUnknownConsiders preferences, distribution, and budget impactsExplicitly considers use of multi-attribute theory to assist with decisions
      Intervention components and comparators
       All components and comparators consideredOnly clinical benefit as gain in survival, major toxicity, and costs of drugs and supportive care for drug delivery; patient co-pay for drugs included; other effects and costs excluded; update to framework includes lower grade toxicitiesEffectiveness, safety (i.e., major toxicity), quality of evidence, consistency of evidence, and affordabilityImplicit in CEA framework, but not specified; framework used as a broad templateVery broad horizon and effects considered in a CEA framework
      Examines only one drug and one control drugNo direct comparisons among treatment modalities
       Toxicity/side effects consideredYes, but not in costs for hospitalization etc.For some regimensImplicit but not specifiedYes
       Considers no treatment as a comparator (e.g., “watchful waiting” or “palliative care”)NoFor some treatment options (surgery, radiation, drugs)UnknownNA
      Data sources
       Trials or observational studies vs. expert opinion; quality of dataRCT and expert opinionRCT and expert opinionSpecifies US data; data rated from A to C on the basis of a formal review of the evidence quality; includes randomized trials and observational dataVaccine trials using intention-to- treat results
      Economics/costs
       All costs stemming from the intervention includedOnly drug costs; drug delivery and supportive care for drug deliveryDrug cost, supportive care, drug administration, and toxicity costs; costs ranked subjectively on an “affordability scale” from 1 to 5 (inexpensive to very expensive)Includes all costs in CEA frameworkIncludes all costs in CEA framework
      Includes patient co-pays for drugs but not time costs
       Budget impact assessedBudget impact not consideredBudget impact not consideredUnknown/UnstatedUnknown/Unstated
       Uncertainty and information gaps consideredNot consideredNot consideredNot explicitly considered; does include an unknown category for valueYes
      Conflicts of interestNot explicitly addressedNot explicitly addressedExplicitly addressedExplicitly addressed
      Disclosures included in journal articlePotential conflicts because value primarily determined by oncologists who deliver (and bill) care<50% of writing committee required to not have conflicts; relationships with drug or device companies allowed and disclosed onlineDisclosure included
      Potential conflicts because value primarily determined by oncologists who deliver (and bill) care
      Sponsoring organization has financial stake? Declared?Yes; NoYes; NoNo; NoEmployees of drug or vaccine companies excluded; all conflicts of relationships declared
      CEA, cost-effectiveness analysis; CHOICE, CHOosing Interventions that are Cost-Effective; GDP, gross domestic product; NA, not applicable; QALY, quality-adjusted life-year; RCT, randomized controlled trial; WHO, World Health Organization.

      Oncology

      The oncology frameworks largely focus on the value of new drugs, driven by the more rapid escalation of drug costs relative to other components of cancer care, and the impact of drug costs on patient co-payments [
      • Schnipper L.E.
      • Bastian A.
      New frameworks to assess value of cancer care: strengths and limitations.
      ,

      Henry J; Kaiser Family Foundation. 2015 Employer health benefits survey. Available from: http://kff.org/report-section/ehbs-2015-summary-of-findings/. [Accessed May 18, 2016].

      ,
      • Shankaran V.
      • Jolly S.
      • Blough D.
      • Ramsey S.D.
      Risk factors for financial hardship in patients receiving adjuvant chemotherapy for colon cancer: a population-based exploratory analysis.
      ]. In other disease specialties, new treatment paradigms, expensive drugs, and high-priced technology have also stimulated the development of value frameworks.
      The American Society of Clinical Oncology’s Task Force on Value in Cancer Care [
      • Schnipper L.E.
      • Davidson N.E.
      • Wollins D.S.
      • et al.
      American Society of Clinical Oncology statement: a conceptual framework to assess the value of cancer treatment options.
      ,
      • Schnipper L.E.
      • Davidson N.E.
      • Wollins D.S.
      • et al.
      Updating the American Society of Clinical Oncology value framework: revisions and reflections in response to comments received.
      ] and the National Comprehensive Cancer Center Network [

      National Comprehensive Cancer Network. NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines®) with NCCN Evidence Blocks. Available from: http://www.nccn.org/evidenceblocks/pdf. [Accessed August 1, 2016].

      ] both developed value assessments for systemic therapy drugs. Comments about this work recognize that the intended use centers on patient-based treatment decisions rather than on a societal perspective. These frameworks followed many elements recommended in our taxonomy, but they omit several key value components. First, they focus only on clinical outcomes related to survival and drug costs. Second, by limiting the framework to clinical trial evidence about episodes of drug treatment, they also do not permit assessment of the value of a given therapy in the general population [
      • Angelis A.
      • Kanavos P.
      Critique of the American Society of Clinical Oncology value assessment framework for cancer treatments: putting methodologic robustness first.
      ]. Third, the framework excludes other important costs of care, such as future need for radiation, repeat surgery, and palliative care. Fourth, many relevant outcomes needed to determine value from a societal or patient perspective are excluded, including the impact of therapy on quality of life [
      • Basch E.
      Toward a patient-centered value framework in oncology.
      ], perhaps as a result of omission of such data in the original clinical trials of efficacy [
      • Li J.
      • Asabere A.
      • Kelly S.
      • et al.
      Application of ASCO value framework evaluations of net health benefit for oncology drugs launched in the United States between 2013 and 2015.
      ]. The revised framework of the American Society of Clinical Oncology [
      • Schnipper L.E.
      • Davidson N.E.
      • Wollins D.S.
      • et al.
      Updating the American Society of Clinical Oncology value framework: revisions and reflections in response to comments received.
      ] endeavors to capture all types of toxicity, but can succeed only to the extent lower grade adverse events are reported.
      Next, the method used to assign value points is both arbitrary and relies primarily on opinions of providers rather than samples of patients or their families. Finally, there is no accounting for uncertainty in the underlying data or the result, giving a false sense of precision when comparing among treatments.
      Other oncology frameworks include DrugAbacus [

      Memorial Sloan Kettering Cancer Center. Evidence driven drug pricing project. Available from: http://www.drugabacus.org/. [Accessed August 1, 2016].

      ], the European Society for Medical Oncology’s Magnitude of Clinical Benefit Scale [
      • Cherny N.I.
      • Sullivan R.
      • Dafni U.
      • et al.
      A standardised, generic, validated approach to stratify the magnitude of clinical benefit that can be anticipated from anti-cancer therapies: the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS).
      ], and the pan-Canadian Oncology Drug Review [
      • Cheung M.C.
      • Sabharwal M.
      • Chambers A.
      • et al.
      Multiple dimensions of value: evaluative frameworks for new cancer therapies.
      ,
      • Hoch J.S.
      • Sabharwal M.
      Informing Canada’s cancer drug funding decisions with scientific evidence and patient perspectives: the Pan-Canadian Oncology Drug Review.
      ]. These frameworks seek to inform policy rather than clinical decision making. They too focus primarily on drugs, with DrugAbacus evaluating cost of drug development and production versus actual retail pricing. The pan-Canadian Oncology Drug Review is the only value framework in oncology that includes a societal perspective, patient stakeholders, and components of a CEA framework including quality-adjusted life-years; considers evidence beyond trials; and determines how the short-term budget impact of new interventions affects the Canadian health care budget.
      These oncology frameworks were developed by oncologists (with other researchers) who have professional and financial stakes in the drugs they prescribe, and so the possibility of conflict of interest arises. A recent review of oncology drug value frameworks was supported by a pharmaceutical company. The authors declared that they owned stock or consulted for this company [
      • Chandra A.
      • Shafrin J.
      • Dhawan R.
      Utility of cancer value frameworks for patients, payers, and physicians.
      ].

      Cardiology

      The American College of Cardiology and the American Heart Association developed another recent value framework [
      • Anderson J.L.
      • Heidenreich P.A.
      • Barnett P.G.
      • et al.
      ACC/AHA statement on cost/value methodology in clinical practice guidelines and performance measures: a report of the American College of Cardiology/American Heart Association Task Force on Performance Measures and Task Force on Practice Guidelines.
      ] to inform their practice guidelines. This framework rates both the quality of evidence and the value of care, with the latter based on a CEA framework. On the basis of CEA results, they developed a score for value from low to high using the World Health Organization’s CHOICE (CHOosing Interventions that are Cost-Effective) project, including the use of cutoff values of 1 to 3 times the per capita gross domestic product [

      World Health Organization. Choosing interventions that are cost-effective project. Available from: http://www.who.int/choice/cost-effectiveness/en/. [Accessed August 1, 2016].

      ]. Notably, the writing committee had fewer than 50% of members with industry relationships, and all these relationships were published. Although the primary data resources for this evaluation come from independent entities, we note that when committee members are the providers of therapies, they may have inherent conflicts of interest that can cause deliberate or subconscious bias. In general, the American College of Cardiology-American Heart Association framework robustly addresses almost all the key components of our taxonomy but could be improved by more explicit definitions of intervention components, and approaches to uncertainty.

      Vaccines

      Frameworks for the evaluation of vaccines against infectious diseases present many complexities that do not arise with other medical interventions, including the need to capture spread of disease in the population; changes in population’s demographic characteristics because of aging and migration; the natural history of immunity, herd protection, and community well-being; the need to model the population equilibrium; the lack of postmarket surveillance for adverse events; and the ability of the vaccine to prevent disease or disease-related effects (e.g., postherpetic neuralgia) [
      • Pronk N.P.
      • Hernandez L.M.
      • Lawrence R.S.
      An integrated framework for assessing the value of community-based prevention: a report of the Institute of Medicine.
      ,
      • Ultsch B.
      • Damm O.
      • Beutels P.
      • et al.
      Methods for health economic evaluation of vaccines and immunization decision frameworks: a consensus framework from a European Vaccine Economics Community.
      ,
      • Ultsch B.
      • Damm O.
      • Beutels P.
      • et al.
      Methods for health economic evaluations of vaccines—results from an international expert (workshop).
      ]. These factors present unique challenges in defining the perspective, individual preferences versus the effects of treatment of one person on unaffected persons (i.e., herd effects), and time horizon for a value framework [
      ]. Despite this complexity, the European guidelines for vaccine value assessment consider virtually all the components in our taxonomy [
      • Ultsch B.
      • Damm O.
      • Perleth M.
      • Wichmann O.
      Health economics in immunization decision-making—results from a systematic literature research and a stakeholder symposium in Germany.
      ]. These guidelines explicitly address conflicts of interest: the developers excluded employees of drug or vaccine companies and required members to disclose potential conflicts.

      Conclusions

      Although the movement to examine the value of health care is rapidly gaining prominence, no consensus exists on the value framework’s structure and components. Wider implementation of value frameworks may be possible by further use of data from clinical trials and information made available through electronic records [
      • Basch E.
      Toward a patient-centered value framework in oncology.
      ,
      • Fayanju O.M.
      • Mayo T.L.
      • Spinks T.E.
      • et al.
      Value-based breast cancer care: a multidisciplinary approach for defining patient-centered outcomes.
      ]. It will also be useful to include education about value frameworks in medical and public health training [
      • Patel M.S.
      • Davis M.M.
      • Lypson M.L.
      The VALUE framework: training residents to provide value-based care for their patients.
      ,
      • Phelps C.
      • Madhavan G.
      • Rappuoli R.
      • et al.
      Strategic planning in population health and public health practice: a call to action for higher education.
      ]. Our taxonomy could support these efforts and extend previous overviews [
      • Chandra A.
      • Shafrin J.
      • Dhawan R.
      Utility of cancer value frameworks for patients, payers, and physicians.
      ,
      • Sanders G.D.
      • Neumann P.J.
      • Basu A.
      • et al.
      Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine.
      ] by including a comprehensive set of metrics grounded in CEA.
      Source of financial support: This work was funded by a grant from the National Cancer Institute (grant no. U01CA183081) at the National Institutes of Health. This work was also supported in part by National Cancer Institute (grant nos. R35 CA197289 and U01 CA152958).

      References

        • Lawler M.
        • French D.
        • Henderson R.
        • et al.
        Shooting for the moon or flying too near the sun? Crossing the value Rubicon in precision cancer care.
        Public Health Genomics. 2016; 19: 132-136
        • Chandra A.
        • Shafrin J.
        • Dhawan R.
        Utility of cancer value frameworks for patients, payers, and physicians.
        JAMA. 2016; 315: 2069-2070
        • Swank M.F.
        • Brennan L.K.
        • Gentry D.
        • Kemner A.L.
        Using frameworks to diagram value in complex policy and environmental interventions to prevent childhood obesity.
        J Public Health Manag Pract. 2015; 21: S116-S120
        • Rogowski W.
        • Payne K.
        • Schnell-Inderst P.
        • et al.
        Concepts of “personalization” in personalized medicine: implications for economic evaluation.
        Pharmacoeconomics. 2015; 33: 49-59
        • Committee on Valuing Community-Based, Non-Clinical Prevention Programs; Board on Population Health and Public Health Practice; Institute of Medicine
        An Integrated Framework for Assessing the Value of Community-Based Prevention.
        National Academies Press, Washington, DC2012
        • Pronk N.P.
        • Hernandez L.M.
        • Lawrence R.S.
        An integrated framework for assessing the value of community-based prevention: a report of the Institute of Medicine.
        Prev Chronic Dis. 2013; 10: 120323
        • Anderson J.L.
        • Heidenreich P.A.
        • Barnett P.G.
        • et al.
        ACC/AHA statement on cost/value methodology in clinical practice guidelines and performance measures: a report of the American College of Cardiology/American Heart Association Task Force on Performance Measures and Task Force on Practice Guidelines.
        Circulation. 2014; 129: 2329-2345
        • Porter M.E.
        What is value in health care?.
        N Engl J Med. 2010; 363: 2477-2481
        • de Souza J.A.
        • Seiwert T.Y.
        A value framework in head and neck cancer care.
        Am Soc Clin Oncol Educ Book. 2014; : e304-e309
        • Harris R.P.
        • Wilt T.J.
        • Qaseem A.
        A value framework for cancer screening: advice for high-value care from the American College of Physicians.
        Ann Intern Med. 2015; 162: 712-717
        • Thaker N.G.
        • Pugh T.J.
        • Mahmood U.
        • et al.
        Defining the value framework for prostate brachytherapy using patient-centered outcome metrics and time-driven activity-based costing.
        Brachytherapy. 2016; 15: 274-282
        • Teckie S.
        • McCloskey S.A.
        • Steinberg M.L.
        Value: a framework for radiation oncology.
        J Clin Oncol. 2014; 32: 2864-2870
        • Yount K.W.
        • Turrentine F.E.
        • Lau C.L.
        • Jones R.S.
        Putting the value framework to work in surgery.
        J Am Coll Surg. 2015; 220: 596-604
        • Barnighausen T.
        • Bloom D.E.
        • Cafiero-Fonseca E.T.
        • O’Brien J.C.
        Valuing vaccination.
        Proc Natl Acad Sci U S A. 2014; 111: 12313-12319
        • Mitera G.
        • Earle C.
        • Latosinsky S.
        • et al.
        Choosing wisely Canada cancer list: ten low-value or harmful practices that should be avoided in cancer care.
        J Oncol Pract. 2015; 11: e296-e303
      1. National Comprehensive Cancer Network. NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines®) with NCCN Evidence Blocks. Available from: http://www.nccn.org/evidenceblocks/pdf. [Accessed August 1, 2016].

      2. Institute for Clinical and Economic Review. Methods: the ICER approach. Available from: http//:icer-review.org/methodology/icers-methods/. [Accessed August 1, 2016].

        • Schnipper L.E.
        • Davidson N.E.
        • Wollins D.S.
        • et al.
        American Society of Clinical Oncology statement: a conceptual framework to assess the value of cancer treatment options.
        J Clin Oncol. 2015; 33: 2563-2577
        • Schnipper L.E.
        • Davidson N.E.
        • Wollins D.S.
        • et al.
        Updating the American Society of Clinical Oncology value framework: revisions and reflections in response to comments received.
        J Clin Oncol. 2016; 34: 2925-2934
        • Cheung M.C.
        • Sabharwal M.
        • Chambers A.
        • et al.
        Multiple dimensions of value: evaluative frameworks for new cancer therapies.
        J Clin Oncol. 2016; 34: 1428-1429
        • Hoch J.S.
        • Sabharwal M.
        Informing Canada’s cancer drug funding decisions with scientific evidence and patient perspectives: the Pan-Canadian Oncology Drug Review.
        Curr Oncol. 2013; 20: 121-124
        • Cherny N.I.
        • Sullivan R.
        • Dafni U.
        • et al.
        A standardised, generic, validated approach to stratify the magnitude of clinical benefit that can be anticipated from anti-cancer therapies: the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS).
        Ann Oncol. 2015; 26: 1547-1573
      3. Gold M.R. Siegel J.E. Russell L.B. Weinstein M.C. Cost-Effectiveness in Health and Medicine. Oxford University Press, New York, NY1996
      4. Neumann P.J. Sanders G.D. Russell L.D. Cost-Effectiveness in Health and Medicine. 2nd ed. Oxford University Press, New York, NY2017
        • Sanders G.D.
        • Neumann P.J.
        • Basu A.
        • et al.
        Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine.
        JAMA. 2016; 316: 1093-1103
        • Marsh K.
        • IJzerman M.
        • Thokala P.
        • et al.
        Multiple criteria decision analysis for health care decision making—emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force.
        Value Health. 2016; 19: 125-137
      5. International Committee of Medical Journal Editors. Standard statistical evaluation tools for medical research. Available from: http://www.icmje.org/recommendations/browse/roles-and-responsibilities/author-responsibilities--conflicts-of-interest.html. [Accessed October 10, 2016].

      6. Cochrane Training. Cochrane handbook for systematic reviews of interventions. Available from: http://training.cochrane.org/handbook. [Accessed October 10, 2016].

        • Madhavan G.
        • Phelps C.E.
        • Colwell R.R.
        • et al.
        In plain view: a transparent systems approach for enhancing health policy decisions.
        Bridge. 2016; 46: 5-10
        • Kahneman D.
        Thinking, Fast and Slow.
        Farrar, Strass and Giroux, New York, NY2011
        • Schnipper L.E.
        • Bastian A.
        New frameworks to assess value of cancer care: strengths and limitations.
        Oncologist. 2016; 21: 654-658
      7. World Health Organization. Metrics: disability-adjusted life year (DALY). Available from: http://www.who.int/healthinfo/global_burden_disease/metrics_daly/en/. [Accessed August 1, 2016].

        • Malone D.C.
        • Berg N.S.
        • Claxton K.
        • et al.
        International Society for Pharmacoeconomics and Outcomes Research comments on the American Society of Clinical Oncology value framework.
        J Clin Oncol. 2016; 34: 2936-2937
        • Saaty T.L.
        Decision making with the analytic hierarchy process.
        Int J Serv Sci. 2008; 1: 83-98
        • Edwards W.
        • Barron F.H.
        SMARTS and SMARTER: improved simple methods for multiattribute utility measurement.
        Organ Behav Hum Decis Process. 1994; 60: 306-325
        • Barron F.H.
        • Barrett B.E.
        Decision quality using ranked attribute weights.
        Manag Sci. 1996; 42: 1515-1523
        • Schmidt K.
        • Aumann I.
        • Hollander I.
        • et al.
        Applying the analytic hierarchy process in healthcare research: a systematic literature review and evaluation of reporting.
        BMC Med Inform Decis Mak. 2015; 15: 112
      8. Madhavan G. Sangha K. Phelps C. Ranking Vaccines: A Prioritization Framework, Phase I: Demonstration of Concept and a Software Blueprint. National Academies Press, Washington, DC2012
      9. Madhavan G. Sangha K. Phelps C. Ranking Vaccines: A Prioritization Software Tool, Phase II: Prototype of a Decision-Support System. National Academies Press, Washington, DC2013
      10. Madhavan G. Phelps C. Rappuoli R. Ranking Vaccines: Applications of a Prioritization Software Tool, Phase III: Use Case Scenarios and Data Framework. National Academies Press, Washington, DC2014
        • Nelson H.D.
        • Fu R.
        • Cantor A.
        • et al.
        Effectiveness of breast cancer screening: systematic review and meta-analysis to update the 2009 U.S. Preventive Services Task Force recommendation..
        Ann Intern Med. 2016; 164: 244-255
        • Nelson H.D.
        • Pappas M.
        • Cantor A.
        • et al.
        Harms of breast cancer screening: systematic review to update the 2009 U.S. Preventive Services Task Force recommendation.
        Ann Intern Med. 2016; 164: 256-267
      11. Henry J; Kaiser Family Foundation. 2015 Employer health benefits survey. Available from: http://kff.org/report-section/ehbs-2015-summary-of-findings/. [Accessed May 18, 2016].

        • Shankaran V.
        • Jolly S.
        • Blough D.
        • Ramsey S.D.
        Risk factors for financial hardship in patients receiving adjuvant chemotherapy for colon cancer: a population-based exploratory analysis.
        J Clin Oncol. 2012; 30: 1608-1614
        • Angelis A.
        • Kanavos P.
        Critique of the American Society of Clinical Oncology value assessment framework for cancer treatments: putting methodologic robustness first.
        J Clin Oncol. 2016; 34: 2935-2936
        • Basch E.
        Toward a patient-centered value framework in oncology.
        JAMA. 2016; 315: 2073-2074
        • Li J.
        • Asabere A.
        • Kelly S.
        • et al.
        Application of ASCO value framework evaluations of net health benefit for oncology drugs launched in the United States between 2013 and 2015.
        Value Health. 2015; 18: A489
      12. Memorial Sloan Kettering Cancer Center. Evidence driven drug pricing project. Available from: http://www.drugabacus.org/. [Accessed August 1, 2016].

      13. World Health Organization. Choosing interventions that are cost-effective project. Available from: http://www.who.int/choice/cost-effectiveness/en/. [Accessed August 1, 2016].

        • Ultsch B.
        • Damm O.
        • Beutels P.
        • et al.
        Methods for health economic evaluation of vaccines and immunization decision frameworks: a consensus framework from a European Vaccine Economics Community.
        Pharmacoeconomics. 2016; 34: 227-244
        • Ultsch B.
        • Damm O.
        • Beutels P.
        • et al.
        Methods for health economic evaluations of vaccines—results from an international expert (workshop).
        Value Health. 2014; 17: A552
        • Ultsch B.
        • Damm O.
        • Perleth M.
        • Wichmann O.
        Health economics in immunization decision-making—results from a systematic literature research and a stakeholder symposium in Germany.
        Value Health. 2015; 18: A569
        • Fayanju O.M.
        • Mayo T.L.
        • Spinks T.E.
        • et al.
        Value-based breast cancer care: a multidisciplinary approach for defining patient-centered outcomes.
        Ann Surg Oncol. 2016; 23: 2385-2390
        • Patel M.S.
        • Davis M.M.
        • Lypson M.L.
        The VALUE framework: training residents to provide value-based care for their patients.
        J Gen Intern Med. 2012; 27: 1210-1214
        • Phelps C.
        • Madhavan G.
        • Rappuoli R.
        • et al.
        Strategic planning in population health and public health practice: a call to action for higher education.
        Milbank Q. 2016; 94: 109-125