Advertisement

Emerging Good Practices for Transforming Value Assessment: Patients’ Voices, Patients’ Values

Open ArchivePublished:March 08, 2018DOI:https://doi.org/10.1016/j.jval.2017.11.013

      Abstract

      Background

      Patient engagement is a transformative strategy for improving value assessment. US value framework developers have increased engagement activities, but more needs to be learned about how to best achieve meaningful patient engagement in value assessment. The objective was to glean good practices in patient engagement emerging from patient community experiences, to be used in value assessment.

      Methods

      The National Health Council Value Workgroup conducted a survey and held a focus group with its member advocacy organizations to gather experiences with value framework developers and views on emerging good practices.

      Results

      Ten of 13 organizations completed the survey; reporting 13 interactions with four framework developers. Most rated experiences as “good” to “very good.” Emerging good practices included (1) engage early; (2) engage a range of patients; (3) leverage patient-provided information, data resources, and outreach mechanisms; (4) be transparent; and (5) appreciate and accommodate resource constraints. Twelve of 13 organizations participated in the focus group, and this produced 30 emerging good practices in four areas: (1) timing; (2) methodology and data; (3) partnering; and (4) characterizing engagement.

      Discussion

      Patient engagement was limited in early development of value frameworks but has increased in the past few years. Patient groups report positive experiences that can serve as emerging good practices. These groups also reported experienced challenges in their interactions and recommended good practices to mitigate those challenges.

      Conclusions/Recommendations

      The growing pool of patient engagement experiences can be translated into good practices to advance a patient-centered, value-driven health care ecosystem. Lessons learned from these early experiences can help establish recommend emerging good practices that can eventually result in best practices and standards in the field.

      Keywords

      Background

      Value frameworks and assessments have proliferated and gained attention in the United States over the past few years. In 2015, four organizations—the American Society of Clinical Oncology (ASCO) [
      • Schnipper L.
      • Davidson N.
      • Wollins D.
      • et al.
      American Society of Clinical Oncology statement: a conceptual framework to assess the value of cancer treatment options.
      ], the Institute for Clinical and Economic Review (ICER) [

      Institute for Clinical and Economic Review. Value Assessment Project. Retrieved from: http://www.icer-review.org/impact-and outcomes/value-assessment-project/. [Accessed May 25, 2017].

      ], the National Comprehensive Cancer Network (NCCN) [

      National Comprehensive Cancer Network. NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines) with NCCN Evidence Blocks. Retrieved from: http://www.nccn.org/evidenceblocks/. [Accessed May 25, 2017].

      ], and Memorial Sloan Kettering Cancer Center (MSK) [

      Memorial Sloan Kettering Cancer Center. Welcome to Drug Abacus. Available from: http://www.drugabacus.org/drug-abacus-tool/. [Accessed May 25, 2017].

      ]—released frameworks that they describe as intended to support physicians and/or payers in assessing the relative value of treatments. At their inception, the US frameworks were developed and value assessments were produced with little to no patient engagement [

      National Health Council. The patient voice in value: the National Health Council patient-centered value model rubric. Available from: http://www.nationalhealthcouncil.org/sites/default/files/Value-Rubric.pdf. [Accessed May 25, 2017].

      ,
      • Perfetto E.M.
      • Oehrlein E.M.
      • Boutin M.
      • et al.
      Value to whom? The patient voice in the value discussion.
      ]. Currently, there is no one perfect model or framework for value assessment or even one that earns support across all stakeholders [
      • Sorenson C.
      • Lavezzari G.
      • Daniel G.
      • et al.
      Advancing value assessment in the United States: a multistakeholder perspective.
      ]. There is agreement among stakeholders that value frameworks need to better incorporate the patient perspective [

      FasterCures—A Center of the Milken Institute, Avalere Health. Patient-Perspective Value Framework (PPVF) Version 1.0. May 2017. Available from: http://www.fastercures.org/assets/Uploads/PPVF-Version-1.0-Methodology-Report-Final.pdf. [Accessed May 25, 2017].

      ].
      To increase and improve patient engagement, the National Health Council (NHC) in 2016 launched a Value Initiative to support its patient-advocate membership [

      National Health Council. Value Initiative. Available from: http://www.nationalhealthcouncil.org/value-initiative. [Accessed May 25, 2017].

      ]. The value initiative (1) a Value Model Rubric [

      National Health Council. The patient voice in value: the National Health Council patient-centered value model rubric. Available from: http://www.nationalhealthcouncil.org/sites/default/files/Value-Rubric.pdf. [Accessed May 25, 2017].

      ] that provides clarity to the patient community and framework developers/assessors on what constitutes patient-centeredness and engagement in value assessment; (2) a Get-Ready Checklist [

      National Health Council. Get-Ready Checklist. Available from: http://www.nationalhealthcouncil.org/sites/default/files/Get_Ready_Checklist.pdf. [Accessed May 25, 2017].

      ] that provides a guide for patient groups to prepare for engagement; (3) a Value Workgroup (Workgroup) that provides networking and information sharing through voluntary meetings of patient-organization staff with recent experience interacting with value framework developers; (4) a Qualitative Research Study that provides insights on patients’ definitions of value; and (5) a Health economics educational program that provides patient groups basics on economic and value-assessment terms and principles.
      A fundamental premise of the NHC’s Value Initiative is that meaningful patient engagement—in all aspects of value assessment—is a transformative strategy that can improve evaluations of the value of health care interventions to advance a value-driven health care ecosystem in the United States. Meaningful engagement is defined as interactions with the patient community (i.e., patients, caregivers, advocates, and patient advocacy groups) exemplified by direct relationships and partnerships that are bidirectional, reciprocal, and continuous; where communications are open, honest, and clear; and where engagement goals, participants, methods, desired impacts, and actual impacts are clearly outlined and transparent [

      National Health Council. The patient voice in value: the National Health Council patient-centered value model rubric. Available from: http://www.nationalhealthcouncil.org/sites/default/files/Value-Rubric.pdf. [Accessed May 25, 2017].

      ].
      When meaningful engagement takes place, a stronger, more comprehensive value assessment results that is accepted by a greater number of stakeholders. For example, a disease-specific patient group provides information to a framework developer describing heterogeneity treatment response, which has been recognized but not fully understood. The patient group convincingly conveys the downside of step therapy [

      National Psoriasis Foundation. ICER Q&A with Leah Howard. October 2016. Available from: https://www.psoriasis.org/advance/icer. [Accessed May 25, 2017].

      ], providing its own data on patient experiences. The result is a value assessment that highlights the importance of heterogeneity in the condition and recommends avoidance of step therapy. If this report is used for decision making by clinicians, payers, or patients, it will be more widely accepted, have greater utility in real-world care, and will not be disputed by patient and clinicians for not considering the heterogeneity in treatment effect they deal with routinely. Credibility and relevance are enhanced.
      There is recent evidence that patient engagement in value assessment has improved considerably in the last few years [

      National Psoriasis Foundation. ICER Q&A with Leah Howard. October 2016. Available from: https://www.psoriasis.org/advance/icer. [Accessed May 25, 2017].

      ,

      Arthritis Foundation. Arthritis Foundation proclaims a patient advocacy “win” through its work with the Institute for Clinical and Economic Review (ICER). April 2017. Available from: http://www.arthritis.org/about-us/news-and-updates/arthritis-foundation-proclaims-a-patient-advocacy-win-through-its-work-with-icer.php. [Accessed May 25, 2017].

      ]. However, questions remain about the best means to achieve meaningful and effective patient engagement in value assessment and to understand its impact on value assessment findings. This study’s objectives were to (1) gather the patient-community’s first-hand experiences with value framework developers and (2) glean from those experiences emerging good practices in patient engagement that can be disseminated, improved upon, and replicated.

      Methods

       NHC Value Workgroup

      Starting in November 2016, the NHC began convening a voluntary Value Workgroup (Workgroup) comprising patient-organization staff members that had interacted in the past and/or currently are interacting, or anticipated interacting with value framework developers. Twenty patient-organization staff members from 13 organizations participate in monthly meetings via teleconference. The participants tend to be senior management-level staff from patient advocacy organizations (e.g., CEO, VP, manager titles). They also tend to be patients or caregivers themselves with personal experience with the disease or condition they represent. The objectives for the group are networking, information sharing, and education, and monthly updates help to inform and guide the NHC’s Value Initiative. Workgroup members and NHC staff have learned a great deal from one another through these informal interactions. The Workgroup wanted to collect its views and experiences in a more formalized manner and decided to prospectively collect their experience reports and recommendations through a survey of Workgroup members and by using one of their meetings to hold a focus-group discussion.

       Survey

      From April 17 to April 21, 2017, the NHC Value Workgroup conducted an online survey to understand the experiences its members had with value framework developers. The SurveyMonkey questionnaire was designed by the NHC staff and University of Maryland, Baltimore, faculty with Workgroup input and pilot testing. The questionnaire (see Appendix A) was sent to each participating organization (N = 13) with the request that only one survey be completed for each organization. Topics included (a) activities comprising engagement experiences; (b) ratings of experiences with value framework developers; (c) experiences considered an emerging good practice (e.g., positive experience that should be replicated and disseminated); (d) proposed good practices recommended to mitigate challenges experienced; and (e) observed/perceived impact of patient input on value-assessment reports or final products. Open-ended questions also allowed respondents to provide details or other reflections on experiences. Workgroup members were sent four email requests to complete the survey. The group also was reminded about completing the survey during one of its teleconferences. Responses were anonymous.

       Focus Group

      On April 21, 2017, the Workgroup held a 2-hour, focus group discussion via teleconference with members. The purpose was to discuss specifically responses to survey questions (c) and (d) to drill down further on experienced and recommended emerging good practices. The Workgroup discussed positive and challenging experiences with value framework/assessment organizations and further discussed if the positive experiences represented emerging good practices that should be recommended for dissemination. It delved into challenging experiences and good practices that could be recommended to mitigate the challenges. The group came to consensus on experienced and recommended emerging good practices.
      Because the maximum number of participants anticipated in this study could only reach 13, only descriptive analyses were planned.

      Results

       Survey

      Ten of the 13 organizations (77%) completed the SurveyMonkey questionnaire. Of the three non-completers, one organization reported that it did not have any experiences with framework developers and could not answer the questions; one organization experienced a staffing change resulting in loss of Workgroup participation; and one organization did not respond.
      Ten patient organizations reported 13 interactions with four value framework developers. The organizations reported the most common activities experienced while interacting with framework developers (Table 1). The top three most common activities were providing advice/consultation on a condition/disease, patient recruitment, and question/problem development. The least common activities were clinician recruitment, providing input for an economic model, and serving as a member of a governing committee/board.
      Table 1Activities reported by patient advocacy organizations when interacting with value framework developers (survey reported, 10 out of 13 organizations surveyed responded)
      ActivityNumber of organizations
      Note: Some patient advocacy organizations had experiences with multiple value framework developers and provided multiple responses.
      Provide advice/consultation on a condition/disease10
      Patient recruitment7
      Question/problem development7
      Protocol development6
      Providing existing data5
      Presented/testified at a meeting4
      Member of an advisory panel/ Member of a governing committee or board4
      Data collection3
      Clinician recruitment2
      Provide input for an economic model2
      Other: Made public comments in writing1
      low asterisk Note: Some patient advocacy organizations had experiences with multiple value framework developers and provided multiple responses.
      Experience ratings on a list of value framework developers are presented in Table 2. Five groups had interacted with Framework Developer A, seven groups had interacted with B, one group had interacted with C, and one group had interacted with D. No interactions were reported with Framework Developers E and F. An excellent or very good rating was usually accompanied by such comments as “responsive,” “receptive,” “genuinely interested,” or “gets us.” Poor or fair ratings were accompanied by such comments as “unresponsive,” “problems with transparency,” “poor or no processes,” “cherry picking what they will/will not use or respond to,” or “provide no response without explanation.”
      Table 2Ratings of experiences reported by patient advocacy organizations
      Note: Some patient advocacy organizations had experiences with multiple value framework developers.
      engaging with value framework developers (survey reported, 10 of 13 organizations surveyed responded)
      PoorFairGoodVery GoodExcellent
      Value Framework Developer A--2-3
      Value Framework Developer B11-4-
      Value Framework Developer C--1--
      Value Framework Developer D---1-
      Value Framework Developer E-----
      Value Framework Developer F-----
      low asterisk Note: Some patient advocacy organizations had experiences with multiple value framework developers.
      Experienced good practices reported in the survey are listed in Table 3, and recommended good practices are listed in Table 4. Through the survey, the groups enumerated several ideas on emerging good practices based on their experiences, such as being involved early and often, tapping mechanisms patients group have for data collection, and adjusting timelines to accommodate resource-constrained advocacy organizations.
      Table 3Emerging good practice based on patient advocacy organizations’ experiences (survey reported, 5 of 13 organizations surveyed responded)
      • 1.
        Engage as early in the process as possible (two of the five responding organizations answered).
      • Involve patient advocacy organizations from the initial development of the framework, not only in a review capacity later down the road.
      • Engage early and often; be persistent and continuous.
      • 2.
        Engage a range of patients and patient groups (one of the five responding organizations answered).
      • Involve a range of patients to understand the varying needs.
      • 3.
        Leverage patient-provided information and data resources, and outreach mechanisms to collect information from patients (two of the five responding organizations answered).
      • Use existing patent advocacy organization outreach mechanisms that patient groups have developed to gather information from patients.
      • 4.
        Be transparent (two of the five responding organizations answered).
      • Provide redline changes in successive drafts of documents so edits are readily apparent.
      • Catalog and make publicly available all suggestions received during an open-comment period.
      • Provide/use an evaluation framework to triage the comments (e.g., what will be addressed immediately, in the short term, in the long term, or cannot be addressed; a spreadsheet of concerns and how they have or will be addressed).
      • Communicate, provide updates, personal outreach, and flexibility to enhance interactions.
      • 5.
        Appreciate and accommodate patient community resource constraints (one of the five responding organizations answered).
      • Lengthen comment periods to the extent possible to accommodate the resource limitations of patient advocacy organizations.
      Table 4Proposed good practices based on patient advocacy organizations’ recommendations (survey reported, 4 of 13 organizations surveyed responded)
      • 1.
        Engage as early in the process as possible (one of the five responding organizations answered).
      • 2.
        Enhance outreach efforts to develop relationships with patient advocates (two of the five responding organizations answered).
      • Make it clear how patients can make contact to engage. Enhance outreach to patient groups to make initial contact easier.
      • Provide incentive to work collaboratively (patient groups can use the data and analytic results, too).
      • 3.
        Be transparent (three of the five responding organizations answered).
      • Adopt a clear timeline for the analysis and the conditions that must be met before the analysis is performed.
      • Take a balanced approach; denying coverage should not be the intent.
      4. Appreciate and accommodate patient community resource constraints (one of the five responding organizations answered).
      • There is opportunity cost.
      Table 5 lists the respondents’ observed/perceived impacts on value assessment final products. Some respondents reported that not enough time had transpired yet to fully realize impact.
      Table 5Patient advocacy organization reported impact of engagement on value assessment final products (survey reported, 6 of 13 organizations surveyed responded)
      Patient quality-of-life factors not included in initial criteria were added and made more specific to a variety of conditions (one of the six responding organizations answered).
      A subpopulation was highlighted in the related documents as a special need for treatment, as well as in the final vote result in the meeting as a result of our input (one of the six responding organizations answered).
      Changes to the model and analysis (i.e., the number of times patients had to cycle through a medication) after our organization presented our data at a public meeting (one of the six responding organizations answered).
      Incorporation of our organization’s and patients’ input was directly reflected in the final product (one of the six responding organizations answered).
      Not at a final product yet. Cautiously optimistic. The process is not yet complete (two of the six responding organizations answered).

       Focus Group

      The focus group participants included 12 of the 13 patient organizations (92%). There were also representatives from the University of Maryland, Baltimore, who listened in but did not participate in the discussion. The group discussed a wide variety of challenges and frustrations they experienced and then flipped the discussion from challenges to proposed good practices. The consensus list resulted in the 30 emerging good practices in Table 6.
      Table 6Recommended emerging good practices for patient engagement in value assessment
      TypeIt is good practice to…Problem this addresses for the patient community
      TimingEngage patients from the beginning of the process. Developers should seek patient input as early as possible.The patient community is often engaged when it is too late to have impact.
      Engage patient groups during the question- or problem-development stage, when scoping the issue/problem.Being engaged late can be viewed by patients as being thought of as a “rubber stamp” rather than a true partner.
      Begin with qualitative research in the patient community to verify the right questions are asked and assumptions are made.
      Make initial outreach easier for patients, provide a clear process with details on who/how to contact, and ensure proper and reasonable response times for comments.
      Methodology and DataInclude what is important to patients in analyses and assumptions, and find new and rigorous ways to use a range of novel data sources to do so, typically going beyond what is found in published randomized, controlled trial (RCT) results.Assumptions about patient views may be made that are inaccurate, misunderstood, and/or poorly supported.
      Use natural history of disease data from a variety sources (including patient-provided information) in constructing a base case scenario.
      Use well-founded, documented, real-world patient-provided information gained by working collaboratively with the patient community when making assumption about patient outcomes, needs, preferences, etc. (e.g., a registry or survey to fill gaps in literature).
      Patient-provided information on preferred outcomes and experiences often comes from non-RCT data sources, which are often rejected by developers.
      Use the data patient groups often have or can collect to contribute to the value assessment process.The patient community has or can provide data, but may not be asked or their data are not seriously considered.
      Proactively engage with the patient community to review the methodology that will be used and where other, non-RCT data that exists could be inserted.The patient community wants to contribute, but needs to understand the context and ground rules to allocate scarce resources wisely.
      Provide criteria that will be used to assess data for inclusion in an analysis or assessment versus what will be included in a text narrative.
      Engage with multiple patients and/or groups when possible to achieve representativeness.No one patient or group can provide all the information that may be needed or most useful.
      Provide a protocol for how the background literature is selected and incorporated into the report, is properly cited, and real or potential conflicts of interest are acknowledged and disclosed (e.g., sources, sponsorships, affiliations, etc.).This provides transparency to the patient community about why some literature and/or data are included versus excluded.
      Develop and implement a common framework or template set of survey questions (qualitative or quantitative) patient groups can use to collect data that will be used and recognized as has value in the process (e.g., can be used by multiple value framework developers, rigorous quality, applied more universally).Data collected by the patient community are often rejected by developers for various reasons, even when a developer and patient group have worked in partnership on data collection. The reasons are often related to perceived lack of rigor and/or lack of methodology existing to incorporate the data into an analysis.
      Develop and implement methods (e.g., adjustments or sensitivity analyses) and other techniques that can be applied to bring the patient perspective and outcomes most important to patients into a value framework when these outcomes have not been consistently included in RCTs but come from other sources.
      Provide transparent mechanisms for communicating changes and edits to documents over time.The patient community would like to be able to follow document changes, and changes made in response to their own and others’ comments.
      Provide transparent mechanisms for tracking developer responses to comments.
      Partnering with PatientsDevelop a partnership relationship with the patient community, with mutual respect and balance (e.g., a win-win scenario, both sides gain, not just patients giving information, data, time, etc.).The patient community has had a wide range of treatment experiences. Poor experiences include being excluded, treated with disrespect, expected to provide data/services for free (e.g., need to have “skin in the game”), and/or relegated to “outsider” roles. This has improved, but needs further attention.
      Provide a meaningful seat at the table for the patient community (e.g., seat(s) on governance-level steering committee with decision making capacity).
      Understand and, where possible, accommodate the needs and resource limitations of the patient community.The patient community wants to engage but can have limitations and resource constraints, due things like illness (may have travel limitations or require a place to rest periodically during a day-long meeting). Patient advocacy groups can have resource limitations that make quick turnaround times extremely difficult.
      Clearly state the objectives to ensure mutual understanding.
      Conduct a mutual evaluation of the patient community’s and developer’s engagement experience in a value assessment (i.e., Did patients feel engaged, listened to, or was it tokenistic? Did patient input change the approach?).Processes can be improved with a feedback loop.
      Establish a protocol for partnering with stakeholders. The protocol should include but is not limited to: how representativeness will be achieved; the targeted patient population(s); objectives for engagement; expectations/roles; and logistics such as number of hours, meetings, compensation, and anticipated final deliverables.The patient community wants to contribute but needs have a full picture and understand expectations to make an informed commitment to fulfill the role successfully.
      Characterizing EngagementAvoid misunderstandings about whether those who are acknowledged also endorse the findings.Patient–community partners fear their engagement could be mischaracterized or misinterpreted as endorsing final products, conclusions, or recommendations.
      Characterize the engagement accurately when citing or acknowledging patient–community participation; the type and level of engagement should be described; and whether or not those who were engaged endorse the final product.
      Create and use an engagement-activity checklist for developers and the patient community.The patient community may have difficulty articulating how they were or were not engaged. They may not know the full breadth of opportunities. They could use this list to check off the ways they were engaged. Developers could use it to describe in their protocol how they want to or have engaged stakeholders. It can be used in reporting to indicate how various stakeholders were engaged (e.g., similar to the lists used to indicate a researcher’s role in a study when manuscripts are submitted for publication).
      Acknowledge and disclose real, potential, or perceived conflicts of interest by all participants.All stakeholders have real, potential, or perceived conflicts that should be fully disclosed.
      Provide examples of how patient engagement had an impact on work products or learned experiences with engagement.Patient engagement on value assessment is new to the patient community and developers; sharing knowledge and experiences can help to improve processes and outcomes for all.
      Recommendations fell into four categories:
      • 1.
        Timing: These recommendations generally refer to when the engagement takes place, and encourages early and continuous engagement.
      • 2.
        Methodology and Data: This section describes approaches and data sources that enhance inclusion of the patient voice and transparency.
      • 3.
        Partnering with Patients: These recommendations enhance partnership and help to reduce burden on patient groups.
      • 4.
        Characterizing Engagement: These recommendations guide the way patient engagement is described publicly.
      The differentiation between emerging good practices that stemmed from “experienced” versus a “recommended” good practice (to mitigate an experienced challenge) was dropped as the distinction became moot in the focus group discussion. Some participants had an experience they thought would be a good practice, and it was subsequently elaborated upon. Other participants offered a recommended good practice in reaction to a challenge they had experienced, only to find out that someone else in the group had, in fact, experienced it (at least to some degree) and had offered it as an experienced good practice. An example is located under “Timing” in Table 6. One member wished they had been consulted during the scoping phase and others chimed in that they had been. For this reason, differentiating between experienced or recommended good practices became moot, and they were combined as one list of recommended emerging good practices. Therefore, the specific survey results as reported are depicted in Table 3, Table 4 as collected. However, Table 6 presents the information from the discussion where they are combined into one list.

      Discussion

      As the United States moves toward increased consumer-directed health care, there is greater focus on patient-centeredness and engagement in research, medical product development, health policy, health care delivery, and value assessment (often referred to as health technology assessment). It is not yet possible to enumerate “best practices” in meaningful patient engagement in any of these areas because, to date, not enough experience or evaluation data are available to determine what is “best” in this still nascent field [
      • Hoos A.
      • Anderson J.
      • Boutin M.
      • et al.
      Partnering with patients in the development and lifecycle of medicines: a call for action.
      ]. Patient engagement in all aspects of value assessment has the potential to transform value assessments to advance a value-driven health care ecosystem in the United States that is truly patient centered. Therefore, it is useful to leverage opportunities to glean from recent experiences of advocacy organizations that can form the basis for recommended emerging “good” practices, gaining insights into the ongoing impact of the engagement, and paving a path to “best” practices to improve engagement in the future.
      Patient advocates are becoming involved in a range of activities with framework developers (see Table 1). This helps confirm the expanding role they can play and the opportunities for contribution. During the focus group discussion, it was suggested that the wide-ranging list of activities be fleshed out and offered as a potential tool that can help patient groups describe how they have or can be involved and for framework developers to understand the breadth of potential opportunities for patient engagement. This tool could be the basis for characterizing patient engagement in a value assessment and elsewhere. That is, the patient group can check off the ways it perceives it was engaged, and value assessors can use the list to characterize the engagement that occurred to inform their assessment.
      These data serve as one snapshot in time as the environment continues to evolve. During the focus group discussion, on numerous occasions, it was pointed out that interactions between patient advocacy groups and value framework developers/assessors had expanded, changed, and improved since 2015. The cross-sectional ratings portion of the survey did not capture change over time, but the qualitative themes from the focus group indicated that change was, indeed, occurring. At that time, many patient groups did not know about value assessments at all, thus having no engagement. A smaller portion of the groups that were aware of value frameworks attempted to interact with value framework developers but had had negative experiences. However, recently, more engagement and more positive experiences were reported by patient advocacy groups. This engagement is likely to increase over time. Therefore, the findings from this study are also likely to change with increasing experiences of patient advocacy groups.
      It is worth noting that some of the recommended best practices offered are consistent with practices that are becoming more common in health technology assessment programs outside the United States. For example, the recommendation that framework developers “provide redline changes in successive drafts of documents so edits are readily apparent” is consistent with a change that the Canadian Health Technology Assessment body (CADTH) made to its evaluations. It now includes a paragraph on how and in which ways patient involvement impacted their decision making [

      Canadian Agency for Drugs and Technologies in Health. CADTH common drug review patient input. Available from: https://cadth.ca/about-cadth/what-we-do/products-services/cdr/patient-input. [Accessed May 25, 2017].

      ]. Additionally, the recommendations are in alignment with the way the Patient-Centered Outcomes Research Institute (PCORI) Engagement Rubric [

      Engagement Rubric PCORI. Patient-Centered Outcomes Research Institute. [Updated June 2016]. Patient-Centered Outcomes Research Institute. February 4, 2014. Available from: http://www.pcori.org/sites/default/files/Engagement-Rubric.pdf. [Accessed November 16, 2017].

      ] encourages patients and other stakeholders be involved in research. Although the focus and output are different, many themes are the same. For example, the PCORI Engagement Rubric describes the need for patient involvement in articulating a research question early in the research process. Similarly, the good practice recommendations encourage engagement with patients from the beginning of the value assessment process.
      Another key point is that the impact participants believed their input had was somewhat inconsistent across groups, with some reporting they had had an impact and others reporting it was too soon to tell what their impact was on the process. Of the 13 interactions the 10 groups had, four advocates reported they perceived they had some impact on the final product. Two others reported that it was just too soon for them to know but that they were “cautiously optimistic.” The remaining four groups did not answer this question and may possibly have skipped it because they also believed it was too soon to tell, rather than implying that they believed they had had no impact.
      In the focus group discussion, many participants reported their communications and interactions with value framework developers had improved over the last few years. But they also agreed that there remains a great deal of work to be done for true partnership to take place and the good practices proposed are a start to framing what true partnerships should entail. However, the perception that engagement had improved since early 2016 has important implications for interpreting the data. Notably, most groups had interactions with only one or two framework developers. Therefore, experiences were somewhat limited, and it is impossible to compare the framework developers with such limited data. However, even within the two framework developers with the most interactions (A and B), there was some variability within the ratings. One reason for this is that framework developers began changing their processes and procedures in 2016 with regard to patient engagement [
      • Perfetto E.M.
      • Oehrlein E.M.
      • Boutin M.
      • et al.
      Value to whom? The patient voice in the value discussion.
      ]. The interpretation could be that Framework Developer B’s poor and fair ratings were based on older experiences and that the very good ratings are more recent. One responder who gave Developer B a poor rating commented in the survey that the group had interacted early on, understood the process had changed since that time, and was looking forward to more collaborative interactions.
      It is clear the patient groups would like to see a variety of data and data sources used beyond what is published in randomized controlled trial results. The need for a range of data may seem to place a new and large burden on value framework developers. However, the respondents’ intent seemed directed to encouraging value framework developer openness to novel data sources, especially those from the patient community and not that the data needed to be necessarily collected by framework developers. They are eager to share the data they have available (e.g., registries and surveys), especially when it contains outcomes important to patients. This approach can open new avenues to framework developers; however, they need to believe that the data are valid, reliable, and generalizable and can be trusted. It will take time for the patient community and framework developers to develop this mutual trust.
      To help foster trust and better relationships between the patient community and framework developers, several of the noted recommendations (see Table 6) address logistical or operational issues. For example, the patient groups recommended framework developers provide clear processes to improve communication (e.g., Who to and how to contact for the framework developer organization?), including transparency on how background literature and data are selected, mechanisms for communicating changes and edits to documents, and tracking the responses of framework developers to comments over time; and for characterizing how patients were engaged in the value assessment.
      Last, it may seem as though implementation of all the best practices would be tremendously burdensome on value framework developers and that they are being asked to do more than their scope can allow or to do things they are not equipped for or have expertise in. Patient groups have seen a great deal of burden also when encountering a value assessment for their disease (e.g., having to hire consultants to help them understand and respond). This is a reasonable concern by both parties and one that should be part of the dialogue. With patient input from the early stages of framework development and with the establishment of sound, trusted relationships, some of the activities can be done collaboratively, thus easing the burden for all.
      It is important to consider the limitations of this study. It included a very small, convenience sample of patient groups. The sample may potentially be biased because it is not a random sample. The Workgroup that comprised the sample included patient advocacy group staff members, who had come to know each other and had been in contact regarding the topic of interest. However, at this time, only a small number of patient groups have had interactions on value assessment. It would be difficult to randomly derive a sample of patients or advocates who could respond to the questions asked. As one example, some members joined the Workgroup because they anticipated that they would be having interactions with a value framework developer in the following year. However, for various reasons, that did not happen (e.g., because of lack of approval of the product(s) that would have been the focus of the value assessment). Thus, one Workgroup member organization reported that it did not have experiences to report and could not complete the questionnaire. A large majority of patient advocacy organizations would be in the same situation, making this convenience sample possibly one of the only ways to elicit these experiences at this time.

      Conclusions/Recommendations

      Although patient engagement was limited in the development of early value frameworks, patient advocacy group engagement with value framework developers and assessors has increased in the past 2 years. Groups report positive experiences in engagement that can serve as emerging good practices. They have also experienced challenges in their interactions and proposed recommendations on good practices they believe can improve engagement experiences.
      The NHC will continue to provide leadership in this area by encouraging use of these emerging good practices and capturing data on patient experiences, monitoring trends, and growing the pool of patient engagement experiences that can be translated into good practices to advance a patient-centered, value-driven health care ecosystem. Learnings from these early experiences can help establish recommend emerging good practices that can eventually get the field to best practices and standards over time.

      Funding

      No financial support was received for the development of this article. However, the National Health Council receives membership and sponsorship funds from various organizations. Members and sponsors are listed on our website: nationalheathcouncil.org.

      Acknowledgements

      The authors, as the writing group for the National Health Council (NHC) Value Workgroup, submitted this article on behalf of the NHC. The NHC Value Workgroup includes the American Heart Association, the Arthritis Foundation, the Asthma and Allergy Foundation of America, the Cancer Support Community, the Familial Hypercholesterolemia Foundation, the Lung Cancer Alliance, the Immune Deficiency Foundation, the National Bone Health Alliance, the National Eczema Association, the National Multiple Sclerosis Society, the National Patient Advocate Foundation, the National Psoriasis Foundation, and the Parent Project Muscular Dystrophy. We acknowledge the contributions of all of the members of the Workgroup as well as that of Dr. Joey Mattingly, Assistant Professor, University of Maryland, Baltimore, and Ms. Elizabeth Oehrlein, PhD Candidate, University of Maryland, Baltimore.

      Supplementary Material

      References

        • Schnipper L.
        • Davidson N.
        • Wollins D.
        • et al.
        American Society of Clinical Oncology statement: a conceptual framework to assess the value of cancer treatment options.
        J Clin Oncol. 2015; 33
      1. Institute for Clinical and Economic Review. Value Assessment Project. Retrieved from: http://www.icer-review.org/impact-and outcomes/value-assessment-project/. [Accessed May 25, 2017].

      2. National Comprehensive Cancer Network. NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines) with NCCN Evidence Blocks. Retrieved from: http://www.nccn.org/evidenceblocks/. [Accessed May 25, 2017].

      3. Memorial Sloan Kettering Cancer Center. Welcome to Drug Abacus. Available from: http://www.drugabacus.org/drug-abacus-tool/. [Accessed May 25, 2017].

      4. National Health Council. The patient voice in value: the National Health Council patient-centered value model rubric. Available from: http://www.nationalhealthcouncil.org/sites/default/files/Value-Rubric.pdf. [Accessed May 25, 2017].

        • Perfetto E.M.
        • Oehrlein E.M.
        • Boutin M.
        • et al.
        Value to whom? The patient voice in the value discussion.
        Value Health. 2017; 20: 286-291
        • Sorenson C.
        • Lavezzari G.
        • Daniel G.
        • et al.
        Advancing value assessment in the United States: a multistakeholder perspective.
        Value Health. 2017; 20: 299-307
      5. FasterCures—A Center of the Milken Institute, Avalere Health. Patient-Perspective Value Framework (PPVF) Version 1.0. May 2017. Available from: http://www.fastercures.org/assets/Uploads/PPVF-Version-1.0-Methodology-Report-Final.pdf. [Accessed May 25, 2017].

      6. National Health Council. Value Initiative. Available from: http://www.nationalhealthcouncil.org/value-initiative. [Accessed May 25, 2017].

      7. National Health Council. Get-Ready Checklist. Available from: http://www.nationalhealthcouncil.org/sites/default/files/Get_Ready_Checklist.pdf. [Accessed May 25, 2017].

      8. National Psoriasis Foundation. ICER Q&A with Leah Howard. October 2016. Available from: https://www.psoriasis.org/advance/icer. [Accessed May 25, 2017].

      9. Arthritis Foundation. Arthritis Foundation proclaims a patient advocacy “win” through its work with the Institute for Clinical and Economic Review (ICER). April 2017. Available from: http://www.arthritis.org/about-us/news-and-updates/arthritis-foundation-proclaims-a-patient-advocacy-win-through-its-work-with-icer.php. [Accessed May 25, 2017].

        • Hoos A.
        • Anderson J.
        • Boutin M.
        • et al.
        Partnering with patients in the development and lifecycle of medicines: a call for action.
        Ther Innov Reguly Sci. 2015; 49: 929-939
      10. Canadian Agency for Drugs and Technologies in Health. CADTH common drug review patient input. Available from: https://cadth.ca/about-cadth/what-we-do/products-services/cdr/patient-input. [Accessed May 25, 2017].

      11. Engagement Rubric PCORI. Patient-Centered Outcomes Research Institute. [Updated June 2016]. Patient-Centered Outcomes Research Institute. February 4, 2014. Available from: http://www.pcori.org/sites/default/files/Engagement-Rubric.pdf. [Accessed November 16, 2017].