Advertisement

Patient-Reported Outcome Measures in the Food and Drug Administration Pilot Compendium: Meeting Today’s Standards for Patient Engagement in Development?

Open ArchivePublished:March 09, 2018DOI:https://doi.org/10.1016/j.jval.2018.01.004

      Abstract

      Background

      In 2016, the Food and Drug Administration (FDA) released a Pilot Clinical Outcome Assessment Compendium (COA Compendium) intended to foster patient-focused drug development (PFDD). However, it is unclear whether patient perspectives were solicited during development or validation of the included patient-reported outcome (PRO) measures.

      Objective

      To examine the pedigree of a sample of measures included in the COA Compendium.

      Methods

      PROs included in chapters 1 or 2 of the COA Compendium were extracted and three reviewers independently searched PubMed and Google to identify information on measure pedigree. Data on method and stage of measure development where patient engagement took place were documented.

      Results

      Among the 26 evaluated PRO measures, we were unable to identify information on development or validation on nearly half the sample (n = 12). Among the remaining 14 measures, 5 did not include any evidence of patient engagement; 2 engaged patients during concept elicitation only; 1 engaged patients during psychometric validation only; and 6 engaged patients during both concept elicitation and cognitive interviewing. Measures either previously qualified or submitted for qualification were more likely to include patient engagement.

      Conclusions

      For the FDA Pilot COA Compendium to fulfill its purpose of fostering PFDD, it needs fine-tuning to reflect today’s standards, improving transparency and facilitating clear identification of included measures so that the level of patient engagement, among other factors, can be properly assessed. Suggested improvements include identifying clinical trials that correspond to the COA Compendium’s use in drug development; more clearly identifying which measure is referred to; and including only those measures that already qualified or undergoing qualification.

      Keywords

      Introduction

      Over the past decade, the US Food and Drug Administration (FDA) has led a movement to improve the validity and relevance of clinical outcome assessment (COA) tools to support labeling [

      Food and Drug Administration. Clinical Outcome Assessment Qualification Program submissions. Available from: https://www.fda.gov/Drugs/DevelopmentApprovalProcess/DrugDevelopmentToolsQualificationProgram/ucm450689.htm. [Accessed June 1, 2017].

      ]. To help drug and COA instrument developers navigate these changes, the FDA published a “Roadmap to Patient-Focused Outcome Measurement in Clinical Trials,” along with the 2009 Guidance on Patient-Reported Outcomes (PROs) [

      Food and Drug Administration. Roadmap to patient-focused outcome measurement in clinical trials. Available from: http://www.fda.gov/downloads/Drugs/DevelopmentApprovalProcess/DrugDevelopmentToolsQualificationProgram/UCM370174.pdf. [Accessed June 1, 2017].

      ,
      Food and Drug Administration
      ]. Adherence to the guidance improves the likelihood that COAs (PROs, clinician-reported outcomes, observer-reported outcomes, and performance outcomes) used to measure treatment benefit in drug approval trials will identify meaningful treatment benefit (e.g., how a patient survives, feels, or functions) as defined by patients [

      Food and Drug Administration. Roadmap to patient-focused outcome measurement in clinical trials. Available from: http://www.fda.gov/downloads/Drugs/DevelopmentApprovalProcess/DrugDevelopmentToolsQualificationProgram/UCM370174.pdf. [Accessed June 1, 2017].

      ]. They also promote rigorous methodology in developing and validating COAs by describing FDA considerations on content validity and reliability, among others.
      In 2016, the FDA released a Pilot Clinical Outcome Assessment Compendium (COA Compendium) [

      Food and Drug Administration. Pilot Clinical Outcome Assessment Compendium (COA Compendium) Version 1. Available from: https://www.fda.gov/downloads/Drugs/DevelopmentApprovalProcess/DevelopmentResources/UCM481225.pdf. [Accessed June 29, 2016].

      ]. The COA Compendium is a list of COAs used as primary or secondary end points in trials and discussed in product labels between 2003 and 2014. The COA Compendium is part of FDA’s efforts to “foster patient-focused drug development (PFDD) by collating and summarizing COA information for many different diseases and conditions into a single resource intended to: (1) facilitate communication; (2) provide clarity and transparency; and (3) be used as a starting point for early drug development” [

      Food and Drug Administration. Enhancing the development and use of patient-reported outcomes in drug development: July 16, 2014 meeting summary. Available from: https://www.brookings.edu/wp-content/uploads/2014/07/PRO-Expert-workshop-1_Meeting-Summary.pdf. [Accessed June 1, 2017].

      ,
      • Perfetto E.M.
      • Burke L.
      • Oehrlein E.M.
      • Epstein R.S.
      Patient-focused drug development: a new direction for collaboration.
      ,

      Food and Drug Administration. Pilot Clinical Outcome Assessment Compendium (COA Compendium) Webinar (March 8, 2016). Available from: https://www.fda.gov/downloads/Drugs/DevelopmentApprovalProcess/SmallBusinessAssistance/UCM489439.pdf. [Accessed June 25, 2016].

      ]. It seeks to help drug developers overcome some of the logistical barriers related to including COAs in clinical trials, particularly in identifying existing measures or measures that could be modified on the basis of context of use [

      Food and Drug Administration. Roadmap to patient-focused outcome measurement in clinical trials. Available from: http://www.fda.gov/downloads/Drugs/DevelopmentApprovalProcess/DrugDevelopmentToolsQualificationProgram/UCM370174.pdf. [Accessed June 1, 2017].

      ].
      Despite FDA’s disclaimers on what the COA Compendium includes and does not include (see Table 1, Table 2), the COA Compendium may be perceived as an endorsement of included measures [

      Food and Drug Administration. Clinical Outcome Assessment Qualification Program submissions. Available from: https://www.fda.gov/Drugs/DevelopmentApprovalProcess/DrugDevelopmentToolsQualificationProgram/ucm450689.htm. [Accessed June 1, 2017].

      ]. Given the time frame of COAs eligible for inclusion in the COA Compendium, many are likely to predate the December 2009 release of FDA’s Guidance on PRO Measures to Support Labeling Claims [
      Food and Drug Administration
      ]. There are concerns that the COA Compendium includes measures that would not be accepted as a well-defined and reliable assessment of a specified concept of interest. Furthermore, although the COA Compendium is intended to “foster patient-focused drug development,” there are concerns that COA Compendium-listed measures do not measure a concept of interest that reflects outcomes that matter to patients. Indeed, with the exception of just two measures (Evaluating Respiratory Symptoms in Chronic Obstructive Pulmonary Disease [E-RS: COPD] and Exacerbations of Chronic Pulmonary Disease Tool [EXACT]), the rest of the measures included in the COA Compendium have not gone through the formal COA qualification process and references to published validation studies are not included in the COA Compendium [

      Food and Drug Administration. Pilot Clinical Outcome Assessment Compendium (COA Compendium) Version 1. Available from: https://www.fda.gov/downloads/Drugs/DevelopmentApprovalProcess/DevelopmentResources/UCM481225.pdf. [Accessed June 29, 2016].

      ]. While measures can be well developed and suitable for the assessment of treatment benefit without undergoing FDA’s qualification program, for example, COAs developed as part of an individual drug development program, data on these measures are not required to be made publicly available. Thus, measure developers may or may not have followed the 2009 guidance or provide sufficient information to document validity or reliability.
      Table 1FDA-Identified key considerations of the COA Compendium

      Food and Drug Administration. Pilot Clinical Outcome Assessment Compendium (COA Compendium) Version 1. Available from: https://www.fda.gov/downloads/Drugs/DevelopmentApprovalProcess/DevelopmentResources/UCM481225.pdf. [Accessed June 29, 2016].

      • “The COA Compendium (PDF) is not a comprehensive list of clinical outcome assessments and is not intended to replace either existing disease-specific guidance or key interactions with FDA concerning drug development (e.g., during pre-IND meetings). Inclusion of a clinical outcome assessment in the COA Compendium is not intended to indicate that the measure is or should be the sole (or primary) determinant of a clinical benefit in a clinical trial.”
      • “Drug sponsors are strongly encouraged to seek advice from the relevant Office of New Drug (OND) review division early in drug development to discuss the selection and implementation of the clinical outcome assessment specific to their program, irrespective of whether the disease, condition, indication, claim, or clinical outcome assessment is included in the COA Compendium.”
      • “Some of the clinical outcome assessments listed in the COA Compendium may be protected by proprietary rights, and in some cases, a royalty and fee may be charged by the copyright owners for their authorized use. The inclusion of a clinical outcome assessment in the COA Compendium does not equate to an endorsement by FDA.”
      COA, clinical outcome assessment; FDA, Food and Drug Administration.
      Table 2FDA-Identified limitations associated with the COA Compendium
      • Rahman A.
      • Rizwan S.
      • Waycaster C.
      • Wall G.M.
      Pooled analysis of two clinical trials comparing the clinical outcomes of topical ciprofloxacin/dexamethasone otic suspension and polymyxin B/neomycin/hydrocortisone otic suspension for the treatment of acute otitis externa in adults and children.
      • “The COA Compendium is not a comprehensive list of all medical conditions or clinical outcome assessments that could potentially support labeling claims.
        • o
          It should be underscored that the current pilot version of the COA Compendium is limited in scope—that is, it is primarily based on the retrospective review of NME labeling approved from 2003 to 2014 and excludes all efficacy supplements.
        • o
          Clinical outcome assessments not included in the COA Compendium should also be considered during drug development, as appropriate, especially those supported by data, literature, and good measurement principles.
      • The COA Compendium is not a replacement for interactions with appropriate FDA review divisions nor does it supersede existing disease-specific guidance. For example:
        • o
          The COA Compendium omits critical aspects of how a listed clinical outcome assessment could be implemented in a clinical trial (e.g., clinical trial design).
        • o
          Inclusion of a clinical outcome assessment in the COA Compendium is not intended to indicate that the measure is or should be the sole determinant of a clinical benefit in a clinical trial. Other assessments, such as overall survival, may be critical drivers of establishing efficacy or clinical benefit.
        • o
          End-point hierarchy and selection of key outcome assessments are always specific to the context of a drug candidate in a therapeutic area and should be discussed with the appropriate review division prior to initiating clinical trials.”
      COA, clinical outcome assessment; FDA, Food and Drug Administration.
      Furthermore, there is no documentation included in the COA Compendium on the level of patient engagement in the development of any of the included measures. With heightened awareness of the need for patient engagement in all aspects of health care and research, it would inform use and interpretation to understand whether the PRO measures included in the COA Compendium were grounded with patient input from inception. To better understand the pedigree of measures listed in the COA Compendium and to gauge its potential contribution to PFDD, this scoping review attempted to identify when and how patients were engaged during the development of a sample of PRO measures included in the COA Compendium.

      Scoping Review Methods

      COA measures included in either the COA Compendium’s first (Office of Microbial Products) or second (Office of Drug Evaluation I) chapter were stratified by type of COA (clinician-reported outcome, observer-reported outcome, PRO, or performance outcome). These chapters were selected to cover a broad range of products including anti-infective, antiviral, transplant, ophthalmology, cardiovascular, renal, neurology, and psychiatry. The list of PROs was extracted from the COA Compendium chapters as the study sample. Three reviewers independently searched PubMed and Google to identify information on the origins and validation of these PRO measures. Searches were conducted between February and July 2017.
      To identify patient engagement activities, reviewers were instructed to look for any information related to the following: How was concept elicitation operationalized? Was cognitive debriefing mentioned? If yes, how was it operationalized? Were patients involved as test subjects only? Reviewers were also asked to document any other information pertinent to understanding how the measure was developed or validated, especially with regard to patient involvement.
      The first two reviewers captured all information in a data extraction template, which was then verified by a third researcher, an expert in PRO development and validation, through repetition of the scoping review methods. If patient engagement was noted to have taken place, data on patient engagement method used and stage of measure development where patient engagement took place were documented and all sources were cataloged and recorded to facilitate comparisons between reviewer findings [
      • Miles M.B.
      • Huberman A.M.
      • Saldana J.
      Qualitative Data Analysis: A Methods Sourcebook.
      ]. Discrepancies in intercoder reliability were addressed through data triangulation between the study team lead and the PRO expert. The discrepancies were discussed and an agreement was reached regarding whether patient engagement took place and the level at which patient engagement occurred. Criteria are described in detail in Table 3. For the instruments for which the search yielded no information on development or validation, instrument developer identity and contact information was also unfortunately not identifiable, preventing further analysis.
      Table 3Scoping review results by level of patient engagement
      Level of patient engagementNo. of PROs (N = 26)No. submitted for qualification (N = 8)PRO Type
      Identifiable and search possibleN = 14N = 8
      • Engagement identified:N = 6N = 6• Disease-specific questionnaire (N = 6)
      • Concept elicitation and cognitive interviewing
      • Identified but weak:N = 3N = 1• Disease-specific questionnaire (N = 3)
      • Patients included as study subjects for psychometric validation only; OR
      • Patients included for concept elicitation only
      • No engagement identifiedN = 5N = 1
      • Disease-specific questionnaire (N = 1)
      • Disease-specific patient diary (N = 2)
      • Disease-specific PRO rating scale (N = 2)
      • No information identified for patient engagement during PRO development
      Not identifiable enough to support a searchN = 12N = 0
      • Disease-specific patient diary (N = 9)
      • Disease-specific PRO four-point pain scale (N = 1)
      • Disease-specific PRO five-point itching scale (N = 1)
      • Disease-specific PRO VAS and six-point numeric pain rating scale (N = 1)
      • Unable to locate information on development or validation
      PRO, patient-reported outcome; VAS, visual analogue scale.

      Scoping Review Results

      In total, 26 PROs were identified among all COA measures listed in the first two chapters of the COA Compendium. Among the 26 evaluated PROs, no information was identified on the development or validation of nearly half (n = 12). These included primarily patient diaries (n = 9) and visual analogue or other numeric scales (n = 3). Among the PRO measures for which information on development or validation was identified, 6 of 14 had clearly documented patient involvement during the PRO instrument’s development (see Table 3). Patient involvement included patients’ participation in concept elicitation/item generation (n = 6) achieved through focus groups (n = 1) and/or interviews (n = 6). Other PROs (n = 3) included very limited patient involvement, for example, to validate clinician-generated items (n = 2), through a “patient focus group” to complement medical literature and as expert input with no details provided (n = 1). Finally, among PRO measures for which information on development/validation was identified, 5 of 14 included no information on patient engagement. Information on an additional PRO was located; however, this PRO relied solely on clinical expertise (n = 1).
      During the initial review, reviewers agreed on the categorization of patient engagement level for all except one PRO measure, the 12-Item Multiple Sclerosis Walking Scale. It was determined that although patient engagement was referenced during the generation of items, no cognitive debriefing of the instrument occurred. A decision was made to categorize the measure as reflecting “identified but weak” patient engagement rather than “identified” patient engagement.

      Discussion

      A key focus of PFDD is to use meaningful patient engagement to capture concepts of interest most important to patients [
      • Perfetto E.M.
      • Burke L.
      • Oehrlein E.M.
      • Epstein R.S.
      Patient-focused drug development: a new direction for collaboration.
      ]. It is essential to involve patients in the development and validation of PRO measures in order to translate what is most important to them into end points in clinical trials. Downstream PFDD efforts, such as patient engagement during value assessments or individual treatment choice, rely on the availability of patient-centered outcomes information to inform decision making. FDA’s “Roadmap to Patient-Focused Outcome Measurement in Clinical Trials,” the 2009 Guidance on Patient-Reported Outcomes (PROs), and the International Society for Pharmacoeconomics and Outcomes Research’s report on emerging good practices provide the blueprints for capturing relevant treatment benefit as characterized by patients themselves [
      • Benjamin K.
      • Vernon M.K.
      • Patrick D.L.
      • et al.
      Patient-reported outcome and observer-reported outcome assessment in rare disease clinical trials: an ISPOR COA Emerging Good Practices Task Force Report.
      ,
      • Patrick D.L.
      • Burke L.B.
      • Gwaltney C.J.
      • et al.
      Content validity—establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO Good Research Practices Task Force report: part 1—eliciting concepts for a new PRO instrument.
      ,
      • Patrick D.L.
      • Burke L.B.
      • Gwaltney C.J.
      • et al.
      Content validity—establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO Good Research Practices Task Force report: part 2—assessing respondent understanding.
      ,
      • Rothman M.
      • Burke L.
      • Erickson P.
      • et al.
      Use of existing patient-reported outcome (PRO) instruments and their modification: the ISPOR Good Research Practices for Evaluating and Documenting Content Validity for the Use of Existing Instruments and Their Modification PRO Task Force report.
      ].
      Our findings indicate that most of the PRO measures in the sample examined violate a fundamental premise of PFDD and COA development: that patients are engaged in the development. Only six measures included patient engagement during both concept elicitation and cognitive interviewing. Because patient engagement is a recognized step in adhering to FDA’s Roadmap to Patient-Focused Outcome Measurement in Clinical Trials, it is likely that most PROs in our sample, and likely within the COA Compendium as a whole, would not be qualified by the FDA. In addition, this review identified a number of considerations for researchers interested in using the current COA Compendium, but also for FDA as it refines the document.

       Time Frame of COA Measures Included in the COA Compendium Predates FDA’s 2009 Guidance on PROs

      Draft FDA Guidance on developing PROs for Use in Medical Product Development to Support Labeling Claims was released in 2006 and final Guidance in December 2009 [
      Food and Drug Administration
      ,
      Guidance for Industry
      Patient-reported outcome measures: use in medical product development to support labeling claims.
      ]. As such COA measures included in labeling before 2009 did not have exposure to FDA final guidance recommendations during development. Similarly, even among products approved after 2009, clinical development plans were likely finalized before the release of the final guidance. A recent review of PRO instruments for heart failure by Psotka et al. [
      • Psotka M.A.
      • von Maltzahn R.
      • Anatchkova M.
      • et al.
      Patient-reported outcomes in chronic heart failure: applicability for regulatory approval.
      ] concluded that none of the currently available PRO instruments would meet all of FDA’s recommendations. Among these is the Kansas City Cardio-Myopathy Questionnaire (KCCQ; published in 2000), which is included in the COA Compendium [
      • Green C.P.
      • Porter C.B.
      • Bresnahan D.R.
      • Spertus J.A.
      Development and evaluation of the Kansas City Cardiomyopathy Questionnaire: a new health status measure for heart failure.
      ]. Although the KCCQ developers met many of FDA’s recommendations, including performing patient focus groups to inform item generation, Psotka et al.’s [
      • Psotka M.A.
      • von Maltzahn R.
      • Anatchkova M.
      • et al.
      Patient-reported outcomes in chronic heart failure: applicability for regulatory approval.
      ] review suggests that the KCCQ did not have documented content validity and the recall period may deviate from FDA recommendations.

       “Better” PRO Measures May Be Currently in the Pipeline

      Because the COA Compendium includes only those measures that were included in product labels or went through the qualification process (to date only two measures), more patient-centered and methodologically rigorous PROs may exist, but are not yet included in the COA Compendium. In their comment letter to the FDA, the International Society for Quality of Life Research (ISOQOL) questioned FDA’s decision to include COAs simply based on precedence [

      International Society for Quality of Life Research. ISOQOL commentary on FDA Compendium. Available from: http://www.isoqol.org/UserFiles/file/ISOQOLFDACompendiumCommentaryFinalDraft.pdf. [Accessed July 19, 2017].

      ]. For example, the COA Compendium includes a patient diary for Clostridium difficile-associated diarrhea. This review did not yield any information on the development or validation of that patient diary. However, a different PRO measure for the same illness, the Cdad-Daysyms™, which was developed on the basis of FDA’s PRO Guidance, was identified during the review [
      • Kleinman L.
      • Talbot G.H.
      • SchÃler R.
      • et al.
      Cdad-Daysymsâ: a new patient-reported outcome tool for Clostridium difficile-associated diarrhea.
      ]. Items for the Cdad-Daysyms were drafted following interviews with patients and refined based on subsequent patient interviews. Although the Cdad-Daysyms measure has not yet been “qualified” by the FDA, it is likely a better starting place for drug developers than the previously used patient diaries. However, it is not yet included in the COA Compendium as its development was post-2013.
      In addition, the absence of publicly available information for measures developed by sponsors for specific submissions can serve as a barrier to uptake or critical evaluation of a measure’s development or validation [
      • Frost M.H.
      • Reeve B.B.
      • Liepa A.M.
      • et al.
      What is sufficient evidence for the reliability and validity of patient-reported outcome measures?.
      ,
      • Doward L.C.
      • Gnanasakthy A.
      • Baker M.G.
      Patient-reported outcomes: looking beyond the label claim.
      ,
      • Newman J.C.
      • Feldman R.
      Copyright and open access at the bedside.
      ]. Because measures listed in the COA Compendium are based on precedence, proprietary measures may be included in the COA Compendium. This (unavailability of information) is likely to be exacerbated as drug developers, increasingly for rare diseases, develop disease-specific measures for specific conditions and it may not be in their interest to share for strategic reasons [
      • Canestaro W.J.
      • Edwards T.C.
      • Patrick D.L.
      Systematic review: patient-reported outcome measures in coeliac disease for regulatory submissions.
      ].

       Lack of Granularity Makes It Difficult to Track the Pedigree of Instruments

      This scoping review yielded little or no data on the pedigree of approximately half the sampled PROs (n = 12). COA Compendium descriptions are vague and products approved using those end points are not identified in the COA Compendium. Thus, it is difficult and time consuming to try to identify specifically which PRO measures are being referred to, especially for patient diaries or where visual analogue scales are referenced. This is consistent with ISOQOL’s suggestions for improving the utility of the COA Compendium. ISOQOL suggested that it would be “useful to know which year and how often the individual COAs appeared in product labels.” It also recommend further details on “the context of use, the labeling claim and the place of the COA in the hierarchy of the endpoints” [

      International Society for Quality of Life Research. ISOQOL commentary on FDA Compendium. Available from: http://www.isoqol.org/UserFiles/file/ISOQOLFDACompendiumCommentaryFinalDraft.pdf. [Accessed July 19, 2017].

      ]. Others have similarly called for more granular information to increase the COA Compendium’s utility [

      Letter to FDA: Re: Docket No. FDA-2015-N-5106: Clinical Outcome Assessment Compendium. Available from: https://www.bio.org/sites/default/files/2016-03-14%20BIO%20Comments%20on%20COA%20Compendium%20FINAL.pdf. [Accessed July 19, 2017].

      ].
      As an example, the COA Compendium refers to a five-point verbal rating scale to measure patient satisfaction with treatment for varicose veins. Because there is no further information provided, including no reference to which products they were used for, identifying more information about the five-point scale is challenging. During the scoping review, we identified a clinical trial that had included a five-point PA-V3 score, which is a patient’s self-assessment of the appearance of one’s varicose veins [
      • Todd K.L.
      • Wright D.I.
      Durability of treatment effect with polidocanol endovenous microfoam on varicose vein symptoms and appearance (VANISH-2).
      ]. However, only a conference abstract describing its validation was identified [
      Validation of new patient/clinician-reported outcome tool—capturing the visual impact of varicose veins.
      ]. A 2008 review by Vasquez and Munschauer [
      • Vasquez M.A.
      • Munschauer C.E.
      Venous Clinical Severity Score and quality-of-life assessment tools: application to vein practice.
      ] identified a number of quality-of-life instruments for use in vein disease; however, none of these measures examined patient satisfaction with treatment. More recently, the VVSymQ® instrument was developed with FDA Guidance serving as a framework for development; however, this measure is not included in the COA Compendium [
      • Paty J.
      • Turner-Bowker D.M.
      • Elash C.A.
      • Wright D.
      The VVSymQ(R) instrument: use of a new patient-reported outcome measure for assessment of varicose vein symptoms.
      ,
      • Wright D.D.
      • Paty J.
      • Turner-Bowker D.M.
      • Bradbury A.
      Psychometric evaluation of a new patient-reported outcome (PRO) symptom diary for varicose veins: VVSymQ(R) Instrument.
      ,
      • Paty J.
      • Elash C.A.
      • Turner-Bowker D.M.
      Content validity for the VVSymQ(R) Instrument: a new patient-reported outcome measure for the assessment of varicose veins symptoms.
      ].

       Reliability of Some Measures Is Suspect

      Related to the lack of granularity is the inability to assess reliability. For example, the COA Compendium includes the “5-point itching severity numerical rating scale with half-unit increments” for allergic conjunctivitis. The scoping review yielded information on an itching scale for allergic conjunctivitis originating in the 1980s. Nevertheless, it is unclear how or when it was developed. It seems that Abelson et al. [
      • Abelson M.B.
      • Chambers W.A.
      • Smith L.M.
      Conjunctival allergen challenge: a clinical approach to studying allergic conjunctivitis.
      ] developed the scale and published it as a table in one article, but the development process was not described. A 2003 follow-up article by Abelson et al. [
      • Abelson M.B.
      • Loeffler O.
      Conjunctival allergen challenge: models in the investigation of ocular allergy.
      ] criticized the reliability of these scales, stating that “diaries contain a high level of subjectivity owing to differences in symptom interpretation among people.” Thus, if this is the “5-point itching severity numerical rating scale with half-unit increments” referenced by the COA Compendium, it is very unlikely that patient engagement was incorporated into instrument development; it is also unlikely that the measure would meet other requirements outlined within FDA Guidance.
      Although including PROs or other COAs with limited content validity and reliability may appear useful in providing a “starting point” for developing a new measure, in fact, inclusion of these measures may be counterproductive. For example, a review of reasons for rejection of PRO label claims found that “fit for purpose” issues, such as insufficient data documenting the validity of instruments, was the most common reason for the denial of PRO data to support labeling claims [
      • DeMuro C.
      • Clark M.
      • Mordin M.
      • et al.
      Reasons for rejection of patient-reported outcome label claims: a compilation based on a review of patient-reported outcome use among new molecular entities and biologic license applications, 2006–2010.
      ]. In certain diseases, PROs are the only information from which treatment benefit can be derived. In these instances, PRO pedigree is particularly important. If developers and users are not following the 2009 guidelines or at least not providing sufficient information that they did so, patient input is only one of the problems preventing the use of COAs in labeling.
      To illustrate: The COA Compendium describes the following measure for acute otitis externa: a “composite of clinician-reported outcome for erythema and edema, and patient-reported outcome for pain (4-point scale).” The scoping review uncovered the use of a four-point scale for inflammation and edema and a binomial scale for ear tenderness and discharge as primary outcomes (otic inflammation, edema, pain, and discharge) in a study examining a treatment for acute otitis externa in adults and children [
      • Rahman A.
      • Rizwan S.
      • Waycaster C.
      • Wall G.M.
      Pooled analysis of two clinical trials comparing the clinical outcomes of topical ciprofloxacin/dexamethasone otic suspension and polymyxin B/neomycin/hydrocortisone otic suspension for the treatment of acute otitis externa in adults and children.
      ]. Patient engagement was not reported to be involved for the development of these scales. If pain is a COA, the FDA PRO guidance clearly states that the patient should be queried [
      Food and Drug Administration
      ]. Both the COA Compendium’s lack of granularity to specify the exact instrument and the likelihood the PRO is not patient informed as uncovered by the scoping review would lead to a failure to meet the requirements of FDA’s PRO Guidance. This raises issue not only about the validity of the PRO included in the COA Compendium but also about the practical use and application of the COA Compendium.

       Patient Diaries and Visual Analogue Scales Require Particular Scrutiny

      It was particularly difficult to identify development or validation information regarding patient diaries and visual analogue or other numeric scales. For example, among 11 patient diaries in our sample, we were unable to identify any validation information for 9 of them. When information was identified, it was unlikely to include evidence of any patient engagement (the 2 patient diaries with information fell under “no engagement identified” category). For example, the COA Compendium references a patient diary for non–24-hour sleep-wake disorder. Although we did not identify a diary specific to non–24-hour sleep-wake disorder, this may refer to the Pittsburgh Sleep Diary. Although methods for deriving items in the Pittsburgh Sleep Diary are not described in detail, it appears that the diary represents a compilation of several older instruments. The 1994 instrument does not specifically mention patient engagement in its development article [
      • Monk T.H.
      • Reynolds III, C.F.
      • Kupfer D.J.
      • et al.
      The Pittsburgh Sleep Diary.
      ]. Similarly, a patient diary for traveler’s diarrhea is referenced, but not specifically named. Although there are data-collection diaries used in clinical trials as primary end points, no information on the development of these diaries could be found [
      • Dupont H.L.
      • Jiang Z.D.
      • Belkind-Gerson J.
      • et al.
      Treatment of travelers’ diarrhea: randomized trial comparing rifaximin, rifaximin plus loperamide, and loperamide alone.
      ]. The studies simply state that a diary was used for the primary end point in the clinical trial.

      Improving the COA Compendium to Foster PFDD

      Difficulty locating information on the validation of almost half of the measures included in this review raises wider questions about the validity of outcome information included in labels of already approved medications. Among sampled PRO measures, only 6 of 26 included patient engagement during both concept elicitation and cognitive interviewing, indicating that most included measures may do little to advance the COA Compendium’s stated goal of “fostering PFDD.” This review provides an eye-opening reminder that historic norms of using “expert opinion” to create PRO measures cannot be the basis of patient-centered or evidence-based medicine. For the COA Compendium to help facilitate PFDD, several improvements could be made. These include more clearly identifying which measure is being referred to; identifying literature that reports the development of the measure; identifying clinical trials that correspond to COA Compendium’s use in drug development; and including only those measures that are already qualified or undergoing qualification. References to the development or clinical trial use of the COAs are instrumental in understanding the pedigree of the COA. This would help researchers and others identify 1) whether the instrument is methodologically valid; 2) characteristics of patients engaged; and 3) how patients were engaged throughout the development and validation processes. Increased granularity by naming the specific instrument rather than simply stating “patient diary” would also increase transparency. Clearly articulated guidelines and methodology for COA inclusion in the COA Compendium would also improve usability. This review found that PROs submitted for qualification, even if they had not yet been qualified, were more likely to have included a high level of patient engagement during development and validation. Thus, the FDA may wish to include only those measures that are already qualified or currently undergoing qualification.

      Study Limitations

      This scoping review examined only the first two chapters of the Pilot COA Compendium. There may be other important considerations for researchers interested in using the document that did not come to light reviewing these two chapters alone. In addition, emerging themes may not apply to all therapeutic areas, or to COAs beyond PROs (e.g., clinician engagement in development of clinician-reported outcomes). Three research assistants identified information regarding the development and validation of the sampled PRO measures. Although best effort was made to identify all relevant information, it is possible that some information may have been missed.

      Conclusions

      At the heart of patient-centered measurement is the patient. The results of this scoping review highlight changes seen in PRO development over the last 20 years, from little to no patient engagement to a standard of requiring patient engagement from the beginning. Despite wide dissemination of these expectations, it appears that lack of patient input is a symptom of a more important problem: the 2009 guidance is not being followed systematically, or at least developers are not providing sufficient information that they did so. Proper development of PRO measures is critical for all downstream PFDD activities. Without meaningful patient engagement in the development of PRO measures, the reliability and true value-add in patient-centered benefit-risk assessment, patient-centered value assessment, and health technology assessment become uncertain. Weaknesses in validity and lack of transparency in development for most of the measures included in this review raise issue regarding not only the validity of measures included in the COA Compendium but also about the practical use and application of the COA Compendium. For the FDA COA Compendium to fulfill its purpose of fostering PFDD, the COA Compendium needs fine-tuning to reflect today’s standards, improving transparency and facilitating clear identification of included measures so that level of patient engagement, among other factors, can be properly assessed.

      References

      1. Food and Drug Administration. Clinical Outcome Assessment Qualification Program submissions. Available from: https://www.fda.gov/Drugs/DevelopmentApprovalProcess/DrugDevelopmentToolsQualificationProgram/ucm450689.htm. [Accessed June 1, 2017].

      2. Food and Drug Administration. Roadmap to patient-focused outcome measurement in clinical trials. Available from: http://www.fda.gov/downloads/Drugs/DevelopmentApprovalProcess/DrugDevelopmentToolsQualificationProgram/UCM370174.pdf. [Accessed June 1, 2017].

        • Food and Drug Administration
        Guidance for Industry. Patient-reported outcome measures: use in medical product development to support labeling claims. 2009; (Available from) ([Accessed June 29, 2016].)
      3. Food and Drug Administration. Pilot Clinical Outcome Assessment Compendium (COA Compendium) Version 1. Available from: https://www.fda.gov/downloads/Drugs/DevelopmentApprovalProcess/DevelopmentResources/UCM481225.pdf. [Accessed June 29, 2016].

      4. Food and Drug Administration. Enhancing the development and use of patient-reported outcomes in drug development: July 16, 2014 meeting summary. Available from: https://www.brookings.edu/wp-content/uploads/2014/07/PRO-Expert-workshop-1_Meeting-Summary.pdf. [Accessed June 1, 2017].

        • Perfetto E.M.
        • Burke L.
        • Oehrlein E.M.
        • Epstein R.S.
        Patient-focused drug development: a new direction for collaboration.
        Med Care. 2015; 53: 9-17
      5. Food and Drug Administration. Pilot Clinical Outcome Assessment Compendium (COA Compendium) Webinar (March 8, 2016). Available from: https://www.fda.gov/downloads/Drugs/DevelopmentApprovalProcess/SmallBusinessAssistance/UCM489439.pdf. [Accessed June 25, 2016].

        • Miles M.B.
        • Huberman A.M.
        • Saldana J.
        Qualitative Data Analysis: A Methods Sourcebook.
        Sage Publishing, Thousand Oaks, CA1994
        • Benjamin K.
        • Vernon M.K.
        • Patrick D.L.
        • et al.
        Patient-reported outcome and observer-reported outcome assessment in rare disease clinical trials: an ISPOR COA Emerging Good Practices Task Force Report.
        Value Health. 2017; 20: 838-855
        • Patrick D.L.
        • Burke L.B.
        • Gwaltney C.J.
        • et al.
        Content validity—establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO Good Research Practices Task Force report: part 1—eliciting concepts for a new PRO instrument.
        Value Health. 2011; 14: 967-977
        • Patrick D.L.
        • Burke L.B.
        • Gwaltney C.J.
        • et al.
        Content validity—establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO Good Research Practices Task Force report: part 2—assessing respondent understanding.
        Value Health. 2011; 14: 978-988
        • Rothman M.
        • Burke L.
        • Erickson P.
        • et al.
        Use of existing patient-reported outcome (PRO) instruments and their modification: the ISPOR Good Research Practices for Evaluating and Documenting Content Validity for the Use of Existing Instruments and Their Modification PRO Task Force report.
        Value Health. 2009; 12: 1075-1083
        • Guidance for Industry
        Patient-reported outcome measures: use in medical product development to support labeling claims.
        Draft guidance. 2006; (Available from:) ([Accessed June 29, 2017].)
        • Psotka M.A.
        • von Maltzahn R.
        • Anatchkova M.
        • et al.
        Patient-reported outcomes in chronic heart failure: applicability for regulatory approval.
        JACC Heart Fail. 2016; 4: 791-804
        • Green C.P.
        • Porter C.B.
        • Bresnahan D.R.
        • Spertus J.A.
        Development and evaluation of the Kansas City Cardiomyopathy Questionnaire: a new health status measure for heart failure.
        J Am Coll Cardiol. 2000; 35: 1245-1255
      6. International Society for Quality of Life Research. ISOQOL commentary on FDA Compendium. Available from: http://www.isoqol.org/UserFiles/file/ISOQOLFDACompendiumCommentaryFinalDraft.pdf. [Accessed July 19, 2017].

        • Kleinman L.
        • Talbot G.H.
        • SchÃler R.
        • et al.
        Cdad-Daysymsâ: a new patient-reported outcome tool for Clostridium difficile-associated diarrhea.
        Value Health. 2014; 17: A569
        • Frost M.H.
        • Reeve B.B.
        • Liepa A.M.
        • et al.
        What is sufficient evidence for the reliability and validity of patient-reported outcome measures?.
        Value Health. 2007; 10: S94-S105
        • Doward L.C.
        • Gnanasakthy A.
        • Baker M.G.
        Patient-reported outcomes: looking beyond the label claim.
        Health Qual Life Outcomes. 2010; 8: 89
        • Newman J.C.
        • Feldman R.
        Copyright and open access at the bedside.
        N Engl J Med. 2011; 365: 2447-2449
        • Canestaro W.J.
        • Edwards T.C.
        • Patrick D.L.
        Systematic review: patient-reported outcome measures in coeliac disease for regulatory submissions.
        Aliment Pharmacol Ther. 2016; 44: 313-331
      7. Letter to FDA: Re: Docket No. FDA-2015-N-5106: Clinical Outcome Assessment Compendium. Available from: https://www.bio.org/sites/default/files/2016-03-14%20BIO%20Comments%20on%20COA%20Compendium%20FINAL.pdf. [Accessed July 19, 2017].

        • Todd K.L.
        • Wright D.I.
        Durability of treatment effect with polidocanol endovenous microfoam on varicose vein symptoms and appearance (VANISH-2).
        J Vasc Surg Venous Lymphat Disord. 2015; 3 (e1): 258-264
      8. Validation of new patient/clinician-reported outcome tool—capturing the visual impact of varicose veins.
        Poster presented at: Drug Information Association 48th annual meeting, Philadelphia, PA2012: 24-28 (June 2012. Available from:)
        • Vasquez M.A.
        • Munschauer C.E.
        Venous Clinical Severity Score and quality-of-life assessment tools: application to vein practice.
        Phlebology. 2008; 23: 259-275
        • Paty J.
        • Turner-Bowker D.M.
        • Elash C.A.
        • Wright D.
        The VVSymQ(R) instrument: use of a new patient-reported outcome measure for assessment of varicose vein symptoms.
        Phlebology. 2016; 31: 481-488
        • Wright D.D.
        • Paty J.
        • Turner-Bowker D.M.
        • Bradbury A.
        Psychometric evaluation of a new patient-reported outcome (PRO) symptom diary for varicose veins: VVSymQ(R) Instrument.
        Patient. 2016; 9: 335-348
        • Paty J.
        • Elash C.A.
        • Turner-Bowker D.M.
        Content validity for the VVSymQ(R) Instrument: a new patient-reported outcome measure for the assessment of varicose veins symptoms.
        Patient. 2017; 10: 51-63
        • Abelson M.B.
        • Chambers W.A.
        • Smith L.M.
        Conjunctival allergen challenge: a clinical approach to studying allergic conjunctivitis.
        Arch Ophthalmol. 1990; 108: 84-88
        • Abelson M.B.
        • Loeffler O.
        Conjunctival allergen challenge: models in the investigation of ocular allergy.
        Curr Allergy Asthma Rep. 2003; 3: 363-368
        • DeMuro C.
        • Clark M.
        • Mordin M.
        • et al.
        Reasons for rejection of patient-reported outcome label claims: a compilation based on a review of patient-reported outcome use among new molecular entities and biologic license applications, 2006–2010.
        Value Health. 2012; 5: 443-448
        • Rahman A.
        • Rizwan S.
        • Waycaster C.
        • Wall G.M.
        Pooled analysis of two clinical trials comparing the clinical outcomes of topical ciprofloxacin/dexamethasone otic suspension and polymyxin B/neomycin/hydrocortisone otic suspension for the treatment of acute otitis externa in adults and children.
        Clin Ther. 2007; 29: 1950-1956
        • Monk T.H.
        • Reynolds III, C.F.
        • Kupfer D.J.
        • et al.
        The Pittsburgh Sleep Diary.
        J Sleep Res. 1994; 3: 111-120
        • Dupont H.L.
        • Jiang Z.D.
        • Belkind-Gerson J.
        • et al.
        Treatment of travelers’ diarrhea: randomized trial comparing rifaximin, rifaximin plus loperamide, and loperamide alone.
        Clin Gastroenterol Hepatol. 2007; 5: 451-456