Advertisement

Methods for Developing Patient-Reported Outcome-Based Performance Measures (PRO-PMs)

Open ArchivePublished:May 20, 2015DOI:https://doi.org/10.1016/j.jval.2015.02.018

      Abstract

      Objective

      To recommend methods for assessing quality of care via patient-reported outcome-based performance measures (PRO-PMs) of symptoms, functional status, and quality of life.

      Methods

      A Technical Expert Panel was assembled by the American Medical Association–convened Physician Consortium for Performance Improvement. An environmental scan and structured literature review were conducted to identify quality programs that integrate PRO-PMs. Key methodological considerations in the design, implementation, and analysis of these PRO-PM data were systematically identified. Recommended methods for addressing each identified consideration were developed on the basis of published patient-reported outcome (PRO) standards and refined through public comment. Literature review focused on programs using PROs to assess performance and on PRO guidance documents.

      Results

      Thirteen PRO programs and 10 guidance documents were identified. Nine best practices were developed, including the following: provide a rationale for measuring the outcome and for using a PRO-PM; describe the context of use; select a measure that is meaningful to patients with adequate psychometric properties; provide evidence of the measure’s sensitivity to differences in care; address missing data and risk adjustment; and provide a framework for implementation, interpretation, dissemination, and continuous refinement.

      Conclusion

      Methods for integrating PROs into performance measurement are available.

      Keywords

      Background

      A foundation for continuous quality improvement is to measure and compare care across practices and providers to translate successful management strategies to others [

      U.S. Department of Health and Human Services. 2012 Annual Progress Report to Congress on the National Strategy for Quality Improvement in Health Care (National Quality Strategy). Available from: http://www.ahrq.gov/workingforquality/nqs/nqs2012annlrpt.pdf. [Accessed April 4, 2014].

      ]. Performance measurement has traditionally relied on routinely collected clinical information such as rates of hospital readmission, infections, procedural complications, survival, or laboratory values. But the ultimate impact on outcomes experienced by patients, such as symptoms, functional status, and health-related quality of life, have rarely been assessed.
      Collection and analysis of patient-reported outcome (PRO) measures is increasingly considered a standard approach for evaluating these experiences [
      • Brundage M.
      • Blazeby J.
      • Revicki D.
      • et al.
      Patient-reported outcomes in randomized clinical trials: development of ISOQOL reporting standards.
      ,

      U.S. Food and Drug Administration. Guidance for Industry. Patient-reported outcome measures: use in medical development to support labeling claims. December 2009. Available from: http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM193282.pdf. [Accessed April 4, 2014].

      ,
      Scientific Advisory Committee of the Medical Outcomes Trust
      Assessing health status and quality of life instruments: attributes and review criteria.
      ,

      European Medicines Agency. Reflection paper on the regulatory guidance for use of health-related quality of life (HRQOL) measures in the evaluation of medicinal products. July 2005. Available from: www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2009/09/WC500003637.pdf. [Accessed April 4, 2014].

      ]. A PRO is defined as information about the status of a patient’s health condition that comes directly from the patient, without interpretation of the patient’s response by a clinician or anyone else [

      U.S. Food and Drug Administration. Guidance for Industry. Patient-reported outcome measures: use in medical development to support labeling claims. December 2009. Available from: http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM193282.pdf. [Accessed April 4, 2014].

      ]. A patient-reported outcome measure (PROM) is a questionnaire used to elicit information directly from respondents. Inclusion of patients’ direct reports about how they feel and function in quality assessment programs through the use of patient-reported outcome-based performance measures (PRO-PMs), and particularly in accountability and value-based payment initiatives, would increase the patient-centeredness of these activities [
      Methodology Committee of the Patient-Centered Outcomes Research Institute (PCORI)
      Methodological standards and patient-centeredness in comparative effectiveness research: the PCORI perspective.
      ,
      • Blumenthal D.
      • Dixon J.
      Health-care reforms in the USA and England: areas for useful learning.
      ,
      • Selby J.V.
      • Beal A.C.
      • Frank L.
      The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda.
      ]. PRO measurement is already common in clinical trials and is of rising interest in comparative effectiveness research, routine clinical practice, and electronic medical record systems [
      • Gnanasakthy A.
      • Mordin M.
      • Clark M.
      • et al.
      A review of patient-reported outcome labels in the United States: 2006 to 2010.
      ,
      • Wu A.W.
      • Snyder C.
      • Clancy C.M.
      • Steinwachs D.M.
      Adding the patient perspective to comparative effectiveness research.
      ,
      • Wu A.W.
      • Snyder C.
      Getting ready for patient-reported outcomes measures (PROMs) in clinical practice.
      ,
      • Abernethy A.P.
      • Herndon II, J.E.
      • Wheeler J.L.
      • et al.
      Improving health care efficiency and quality using tablet personal computers to collect research-quality, patient-reported data.
      ,
      • Bennett A.V.
      • Jensen R.E.
      • Basch E.
      Electronic patient-reported outcome systems in oncology clinical practice.
      ,
      • Basch E.
      • Abernethy A.P.
      Supporting clinical practice decisions with real-time patient-reported outcomes.
      ].
      Beyond patient-centeredness, there are additional rationales to include PROs in performance measurement. Recent data suggest that patients’ self-reported symptoms and health status are associated with the use of medical services (e.g., emergency room visits and hospitalizations), costs, outpatient medication compliance, and survival [
      • Harrison P.L.
      • Pope J.E.
      • Coberley C.R.
      • Rula E.Y.
      Evaluation of the relationship between individual well-being and future health care utilization and cost.
      ,
      • Wagner L.I.
      • Zhao F.
      • Chapman J-AW
      • et al.
      Patient-reported predictors of early treatment discontinuation: NCIC JMA.27/E1Z03. San Antonio Breast Cancer Symposium, San Antonio, TX; 2011.
      ,
      • Chan P.S.
      • Soto G.
      • Jones P.G.
      • et al.
      Patient health status and costs in heart failure: insights from the eplerenone post-acute myocardial infarction heart failure efficacy and survival study (ephesus).
      ,
      • Gotay C.C.
      • Kawamoto C.T.
      • Bottomley A.
      • Efficace F.
      The prognostic significance of patient-reported outcomes in cancer clinical trials.
      ]. The process of patient self-reporting itself can improve symptom management, quality of life, communication, and satisfaction with care [
      • Valderas J.M.
      • Kotzeva A.
      • Espallargues M.
      • et al.
      The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature.
      ,
      • Chen J.
      • Ou L.
      • Hollis S.J.
      A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting.
      ,
      • Detmar S.B.
      • Muller M.J.
      • Schornagel J.H.
      • et al.
      Health-related quality-of-life assessments and patient-physician communication: a randomized controlled trial.
      ,
      • Velikova G.
      • Booth L.
      • Smith A.B.
      • et al.
      Measuring quality of life in routine oncology practice improves communication and patient well-being: a randomized controlled trial.
      ]. Moreover, symptoms and functional status impairment are far more common than serious complications of treatment, such as hospitalizations or death [
      • Basch E.
      The missing voice of patients in drug-safety reporting.
      ]. As the ultimate end users of services, patients selecting a treatment or provider may have interest in outcomes based on previous reports of patients like themselves.
      There is currently limited understanding in the PRO methodology community about performance measurement procedures, and a similarly limited understanding in the performance measurement community about methodological challenges involved with developing, administering, and analyzing PRO data. Therefore, there is a need for a practical blueprint to bring these two fields together and describe methodological best practices for developing, testing, implementing, and interpreting PRO performance measures that can be used as criteria by measure developers and credentialing organizations to evaluate candidate measures.

      Methods

       Technical Expert Panel

      The American Medical Association–convened Physician Consortium for Performance Improvement, which has overseen the specification and testing of health care performance measures in the United States for more than 10 years, assembled a Technical Expert Panel to develop methodological best practices for guiding the development and evaluation of PRO measures in performance evaluation. The panel consisted of experts in performance measurement, PROs, clinical research, health services research, as well as clinical practitioners and patient representatives (represented by the authors of this article). By design, the panel’s focus was restricted to traditional domains measured by PROs, including symptoms, functional status, and health-related quality of life, and did not encompass patient-reported health-related behaviors or satisfaction with care (sometimes referred to as patient-reported experience measures, e.g., patient perceptions of clinic wait times or staff attentiveness), which are related but distinct areas of measurement with a more extensive history in performance evaluation [
      • Jha A.K.
      • Orav E.J.
      • Zheng J.
      • Epstein A.M.
      Patients’ perception of hospital care in the United States.
      ,
      • Lansky D.
      Overview: performance measures--the next generation.
      ,

      Bethell C, for FACCT (Foundation for Accountablity). Patient-centered care measures for the National Health Care Quality Report. May 2000. Available from: http://www.policyarchive.org/handle/10207/bitstreams/95530.pdf. [Accessed April 4, 2014].

      ,

      Lansky D., Bethell C, for FACCT (Foundation for Accountability). Consumers and quality: What do they know? What do they want? Results from FACCT Consumer Research 1996-2000. Available from: http://www.policyarchive.org/handle/10207/bitstreams/95534.pdf. [Accessed April 4, 2014].

      ].

       Environmental Scan

      The panel first completed an environmental scan (including a structured literature review) of existing pertinent initiatives using PROs for quality assessment (“use cases”), and related methods guidance documents. The purpose of finding use cases was both to evaluate the current state of this field and to identify areas warranting best practice recommendations.
      The literature review was initially conducted in January 2012 and updated in April 2013, via the PubMed electronic database consisting of two searches, the first focused on published reports of research using PROs to assess performance and the second focused on existing salient guidance documents. Search terms are listed in the Appendix in Supplemental Materials found at doi:10.1016/j.jval.2015.02.018.
      Identified publication titles and abstracts were reviewed by two members of the panel and categorized independently as not relevant, maybe relevant, or relevant, with citations considered not relevant by both researchers eliminated and remaining citations presented to the panel for evaluation. Relevance was based on two required criteria: 1) elicitation of information about symptoms, functional status, or health-related quality of life directly from patients using PRO measures and 2) an objective of measuring or improving performance/quality of care delivery. Subsequent steps included assessment of the quality of each identified study and review of references to find additional potentially relevant articles.
      The environmental scan also consisted of a Google search using permutations of the same terms, which were reviewed in real time for performance measurement initiatives using PROs, as well as existing PRO guidance documents. These searches were supplemented by results of reports contracted by the Patient-Centered Outcomes Research Institute, which are publicly available [

      Butt Z, Reeve B. Enhancing the patient’s voice: standards in the design and selection of Patient-Reported Outcomes Measures (PROMs) for use in patient-centered outcomes research. A report prepared for the Patient-Centered Outcomes Research Institute (PCORI). Available from: www.pcori.org/assets/Enhancing-the-Patients-Voice-Standards-in-the-Design-and-Selection-of-Patient-Reported-Outcomes-Measures-for-Use-in-Patient-Centered-Outcomes-Research.pdf. [Accessed April 4, 2014].

      ,

      Acaster S, Cimms T, Lloyd A. Development of a methodological standards report. The design and selection of patient-reported outcomes measures (PROMS) for use in patient-centered outcomes research. A report prepared for the Patient-Centered Outcomes Research Institute (PCORI). Available from: http://www.pcori.org/assets/The-Design-and-Selection-of-Patient-Reported-Outcomes-Measures-for-Use-in-Patient-Centered-Outcomes-Research.pdf. [Accessed April 4, 2014].

      ], and distribution of an e-mail inquiry to an assembled list of investigators involved in PRO research and performance measurement research.
      Identified use cases were reviewed by the panel—with direct communication with investigators whenever possible—to identify the design and measures used for each initiative. “Key findings” were assembled for each use case on the basis of published data and investigator communication, encompassing the a priori selected areas of sample size, recruitment and response/retention rates, project duration, and observed results. “Lessons learned” were assembled encompassing the a priori selected areas of bias and adjustment, compliance and missing data, and feasibility.

      Results

       Use Cases and Guidance Documents

      Of 21,342 titles identified by the literature search, 6 publications were identified that described initiatives in which PROs were used in performance evaluation [
      • Vickers A.J.
      • Sjoberg D.
      • Basch E.
      • et al.
      How do you know if you are any good? A surgeon performance feedback system for the outcomes of radical prostatectomy.
      ,
      • Chen P.C.
      • Lee Y.C.
      • Kuo R.N.
      Differences in patient reports on the quality of care in a diabetes pay-for-performance program between 1 year enrolled and newly enrolled patients.
      ,
      • Weinstein J.N.
      • Tosteson T.D.
      • Lurie J.D.
      • et al.
      Surgical vs. nonoperative treatment for lumbar disk herniation: the Spine Patient Outcomes Research Trial (SPORT): a randomized trial.
      ,
      • Jones N.
      • Jones S.L.
      • Miller N.A.
      The Medicare Health Outcomes Survey program: overview, context, and near-term prospects.
      ,
      • Gilbert J.E.
      • Howell D.
      • King S.
      • et al.
      Quality improvement in cancer symptom assessment and control: the Provincial Palliative Care Integration Project (PPCIP).
      ,
      • Wu A.W.
      • Young Y.
      • Skinner E.A.
      • et al.
      Quality of care and outcomes of adults with asthma treated by specialists and generalists in managed care.
      ]. Three additional relevant initiatives were identified via the Google search [
      • Appleby J.
      • Devlin N.
      Measuring Success in the NHS: Using Patient-Assessed Health Outcomes to Manage the Performance of Health Care Providers.
      ,

      Swedish Rheumatology Registers (sRR). Annual report from sRR. 2011. Available from: www.swerre.se/cms/sv/provider/reports. [Accessed April 4, 2014].

      ,

      National Health Service, National Clinical Audit Support Programme (NCASP). Overview of Mastectomy and Breast Reconstruction (MBR) audit for women in England and Wales. Available from: www.ic.nhs.uk/mbr. [Accessed April 4, 2014].

      ] and an additional four via the e-mail distribution [

      Centers for Medicare & Medicaid Services. Transcatheter aortic valve replacement (TAVR) National Coverage Determination for Coverage with Evidence Development (CED). Available from: http://www.cms.gov/Medicare/Medicare-General-Information/MedicareApprovedFacilitie/Transcatheter-Aortic-Valve-Replacement-TAVR.html. [Accessed April 4, 2014].

      ,

      The Commonwealth Fund. Purchasing high performance Boeing’s nurse case managers cut per capita costs by 20 percent. Available from: www.commonwealthfund.org/Newsletters/Purchasing-High-Performance/2011/March-29-2011/Case-Study/Boeings-Nurse-Case-Managers.aspx. [Accessed April 4, 2014].

      ,

      Minnesota Community Measurement. Minnesota Health Scores overview. Available from: www.mnhealthscores.org. [Accessed April 4, 2014].

      ,

      Royal College of Obstetricians and Gynaecologists. Heavy Menstrual Bleeding Audit Programme description. Available from: http://www.rcog.org.uk/orca/audit. [Accessed April 4, 2014].

      ]. These 13 use cases are described below and in Table 1, including key findings and lessons learned.
      Table 1Example use cases identified via literature and landscape overviews, using patient-reported outcomes in performance evaluation projects or programs.
      It is recognized that this list may not be exhaustive.
      NameDesignMeasuresFindingsLessons learned
      1. English National Health Service’s (NHS’s) PROMs Program
      • Appleby J.
      • Devlin N.
      Measuring Success in the NHS: Using Patient-Assessed Health Outcomes to Manage the Performance of Health Care Providers.
      ,
      • Hutchings A.
      • Grosse Frie K.
      • Neuburger J.
      • et al.
      Late response to patient reported outcome questionnaires was associated with worse outcome.
      All providers to English NHS patients undergoing elective total hip replacement, total knee replacement, varicose vein surgery, and inguinal hernia repair required to offer baseline preprocedure PRO questionnaire and postprocedure PRO questionnaire after 3 (varicose vein and hernia) or 6 (hip and knee replacement) moEQ-5D, Oxford Hip Score, Oxford Knee Score, Aberdeen Varicose Vein Questionnaire, postoperative questions about complications and single transitional items on overall view of resultsBetween April 2009 and March 2011, 485,199 eligible patients with 329,841 preoperative questionnaires returned (68% recruitment rate), and 314,488 postoperative questionnaires mailed out with 253,135 returned (81% response rate). Overall improvements observed across patients. Comparisons of performance conducted at “hospital trust” level rather than single hospital or single provider level (many hospital trusts consist of only one hospital)
      • Recruitment rates higher for more major surgery (~80%) than for minor procedures (~50%)
      • Recruitment bias disfavors patients who are nonwhite, have lower socioeconomic status, and are older and varies considerably between sites (20%–100%), necessitating case-mix adjustment in analyses
      • Response bias disfavors patients who are nonwhite, younger, and male, have lower socioeconomic status, and with more comorbidity. This risks underdetecting poor quality because patients with worse outcomes may be less likely to return a questionnaire
      • For major surgery, disease-specific PROs delineate providers as above or below average performance, with correlation with generic health state measures. However, ratings of some providers depend on the choice of the measure
      • Mandatory systemwide participation optimizes response rates
      2. Postprostatectomy quality assessment program
      • Vickers A.J.
      • Sjoberg D.
      • Basch E.
      • et al.
      How do you know if you are any good? A surgeon performance feedback system for the outcomes of radical prostatectomy.
      ,
      • Vickers A.J.
      • Savage C.J.
      • Shouery M.
      • et al.
      Validation study of a web-based assessment of functional recovery after radical prostatectomy.
      Patients returning for follow-up after radical prostatectomy for localized prostate cancer administered online questionnaire at home at prespecified time points. Comparisons of performance are conducted at the individual provider level, with an electronic dashboard showing surgeons their risk-adjusted results compared with those of their colleagues. Patients can access graphs showing progress over time in comparison to similar patients15 validated items pertaining to urinary, erectile, and bowel dysfunction, as well as overall quality of lifeSince 2010, more than 2000 men have completed more than 5000 questionnaires. Compliance is 75%, with imputation techniques providing data for 85% of the patients. Investigations of surgical techniques at bladder neck prompted by observation of better symptomatic results by some surgeons
      • Individual providers incented to participate because obtaining PROs before a clinic visit can streamline the consultation; providers can privately compare their own outcomes with those of others and track performance over time
      • Critical driver of high patient compliance is that patients know that their questionnaire responses are seen by the doctor and used in the clinical consultation
      • Electronic administration of questionnaires pushed out via automated e-mails, with backup data collection via staff-distributed iPads in clinics, optimizes compliance
      • Electronic PRO system is inexpensive
      • PROs obtained for clinical purposes can be used as end points of observational and experimental research
      3. Centers for Medicare & Medicaid Services (CMS)-approved registry for percutaneous transcatheter aortic valve replacement (TAVR) coverage

      Centers for Medicare & Medicaid Services. Transcatheter aortic valve replacement (TAVR) National Coverage Determination for Coverage with Evidence Development (CED). Available from: http://www.cms.gov/Medicare/Medicare-General-Information/MedicareApprovedFacilitie/Transcatheter-Aortic-Valve-Replacement-TAVR.html. [Accessed April 4, 2014].

      ,

      Society of Thoracic Surgeons and American College of Cardiology. TVT registry for transcatheter aortic valve replacement (TAVR) procedure. Available from: https://www.ncdr.com/TVT/Home/Default.aspx. [Accessed April 4, 2014].

      In 2012, the CMS approved a National Coverage Determination (NCD) for TAVR. The NCD permits coverage at qualified sites using an FDA-approved device for FDA-labeled indications, and requires all patients to be included in a national TAVR registry. The STS/ACC TVT Registry is the first approved registry for this purpose, and was developed by the Society of Thoracic Surgeons (STS) and the American College of Cardiology (ACC)Kansas City Cardiomyopathy Questionnaire (KCCQ) Short FormPublic program specifications require the inclusion of patient-reported outcomes. The prospective registry launched in Summer 2012 with participation required for CMS reimbursement
      • Program demonstrates CMS interest in understanding patient symptom experience and self-reported health status
      • Compliance and results pending (registry recently launched). Data anticipated to benchmark performance and identify patient and operator/center sources of variability, and characterize the symptom burden of patients undergoing TAVR
      • Program will reflect the feasibility of collecting data about how patients feel and how they improve after treatment as a component of CMS-required evidence
      4. Diabetes pay-for-performance PRO Evaluation
      • Chen P.C.
      • Lee Y.C.
      • Kuo R.N.
      Differences in patient reports on the quality of care in a diabetes pay-for-performance program between 1 year enrolled and newly enrolled patients.
      Cross-sectional telephone interview to measure quality of care for patients with diabetes mellitus in pay-for-performance program. Newly enrolled patients (<3 mo in program) compared with long-term participants (>1 y in program)Satisfaction, compliance, lifestyle/habits (exercise, diet), and self-care questionnaire46% response rate with 1796 respondents with complete information (1238 case group, 558 control group). Imputation methods used. Long-term participants had better outcomes than did new enrollees including self-care (including foot care), exercise, diet, medication compliance, and satisfaction
      • Range of patient-reported metrics includes medication compliance, self-care, and lifestyle, which can be used to evaluate performance
      • Case-control design is feasible in this context
      • Cross-sectional design is feasible, although methods to enhance response rates might minimize missing data
      5. Dartmouth-Hitchcock Spine Center project
      • Weinstein J.N.
      • Tosteson T.D.
      • Lurie J.D.
      • et al.
      Surgical vs. nonoperative treatment for lumbar disk herniation: the Spine Patient Outcomes Research Trial (SPORT): a randomized trial.
      ,

      Dartmouth Institute for Health Policy and Clinical Practice. Peer reviewed technical report: using patient-reported information to improve health outcomes and health care value: case studies from Dartmouth, Karolinska, and Group Health. 2012. Available from: http://tdi.dartmouth.edu/images/uploads/tdi_tr_pri_ia_sm.pdf. [Accessed April 4, 2014].

      Patients complete 30-min online questionnaire at baseline and before each follow-up visit. Summary reports with longitudinal trends given to providers. Population-level data reported publicly online including the proportion of patients who experience improved pain and functioning after surgical or nonsurgical interventions, as well as the proportion of patients with numbness and/or weaknessSymptom, functional status/activity, weakness, employment status, and lifestyle questionnaire. Patient-reported data combined with other clinical and cost outcomes in analysesAll spine clinic patients participate. Data shared at the patient and provider level to guide clinical practice, and at the institutional level for public reporting
      • Minimize response burden for patients by using brief items and questionnaires that are valid, reliable, and sensitive to change over time
      • Questions should have clear purpose, be salient to patients and care teams (i.e., match needs and preferences), and be actionable
      • Summary reports for patients and clinicians enhance engagement
      • Collaboration with key nonclinical stakeholders (employers, purchasers, researchers, patient advocates) and financial incentives or cost-sharing can enhance adoption and use
      • Importance of case-mix adjustment
      6. Swedish Rheumatology Quality Register

      Swedish Rheumatology Registers (sRR). Annual report from sRR. 2011. Available from: www.swerre.se/cms/sv/provider/reports. [Accessed April 4, 2014].

      ,

      Dartmouth Institute for Health Policy and Clinical Practice. Peer reviewed technical report: using patient-reported information to improve health outcomes and health care value: case studies from Dartmouth, Karolinska, and Group Health. 2012. Available from: http://tdi.dartmouth.edu/images/uploads/tdi_tr_pri_ia_sm.pdf. [Accessed April 4, 2014].

      Patients with rheumatologic illnesses being treated at participating clinics complete a 10–15-min questionnaire (paper or online) before clinic visits. Summary reports reviewed by patients and providers at visits. Data linked to national EHR systemPain, swollen and/or tender joints, general well-being, daily activities, health state (EQ-5D), ability to workProgram operating since 1995. By 2012, 64 clinics participating, with 25 clinics using online forms. Data shared at the patient and provider level to guide clinical practice, and at the national level for comparing change scores over time between practices
      • Migration to electronic system from paper is feasible but requires local engagement
      • Integration of electronic PROs with a national EHR system is feasible
      • Data collected to enhance clinical practice and patient-provider communication can also be used to compare performance between clinics
      • Importance of case-mix adjustment
      7. Boeing Intensive Outpatient Care Program (IOCP) pilot project

      The Commonwealth Fund. Purchasing high performance Boeing’s nurse case managers cut per capita costs by 20 percent. Available from: www.commonwealthfund.org/Newsletters/Purchasing-High-Performance/2011/March-29-2011/Case-Study/Boeings-Nurse-Case-Managers.aspx. [Accessed April 4, 2014].

      Patients participating in a purchaser-sponsored patient-centered medical home pilot program complete questions about physical and mental healthPhysical and mental functioning questions, and patient-reported employment informationStarting in 2007, 740 employees and dependents enrolled, with 15% improvement in physical functioning scores, 16% improvement in mental functioning scores, and 56% improvement in patient-reported missed workdays
      • Patient-reported data reflect improved quality of care and correlate with other metrics of performance
      • Feasible to integrate PROs into performance-improvement measurement efforts
      8. Mastectomy and Breast Reconstruction (MBR) Audit

      National Health Service, National Clinical Audit Support Programme (NCASP). Overview of Mastectomy and Breast Reconstruction (MBR) audit for women in England and Wales. Available from: www.ic.nhs.uk/mbr. [Accessed April 4, 2014].

      All adult women in England and Wales undergoing mastectomy and/or breast reconstruction for breast cancer complete questions about satisfaction with results of surgery, satisfaction with provider(s), symptoms, sexual function, and quality of life longitudinallyBREAST-QBetween 2008 and 2009, 18,216 women aged 16 y and older enrolled with follow-up data now collected 3 and 18 mo after surgery. At 18 mo, among women undergoing mastectomy without breast reconstruction, 83% were satisfied with how they looked clothed and 42% unclothed vs. women undergoing immediate reconstruction among whom 90% were satisfied clothed and 59% unclothed vs. women undergoing delayed reconstruction among whom 93% were satisfied clothed and 76% unclothed
      • Context-specific questions designed for a particular patient population and procedure provides meaningful information to advise future patients and decision makers
      • Feasible to implement a universal longitudinal PRO data collection initiative in a targeted population
      9. Minnesota Community Measurement Depression and Orthopedic Measures

      Minnesota Community Measurement. Minnesota Health Scores overview. Available from: www.mnhealthscores.org. [Accessed April 4, 2014].

      State-mandated program for primary care and psychiatric practices across Minnesota to collect and submit longitudinal patient depression scores via secure EMR portal, and pilot for state-mandated program for orthopedic and neurosurgical practices across Minnesota to collect and submit longitudinal patient functional status scores via secure EMR portalPHQ-9 (for depression program); Oxford Knee Score, Oswestry Disability Index, VAS Pain Scale (for orthopedic program)For depression program, since 2009, patients have reported baseline and follow-up depression scores at 6- and 12-mo time points (±1 mo). Currently, ~80% of eligible Minnesota practices participate. Six-month depression remission rates remain low overall (6% statewide), with some clinics achieving higher rates (15%–24%). Practice-level support resources for patient care management and between-visit communication associated with higher remission rates. Six-month follow-up rate is 25% (N = 75,953) but improving annually. Orthopedic program currently in data collection phase
      • Primary care clinicians accept depression score collection and reporting
      • Many patients are lost to follow-up, suggesting need for backup data collection methods
      • Minnesota Health Reform mandated the collection of this measure from all physician clinics in the state of Minnesota (primary care and psychiatry). Before this point in time, ~70% of the clinics participated voluntarily
      • The PHQ-9 tool was relatively easy for software vendors to add to the EMR, easing data collection and extraction burden. PHQ-9 coded for LOINC, which supports eMeasure specifications
      10. Medicare (CMS) Health Outcomes Survey (HOS)
      • Jones N.
      • Jones S.L.
      • Miller N.A.
      The Medicare Health Outcomes Survey program: overview, context, and near-term prospects.
      ,

      Centers for Medicare & Medicaid Services. Medicare Health Outcomes Survey: program overview. Available from: http://www.hosonline.org/Content/Default.aspx. [Accessed April 4, 2014].

      Administered by the CMS, with assistance by the National Committee for Quality Assurance (NCQA). All managed care plans with Medicare Advantage contracts must participate, with a goal to collect data toward quality improvement, accountability, and public reporting, ultimately as part of the Five-Star Quality Rating system for the Medicare Advantage PlansHOS (includes specific patient-reported items to assess selected symptoms, functional status, mental health, comorbidities, and other questions toward case-mix and risk adjustment)Since 1998, the HOS has been administered annually to a random sample of Medicare beneficiaries from each participating Medicare Advantage Organization, with repeat survey administration 2 y later, with an annual cohort sample size randomly chosen in those plans over 1000. Compliance rates range between about 50% and 85% by year of cohort. Performance of plans is classified for a mental summary score and physical summary as being better, same, or worse than expected. HOS survey results are included as part of the Medicare Advantage Quality and Performance Plan Ratings System to be used by the CMS as the basis for quality bonus payments
      • A payer-developed PRO performance evaluation strategy is feasible
      • PRO data can be combined with other performance measures
      • PRO data can be used within the context of a pay-for-performance program
      • Two-year follow-up is feasible, although response rates vary from year to year
      11. Cancer Care Ontario Chemotherapy Symptoms
      • Gilbert J.E.
      • Howell D.
      • King S.
      • et al.
      Quality improvement in cancer symptom assessment and control: the Provincial Palliative Care Integration Project (PPCIP).
      Across the Canadian province of Ontario, internet kiosks installed in outpatient oncology clinics for patient voluntary symptom self-reporting at clinic visitsEdmonton Symptom Assessment Scale (ESAS)Program administered in 2006. Performance measures included proportion of patients completing the ESAS at initial and follow-up visits; proportion of patients with severe symptoms whose symptoms are improved at subsequent assessment; proportion of patients self-reporting depression who are further assessed or referred for psychosocial resources within 72 h. Within 1 y of implementation, improvements seen in symptom screening rates and symptom control
      • Patient self-reporting can be used as both a process (% completing ESAS) and an outcome measure (symptom improvement over time; timely referral) in the same implementation
      • Regional entities overseeing care can be engaged by a central administration to implement a systemwide program
      • Compliance rates may not be optimal with “voluntary” patient reports via clinic-based kiosks. Patients may have other priorities at clinic visits, and those too ill to attend visits are missed
      12. Managed Health Care Association (MHCA) Outcomes Management System Consortium Asthma Survey
      • Wu A.W.
      • Young Y.
      • Skinner E.A.
      • et al.
      Quality of care and outcomes of adults with asthma treated by specialists and generalists in managed care.
      Cross-sectional survey mailed to adult asthma patients enrolled in 12 managed care organizationsQuestionnaire including asthma symptoms and overall health status itemsSurvey mailed to 1954 patients at a single time point. Primary finding related to patient-reported data is greater likelihood of improved asthma symptoms when treatment is under the supervision of a specialty-trained (i.e., pulmonologist) rather than generalist physician
      • Cross-sectional surveys can provide between-practitioner–level data
      • Consortia of payer organizations may be used as a mechanism for administering surveys to patients
      13. English Heavy Menstrual Bleeding Audit

      Royal College of Obstetricians and Gynaecologists. Heavy Menstrual Bleeding Audit Programme description. Available from: http://www.rcog.org.uk/orca/audit. [Accessed April 4, 2014].

      All adult women in England attending NHS gynecological outpatient clinics for heavy menstrual bleeding complete questions about symptoms, functional status, and health-related quality of life. Mailed questionnaire 1 y laterUterine Fibroid Symptoms and QoL Questionnaire (UFS-QOL), EQ-5DBaseline questionnaires administered to 15,812 women between February 2011 and January 2012 (recruitment rate ~30%), with follow-up questionnaires sent 1 y later; 148 of 154 hospital trusts participating. One-year follow-up underway
      • Recruitment of patients referred to specialist outpatient clinic much lower than recruitment of those undergoing surgical procedure/treatment (compared with other national England programs)
      • Difficult to determine recruitment rate in outpatient clinic because hospital administrative data on outpatients are limited (no diagnostic data) compared with data on inpatients
      BREAST-Q, Breast Reconstruction Questionnaire; EHR, electronic health record; EMR, electronic medical record; EQ-5D, EuroQol five-dimensional questionnaire; FDA, Food and Drug Administration; LOINC, Logical Observation Identifiers Names and Codes; PHQ-9, Patient Health Questionnaire-9; PRO, patient-reported outcome; PROMs, patient-reported outcome measures; TVT, Transcatheter Valve Trial; VAS, visual analogue scale.
      low asterisk It is recognized that this list may not be exhaustive.
      A classification challenge in identifying use cases is that there are many initiatives using PROs in clinical practice, in observational research/comparative effectiveness research, or in integrating PROs with electronic health records. Although these projects include closely related design elements and can be instructive, they are not performance-improvement oriented per se and were therefore excluded from the analysis. Conversely, there are programs that incorporate measures in which patient experiences such as pain are expected to be elicited by providers from patients and documented, but do not require a specific PRO measure to be used. To be included in the use cases, an example had to include not only the administration of PRO measures but also a research design intended to measure performance. It is recognized that other use cases exist that were not included, and the panel felt that the identified cases provided a sufficient basis for developing best practices.
      Beyond the use cases, 10 PRO methodological guidance documents pertaining to other health care areas were identified and considered by the panel in establishing best practices [

      U.S. Food and Drug Administration. Guidance for Industry. Patient-reported outcome measures: use in medical development to support labeling claims. December 2009. Available from: http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM193282.pdf. [Accessed April 4, 2014].

      ,
      Scientific Advisory Committee of the Medical Outcomes Trust
      Assessing health status and quality of life instruments: attributes and review criteria.
      ,

      European Medicines Agency. Reflection paper on the regulatory guidance for use of health-related quality of life (HRQOL) measures in the evaluation of medicinal products. July 2005. Available from: www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2009/09/WC500003637.pdf. [Accessed April 4, 2014].

      ,
      • Mokkink L.B.
      • Terwee C.B.
      • Patrick D.L.
      • et al.
      The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes.
      ,
      • Coons S.J.
      • Gwaltney C.J.
      • Hays R.D.
      • et al.
      Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO Good Research Practices Task Force report.
      ,
      • Wild D.
      • Grove A.
      • Martin M.
      • et al.
      Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (PRO) measures: report of the ISPOR Task Force for Translation and Cultural Adaptation.
      ,

      Johnson C, Aaronson N, Blazeby JM, et al. EORTC Quality of Life Group: guidelines for developing questionnaire modules. 2011. Available from: http://groups.eortc.be/qol/Pdf%20presentations/Guidelines%20for%20Developing%2 0questionnaire-%20FINAL.pdf. [Accessed April 4, 2014].

      ,
      • Rothman M.
      • Burke L.
      • Erickson P.
      • et al.
      Use of existing patient-reported outcome (PRO) instruments and their modification: the ISPOR Good Research Practices for Evaluating and Documenting Content Validity for the Use of Existing Instruments and Their Modification PRO Task Force Report.
      ,
      • Revicki D.A.
      • Osoba D.
      • Fairclough D.
      • et al.
      Recommendations on health-related quality of life research to support labeling and promotional claims in the United States.
      ]. These documents encompass both regulatory and nonregulatory uses of PRO measures largely in the clinical research context.

       Analysis of Use Cases

      Overall, the use cases and guidance documents underscore that PROs are appropriate when information sought is best known by the patient, for example, symptoms. Outcomes and analysis results should be important and meaningful to patients in a prespecified target population in a prespecified context of use. Similarly, measured outcomes should be sensitive to changes in clinical practice and relevant to clinicians and other decision makers (i.e., actionable). If an outcome cannot be improved by a change in practice, it is likely not appropriate for performance measurement.
      The importance of pilot work to evaluate response rates, sensitivity to within- and between-practice variations, appropriate unit(s) of analysis, and barriers to implementation, as well as continued evaluation of programs to improve implementation strategies, is also emphasized.
      Use of measures with well-established psychometric properties is a common theme in the identified use cases and particularly in guidance documents, including qualitative and quantitative data to establish that patients in a target population understand terms and that terms map to underlying outcomes of interest; construct validity; reliability; sensitivity to change over time; appropriate recall period; meaningfulness of score thresholds and changes to patients; and adequacy of cultural, linguistic, and mode adaptation as needed.
      The availability and use of methods to optimize response rates and minimize missing data is an almost universal theme, with approaches to optimize response rates including centralized data collection, reminder mechanisms, mandatory collection of information in practices (rather than voluntary or subpopulation collection), partnership with providers and communities, and use of proxy/surrogate reporting.
      Most use cases include a risk/case-mix adjustment strategy. In some instances, adjustment for response rates is considered in analyses to avoid response bias; that is, patients with worse outcomes are less likely to return a questionnaire, so those providers who are most vigilant about obtaining follow-up questionnaires may recover a higher proportion of responses from sicker individuals with worse outcomes, making their results look worse if analyses are unadjusted.
      An a priori plan for data analysis, interpretation, and dissemination is emphasized. This includes a plan for handling missing data using imputation methods because patient-reported data are often missing not at random [
      • Fairclough D.L.
      • Peterson H.F.
      • Cella D.
      • Bonomi P.
      Comparison of several model-based methods for analysing incomplete quality of life data in cancer clinical trials.
      ]. Other issues include dealing with multiple end points/comparisons, measuring cross-sectional differences versus changes over time, adequacy of sample size and potential overpowering, and consideration of outliers in analyses. Most of these issues are not unique to PRO-based analyses.

      Best Practice Recommendations

      Five a priori areas of review guided the panel in analyzing use cases and guidelines to extract best practices that facilitate the development of clinically meaningful and scientifically robust performance measures: selection of outcomes, development/selection of metrics, implementation, analysis, and reporting/dissemination. This analysis yielded nine best practices that are presented in Table 2 along with specific questions that can be considered to determine whether a measure abides by the recommendations. Panel agreement was unanimous on each; refinements were made on the basis of public comment (described below). The best practices were approved by the Physician Consortium for Performance Improvement Measures Implementation and Informatics Committee.
      Table 2Methodological best practices and associated considerations for developing and evaluating proposed patient-reported outcome performance measures (PRO-PMs).
      Best practiceConsiderationsCorresponding National Quality Forum (NQF) measure evaluation criteria
      Details available from www.qualityforum.org/docs/measure_evaluation_criteria.aspx.
      1. A rationale for measuring the outcome should be described
      • Is a knowledge gap described and justified?
      • Is there evidence that the outcome is meaningful and important to patients, caregivers, and/or other stakeholders?
      • How does patient self-reporting in particular address the gap?
      • Are patients the most appropriate source of information?
      • Evidence, Performance Gap, and Priority (Impact)—Importance to Measure and Report
      2. The intended context of use should be described and justified
      • Is the intended context of use clearly described and justified?
      • How is information from the measure expected to inform change in practice to improve performance in the intended context of use?
      • How will the nominated measure complement other measures to improve understanding of performance in the intended context of use?
      • Is there variability in the outcome at the practice or practitioner level?
      • Evidence, Performance Gap, and Priority (Impact)—Importance to Measure and Report
      • Comparison to Related or Competing Measures
      3. The measure should be adequately developed for the intended context of use (or a similar context of use), including demonstration of meaningfulness and importance to patients as well as adequate psychometric properties
      • Is the underlying concept to be measured clearly identified (e.g., postchemotherapy nausea)?
      • Is there prior or planned qualitative work in a patient population similar to the intended context of use demonstrating understanding of terminology and mapping of the terminology to the underlying concept(s) of interest?
      • Is there evidence of adequate psychometric properties of the measure, including construct validity and reliability; meaningfulness of score changes in a comparable population; and reasonableness of the recall period?
      • Reliability and Validity—Scientific Acceptability of Measure Properties
      4. There should be prior or planned work using the measure in the intended context of use (or a similar context of use), demonstrating that it is sensitive to change and clinically actionable
      • Has the measure been shown to detect changes over time or differences between known patient groups, practices, and/or procedures?
      • Does the measure detect change in clinical action(s)?
      • Is there evidence that there is not a floor or ceiling effect of the measure in the intended context of use?
      • Feasibility
      • Usability and Use
      5. There should be a recommended implementation strategy for the measure in the intended context of use
      • Is there a rationale for an administration mode (e.g., paper, electronic) and schedule (e.g., timing of follow-up evaluations)?
      • Is there a plan to maximize recruitment and response rates (e.g., backup data collection plan for nonrespondents)?
      • Is proxy or surrogate reporting considered allowable?
      • Is there a plan to accurately identify patients in the target population and calculate the denominator (i.e., number of people who were asked to complete the measure)?
      • Feasibility
      6. There should be a recommended analysis plan, including risk-adjustment strategy, missing data approach, and power calculation
      • Is there a well-justified a priori risk-adjustment or stratification strategy based on evidence?
      • Is there a plan to adjust analyses for case mix, recruitment bias, and response bias?
      • Is there a plan for imputing missing data, with sensitivity analyses?
      • What sample sizes are necessary for planned analyses?
      • Reliability and Validity—Scientific Acceptability of Measure Properties
      7. There should be a recommended framework for interpreting results, including unit(s) of analysis and meaningful score thresholds
      • What unit of analysis is recommended (e.g., hospital system, hospital, individual practice, individual practitioner, and patient-level)?
      • What metrics should be used to reflect performance (e.g., proportion of patients achieving a specific score change; proportion of providers who are outliers)?
      • How are results of different PRO measures that may not agree with each other considered?
      • Reliability and Validity—Scientific Acceptability of Measure Properties
      8. There should be a recommended approach for reporting and disseminating results
      • Is there a suggested approach for packaging and presenting reports to practices, providers, and/or patients?
      • Usability and Use
      9. There should be a recommended approach for assessing the impact of the measurement approach itself during/after implementation
      • How can the ability to distinguish between units of analysis be assessed?
      • How can the risk assessment strategy be assessed or revised?
      • Comparison to Related or Competing Measures
      The panel concluded that a PRO measure should be considered within the proposed context of use and implementation strategy. Therefore, information about these should be provided with candidate measures. It is acknowledged that not all the information specified in the table will be available or appropriate for all approaches, especially in these early days of developing and testing PRO performance measures. But the considerations should be addressed, and some may be tested within early implementation or pilot work.
      The panel concluded that an actionable PRO measure is one that can identify patients for whom changes in care might be warranted and detects changes in outcomes after treatment. Demonstration of this capacity is advisable before widespread implementation. Because the use of PROs in performance evaluation has been uncommon in the past, it is acknowledged that research in other related contexts may be relied upon initially for such evidence, such as use of a measure in clinical trials or comparative effectiveness research. When adapting an existing PROM for use in performance measurement (e.g., a measure initially developed for use in clinical research), it should be considered whether it 1) assesses outcomes that are meaningful in the population/context of interest and are actionable; 2) has been found understandable to patients in qualitative assessments; 3) demonstrates validity, reliability, and responsiveness; and 4) is feasible to implement (e.g., is not excessively lengthy, has available language translations if necessary, and can be administered electronically if necessary). Specific approaches for adapting PROMs into PRO-PMs have previously been described by the National Quality Forum (NQF), as well as details about the distinctions between PROs, PROMs, and PRO-PMs [
      National Quality Forum
      Patient Reported Outcomes (PROs) in Performance Measurement.
      ].
      The goal of measuring PROs and rewarding performance on the basis of PROs is to encourage clinicians and organizations to adopt procedures that improve outcomes experienced by patients. Accurate measurement of PROs will be hindered if variation in performance among providers primarily reflects differences in the underlying populations served and adjustment methods are not used. Quantifying changes in scores over time compared with baseline, rather than relying on cross-sectional analyses, allows patients to serve as their own controls and accounts for baseline health status. It is acknowledged that risk-adjustment approaches may be refined over time in a particular program and that risk adjustment may be deemed unnecessary in some contexts (which should be demonstrated with empiric evidence). A practical example of how the best practices may be used to develop a measure is provided in Table 3.
      Table 3Example of how the best practices can be used to develop a measure.
      A large network of community oncology practices is interested in understanding differences between practices in postchemotherapy nausea control as a potential quality measure. The rationale for this measure is published cross-sectional data about rates of nausea, and about variable compliance with published antiemetic guidelines (best practice 1). The purpose of the measure is to understand whether there are variable rates of nausea control between practices that might be improved through educational programs or feedback to providers (best practice 2). An existing nausea measure from the National Cancer Institute’s (NCI’s) PRO-CTCAE symptom library is selected on the basis of published qualitative and quantitative data in a similar community practice context, providing evidence that the measure evaluates a symptom that is important and meaningful to patients, with acceptable validity and reliability. Relevant permissions from the NCI are obtained (best practices 3 and 4). A multidisciplinary planning panel, including patient representatives, identifies the pertinent population as patients receiving moderately or highly emetogenic chemotherapy according to existing criteria (best practice 2). It is determined on the basis of a literature review to use an automated telephone system (IVRS) with live interviewer backup to assess symptoms at baseline (and to collect baseline data for risk adjustment), as well as daily following chemotherapy for 7 d (best practice 5). The proportion of patients experiencing moderate and severe nausea at more than one time point following chemotherapy at the practice level is chosen as the a priori primary end point, with risk adjustment for age, comorbidities, stage of cancer, number of prior lines of chemotherapy, time since diagnosis, baseline quality of life (via a PROMIS single measure), baseline nausea, and comorbidities (best practices 6 and 7). Exploratory analyses will be used to evaluate alternative end points in the analysis phase and refine the risk adjustment model. A sample size is determined by a biostatistician on the basis of published effect sizes for similar patients, accounting for anticipated missing data and attrition (best practice 6). Results will be fed back to practices with a planned onsite educational program offered to sites with lower performance (best practice 8). Follow-up assessment of practices following educational programs is planned to assess the impact of the programs, in addition to eliciting provider feedback about the program (best practice 9).
      IVRS, interactive voice response system; PRO-CTCAE, Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events; PROMIS, Patient-Reported Outcomes Measurement Information System.

       Harmonization with NQF White Paper and Public Comment

      During the period of development of these best practices, the NQF produced a white paper outlining a pathway for PRO-PMs [
      National Quality Forum
      Patient Reported Outcomes (PROs) in Performance Measurement.
      ,
      • Basch E.
      • Torda P.
      • Adams K.
      Patient-reported outcomes-based performance measures.
      ]. Because of the importance of the NQF as the major US organization for endorsing performance measures, it was considered important by the panel to harmonize the panel’s and the NQF’s efforts. Therefore, the panel’s best practices and the NQF pathway were mapped to each other in a table to assist measure developers who may wish to understand how the different recommendations relate to each other (Table 2).
      A draft of the best practice recommendations was posted on the Physician Consortium for Performance Improvement Web site during a 3-week public comment period. Input was elicited for each individual recommendation, and overall. The public comment process yielded feedback from 15 national and international organizations. No major substantive changes were suggested. Each comment was reviewed in a panel meeting with individual responses generated, and alterations to the recommendations were made on the basis of panel consensus.

       Purposes for Using PRO-PMs

      On the basis of the use cases, the panel identified five purposes for which PRO measures may be used in performance evaluation (Table 4). An underlying premise is that PROs should be used when information sought is best known by the patient, for example, symptoms. This information can be used along with information from other sources, such as the medical chart. The table provides illustrations of how each of the five purposes could be pursued in specific contexts of use. These purposes are not mutually exclusive, and can be addressed together in a given assessment program, sometimes within the same data collection strategy.
      Table 4Various purposes for measuring patient-reported outcomes (PROs) to assess performance, and examples.
      Purpose of PRO measureExample
      A. Assess appropriateness of patients administered a particular treatmentMeasure the proportion of patients with baseline angina before angioplasty
      B. Assess appropriateness of treatment administered to particular patientsMeasure the proportion of patients with postchemotherapy nausea for whom appropriate antiemetics are prescribed
      C. Assess impact of clinical and/or self-care management administered for a particular conditionMeasure the proportion of patients with asthma experiencing improvement in dyspnea at prespecified time points
      D. Support or complement other performance measuresMeasure postprostatectomy erectile dysfunction and urinary incontinence in combination with surgical margins and local recurrence rates
      E. Assess the process of using PROs in clinical and/or self-care practiceMeasure the proportion of patients from whom direct reports of salient symptoms are collected at prespecified time points

      Discussion

      This article presents use cases of PRO-PM development and implementation that along with several identified guidance documents informed the development of best practices. A group of purposes for developing PRO-PMs and using the best practices is also presented to guide measure developers in thinking about how PROs might best be used in a given context.
      A key underlying premise of the best practices is that evaluation of a proposed PRO performance measure warrants more than simply understanding whether the measure is valid and reliable—it requires an overall measurement strategy that takes into account the intended context of use, population, and meaningfulness to patients. It is acknowledged that at the current time, there is limited experience with these approaches, and the best practices are anticipated to evolve as evidence accumulates. It is also recognized that overly stringent standards should not hinder the endorsement of projects that may enrich knowledge and advance this area. Not all best practices will be abided by, or are appropriate, in all proposals.
      Among the five identified “purposes” for collecting PRO data in performance measurement in Table 4, the fifth, “Assess use of patient-reported outcomes to enhance clinical practice,” is perhaps the most novel. Although there is increasing interest in routinely collecting PRO measures to enhance clinical management, this is still not standard practice [
      • Gnanasakthy A.
      • Mordin M.
      • Clark M.
      • et al.
      A review of patient-reported outcome labels in the United States: 2006 to 2010.
      ,
      • Abernethy A.P.
      • Herndon II, J.E.
      • Wheeler J.L.
      • et al.
      Improving health care efficiency and quality using tablet personal computers to collect research-quality, patient-reported data.
      ,
      • Basch E.
      • Abernethy A.P.
      Supporting clinical practice decisions with real-time patient-reported outcomes.
      ]. Many operational questions remain however: how should this information optimally be collected (paper vs. electronic; in waiting rooms vs. from home between visits); when should initial and follow-up assessments be conducted; who should review this information and when; what specific information should be collected; should clinical actions resulting from this information be tracked? These questions merit future research, and approaches are expected to vary depending on the intended context of use of a measure.
      Moving forward, there are a number of challenges, some which are inherent to PRO measurement, and some that apply to performance measurement more broadly. Maximizing recruitment and response rates (particularly from the sickest and busiest patients) and analyzing missing data are perennial challenges in PRO assessment [
      • Fairclough D.L.
      • Peterson H.F.
      • Cella D.
      • Bonomi P.
      Comparison of several model-based methods for analysing incomplete quality of life data in cancer clinical trials.
      ]. Poor performance may be underdetected if ill patients systematically do not self-report, and sites that more effectively encourage sicker patients to return questionnaires may appear to have worse quality, unless analyses are adjusted for response bias. An additional challenge is identifying formats for reporting findings in a manner that is accessible, interpretable, meaningful, and actionable across stakeholders [
      • Brundage M.
      • Bass B.
      • Davidson J.
      • et al.
      Patterns of reporting health-related quality of life outcomes in randomized clinical trials: implications for clinicians and quality of life researchers.
      ].

      Conclusions

      Enthusiasm to elicit the patient perspective in performance evaluation is evolving within a broader context of patient-centeredness in health care delivery, research, and policy [
      • Selby J.V.
      • Beal A.C.
      • Frank L.
      The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda.
      ]. The overall goal of developing best practices for PRO-PMs is to provide a framework and context-appropriate specifications for supporting the development of patient-centered performance measurement strategies. It is the hope of the authors that these best practices will encourage future development of rigorous approaches that help better characterize the impact of care on the patient experience, and enable mechanisms for improving that experience across populations.
      Source of financial support: The American Medical Association provided administrative support for the development of this article.

      Supplementary Materials

      References

      1. U.S. Department of Health and Human Services. 2012 Annual Progress Report to Congress on the National Strategy for Quality Improvement in Health Care (National Quality Strategy). Available from: http://www.ahrq.gov/workingforquality/nqs/nqs2012annlrpt.pdf. [Accessed April 4, 2014].

        • Brundage M.
        • Blazeby J.
        • Revicki D.
        • et al.
        Patient-reported outcomes in randomized clinical trials: development of ISOQOL reporting standards.
        Qual Life Res. 2013; 22: 1161-1175
      2. U.S. Food and Drug Administration. Guidance for Industry. Patient-reported outcome measures: use in medical development to support labeling claims. December 2009. Available from: http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM193282.pdf. [Accessed April 4, 2014].

        • Scientific Advisory Committee of the Medical Outcomes Trust
        Assessing health status and quality of life instruments: attributes and review criteria.
        Qual Life Res. 2002; 11: 193-205
      3. European Medicines Agency. Reflection paper on the regulatory guidance for use of health-related quality of life (HRQOL) measures in the evaluation of medicinal products. July 2005. Available from: www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2009/09/WC500003637.pdf. [Accessed April 4, 2014].

        • Methodology Committee of the Patient-Centered Outcomes Research Institute (PCORI)
        Methodological standards and patient-centeredness in comparative effectiveness research: the PCORI perspective.
        JAMA. 2012; 307: 1636-1640
        • Blumenthal D.
        • Dixon J.
        Health-care reforms in the USA and England: areas for useful learning.
        Lancet. 2012; 380: 1352-1357
        • Selby J.V.
        • Beal A.C.
        • Frank L.
        The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda.
        JAMA. 2012; 307: 1583-1584
        • Gnanasakthy A.
        • Mordin M.
        • Clark M.
        • et al.
        A review of patient-reported outcome labels in the United States: 2006 to 2010.
        Value Health. 2012; 15: 437-442
        • Wu A.W.
        • Snyder C.
        • Clancy C.M.
        • Steinwachs D.M.
        Adding the patient perspective to comparative effectiveness research.
        Health Aff (Millwood). 2010; 29: 1863-1871
        • Wu A.W.
        • Snyder C.
        Getting ready for patient-reported outcomes measures (PROMs) in clinical practice.
        Healthc Pap. 2011; 11: 48-53
        • Abernethy A.P.
        • Herndon II, J.E.
        • Wheeler J.L.
        • et al.
        Improving health care efficiency and quality using tablet personal computers to collect research-quality, patient-reported data.
        Health Serv Res. 2008; 43: 1975-1991
        • Bennett A.V.
        • Jensen R.E.
        • Basch E.
        Electronic patient-reported outcome systems in oncology clinical practice.
        CA Cancer J Clin. 2012; 62: 336-347
        • Basch E.
        • Abernethy A.P.
        Supporting clinical practice decisions with real-time patient-reported outcomes.
        J Clin Oncol. 2011; 29: 954-956
        • Harrison P.L.
        • Pope J.E.
        • Coberley C.R.
        • Rula E.Y.
        Evaluation of the relationship between individual well-being and future health care utilization and cost.
        Popul Health Manag. 2012; 15: 325-330
        • Wagner L.I.
        • Zhao F.
        • Chapman J-AW
        • et al.
        Patient-reported predictors of early treatment discontinuation: NCIC JMA.27/E1Z03. San Antonio Breast Cancer Symposium, San Antonio, TX; 2011.
        Cancer Res. 2011; 71: 662
        • Chan P.S.
        • Soto G.
        • Jones P.G.
        • et al.
        Patient health status and costs in heart failure: insights from the eplerenone post-acute myocardial infarction heart failure efficacy and survival study (ephesus).
        Circulation. 2009; 119: 398-407
        • Gotay C.C.
        • Kawamoto C.T.
        • Bottomley A.
        • Efficace F.
        The prognostic significance of patient-reported outcomes in cancer clinical trials.
        J Clin Oncol. 2008; 26: 1355-1363
        • Valderas J.M.
        • Kotzeva A.
        • Espallargues M.
        • et al.
        The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature.
        Qual Life Res. 2008; 17: 179-193
        • Chen J.
        • Ou L.
        • Hollis S.J.
        A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting.
        BMC Health Serv Res. 2013; 13: 211
        • Detmar S.B.
        • Muller M.J.
        • Schornagel J.H.
        • et al.
        Health-related quality-of-life assessments and patient-physician communication: a randomized controlled trial.
        JAMA. 2002; 288: 3027-3034
        • Velikova G.
        • Booth L.
        • Smith A.B.
        • et al.
        Measuring quality of life in routine oncology practice improves communication and patient well-being: a randomized controlled trial.
        J Clin Oncol. 2004; 22: 714-724
        • Basch E.
        The missing voice of patients in drug-safety reporting.
        N Engl J Med. 2010; 362: 865-869
        • Jha A.K.
        • Orav E.J.
        • Zheng J.
        • Epstein A.M.
        Patients’ perception of hospital care in the United States.
        N Engl J Med. 2008; 359: 1921-1931
        • Lansky D.
        Overview: performance measures--the next generation.
        Jt Comm J Qual Improv. 1996; 22: 439-442
      4. Bethell C, for FACCT (Foundation for Accountablity). Patient-centered care measures for the National Health Care Quality Report. May 2000. Available from: http://www.policyarchive.org/handle/10207/bitstreams/95530.pdf. [Accessed April 4, 2014].

      5. Lansky D., Bethell C, for FACCT (Foundation for Accountability). Consumers and quality: What do they know? What do they want? Results from FACCT Consumer Research 1996-2000. Available from: http://www.policyarchive.org/handle/10207/bitstreams/95534.pdf. [Accessed April 4, 2014].

      6. Butt Z, Reeve B. Enhancing the patient’s voice: standards in the design and selection of Patient-Reported Outcomes Measures (PROMs) for use in patient-centered outcomes research. A report prepared for the Patient-Centered Outcomes Research Institute (PCORI). Available from: www.pcori.org/assets/Enhancing-the-Patients-Voice-Standards-in-the-Design-and-Selection-of-Patient-Reported-Outcomes-Measures-for-Use-in-Patient-Centered-Outcomes-Research.pdf. [Accessed April 4, 2014].

      7. Acaster S, Cimms T, Lloyd A. Development of a methodological standards report. The design and selection of patient-reported outcomes measures (PROMS) for use in patient-centered outcomes research. A report prepared for the Patient-Centered Outcomes Research Institute (PCORI). Available from: http://www.pcori.org/assets/The-Design-and-Selection-of-Patient-Reported-Outcomes-Measures-for-Use-in-Patient-Centered-Outcomes-Research.pdf. [Accessed April 4, 2014].

        • Vickers A.J.
        • Sjoberg D.
        • Basch E.
        • et al.
        How do you know if you are any good? A surgeon performance feedback system for the outcomes of radical prostatectomy.
        Eur Urol. 2012; 61: 284-289
        • Chen P.C.
        • Lee Y.C.
        • Kuo R.N.
        Differences in patient reports on the quality of care in a diabetes pay-for-performance program between 1 year enrolled and newly enrolled patients.
        Int J Qual Health Care. 2012; 24: 189-196
        • Weinstein J.N.
        • Tosteson T.D.
        • Lurie J.D.
        • et al.
        Surgical vs. nonoperative treatment for lumbar disk herniation: the Spine Patient Outcomes Research Trial (SPORT): a randomized trial.
        JAMA. 2006; 296: 2441-2450
        • Jones N.
        • Jones S.L.
        • Miller N.A.
        The Medicare Health Outcomes Survey program: overview, context, and near-term prospects.
        Health Qual Life Outcomes. 2004; 2: 33-43
        • Gilbert J.E.
        • Howell D.
        • King S.
        • et al.
        Quality improvement in cancer symptom assessment and control: the Provincial Palliative Care Integration Project (PPCIP).
        J Pain Symptom Manage. 2012; 43: 663-678
        • Wu A.W.
        • Young Y.
        • Skinner E.A.
        • et al.
        Quality of care and outcomes of adults with asthma treated by specialists and generalists in managed care.
        Arch Intern Med. 2001; 161: 2554-2560
        • Appleby J.
        • Devlin N.
        Measuring Success in the NHS: Using Patient-Assessed Health Outcomes to Manage the Performance of Health Care Providers.
        King’s Fund, London2004 (Available from: http://www.staff.city.ac.uk/n.j.devlin/measuring%20success%20in%20the%20NHS.pdf. [Accessed April 4, 2014])
      8. Swedish Rheumatology Registers (sRR). Annual report from sRR. 2011. Available from: www.swerre.se/cms/sv/provider/reports. [Accessed April 4, 2014].

      9. National Health Service, National Clinical Audit Support Programme (NCASP). Overview of Mastectomy and Breast Reconstruction (MBR) audit for women in England and Wales. Available from: www.ic.nhs.uk/mbr. [Accessed April 4, 2014].

      10. Centers for Medicare & Medicaid Services. Transcatheter aortic valve replacement (TAVR) National Coverage Determination for Coverage with Evidence Development (CED). Available from: http://www.cms.gov/Medicare/Medicare-General-Information/MedicareApprovedFacilitie/Transcatheter-Aortic-Valve-Replacement-TAVR.html. [Accessed April 4, 2014].

      11. The Commonwealth Fund. Purchasing high performance Boeing’s nurse case managers cut per capita costs by 20 percent. Available from: www.commonwealthfund.org/Newsletters/Purchasing-High-Performance/2011/March-29-2011/Case-Study/Boeings-Nurse-Case-Managers.aspx. [Accessed April 4, 2014].

      12. Minnesota Community Measurement. Minnesota Health Scores overview. Available from: www.mnhealthscores.org. [Accessed April 4, 2014].

      13. Royal College of Obstetricians and Gynaecologists. Heavy Menstrual Bleeding Audit Programme description. Available from: http://www.rcog.org.uk/orca/audit. [Accessed April 4, 2014].

        • Mokkink L.B.
        • Terwee C.B.
        • Patrick D.L.
        • et al.
        The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes.
        J Clin Epidemiol. 2010; 63: 737-745
        • Coons S.J.
        • Gwaltney C.J.
        • Hays R.D.
        • et al.
        Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO Good Research Practices Task Force report.
        Value Health. 2009; 12: 419-429
        • Wild D.
        • Grove A.
        • Martin M.
        • et al.
        Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (PRO) measures: report of the ISPOR Task Force for Translation and Cultural Adaptation.
        Value Health. 2005; 8: 94-104
      14. Johnson C, Aaronson N, Blazeby JM, et al. EORTC Quality of Life Group: guidelines for developing questionnaire modules. 2011. Available from: http://groups.eortc.be/qol/Pdf%20presentations/Guidelines%20for%20Developing%2 0questionnaire-%20FINAL.pdf. [Accessed April 4, 2014].

        • Rothman M.
        • Burke L.
        • Erickson P.
        • et al.
        Use of existing patient-reported outcome (PRO) instruments and their modification: the ISPOR Good Research Practices for Evaluating and Documenting Content Validity for the Use of Existing Instruments and Their Modification PRO Task Force Report.
        Value Health. 2009; 12: 1075-1083
        • Revicki D.A.
        • Osoba D.
        • Fairclough D.
        • et al.
        Recommendations on health-related quality of life research to support labeling and promotional claims in the United States.
        Qual Life Res. 2000; 9: 887-900
        • Fairclough D.L.
        • Peterson H.F.
        • Cella D.
        • Bonomi P.
        Comparison of several model-based methods for analysing incomplete quality of life data in cancer clinical trials.
        Stat Med. 1998; 17: 781-796
        • National Quality Forum
        Patient Reported Outcomes (PROs) in Performance Measurement.
        National Quality Forum, Washington, DC2013 (Available from http://www.qualityforum.orgProjectsn-rPatient-Reported_OutcomesPatient-Reported_Outcomes.aspx)
        • Basch E.
        • Torda P.
        • Adams K.
        Patient-reported outcomes-based performance measures.
        JAMA. 2013; 310: 139-140
        • Brundage M.
        • Bass B.
        • Davidson J.
        • et al.
        Patterns of reporting health-related quality of life outcomes in randomized clinical trials: implications for clinicians and quality of life researchers.
        Qual Life Res. 2011; 20: 653-664
        • Hutchings A.
        • Grosse Frie K.
        • Neuburger J.
        • et al.
        Late response to patient reported outcome questionnaires was associated with worse outcome.
        J Clin Epidemiol. 2013; 66: 218-225
        • Vickers A.J.
        • Savage C.J.
        • Shouery M.
        • et al.
        Validation study of a web-based assessment of functional recovery after radical prostatectomy.
        Health Qual Life Outcomes. 2010; 8: 82-88
      15. Society of Thoracic Surgeons and American College of Cardiology. TVT registry for transcatheter aortic valve replacement (TAVR) procedure. Available from: https://www.ncdr.com/TVT/Home/Default.aspx. [Accessed April 4, 2014].

      16. Dartmouth Institute for Health Policy and Clinical Practice. Peer reviewed technical report: using patient-reported information to improve health outcomes and health care value: case studies from Dartmouth, Karolinska, and Group Health. 2012. Available from: http://tdi.dartmouth.edu/images/uploads/tdi_tr_pri_ia_sm.pdf. [Accessed April 4, 2014].

      17. Centers for Medicare & Medicaid Services. Medicare Health Outcomes Survey: program overview. Available from: http://www.hosonline.org/Content/Default.aspx. [Accessed April 4, 2014].