Advertisement

Development of System Performance Indicators for Adolescent and Young Adult Cancer Care and Control in Canada

Open ArchivePublished:October 11, 2019DOI:https://doi.org/10.1016/j.jval.2019.08.004

      Highlights

      • Metrics for evaluating and monitoring are essential for successful programs of cancer care and control.
      • Healthcare metrics for adolescent and young adult (AYA) patients with cancer have not been researched adequately.
      • This article provides a first set of indicators in Canada for AYAs (15-39 years of age) with cancer, a population with special needs who have been systematically under-served.
      • The proposed indicators can serve as metrics to evaluate and monitor changes in AYA cancer care and control.

      Abstract

      Objectives

      To develop an expert-group, consensus-based list of system performance indicators to be used for monitoring, evaluating, and benchmarking progress for cancer care and control in adolescents and young adults (AYAs) in Canada.

      Methods

      A national multidisciplinary panel of AYA oncology experts was convened; they prepared a literature review and undertook a brainstorming exercise to create a comprehensive list of indicators based on a previously defined framework for AYA cancer care and control in Canada. A modified Delphi process was then undertaken to cull the list based on 3 quick screen criteria. Three rounds of ranking were required. The fourth stage employed a face-to-face meeting, and the final stage utilized a survey to rank the indicators on the basis of importance and feasibility.

      Results

      Nineteen participants contributed to the 5-stage process. From an initial list of 114 indicators, 14 were ultimately endorsed, representing 5 themes: active care, survivorship, psychosocial issues, palliative care, and research. The 5 highest ranked indicators were assessed as very to moderately feasible, with only a single indicator (clinical trial enrollment) in the top 5 assigned a least feasible ranking.

      Conclusion

      The 14 indicators provide a starting point for the development of a standard set of metrics for AYA cancer care and control in Canada and have potential for international utility.

      Keywords

      Introduction

      There have been considerable efforts in several high-income countries to improve the care of adolescents and young adults (AYAs) with cancer,
      • Barr R.D.
      • Ferrari A.
      • Ries L.
      • Whelan J.
      • Bleyer W.A.
      Cancer in adolescents and young adults. A narrative review of the current status and a view of the future.
      a group defined as those 15 to 39 years of age at the time of diagnosis.
      Department of Health and Human Services, National Institutes of Health, National Cancer Institute, LiveStrong Young Adult Alliance
      Closing the gap: research and care imperatives for adolescents and young adults with cancer: a report of the Adolescent and Young Adult Oncology Progress Review Group.
      AYAs with cancer have many unmet needs, which are distinct from those of pediatric and older adult cancer patients.
      • Ferrari A.
      • Albritton K.
      • Osborn M.
      • et al.
      Access and models of care.
      The biology of their disease and their stage in development make them a unique cancer population who require tailored interventions to address their specific needs
      • Zebrack B.
      • Mathews-Bradshaw B.
      • Siegel S.
      • Alliance L.Y.A.
      Quality cancer care for adolescents and young adults: a position statement.
      and specific measurements of the performance of these interventions on AYA cancer care and control. Canada is a nation with both population-based cancer control programs and an effective public health system
      • Sutcliffe S.B.
      A review of Canadian health care and cancer care systems.
      ; however, it still faces many challenges to providing optimal cancer care and control for AYAs. Although the funding for healthcare is in part federal, the health system is administered by each of the individual provinces and territories, making the implementation of national programs and standardized data collection politically and administratively challenging. To improve outcomes in AYAs with cancer and overcome these challenges, a national initiative is required.
      • Ferrari A.
      • Albritton K.
      • Osborn M.
      • et al.
      Access and models of care.
      • Bleyer A.
      Cancer in adolescents and young adults: latest estimates of U.S. survival and years of life affected.
      Key to such an investment are metrics for monitoring and evaluating AYA cancer care and control.
      Approximately 8000 AYAs receive a diagnosis of cancer each year in Canada and more than 80% survive their disease.

      Canadian Partnership Against Cancer. Adolescent & Young Adults with Cancer: Technical Report. Toronto, ON: 2017.

      In 2008 a National Task Force was formed to address AYA cancer care and control with funding from the Canadian Partnership Against Cancer (CPAC). The Task Force’s mission was “to ensure that AYA Canadians with cancer and AYA survivors of cancer have prompt, equitable access to the best care, and to establish and support research to identify how their health outcomes and health-related quality of life can be optimised.”
      • Barr R.
      • Rogers P.
      • Schacter B.
      Adolescents and young adults with cancer: towards better outcomes in Canada.
      The Task Force held an international workshop in 2010 to develop a framework for AYA cancer care and control in Canada
      and another in 2012 to develop a plan for action.
      • Rogers P.C.
      • De Pauw S.
      • Schacter B.
      • et al.
      A process for change in the care of adolescents and young adults with cancer in Canada. “Moving to action”: the second Canadian international workshop. International perspectives on AYAO, part 1.
      • Fernandez C.
      • Fraser G.A.
      • Freeman C.
      • et al.
      Principles and recommendations for the provision of healthcare in Canada to adolescent and young adult–aged cancer patients and survivors.
      Stakeholders at these workshops came from all provinces with representatives from health professionals, administrators, researchers, nongovernmental organizations (NGOs), and survivors. The framework identified themes with key recommendations and priority areas, which needed to be addressed within the Canadian context to improve care and outcomes for AYAs with cancer.
      • Fernandez C.
      • Fraser G.A.
      • Freeman C.
      • et al.
      Principles and recommendations for the provision of healthcare in Canada to adolescent and young adult–aged cancer patients and survivors.
      To help facilitate and monitor expected improvements based on this framework, it became important to determine appropriate system performance indicators. Relevant metrics can identify opportunities to improve quality of care and benchmarks to achieve short-term, medium-term, and long-term outcome-improvement goals. AYA-specific metrics would help to inform decisions regarding policy and resource allocation within the healthcare system.
      Previous work on indicator development for cancer care and control in Canada has focused predominantly on the separate pediatric and adult age groups.
      • Bradley N.M.E.
      • Robinson P.D.
      • Greenberg M.L.
      • et al.
      Measuring the quality of a childhood cancer care delivery system: assessing stakeholder agreement.
      • Bradley N.M.E.
      • Robinson P.D.
      • Greenberg M.L.
      • et al.
      Measuring the quality of a childhood cancer care delivery system: quality indicator development.
      • Greenberg A.
      • Angus H.
      • Sullivan T.
      • et al.
      Development of a set of strategy-based system-level cancer care performance indicators in Ontario, Canada.
      • Barbera L.
      • Paszat L.
      • Chartier C.
      Indicators of poor quality end-of-life cancer care in Ontario.
      • Gagliardi A.R.
      • Fung M.F.K.
      • Langer B.
      • et al.
      Development of ovarian cancer surgery quality indicators using a modified Delphi approach.
      • Gagliardi A.R.
      • Simunovic M.
      • Langer B.
      • et al.
      Development of quality indicators for colorectal cancer surgery, using a 3-step modified Delphi approach.
      Canadian Partnership Against Cancer (CPAC)
      Systems performance: indicator index.
      Recently a comprehensive set of pediatric cancer indicators was developed by the Pediatric Oncology Group of Ontario (POGO), which included a few specific indicators for adolescents between 15 and 18 years of age.
      • Bradley N.M.E.
      • Robinson P.D.
      • Greenberg M.L.
      • et al.
      Measuring the quality of a childhood cancer care delivery system: quality indicator development.
      At a national level, CPAC has developed indicators for cancer control that are included in its system performance reports, but these are not focused specifically on cancer in AYAs.
      Canadian Partnership Against Cancer (CPAC)
      Systems performance: indicator index.
      In April 2017, based on the accomplishments of the indicator working group of the AYA Task Force, CPAC released a system performance report on indicators for AYA cancer care in Canada.

      Canadian Partnership Against Cancer. Adolescent & Young Adults with Cancer: Technical Report. Toronto, ON: 2017.

      These indicators were selected on the basis of current feasibility to accrue the appropriate data from an initial list produced by the Task Force. Greenberg et al also summarized metrics for AYA cancer care, which were identified through brainstorming with stakeholders.
      • Greenberg M.
      • Klassen A.
      • Gafni A.
      • et al.
      Outcomes and metrics.
      No further effort was made to develop and finalize the list since that publication. Internationally, some work has been done in Australia to develop indicators for AYA cancer care,
      Adolescent and Young Adult Working Party of the Statewide Cancer Clinical Network
      South Australian adolescent and young adult cancer care pathway.
      but much more remains to be undertaken in this important area. It is crucial to build on this work with a rigorous approach to ensure that indicators address key priorities and recommendations for AYA cancer care and control in Canada and to allow for stakeholder involvement to help improve the uptake and use of the indicators.
      The overall objective of this study was to develop an expert-group, consensus-based list of system performance indicators to be used for monitoring, evaluating, and benchmarking progress for cancer care and control in AYAs in Canada. Our secondary objective was to create a refined list of indicators based on feasibility and the availability of relevant data.

      Methods

      Methods for indicator development were based largely on those used by POGO in their compilation of a list of indicators for childhood cancer.
      • Bradley N.M.E.
      • Robinson P.D.
      • Greenberg M.L.
      • et al.
      Measuring the quality of a childhood cancer care delivery system: assessing stakeholder agreement.
      • Bradley N.M.E.
      • Robinson P.D.
      • Greenberg M.L.
      • et al.
      Measuring the quality of a childhood cancer care delivery system: quality indicator development.
      Briefly, the methods entail the formation of an expert group who develop a comprehensive list, followed by a modified Delphi
      • Boulkedid R.
      • Abdoul H.
      • Loustau M.
      • Sibony O.
      • Alberti C.
      Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review.
      process to arrive at a culled list. A final assessment of importance and feasibility was undertaken by survey. Figure 1 provides an overview of the process used in this study.
      Figure thumbnail gr1
      Figure 1Overview of modified Delphi process.

      Participants

      An invitation was sent to participants who had contributed to the Canadian Task Force on AYAs with Cancer or who had attended at least one of its workshops. Responding participants were selected to ensure representation from all stakeholder groups and to engage as many provinces as possible. The group was referred to as the system performance metrics (SPM) group.

      Framework

      The SPM group reviewed the principles and recommendations for AYA cancer care and control that were created from the 2010 multistakeholder workshop and revised at the 2012 workshop.
      • Fernandez C.
      • Fraser G.A.
      • Freeman C.
      • et al.
      Principles and recommendations for the provision of healthcare in Canada to adolescent and young adult–aged cancer patients and survivors.
      These workshops involved input from 100 individuals representing healthcare professionals, administrators, survivors, advocates, and international content experts.

      Canadian Partnership Against Cancer. Adolescent & Young Adults with Cancer: Technical Report. Toronto, ON: 2017.

      The resulting comprehensive framework addressed 7 themes: active therapy and supportive care; psychosocial needs; palliation and symptom management; survivorship; research and metrics; education, awareness, and advocacy; and prevention (Figure 2).
      • Rogers P.C.
      • De Pauw S.
      • Schacter B.
      • et al.
      A process for change in the care of adolescents and young adults with cancer in Canada. “Moving to action”: the second Canadian international workshop. International perspectives on AYAO, part 1.
      • Fernandez C.
      • Fraser G.A.
      • Freeman C.
      • et al.
      Principles and recommendations for the provision of healthcare in Canada to adolescent and young adult–aged cancer patients and survivors.
      Figure thumbnail gr2
      Figure 2Indicator development framework.
      Adapted from Fernandez et al.
      • Fernandez C.
      • Fraser G.A.
      • Freeman C.
      • et al.
      Principles and recommendations for the provision of healthcare in Canada to adolescent and young adult–aged cancer patients and survivors.

      Development of a Comprehensive List of Potential AYA Cancer Care Indicators

      A literature review was undertaken of both the scientific and gray literature to provide the foundation for the work of the SPM group. The Ovid database was utilized to search the scientific literature based on a modified search strategy, adapted from the systematic review conducted by Bradley et al (see Appendix 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2019.08.004).
      • Bradley N.M.E.
      • Robinson P.D.
      • Greenberg M.L.
      • et al.
      Measuring the quality of a childhood cancer care delivery system: quality indicator development.
      Search strategy constructs included: cancer, quality indicators, healthcare, adolescents, and young adults. Titles and abstracts were reviewed in EndNote, version X7.7.1, for relevance to AYA cancer and quality of care. Only work focused on cancer specifically in the AYA age range (15-39 years) was included. The following data were extracted: age range(s), indicator, and indicator definition. Selected indicators were categorized by framework theme. Queries were sent to international AYA cancer groups regarding their use of indicators, including CanTeen (Australia), Teenage Cancer Trust (UK), Children’s Oncology Group (USA), and Critical Mass (USA). The websites of provincial cancer agencies and the Rossy Cancer Network were viewed to collect information regarding cancer indicator work at the provincial level within Canada. For the gray literature search, Google was searched using the constructs described earlier. Search results were reviewed for items focused specifically on indicators for cancer care in AYAs. In addition, SPM group members completed a brainstorming exercise wherein they provided indicators or measurement constructs they thought would be important to capture based on the framework. A worksheet (see Appendix 2 in Supplemental Materials) was emailed to participants along with the framework document. Results were aggregated and duplicate indicators and concepts were removed. Indicators were grouped according to the 7 framework themes.
      • Fernandez C.
      • Fraser G.A.
      • Freeman C.
      • et al.
      Principles and recommendations for the provision of healthcare in Canada to adolescent and young adult–aged cancer patients and survivors.

      Item Reduction

      To reduce the list of indicators compiled from the literature review and brainstorming exercise, two rounds of item reduction were conducted using surveys. Item reduction deviated somewhat from the POGO methods.
      • Bradley N.M.E.
      • Robinson P.D.
      • Greenberg M.L.
      • et al.
      Measuring the quality of a childhood cancer care delivery system: assessing stakeholder agreement.
      • Bradley N.M.E.
      • Robinson P.D.
      • Greenberg M.L.
      • et al.
      Measuring the quality of a childhood cancer care delivery system: quality indicator development.
      During this stage, POGO utilized 4 expert reviewers to assess 4 quick screen criteria: importance, relevance, applicability to the mission statement, and alignment with strategic objectives. We aimed to maximize stakeholder involvement and so decided to include all stakeholders rather than limit it to expert review. Maximizing stakeholder involvement was thought to be important because of the nature of AYA cancer care across pediatric and adult institutions and the current lack of formal entities addressing AYA cancer care in Canada. To gain “buy-in” from all stakeholder groups, we believe it was important to include them throughout the entire process. From the POGO experience,
      • Bradley N.M.E.
      • Robinson P.D.
      • Greenberg M.L.
      • et al.
      Measuring the quality of a childhood cancer care delivery system: assessing stakeholder agreement.
      overall importance and relevance were selected because it was thought that alignment with mission statements and objectives was covered by the use of the AYA framework. Importance addressed whether the indicator captured a critical aspect of care for AYA cancer patients. Relevance captured the use of the indicator for measuring quality care specific to AYA, rather than to general cancer care. The last criterion represented a simplification of POGO’s primary selection criteria
      • Bradley N.M.E.
      • Robinson P.D.
      • Greenberg M.L.
      • et al.
      Measuring the quality of a childhood cancer care delivery system: assessing stakeholder agreement.
      to capture the “usefulness” of the indicator in quality improvement. Usefulness captured the face validity of the indicator, which, if implemented, would make sense with the potential for change. The 7 POGO primary selection criteria were simplified to reduce the survey burden without hindering the judgments of respondents.
      Each SPM group member assessed each indicator based on the responses to the 3 screening criteria of importance, relevance, and usefulness. Survey responses were dichotomous (yes/no) to have respondents commit to whether the indicator met the criteria. An overall score for each indicator from each participant was calculated by summing the total number of “yes” responses. Therefore the highest possible score for each indicator was 3. The proportion of all respondents providing a total score of 3 was calculated for each indicator. During the POGO reduction rounds, the agreement of 4 experts was the criterion for inclusion of indicators.
      • Bradley N.M.E.
      • Robinson P.D.
      • Greenberg M.L.
      • et al.
      Measuring the quality of a childhood cancer care delivery system: quality indicator development.
      Because we modified the process to maintain stakeholder participation, we instituted a cut-point of 70% for the first two surveys. This cut-point was 10% less than the higher agreement threshold of 80% used in the consensus meeting because we wanted to be more conservative, given the potential variability of survey responses from a wide range of respondents. For the first round, indicators were retained if 70% or more of the respondents had a total score of 3 for the specific indicator. Indicators meeting this criterion were included in the second round survey after further wording clarification was undertaken by the group. The second round survey followed the same format and analysis plan as the first round survey. The chosen cut-point was altered for the second reduction round with a more conservative cut-point of 60% implemented because of a smaller number of survey respondents. The group required that one indicator be retained under each theme until the final meeting, regardless of score and cut-point. When no indicator was selected, the indicator that was closest to the cut-point was retained. In the event of a tie, both indicators were kept. After completion of the second round survey, the CPAC technical report, based on a selection of feasible indicators from the brainstorming round of this work, was released.

      Canadian Partnership Against Cancer. Adolescent & Young Adults with Cancer: Technical Report. Toronto, ON: 2017.

      A review of this list was conducted by the group in a teleconference to determine whether any of the previously discarded indicators should be rescued and considered at the final meeting.

      Final Meeting

      The third and final round of consensus-building was established via a face-to-face meeting. In advance of the meeting, group members were invited to complete a survey scoring the remaining indicators based on the 3 criteria used in the previous 2 survey rounds: importance, relevance, and usefulness. The assessment scale was a 6-point scale ranging from 0 (not at all) to 5 (highest); a large scale was used to provide more information regarding each criterion for ranking purposes. Results from this survey were summarized and distributed to the group 1 week before the meeting. Analyses were conducted in SPSS 25.0 (IBM SPSS Statistics, version 25, IBM Corp) and included summary statistics (mean, standard deviation, median, minimum, and maximum), agreement statistics, and overall within group ranking. Ranking was based on percent agreement ordered by importance, relevance, and usefulness. Participants were asked to review the results before the meeting and complete a further survey selecting indicators, which they would endorse based on these results. Cutoff values were chosen to correspond with those described in the stakeholder agreement stage of the POGO indicator selection process.
      • Bradley N.M.E.
      • Robinson P.D.
      • Greenberg M.L.
      • et al.
      Measuring the quality of a childhood cancer care delivery system: quality indicator development.
      If an indicator received an endorsement of 80% or greater on the survey, the indicator was considered endorsed and excluded from further discussion. The final meeting was held in Toronto in October 2018. Members were invited to attend in person or via teleconference. The first part of the meeting provided a review of the process to date and a more in-depth review of the results of the surveys completed before the meeting. After discussion, a second survey was circulated to the group, and they were asked to review and endorse (yes/no) up to 50% of the remaining indicators based on the criteria presented.
      • Bradley N.M.E.
      • Robinson P.D.
      • Greenberg M.L.
      • et al.
      Measuring the quality of a childhood cancer care delivery system: quality indicator development.
      Indicators not receiving any endorsement after this round were considered not endorsed and removed from the process. The final part of the meeting involved an in-depth discussion of each of the remaining indicators. A third round survey was circulated after the discussion, and participants were asked to select the remaining indicators they would endorse. Any indicator receiving an endorsement of 80% or greater on this final survey was considered endorsed, with all other indicators considered not endorsed.

      Ranking

      A survey was sent to participants after the final meeting to rank the endorsed indicators for both current feasibility and importance and to elicit potential data sources for each indicator. For the feasibility criterion, participants were asked, “Given current data availability, how feasible would it be to implement this indicator nationally?” The importance criterion was worded as in previous surveys. Both criteria were rated on a scale of 0 (not at all) to 10 (highest). The mean was calculated for the criteria and indicators were rank-ordered based on the mean from highest to lowest for both feasibility and importance. Ties in rankings were broken by totaling the number of 9 and 10 ratings received by the indicators; the higher ranking was assigned to the indicator with more 9+10 ratings. Final indicator selections were mapped onto the quality dimensions from the Cancer Quality System Index (CQSI)
      • Cancer Quality Council of Ontario
      Cancer System Quality Index (CSQI): quality dimensions.
      to support the applicability of the indicators for improving the quality of the cancer system for AYAs.

      Results

      Participants

      Nineteen participants were involved throughout the various stages in the process, but not all participants contributed to each phase. Variation in participation stemmed from multiple factors, including changes in employment, length of time to complete process, and competing priorities. Participants included representatives from 7 Canadian provinces and the United States. Stakeholders consisted of pediatric oncologists, adult oncologists, nurses, researchers, administrators, nongovernmental organizations or charity representatives, and a patient representative. Sustained engagement was maintained for 8 representatives, who participated in the majority of stages throughout the indicator selection process. Because of the time span of the work, maintaining engagement for all members was challenging.

      Comprehensive Indicator List

      A total of 5287 articles were identified from the literature review; after review, 93 were deemed relevant to cancer care indicators. Upon further review, only two articles were judged to be relevant to indicators for AYA cancer care and control: the article by Greenberg et al
      • Greenberg M.
      • Klassen A.
      • Gafni A.
      • et al.
      Outcomes and metrics.
      on AYA outcomes and metrics and the POGO childhood cancer indicator study that included adolescent-specific indicators.
      • Bradley N.M.E.
      • Robinson P.D.
      • Greenberg M.L.
      • et al.
      Measuring the quality of a childhood cancer care delivery system: assessing stakeholder agreement.
      • Bradley N.M.E.
      • Robinson P.D.
      • Greenberg M.L.
      • et al.
      Measuring the quality of a childhood cancer care delivery system: quality indicator development.
      • Greenberg M.
      • Klassen A.
      • Gafni A.
      • et al.
      Outcomes and metrics.
      The gray literature search provided a report on optimizing outcomes for all South Australian AYAs with a cancer diagnosis.
      Adolescent and Young Adult Working Party of the Statewide Cancer Clinical Network
      South Australian adolescent and young adult cancer care pathway.
      This listed 15 indicators, focused on the 15- to 25-year-old age group, on the themes of active care, survivorship, palliative care, and cancer awareness. The brainstorming exercise was completed by 11 of 19 respondents. Results from the brainstorming were combined with the literature review, resulting in 131 indicators. Seventeen were considered duplicates, leaving 114 indicators for further consideration.

      Indicator List Reduction

      Forty-three indicators remained after the first survey, which was completed by 13 of 18 respondents (Table 1). The group clarified the indicator text afterward from survey feedback and group discussion. The survey was updated based on the revised indicator list and redistributed to the group. The second survey was completed by 8 of 18 respondents. Participation in the second survey dropped because of competing priorities with the publication of the CPAC AYA system performance report.
      • Barr R.
      • Rogers P.
      • Schacter B.
      Adolescents and young adults with cancer: towards better outcomes in Canada.
      The 60% endorsement criterion was met by 29 indicators, with one indicator (“proportion of oncology professionals with AYA certification/expertise”) being retained to ensure that an indicator was included under the education theme. The group agreed to rescue the “research funding” indicator from the CPAC report list, as it had been discarded at an early stage in the process. Thirty-one indicators were selected to continue for review; these are listed in Table 2.
      Table 1Demographics.
      Brainstorming

      N = 11
      Survey 1

      N = 13
      Survey 2

      N = 8
      Pre-meeting survey

      N = 11
      Final meeting and ranking

      N = 10
      n%n%n%n%n%
      Province
       British Columbia19.1%215.4%337.5%218.2%110.0%
       Alberta19.1%17.7%112.5%00.0%00.0%
       Manitoba218.2%215.4%112.5%218.2%220.0%
       Ontario327.3%430.8%112.5%545.5%550.0%
       Quebec00.0%17.7%00.0%00.0%00.0%
       Nova Scotia218.2%215.4%225.0%218.2%220.0%
       Newfoundland19.1%17.7%00.0%00.0%00.0%
       USA19.1%00.0%00.0%00.0%00.0%
      Sex
       Male763.6%969.2%450.0%763.6%770.0%
       Female436.4%430.8%450.0%436.4%330.0%
      Stakeholder Group
       Pediatric Oncologist218.2%538.5%337.5%436.4%440.0%
       Adult Oncologist19.1%323.1%112.5%327.3%220.0%
       Nurse218.2%00.0%112.5%19.1%110.0%
       Researcher218.2%215.4%112.5%19.1%110.0%
       Administrator19.1%215.4%112.5%19.1%110.0%
       NGO/Charity218.2%17.7%00.0%19.1%110.0%
       Patient Representative19.1%00.0%00.0%00.0%00.0%
      NGO indicates nongovernmental organization.
      Table 2Summary statistics for pre-meeting survey.
      Importance

      Is this indicator important and appropriate from a systems perspective for managing the care of AYA cancer patients?
      Relevance

      Does this indicator capture an issue relevant to improving quality of care for AYAs with cancer?
      Usefulness

      If initiatives were taken in the healthcare system, would we expect to see change in this indicator?
      Rank
      IndicatorMedMinMax%MedMinMax%MedMinMax%OverallWithin Group
      Active
       Proportion of AYA patients who had fertility preservation discussion before treatment53590.90%51590.90%53581.80%31
       Proportion of AYA patients treated according to a clinical trial protocol42581.80%52581.80%43563.60%102
       Time from first healthcare visit with symptom to diagnosis52572.70%52590.90%40572.70%143
       Proportion of AYA diagnosed with cancer who have met with or are referred to a social worker42572.70%42554.50%41554.50%204
       Proportion of AYA diagnosed with cancer reporting positive ratings for satisfaction of active care42563.60%42554.50%41563.60%225
       Proportion of patients presented at multidisciplinary team meeting42563.60%42554.50%32545.50%236
       Proportion of AYA diagnosed with cancer who report having knowledge about diagnosis and treatment options43560.00%32545.50%41554.50%247
       Proportion of AYA patients referred for fertility treatment41554.50%42572.70%52563.60%268
      Survivorship
       Proportion of patients who are provided with a treatment summary at end of treatment (record of cancer treatment received)53590.90%53581.80%53581.80%51
       Proportion of AYA patients who have access to age-appropriate resources for educational, vocational, and psychosocial support42581.80%53581.80%42572.70%82
       Proportion of AYA patients with cancer who received reproductive counseling in follow-up care52581.80%52581.80%53572.70%93
       Proportion of female cancer survivors who received chest radiation and have been screened for breast cancer according to follow-up guidelines52581.80%52572.70%52572.70%124
       Mean HRQL score of cancer survivors aged 15-39 years post treatment41572.70%42581.80%42554.50%165
       Proportion of treatment centers with AYA-specific survivorship programs42563.60%31536.40%42572.70%256
      Psychosocial
       Proportion of programs that have psychology or psychiatry support available for AYA patients53590.90%52563.60%42572.70%71
       Proportion of AYA patients screened for distress51572.70%51581.80%52581.80%152
       Proportion of AYA patients identified to have distress through screening41554.50%41563.60%42554.50%273
      Palliation
       Proportions of centers offering AYA-specific palliative care services52590.90%43590.90%42581.80%41
       Proportion of facilities offering palliative home care programs for AYA patients42581.80%43563.60%42554.50%132
       Proportion of AYA patients who die in hospital31545.50%40554.50%30527.30%313
      Research
       AYA progression or event-free survival545100.00%555100.00%52572.70%11
       AYA overall survival52590.90%52590.90%53590.90%22
       Cause of death (short and long): proportion of AYA diagnosed with cancer who die of their disease at 5, 10, and 20 years post diagnosis52590.90%52581.80%42563.60%63
       Proportion of AYA enrolled into clinical trials53581.80%52581.80%42563.60%114
       Proportion of AYA diagnosed with cancer who are also diagnosed with at least one chronic condition other than original cancer42572.70%52572.70%42554.50%175
       Proportion of AYA patients accrued to treatment clinical trials41572.70%52563.60%41554.50%186
       Proportion of funded grants for AYA cancer research41554.50%42554.50%31527.30%287
       Proportion of GPs reporting that they feel comfortable recognizing common presenting symptoms of AYA cancers31545.50%42554.50%31536.40%308
      Education
       Proportion of oncology professionals with AYA certification/expertise31445.50%42554.50%41572.70%291
      Prevention
       HPV vaccination rate (primary prevention)51572.70%41563.60%50554.50%191
       HPV vaccination rate in AYA who are survivors of cancer41563.60%41563.60%40554.50%212
      AYA indicates adolescent and young adult; GP, general practitioner; HPV, human papilloma virus.

      Final Meeting

      All 11 participants invited to the final meeting completed the survey. Results of the premeeting survey are provided in Table 2. Nine participants were able to attend the final meeting, 4 in person and 5 via teleconference. Table 3 summarizes the results of the 3 endorsement rounds at which a total of 15 indicators were endorsed from 5 of the 7 framework themes. There were no indicators endorsed for the prevention or education and awareness themes. Owing to the similarities between the 2 survival indicators endorsed by the group, it was agreed to create 1 survival indicator with 2 metrics for a final total of 14 indicators.
      Table 3Summary of endorsement results from final meeting.
      RoundEndorsed

      Indicator (% agreement)
      Not endorsed

      Indicator (% agreement)
      Round 1 n = 10
      • Active care
        • Survival indicators:
          • AYA patient progression or event-free survival (90%)
          • AYA patient overall survival (100%)
        • Proportion of AYA patients who had fertility preservation discussion before treatment (90%)
      • Survivorship
        • Proportion of patients who are provided with a treatment summary at end of treatment (90%)
        • Proportion of AYA patients who have access to age-appropriate resources for educational, vocational, and psychosocial support (90%)
        • Proportion of female cancer survivors who received chest radiation who have been screened for breast cancer according to follow-up guidelines (80%)
        • Proportion of AYA patients with cancer who received reproductive counseling in follow-up care (80%)
      • Psychosocial
        • Proportion of programs that have psychology or psychiatry support available for AYA patients (80%)
      • Palliation
        • Proportions of centers offering AYA-specific palliative care services (80%)
      • Research
        • Cause of death (short and long): proportion of AYA survivors who die of their disease post diagnosis (80%)
        • Proportion of AYA enrolled into clinical trials (80%)
      Round 2 n = 9
      • Prevention
        • HPV vaccination rates in AYA patients who are survivors of cancer (0%)
      Round 3 n = 9
      • Active care
        • Proportion of patients presented at multidisciplinary team meeting (89%)
      • Psychosocial
        • Proportion of AYA diagnosed with cancer who have met with a psychosocial oncology professional (100%)
        • Proportion of AYA patients screened for distress with standardized AYA-specific tools (CDS-AYA or other) (89%)
      • Research
        • Proportion of funded grants for AYA cancer research (based on both number and value) (89%)
      • Survivorship
        • Mean standardized HRQL score of cancer survivors aged 15-39 years post treatment (78%)
        • Proportion of treatment centers with AYA-specific survivorship programs (44%)
        • Proportion of AYA diagnosed with cancer who are also diagnosed with at least one chronic condition other than original cancer (22%)
      • Active care
        • Time from first healthcare visit with symptom to diagnosis (33%)
        • Proportion of AYA patients referred for fertility treatment (22%)
        • Proportion of AYA patients treated according to a clinical trial protocol (11%)
        • Proportion of AYA diagnosed with cancer who report having knowledge about diagnosis and treatment options (11%)
        • Proportion of AYA diagnosed with cancer reporting positive ratings for satisfaction of active care (0%)
        • Proportion of AYA patients accrued to treatment clinical trials (0%)
      • Palliation
        • Proportion of AYA patients who die in hospital (33%)
        • Proportion of facilities offering palliative homecare programs for AYA patients (0%)
      • Psychosocial care
        • Proportion of AYA patients identified to have distress through screening (0%)
      • Research
        • Proportion of GPs reporting that they feel comfortable recognizing common presenting symptoms of AYA cancers (0%)
      • Prevention
        • HPV vaccination rate (0%)
      • Education
        • Proportion of oncology professionals with AYA certification/expertise (11%)
      AYA indicates adolescent and young adult; GP, general practitioner; HPV, human papilloma virus.

      Ranking

      The ranking survey was completed by 10 of the 11 invited respondents. Final rankings of endorsed indicators, together with potential data sources, are displayed in Table 4, mapped onto both the framework and CSQI quality dimensions. Four of the 5 most important indicators were considered to be moderately to very feasible. Clinical trial enrollment was the only top 5 indicator for importance considered to be the least feasible to obtain.
      Table 4Ranking of final indicators for importance.
      IndicatorFramework themeFramework

      recommendation or priority addressed by indicator*From Fernandez et al
      • Fernandez C.
      • Fraser G.A.
      • Freeman C.
      • et al.
      Principles and recommendations for the provision of healthcare in Canada to adolescent and young adult–aged cancer patients and survivors.
      ImportancePotential data sourceCancer System Quality Index quality dimension
      • Cancer Quality Council of Ontario
      Cancer System Quality Index (CSQI): quality dimensions.
      Survival: a) AYA patient overall survival; b) AYA patient progression or event-free survivalActive care

      and

      survivorship
      1. Services must be provided to address the unique needs of AYAs with cancer and survivors of cancer in childhood, adolescence, and young adulthood to redress inequities in the care provided to this group relative to both younger and older cancer patients.

      4. Implementation of life-long monitoring and follow-up of survivors of cancer in childhood, adolescence, and young adulthood will provide economic and other societal benefits and help mitigate late- or long-term treatment effects.
      1Overall survival can be obtained from provincial cancer registries; however, event-free survival requires chart abstraction and is less feasible.Effective
      Proportion of AYA patients who have access to age-appropriate resources for educational, vocational, and psychosocial supportSurvivorship4.4 Rehabilitation services should be available to provide quality evidence-based services meeting the range of AYA cancer survivors’ rehabilitation needs (including physical, psychosocial, occupational, and educational).2Survey of services offered by institutionsResponsive/integrated
      Cause of death among cancer survivors (short and long): proportion of AYA diagnosed with cancer who die of their disease at 5, 10, and 20 years post diagnosisSurvivorship4. Implementation of life-long monitoring and follow-up of survivors of cancer in childhood, adolescence, and young adulthood will provide economic and other societal benefits and help mitigate late- or long-term treatment effects.3Information is available in administrative databases but there are potential limitations because of a lack of consistency when coding the cause of death
      • Alexander S.
      • Pole J.D.
      • Gibson P.
      • et al.
      Classification of treatment-related mortality in children with cancer: a systematic assessment.
      • Pole J.D.
      • Gibson P.
      • Ethier M.-C.
      • et al.
      Evaluation of treatment-related mortality among paediatric cancer deaths: a population based analysis.
      • Tran T.H.
      • Lee M.
      • Alexander S.
      • et al.
      Lack of treatment-related mortality definitions in clinical trials of children, adolescents and young adults with lymphomas, solid tumors and brain tumors: a systematic review.
      • Schaffar R.
      • Rapiti E.
      • Rachet B.
      • et al.
      Accuracy of cause of death data routinely recorded in a population-based cancer registry: impact on cause-specific survival and validation using the Geneva Cancer Registry.
      Integrated
      Proportion of female cancer survivors who received chest radiation and have been screened for breast cancer according to follow-up guidelinesSurvivorship4.3 Healthcare services and survivorship follow-up guidelines should be available to provide quality, evidence-based, long-term follow-up care for AYA cancer survivors; these healthcare services should be risk-stratified, considerate of patient needs and preferences, and easily integrated into the local healthcare delivery system.4Administrative databases contain this information but difficulty may arise when trying to combine treatment dataEffective/integrated
      Proportion of AYA patients enrolled in clinical trialsActive care and

      research
      1.5 Opportunities for AYAs with cancer to participate in appropriate clinical research trials must be increased and such patients should be offered entry into any appropriate clinical research trial for which they are eligible.

      5.5 Assessment of determinants of AYA patients’ access to clinical trials, the study of tumor biology, and translational research.
      5Chart abstractionEffective/integrated
      Proportion of AYA patients who had fertility preservation discussion before treatmentActive care1.7 Fertility risks and options for considering or not considering fertility preservation must be discussed with each patient.6Chart abstraction. This information was the goal of the Canadian Oncofertility Database.
      • Alexander S.
      • Pole J.D.
      • Gibson P.
      • et al.
      Classification of treatment-related mortality in children with cancer: a systematic assessment.
      Responsive/accessible
      Proportion of patients who are provided with a treatment summary at end of treatment (record of cancer treatment received)Survivorship4.2 Every AYA cancer survivor should be provided with a record to facilitate transition.7Chart abstractionEffective/responsive
      Proportion of AYA diagnosed with cancer who have met with a psychosocial oncology professionalActive care

      and

      psychosocial
      1.2 Age-appropriate care should be delivered and/or supported by interdisciplinary teams populated with age- and disease-specific medical and psychosocial experts able to effectively communicate and provide evidence-based care, including age-appropriate and developmentally appropriate supportive and psychosocial care.

      2.2 The interdisciplinary team should have access to experts in AYA psychosocial care and their services should be offered to all patients and their families.
      8Chart abstractionResponsive/safe
      Proportion of centers offering AYA-specific palliative care servicesPalliation3.4 AYA-specific interdisciplinary palliative care teams should be established throughout Canada; these teams should be flexible and able to work in both pediatric and adult facilities, as well as in a virtual environment to support patients being cared for in smaller communities or at home.9Survey of services offered by institutionsAccessible/responsive
      Proportion of programs that have psychology or psychiatry support available for AYA patientsPsychosocial2.2 The interdisciplinary team should have access to experts in AYA psychosocial care and their services should be offered to all patients and their families.10Survey of services offered by institutionsAccessible/responsive
      Proportion of patients presented at multidisciplinary team meetingActive care1.3 AYA cancer patients must be recognized as a special population at a critical developmental stage between childhood dependency and adult independence. Systems should be in place to ensure interdisciplinary collaboration, coordination, and transition between pediatric and adult healthcare providers, and to develop and promote linkages with relevant community-based resources.11Chart abstractionIntegrated/efficient
      Proportion of AYA with cancer who received reproductive counseling in follow-up careSurvivorship4.1 Discussion and education about the survivorship phase should begin during active treatment to prepare the patient and family for the transition to survivorship; this must include discussion of potential long-term and late effects of treatment.12Chart abstractionResponsive/integrated
      Proportion of AYA patients screened for distress with standardized AYA-specific tools (CDS-AYA or other)Psychosocial2.1 Implementation of routine psychosocial screening of AYAs at diagnosis and intervals throughout the illness trajectory to provide opportunities for early or prophylactic intervention.13Some administrative databases capture distress screening; however, currently no AYA-specific screening tools are available for use.Effective/responsive
      Proportion of funded grants for AYA cancer researchResearchResearch and the establishment of outcome metrics are required to investigate issues critical to AYAs with cancer and survivors of cancer in childhood, adolescence, and young adulthood to target interventions and healthcare policy to improve all phases of the cancer journey.14In Canada, data available from Canadian Cancer Research Alliance database. Nevertheless, extracting AYA research requires a systemic review because coding for age is dichotomous in this database, pediatric or adult.Equitable
      Note. Data source coded by level of feasibility: ∗ indicates very feasible, means moderately feasible, and is least feasible.
      AYA indicates adolescent and young adult

      Discussion

      This modified Delphi process produced 14 indicators, which covered all of the themes identified at the 2010 AYA cancer care workshop
      • Fernandez C.
      • Fraser G.A.
      • Freeman C.
      • et al.
      Principles and recommendations for the provision of healthcare in Canada to adolescent and young adult–aged cancer patients and survivors.
      to monitor, evaluate, and benchmark cancer care and control for AYAs in Canada. No indicators were identified in the 2 additional themes, education and prevention, which were added to the framework after the 2012 international workshop.
      • Rogers P.C.
      • De Pauw S.
      • Schacter B.
      • et al.
      A process for change in the care of adolescents and young adults with cancer in Canada. “Moving to action”: the second Canadian international workshop. International perspectives on AYAO, part 1.
      Many of the endorsed indicators work in concert, such that improvements in one indicator would lead to long-term improvement in indicators in other areas. For example, it has been suggested that multidisciplinary care is necessary to improve fertility preservation uptake in AYA cancer patients.
      • Nagel K.
      • Cassano J.
      • Wizowski L.
      • Neal M.S.
      Collaborative multidisciplinary team approach to fertility issues among adolescent and young adult cancer patients.
      It is likely, therefore, that increases in the number of patients presented at multidisciplinary meetings would lead to improved rates of referrals to psychosocial oncology professionals and to an increase in the number of patients who have referrals for fertility preservation because these items would be part of discussions at the meeting regarding comprehensive patient care.
      A major challenge to metrics for AYA cancer care and control is the collection of reliable age-specific data, in areas important and relevant to AYAs, which are comparable across provinces. AYAs represent a small subpopulation in cancer care and control; without provincial co-operation on data collection initiatives, barriers will remain to reporting metrics at both the provincial and national levels. Enrollment in clinical trials was identified as an important metric in this process. Nevertheless, obtaining information regarding this metric remains challenging. In Canada, there is no comprehensive national data source for clinical trial enrollment that contains age as a variable, with much of the data retained by individual sponsors of the trials.
      • Hay A.
      • Rae C.
      • Fraser G.
      • et al.
      Accrual of adolescents and young adults with cancer to clinical trials.
      Oncofertility is another area in which access to high-quality national data remains a challenge. Ronn et al
      • Ronn R.
      • Holzer H.
      Oncofertility in Canada: an overview of Canadian practice and suggested action plan.
      highlighted the need for a national database to capture important variables for oncofertility in their plan for action on this challenge in Canada. Because many fertility clinics are private facilities in Canada, data on visits to these clinics are not generally available in any administrative databases but are retained within individual clinics. Initiatives such as the Canadian Oncofertility Database
      Cancer Knowledge Network launches the first Canadian Oncofertility Database. The Oncofertility Consortium.
      will greatly aid efforts to monitor and evaluate metrics in oncofertility. Small changes to current data collection could also enhance the feasibility of collecting data for AYA indicators. For example, if the AYA age group was identified and denoted when collecting administrative information on research grants, the AYA cancer research investment indicator identified in this process would be more readily available and likely have greater accuracy. Currently, identifying AYA-related studies is resource intensive and judgment-based, requiring a systematic approach based on abstracts in a national funding database.

      Canadian Partnership Against Cancer. Adolescent & Young Adults with Cancer: Technical Report. Toronto, ON: 2017.

      A focus of discussion at the final meeting was on whether an indicator was important for cancer in general or was specific to AYA. Some indicators, such as “place of death,” are already collected nationally for cancer, and although some represent important metrics to capture for general use and review, these were not considered to be specific to AYA. It was noted that, for national indicators that are considered important to AYAs but not specific to the unique needs of this population, there should be concerted efforts to ensure reporting on the AYA age subgroup. Indicators specified in this article are meant to be a distinct set for the AYA cancer population and not a set to supplement the existing national indicators. We believe that reporting indicators for AYA cancer care and control in a regular report, such as the CPAC AYA cancer system performance report,

      Canadian Partnership Against Cancer. Adolescent & Young Adults with Cancer: Technical Report. Toronto, ON: 2017.

      would be beneficial in monitoring and evaluating care and control in this population and increasing awareness regarding cancer in AYAs.
      The education theme added to the framework after the 2012 workshop discusses the need for the development of AYA cancer care and control expertise in all fields to form multidisciplinary teams through training and education programs.
      • Rogers P.C.
      • De Pauw S.
      • Schacter B.
      • et al.
      A process for change in the care of adolescents and young adults with cancer in Canada. “Moving to action”: the second Canadian international workshop. International perspectives on AYAO, part 1.
      The process reported here did not endorse any indicators for this theme, despite the group’s recognition of its importance to the quality of AYA cancer care. The group agreed that the proposed indicator in its current state would not be useful. Professional programs for developing AYA cancer management expertise in Canada are in their early stages, including a newly developed program from the Royal College of Physicians and Surgeons of Canada.
      • Rogers P.C.
      • De Pauw S.
      • Schacter B.
      • et al.
      A process for change in the care of adolescents and young adults with cancer in Canada. “Moving to action”: the second Canadian international workshop. International perspectives on AYAO, part 1.
      In the future, as programs to develop AYA expertise within healthcare professions are devised and implemented, this education indicator should be revisited. Prevention was another theme for which indicators were not endorsed. Although prevention was acknowledged by the group as important for AYA cancer control, there was agreement that a lack of research specific to AYA cancer prevention makes it difficult to identify any indicators specific to this group.
      To ensure the successful implementation of indicators for AYA cancer care and control, the process should start with a few feasible and impactful indicators. This would help to develop a pattern of practice and build capacity and buy-in for the implementation of additional indicators. The survival indicator was determined to be the most important and feasible to implement. This indicator has been well defined for use in AYA cancer by the April 2017 CPAC report.

      Canadian Partnership Against Cancer. Adolescent & Young Adults with Cancer: Technical Report. Toronto, ON: 2017.

      Other indicators for initial implementation include availability of AYA-specific services and AYA cancer research investment. Other indicators, although easily obtainable from administrative databases, have limitations such as consistency of coding (eg, “cause of death”)
      • Alexander S.
      • Pole J.D.
      • Gibson P.
      • et al.
      Classification of treatment-related mortality in children with cancer: a systematic assessment.
      • Pole J.D.
      • Gibson P.
      • Ethier M.-C.
      • et al.
      Evaluation of treatment-related mortality among paediatric cancer deaths: a population based analysis.
      • Tran T.H.
      • Lee M.
      • Alexander S.
      • et al.
      Lack of treatment-related mortality definitions in clinical trials of children, adolescents and young adults with lymphomas, solid tumors and brain tumors: a systematic review.
      • Schaffar R.
      • Rapiti E.
      • Rachet B.
      • et al.
      Accuracy of cause of death data routinely recorded in a population-based cancer registry: impact on cause-specific survival and validation using the Geneva Cancer Registry.
      or do not include variables that would allow for the identification of AYA age subgroups.

      Canadian Partnership Against Cancer. Adolescent & Young Adults with Cancer: Technical Report. Toronto, ON: 2017.

      • Hay A.
      • Rae C.
      • Fraser G.
      • et al.
      Accrual of adolescents and young adults with cancer to clinical trials.
      Indicators that are ranked highly should be considered for targeted investment to improve feasibility and availability from organizations such as provincial and national cancer agencies. The new Adolescent and Young Adult National Network at CPAC will play an important role in implementation and reporting of indicators in this area because it is one in which CPAC has developed considerable expertise.
      A limitation of this study is a lack of representation from all provinces. Ideally, there would have been representation of all provinces and stakeholders at every stage, but because of the limited availability of members and their changing roles throughout the study, participation varied during the process. Nevertheless, regardless of representation, many, if not all, of the endorsed indicators are likely to be important for all provinces. Overall the process had strong representation from members with substantial experience in the care of AYA patients and from those familiar with AYA cancer data and research, despite the variability in the number of participants during the project. Another weakness in the selection process was the lack of continued engagement of patient representatives throughout all the stages. Nevertheless, there was substantial patient representation during the process to develop the framework that formed the basis of indicator selection. The selected indicators are derived from a framework, which was built on substantial robust representation from provinces and stakeholder groups, with more than 100 participants, including patient and family representatives, providing input into the development of the framework.
      • Fernandez C.
      • Fraser G.A.
      • Freeman C.
      • et al.
      Principles and recommendations for the provision of healthcare in Canada to adolescent and young adult–aged cancer patients and survivors.
      Another limitation of this study is that the indicators selected were not evaluated against quality indicator criteria and benchmarks were not described. Further work needs to be done to validate the chosen indicators and develop appropriate benchmarks.
      Future research and policy development should focus on new indicators in important areas not addressed by the framework of Fernandez et al.
      • Fernandez C.
      • Fraser G.A.
      • Freeman C.
      • et al.
      Principles and recommendations for the provision of healthcare in Canada to adolescent and young adult–aged cancer patients and survivors.
      One of these areas should be health economics. Greenberg et al
      • Greenberg M.
      • Klassen A.
      • Gafni A.
      • et al.
      Outcomes and metrics.
      included health economics as a section in their summary of potential AYA cancer metrics, proposing measures such as quality-adjusted life-years. Health economics was deemed important to help monitor and evaluate the use of resources to ensure efficiency, effectiveness, and sustainability of AYA cancer programs.

      Conclusion

      Improvements in care and outcomes for AYAs with cancer will require co-operation and co-ordination at multiple levels of the Canadian healthcare system. Developing important metrics relevant to the needs of this population, which can be compared provincially, nationally, and internationally, is required. This is necessary to facilitate the expected improvement of cancer care and control and clinical outcomes. Objective measurements are required to evaluate whether the changes to AYA cancer care and control result in the expected improvements. The indicators presented in this article provide a starting point for the development of a standard set of metrics for AYA cancer care and control in Canada and may prove to be of international utility.

      Acknowledgments

      The authors acknowledge valuable contributions from the Pediatric Oncology Group of Ontario (POGO), the Atlantic Provinces Pediatric Hematology Oncology Network, the Canadian Partnership Against Cancer, Dr. Mark Greenberg at the Pediatric Oncology Group of Ontario. Dr. Paul Grundy at the Canadian Partnership Against Cancer (CPAC), Rebecca Block from Critical Mass (US), and Annie Lok. Other members of the System Performance Group include Dr. Kristin Marr, Tim Buckland, and Sonja De Pauw.
      Source of financial support: This study was supported in part by the Canadian Partnership Against Cancer.

      Supplemental Material

      References

        • Barr R.D.
        • Ferrari A.
        • Ries L.
        • Whelan J.
        • Bleyer W.A.
        Cancer in adolescents and young adults. A narrative review of the current status and a view of the future.
        JAMA Pediatr. 2016; 170: 495-501
        • Department of Health and Human Services, National Institutes of Health, National Cancer Institute, LiveStrong Young Adult Alliance
        Closing the gap: research and care imperatives for adolescents and young adults with cancer: a report of the Adolescent and Young Adult Oncology Progress Review Group.
        • Ferrari A.
        • Albritton K.
        • Osborn M.
        • et al.
        Access and models of care.
        in: Bleyer A. Barr R. Ries L. Whelan J. Ferrari A. Cancer in Adolescents and Young Adults. 2nd ed. Springer, Cham, Switzerland2017: 509-547
        • Zebrack B.
        • Mathews-Bradshaw B.
        • Siegel S.
        • Alliance L.Y.A.
        Quality cancer care for adolescents and young adults: a position statement.
        J Clin Oncol. 2010; 28: 4862-4867
        • Sutcliffe S.B.
        A review of Canadian health care and cancer care systems.
        Cancer. 2011; 117: 2241-2244
        • Bleyer A.
        Cancer in adolescents and young adults: latest estimates of U.S. survival and years of life affected.
        J Adolesc Young Adult Oncol. 2011; 1: 37-42
      1. Canadian Partnership Against Cancer. Adolescent & Young Adults with Cancer: Technical Report. Toronto, ON: 2017.

        • Barr R.
        • Rogers P.
        • Schacter B.
        Adolescents and young adults with cancer: towards better outcomes in Canada.
        Preamble. Cancer. 2011; 117: 2239-2240
      2. Barr R. Rogers P. Schacter B. Adolescents and young adults with cancer: towards better outcomes in Canada. Cancer. 2011;117(S10): 2239-2354
        • Rogers P.C.
        • De Pauw S.
        • Schacter B.
        • et al.
        A process for change in the care of adolescents and young adults with cancer in Canada. “Moving to action”: the second Canadian international workshop. International perspectives on AYAO, part 1.
        J Adolesc Young Adult Oncol. 2013; 2: 72-76
        • Fernandez C.
        • Fraser G.A.
        • Freeman C.
        • et al.
        Principles and recommendations for the provision of healthcare in Canada to adolescent and young adult–aged cancer patients and survivors.
        J Adolesc Young Adult Oncol. 2011; 1: 53-59
        • Bradley N.M.E.
        • Robinson P.D.
        • Greenberg M.L.
        • et al.
        Measuring the quality of a childhood cancer care delivery system: assessing stakeholder agreement.
        Value Health. 2013; 16: 639-646
        • Bradley N.M.E.
        • Robinson P.D.
        • Greenberg M.L.
        • et al.
        Measuring the quality of a childhood cancer care delivery system: quality indicator development.
        Value Health. 2013; 16: 647-654
        • Greenberg A.
        • Angus H.
        • Sullivan T.
        • et al.
        Development of a set of strategy-based system-level cancer care performance indicators in Ontario, Canada.
        Int J Qual Health Care. 2005; 17: 107-114
        • Barbera L.
        • Paszat L.
        • Chartier C.
        Indicators of poor quality end-of-life cancer care in Ontario.
        J Palliat Care. 2006; 22: 12-17
        • Gagliardi A.R.
        • Fung M.F.K.
        • Langer B.
        • et al.
        Development of ovarian cancer surgery quality indicators using a modified Delphi approach.
        Gynecol Oncol. 2005; 97: 446-456
        • Gagliardi A.R.
        • Simunovic M.
        • Langer B.
        • et al.
        Development of quality indicators for colorectal cancer surgery, using a 3-step modified Delphi approach.
        Can J Surg. 2005; 48: 441-452
        • Canadian Partnership Against Cancer (CPAC)
        Systems performance: indicator index.
        http://www.systemsperformance.ca
        Date accessed: December 21, 2018
        • Greenberg M.
        • Klassen A.
        • Gafni A.
        • et al.
        Outcomes and metrics.
        Cancer. 2011; 117: 2342-2350
        • Adolescent and Young Adult Working Party of the Statewide Cancer Clinical Network
        South Australian adolescent and young adult cancer care pathway.
        (Accessed May 16, 2019)
        • Boulkedid R.
        • Abdoul H.
        • Loustau M.
        • Sibony O.
        • Alberti C.
        Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review.
        PLoS One. 2011; 6: e20476
        • Cancer Quality Council of Ontario
        Cancer System Quality Index (CSQI): quality dimensions.
        • Nagel K.
        • Cassano J.
        • Wizowski L.
        • Neal M.S.
        Collaborative multidisciplinary team approach to fertility issues among adolescent and young adult cancer patients.
        Int J Nurs Pract. 2009; 15: 311-317
        • Hay A.
        • Rae C.
        • Fraser G.
        • et al.
        Accrual of adolescents and young adults with cancer to clinical trials.
        Curr Oncol. 2016; 23: e81-e85
        • Ronn R.
        • Holzer H.
        Oncofertility in Canada: an overview of Canadian practice and suggested action plan.
        Curr Oncol. 2013; 20: e465-e474
      3. Cancer Knowledge Network launches the first Canadian Oncofertility Database. The Oncofertility Consortium.
        • Alexander S.
        • Pole J.D.
        • Gibson P.
        • et al.
        Classification of treatment-related mortality in children with cancer: a systematic assessment.
        Lancet Oncol. 2015; 16: e604-e610
        • Pole J.D.
        • Gibson P.
        • Ethier M.-C.
        • et al.
        Evaluation of treatment-related mortality among paediatric cancer deaths: a population based analysis.
        Br J Cancer. 2017; 116: 540-545
        • Tran T.H.
        • Lee M.
        • Alexander S.
        • et al.
        Lack of treatment-related mortality definitions in clinical trials of children, adolescents and young adults with lymphomas, solid tumors and brain tumors: a systematic review.
        BMC Cancer. 2014; 14: 612
        • Schaffar R.
        • Rapiti E.
        • Rachet B.
        • et al.
        Accuracy of cause of death data routinely recorded in a population-based cancer registry: impact on cause-specific survival and validation using the Geneva Cancer Registry.
        BMC Cancer. 2013; 13: 609