Use of Structured Template and Reporting Tool for Real-World Evidence Template for Critical Appraisal of the Quality of Reporting of Real-World Evidence Studies: A Systematic Review

  • Author Footnotes
    ∗ Kapil Khambholja and Manish Gehani contributed equally to this work.
    Kapil Khambholja
    Correspondence: Kapil Khambholja, PhD, Department of Medical Writing and Real-world Evidence, Genpro Research Inc, 303 Wyman St, Ste 300, Waltham, MA 02451, USA.
    ∗ Kapil Khambholja and Manish Gehani contributed equally to this work.
    Department of Medical Writing and Real World Evidence, Genpro Research Inc, Waltham, MA, USA
    Search for articles by this author
  • Author Footnotes
    ∗ Kapil Khambholja and Manish Gehani contributed equally to this work.
    Manish Gehani
    ∗ Kapil Khambholja and Manish Gehani contributed equally to this work.
    Department of Medical Writing and Real World Evidence, Genpro Research Pvt Ltd, Thiruvananthapuram, India
    Search for articles by this author
  • Author Footnotes
    ∗ Kapil Khambholja and Manish Gehani contributed equally to this work.
Published:October 06, 2022DOI:


      • Real-world evidence (RWE) studies are increasingly being used to generate supportive evidence regarding safety, effectiveness, and generalizability of medicinal products or interventions for healthcare decision makers, whereas methods and findings of these studies are full of ambiguity, assumptions, or misinterpretations.
      • This study did critical assessment of the quality of reporting of the published RWE studies using the Structured Template and Reporting Tool for RWE template and uncovered many deficiencies, which could have affected the robustness of the evidence.
      • Structured Template and Reporting Tool for RWE template has a potential of improving robustness of the RWE studies and should be used for planning, implementing, and reporting such studies.


      Real-world evidence (RWE) studies are increasingly being used to support healthcare decisions. Various frameworks, tools, and checklists exist for ensuring quality of real-world data, designing robust studies, and assessing potential for bias. In January 2021, Structured Template and Reporting Tool for RWE (STaRT-RWE) was released to further reduce ambiguity, assumptions, and misinterpretation while planning, implementing, and reporting RWE studies of the safety and effectiveness of treatments. The objective of this study was to identify gaps in the reporting quality of published RWE studies by using this template for critical appraisal.


      Two reviewers conducted a keyword search on PubMed for free-full-text research articles using real-world data, RWE design, and safety with or without effectiveness outcomes of a medicinal product or intervention in humans of any age or gender, published in English between January 13, 2021, and January 13, 2022. Assessment of risk of bias was done using Assessment of Real-World Observational Studies critical appraisal tool. Deficiencies in methods and findings as per STaRT-RWE template were reported as frequencies.


      A total of 54 of 2374 retrieved studies were included in the review. Based on the STaRT-RWE template, the studies inadequately reported empirically defined covariates, power and sample size calculation, attrition, sensitivity analyses, index date (day 0) defining criterion, predefined covariates, outcome, metadata about data source and software, objective, inclusion and exclusion criteria, analysis specifications, and follow-up.


      The use of STaRT-RWE template along with its tables, design diagram, and library of published studies has a potential of improving robustness of RWE studies.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Value in Health
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Jaksa A.
        • Mahendraratnam N.
        Learning from the past to advance tomorrow’s real-world evidence: what demonstration projects have to teach us.
        J Comp Eff Res. 2021; 10: 1169-1173
        • Katkade V.B.
        • Sanders K.N.
        • Zou K.H.
        Real world data: an opportunity to supplement existing evidence for the use of long-established medicines in health care decision making.
        J Multidiscip Healthc. 2018; 11: 295-304
        • Wang S.V.
        • Pinheiro S.
        • Hua W.
        • et al.
        STaRT-RWE: structured template for planning and reporting on the implementation of real world evidence studies.
        BMJ. 2021; 372: m4856
        • Langan S.M.
        • Schmidt S.A.
        • Wing K.
        • et al.
        The reporting of studies conducted using observational routinely collected health data statement for pharmacoepidemiology (RECORD-PE).
        BMJ. 2018; 363: k3532
        • Kurz X.
        • Perez-Gutthann S.
        • ENCePP Steering Group
        Strengthening standards, transparency, and collaboration to support medicine evaluation: ten years of the European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP).
        Pharmacoepidemiol Drug Saf. 2018; 27: 245-252
        • Coles B.
        • Tyrer F.
        • Hussein H.
        • Dhalwani N.
        • Khunti K.
        Development, content validation, and reliability of the Assessment of Real-World Observational Studies (ArRoWS) critical appraisal tool.
        Ann Epidemiol. 2021; 55 (e15): 57-63
      1. REQueST tool and its vision paper. European Network for Health Technology Assessment.
        • Gatto N.M.
        • Campbell U.B.
        • Rubinstein E.
        • et al.
        The structured process to identify fit-for-purpose data: a data feasibility assessment framework.
        Clin Pharmacol Ther. 2022; 111: 122-134
        • Gatto N.M.
        • Reynolds R.F.
        • Campbell U.B.
        A structured preapproval and postapproval comparative study design framework to generate valid and transparent real-world evidence for regulatory decisions.
        Clin Pharmacol Ther. 2019; 106: 103-115
      2. A road map for developing study endpoints in real-world settings. Duke-Margolis Center for Health Policy.
        • Digital medicine society
        The Playbook. Digital Clinical Measures.
        Date accessed: July 11, 2022
        • Berger M.L.
        • Sox H.
        • Willke R.J.
        • et al.
        Good Practices for real-world data studies of treatment and/or comparative effectiveness: recommendations from the Joint ISPOR-ISPE Special Task Force on real-world evidence in health care decision making.
        Value Health. 2017; 20: 1003-1008
        • Schneeweiss S.
        • Rassen J.A.
        • Brown J.S.
        • et al.
        Graphical depiction of longitudinal study designs in health care databases.
        Ann Intern Med. 2019; 170: 398-406
        • Chaparro M.
        • Garre A.
        • Iborra M.
        • et al.
        Effectiveness and safety of ustekinumab in ulcerative colitis: real-world evidence from the ENEIDA registry.
        J Crohns Colitis. 2021; 15: 1846-1851
        • Thaler F.S.
        • Zimmermann L.
        • Kammermeier S.
        • et al.
        Rituximab treatment and long-term outcome of patients with autoimmune encephalitis: real-world evidence from the GENERATE registry.
        Neurol Neuroimmunol Neuroinflamm. 2021; 8e1088
        • Rigamonti A.E.
        • Caroli D.
        • Grugni G.
        • Cella S.G.
        • Sartorio A.
        Frequent medical supervision increases the effectiveness of a longitudinal multidisciplinary body weight reduction program: a real-world experience in a population of children and adolescents with obesity.
        Nutrients. 2021; 13: 3362
        • Schechter M.
        • Melzer-Cohen C.
        • Rozenberg A.
        • et al.
        Cardiorenal outcomes with sodium/glucose cotransporter-2 inhibitors in patients with type 2 diabetes and low kidney risk: real world evidence.
        Cardiovasc Diabetol. 2021; 20: 169
        • Khunti K.
        • Knighton P.
        • Zaccardi F.
        • et al.
        Prescription of glucose-lowering therapies and risk of COVID-19 mortality in people with type 2 diabetes: a nationwide observational study in England.
        Lancet Diabetes Endocrinol. 2021; 9: 293-303
        • Kanamori R.
        • Yamane S.
        • Seto T.
        Long-term safety and effectiveness of thyrotropin Alfa in Japanese patients: a post-marketing surveillance study.
        Adv Ther. 2021; 38: 4949-4960
        • Singal A.G.
        • Nagar S.P.
        • Hitchens A.
        • Davis K.L.
        • Iyer S.
        Real-world effectiveness of lenvatinib monotherapy among unresectable hepatocellular carcinoma patients in the USA.
        Future Oncol. 2021; 17: 2759-2768
        • Hsu C.W.
        • Tseng W.T.
        • Wang L.J.
        • Yang Y.H.
        • Kao H.Y.
        • Lin P.Y.
        Comparative effectiveness of antidepressants on geriatric depression: real-world evidence from a population-based study.
        J Affect Disord. 2022; 296: 609-615
        • Kirchgesner J.
        • Desai R.J.
        • Beaugerie L.
        • Kim S.C.
        • Schneeweiss S.
        Calibrating real-world evidence studies against randomized trials: treatment effectiveness of infliximab in Crohn’s disease.
        Clin Pharmacol Ther. 2022; 111: 179-186
        • Bird P.
        • Littlejohn G.
        • Butcher B.
        • et al.
        Real-world evaluation of effectiveness, persistence, and usage patterns of monotherapy and combination therapy tofacitinib in treatment of rheumatoid arthritis in Australia.
        Clin Rheumatol. 2022; 41: 53-62
        • de Vlam K.
        • Toukap A.N.
        • Kaiser M.J.
        • et al.
        Real-world efficacy and safety of apremilast in Belgian patients with psoriatic arthritis: results from the prospective observational APOLO study.
        Adv Ther. 2022; 39: 1055-1067
        • Sun C.
        • Li Z.
        • Shi Z.
        • Li G.
        Current diagnosis and treatment of acute pancreatitis in China: a real-world, multicenter study.
        BMC Gastroenterol. 2021; 21: 210
        • Pawlowski C.
        • Lenehan P.
        • Puranik A.
        • et al.
        FDA-authorized mRNA COVID-19 vaccines are effective per real-world evidence synthesized across a multi-state health system.
        Med (N Y). 2021; 2 (e8): 979-992
        • Lattanzi S.
        • Canafoglia L.
        • Canevini M.P.
        • et al.
        Adjunctive brivaracetam in focal epilepsy: real-world evidence from the BRIVAracetam add-on First Italian netwoRk STudy (BRIVAFIRST).
        CNS Drugs. 2021; 35 ([published correction appears in CNS Drugs. 2021;35(12):1329-1331]): 1289-1301
        • Snell Taylor S.J.
        • Nielson C.M.
        • Breskin A.
        • et al.
        Effectiveness and safety of romiplostim among patients with newly diagnosed, persistent and chronic immune thrombocytopenia in European clinical practice.
        Adv Ther. 2021; 38: 2673-2688
        • Agrawal C.
        • Goyal P.
        • Agarwal A.
        • et al.
        Multicentric real world evidence with palbociclib in hormone positive HER2 negative metastatic breast cancer in Indian population.
        Sci Rep. 2021; 1116236
        • Nunes A.P.
        • Seeger J.D.
        • Stewart A.
        • Gupta A.
        • McGraw T.
        Retrospective observational real-world outcome study to evaluate safety among patients with erectile dysfunction (ED) with co-possession of tadalafil and anti-hypertensive medications (anti-HTN).
        J Sex Med. 2022; 19: 74-82
        • Muro S.
        • Suzuki M.
        • Nakamura S.
        • et al.
        Real-world effectiveness of early intervention with fixed-dose tiotropium/olodaterol vs tiotropium in Japanese patients with COPD: a high-dimensional propensity score-matched cohort analysis.
        Respir Res. 2021; 22: 180
        • Shu W.
        • Du F.
        • Bai J.S.
        • Yin L.Y.
        • Duan K.W.
        • Li C.W.
        A real-world evidence-based management of HIV by differential duration HAART treatment and its association with incidence of oral lesions.
        Curr HIV Res. 2022; 20: 91-99
        • Shore N.D.
        • Ionescu-Ittu R.
        • Laliberté F.
        • et al.
        Beyond frontline therapy with abiraterone and enzalutamide in metastatic castration-resistant prostate cancer: a real-world US study.
        Clin Genitourin Cancer. 2021; 19: 480-490
        • McCullough P.A.
        • Mehta H.S.
        • Barker C.M.
        • et al.
        Mortality and guideline-directed medical therapy in real-world heart failure patients with reduced ejection fraction.
        Clin Cardiol. 2021; 44: 1192-1198
        • Hsieh S.T.
        • Ho H.F.
        • Tai H.Y.
        • et al.
        Real-world results of immune checkpoint inhibitors from the Taiwan National Health Insurance registration system.
        Eur Rev Med Pharmacol Sci. 2021; 25: 6548-6556
        • Cahill S.P.
        • Lunn S.E.
        • Diaz P.
        • Page J.E.
        Evaluation of patient reported safety and efficacy of cannabis from a survey of medical cannabis patients in Canada.
        Front Public Health. 2021; 9626853
        • Jones N.K.
        • Rivett L.
        • Seaman S.
        • et al.
        Single-dose BNT162b2 vaccine protects against asymptomatic SARS-CoV-2 infection.
        Elife. 2021; 10e68808
        • Tai Y.H.
        • Chang C.C.
        • Yeh C.C.
        • Cherng Y.G.
        • Chen T.L.
        • Liao C.C.
        Adverse outcomes after noncardiac surgery in patients with aortic stenosis.
        Sci Rep. 2021; 1119517
        • Aggarwal V.
        • Sabry W.
        • Elemary M.
        • et al.
        Real world clinical experience of biosimilar G-CSF (Grastofil) for autologous peripheral blood stem cell mobilization: Single Center experience in Canada following early adoption.
        Curr Oncol. 2021; 28: 1571-1580
        • Ahmat A.N.M.F.
        • Wan Puteh S.E.
        • Yusak S.
        Efficacy of rivaroxaban use in solid tumour malignancy: experience from a tertiary care Cancer Centre.
        Asian Pac J Cancer Prev. 2021; 22: 3601-3606
        • Hou J.
        • Kim N.
        • Cai T.
        • et al.
        Comparison of dimethyl fumarate vs fingolimod and rituximab vs natalizumab for treatment of multiple sclerosis.
        JAMA Netw Open. 2021; 4e2134627
        • Rajamand Ekberg N.
        • Bodholdt U.
        • Catarig A.M.
        • et al.
        Real-world use of once-weekly semaglutide in patients with type 2 diabetes: results from the SURE Denmark/Sweden multicentre, prospective, observational study.
        Prim Care Diabetes. 2021; 15: 871-878
        • Rudofsky G.
        • Catarig A.M.
        • Favre L.
        • et al.
        Real-world use of once-weekly semaglutide in patients with type 2 diabetes: results from the SURE Switzerland multicentre, prospective, observational study.
        Diabetes Res Clin Pract. 2021; 178108931
        • Efthimios D.
        • Georgios K.
        • Antonia A.
        • Rania G.
        • Maria-Eleutheria E.
        • TOPICS Study Group
        Long-term effectiveness of natalizumab in patients with relapsing-remitting multiple sclerosis treated in the routine care in Greece: results from the multicenter, observational 5-year prospective study 'TOPICS Greece'.
        Clin Drug Investig. 2021; 41: 865-874
        • Agra-Bermejo R.
        • Cordero A.
        • Veloso P.R.
        • et al.
        Long term prognostic benefit of complete revascularization in elderly presenting with NSTEMI: real world evidence.
        Rev Cardiovasc Med. 2021; 22: 475-482
        • Nanchen D.
        • Carballo D.
        • Bilz S.
        • et al.
        Effectiveness, adherence, and safety of evolocumab in a Swiss multicenter prospective observational study.
        Adv Ther. 2022; 39: 504-517
        • Erman M.
        • Biswas B.
        • Danchaivijitr P.
        • et al.
        Prospective observational study on pazopanib in patients treated for advanced or metastatic renal cell carcinoma in countries in Asia Pacific, North Africa, and Middle East regions: PARACHUTE study.
        BMC Cancer. 2021; 21 ([published correction appears in BMC Cancer. 2021;21(1):1191]): 1021
        • Visaria J.
        • Uzoigwe C.
        • Swift C.
        • Dang-Tan T.
        • Paprocki Y.
        • Willey V.J.
        Real-world effectiveness of once-weekly semaglutide from a US commercially insured and medicare advantage population.
        Clin Ther. 2021; 43: 808-821
        • Arnold Egloff S.A.
        • Junglen A.
        • Restivo J.S.
        • et al.
        Convalescent plasma associates with reduced mortality and improved clinical trajectory in patients hospitalized with COVID-19.
        J Clin Invest. 2021; 131
        • Blanc-Durand F.
        • Florescu M.
        • Tehfe M.
        • et al.
        Improvement of EGFR testing over the last decade and impact of delaying TKI initiation.
        Curr Oncol. 2021; 28: 1045-1055
        • Cowie M.R.
        • Flett A.
        • Cowburn P.
        • et al.
        Real-world evidence in a national health service: results of the UK CardioMEMS HF System Post-Market Study.
        ESC Heart Fail. 2022; 9: 48-56
        • Trifirò G.
        • Isgrò V.
        • Ingrasciotta Y.
        • et al.
        Large-scale postmarketing surveillance of biological drugs for immune-mediated inflammatory diseases through an Italian distributed multi-database healthcare network: the VALORE project.
        BioDrugs. 2021; 35: 749-764
        • Molinier O.
        • Besse B.
        • Barlesi F.
        • et al.
        IFCT-1502 CLINIVO: real-world evidence of long-term survival with nivolumab in a nationwide cohort of patients with advanced non-small-cell lung cancer.
        ESMO Open. 2022; 7100353
        • Mark T.
        • Falkenstein A.
        • Kish J.
        Real-world outcomes of pomalidomide therapy after lenalidomide induction in relapsed/refractory multiple myeloma.
        Future Oncol. 2022; 18: 553-564
        • Rudrapatna V.A.
        • Glicksberg B.S.
        • Butte A.J.
        Utility of routinely collected electronic health records data to support effectiveness evaluations in inflammatory bowel disease: a pilot study of tofacitinib.
        BMJ Heal Care Inform. 2021; 28e100337
        • Überall M.A.
        • Bösl I.
        • Hollanders E.
        • Sabatschus I.
        • Eerdekens M.
        Localized peripheral neuropathic pain: topical treatment with lidocaine 700 mg medicated plaster in routine clinical practice.
        Pain Manag. 2022; 12: 521-533
        • Überall M.A.
        • Eerdekens M.
        • Hollanders E.
        • Bösl I.
        • Sabatschus I.
        Lidocaine 700 mg medicated plaster for postherpetic neuralgia: real-world data from the German Pain e-Registry.
        Pain Manag. 2022; 12: 195-209
        • Pawlowski C.
        • Rincón-Hekking J.
        • Awasthi S.
        • et al.
        Cerebral venous sinus thrombosis is not significantly linked to COVID-19 vaccines or non-COVID vaccines in a large multi-state health system.
        J Stroke Cerebrovasc Dis. 2021; 30105923
        • Lee J.
        • Choi J.
        • Yum Y.
        • et al.
        Clinical effectiveness and safety of amlodipine/losartan-based single-pill combination therapy in patients with hypertension: findings from real-world, multicenter observational databases.
        J Clin Hypertens (Greenwich). 2021; 23: 1975-1983
        • Rahhal A.
        • Khir F.
        • Orabi B.
        • et al.
        A comparative study of high-intensity rosuvastatin versus atorvastatin therapy post-acute coronary syndrome using real-world data.
        Curr Probl Cardiol. 2022; 47100956
        • Epstein S.
        • Xia Z.
        • Lee A.J.
        • et al.
        Vaccination against SARS-CoV-2 in neuroinflammatory disease: early safety/tolerability data.
        Mult Scler Relat Disord. 2022; 57103433
        • Perez C.
        • Rojo E.
        • Margarit C.
        • et al.
        24-month real-world study of spinal cord stimulation in failed back surgery patients with refractory pain.
        Pain Physician. 2021; 24: 479-488
        • Serio B.
        • Giudice V.
        • Guariglia R.
        • et al.
        Prophylactic letermovir decreases cytomegalovirus reactivation after stem cell transplantation: a single-center real-world evidence study.
        Infenz Med. 2021; 29: 102-113
        • Bi J.
        • Khan A.
        • Tang J.
        • et al.
        Targeting glioblastoma signaling and metabolism with a re-purposed brain-penetrant drug.
        Cell Rep. 2021; 37109957
        • Loucera C.
        • Peña-Chilet M.
        • Esteban-Medina M.
        • et al.
        Real world evidence of calcifediol or vitamin D prescription and mortality rate of COVID-19 in a retrospective cohort of hospitalized Andalusian patients.
        Sci Rep. 2021; 1123380
        • Rentsch C.T.
        • Beckman J.A.
        • Tomlinson L.
        • et al.
        Early initiation of prophylactic anticoagulation for prevention of coronavirus disease 2019 mortality in patients admitted to hospital in the United States: cohort study.
        BMJ. 2021; 372: n311
        • Adir Y.
        • Humbert M.
        • Saliba W.
        COVID-19 risk and outcomes in adult asthmatic patients treated with biologics or systemic corticosteroids: nationwide real-world evidence.
        J Allergy Clin Immunol. 2021; 148 (e13): 361-367
        • Page M.J.
        • McKenzie J.E.
        • Bossuyt P.M.
        • et al.
        The PRISMA 2020 statement: an updated guideline for reporting systematic reviews.
        BMJ. 2021; 372: n71
      3. Gehani M, Khambholja K. Data for: use of STaRT-RWE template for critical appraisal of the quality of reporting of real world evidence studies: a systematic review. 2022. doi:10.7910/DVN/VJHPHY

      4. equator network. Enhancing the QUAlity and Transparency Of health Research. What is a reporting guideline?.
        Date accessed: July 11, 2022
        • Dhammi I.K.
        • Haq R.U.
        Standard format for writing a manuscript: a guide to authors.
        Indian J Orthop. 2018; 52: 341
        • Moher D.
        • Hopewell S.
        • Schulz K.F.
        • et al.
        CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials.
        BMJ. 2010; 340: 869
        • Cohen J.F.
        • Korevaar D.A.
        • Altman D.G.
        • et al.
        STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration.
        BMJ Open. 2016; 6e012799
        • Cuschieri S.
        The STROBE guidelines.
        Saudi J Anaesth. 2019; 13: S31-S34
        • Swart E.
        • Schmitt J.
        [Standardized reporting of secondary data analyses (STROSA)—a recommendation].
        Z Evid Fortbild Qual Gesundhwes. 2014; 108: 511-516