A Framework for Using Real-World Data and Health Outcomes Modeling to Evaluate Machine Learning–Based Risk Prediction Models

Published:December 22, 2021DOI:



      We propose a framework of health outcomes modeling with dynamic decision making and real-world data (RWD) to evaluate the potential utility of novel risk prediction models in clinical practice. Lung transplant (LTx) referral decisions in cystic fibrosis offer a complex case study.


      We used longitudinal RWD for a cohort of adults (n = 4247) from the Cystic Fibrosis Foundation Patient Registry to compare outcomes of an LTx referral policy based on machine learning (ML) mortality risk predictions to referral based on (1) forced expiratory volume in 1 second (FEV1) alone and (2) heterogenous usual care (UC). We then developed a patient-level simulation model to project number of patients referred for LTx and 5-year survival, accounting for transplant availability, organ allocation policy, and heterogenous treatment effects.


      Only 12% of patients (95% confidence interval 11%-13%) were referred for LTx over 5 years under UC, compared with 19% (18%-20%) under FEV1 and 20% (19%-22%) under ML. Of 309 patients who died before LTx referral under UC, 31% (27%-36%) would have been referred under FEV1 and 40% (35%-45%) would have been referred under ML. Given a fixed supply of organs, differences in referral time did not lead to significant differences in transplants, pretransplant or post-transplant deaths, or overall survival in 5 years.


      Health outcomes modeling with RWD may help to identify novel ML risk prediction models with high potential real-world clinical utility and rule out further investment in models that are unlikely to offer meaningful real-world benefits.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Value in Health
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Wessler B.S.
        • Paulus J.
        • Lundquist C.M.
        • et al.
        Tufts PACE clinical predictive model registry: update 1990 through 2015.
        Diagn Progn Res. 2017; 1: 20
        • Kelly C.J.
        • Karthikesalingam A.
        • Suleyman M.
        • Corrado G.
        • King D.
        Key challenges for delivering clinical impact with artificial intelligence.
        BMC Med. 2019; 17: 195
        • Ramos K.J.
        • Quon B.S.
        • Psoter K.J.
        • et al.
        Predictors of non-referral of patients with cystic fibrosis for lung transplant evaluation in the United States.
        J Cyst Fibros. 2016; 15: 196-203
        • Ramos K.J.
        • Somayaji R.
        • Lease E.D.
        • Goss C.H.
        • Aitken M.L.
        Cystic fibrosis physicians’ perspectives on the timing of referral for lung transplant evaluation: a survey of physicians in the United States.
        BMC Pulm Med. 2017; 17: 21
        • Yin J.
        • Ngiam K.Y.
        • Teo H.H.
        Role of artificial intelligence applications in real-life clinical practice: systematic review.
        J Med Internet Res. 2021; 23e25759
        • Dekker F.W.
        • Ramspek C.L.
        • van Diepen M.
        Con: most clinical risk scores are useless.
        Nephrol Dial Transplant. 2017; 32: 752-755
        • Vollmer S.
        • Mateen B.A.
        • Bohner G.
        • et al.
        Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness [published correction appears in BMJ. 2020;369:m1312].
        BMJ. 2020; 368: l6927
        • Moons K.G.
        • Altman D.G.
        • Vergouwe Y.
        • Royston P.
        Prognosis and prognostic research: application and impact of prognostic models in clinical practice.
        BMJ. 2009; 338: b606
        • Steyerberg E.W.
        • Moons K.G.
        • van der Windt D.A.
        • et al.
        Prognosis Research Strategy (PROGRESS) 3: prognostic model research.
        PLoS Med. 2013; 10e1001381
        • Reilly B.M.
        • Evans A.T.
        Translating clinical research into clinical practice: impact of using prediction rules to make decisions.
        Ann Intern Med. 2006; 144: 201-209
        • Khalifa M.
        • Magrabi F.
        • Gallego Luxan B.
        Evaluating the impact of the grading and assessment of predictive tools framework on clinicians and health care professionals’ decisions in selecting clinical predictive tools: randomized controlled trial.
        J Med Internet Res. 2020; 22e15770
        • Goto T.
        • Camargo Jr., C.A.
        • Faridi M.K.
        • Freishtat R.J.
        • Hasegawa K.
        Machine learning–based prediction of clinical outcomes for children during emergency department triage.
        JAMA Network Open. 2019; 2e186937
        • Osawa I.
        • Goto T.
        • Yamamoto Y.
        • Tsugawa Y.
        Machine-learning-based prediction models for high-need high-cost patients using nationwide clinical and claims data.
        NPJ Digit Med. 2020; 3: 148
        • Kappen T.H.
        • van Loon K.
        • Kappen M.A.
        • et al.
        Barriers and facilitators perceived by physicians when using prediction models in practice.
        J Clin Epidemiol. 2016; 70: 136-145
        • Bate L.
        • Hutchinson A.
        • Underhill J.
        • Maskrey N.
        How clinical decisions are made.
        Br J Clin Pharmacol. 2012; 74: 614-620
        • van Giessen A.
        • Peters J.
        • Wilcher B.
        • et al.
        Systematic review of health economic impact evaluations of risk prediction models: stop developing, start evaluating.
        Value Health. 2017; 20: 718-726
        • Siontis K.C.
        • Siontis G.C.
        • Contopoulos-Ioannidis D.G.
        • Ioannidis J.P.
        Diagnostic tests often fail to lead to changes in patient outcomes.
        J Clin Epidemiol. 2014; 67: 612-621
        • Vickers A.J.
        • Cronin A.M.
        Traditional statistical methods for evaluating prediction models are uninformative as to clinical value: towards a decision analytic framework.
        Semin Oncol. 2010; 37: 31-38
        • Vickers A.J.
        • Elkin E.B.
        Decision curve analysis: a novel method for evaluating prediction models.
        Med Decis Making. 2006; 26: 565-574
        • Kerem E.
        • Reisman J.
        • Corey M.
        • Canny G.J.
        • Levison H.
        Prediction of mortality in patients with cystic fibrosis.
        N Engl J Med. 1992; 326: 1187-1191
        • Mayer-Hamblett N.
        • Rosenfeld M.
        • Emerson J.
        • Goss C.H.
        • Aitken M.L.
        Developing cystic fibrosis lung transplant referral criteria using predictors of 2-year mortality.
        Am J Respir Crit Care Med. 2002; 166: 1550-1555
        • Aaron S.D.
        • Chaparro C.
        Referral to lung transplantation--too little, too late.
        J Cyst Fibros. 2016; 15: 143-144
        • Buzzetti R.
        • Alicandro G.
        • Minicucci L.
        • et al.
        Validation of a predictive survival model in Italian patients with cystic fibrosis.
        J Cyst Fibros. 2012; 11: 24-29
        • Liou T.G.
        • Adler F.R.
        • Fitzsimmons S.C.
        • Cahill B.C.
        • Hibbs J.R.
        • Marshall B.C.
        Predictive 5-year survivorship model of cystic fibrosis.
        Am J Epidemiol. 2001; 153: 345-352
        • Nkam L.
        • Lambert J.
        • Latouche A.
        • Bellis G.
        • Burgel P.R.
        • Hocine M.N.
        A 3-year prognostic score for adults with cystic fibrosis.
        J Cyst Fibros. 2017; 16: 702-708
        • Liu Y.
        • Vela M.
        • Rudakevych T.
        • Wigfield C.
        • Garrity E.
        • Saunders M.R.
        Patient factors associated with lung transplant referral and waitlist for patients with cystic fibrosis and pulmonary fibrosis.
        J Heart Lung Transplant. 2017; 36: 264-271
        • Mitchell A.B.
        • Glanville A.R.
        Lung transplantation: a review of the optimal strategies for referral and patient selection.
        Ther Adv Respir Dis. 2019; 131753466619880078
        • Thabut G.
        • Christie J.D.
        • Mal H.
        • et al.
        Survival benefit of lung transplant for cystic fibrosis since lung allocation score implementation.
        Am J Respir Crit Care Med. 2013; 187: 1335-1340
        • Vock D.M.
        • Tsiatis A.A.
        • Davidian M.
        • et al.
        Assessing the causal effect of organ transplantation on the distribution of residual lifetime.
        Biometrics. 2013; 69: 820-829
        • Vock D.M.
        • Durheim M.T.
        • Tsuang W.M.
        • et al.
        Survival benefit of lung transplantation in the modern era of lung allocation.
        Ann Am Thorac Soc. 2017; 14: 172-181
        • Knapp E.A.
        • Fink A.K.
        • Goss C.H.
        • et al.
        The Cystic Fibrosis Foundation Patient Registry. Design and methods of a national observational disease registry.
        Ann Am Thorac Soc. 2016; 13: 1173-1179
        • Ramos K.J.
        • Sykes J.
        • Stanojevic S.
        • et al.
        Survival and lung transplant outcomes for individuals with advanced cystic fibrosis lung disease living in the United States and Canada: an analysis of national registries.
        Chest. 2021; 160: 843-853
        • Stephenson A.L.
        • Ramos K.J.
        • Sykes J.
        • et al.
        Bridging the survival gap in cystic fibrosis: an investigation of lung transplant outcomes in Canada and the United States.
        J Heart Lung Transplant. 2021; 40: 201-209
        • R Core Team
        R: A language and environment for statistical computing.
        R Foundation for Statistical Computing, Vienna, Austria2021
        • Polley E.
        • LeDell E.
        • Kennedy C.
        • van der Laan M.
        SuperLearner: Super Learner Prediction.
        • van der Laan M.J.
        • Polley E.C.
        • Hubbard A.E.
        Super learner.
        Stat Appl Genet Mol Biol. 2007; 6 (article25)
        • Quanjer P.H.
        • Stanojevic S.
        • Cole T.J.
        • et al.
        Multi-ethnic reference values for spirometry for the 3-95-yr age range: the global lung function 2012 equations.
        Eur Respir J. 2012; 40: 1324-1343
        • Franklin J.M.
        • Schneeweiss S.
        • Polinski J.M.
        • Rassen J.A.
        Plasmode simulation for the evaluation of pharmacoepidemiologic methods in complex healthcare databases.
        Comput Stat Data Anal. 2014; 72: 219-226
        • Vaughan L.K.
        • Divers J.
        • Padilla M.
        • et al.
        The use of plasmodes as a supplement to simulations: a simple example evaluating individual admixture estimation methodologies.
        Comput Stat Data Anal. 2009; 53: 1755-1766
        • Thompson D.
        • Waisanen L.
        • Wolfe R.
        • Merion R.M.
        • McCullough K.
        • Rodgers A.
        Simulating the allocation of organs for transplantation.
        Health Care Manag Sci. 2004; 7: 331-338
        • Bansal A.
        • Heagerty P.J.
        A tutorial on evaluating the time-varying discrimination accuracy of survival models used in dynamic decision making.
        Med Decis Making. 2018; 38: 904-916
        • Alkhateeb A.A.
        • Lease E.D.
        • Mancl L.A.
        • Chi D.L.
        Untreated dental disease and lung transplant waitlist evaluation time for individuals with cystic fibrosis.
        Spec Care Dentist. 2021; 41: 489-497
        • Egan T.M.
        • Murray S.
        • Bustami R.
        • et al.
        Development of the new lung allocation system in the United States.
        Am J Transplant. 2006; 6: 1212-1227
        • Chambers D.C.
        • Yusen R.D.
        • Cherikh W.S.
        • et al.
        The registry of the International Society for Heart and Lung Transplantation: thirty-fourth adult lung and heart-lung transplantation report—2017; focus theme: allograft ischemic time.
        J Heart Lung Transplant. 2017; 36: 1047-1059
        • Moons K.G.
        • Kengne A.P.
        • Grobbee D.E.
        • et al.
        Risk prediction models: II. External validation, model updating, and impact assessment.
        Heart. 2012; 98: 691-698
        • Wallace E.
        • Smith S.M.
        • Perera-Salazar R.
        • et al.
        Framework for the impact analysis and implementation of Clinical Prediction Rules (CPRs).
        BMC Med Inform Decis Mak. 2011; 11: 62
        • Schaafsma J.D.
        • van der Graaf Y.
        • Rinkel G.J.
        • Buskens E.
        Decision analysis to complete diagnostic research by closing the gap between test characteristics and cost-effectiveness.
        J Clin Epidemiol. 2009; 62: 1248-1252
        • Steyerberg E.W.
        • Vickers A.J.
        • Cook N.R.
        • et al.
        Assessing the performance of prediction models: a framework for traditional and novel measures.
        Epidemiology. 2010; 21: 128-138
        • Weill D.
        Lung transplantation: indications and contraindications.
        J Thorac Dis. 2018; 10: 4574-4587
        • Lynch 3rd, J.P.
        • Sayah D.M.
        • Belperio J.A.
        • Weigt S.S.
        Lung transplantation for cystic fibrosis: results, indications, complications, and controversies.
        Semin Respir Crit Care Med. 2015; 36: 299-320