If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Correspondence: Zsombor Zrubka, MD, PhD, Health Economics Research Center; University Research and Innovation Center; Óbuda University, Bécsi út 96/b, Budapest 1034, Hungary.
Health Economics Research Center, University Research and Innovation Center, Óbuda University, Budapest, HungaryCorvinus Institue for Advanced Studies, Corvinus University of Budapest, Budapest, Hungary
Institute of Economics, Centre for Economic and Regional Studies, Budapest, HungaryInstitute of Economics, Corvinus University of Budapest, Budapest, Hungary
Health Economics Research Center, University Research and Innovation Center, Óbuda University, Budapest, HungaryCorvinus Institue for Advanced Studies, Corvinus University of Budapest, Budapest, Hungary
Despite the vast amount of real-life data accumulated in healthcare, EQ-5D index scores are frequently lacking for health economic analyses and have to be estimated.
•
We predicted 3 level version of EQ-5D (EQ-5D-3L) index scores in a large heterogenous data set of population surveys and clinical studies using eXtreme Gradient Boosting classification, eXtreme Gradient Boosting regression, and ordinary least squares regression. Regression methods outperformed classification in terms of prediction accuracy and bias. The performance of the 3 methods depended on the applied evaluation criteria, the target population, the included predictors, and the EQ-5D-3L index score range.
•
The prediction accuracy of individual EQ-5D-3L index scores was inadequate for the majority of respondents. For the evaluation of personalized health interventions, we encourage the systematic collection of patient-reported outcomes such as EQ-5D with the involvement of artificial intelligence experts and outcomes researchers to enhance the value of accumulating data in health systems.
Abstract
Objectives
This study aimed to evaluate the performance of machine learning and regression methods in the prediction of 3-level version of EQ-5D (EQ-5D-3L) index scores from a large diverse data set.
Methods
A total of 30 studies from 3 countries were combined. Predictions were performed via eXtreme Gradient Boosting classification (XGBC), eXtreme Gradient Boosting regression (XGBR) and ordinary least squares (OLS) regression using 10-fold cross-validation and 80%/20% partition for training and testing. We evaluated 6 prediction scenarios using 3 samples (general population, patients, total) and 2 predictor sets: demographic and disease-related variables with/without patient-reported outcomes. Model performance was evaluated by mean absolute error and percent of predictions within clinically irrelevant error range and within correct health severity group (EQ-5D-3L index <0.45, 0.45-0.926, >0.926).
Results
The data set involved 26 318 individuals (clinical settings n = 6214, general population n = 20 104) and 26 predictor variables plus diagnoses. Using all predictors and the total sample, mean absolute error values were 0.153, 0.126, and 0.131, percent of predictions within clinically irrelevant error range were 47.6%, 39.5%, and 37.4%, and within the correct health severity group were 56.3%, 64.9%, and 63.3% by XGBC, XGBR, and OLS, respectively. The performance of models depended on the applied evaluation criteria, the target population, the included predictors, and the EQ-5D-3L index score range.
Conclusions
Regression models (XGBR and OLS) outperformed XGBC, yet prediction errors were outside the clinically irrelevant error range for most respondents. Our results highlight the importance of systematic patient-reported outcome (EQ-5D) data collection. Dialogs between artificial intelligence and outcomes research experts are encouraged to enhance the value of accumulating data in health systems.
Patient-reported outcomes (PROs) reflect disease burden or treatment effectiveness from the patients’ perspective. The value of PROs in improving health system performance and individual health outcomes has been demonstrated in multiple settings.
Recommendations to OECD Ministers of Health from the high level reflection group on the future of health statistics: strengthening the international comparison of health system performance through patient-reported indicators. OECD.
Which multi-attribute utility instruments are recommended for use in cost-utility analysis? A review of national health technology assessment (HTA) guidelines.
Recommendations to OECD Ministers of Health from the high level reflection group on the future of health statistics: strengthening the international comparison of health system performance through patient-reported indicators. OECD.
With the gradual implementation of electronic health records and harmonized statistical data collections (eg, European Health Survey), a large amount of administrative health data is being collected.
Nevertheless, because of varying rules of data sharing, standards of interoperability, available infrastructure or level of stakeholder collaboration, and data sets, which are usually collected at different time points for different purposes with different methods in the health data ecosystem, are difficult to connect.
For example, the Minimum European Health Module is a PRO measure collected regularly in Eurostat population surveys while hardly used in clinical trials. The EQ-5D questionnaire has been increasingly used in clinical trials, health surveys, and registries,
Collection and use of EQ-5D for follow-up, decision-making, and quality improvement in health care — the case of the Swedish National Quality Registries.
The accumulating big data are typically unstructured, heterogenous, and incomplete, which may hamper the analysis using standard regression methods, whereas novel machine learning (ML) approaches may offer advantages in such data sets.
For calculating quality-adjusted life-years in health economic analyses, EQ-5D values are often missing and have to be estimated from other health measures.
Therefore, the question arises whether EQ-5D index scores can be predicted from a large diverse data set combined from multiple sources and whether novel analytical methods offer advantages over conventional regression techniques.
Over the past 15 years, we collected EQ-5D-3L data in 30 studies from 26 318 individuals in a variety of settings and designs.
Health-related costs of diabetes mellitus in adults treated with insulin. Cross-sectional survey of 480 patients in general practice and outpatient settings.
Health related quality of life in patients with bladder cancer: a cross-sectional survey and validation study of the Hungarian version of the Bladder Cancer Index.
As a model of the heterogenous sociodemographic and disease-related variables that can be yielded from real-world electronic health records, a combined anonymous data set was created by applying uniform data-management rules for standard sociodemographic and healthcare-related variables.
This study aimed to evaluate the performance of ML and ordinary least squares (OLS) regression in the prediction of individual-level EQ-5D-3L index scores from variables routinely collected in observational studies in various patient populations and the general population.
Methods
EQ-5D-3L
The EQ-5D-3L questionnaire consists of 2 parts. The descriptive system assesses self-reported health in 5 dimensions: mobility, self-care, usual activities, pain/discomfort, and anxiety/depression. In each dimension, respondents can describe their current health with one of the following 3 categories: no problems, some problems, and severe problems. The descriptive system defines 243 (35) distinct health states.
The EQ-5D-3L index scores (utilities) attached to each health state are measured in valuation studies and reflect societal preferences. In this study, we applied the UK EQ-5D-3L index value set (range −0.594 to 1.000).
The EQ-5D-3L index score of 1 represents perfect health, 0 represents death, and negative values represent “worse than death” health states. The second part of the instrument is a 20-cm vertical EuroQol visual analog scale (EQ-VAS) for the measurement of current health ranging from 0 (worst imaginable health) to 100 (best imaginable health).
Study Population
Data were collected in Hungary, Poland, and Slovenia. These countries have EQ-5D-3L value sets
Health-related costs of diabetes mellitus in adults treated with insulin. Cross-sectional survey of 480 patients in general practice and outpatient settings.
Health related quality of life in patients with bladder cancer: a cross-sectional survey and validation study of the Hungarian version of the Bladder Cancer Index.
From Poland, we included 504 patients from cohort studies involving measurements before, during, or after hospitalization because of stroke (cerebrovascular accident, n = 397)
From Slovenia, 3290 patients were included from a cohort study comparing the outcomes of various health programs across hospitals including conditions such as gonarthrosis, coxarthrosis, intervertebral disc disease, urinary incontinence, carpal tunnel syndrome, inguinal hernia, varicose veins, osteosynthesis removal, and shoulder lesions (PSI, n = 3290),
The detailed description of the involved study populations is provided in the reference publications. We involved patients from the original databases without any other restrictions if EQ-5D-3L values were available; therefore, the number of eligible patients for our study may have differed slightly from the reference publications.
Database and Variables
We partitioned the database into a general population
Health-related costs of diabetes mellitus in adults treated with insulin. Cross-sectional survey of 480 patients in general practice and outpatient settings.
Health related quality of life in patients with bladder cancer: a cross-sectional survey and validation study of the Hungarian version of the Bladder Cancer Index.
sample. The database structure is summarized by listing key study characteristics and nonmissing values for patient-level explanatory variables in Appendix Table 1 of Appendix 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.01.024. The dependent variable was the EQ-5D-3L index score.
Patient-level predictor variables were organized into 2 groups.
Demographic and disease-related variables
Predictors in this group included age, gender, education, place of residence, family status, employment, relative income (net personal income as a percent of the study year’s national average net income), setting (outpatient, hospitalized, and postoperative in the case of documented surgery within 3-6 months), number of general practitioner visits, any general practitioner visit, number of specialist visits, and any specialist visits or hospitalizations in the past year. In the case of missing data, we assumed that specialist visits happened for patients recruited in outpatient specialist centers,
We recorded whether patients were informal care recipients, weight, height, and body mass index. Physician-reported outcomes were transferred to a standard scale, where 0 represents the worst and 1 the best possible health status measurable with the given instrument. The included physician-reported instruments (acronym; score of worst health status—score of best health status) were the clinician-reported VAS (0-100)
Health-related costs of diabetes mellitus in adults treated with insulin. Cross-sectional survey of 480 patients in general practice and outpatient settings.
Health related quality of life in patients with bladder cancer: a cross-sectional survey and validation study of the Hungarian version of the Bladder Cancer Index.
). An overall dummy variable indicated the presence of any disease. Appendix Table 2 of Appendix 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.01.024 summarizes the number of patients in each study with conditions categorized under the International Classification of Diseases, Tenth Revision, codes and the International Classification of Diseases, Tenth Revision, main chapters.
: self-rated health and the limitations because of health problems (Global Activity Limitation Indicator). Scores from PRO instruments were transferred to a standard scale, with 0 representing the worst and 1 the best health status measurable with the given instrument. The applied PRO instruments (acronym; score of worst health status—score of best health status) were the Health Assessment Questionnaire Disability Index (3-0)
Development and validation of the Bladder Cancer Index: a comprehensive, disease specific measure of health related quality of life in patients with localized bladder cancer.
Missing data were handled via the indicator method. We imputed zeros for all missing values and generated a dummy indicator for each predictor denoting missing values. The indicator was set as missing in those general population studies, where self-reported conditions were not inquired. In contrast, the disease dummy was set as absent for those patients who were asked about the presence of a disease and responded negatively. Comorbidities were not recorded in patient populations; therefore, the disease dummy was set as missing except for the index conditions.
Prediction models
EQ-5D-3L index scores were predicted by OLS regression, eXtreme Gradient Boosting (XGBoost) classification (XGBC), and XGBoost regression (XGBR). A regular winner of ML competitions, XGBoost is a highly scalable and computationally efficient implementation of gradient boosted trees. Boosted decision trees are an ensemble of decision trees added sequentially. Each additional tree is trained to correct the errors of the ensemble of previous trees until no further improvements can be made on a validation data set. Gradient boosting grows the best trees by optimizing a loss function that comprises prediction error and a regularization term, which describes the complexity of the trees. Depending on the loss function, XGBoost can run in classification and regression mode, which predict EQ-5D scores in 243 categories or as a continuous value, respectively.
Chen T, Guestrin C. XGBoost: a scalable tree boosting system. Paper presented at: KDD’16: 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; New York; 2016.
Patients with multiple measurements were entered as unique records. No weights were applied. In the OLS model, age was split into 5-year categories, and a piecewise model was fit on EQ-VAS scores with different slopes for the 0 to 34 and 35 to 100 value ranges. We entered predictors without interactions but explored an OLS model with interaction between disease dummies, gender and age. For XGBoost, default settings were retained for most parameters after initial exploration and monitoring of train and test errors. The learning rate parameter was set to 0.1, number of trees were set to 20 to improve speed of classification, and the L1 regularization term of regression was set to 0.9 to avoid overfitting.
We performed predictions in 6 scenarios involving the general population sample (“Pop”), the patient sample (“Pts”), and the entire sample (“Total”), by using only demographic and disease predictors (“Base”) and adding PRO predictors (“PRO”). Model training and evaluation were performed via 10-fold cross-validation, using a randomly selected 80% of the data set for training and 20% for testing. OLS coefficient estimates, XGBoost settings, and feature importance tables for the PRO scenarios are presented in Appendix 2 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.01.024 (Appendix Tables 3-5 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.01.024).
Evaluation of Model Performance
Models were compared via the “mean absolute error” (MAE) of prediction as an intuitive and stabile measure when comparing scenarios with different sample sizes.
yet greater errors in either direction are undesirable, we calculated the percent of predictions within the “clinically irrelevant error range” (eg, predictions within true ± MCD range). Third, we assessed prediction bias via observing mean prediction errors. Finally, we calculated the percent of predictions within the “correct health severity group.” For this, according to the trimodal distribution of EQ-5D-3L, index values > 0.926 (eg, 1-MCD) denoted “full health,” values between 0.926 and 0.45 denoted “medium health,” and values < 0.45 denoted “low health.”
(In “full health,” true EQ-5D-3L index scores are equal to 1, whereas predictions can take values > 0.926.)
To evaluate the reliability of predictions, from the 10 cross-validation sets, we calculated 95% confidence intervals for the evaluation metrics using the following formula:
(1)
where is the mean and SD is the SD of the evaluation metric (eg, MAE) of the k cross-validation sets.
we evaluated whether individuals in different health statuses may be affected adversely because of prediction error. In addition to mean prediction error, we quantified the percent of predictions within clinically irrelevant error range and the percent of predictions within the correct health severity group across the full range of the true EQ-5D index. (Lower values denote greater risk of flawed predictions.) Second, assuming that decisions would be based on predicted and not the unknown true values, we evaluated bias and the probability of accurate predictions across the range of predicted EQ-5D-3L index values (eg, positive predictive value).
The database contained 28 862 records of 26 318 individuals. Cross-sectional studies of the general population provided 20 104 records, whereas 8758 records were from cross-sectional and cohort studies involving 6214 patients (single measurement, n = 3753; 2 measurements, n = 2378; 3 measurements, n = 83). Most data originated from Hungary (16 862 records; 58.4%), followed by Slovenia (6507 records, 22.6%) and Poland (5493 records, 19.0%). Counting the diagnosis related dummies as one variable, the 28 predictor variables contained 64.1% missing values. There were 214 missing data patterns across observations with a range of missing variables from 7 to 23. There were no complete cases in the data set. The predictor variables and missing values are summarized in Table 1.
Table 1Summary of predictor variables and missing values.
Mean (SD) EQ-5D-3L index scores were 0.847 (0.198), 0.665 (0.317), and 0.792 (0.254) in the general population, patients, and the total sample, respectively. In the general population sample, 3.7%, 49.0%, and 47.3%; in the patient sample, 14.2%, 62.3%, and 23.1%; and in the total sample, 6.9%, 53.2%, and 39.9% had EQ-5D-3L index scores in the low, medium, and full health categories, respectively.
The Distribution of True and Predicted EQ-5D-3L Index Scores
The distribution of XGBC predictions resembled the trimodal distribution of true EQ-5D scores, yet predictions of full health were more frequent. The distributions of XGBR and OLS were unimodal with left skew and a peak below full health (Appendix Fig. 1 of Appendix 2 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.01.024). The range of full health across all scenarios was 23.1% to 47.2% for true EQ-5D-3L values, 43.4% to 93.9% for XGBC, 1.0% to 26.2% for XGBR, and 1.9% to 19.2% for OLS. XGBR predictions exceeded the value of 1 less frequently than those of OLS.
Accuracy of Predictions
In all scenarios, MAE was greatest for XGBC and lowest for XGBR followed closely by OLS. In the PRO scenario, MAE was 0.126, 0.113, and 0.118 in the population sample; 0.200, 0.159, and 0.162 in the patient sample; and 0.153, 0.126, and 0.131 in the total sample using XGBC, XGBR, and OLS, respectively. Adding PROs to demographic and disease-related predictors decreased MAE on average by 0.022 (Fig. 1).
On the contrary, the percent of predictions within the clinically irrelevant error range (true ± MCD) was highest for XGBC with 51.9%, 39.1%, and 47.6%, followed by XGBR with 41.7%, 34.2%, and 39.5% and OLS with 38.2%, 33.1%, and 37.4% in the PRO scenario for the general population, patients, and the total sample, respectively. Adding PROs to the base predictors increased the percent of predictions within the clinically irrelevant error range on average by 6.6% (Fig. 2). Although mean prediction error for XGBR and OLS predictions was nearly zero in all scenarios, XGBC showed positive bias with mean error exceeding the MCD all scenarios (range 0.086-0.097) (Appendix Fig. 2 of Appendix 3 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.01.024).
Figure 2Percentage of predictions within clinically irrelevant error range (true ± MCD).
In terms of the percent of predictions within the correct health severity group, XGBR followed closely by OLS outperformed XGBC. In the PRO scenario for the general population, patient, and total samples, 57.2%, 58.1%, and 56.3% of predictions using XGBC; 63.2%, 68.5%, and 64.9% using XGBR; and 60.5%, 68.2%, and 63.1% using OLS fell within the correct health severity group, respectively (Fig. 3). The narrow 95% confidence interval ranges suggested that the predictions of all 3 methods were reliable through the cross-validation rounds. The performance of OLS models with or without interaction terms was rather similar (Appendix Fig. 3 of Appendix 3 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.01.024).
Figure 3Percentage of predictions in the correct health severity group.
XGBC often predicted full across the entire range of true EQ-5D-3L index values. The scatterplots of predicted over true values suggested that adding PROs to base predictors improved prediction accuracy mainly in the low health region and in patients (Appendix Fig. 4 of Appendix 3 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.01.024). Regression to the mean was observed with all methods with positive bias (range 0.59-1.39) in low health and slight negative bias (range −0.01 to −0.23) in full health (Appendix Fig. 5 of Appendix 3 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.01.024).
Accuracy of Predictions by True EQ-5D-3L Index Scores
Figure 4 illustrates the percent of predictions within clinically irrelevant error range by true EQ-5D-3L index scores. XGBC predictions were most accurate in the full health range, whereas XGBR and OLS predictions were most accurate in medium health. All methods were least accurate in the low health range, which improved after adding PROs to the base predictors. The proportions of predictions within the correct health severity category are depicted in Appendix Figure 6 of Appendix 3 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.01.024.
Figure 4Percentage of predictions within clinically irrelevant error range by scenario and true EQ-5D-3L index scores.
Accuracy of Predictions by Predicted EQ-5D-3L Index Scores
If full health was predicted by XGBR or OLS, those values were mostly within, whereas full health predictions by XGBC were mostly outside the clinically irrelevant error range (Fig. 5). If medium health was predicted, the accuracy of the 3 methods was similar, albeit moderate. Low health predictions were the least accurate, which improved when PROs were added to base predictors, especially in patients. The pattern was similar for predictions in the correct health severity group. Although XGBC predictions were most accurate in the medium health range, the accuracy of XGBR and OLS improved from low health toward the full health range (Appendix Fig. 7 of Appendix 3 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.01.024). Although mean prediction error of XGBR and OLS was approximately zero across the entire range, the bias of XGBC depended on the predicted EQ-5D-3L index values (Appendix Fig. 8 of Appendix 3 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.01.024).
Figure 5Percentage of predictions within clinically irrelevant error range by scenario and predicted EQ-5D-3L index scores.
We predicted EQ-5D-3L index scores via XGBC, XGBR, and OLS regression in a large international database combining multiple studies among patients and the general population with diverse predictors and a large amount of missing data. Across scenarios involving patients and the general population with and without PRO predictors, the percent of predictions within the clinically irrelevant error (true ± MCD) range were highest for XGBC and lowest for OLS with XGBR coming close. Nevertheless, MAE of prediction was lowest for XGBR followed by OLS and XGBC. Predictions with XGBC were biased. The performance of the 3 methods depended on the evaluation criteria, the target population, the predictor variables, and the EQ-5D-3L index range. Adding PROs to the demographic and disease-related predictors improved the accuracy of predictions.
Several studies have already used ML to predict EQ-5D values as a binary threshold
Markov blanket-based approach for learning multi-dimensional Bayesian network classifiers: an application to predict the European Quality of Life-5 Dimensions (EQ-5D) from the 39-item Parkinson’s Disease Questionnaire (PDQ-39).
predicted EQ-5D-3L index scores from the 39-item Parkinson’s Disease Questionnaire using multidimensional Bayesian network classifiers (MAE for OLS 0.350; MAE for multidimensional Bayesian network classifier 0.174). Gutacker et al
Mapping MacNew Heart Disease Quality of Life Questionnaire onto country-specific EQ-5D-5L utility scores: a comparison of traditional regression models with a machine learning technique.
mapped heart disease-specific quality of life to EQ-5D-5L index scores using econometric models and deep neural network algorithm (MAE for OLS 0.090-0.129; MAE for deep neural network 0.076-0.105). Recently, Mlynczak et al
applied random forest for assessing the construct validity of EQ-5D-5L (MAE ≤ root mean square error 0.121). In these studies, when compared, ML usually outperformed traditional econometric methods.
Markov blanket-based approach for learning multi-dimensional Bayesian network classifiers: an application to predict the European Quality of Life-5 Dimensions (EQ-5D) from the 39-item Parkinson’s Disease Questionnaire (PDQ-39).
Mapping MacNew Heart Disease Quality of Life Questionnaire onto country-specific EQ-5D-5L utility scores: a comparison of traditional regression models with a machine learning technique.
Advanced econometric models were also used to predict EQ-5D index values accommodating its multimodal distribution and upper limit at full health. In a rheumatoid arthritis data set, MAE was 0.1505 with linear regression, 0.1508 with a random effects Tobit model, 0.1508 with an adjusted limited variable model (treating EQ-5D index score predictions > 0.883 as 1), and 0.1438 with a random effects adjusted limited variable mixture model.
The strength of our study is that the analysis was performed on a large and diverse data set of multiple studies resembling real-world data connected in health data ecosystems.
MAE was comparable with previous studies using ML or regression methods. We evaluated prediction accuracy via the percent of predictions within the “clinically irrelevant error range” by splitting absolute error into “irrelevant” (≤MCD) and “relevant” (>MCD) values. We argue that there are no established criteria for further classifying errors into “large” or “acceptable” ones. Nevertheless, erroneous predictions in any direction of any magnitude may negatively affect decisions. Therefore, by conveying clinically relevant information about the shape of error distribution, this metric has merit in the evaluation of predictive models in healthcare.
Our study has limitations. Despite the full data matrices of individual studies, the joint database had a large proportion of missing data, which was handled via the missing indicator method. Although this method has been criticized for its biasedness,
Song M, Zhou X, Pazaris M, Spiegelman D. The missing covariate indicator method is nearly valid almost always. Preprint. Posted online October 30, 2021. ArXiv:211100138. https://doi.org/10.48550/arXiv.2111.00138.
In our study, mean prediction error using XGBR and OLS was close to zero in all scenarios, whereas XGBC predictions were positively biased. Nevertheless, the performance of XGBoost is usually not affected by the imputation of missing data.
We did not apply multiple imputation techniques to prevent leakage of information and interference with the prediction methods. Therefore, the information contained in the data set probably could not be used to its full capacity. The potential effect of missing data structures on the predictive performance of the models deserves further exploration, along with the use of more advanced data imputation techniques such as multiple imputation
Although more advanced regression models are available to accommodate the multimodal distribution and upper limit of EQ-5D index scores at full health, simple OLS models are commonly applied to predict individual utilities.
Adding further interaction terms did not affect the performance of our OLS model. Nevertheless, variable selection via LASSO regression from many interacted predictors has been shown to improve the performance of OLS in predicting EQ-5D-5L index scores.
In addition, predictions were performed on unweighted data, which, through the nonrepresentative proportions of patients in the sample, may have introduced bias to the prediction results. Therefore, the external validity of our prediction models is probably limited and the accuracy of predictions may be further improved. As a future area of research, alternative prediction techniques, a combination of methods based on their performance in various EQ-5D-3L ranges, imputation of missing data, and weights reflecting the structure of the average population and disease epidemiology could be applied to improve the accuracy of predictions of individual EQ-5D-3L index scores. Furthermore, the external validity of the prediction models should be tested on multiple study populations that were not included in the training phase.
Conclusions
In a large database of EQ-5D-3L studies, prediction errors of EQ-5D-3L index scores using XGBC, XGBR, and OLS were greater than the MCD for most respondents and depended on the applied method, performance evaluation criteria, the target population, applied predictors, and the EQ-5D-3L range. The performance of XGBR slightly exceeded OLS in most evaluation measures. Regression methods outperformed XGBC in terms of prediction accuracy and bias.
Our results warn against overoptimistic expectations and prompt for care when using ML for predicting individual patient-reported health outcomes. We recommend the systematic and widespread collection of real-world PRO data using standardized PRO measures, including EQ-5D. In addition, we encourage dialogs between artificial intelligence and outcomes research experts to enhance the value of accumulating data in health systems.
Acquisition of data: Golicki, Prevolnik-Rupel, Ogorevc, Gulácsi, Péntek
Analysis and interpretation of data: Zrubka, Csabai, Hermann
Drafting of the manuscript: Zrubka, Péntek
Critical revision of the paper for important intellectual content: Zrubka, Csabai, Hermann, Golicki, Prevolnik-Rupel, Ogorevc, Gulácsi, Péntek
Statistical analysis: Zrubka, Csabai, Hermann
Provision of study materials or patients: Golicki, Prevolnik-Rupel, Ogorevc, Gulácsi, Péntek
Obtainingfunding: Zrubka, Gulácsi
Supervision: Gulácsi, Péntek
Conflict of Interest Disclosures: Drs Zrubka, Csabai, Hermann, Golicki, Prevolnik-Rupel, Gulácsi and Péntek reported receiving funding from project number TKP2020-NKA-02 implemented with the support provided from the National Research, Development, and Innovation Fund of Hungary, financed under the Tématerületi Kiválósági Program funding scheme. Dr Zrubka reported receiving grants or contracts from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (grant agreement number 679681); reported receiving consulting fees from Roche Hungary; and reported serving as a member of the ISPOR Digital Health Special Interest Group as co-chair of a key scientific project. Dr Csabai reported receiving funding from the Ministry of Innovation and Technology National Research Development and Innovation Office in the framework of the Artificial Intelligence National Laboratory Program (MILAB). Dr Golicki reported receiving research grants and travel reimbursement from the EuroQol Group. Dr Péntek reported receiving funding from project number 2019-1.3.1-KK-2019-00007 implemented with the support provided from the National Research, Development and Innovation Fund of Hungary, financed under the 2019-1.3.1-KK funding scheme during writing this study. Drs Golicki, Prevolnik-Rupel, and Péntek reported being members of the EuroQol Group. No other disclosures were reported.
Funding/Support: Under project number TKP2020-NKA-02, this research has been implemented with the support provided from the National Research, Development and Innovation Fund of Hungary, financed under the Tématerületi Kiválósági Program funding scheme.
Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Recommendations to OECD Ministers of Health from the high level reflection group on the future of health statistics: strengthening the international comparison of health system performance through patient-reported indicators. OECD.
Which multi-attribute utility instruments are recommended for use in cost-utility analysis? A review of national health technology assessment (HTA) guidelines.
Collection and use of EQ-5D for follow-up, decision-making, and quality improvement in health care — the case of the Swedish National Quality Registries.
Health-related costs of diabetes mellitus in adults treated with insulin. Cross-sectional survey of 480 patients in general practice and outpatient settings.
Health related quality of life in patients with bladder cancer: a cross-sectional survey and validation study of the Hungarian version of the Bladder Cancer Index.
Development and validation of the Bladder Cancer Index: a comprehensive, disease specific measure of health related quality of life in patients with localized bladder cancer.
Chen T, Guestrin C. XGBoost: a scalable tree boosting system. Paper presented at: KDD’16: 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; New York; 2016.
Markov blanket-based approach for learning multi-dimensional Bayesian network classifiers: an application to predict the European Quality of Life-5 Dimensions (EQ-5D) from the 39-item Parkinson’s Disease Questionnaire (PDQ-39).
Mapping MacNew Heart Disease Quality of Life Questionnaire onto country-specific EQ-5D-5L utility scores: a comparison of traditional regression models with a machine learning technique.
Song M, Zhou X, Pazaris M, Spiegelman D. The missing covariate indicator method is nearly valid almost always. Preprint. Posted online October 30, 2021. ArXiv:211100138. https://doi.org/10.48550/arXiv.2111.00138.
00101-2&id=gr1.jpg){kind=link}
00101-2&id=gr2.jpg){kind=link}
00101-2&id=gr3.jpg){kind=link}
00101-2&id=gr4.jpg){kind=link}
00101-2&id=gr5.jpg){kind=link}