Modeling Early Warning Systems: Construction and Validation of a Discrete Event Simulation Model for Heart Failure

Objectives: Developing and validating a discrete event simulation model that is able to model patients with heart failure managed with usual care or an early warning system (with or without a diagnostic algorithm) and to account for the impact of individual patient characteristics in their health outcomes. Methods: The model was developed using patient-level data from the Trans-European Network – Home-Care Management System study. It was coded using RStudio Version 1.3.1093 (version 3.6.2.) and validated along the lines of the Assessment of the Validation Status of Health-Economic decision models tool. The model includes 20 patient and disease characteristics and generates 8 different outcomes. Model outcomes were generated for the base-case analysis and used in the model validation. Results: Patients managed with the early warning system, compared with usual care, experienced an average increase of 2.99 outpatient visits and a decrease of 0.02 hospitalizations per year, with a gain of 0.81 life years (0.45 quality-adjusted life years) and increased average total costs of V11249. Adding a diagnostic algorithm to the early warning system resulted in a 0.92 life year gain (0.57 quality-adjusted life years) and increased average costs of V9680. These patients experienced a decrease of 0.02 outpatient visits and 0.65 hospitalizations per year, while they avoided being hospitalized 0.93 times. The model showed robustness and validity of generated outcomes when comparing them with other models addressing the same problem and with external data. Conclusions: This study developed and validated a unique patient-level simulation model that can be used for simulating a wide range of outcomes for different patient subgroups and treatment scenarios. It provides useful information for guiding research and for developing new treatment options by showing the hypothetical impact of these interventions on a large number of important heart failure outcomes.


Introduction
Decision-analytical models (henceforth models) are key instruments in the toolbox of health economists. Models are the resource by which researchers represent the complex reality in a more simplistic and comprehensible manner or by which experiments that are infeasible or impracticable are simulated. 1 In the health-economic context, through exploring hypothetical scenarios and alternative treatment strategies to identify the most efficient allocation of healthcare resources, models are used to inform decisions when significant real-world data are not available. 2 Heart failure (HF) is a complex clinical syndrome that results from any structural or functional impairment of ventricular filling or ejection of blood. 3,4 HF is characterized by typical symptoms such as breathlessness, ankle swelling, and fatigue and signs such as elevated jugular venous pressure, pulmonary crackles, and peripheral edema. 5 The main disease severity indicator used to describe HF is based on measurement of the left ventricular ejection fraction, which results in a distinction between HF and preserved, mid-range, and reduced ejection fractioneach with different underlying etiologies, demographics, comorbidities, and response to therapies. 6 The New York Heart Association (NYHA) functional classification is an alternative classification system that is used to describe the severity of symptoms and exercise intolerance, providing useful and complementary information about the presence and severity of the disease and thus guiding patient pathways in HF treatment. 7 HF is a major health concern associated with significant morbidity, mortality, and reduced quality of life for patients. From a medical perspective, the goals of managing patients with HF consist of improving their clinical status, functional capacity, and quality of life; preventing hospital admissions; and reducing mortality. [8][9][10] Early warning systems (EWS) in the context of healthcare are timely surveillance systems that collect clinical information to anticipate health deterioration and trigger prompt intervention, thus improving prognosis and treatment outcomes. 11 Broadly speaking, EWS consist of 3 main elements: (1) monitoring and collection of clinical data (eg, vital signs, biomarkers, self-reported health status); (2) a framework allowing for the identification of patterns and trends in these data, indicating significant changes in the health status of the patients; and (3) the establishment of pre-determined conditionssuch as the existence of statistically uncommon patterns in the data, threshold values or ranges for specific parameters within the collected data, or the presence of a singular combination of signs and symptomsthat trigger an alarm and follow-up actions. 12 Diagnostic algorithms (DAs) are predictive mathematical relationships that use a wide range of data collected by EWS for calculating the likelihood of an event (eg, hospitalization or death). These algorithms are used for assisting medical personnel in their decision-making process [13][14][15][16] by translating their output into clinical decision rules for clinical practice, for instance, by prioritizing patients according to their likelihood of having an event or by raising an action-triggering alarm if the probability of having that event exceeds a pre-defined threshold. 17 A previous systematic literature review of models used in the economic evaluation of EWS for the management of patients with HF found that all published models were either decision trees or Markov models. 12 Nevertheless, owing to the specific features of EWS in the context of HF, the flexibility for modeling complex systems provided by discrete event simulation (DES) models makes them an arguably better option for the assessment of the (cost-)effectiveness of EWS. [18][19][20][21][22] DES or patient-level models (both terms will be used interchangeably henceforth) are a type of model that has been increasingly used in the health economics field, not only because of the advances in computing technology and dedicated software but also because of their flexibility and potential for modeling complex diseases. [18][19][20]23 One of the main advantages of DES modeling is the ability to use individual patient characteristics as explanatory variables for predicting disease pathways of simulated patients. To compare the cost-effectiveness of treatment strategies targeted at changing individual patient characteristics, DES models accounting for those characteristics and outputting a wide variety of (intermediate) outcomes are desirable. Nevertheless, to be useful tools for decision making regarding the problem at hand, DES models must accurately reflect disease pathways and their management. 24 The 2 main objectives of this study were (1) developing a DES modeling framework for patients with HF managed with EWSwith and without a DAthat is able to model patients across the whole treatment pathway until death, taking into account the evolution and impact of individual patient characteristics in the outcomes of each individual patient, and (2) justifying the model structure chosen and validating the model through the use of the Assessment of the Validation Status of Health-Economic (AdViSHE) questionnaire and the model outcomes generated in the base-case analysis.

Starting Population of the Model
The starting population of the model consisted of the patients who participated in the Trans-European Network -Home-Care Management System (TEN-HMS) study. 25 This trial investigated the impact of using home telemonitoring (HTM; n = 168), nurse telephone support (NTS; n = 173), and usual care (UC; n = 85) in hospital admissions, hospital days, and rates of mortality. Patientlevel data from the trial were used in the construction and validation of the model.
The simulated model population consisted of a set of randomly drawn patients (with replacement) from the database containing the patient-level data of the starting population. The baseline characteristics of the starting population and of the simulated model population for 1000 patients are presented in Table 1.
Three interventions were considered in the model: (1) UC, patient management plan implemented by the patient's primary care physician; (2) EWS (EWS without a DA), proxied by HTM (described in detail in the TEN-HMS original publication 25 ); and (3) EWS 1 DA (EWS with a DA), intervention, (2) with the addition of a DA (described in the following section).

Conceptualization of EWS and the DA for the Management of HF
We conceptualized the EWS and the DA for the management of HF in the model from a clinical perspective; that is, we have not simulated their impact in the actual pathogenetic process of the disease but rather how they manifest in clinical practice through their impact on each of the events considered in the model. In the scope of HF, the EWS collects clinical information such as vital signs, biomarkers, and inputs from surveysdaily in our caseand uses it for changing the chance of death and hospitalization. The effect of the EWS is captured by the difference of time-to-hospitalization and time-to-death of HTM (the EWS in the context of our analysis) compared with UC. The additional effect of the DA is captured by the possibility of avoiding hospitalizations as described in the following paragraphs.
In our instance, the DA is a mathematical feature that uses clinical data for calculating the likelihood of hospitalization and raises an action-triggering alarm if the probability of being hospitalized exceeds a pre-defined threshold. It is added to the EWS as a way of automatically analyzing the collected data in the EWS. In this framework, we can interpret the alarm as a diagnostic test: if an alarm is raised, the test is positive; if not, the test is negative. We can then consider the event of interest (hospitalization) as "having disease" and not being hospitalized as "not having disease." The interpretation of the statistical measures of the performance of a binary classification test in the context of the model can be described as follows: (1) when the simulated event is a hospitalization, the sensitivity represents the probability of correctly detecting that hospitalization. The final probability of avoiding a hospitalization can be achieved by multiplying the sensitivity of the test by the probability of avoiding a hospitalization in the case of having correctly predicted it (eg, assuming the sensitivity of the alarm is 0.8 and that 80% of the correctly predicted admissions can be avoided, then 0.8 3 80% = 64% is the overall probability of avoiding a hospitalization). (2) Regardless of the simulated event, there are as many diagnostic tests as there were days elapsed between the previous event and the current one. The model calculates the number of false positives (alarms for which there were no hospitalization) in that period by multiplying the number of elapsed days by the false-positive rate (FPR) of the DA (eg, if there were 45 days between the previous and the current events and the FPR of the DA is 0.40, there were 18 false alarms during the period between both events).

Model Structure
The main elements of the model are entities, attributes, events, procedures, outcomes, and relationships. The entity is the modeling representation of the patient (hereafter treated in the masculine form). Attributes are the characteristics of that patient, which can either be fixed throughout the simulation (eg, history of myocardial infarction) or change over time (eg, age). Events are relevant moments in the simulation that are recorded for reconstructing the clinical history of the entity; the model determines which event will happen next by calculating the lowest time-to-event of competing events. Procedures are the means by which the model processes events, following a decision-analytical logic that simulates the clinical pathway of the entity. During each procedure, attributes of the entity are re-evaluated and updated, and outcomes are generated and recorded. Outcomes are the elements that aggregate the information generated by the model and that allow for drawing conclusions from the performed simulations. Relationships are the model elements that link entities, attributes, events, procedures, and outcomes together through mathematical and logical terms defined in the model's code.
For ease of description of the model flow, elements are enclosed within ,., each with a subscript, depending on the type of element we are referring to (Ent, entity; A, attribute; E, event; Proc, procedure; O, outcome). At the start of simulation, a ,patient. Ent is randomly drawn (with replacement) from the database containing the patient-level data of the starting population (patients participating in the TEN-HMS trial). Attributes are assigned to ,patient. Ent based on the patient characteristics found at baseline in the dataset and calculates the time-to-event for each of the following competing events: ,outpatient.visit. E , ,hospitalisation. E , and ,death. E . Time-to-event depends on the individual attributes of the ,patient. Ent at the time of the simulation. The lowest time-to-event determines which event will be processed next. The event is renamed as a procedure and a decision-analytical logic for each of the different procedures determines the pathway of the patient. In ,outpatient.visit. Proc  if not, the model updates attributes and the ,patient. Ent goes back to ,next.event. Proc . In ,death. Proc , the model follows these sequential steps: (1) recording time, costs, life years, and QALYs; (2) updating attributes; (3) computing total outcomes for the simulation; and (4) removing ,patient. Ent from the simulation (see Fig. 1 for a diagrammatic representation of the model structure).
Each ,patient. Ent created in the model runs through the simulation 3 timesone for each of the interventions under analysis.

Patient Attributes and Regression Equations
A study by Pocock et al 26 identified the following as significant independent predictors of mortality in patients with HF: age, ejection fraction, NYHA class, serum creatinine, diabetes, not prescribed beta-blocker, systolic blood pressure, body mass index, time since diagnosis, smoking status, chronic obstructive pulmonary disease, gender, and not prescribed angiotensin-converting enzyme inhibitor or angiotensin-receptor blockers. These variables were all present in our dataset and were used in the model to predict time-to-death. We also used these variables to predict time-to-hospitalization, because it seems reasonable to assume that the pathophysiological mechanisms leading to death in HF are the same that lead to hospitalizations. The summary and the definitions of the parameters used in the regression equations and in the model are presented in Table 2.

Time-to-Event Calculations
We estimated Kaplan-Meier (KM) curves for death and hospitalization using the patient-level data for the UC and HTM populations of the TEN-HMS trial. We then fitted the most common parametric distributionsexponential, Weibull, log-normal, log-logistic, Gompertz, and generalized gammato the KM curves

Death in the Hospital
When a patient is hospitalized, there is a chance of dying in the hospital. For predicting it, we ran a logistic regression where the probability of dying in the hospital is explained by age, gender, history of myocardial infarction, history of chronic atrial fibrillation, comorbidities (diabetes or chronic obstructive pulmonary disease), and the number of previous hospitalizations (see Appendix 2 in Supplemental Materials found at https://doi.org/1

Resource Use and Costs
The model distinguishes among yearly cost of maintenance treatment for UC and for EWS, costs related to the management of false-positive alarms, and event costs (outpatient visit, hospitalization, and death). Costs of maintenance treatment and alarm management depend on the time elapsed between simulated events and are continuously discounted, whereas event costs are accounted for at time of occurrence and are discretely discounted.

Utilities
Utility is a patient attribute assigned at the start of the simulation according to the NYHA class at baseline. The mean utility values per NYHA class used were reported elsewhere 27 (0.88, 0.71, 0.61, and 0.49 for NYHA classes I, II, III, and IV, respectively). Every time an outpatient visit or a hospitalization is processed, the patient utility is updated via a multiplier. For instance, if the utility at the start of the simulation is 0.80 and the multiplier for hospitalization is 0.85, the updated utility of that patient after being hospitalized is 0.80 3 0.85 = 0.68, which remains the utility for the patient until the next event is processed. The decrease in utility in the simulation is limited to the utility found for NYHA class IV.

Model Outcomes
The following outcomes are calculated from the model: number of events per type (referred to as intermediate outcomes), total costs, total life years, total QALYs, and incremental costeffectiveness ratios.
The costs in the model are calculated by adding the discrete costs for each event (outpatient visit, hospitalization, and death) and the cost of maintenance treatment for the intervention. Life years correspond to the elapsed time between the creation of the patient and his death and consequent removal from the simulation. QALYs are obtained through weighing life years with patient utilities over time. The incremental cost-effectiveness ratios were calculated as the difference in the total average costs per patient divided by the difference in the average number of QALYs per patient (V/QALY) between 2 alternative treatment options.
Because outcomes are recorded for each simulated patient, the model allows for extracting the individual patient history for every simulation. See Table 2 for a summary of the parameters used in the model.

Base-Case Analysis
The base-case number of simulations in the deterministic analysis was set to 1000 patients, because this number gave stable results while keeping the running time reasonable. For the base-case analysis, the Weibull distribution was used for extrapolating time-to-death and the log-normal distribution for extrapolating time-to-hospitalization. Distributions were chosen according to the recommendations issued by the Decision Support Unit commissioned by the National Institute for Health and Care Excellence 28 (details can be found in Appendix 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2 021.04.004). The time-to-outpatient visit was set to 0.234 years (approximately every 2.8 months) for UC and 0.141 years (approximately every 1.7 months) for EWS, following the data reported in the TEN-HMS study. 25 The utility multipliers were set to 1 for an outpatient visit (assuming no utility changes resulting from an outpatient visit) and 0.82 for hospitalization, which corresponds to the decrease in utility resulting from a transition from NYHA class 3 to 4 that was found in a previous study estimating QALY weights based on NYHA functional class in an elderly population with HF. 29 The sensitivity of the DA was set to 0.96 and the FPR to 0.54, representing the Youden point of the receiver operating characteristic curve provided by the manufacturer. The probability of avoiding a hospitalization in the case of having correctly predicted it was set to 0.5, as reported elsewhere. 30 A summary of input costs and respective sources is presented in Table 3. The costs are reported in euros and adjusted to 2020 rates based on the Dutch consumer price index. 31 The costs presuppose a healthcare perspective, because it is likely that in The Netherlands there will be healthcare insurers that will decide upon the availability of EWS to patients. 32 Costs and health outcomes were discounted at 4.0% and 1.5%, respectively, according to Dutch guidelines. 33

Probabilistic Sensitivity Analysis
In addition to the patient heterogeneity stemming from the variation in the patient population at baseline, the model includes 2 other types of uncertainty: (1) stochastic uncertainty, which is the uncertainty owing to the randomness of drawing values from probability distributions during the simulation, and (2) parameter uncertainty, which is the uncertainty associated with the coefficients of the regression equations and with the remaining model input parameters.
Accounting on the above, the probabilistic sensitivity analysis was implemented as a double loop: an inner loop in which a predetermined number of patients are sampled with replacement from the baseline population and an outer loop in which values of the input parameters of the model are randomly drawn. This approach is similar to other published and validated patient-level simulation models. 34

Model Development, Coding, and Validation
The model was developed using the R software 35 and it consists of 4 R files: (1) the survival analyses; (2) the logistic regression model for calculating the probability of a patient dying in the hospital; (3) the model functions, which can be seen as the model engine; and (4) the model script where the user can define the model inputs, run the model, and output results. The full code can be found on GitHub (https://github.com/fernandoalbuquerquealmeida/EWS_ HF_DES_model).
We used the AdViSHE decision models tool for having a structured view on the main topics regarding the validation of the model. 36

Base-Case Analysis
The average model results per patient over lifetime are presented in Table 4

Model Validation
The validation of the model outcomes found a slightly higher mortality for the simulated population than the available data from the TEN-HMS trial 25 : 52.8% and 40.8% in our simulation versus 51.0% for UC and 34.0% for EWS at day 450 in the trial. The percentage of estimated deaths in our simulation was also slightly higher than what would be predicted using the model published by Pocock et al. 26 The percentage of deaths after 1 year in our population estimated by the KM method was 37.8% for UC and   26 should not yield exactly the same results, because the considered populations are not exactly the same, both in terms of the patient characteristics at baseline, which are predictors of their survival, and the sample size generating the results. It is still worthwhile mentioning that the direction of the impact of the predictors for mortality in our model was the same as observed by Pocock et al 26 for all variables except smoking and time of diagnosis. In our model, smoking was associated with a lower probability of dying (although with almost no effect) and the time since the first diagnosis of HF being lower than 18 months (see Appendix 1A in Supplemental Materials found at https://doi.org/10.1016/j.jval. 2021.04.004 for further details). There were 1.69 hospitalizations per life year in the UC population and 1.67 hospitalizations per life year in the EWS population observed in the model. These hospitalization rates were about one-third higher than those observed in the TEN-HMS trial 25 (1.25 and 1.22, respectively, for UC and EWS). The increased hospitalization rates can be partly explained by the additional survival considered in the model compared with the TEN-HMS trial, especially when weighing in the fact that increased age reduces time-to-hospitalization, and by the lower time-to-outpatient visit used in the base-case analysis than the input used for selecting the parametric model (see Appendix 1B in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.04.004 for further details).
When comparing the outcomes of the model with other models addressing similar problems, we found comparable deterministic results with the ones found by Grustam et al. 27 Nevertheless, it should be noted that their study did not estimate the (cost-)effectiveness of EWS 1 DA. The comparisons among total costs, life years, and QALYs for UC and EWS are presented in Table 5.
For a systematic overview on the topics related to the model validation, please consult the filled-in AdViSHE questionnaire in the Supplemental Material II (available online).

Discussion
This study aimed at developing a health-economic patientlevel simulation model for HF that included a wide variety of HF patient characteristics and that simulated changes in these characteristics and their subsequent impact on a broad set of outcomes. The modeling framework should be able to model patients managed with an EWS, with or without the use of a DA.
We had access to a comprehensive patient-level dataset generated in the TEN-HMS study 25 that contained the critical factors for prognosis as identified previously by Pocock et al. 26 The limitations of the database consisted of the relatively small sample size, the inevitable missing data on some of the variables, and referring to 2005, 25 which can overlook the changes in clinical practice that occurred ever since. 37 Nevertheless, it ought to be mentioned that patient-level simulation modeling in R has the clear advantage of allowing the adaptation of the code for using other available databasesas long as they include the patient and disease characteristics used in the modelfor estimating the regression equations and for performing an external validation of the model results without changing the core model structure.
In total, we included 20 patient and disease characteristics and 8 different outcomes in the model, which allowed for an adequate description of patients with HF across their treatment pathway until death. These characteristics make our model unique, because, to the best of our knowledge, there are not any previously published models in HF that are able to take into account individual patient characteristics for generating suitable outcomes for our target population. 12 Disease pathways and health outcomes in HFalike other chronic diseasesare strongly influenced by the individual characteristics of the patients. 5,26,[38][39][40] Therefore, it is crucial that the type of model chosen allows for recording the individual patient experience and the variation of their individual characteristics over time. In this regard, Markov models have 3 critical shortcomings compared with patient-level simulations: (1) the definition of health states may preclude considering inter-patient variability, (2) the fixed cycle length does not allow for exploring the effects of changing the frequency of events that impact individual patient characteristics (eg, outpatient visits), and (3) the "lack of memory" regarding the treatment history of a patient when in fact the treatment options of chronic patients normally depend on the previous treatment sequencing and experiences with those treatments. 41 Conversely, DES models can address a wide range of problems, because health-economic modeling using events is a more flexible approach than using health states. Furthermore, DES models use patient attributes, which can change over time and affect time-toevent calculation, to properly model competing risks. Because the DES models approach patients individually, they are a better alternative for dealing with heterogeneous populations. DES models are perceived as a better option for conveying the message to non-modeling experts, because they consist of a more compact representation of the conceptual model, avoiding, for instance, the problem of overcomplicated Markov chains through state explosion. Furthermore, in the eventuality of limited data, DES models also provide a substantial advantage, because the inadequacy of the data is not built into the structure of the model; the simulation can be designed to properly reflect the problem under analysis and perform exploratory analyses with limited data and bestguess estimates. [42][43][44][45] Therefore, although there is a need of a detailed and comprehensive database for estimating the regression equations governing the time-to-event calculations, after the development and validation of the model, which was the goal of our study, it is possible to test a wide variety of scenarios and perform subgroup analyses by changing the settings of the model and the simulated model population.
Building on the specific features of DES modeling, it is of the utmost importance to stress the ability of our model to estimate health outcomes for the EWS 1 DA intervention, with particular attention to its DA feature. In an EWS setting, clinical information is usually assessed by a clinical team who is prompted to act based on clinical decision rules defined for specific combinations of the monitored parameters and the assessment of the clinical picture at any given time. Nevertheless, evidence shows that data-driven approaches such as DAs looking at trends and patterns of recorded parameters change seem to improve the accuracy of detecting events compared with clinical decision rules. [46][47][48][49] When taking into account the conceptualization of the DA (see Methods section), because the model only needs a figure for sensitivity and specificity for accounting for the DA, it easily allows for analyzing the (cost-) effectiveness of the EWS 1 DA intervention at any given point of the receiver operating characteristic curve of the DA. In other words, the model permits judging on the best operating point for the DA to optimize the cost-effectiveness of the intervention, which is crucial for making informed decisions on the adoption of a particular DA. Additionally, we can think of our model as a bridge between cost-effectiveness and the huge potentialities of artificial intelligence and machine learning for improving the quality of those decisions, not only by reducing uncertainty through the continuous incorporation of big data collected by the EWS and other data sources but also by constantly improving the DA prediction capabilities through machine learning, thereby determining the best follow-up actions from the results of the DA. 50,51 We can further envision a more comprehensive model to which our model is only but a piece that is generating the cost-effectiveness results. Going one step deeper, we can think of the cost-effectiveness results themselves as another piece of information used by the DA for improving its predictions.
Although it reflects the disease pathways in HF and uses HTM as an example of an EWS, the model was developed to be easily adaptable for other type of EWS interventions used in chronic disease management. For instance, the time-to-outpatient visit, which can be easily changed in the model by the user, can be set according to the specific treatment guidelines for any given population suffering from a chronic disease. In our case, the EWS had an effect in both time-to-hospitalization and time-to-death. Nevertheless, other events can be considered when conceptualizing the model for other chronic diseases; the logic used for modeling hospitalization and death in our model can be repeated for as many events as needed. Focusing on the DA, it should be noted that this feature affected the outcomes of the simulated patient by avoiding hospitalizations (having an impact in costs and health outcomes). Avoiding hospitalizations, in turn, affects the disease pathways of the simulated patient and has an impact on recorded outcomes. This logic can be used with other EWS for events a DA is intended to avoid in the management of any other chronic disease.
Concerning the validation of the model, the face validity of the conceptual model was underpinned by the opinions of both experts in the field of health-economic modeling and a multidisciplinary team of experts in the field of clinical technical solutions development for HF. All the performed tests revealed that our model was robust and able to generate health outcomes comparable with those estimated by other models addressing similar problems and those obtained from empirical data. On the comparison with other models, it should be stressed that we found fewer life years and QALYs than Grustam et al. 27 In their study, the authors assumed that the transition probabilities measured in the time frame of 240 to 450 days in the original study continue unaltered for 20 years. Given the average age of the patients included in the model (67 years old) and their very poor health state, it seems unlikely that their transition probabilities would remain the same for the following 20 years. Therefore, the fewer QALYs found in our study are a consequence of the higher mortality that was found using the parametric survival modeling approach that we took and the assumption that there is a utility change similar to the one observed for a change to the next worse NYHA class from a hospitalization, which occurs more frequently than the health state transitions in the study by Grustam et al. 27 The AdViSHE questionnaire proved to be a useful tool in the process of model validation, both for guiding in the model development and for identifying areas for improvement (see Appendix II in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021. 04.004 for further details). On that note, there are a few shortcomings of the model that ought to be discussed.
Although the model allows for updating patient characteristics at the occurrence of each event, we did not have information on the evolution of some patient characteristics and we could not update patient attributes accordingly. Conceptually, it would be ideal to have equations describing the trends of the patient characteristics, eventually with a link to changes in the medication that could be modeled during the outpatient visit procedure.
The model outcomes are representative for the group of patients who participated in the TEN-HMS trial, which are mainly patients with severe HF who have been previously hospitalized. It ought to be said that patients with HF enrolled in clinical trials of EWS usually have similar characteristics to the TEN-HMS patients and, as such, results could be projected for those patients using the model. Nevertheless, because regression equations were estimated using the database obtained from the TEN-HMS trial, extrapolation of the results to the general HF population should be done with care. It would be interesting to re-estimate the model equations using real-world evidence for a more representative HF population to assess whether there are significant differences in estimated outcomes. In doing so, the model would be able to be used for a larger proportion of patients with HFfor example, an HF population with milder symptoms and treated in primary care who could also be candidates for an EWS. Nevertheless, it should be noted that building a DES model is an extensively datademanding exercise that requires a wide range of patient-level data for building and validating the model. Unfortunately, patient-level data are not widely available, particularly in the realworld setting, and they tend to be characterized by a lot of missing data, which leave the developer with a dilemma on how to handle those without biasing the outcomes of the model. [52][53][54][55][56] Further on the issue of data, in our particular case, we did not have information that would allow us to determine the impact of patient characteristics in outpatient visits. If we would have been able to do so, we could have incorporated in the model a relationship between patient characteristics and outpatient visits, which could result, for instance, in a change in medication. The change in medication in turn could impact the disease pathways in the model and, as a consequence, the outcomes of simulated patients. This would arguably be of added value from a conceptual point of view and for the sake of increased face validity of the model in the eyes of the layperson in health economicsas it is often the case of some decision makers.
We also regret not having access to another database with patient-level data, which would have been worthwhile for increasing the sample size of our data inputs (thus reducing uncertainty) and for validating the model through assessing outcomes using alternative input data. Yet again, data availability and the real world hardly go hand in hand.
In conclusion, the developed model is a unique patient-level simulation model that includes many of the patient and disease characteristics that are considered important for prognosis and treatment of patients with HF. The model can be used for simulating a wide range of outcomes for different patient subgroups. More specifically, the model can provide useful information for guiding research and for the development of new treatment options, with a particular focus on EWS and the operationalization of DA, by showing the possible impact of these interventions on a large number of important HF outcomes. doi: https://doi.org/10.1016/j.jval.2021.04.004