INTRODUCTION

Patients readmitted to intensive care unit (ICU) carry a two- to fivefold higher rates of mortality, and two fold increase in hospital length of stay.1,2,3 Previous studies have shown that certain patient and process factors are associated with higher rates of ICU readmissions and increased mortality.4,5,6,7,8,9 Examples of patient factors include male sex, severity of illness scores, history of specific medical conditions such as diabetes or cancer, interventions such as continuous renal replacement therapy, vasopressor use, and mechanical ventilation, as well as clinical parameters such as white blood cell count and heart rate which have all been associated with higher rates of readmissions.4, 5 Process factors such as out-of-hours ICU discharges have also been linked to higher rates of ICU readmissions and increased mortality.5,6,7,8,9

Considering the high mortality and morbidity associated with ICU readmissions, models have previously been developed to predict which patients are at a higher risk of readmissions. Many of these models rely mostly on patient factors, physiological parameters, and/or clinical interventions to predict readmissions.10, 11 In reality, however, both patient and process factors work in tandem to increase the likelihood of a patient being readmitted to the ICU.8 When scoring tools incorporating both patient and process factors have been developed to assist in data analysis, they are either complex or rely heavily on specific computer algorithms with access to patient data in order to calculate risk scores.12,13,14 A simple scoring tool that is based on a predictive model that comprehensively takes into account not only patient factors such as physiological parameters and clinical interventions but also process factors such as weekend and out-of-hours discharges into a single model would address the deficiencies in the previous work in this area and provide a clinically useful tool to enhance patient care. In this study, we rigorously examine the patient and process factors that might affect readmission of patients to the medical intensive care unit (MICU) and then attempt to develop and validate a simple, usable prediction model that takes into consideration factors found to significantly affect MICU readmissions based on this analysis.

METHODS

We conducted a retrospective chart review of patients admitted between June 2016 and May 2017 to the MICU of the Robert Wood Johnson University Hospital, New Brunswick, NJ, a tertiary care referral center which is the primary teaching hospital of Rutgers Robert Wood Johnson Medical School. MICU readmission was defined as first admission back to ICU for any cause during the same hospital stay. Subsequent MICU readmissions after the first readmission were excluded in this study. We included all patients except those who were less than 18 years of age, pregnant, planned for hospice or comfort care at discharge from the ICU or general wards, or MICU discharged patients who died on the general wards before being readmitted to the ICU. Patients who were “Do Not Attempt to Resuscitate/Do Not Intubate” but still elected for intensive level of care were included in the study. Based on literature review and clinical judgment, we collected data on variables that were likely to increase the risk of readmissions as mentioned in Supplement Table 1.4,5,6,7,8,9 MICU admission diagnosis was determined through chart review by clinicians and categories were assigned based on the one or more diagnoses that best reflected the reason for ICU level of care.

The study protocol was reviewed and approved by the Institutional Review Board of Robert Wood Johnson Medical School.

Statistical Analysis

The data consisted of training and validation datasets. On the training patient population admitted to the MICU between June 2016 and November 2016, we performed univariable analysis to select variables that were associated with increased risk of MICU readmission. Chi-square was used to assess significance of nominal categorical variables and Mann-Whitney U test was used to assess continuous variables. Variables with 2-tailed p values less than 0.05 were considered for the multivariable logistic regression model. In the logistic regression model, backward elimination with a threshold p value of 0.15 was used. The coefficients of the remaining variables were included in the final model. The reason for keeping p value of 0.15 in backward elimination was to be conservative given the small sample size, and consequently not eliminate clinically important risk factors from the final model.

To create a scoring tool, the score weight for each variable was equal to the variable’s regression coefficient divided by the smallest regression coefficient in the model, rounded to ± 1 or to the nearest multiple of 10. Applicability, efficiency, and ease of use were the guiding principles behind our decision to round like this. However, we did not want to lose discrimination power of the model in the process of making it simpler. Hence, we compared both multivariable logistic regression model and the scoring tool on the validation cohort.

The scoring tool was validated on the patient population admitted to the MICU from December 2016 to May 2017. The fit of the scoring tool on the validation dataset was assessed based on the Hosmer-Lemeshow goodness-of-fit test because this test has asymptotic properties that work well for relatively smaller studies. Total scores of patients in the validation dataset were retrospectively calculated and then grouped into increments of 10 s to assess risk of readmission based on increment of scores. The accuracy of the predictive model in discriminating which patients were readmitted and which were not was evaluated and validated using area under the receiver operating characteristic (AUROC) curves.

RESULTS

Training Population

A total of 535 patients admitted to the MICU between June 2016 and November 2016 fulfilled the initial inclusion criteria. Of those patients, 89 patients expired in the MICU during the first admission, 24 died on the general wards after being discharged from the MICU, and so were excluded from analysis. Thirteen patients with missing data were also excluded from the analysis. Of the remaining patients, the 36 patients who were readmitted yielded a readmission rate of 8.8%. The results of the univariable analysis of all the variables are provided in the Supplement Table 1. Final variables included in the model at the end of multivariable regression analysis are listed in Table 1. As outlined in Table 2, the scoring tool developed included the following variables: MICU admission diagnosis of sepsis, intubation during MICU stay, duration of mechanical ventilation, tracheostomy during MICU stay, non-emergency department admission source to MICU, weekend MICU discharge, and length of stay in the MICU.

Table 1 Logistic Regression Model (Without APACHE III) to Predict MICU Readmission
Full size table
Table 2 Scoring Tool
Full size table

Validation Population

A total of 591 patients were admitted to the MICU between December 2016 and May 2017. There were 94 patients who died in the MICU on first admission and 17 who died on the general wards after being discharged from the MICU. Six patients with missing data were excluded from the analysis. The 43 patients who were readmitted yielded a readmission rate of 9.1% which is similar to 8.8% found in the training dataset. Readmission rates in both training and validation cohorts fall within the range of 1.2 to 14.5% previously reported in the literature.3 Training and validation populations were also similar in terms of the variables included in the final model as shown in Supplement Table 2. The AUROC of the logistic regression model with the seven variables was 0.77 (95% CI, 0.69–0.85). The AUROC of the scoring tool on the validation population was 0.76 (95% CI, 0.68–0.84), and is demonstrated in Fig 1 and also in Supplement Fig 1. Distribution of the percentage of patients readmitted according to the scores in increments of 10 is shown in Fig 2 and Supplement Table 3. The fit of the scoring tool on the validation dataset assessed by Hosmer-Lemeshow goodness-of-fit test was p = 0.644.

Figure 1
figure 1

ROC curves for the scoring tool, logistic regression model without APACHE III, and logistic regression model with APACHE III on the validation cohort in discriminating MICU readmissions. Area under the ROC curve of the scoring tool (0.76; 95% CI, 0.68–0.84), of the logistic regression model without APACHE III (0.77; 95% CI, 0.69–0.85), and logistic regression model with APACHE III (0.79; 95% CI, 0.72–0.86). ROC receiver operating characteristic, MICU medical intensive care unit, CI confidence interval.

Full size image
Figure 2
figure 2

Distribution of non-readmitted and readmitted patients in the validation cohort based on their calculated score grouped in increments of 10. The line represents the percentage of readmitted patients in each of the groups.

Full size image

Additionally, our initial model did include APACHE (Acute Physiology and Chronic Health Evaluation) III as it was found to be statistically significant in the multivariable logistic regression analysis. APACHE III is a severity of illness score initially developed to predict hospital mortality of critically ill hospitalized patients. However, it was removed during further analysis as calculating APACHE III is complex. APACHE III scoring data are also not uniformly available in all ICUs. The AUROC of the logistic regression model with APACHE III and the seven variables mentioned above on the validation dataset was 0.79 (95% CI, 0.72–0.86), and the model is listed in Supplement Table 4. The AUROC of the scoring tool and the logistic regression models with and without APACHE scores were similar and between 0.75 and 0.80 demonstrating that all had good discrimination in terms of predicting MICU readmissions as presented in Fig 1.

DISCUSSION

Here, we present a seven-variable easy-to-use scoring tool developed to predict the risk of readmission of MICU patients after detailed examination of the potential variables that might affect readmission. We created logistic regression models with and without APACHE III score on a training population of 409 patients. Next, we developed a scoring tool that was simple and that incorporated both patient and process factors. The two logistic regression models and the scoring tool were subsequently validated on a separate validation cohort of 474 patients. The AUROC of all three models were between 0.75 and 0.80 meaning they all had good discrimination in predicting patients at risk of MICU readmissions.

Sepsis has been consistently found to be associated with increased unplanned 30-day readmissions to the hospital. A study of nearly 3300 hospitals and over 600,000 sepsis hospitalizations of Medicare patients found 30-day hospital readmissions rate of approximately 29%.15 A retrospective study of 14 million hospitalizations found that sepsis was the leading cause of 30-day hospital readmissions.16 These studies have linked sepsis diagnosis with hospital readmissions, but not ICU readmissions. Our study serves as one of the first to show an association between a diagnosis of sepsis at MICU admission and subsequent MICU readmissions.

A declining respiratory status during ICU admission has been linked with increased rates of readmissions.17, 18 In the training cohort, patients readmitted had an average duration of mechanical ventilation of 3.9 days while non-readmitted patients were on mechanical ventilation for an average of 1.2 days. So it may seem counterintuitive that even though duration of mechanical ventilation is associated with higher readmission, the scoring tool deducts a point for every day a patient is getting mechanical ventilation. The reason for this is the high correlation between the three variables: the use of mechanical ventilation, its duration, and length of MICU stay. Since we already account for mechanical ventilation with 10 points irrespective of the duration, the scoring tool prevents double penalizing a patient for having a prolonged length of stay due to a prolonged duration of mechanical ventilation.

Tracheostomy has been associated with complications such as tube blockage causing respiratory failure, respiratory infections, and bleeding.19 Although there is limited information in the literature about relationship between tracheostomy and MICU readmissions, studies have shown tracheostomy to be associated with greater 30-day hospital readmissions.20 More research in the area of tracheostomy and MICU readmissions is needed.

In our institution as in many other large tertiary referral centers, patients who are admitted from hospital general wards or transferred from outside hospitals tend to be sicker with higher severity of illness scores, are at risk for greater morbidity and mortality, and tend to have a worse prognosis compared to patients admitted directly to MICU from the emergency department even after controlling for patient factors such as age and severity of illness.21, 22 This suggests that there may be other process factors such as time to acceptance after referral, physical consequences of transport, duration and mode of transport, and expertise of the transport personnel affecting outcomes in these patients.23 A combination of sicker patients at baseline and transfer process from non-ED sources may predispose these patients to a worse prognosis at initial MICU admission, and if they survive, according to our analysis, there is a greater chance of MICU readmission.

We also observed increased rates of readmissions for patients discharged from ICU to the general wards on weekends. It seems logical to assume that at most institutions ICUs are usually adequately staffed even on the weekends because of the illness severity of the patients in the unit. However, it is possible that medical general wards may have limited staff over the weekends, with reduced access to key services.24 We hypothesize that patients discharged from ICU to general wards on weekends face this discrepancy in staffing and potentially required services at a crucial transition time.

Our model also demonstrates the importance of ICU length of stay in predicting a patient’s risk of readmission to the ICU. Patients with prolonged ICU stay usually have higher severity of illness, greater interventions such as mechanical ventilation, and multiple organ and physiological disturbances.25 Prolonged ICU length of stay may indirectly reflect greater severity of illness that increases a patient’s risk of readmission.

Markazi-Moghaddam et al.26 in their systematic review of risk prediction models for ICU readmissions identified five previous studies in the literature that described development of ICU readmissions prediction models. Of the five models described, the model by Fialho et al.11 only considered physiological parameters and not process variables while constructing their model. Models by Badawi and Breslow,13 Frost et al.,27 Magruder et al.,28 and Jo et al.4 do consider both patient and process factors; however, they all have certain limitations that our model addresses. Unlike our model, Badawi and Breslow’s13 prediction model is very complex with 23 variables, cannot be performed manually, and needs to be programmed into an electronic decision support system. Similarly, the scoring nomogram developed by Frost et al.27 has eight variables, including APACHE II which itself is a complex scoring tool with 14 variables. As such, the models by Badawi and Breslow13 and Frost et al.27 are not efficient for a clinician who is trying to quickly but accurately risk stratify a patient at the time of ICU discharge. The scoring tool by Magruder et al.28 is easy to calculate, but compared to our scoring tool which has been internally validated on a second separate cohort of patients, it has not been validated and also lacks generalizability as it is primarily developed on data from patients undergoing cardiac surgery. Jo et al.4 excluded patients who were not intubated or extubated within 48 h after intubation which we believe results in lack of generalizability as a significant proportion of critical care patients do not undergo mechanical ventilation during their ICU stay. Because we did not exclude patients on the basis of whether or not they were intubated, our scoring tool actually accounts appropriately for the respiratory status of the patients. This is evident by the fact that two of the seven variables in our scoring tool (intubation during MICU stay and duration of mechanical ventilation during MICU stay) reflect the respiratory status of the patients. Our scoring tool overcomes the limitations of the previous models while maintaining a good discrimination. The AUROC of our scoring tool is 0.76 which is similar to 0.66–0.81 of the previous five models, and thereby our scoring tool provides the ease of use, efficiency, and applicability while maintaining a good and comparable discrimination.4, 11, 13, 26,27,28

We believe that our scoring tool can be utilized by clinicians not only in the ICU but also on the wards or step-down units they are discharged to. In the ICU, clinicians can calculate the readmission score to risk stratify patients at the time of ICU discharge. Patients with higher scores can be discharged to step-down units or general wards with better nursing-patient ratio where patients can be monitored more closely. Similarly, clinicians on the general wards or step-down units can calculate readmission risk score of a patient recently transferred to their floor from ICU. This can lead to close monitoring, early goals of care discussions, and proactive plans for transfer back to ICU for patients with high scores in case a patient’s condition deteriorates.

There are certain limitations of this study. The scoring tool is derived and internally validated based on the patient information at one MICU in a single tertiary care hospital. Also, data collection is retrospective in nature, and sample size is small. We believe multicenter, larger prospective studies based on our analysis would be needed to further validate the scoring tool.

Moreover, we recognize that the events per variable (EPV) have been suggested to be approximately 10. However, given the small study size, the EPV in our prediction model building process was close to 5. We believe that despite this limitation, the final set of predictors in the model are clinically important variables for predicting MICU readmissions. The validation of the model in the validation dataset was good which gives us some reassurance that the power of the model to identify important risk factors was reasonable.

In selecting variables for the study, we did literature review and used clinical judgment to include as many variables as possible. Previous and future literature may show other variables to be associated with higher MICU readmission. We also realize that some clinicians may want to have a severity of illness score and/or mortality predictor scores to calculate risk of readmission. Future models can use our scoring tool as a starting point and modify it to achieve greater discrimination power in accordance with the latest clinical information available in the literature at that time.

Additionally, since different hospitals have variety of intensive care organizational structures, we believe that it may limit generalizability of the scoring tool. For instance, during the study period, our hospital did not have a specialized neurocritical care unit and thus our MICU accepted many patients with primary neurologic diagnosis. Out of 145 patients with neurologic admission diagnosis in the training cohort, 120 patients only had neurologic diagnosis without any other concurrent diagnosis. Twenty-five patients also had concurrent diagnosis groups such as sepsis, respiratory, and cardiogenic. Also, our hospital did not have a step-down or intermediate-care unit, but did have “cluster floors” with nursing to patient ratio of 1:4 for patients who may require a more dedicated nursing care and management. Additionally, our training dataset was on patients admitted from June 2016 to November 2016, and the validation dataset was on patients admitted from December 2016 to May 2017. However, considering that the readmission rates and the baseline characteristics of the variables included in the model were similar between both periods, we do not believe that seasonality affected our final results.

CONCLUSION

In this study of patients discharged from the MICU, we examined and identified patient and process factors that might predispose them to MICU readmission. Based on the factors found to be significantly associated with MICU readmissions, we created a simple scoring tool that we validated on a second cohort of patients. This scoring tool may be used by clinicians both in MICU and the general wards to stratify patients at risk of readmission and provide appropriate level of monitoring and planning they need. Additionally, this is one of the first studies to show an association between MICU admission diagnosis of sepsis and MICU readmissions.