Accuracy of Prospective Predictions of 30-Day Hospital Readmission

This study received institutional review board approval. The study setting is a freestanding psychiatric hospital with six subspecialty inpatient units and a satellite site with a general psychiatry inpatient service. Between May 1, 2013, and August 31, 2013, inpatient social work case managers were asked to complete a brief survey for each discharged inpatient who was an enrollee of a private insurance health plan. The plan provided readmission utilization data for the patient during the 30-day postdischarge period. All participants provided written informed consent. The health plan utilization information was used to calculate prediction accuracy. The four-month observation window was selected on the basis of historical data on the hospital’s readmission rate for the health plan, and because a window of that length was anticipated to provide a sample size of readmitted patients that would provide at least 80% power for our analyses.

The survey consisted of two sections. Part 1 asked case managers, “On a scale from 1 to 5 below, please rate how strongly you have concerns that this patient may need a mental health or substance use disorder readmission within 30 days of discharge from here?” A response of 1 indicated not concerned or low risk, and a response of 5 indicated very concerned or high risk. Part 2 asked objective categorical questions about the patient that may be related to 30-day readmission, for example, questions suggested either by prior literature or by the clinical face validity of particular characteristics. These questions related to active substance use disorder, employment, homelessness, medication adherence, and family involvement during hospitalization. The survey also asked whether the team felt the discharge was premature, whether the aftercare plan included an appointment with a new outpatient clinic or provider, whether the aftercare plan was the preferred or first recommended choice, and whether the patient received an appointment with an outpatient provider within seven days of discharge. Respondents were instructed to leave items blank if they were not confident about the correct response.

We calculated sensitivity, specificity, and positive predictive value (PPV) of the case managers’ Likert-scale ratings to examine the optimal cutoff for predicting readmission. We conducted bivariate analyses of the case managers’ concern ratings (categorical variable) and responses to objective categorical questions in the survey (binary variable) to determine if any were associated with 30-day readmission. Variables with p values ≤.10 were included in a multivariable logistic regression model, with stepwise removal of explanatory variables no longer statistically significant. For low-frequency events, we used Firth’s penalized likelihood logistic regression. We calculated areas under the curve (AUC) receiver operating characteristics (ROC) for the final model. All analyses adjusted for potential clustering of observations within case managers and were conducted with SAS, version 9.2, and Stata, version 13.

During this time period, 323 patients who were enrolled in the private health plan were discharged; surveys were completed for 87% (N=282). Thirty-four of 37 case managers (92%) participated. There was no difference in readmission rates between patients for whom surveys were or were not completed at discharge. Patients for whom surveys were completed had a mean±SD age of 40.5±17.3, and the greatest number were treated in the unit for detoxification of patients with co-occurring mental and substance use disorders (N=79, 28%). Others were treated in the units for depression and anxiety (N=62, 22%), psychotic disorders (N=14, 14%), geriatric psychiatry (N=25, 9%), and dissociative disorders (N=41, 15%) or in the general unit at the satellite site (N=36, 13%).

Twenty-eight patients (10%) had a behavioral health readmission within 30 days. Readmission rates for case manager concern levels of 3, 4, and 5 were 13.1%, 16.7%, and 17.6%, respectively. [A table displaying sensitivity and specificity for alternate cutoffs is available in an online supplement to this report.]

No patients had a repeat index admission during the study. In the unadjusted results, concern levels ≥3 resulted in sensitivity of 86%, specificity of 37%, and PPV of 13%. A cutoff score ≥4 resulted in sensitivity of 39%, specificity of 78%, and PPV of 17%.

In bivariate analyses, only student status and aftercare appointments within seven days of discharge were associated with readmission at the p<.10 level (Table 1). Responses for medication adherence were missing for 44 (16%) patients, but otherwise few data were missing.

The final multivariable model indicated that case manager concern level (odds ratio [OR]=1.52, 95% confidence interval [CI]=1.02–2.27, p=.04) and an appointment scheduled within seven days of discharge (OR=3.23, CI=1.16–9.00, p=.03) were associated with greater odds of 30-day readmission and had an AUC of .67 (CI=.58–78). Thus both variables were significant and independent predictors of 30-day readmission.

Inpatient case managers’ prospective concern level was associated with greater odds of 30-day readmission, but it was not overall highly accurate. Still, our findings indicate that these predictions can be useful in identifying a significant proportion of patients at higher risk of 30-day readmission. In our population, with an approximately 10% readmission rate, a targeted intervention to reduce readmissions using a cutoff of ≥3 would mean that 65% of the sample would have received additional resources or attention to prevent readmission, including 86% of individuals who actually were readmitted. Using a cutoff ≥4 would mean that 24% of the sample—and 40% of the population that was actually readmitted—would have received additional resources.

A program with resource constraints may choose to implement two levels of intervention, depending on the patients’ cutoff scores. Patients who meet a lower cutoff for intervention, who represent a broader proportion of the at-risk population, might receive a postdischarge phone call, whereas patients who meet a higher cutoff threshold might receive more resource-intensive interventions, such as a home visit. Even if some high-risk patients are missed, identifying a significant proportion of higher-risk individuals could provide value.

Among the objective patient characteristics, only one—a follow-up visit scheduled for within seven days of discharge—was (negatively) associated with readmission. Not obtaining an appointment in the seven days after discharge may occur because of patient preferences—for example, patients with an ongoing relationship with a provider may prefer to schedule their own appointments or patients may be reluctant to schedule a follow-up visit in general—or practices by the outpatient clinic—for example, clinic personnel directly notify new patients about a discharge appointment or had no appointments available within seven days of discharge. Nevertheless, our results indicate that appointments not scheduled within seven days after discharge, regardless of the reason, are an independent risk of 30-day readmission. Research is needed to learn if improving scheduled appointment rates reduces 30-day readmission.

Several patient characteristics that have been associated with readmission in prior literature were not associated with readmission in this study. Possibly our study was underpowered to detect statistically significant associations between 30-day readmission and these characteristics. Consistent with prior literature, however, a comparison of the distribution of these characteristics among patients who were or were not readmitted suggests that there was an association in the expected direction between readmission and many characteristics, such as substance use disorders, employment status, homelessness, new or significant stressors, and new aftercare providers. Nonetheless, case manager predictions and scheduled follow-up within seven days of discharge were more strongly associated with readmission compared with these other characteristics. Possibly, that is because case manager concern levels incorporate a variety of risk and mitigating factors. Also, in contrast to prior literature (10,15), we found similar readmission rates for medication-adherent and -nonadherent patients. However, case managers skipped this question most frequently (16%), suggesting that they were unsure of the response. Possibly, this information was not missing at random, which may have affected our results.

A limitation was that although the survey included questions about patient characteristics that would be readily known by case managers, we did not verify this information with chart review, and it is possible that there were some inaccuracies in case managers’ responses. To minimize this possibility, we requested that the case managers avoid completing survey items if they were not confident of the response. The missing responses for medication adherence suggest that respondents followed these instructions.

Another limitation was that the study was conducted in a single hospital, which may limit generalizability. To mitigate this limitation, we included a diverse patient population from multiple inpatient units, including general and subspecialty units as well as an off-site satellite program. Further strengthening our findings was the high rate of case manager participation (92%) and survey completion (87%). Still, there may be hospital organizational characteristics that affected readmission prediction accuracy and limit the generalizability of our findings. For example, hospitals may vary in case managers’ ability to adequately interact with patients or family members (because of caseload constraints), which could impact their knowledge of pertinent patient history or characteristics and affect the accuracy of their concern level in predicting readmission.

Finally, although concern levels by inpatient case managers about 30-day readmission can be a very “user friendly” method to identify some patients at higher risk of readmission, we found few objective predictors of readmission. These results limit our ability to identify more systematic interventions that could be tailored to patients, other than scheduling appointments within seven days postdischarge. Future research is needed to test whether clinician ratings, coupled with an additional process that can help avoid rehospitalization for targeted patients, can successfully reduce readmissions. Such a process could include, for example, a clinical case conference to discuss patients who score above a 30-day readmission risk threshold. Similarly, our results may not generalize to interventions in which the same clinician who provides the ratings is given additional tasks on the basis of those ratings, for example, making additional postdischarge calls to patients, possibly without supplemental resources.

This study demonstrates that a simple Likert scale completed by inpatient social worker case managers can successfully identify a significant subset of patients who are at higher risk of 30-day readmission, albeit at the cost of misidentifying a relatively large proportion of patients who are not. Not having a scheduled follow-up within seven days of discharge improved the prediction accuracy of case manager concern level for 30-day readmission. Such information could be useful for targeting new interventions for patients determined to be at high risk, helping to reduce 30-day readmissions of patients with psychiatric illnesses.

The authors thank Optum for providing health plan readmission data for the study population.