Predictive Models and the Effectiveness of Strategies for Improving Outpatient Follow-Up Under Managed Care
Abstract
OBJECTIVES: This study tested the accuracy of models for predicting rehospitalization in a managed behavioral health organization and tested the effectiveness of different care management strategies for enhancing outpatient treatment follow-up. METHODS: In a controlled study, patients with an inpatient mental health or substance use admission received one of three types of care management, distinguished by the level of care managers' involvement in discharge planning and postdischarge outreach: usual (N=31), enhanced (N=94), and intensive (N=74). The groups were compared with each other and with a cohort admitted in the year before the study that received usual care management (N=192) to determine whether differences existed in time to outpatient follow-up, amount of postdischarge care, and rehospitalization at 30, 60, and 180 days. RESULTS: No differences between groups were found. The majority of patients (69 percent) received outpatient care within 30 days of discharge. Prediction models using logistic regression suggested that the number of clinical and sociodemographic risk factors identified by care managers was related to the rate of rehospitalization at 60 and 180 days. Patients authorized to receive intermediate care (partial hospitalization or residential care) and those who failed to attend intermediate care if it was authorized were more likely than other patients to be rehospitalized at 30, 60, and 180 days. CONCLUSIONS: Outpatient follow-up after psychiatric hospitalization did not improve with increasingly intensive discharge planning and outreach. Improvement in prediction of risk of rehospitalization may increase opportunities to provide intensive interventions for difficult-to-engage patients.
The difficulty in engaging psychiatric inpatients in meaningful outpatient treatment after hospital discharge has been a long-standing concern of organized care systems. In previous studies conducted mainly in public-sector general hospitals and the Department of Veterans Affairs system, researchers have estimated that 40 to 60 percent of inpatients fail to connect with an outpatient clinician (1). Two studies suggested that rates of outpatient treatment 15 to 30 days after discharge in populations insured by managed behavioral health organizations may be higher, in the range of 70 to 80 percent (2,3). In one study, rates of outpatient treatment within 30 days of discharge increased from 67 percent to 83 percent after a managed behavioral health organization began managing care for a privately insured plan (2). In a pre-post study, enhancement of discharge planning in a managed behavioral health organization (United Behavioral Health, Health Plan Division) raised outpatient follow-up attendance to 78 percent, an increase of 87 percent from a baseline level of 55 percent (3).
An unresolved question in the literature is whether a patient's failure to engage in outpatient treatment after discharge is determined by characteristics of the patient, such as age, sex, educational level, and diagnosis, or by aspects of the care system, such as the availability of services or benefit restrictions. A comprehensive review of studies published between 1974 and 1994 concluded that various patient characteristics were relatively weak predictors of outpatient follow-up compared with measures of system responsiveness (1). This conclusion was supported in a recent study in which the effects of several patient characteristics were analyzed in a sample of 229 public psychiatric patients discharged from a general hospital (4). The predictive value of patient characteristics was heavily outweighed by the predictive value of whether three clinical interventions had taken place: communication between inpatient staff and outpatient clinicians about the patient's discharge plans, the patient's starting the outpatient program before discharge, and involvement of the family in treatment. Although these three factors were strongly predictive, the factor associated with the highest outpatient follow-up rate (62 percent) was starting the outpatient program before discharge. The average outpatient follow-up rate reported by this study was only 35 percent at 30 days, which was consistent with rates reported in earlier studies (1).
Interpretation of the existing literature is difficult, because most studies have been conducted in systems with relatively poor outpatient follow-up rates, have used uncontrolled designs, and have examined patient characteristics in isolation rather than by using multivariate statistical procedures (1). Perhaps counterintuitively, patient characteristics may be more predictive of risk of poor outpatient follow-up in systems where follow-up is uniformly high. Consistent with a model described by Klinkenberg and Calsyn (1) for predicting receipt of outpatient care, where system responsiveness is homogenized because it is consistently high, patient characteristics and other situational factors may be the strongest predictors of postdischarge follow-up.
In the study reported here, we sought to test the relative effects of patient characteristics and system characteristics in facilitating outpatient follow-up and reducing rehospitalization. We did this in two ways: first, by manipulating the intensity of postdischarge care management in a controlled study and observing rates of outpatient follow-up and rehospitalization, and, second, by testing prediction models of rehospitalization under conditions in which the managed behavioral health organization has ensured the availability of behavioral health services. The explicit premise in this study is that repeated hospitalizations represent a failure of the clinical service system.
Methods
This study was part of a quality improvement program initiated in 2000 by United Behavioral Health, a national managed behavioral health organization, to identify high-risk patients and to examine the influence of care managers in enhancing outpatient follow-up and reducing subsequent rehospitalization. Two aspects of the program are noteworthy. First, in developing a model for prediction of rehospitalization, the program did not rely solely on administrative data sources; it also used data on the presence of six clinical risk factors on the basis of judgments made by care managers during the course of their review of inpatient care. In addition, the quality improvement program was implemented in the context of a controlled study rather than a pre-post study or nonequivalent group comparisons. This latter aspect is critical in disentangling the influence of care management and patient characteristics.
Sample
The study was based on records for all patients from three national self-insured private employer health plans who were admitted to an acute inpatient setting with a mental health or substance use diagnosis during two periods—May through September 1999 and May through September 2000. During the study periods, the health plans represented 209,011 covered lives, and a total of 391 patients, including insured employees and dependents, had an inpatient mental health or substance use admission. The research project was approved by the employers before the study. No information about individuals was released to the employers.
Inpatients admitted during the 2000 study period were assigned in alternating weeks to receive one of three levels of care management from the managed behavioral health organization: usual care management, enhanced care management, or intensive care management. Patients from one employer, the largest of the three, had been receiving enhanced care management before initiation of the study. As a result, these members were assigned to either enhanced care management or intensive care management on alternating weeks. Using this procedure, 31 patients were assigned to receive usual care management, 94 to receive enhanced care management, and 74 to receive intensive care management. An additional 192 patients admitted during the period from May through September 1999 were included to allow comparison between current and previous patterns of care. These 192 patients were not included in the models predicting rehospitalization.
Levels of care management
Usual care management involved regular reviews of medical necessity and the appropriateness of the treatment plan during the inpatient stay. The reviews were conducted by care managers who were experienced and licensed clinicians. The managed behavioral health organization requested that the patient's inpatient care staff arrange an outpatient follow-up appointment within 72 hours after discharge. The only intervention provided by the care manager after discharge was one phone call to the patient within 24 hours of discharge to verify that an outpatient appointment had been made and to remind the patient to attend. The care manager initiated no additional contacts with either the inpatient or outpatient care staff.
Enhanced care management consisted of usual care management during the inpatient stay and supplementary follow-up provided by intake counselors. The intake counselors were master's-level clinicians, most of whom had relatively little direct clinical experience and were not licensed. They acted as discharge planners by contacting inpatient facilities before discharge and ensuring that the facility had scheduled outpatient treatment within 72 hours of discharge for patients who were leaving inpatient care. The intake counselor contacted the patient within 24 hours of discharge and reiterated the importance of follow-up treatment. The intake counselor also called the outpatient clinician within 48 hours of the appointment to determine whether the patient had complied with the discharge plan. If the patient did not attend, calls were made weekly to urge adherence to the plan. If the patient was dissatisfied with the outpatient clinician, additional referrals were offered. The intake counselor continued efforts to call the patient and the outpatient clinician for as long as two weeks after discharge in an attempt to ensure that follow-up treatment was initiated.
Intensive care management was a more time-consuming and costly intervention, provided by experienced and licensed clinicians. The intensive care manager contacted the patient and the inpatient staff as soon after admission as possible to ensure that the patient had an appropriate aftercare plan. The intensive care manager worked with the outpatient clinician to identify any early signs of the patient's relapse and to intensify treatment if necessary, was involved in treatment planning with the outpatient clinician, and reached out to the patient in cases of noncompliance. The intensive care manager ensured that an outpatient appointment was scheduled within 72 hours of inpatient discharge and coordinated treatment across all services involved in the patient's care. The care manager also conducted telephone reviews with the outpatient clinician every three weeks for the first 12 weeks after discharge and contacted the patient within 24 hours after the first scheduled appointment to verify that it occurred. Patients who were authorized to receive partial hospitalization were contacted by the care manager weekly.
Measures
Measures included number of days to first rehospitalization within six months after discharge and number of days to first outpatient or intermediate care—that is, partial hospitalization or residential care—visit. In addition, we examined total use of inpatient, outpatient, and intermediate care in the six months after discharge.
Before discharge, care managers rated six factors thought to predict rehospitalization: a diagnosis of schizophrenia or bipolar disorder, the presence of dual diagnoses, a history of noncompliance with treatment, a significant lack of social supports or family or housing problems, a history of one or more acute admissions in the 12 months before the index admission, and having a child aged 13 years or younger. The risk factors were summed to create a single risk scale on which scores could range from 0, no risk factors identified by the care manager, to 6, all six risk factors identified by the care manager.
Additional risk factors identified from United Behavioral Health's claims and authorization data included whether the inpatient attending psychiatrist was the treating outpatient psychiatrist, whether the patient was authorized to receive intermediate care, whether the patient failed to attend the outpatient or intermediate care appointment, and the patient's age, sex, and relationship to the managed behavioral health organization subscriber.
Statistical analyses
Survival analysis was used to model the number of days to first rehospitalization, outpatient visit, and intermediate care. General linear regression models developed with SAS statistical software (5) were used for between-groups comparisons of the amount of services used after discharge. Logistic regression models were used to predict rehospitalization within 30, 60, and 180 days of discharge on the basis of risk scores assigned by the care manager and other patient characteristics identified from administrative data. Analyses based on a simple count of the number of risk factors identified by the care manager was compared with analyses in which each of the six risk factors was entered separately into a logistic regression model. The two approaches had equivalent overall predictive accuracy in correctly identifying high-risk and low-risk patients, so we adopted the simpler checklist approach to measuring risk. Risk cutoffs were set to minimize the rate of false-positive and false-negative cases (6). False-positive cases involved patients who were predicted to be rehospitalized but were not. False-negative cases involved patients who were predicted not to be rehospitalized but were rehospitalized.
Results
The sociodemographic and clinical characteristics of the study groups are presented in Table 1. The results of group comparisons suggested that the assignment procedures produced equivalent groups, with two exceptions. Patients in the usual care management group were about six years younger on average than those in the other groups, and care managers identified more patients with dual diagnoses in the intensive care management group than in the other two care management groups.
Effects of care management levels
The care management groups did not differ in their time to outpatient treatment. Figure 1 displays the adjusted cumulative probability of receiving intermediate care or any care as a function of days since discharge. Clearly, follow-up with outpatient services was no more likely with intensive care management or enhanced care management than with usual care management or with usual care management in the year before the quality improvement program. Estimates from the proportional hazards regression model suggested that 46 percent (95 percent confidence interval [CI]=42 percent to 51 percent) of patients received some aftercare within seven days of discharge and 69 percent within 30 days (95 percent CI=65 percent to 73 percent).
To explore whether intensified care management affected the likelihood of ongoing adherence after the initial visit, we examined measures of the total amount of mental health care delivered after discharge. Table 2 lists the mean number of inpatient days, outpatient visits, intermediate care days, and other service units for each group. No statistically significant differences were observed across groups in any measures of postdischarge mental health service use.
Not surprisingly, intensified care management also did not have an effect on the time to rehospitalization. Proportional hazards regression analysis comparing study groups found no differences in community survival for individuals in the enhanced care management (β=.26, p=.30), intensive care management (β=.37, p=.17), and usual care management (β=.17, p=.69) groups compared with community survival in the year before the quality improvement program. Overall, the model fit was poor (χ2=4.7, df=5, p=.44).
Predictors of rehospitalization
The central hypothesis of this study was that rehospitalization may be more strongly determined by patient characteristics when the availability of mental health services is uniformly high. The fact that proactively increasing the availability of services to discharged patients did not increase outpatient follow-up and did not reduce rehospitalization is consistent with this hypothesis. The predictive models developed in this study provided more direct support for this hypothesis.
Logistic regression models were used to predict rehospitalization at 30, 60, and 180 days. Because care managers' ratings were initiated at the start of the quality improvement program, the logistic regression analyses were computed with data from the 199 patients in the three care management groups. Predictors included the care manager's risk score in addition to each of the characteristics listed in Table 1. Nonsignificant risk factors were eliminated from the logistic models, leaving the most parsimonious set. In each regression model, being a dependent of the insured party, having a higher risk score assigned by the care manager, being authorized for intermediate care rather than outpatient care after hospitalization, and failing to attend intermediate care appointments were predictive of rehospitalization at each follow-up point. Table 3 shows the odds ratios associated with these factors. In each model, the rates of correct identification of high-risk and low-risk patients were high but included a high rate of false-positive cases, which suggests that the models' predictions were substantially more accurate than random selection would have been.
Discussion
Clinical management across the continuum of care is challenged by the fact that some high-risk patients do not receive appropriate treatment after discharge from a psychiatric hospital. Previous studies have sounded the hopeful note that factors under the control of service systems, such as the amount and type of discharge planning and the delivery of appropriate and intensive follow-up treatment, are more important correlates of rehospitalization than are patient characteristics (1,4). Two important findings in our study call this hypothesis into question, at least for service systems in which outpatient follow-up rates are relatively high.
First, we found that intensification of care management services did not increase outpatient follow-up, nor did it decrease rehospitalization rates. United Behavioral Health's quality improvement program involved two levels of care management intensification, one of which included frequent contacts by experienced clinicians with patients and with all treating outpatient clinicians. Although intensive care managers reported that treating clinicians and patients appreciated the outreach calls, measures of outpatient follow-up and rehospitalization were no better than with usual care management and no better than in the year before the quality improvement program.
It is noteworthy that our not finding a care management effect may have been due to inadequate statistical power, particularly for the usual case management group. Only 31 patients were included in this group because of a preexisting agreement with one employer that guaranteed at least the enhanced care management protocol. In addition, the intensive care management group had the highest number of patients with dual diagnoses, although this group was otherwise comparable to the other groups. This difference may have biased the study toward a null finding.
Second, prediction models showed that clinical and demographic characteristics of patients were significantly associated with risk of rehospitalization. A risk factor score composed of six criteria identified by care managers in the normal course of their review of inpatient care was strongly associated with rehospitalization. Risk factors were objective and readily obtainable from information routinely collected during an inpatient admission. An additional factor determined by the care manager's assessment of risk was also associated with rehospitalization—an authorization by the care manager for partial hospitalization or residential care (a proxy for severity). Finally, the patient's failure to follow through on the authorization for intermediate care also increased the likelihood of rehospitalization. Rates of false-positive cases were high, but not so high that they precluded the utility of the model, especially if effective interventions for engaging high-risk patients can be identified.
Clearly, the care management interventions tested in this study represent only one possible set of solutions to the problem of failed outpatient follow-up. The absence of positive findings suggests that simply increasing proactive, telephone-based interactions with patients and treating clinicians will not improve care processes, especially if the existing system of care is performing well for high-risk patients. The results of some past studies of managed behavioral health organizations and of this study suggest that the likelihood of postdischarge follow-up care is higher in managed behavioral health systems than in unmanaged systems and in public-sector populations (2). Data from the Health Plan Employer Data and Information Set for a large group of private health plans have shown similarly high rates of outpatient follow-up (70.1 percent) (7).
Improvements beyond the levels currently observed in managed behavioral health are unlikely to come through unilateral quality improvement initiatives launched by the managed behavioral health organizations. Interventions involving closer collaboration between managing entities and the facilities and staff that deliver acute care to high-risk patients may be more fruitful. New technologies, such as outcomes management systems and electronic medical records implemented through the Internet and through interactive voice response systems, may provide the needed connectivity across components of the service delivery system. Use of such systems could enhance prediction models by facilitating communication between clinicians and managed behavioral health organizations. Information about patients' noncompliance, early response to treatment, family involvement, and clinical severity after return to the community would be invaluable to systemic efforts to respond with new and intensified interventions and resources such as those implemented in assertive community treatment programs (8). We predict that the results of this study would be replicable in public-sector populations served by assertive community treatment programs in which the level of system responsiveness is high and patients' characteristics could be expected to account for a high proportion of the variance in receipt of aftercare and readmission. Incremental improvements may also come through the use of enhanced monitoring systems to identify the most successful clinicians and programs for treating the most challenging patients, which this study could not address.
Conclusions
Ongoing improvement of outpatient continuity of care for patients who have experienced psychiatric hospitalization did not improve as a result of simple interventions involving telephone outreach and enhanced discharge planning. The study results suggest that patients served in a privately insured managed behavioral health organization who do not participate in outpatient follow-up care and who are at risk of rehospitalization may be less influenced by such interventions than by personal risk factors. The success of prediction models that identify high-risk patients early in treatment may allow for the development of innovative interventions that are more successful with this population. Future research may focus on the practices and clinical support systems that are effective in engaging high-risk patients in treatment and the extent to which better outpatient follow-up lowers the likelihood of rehospitalization.
Acknowledgment
The authors thank Howard H. Goldman, M.D., Ph.D., for comments on a previous version of the manuscript.
Figure 1. Days from discharge to first appointment for any type of aftercare or intermediate care of patients with an inpatient mental health or substance use admission who received one of three levels of care management during a quality improvement program in a managed behavioral health organization or usual care management in the year before program implementationa
a No significant differences between groups
Footnote
The authors are affiliated with United Behavioral Health, 425 Market Street, 27th Floor, San Francisco, California 94105 (e-mail, [email protected]).
References
1.
Klinkenberg WD, Calsyn RJ: Predictors of receipt of aftercare and recidivism among persons with severe mental illness: a review. Psychiatric Services 47:487-496, 1996
2.
Merrick EL: Treatment of major depression before and after implementation of a behavioral health carve-out plan. Psychiatric Services 49:1563-1567, 1998
3.
Nelson EA, Maruish ME, Axler JL: Effects of discharge planning and compliance with outpatient appointments on readmission rates. Psychiatric Services 51:885-889, 2000
4.
Boyer CA, McAlpine DD, Pottick KJ, et al: Identifying risk factors and key strategies in linkage to outpatient psychiatric care. American Journal of Psychiatry 157:1592-1598, 2000
5.
SAS/STAT Users Guide, version 8, vol 1. Cary, NC, SAS Institute, 1989
6.
Hanley JA, McNeil BJ: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143:29-36, 1982
7.
National Committee for Quality Assurance: National Results for Selected 2000 HEDIS and HEDIS/CAHPS Measures. Oct 30, 2000. Available at www.ncqa.org/programs/hedis/00measures.htm
8.
Olfson M: Assertive community treatment: an evaluation of the experimental evidence. Hospital and Community Psychiatry 41:634-641, 1990
Information & Authors
Information
Published In
History
Published online: 1 November 2002
Published in print: November 2002
Authors
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBGet Access
Login options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).