Comparing Quality of Mental Health Care for Public-Sector and Privately Insured Populations
Abstract
OBJECTIVE: This study examined the methodological difficulties of comparing quality of care in large health care systems. It demonstrated methods for measuring quality of mental health care and, using these measures, compared patients from Department of Veterans Affairs (VA) hospitals with privately insured patients. METHODS: Individuals receiving VA inpatient mental health care during the first six months of each fiscal year from 1993 to 1997 were identified from discharge abstracts. A similar cohort of privately insured individuals was identified using MEDSTAT's MarketScan database from 1993 to 1995. Individuals in both cohorts were tracked for six months after discharge. Length of stay, readmission rates, and access to outpatient services were calculated. RESULTS: The private sector outperformed VA on most quality measures, although differences were modest and can likely be explained by the greater severity of illness and social disadvantages of VA patients. Readmission rates increased considerably over time in the private sector, whereas they declined for VA patients. Quality measures varied by diagnosis, with VA performing better than the private sector in treating patients diagnosed with substance abuse and mental disorders not elsewhere classified but worse in treating patients diagnosed with depression. CONCLUSIONS: Although the private sector modestly outperformed VA on most quality measures, VA treats a more troubled population, and it improved markedly over time compared with the private sector. As health systems strive to reduce costs of care, methods for comparing and evaluating the quality of care become increasingly important. However, methodological challenges remain substantial.
Rapid changes in the delivery of health care in recent years have led to increased emphasis on evaluating the quality of care. Quality evaluation on a large scale is extremely difficult, however (1). The Health Plan Employer Data and Information Set (HEDIS) has become the standard tool for comparing health plans (2,3). Even though other measures of mental health quality have recently been considered for use in HEDIS (4), this tool employs only a single measure of the quality of mental health services: the percentage of patients hospitalized for affective disorder who receive outpatient services within 30 days of discharge (2,3).
Over the past four years, the Department of Veterans Affairs (VA) has developed a Mental Health Program Performance Monitoring System. Because it uses a variety of patient-level measures of mental health service delivery, it provides a more comprehensive tool for monitoring the quality of care (5,6).
In response to pressure to reduce the costs of health care, both the private sector and VA have undergone significant changes. The private sector has developed various cost-containment mechanisms, such as utilization review, case management, exclusive contracting arrangements with selected providers, and risk sharing (7,8,9,10,11). In October 1995 VA implemented a major reorganization that produced similar results (12). The VA system was reorganized into 22 distinct networks, each responsible for the health care needs of a geographically defined population. A major goal was to shift the focus of care away from acute inpatient care and toward ambulatory and primary care (13). National performance standards were implemented to encourage this change. These management changes have had significant effects on mental health utilization and costs in both systems (12,14,15).
In the study reported here, we examined the impact of changes in the organization and delivery of mental health care on the quality of mental health care. The Mental Health Program Performance Monitoring System includes a variety of quality measures, which are constructed from readily available administrative databases. Using these measures, we compared the quality of mental health care in VA between 1993 and 1997 and in a sample of privately insured individuals between 1993 and 1995. Because comprehensive assessment of quality must involve trend analysis in addition to assessment of cross-sectional performance, we included 1997 VA data to compare how the systems have changed in response to the introduction of managed care mechanisms.
Methods
Sources of data
VA data for this study came from two national data files that include information from 172 VA medical centers. Patients admitted to the hospital for a psychiatric or substance abuse disorder were identified using the patient treatment file, a discharge abstract of all completed episodes of inpatient care in the VA system. Data on outpatient service use were collected from the outpatient file. Data are available from fiscal year 1993, which began October 1, 1992, to fiscal year 1997, which ended September 30, 1997.
Data on privately insured individuals come from MEDSTAT's MarketScan database, which contains claims information for a national sample of more than seven million covered lives between 1993 to 1995. The claims data, which cover employees and retirees of large corporations and their dependents, were collected from more than 200 different insurance plans, including Blue Cross and Blue Shield plans and third-party administrators.
For both systems, we identified patients admitted to the hospital with a psychiatric or substance use disorder and discharged in the first half of the year. In the VA system behavioral health inpatients were identified by the bed section from which they were discharged. Privately insured behavioral health patients were identified by the primary diagnosis associated with the admission, usually the discharge diagnosis. In both systems, these patients were tracked for 180 days after discharge, and data on both inpatient readmission and outpatient service delivery were recorded.
Quality measures
We defined ten measures to evaluate the quality of mental health care. The inpatient measures include the length of the index hospital stay, the number of days to the first readmission after discharge, and the percentage of patients who were readmitted within 14, 30, or 180 days after discharge. Readmission measures have been shown to be significantly related to clinical outcome (16).
Outpatient follow-up measures included the percentage of patients who had an outpatient visit within 30 or 180 days after discharge, the number of days to the first outpatient visit, the number of outpatient visits in the 180 days after discharge, and a measure of continuity of care. The continuity measure was defined as the number of two-month periods with two or more outpatient visits in the six months after discharge. These measures have also been shown to be significantly related to some clinical outcomes (16).
Patient characteristics
In computing the quality measures described above, we controlled for a number of patient characteristics. We were able to control for age and gender of the patient across both systems, but were unable to control for ethnicity in the privately insured sample because insurance organizations are prohibited from collecting this information. We also could not control for income, disability, or homelessness, all of which are likely to be very different between these two systems.
Besides controlling for demographic characteristics, we assigned each patient to one of five diagnostic groups: schizophrenia (ICD-9 codes 295.00 to 295.99), major depression-bipolar disorder (296.00 to 296.99), substance use (291.00, 292.00, and 303.00 to 305.99), mild-moderate depression (300.40, 300.50, 301.10, 309.10, 309.90, and 311.00), and other psychiatric disorders (290.00 to 312.99 not elsewhere assigned).
The diagnostic group to which a patient was assigned was based on the ICD-9 code representing the primary diagnosis of the index admission. The diagnostic categories are mutually exclusive. Of the diagnoses included in "other psychiatric disorders," the most common were adjustment reaction (25.8 percent for the private sector and 7.3 percent for VA), neurotic disorders (19.3 percent for the private sector and 3.9 percent for VA), and other nonorganic psychoses (13.5 percent for the private sector and 3.9 percent for VA).
Finally, among VA patients, we controlled for whether the patient's illness was connected to military service. Veterans with service-connected disorders have more comprehensive coverage than those whose illness is not connected to service.
Statistical analysis
Analysis proceeded in several steps. For both the VA and privately insured samples, we first identified inpatients discharged in the first half of each year with a behavioral health primary diagnosis. Next we abstracted service utilization data, which we used to construct the quality measures described above. We then calculated adjusted means for each of these measures for both the VA and the privately insured samples, controlling for year, patient age, gender, diagnostic category, whether the patient had dual diagnoses, and, for VA patients only, whether the disorder was service connected.
We ran separate models for the VA and the privately insured samples to control for a different set of patient characteristics in each sample. We used the SAS general linear models procedure (17), with the least-square means option, to calculate the means of our measures. To investigate the interaction of year with patient characteristics, we conducted a series of two-way analyses of covariance in which we looked at differences across the years in the effects of patient characteristics on the dependent variables. (A more detailed description of the procedures is available from the authors.)
Results
Sample characteristics
Table 1 shows sample characteristics. The VA sample was overwhelmingly male, whereas the private-sector sample was more evenly divided across gender and was younger than the VA group. A considerably higher fraction of VA patients were diagnosed as having schizophrenia, and a much lower fraction were diagnosed as having major depression or bipolar disorder. Finally, VA patients were much more likely than privately insured patients to have a dual diagnosis.
These results illustrate the significant differences between the two populations. Individuals eligible for VA services are poor and often homeless (18) and unemployed (6). We expected such a population to be sicker on average than a group of employed individuals with private health insurance, but we could not control for all of the differences between these populations because data on homelessness, income, and disability are not available from the MarketScan database.
Inpatient measures
Table 2 shows the adjusted means for the quality measures for the VA and private-sector systems and the difference between the two systems on these measures. We cannot report statistical tests of the differences across service systems because these models were run separately for each system and could not legitimately be merged. Table 3 presents changes over time within each system, along with data on significant differences.
As Table 2 shows, in 1993 the length of stay was substantially higher in the VA sample than in the privately insured sample, 34.6 days versus 13.1 days. Length of stay declined over time in both systems, although the decline was much faster in the private sector than in the VA system between 1993 and 1995. After 1995, length of stay fell much faster in VA, with the rate of decline exceeding that seen earlier in the privately insured sample.
Rates of readmission within 180 days of discharge were higher in the VA sample in 1993 (36.6 percent compared with 28.2 percent), but the differences were smaller for readmission rates within 30 or 14 days of discharge. In the VA system, long-term readmission rates stayed roughly the same, and short-term readmission rates declined slightly. In the privately insured sample, however, short-term readmission rates increased dramatically—more than 80 percent between 1993 and 1995. As a result, the gap between the VA system and the private sector in readmission rates declined over time. We see a similar pattern in the number of days to the first readmission.
Outpatient measures
In 1993 the percentage of patients receiving outpatient care within 180 days of discharge was fairly high and virtually the same in both systems (71.7 percent for VA and 72.2 percent for the private sector). Patients in the private sector received outpatient care sooner, however. The percentage of patients receiving outpatient care within 30 days of discharge was higher for privately insured patients (59.2 percent versus 47.8 percent), and the number of days to the first outpatient visit was lower (22.5 versus 32.2). These measures improved over time in both service systems, but much more so in the privately insured sample than in the VA sample. Privately insured patients also had a slightly higher number of visits than VA patients in the 180 days after discharge.
The VA outperformed the private sector on one outpatient measure: continuity of care. In 1993 VA patients averaged 1.16 two-month periods with two or more outpatient visits in the six months after discharge, compared with 1.06 in the privately insured sample. This difference declined over time, however, as the continuity measure increased faster for the privately insured sample than for the VA sample.
Differences by diagnostic group
We also stratified the above analyses by diagnostic group. (Data are available from the authors.) In both service systems, patients diagnosed as having schizophrenia or major depression-bipolar disorder had longer lengths of stay and higher readmission rates. Patients diagnosed as having a substance use disorder had the lowest readmission rates, especially short term. Readmission rates generally declined over time, although slightly less so for patients with schizophrenia. Because these patients are sicker, they may be less likely to improve on the measures over time.
Patients with major depression or bipolar disorder had higher rates for outpatient visits within 30 and 180 days after discharge, more outpatient visits, and higher continuity-of-care scores compared with other diagnostic groups. Patients with a substance use disorder had the lowest outpatient admission rates, fewest outpatient visits, and lowest continuity scores, although these measures increased more over time than for other diagnostic groups.
Comparing the two systems across diagnostic groups, the private sector outperformed VA to a modest degree on many of the quality measures. Relative to levels in the private sector, inpatient readmission rates are lowest in the VA system for patients diagnosed as having other psychiatric disorders. VA patients diagnosed as having schizophrenia had lower readmission rates than the private-sector patients in 1993, but relatively high rates in 1995. Patients with major depression-bipolar disorder or mild-moderate depression had the highest readmission rates relative to the private sector. VA patients diagnosed as having a substance use disorder came the closest to their private-sector counterparts on the outpatient measures.
Discussion
In this study, we compared the performance of VA and private-sector systems on ten measures of quality of mental health care, adjusted for age and gender but not for other potentially confounding factors. We found that, overall, private-sector mental health inpatients had shorter lengths of stay, more days to the next inpatient readmission, and lower readmission rates within 14, 30, or 180 days of discharge compared with VA mental health inpatients. Although VA patients had higher continuity-of-care scores, moderately higher proportions of private-sector patients had an outpatient visit within 30 and 180 days after discharge. Private-sector patients also had fewer days to the first outpatient visit and more outpatient visits in the six months after discharge. Readmission rates increased substantially over time for private-sector patients, however, whereas these measures fell in the VA sample. The proportion of patients receiving outpatient care within six months of discharge improved over time for both populations.
When we stratified the analysis by diagnostic group, we found that the private sector outperformed VA to a modest degree for each of the diagnostic groups. The VA quality measures were highest relative to the private sector in treating substance use and other psychiatric disorders. The difference between the service systems was largest for patients in the depression diagnostic groups. We might have expected VA to do relatively well in treating the more severe diagnostic groups, as it has more comprehensive coverage. However, the results did not bear out this expectation.
One possible explanation for this discrepancy is that private-sector patients who are more severely ill may have exhausted their plan benefits. In that case, our claims database would not have captured services used after the claims were no longer paid. The difference in readmission rates may also be biased downward for this reason as the VA system does not impose such benefit limits. In addition, patients in the private sector must follow very specific rules in order for claims to be considered valid by insurance companies. Some claims may not be reimbursed—and therefore would not show up in the database—if these rules are not followed.
Unmeasured differences in the types of patients treated in these service systems are likely to explain some of the observed differences in quality measures between VA and the private sector. In contrast to private-sector patients, VA patients are poor, with an average income of $10,204 (5). They are typically disabled (6), and 35 percent are homeless (18). Further, they generally have more severe disorders than patients in the privately insured sector.
Other studies have found that low-income and homeless patients have longer lengths of stay than other patients (19). A study of hospital discharge data from hospitals in New York City between 1992 and 1993 found that lengths of stay for homeless mentally ill patients averaged 36.2 days (20), just slightly more than the average length of stay of 34.6 days for VA patients in 1993. In addition, poor, disabled, or homeless patients might be more likely to develop complications requiring readmission or be less likely to show up for outpatient follow-up appointments.
This study demonstrates some of the methodological challenges of assessing quality across health care systems. One can argue that low readmission rates or high proportions of discharged inpatients receiving outpatient follow-up care are indications of quality health care. However, low readmission rates may also be a result of restrictive readmission criteria or exhausted plan benefits, and a high proportion of inpatients receiving outpatient care within 30 days of discharge might suggest that patients were pushed out of an inpatient setting to outpatient care too soon. Without detailed information on procedures and outcomes, it is difficult to determine overall quality using administrative data. Another approach, using detailed survey data to compare quality across systems, is feasible (21). It is more costly, however, and relies on much smaller and possibly unrepresentative sample sizes.
Despite the differences observed in our quality measures between VA and private-sector patients—perhaps partly attributable to differences in severity of illness and failure to capture all private-sector service use—it is notable that the systems were roughly similar on most measures. It is also of interest that inpatient readmission rates increased substantially in the private sector while they declined in the VA system, which may reveal significant differences in how the quality of care is changing over time in these two systems.
Do the substantial limitations of our quality measures and the differences in the populations served in the two systems overshadow the usefulness of doing these analyses? In addressing this question, we consider the following thought experiment. Suppose the federal government was considering whether to discontinue VA mental health services, and crucial decision makers asked if data existed to support their decision. We feel that the data from this study, although flawed, would be more useful than no data at all. These data could reasonably support the statement that the VA does less well than the private sector, but that the differences are likely explained by differences in patient characteristics, and that VA performance is therefore roughly equivalent to that of the private sector. If in the VA system, for example, only 10 percent of patients made an outpatient visit in the 30 days following discharge and 40 percent were readmitted within 14 days of discharge, one might come to a more negative conclusion.
This study is useful in that it presents a rare comparison of the operation of two very different sectors of the U.S. health care system. In addition, it sheds light on how changes in the delivery of mental health services in the VA and private-sector systems may have affected the quality of care. This study is also useful in pointing out both the difficulties and the importance of comparing quality across service systems. As government health care systems are increasingly compared with private-sector firms, designing measures for comparing quality across service delivery systems, and for adjusting for differences in patient populations, should be a major priority for health services researchers.
Conclusions
Although methodological challenges, as illustrated here, are substantial, we found that the private sector modestly outperformed VA on most quality measures. However, VA improved markedly over time compared with the private sector, and it treats a more troubled population.
Footnote
The authors are affiliated with the Northeast Program Evaluation Center at the Connecticut-Massachusetts VA Mental Illness Research, Education, and Clinical Center, 950 Campbell Avenue, West Haven, Connecticut 06516 (e-mail, [email protected]), and the department of psychiatry at Yale University School of Medicine.
References
1.
Epstein AM: Performance reports on quality: prototypes, problems, and prospects. New England Journal of Medicine 333:57-61, 1995
2.
Health Plan and Employer Data Information Set, Version 2.5. Washington, DC, National Committee for Quality Assurance, 1995
3.
Druss BG, Rosenheck RA: Evaluation of the HEDIS measure of behavioral health care quality. Psychiatric Services 48:71-75, 1997
4.
Health Plan and Employer Data Information Set, Version 3.0. Washington, DC, National Committee for Quality Assurance, 1998
5.
Rosenheck RA, Cicchetti D: A mental health program report card: a multidimensional approach to performance monitoring in public sector programs. Community Mental Health Journal 34:85-106, 1998
6.
Rosenheck RA, DiLella D: Department of Veterans Affairs National Mental Health Program Performance Monitoring System: Fiscal Year 1998 Report. West Haven, Conn, Northeast Program Evaluation Center, 1999
7.
Pincus HA, Zarin DA, West JC: Peering into the "black box": measuring outcomes of managed care. Archives of General Psychiatry 53:870-877, 1996
8.
Mechanic D, Schlesinger M, McAlpine DD: Management of mental health and substance abuse services: state of the art and early results. Milbank Quarterly 73:19-55, 1995
9.
Frank RG, McGuire TG: Savings from a Medicaid carve-out for mental health and substance abuse services in Massachusetts. Psychiatric Services 48:1147-1152, 1997
10.
Goldman W, McCulloch J, Sturm R: Costs and use of mental health services before and after managed care. Health Affairs 17(2):40-52, 1998
11.
Ma C, McGuire T: Costs and incentives in a behavioral health carve-out. Health Affairs 17(2):53-69, 1998
12.
Leslie DL, Rosenheck RA: Inpatient treatment of comorbid psychiatric and substance abuse disorders: comparison of public sector and privately insured populations. Administration and Policy in Mental Health 36:253-268, 1999
13.
Rosenheck R, Horvath T: The impact of VA reorganization on patterns of mental health care. Psychiatric Services 49:53, 1998
14.
Leslie DL, Rosenheck RA: Changes in inpatient mental health utilization and costs in a privately insured population:1993 to 1995. Medical Care 37:457-468, 1999
15.
Leslie DL, Rosenheck RA: Shifting to outpatient care? Mental health care use and cost under private insurance. American Journal of Psychiatry 156:1250-1257, 1999
16.
Rosenheck RA, Fontana A, Stolar M: Assessing quality of care: administrative indicators and clinical outcomes in posttraumatic stress disorder. Medical Care 37:180-188, 1999
17.
SAS/STAT User's Guide, Vol 2, 1990, p 891
18.
Rosenheck RA, Seibyl CL: Homelessness: health service use and related costs. Medical Care 36:1256-1264, 1998
19.
Epstein AM, Stern RS, Weissman JS: Do the poor cost more? A multihospital study of patients' socioeconomic status and use of hospital resources. New England Journal of Medicine 322:1122-1128, 1990
20.
Salit SA, Kuhn EM, Hartz AJ, et al: Hospitalization costs associated with homelessness in New York City. New England Journal of Medicine 338:1734-1763, 1998
21.
Rosenheck R, Hoff R, Steinwachs D, et al: Benchmarking treatment of schizophrenia: a comparison of service delivery by the national government and by state and local providers. Journal of Nervous and Mental Disease, in press
Information & Authors
Information
Published In
History
Published online: 1 May 2000
Published in print: May 2000
Authors
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBGet Access
Login options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).