Implementing a Statewide Outcomes Management System for Consumers of Public Mental Health Services
Abstract
The authors describe the development and implementation of an outcomes management system designed to measure outcomes and processes of care for public mental health consumers in Arkansas. The public-academic project was implemented in 1995 and is based on the Shewhart-Deming model of continuous quality improvement. All 15 community mental health centers (CMHCs) in the state participate in the project, which prospectively measures longitudinal outcomes of care for the tracer conditions of major depression and schizophrenia. Multiperspective measurement tools are used to measure patients' psychiatric status and general health status at periodic intervals; information is gathered on functioning, symptoms, severity of illness, social factors, demographic characteristics, and quality of life. A problem encountered during implementation was the relatively low rate of referral of patients with the tracer conditions for monitoring. Voluntary rather than mandatory participation in the outcomes management system by the CMHCs as well as clinicians' misperceptions about the system's purpose and concerns about confidentiality may have partly accounted for the low rate.
Compelled by numerous forces such as national health care reform (1) and managed care (2), many states are undergoing reforms aimed at improving the organization and financing of public mental health services (3-5). Issues such as access to care, continuity of care, and outcomes of care are also recent concerns for many states (6-9). Although great interstate variation exists in accountability efforts (10), documenting and improving the effectiveness of mental health services is clearly an urgent issue that has affected not only private payers, but public payers as well.
In the past, techniques such as utilization management and utilization review have been the primary methods for system accountability, with an emphasis on utilization of services (11). However, the difficulty with using utilization review as the sole approach to assessing mental health services is the questionable assumption that service utilization is an adequate measure of treatment needs or effectiveness. In addition, utilization review typically examines only a few components of treatment—the location, amount, and, sometimes, type of care. Although utilization review was initially intended to improve the quality of care, it has resulted in the micromanagement of individual cases and often has left clinicians feeling frustrated that outside care reviewers unfamiliar with their patients are directing care (12).
Current methods of accountability are much more focused on demonstrating how treatment affects outcomes, and these approaches include an increased role for consumers' perspectives of functioning and well-being (13,14). For example, the state of Georgia has undergone a radical restructuring of its public mental health system, shifting from a centralized model of service delivery to one that is decentralized and consumer driven with regional boards whose membership is at least 50 percent consumers or family members (13). These new approaches to accountability, particularly the use of outcomes management systems, promise to go beyond a narrow focus on service utilization to provide valid and reliable information on what types of care are most effective for whom and in what settings and circumstances (15-19).
This paper describes a public-academic collaboration to develop a statewide outcomes management system for measuring outcomes and processes of care for public mental health patients at risk for hospital admission. The system was also designed to provide information, such as about patients' satisfaction with their living situations, that could be used to shape policies. Other goals of the system were to promote successful treatment modalities, such as adequate medication dosing for chronically depressed patients, and to enhance continuous quality improvement initiatives.
Very little research has focused on efforts to achieve large-scale statewide system accountability such as the initiative described here, nor has research adequately addressed the effectiveness of such systems of care (20). However, because of recent national trends to shift care from the hospital to the community (4,7) and local trends in Arkansas involving financial decentralization (21), questions about whether patients are receiving appropriate services and about the extent of these services are at the forefront of many stakeholders' agendas. It is hoped that the discussion of our approach, methods, data collection tools, and processes to facilitate continuous quality improvement, as well as issues encountered during implementation, will help others who are involved in clinical services research and accountability efforts in the public mental health system.
The outcomes management system
Background
Our working model for this considerable task relies heavily on the use of outcomes management consistent with the Shewhart-Deming conceptual model of continuous quality improvement (CQI) (22,23). This approach follows a cyclical process that begins with monitoring clinical performance to identify problems that influence clinical practice patterns and the causes of these problems. Once problems are identified, practice modifications can be recommended and introduced, and the results assessed. Most important, by using the CQI process, information can be fed back to clinicians and administrators to improve a system's clinical performance while also addressing issues of accountability.
Key collaborators in this public-academic venture to develop a statewide outcomes management system include the Health Care Financing Administration and the Arkansas Division of Mental Health Services (DMHS), agencies that both set policies and provide payments; the Center for Mental Healthcare Research, an academic institution that provided the implementation project team; and Arkansas community mental health centers (CMHCs), the provider agencies. The project was conceptualized in the winter of 1994 and implemented the following summer, in 1995.
The involvement of senior organizational leaders, particularly from the DMHS in our case, has been noted as a critical factor in the success of an outcomes management system (17). To facilitate this involvement, before implementation of the system, the project team presented the proposed project to the executive directors of the CMHCs at a regular meeting in the spring of 1995. Voluntary support was necessary from each center director because CMHCs operate as private nonprofit organizations in Arkansas and are not under the direct administrative control of the DMHS.
Clinician support, also an important element of success (17), was fostered in the summer of 1995 by two days of intensive training for all CMHC clinicians participating in the project. The training was given by the project team from the Center for Mental Healthcare Research at a central site to allow the greatest attendance possible. Site-specific inservice training was provided on request, and the project team visited the facility to train CMHC clinicians to tailor data collection to their specific circumstances. Throughout the project, telephone consultation has been provided to CMHC clinicians as needed. The executive director at each CMHC selected a clinical staff member to provide additional training to all clinicians and to coordinate patient enrollment and data collection efforts throughout the project.
The scope and impact of this project is quite large in terms of dollars devoted to services and numbers of providers and patients. For example, although state and federal funds are not the only source of income for CMHCs in Arkansas, approximately $20 million of such funds were disbursed to the CMHCs in fiscal year 1996. Slightly more than 10 percent of this budget (approximately $2.6 million) was earmarked for alternatives to state hospitalization. All 15 CMHCs and their on-site staff throughout the state are involved in the project. In 1996, with a state population of 2,225,000, approximately 60,000 consumers received some type of CMHC service in Arkansas.
Project implementation costs are difficult to estimate, and costs of implementing such a project will vary greatly among different systems depending on the technology and resources that are available. Cost of maintaining the Arkansas outcomes management system will range from $100,000 to $175,000 a year, which is less than 1 percent of the state and federal funds spent annually in the CMHC system.
Some factors that would result in lower costs include care providers' collecting data from consumers at the point of service versus the use of special project assistants, less dependency on outside professional interpretation of data, less reliance on sources outside of the CMHC for providing technical support such as data entry, and sampling only as many consumers as are needed to provide an accurate appraisal of outcomes for the system.
System design
A disorder-specific approach to sampling and assessment was used to examine how patient outcomes relate to treatment provided within the state's public mental health system. Consumers at CMHCs were selected to be monitored by the outcomes management system based on their psychiatric diagnosis. An assessment tool specifically designed to measure the salient aspects of their disease and its treatment was used.
This disease-specific approach, which is based on the tracer methodology, has been favored by others evaluating mental health services (17,24), primarily because it provides relevant clinical and prognostic information for a particular disorder (tracer). Also, it uses specific tracers as an indicator of the overall quality and effectiveness of the system delivering that care (25,26); thus this approach is more economical than one that monitors every condition and every patient.
For several reasons, schizophrenia and depression were chosen as tracer conditions in this outcomes management system to assess the quality and effectiveness of our public mental health services. Schizophrenia and depression have been rated as areas of primary professional concern by those involved in public-academic health linkages (27). Depressive disorders in mental health settings are common (28). In Arkansas the prevalence of schizophrenia and depression is quite high, as is the use of services by persons with these disorders compared with patients who have other diagnoses. For example, a 1994 study using a statewide database of patient characteristics and service use in Arkansas CMHCs showed that individuals with schizophrenia or with depression received 46.8 percent of all CMHC services provided (Zhang M, Smith GR, Heithoff K, et al, unpublished manuscript, 1996). Finally, the economic and social burdens associated with both of these disorders (29-32) make accountability and quality improvement efforts imperative.
Procedures
Recruitment of patients for monitoring and evaluation by the outcomes management system began in August 1995 and ended in December 1996. To be eligible, individuals had to have a diagnosis of major depression or schizophrenia, be 18 years of age or older, and be at risk for hospitalization, as initially determined by a clinician. Patients from each of the state's 15 CMHCs were considered for the depression tracer condition. However, a necessary component of our monitoring approach for patients with schizophrenia was a periodic face-to-face interview; given the costs and constraints involved in such interviews, patients with schizophrenia seen only in the eight CMHCs in central Arkansas were targeted for monitoring by the system.
To examine a relatively homogeneous patient population in terms of diagnosis, individuals with depression who were also known to have severe borderline personality disorder, antisocial personality disorder, or bipolar disorder were excluded from monitoring. Similarly, for the tracer condition of schizophrenia, individuals with schizoaffective disorder, organic psychosis, bipolar disorder, or depression with psychotic features were excluded.
Clinicians at each CMHC selected patients who met the above criteria. After appropriate diagnostic screening and patient consent, clinicians and patients completed the baseline portion of the Depression Outcomes Module (33) or the Schizophrenia Outcomes Module (34). These instruments measure disorder-specific outcomes and prognostic case-mix characteristics as well as general health status, functioning, social factors, demographic characteristics, quality of life, and processes of care (for example, quantity and type of services). The modules are designed to measure outcomes from multiple perspectives and from several sources, including consumers' self-reports, clinicians' ratings, and medical records. They are also relatively brief and feasible to implement in routine mental health care settings (26). To ensure patient privacy, the data collected for this project were secured as if they were research data, so they could not be readily linked to individual patients.
Collection of follow-up data
The protocol for the outcome management system called for collecting follow-up data for patients with depression every four months after baseline for up to one year, or until the depressive disorder resolved, and for patients with schizophrenia every six months after baseline for one year. The follow-up times were chosen based on clinical experience and on literature indicating that they would be reasonable periods for observing clinically meaningful changes (35,36).
For individuals with depression, the follow-up forms were mailed to patients, administered by phone, or completed at the CMHC. For those with schizophrenia, face-to-face follow-up interviews were conducted by a trained research assistant at the CMHCs. The CMHCs provided temporary unused space for the interviews. They also helped locate patients and plan specific times for follow-up interviews.
Medical record review, one of the components of the modules, was performed after the last follow-up by trained research assistants. Diagnoses initially made by CMHC clinicians were verified using a clinician DSM-IV checklist diagnostic verification form, and any further questions about the diagnosis were resolved by a review of the patient's medical records by a research psychiatrist. Additional information collected by chart review included quantity and type of services utilized within the public mental health system, such as day treatment, or utilized elsewhere, such as emergency room services. Information on psychotropic medications prescribed was also obtained. Follow-up of service utilization ended in December 1997, one year after the final recruitment month.
Feedback for CQI
Perhaps the most important part of an outcomes management system is the feedback of information for continuous quality improvement. Not surprisingly, CQI is becoming a major goal in state mental health systems (37). Our approach to CQI consisted of a twofold strategy. The first step involved a meeting between the project team and the CMHC directors to mutually decide how data could best be reported. The second step involved disseminating regular reports to participating stakeholders, starting in July 1996, so that they could conduct site-specific CQI initiatives, especially feedback of information to clinicians.
These regular reports provide a written and graphical summary of primary findings, including patient characteristics, changes in outcomes over time, and mental health services received. The reports are sent bimonthly to CMHC executive directors with a request for input on their concerns about patient care, managed care, and state policies. Regular reports are also sent to the director of the DMHS, the Arkansas Alliance for the Mentally Ill, and the Arkansas Mental Health Council.
When enough data are gathered, all individuals and groups that have been receiving regular reports will receive reports that will include comparisons between large subgroups of patients and analyses of patient and treatment characteristics associated with improved outcomes. It is anticipated that these reports will be ready by mid-1998.
Issues in implementation
Others might benefit from a discussion of several issues related to our experiences in implementing a statewide outcomes management system in the public mental health setting. First, we believe a strength of our approach is that multiple perspectives and sources of outcome are taken into account over time and that all parties invested in the process of public mental health are represented. The people who can make a difference in improving the quality and effectiveness of care—consumers, clinicians, consumer advocates, DMHS administrators, CMHC directors, and other state policy makers—are all involved in the project. In addition, our disorder-specific design and measurement tools are economical and incorporate critical components of case-mix that provide sufficient differential data on which to base sound policy.
A limitation of our outcomes management system is that we have obtained fewer referrals for monitoring than were expected, given the number of known cases of schizophrenia and depression in the state. Although the limited number of referrals is a typical problem encountered in large-scale "real world" evaluations of this type (38), the reasons for the low referral rate are not entirely clear. As in other studies of outcomes management systems, it is possible that referral rates are directly correlated with the number of clinicians at each site who are designated to make referrals and the number of staff in the outcomes management system responsible for data collection (18).
However, other factors accounting for the relatively low number of referrals might include the stringent inclusion criteria, the fact that cooperation by CMHCs was voluntary, and concerns about confidentiality by both providers and consumers. Privacy in outcomes management systems is an ongoing concern. Although individual providers were not specifically monitored in this project, clinicians might have been hesitant to volunteer information they perceived could be used against them.
The CMHCs had no financial front-end investment in the outcomes management system and were strictly voluntary participants. Each CMHC agreed to designate time to enter a patient in the project, amounting to about one hour per participant. The CMHC also provided medical records so that service utilization data could be obtained. However, even though the project was described as an outcomes management system from the start, some CMHC staff members perceived that the data gathering was for academic research. This perception may have given the project lower priority in the CMHC system than if it had been viewed as a practical CQI tool. In light of these observations, it is clear that for an outcomes management system to operate to capacity, providers must value and incorporate it into the service delivery system as a necessary means of measuring the effectiveness of treatment. Research is needed to identify specific techniques for gaining providers' cooperation.
Limitations aside, we suggest that it is feasible to implement an outcomes management system in the public mental health setting, despite common perceptions noted by others that public mental health systems are complex and turbulent systems that do not support research efforts (39). The application of clinical services research and outcomes management systems, such as the one described here, will become an increasingly important issue for political and clinical leaders in public mental health settings to consider.
Particularly as competition in the health care industry continues to intensify, public mental health systems must come to terms with these issues to survive. Not only do such state-of-the-art projects add to our current knowledge about processes of care and effectiveness of treatment, but they also provide guidance in solving practical problems such as case-mix adjustment, outcomes specificity, and quality improvement efforts. Thus they help reduce the burden on public mental health systems that many states currently face.
Footnote
The authors are affiliated with the National Institute of Mental Health's Center for Mental Healthcare Research and the department of psychiatry at the University of Arkansas for Medical Sciences, 4301 West Markham Street, Slot 589, Little Rock, Arkansas 72205. Dr. Booth, Dr. G. Richard Smith, and Ms. Nordquist are also affiliated with the Health Services Research and Development Field Program of the Department of Veterans Affairs.
References
1.
Ross EC, Mazade NA: The Clinton proposed health care reform: implications for persons with serious mental illness. Administration and Policy in Mental Health 21:251-261, 1994
2.
Essock SM, Goldman HH: States' embrace of managed mental health care. Health Affairs 14(3):34-44, 1995
3.
Cuffel BJ, Snowden L, Green RS, et al: The California adult performance outcome survey: preliminary evidence on reliability and validity. Community Mental Health Journal 31:425-436, 1995
4.
Dickey B, Cohen MD: Changing the financing of state mental health programs: using carrots, not sticks, to improve care. Administration and Policy in Mental Health 20:343-355, 1993
5.
Gilchrist LD, Allen DG, Brown L, et al: A public-academic approach to designing a state mental health program evaluation. Evaluation and Program Planning 17:53- 61, 1994
6.
Glover R, Petrila J: Can state mental health agencies survive health care reform? Hospital and Community Psychiatry 45:911- 913, 1994
7.
Rapp CA, Moore TD: The first 18 months of mental health reform in Kansas. Psychiatric Services 46:580-585, 1995
8.
Reinig SP, Hawley R: Realigning local-state relations: grassroots mental health restructuring in the Pacific Northwest. Administration and Policy in Mental Health 22:233-245, 1995
9.
Brown L, Cox GB, Jones WE, et al: Effects of mental health reform on client characteristics, continuity of care, and community tenure. Evaluation and Program Planning 17:63-72, 1994
10.
Davidson H, Schlesinger M, Dorwart RA, et al: State purchase of mental health care: models and motivations for maintaining accountability. International Journal of Law and Psychiatry 14:387-403, 1991
11.
Tischler GL: Utilization management and the quality of care. Hospital and Community Psychiatry 41:1099-1102, 1990
12.
Eddy DM: Clinical decision making: from theory to practice. JAMA 263:287-290, 1990
13.
Elliott RL, Cohen MD, Evans DL: Reforming Georgia's mental health system. Community Mental Health Journal 31:413-423, 1995
14.
Wells KB, Stewart A, Hays RD, et al: The functioning and well-being of depressed patients: results from the Medical Outcomes Study. JAMA 262:914-919, 1989
15.
Ellwood PM: Outcomes management: a technology of patient experience. New England Journal of Medicine 318:1549- 1556, 1988
16.
Flynn L, Steinwachs D: Outcomes roundtable includes all stakeholders. Health Affairs 14(3):269-270, 1995
17.
Smith GR, Fischer EP, Nordquist CR, et al: Implementing outcomes management systems in mental health settings. Psychiatric Services 48:364-368, 1997
18.
Steinwachs DM, Wu AW, Skinner EA: How will outcomes management work? Health Affairs 13(3):153-162, 1994
19.
Wagner J, Gartner CG: Highlights of the 1995 Institute on Psychiatric Services. Psychiatric Services 47:15-20, 1996
20.
Cutler DL, Bigelow D, McFarland B: The cost of fragmented mental health financing: is it worth it? Community Mental Health Journal 28:121-133, 1992
21.
Cuffel BJ, Wait D, Head T: Shifting the responsibility for payment for state hospital services to community mental health agencies. Hospital and Community Psychiatry 45:460-465, 1994
22.
Deming WE: The New Economics for Industry, Education, Government. Cambridge, Massachusetts Institute of Technology, Center for Advanced Engineering Study, 1993
23.
Shewhart WA: The Economic Control of Quality of the Manufactured Product. Milwaukee, ASQC Quality Press, 1980
24.
McGlynn EA, Norquist GS, Wells KB, et al: Quality-of-care research in mental health: responding to the challenge. Inquiry 25:157-170, 1988
25.
Kessner DM, Kalk CE, Singer J: Assessing health quality: the case for tracers. New England Journal of Medicine 288:189-194, 1973
26.
Smith GR, Rost KM, Fischer EP, et al: Assessing the effectiveness of mental health care in routine clinical practice: characteristics, development, and uses of patient outcomes modules. Evaluation and the Health Professions 20:65-80, 1997
27.
O'Hare T, Rodriguez R, Earls E: The current status of academic-mental health linkages in Rhode Island. Community Mental Health Journal 31:209-214, 1995
28.
Rost KM, Smith GR, Burnam MA, et al: Measuring the outcomes of care for mental health problems: the case of depressive disorders. Medical Care 30(suppl 5):266-273, 1992
29.
Health care reform for Americans with severe mental illness: report of the National Advisory Mental Health Council. American Journal of Psychiatry 150:1447-1465, 1993
30.
Eckett S: A commentary: measuring the health economics of depression. Human Psychopharmacology 10:33-37, 1995
31.
Klerman GL, Weissman MM: The course, morbidity, and costs of depression. Archives of General Psychiatry 49:831-834, 1992
32.
Cuffel BJ, Jeste DV, Halpain M, et al: Treatment costs and use of community mental health services for schizophrenia by age cohorts. American Journal of Psychiatry 153:870-876, 1996
33.
Smith GR, Burnam MA, Burns BJ, et al: Outcomes Module for Major Depression. Little Rock, University of Arkansas for Medical Sciences, 1994
34.
Cuffel BJ, Fischer EP, Owen RR, et al: An instrument for measurement of outcomes of care for schizophrenia: issues in development and implementation. Evaluation and the Health Professions 20:96-108, 1997
35.
Prien RF, Carpenter LL, Kupfer DJ: The definition and operational criteria for treatment outcome of major depressive disorder: a review of the current research literature. Archives of General Psychiatry 48:796-800,1991
36.
Keitner GI, Ryan CE, Miller IW, et al: Recovery and major depression: factors associated with twelve-month outcome. American Journal of Psychiatry 149:93-99, 1992
37.
Evans ON, Faulkner LR, Hodo GL, et al: A quality improvement process for state mental health systems. Hospital and Community Psychiatry 43:465-469, 1992
38.
Cox GB, Allen DG, Brown L, et al: Recommendations for state-wide evaluations: lessons relearned. Evaluation and Program Planning 17:97-101, 1994
39.
Hogan MF: Serving the seriously mentally ill: public-academic linkages in services, research, and training, in Public Mental Health Systems and Research. Edited by Wohlford P, Myers HF, Callan JE. Washington, DC, American Psychological Association, 1993
Information & Authors
Information
Published In
History
Published in print: February 1998
Published online: 1 April 2006
Authors
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBLogin options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).