State Mental Health Policy: Less Is More: Virginia's Performance Outcomes Measurement System

An orientation meeting was conducted at each site to present the background, purpose, and expectations of POMS. Local workgroups were established, which included representation from direct service staff members, administrative and management staff members, consumers, and family members. Each location developed a site-specific protocol to guide the implementation of POMS and to ensure compliance with statewide standards. Finally, clinical staff members were trained in administering the assessments.

Initially, 11 standardized instruments were selected for the pilot project, not including the "satisfaction" instruments. Instruments were selected if they seemed to have the potential both to be clinically useful and to serve as good indicators of performance. Standardized assessments varied according to the populations that were being served at the pilot sites and included the Child and Adolescent Functional Assessment Scale (1), the Multnomah Community Ability Scale (2), the Addiction Severity Index (ASI) (3), the MHSIP Symptom Distress Scale (4), and the Specific Level of Functioning Scale (5). In addition, organizational and client data were collected from administrative records.

The Southeastern Rural Mental Health Research Center at the University of Virginia conducted an independent evaluation of the pilot project. Goals of the evaluation were to determine the feasibility of POMS, identify refinements that would enhance the efficiency and effectiveness of statewide implementation, and provide initial estimates of the overall costs. The evaluation involved both qualitative and quantitative methods, including focus groups, clinician ratings of instruments, and an analysis of the time that was associated with completing the various components of the pilot project. The results were used to identify potential problems, so that a refined system could be implemented statewide and so that difficulties that were likely to be encountered during the implementation would be anticipated, if not circumvented. Focus groups were conducted before and after the implementation of the pilot program with each site; with the Central Office of the Department of Mental Health, Mental Retardation, and Substance Abuse Services; and with consumers and family members. Focus groups were held at the national office of the National Alliance for the Mentally Ill in Northern Virginia; at the central office of the Virginia Department of Mental Health, Mental Retardation, and Substance Abuse Services in Richmond; and at locations in Fairfax, Suffolk, Roanoke, Hanover, Petersburg, Alexandria, and Martinsville.

A key component of the pilot test was to determine the perceived accuracy and clinical utility of the instruments. The instruments were rated by the clinicians at the time of intake, at three and six months after admission, and at discharge. To determine the amount of time that was associated with completing the components of the pilot project, time allocation sheets were completed monthly by persons who were involved in the administration, planning, training, data collection, report preparation, or any other aspect of the pilot project. A total of 838 individuals participated in the evaluation.

Audiotape recordings of focus groups were transcribed and analyzed by using QSR NUD*IST, a software package for theme coding and qualitative data analysis (6). QSR NUD*IST is an all-purpose qualitative data analysis system that was designed to aid researchers in handling nonnumerical unstructured data by supporting processes of indexing, searching, and theorizing. The analysis was inductive, meaning that concepts and themes emerged from the data throughout the process of data collection and analysis.

Acceptance. Perhaps most striking was the recognition among clinicians and administrators that greater accountability would be demanded from providers. Overall, participants accepted that a systematic collection of performance indicators was necessary and appropriate.

Problems. The problems with POMS tended to be logistical rather than ideological. Implementation problems occurred at all eight pilot sites. Staff members expressed concern that adequate time was not allocated for training on how to administer the instruments. They also expressed concern that the time required to implement POMS was burdensome to employees and problematic for those with brief tenures, such as students and interns. Some providers commented that time that they spent on POMS detracted attention from their clinical work. Communication problems among local POMS coordinators, management information system staff members, and clinicians were encountered by a majority of sites, including difficulty notifying clinicians and consumers when forms needed to be completed.

Providers reported that some consumers resisted being interviewed, some consumers were suspicious about the purpose of the interview, and some consumers felt coerced because their treatment was court ordered. Many providers complained about the length of time that was required for the interviews, perhaps resulting in insufficient effort to locate consumers for post-admission assessments.

Despite the national prominence of the standardized instruments, their use during the pilot project was controversial. Resistance was due in part to the belief that the instruments were not clinically useful. Focus groups commented that existing administrative data, such as length of stay and frequency of consumer contacts, would provide more useful information. Staff members were concerned that the instruments would be used to determine service eligibility and did not accurately reflect the day-to-day functioning of consumers. Some staff members felt that the instruments served to increase consumer anxiety, paranoia, and fear. An exception was the ASI, which was more accepted because it could be used as an intake interview and did not create much of an additional burden for consumers of substance abuse services.

Participants from all sites reported problems that were related to information technology. Most problems were related to a lack of resources, such as not having enough computer hardware and, especially, not having in-house technical expertise available. Clinicians rated the study instruments for clinical accuracy and utility. The clinicians were asked for ratings at intake, at three and six months after admission, and at discharge. The response rate was poor for the rating survey; only 25 clinicians completed the rating survey. Clinicians who responded tended to rate the accuracy and utility of the instruments as less than satisfactory. However, these ratings should be interpreted in light of the possibility of a negative halo effect. In addition, the poor response rate renders the ratings less reliable.

More individuals completed the time allocation sheets; 838 responses were collected from clinicians, administrators, project managers, trainers, and management information system personnel for up to four data points. The number of individuals who completed the sheets per site per data point ranged from three to 43, with a mean±SD of 23±15.7 individuals. Participants reported that they spent a total of 9,849 hours on POMS during the pilot project, ranging from 516 to 2,162 hours per site. The cost of implementing POMS was calculated by using standard wage rates. However, the costs varied from setting to setting, and estimated costs in other settings will depend on assumptions about what upgrades to existing data collection infrastructure will be needed and what marginal increases in staff time will be necessary.

The sites did not uniformly recruit consumers to take part in the pilot project. In particular, at some sites consumer participation was presented as being optional, and informed consent, or at least assent, was obtained. At other sites the standardized assessments were considered to be a routine part of the intake process and consent was not obtained. Whether consumer participation in POMS is voluntary or mandatory in future POMS efforts should be clarified to ensure consistency with the accepted standards of ethics of conducting service evaluations.

The software and hardware problems arose because of the variety of computing and software systems that evolved over time at the sites, making it difficult to address problems centrally. Uniformity did not exist from locality to locality, and resistance to adopting a uniform system arose, because the localities had made substantial investments in the existing systems. These problems could be solved by expanding existing hardware and software technology.

Suggestions. Participants reported that they were satisfied with the training, although it was time-consuming. Participants also reported that having a project coordinator was critical to the success of the project and that more time needed to be allocated for the coordinator. Furthermore, participants reported that there was a need for additional funding for such things as additional hardware and software and for the additional cost of data entry and training in the use of standardized instruments. The pilot project included a very detailed manual of consumer selection and data collection protocols that proved to be invaluable. This manual should be replicated in any implementation of a performance measurement system. It was also suggested that assessment instruments be normed at a fourth grade reading level.

Limitations. It should be noted that the pilot project involved only a single population at each site, either adults with serious mental illness, children and adolescents with serious emotional disturbances, or substance abusers. Full implementation of POMS would involve all populations that the state authority serves, and administering the complete set of measures would engender even greater burdens on the community mental health centers.