Effectiveness of Two Methods of Obtaining Feedback on Mental Health Services Provided to Anonymous Recipients

NYOMH selected eight sites to participate in the recipient-feedback pilot. These sites consistently served high numbers of individuals and differed from one another geographically, culturally, and ethnically. To explore whether offering a brief questionnaire affected rates of participation in a telephone interview, the evaluators then further divided these eight sites, counterbalancing demographic characteristics of service recipients and indicators of how well sites were functioning, into four sites to offer participation in a telephone interview only and four to offer each service recipient both the telephone interview and the brief questionnaire. Mount Sinai School of Medicine evaluators met with staff at each site to conduct training on the forms and procedures for approaching potential participants.

After measurement development and protocol approval by Mount Sinai School of Medicine's and NYOMH's institutional review boards, the study was conducted between January and April 2003. During this time, counselors at participating sites offered all English- and Spanish-speaking adults who used Project Liberty individual crisis counseling the opportunity to evaluate their service. If respondents agreed to a telephone interview, they could hand in completed (signed and sealed) permission-to-contact forms to their counselors to send on to evaluators, or they could complete forms later and mail them in. If respondents agreed to complete a brief questionnaire, they were given a copy to complete immediately or send in later. Training was provided to counselors so they would be familiar with the purpose of the evaluation and be able to answer any questions from recipients.

Researchers from the Mount Sinai School of Medicine, in consultation with NYOMH and with other evaluators, developed the permission-to-contact form, along with a 102-item, 45-minute telephone interview (for which participants received $20) and the 32-item, brief, ten-minute questionnaire. The permission-to-contact form gathered contact information for individuals who expressed a willingness to participate in a telephone interview. Telephone interview and brief questionnaire items were structured to reflect content used in prior surveys ( 3, 6 ), including reasons for contacting Project Liberty, satisfaction with services, extent of symptoms and functional impairment, demographic information, and referral to other services. The telephone interview included additional items designed to gather more detailed information in each of these domains. Counselors provided research staff with daily reports of the number of Project Liberty service recipients approached, how many declined to take a form (permission-to-contact or brief questionnaire), how many agreed to take a form, and how many returned a sealed form for the counselor to forward to NYOMH.

Descriptive statistics characterized response rates for permission-to-contact forms, the telephone interview, and the brief questionnaire and to examine responses to items evaluating Project Liberty. Chi square goodness-of-fit analyses were used to compare demographic information from the telephone interview and brief questionnaire with demographic information for all individuals using services at the participating Project Liberty sites between February and April 2003.

Because Project Liberty services were anonymous, evaluators determined the number of individuals who were offered the opportunity to provide feedback by asking counselors to report the number of individuals who were offered a permission-to-contact form and brief questionnaire. Compliance with the administrative task of reporting the number of individuals who were offered each form was uneven; hence, we used data from the five sites that were most adherent to calculating response and refusal rates—three sites offered only permission-to-contact forms to clients and two sites offered both permission-to-contact forms and the brief questionnaire—for chi square analyses to examine whether distributing the brief questionnaire appeared to affect response rates for the permission-to-contact form.

A total of 214 Project Liberty service recipients from eight sites completed permission-to-contact forms and agreed to be contacted regarding participation in the telephone interview. Of these, 134 (63 percent) returned the permission-to-contact form by mail, and 80 (37 percent) gave the form to the counselor. Of the 214 recipients, 153 (71 percent) completed the telephone interview; of these, 61 (40 percent) stated they had also completed a brief questionnaire. A total of 107 Project Liberty recipients from four sites returned the brief questionnaire; of these, 73 (68 percent) returned the brief questionnaire by mail, and the remainder gave it to their counselor. Completion of individual items ranged from 75 to 99 percent for the brief questionnaire and consistently more than 97 percent for the telephone interview.

Among the five sites most compliant in recording information on the number of clients approached to participate in the telephone interview, 974 service recipients were offered a permission-to-contact form. Of these, 155 (16 percent) forms were completed—92 (17 percent) of 536 at the three permission-to-contact-only sites compared with 63 (14 percent) of 438 at two sites where both the permission-to-contact form and the brief questionnaire were offered. Similarly, at the two sites most adherent to recording the number of clients who received the brief questionnaire, 441 service recipients were offered the brief questionnaire, and 63 questionnaires (14 percent) were completed.

Both the telephone interview and brief questionnaire sampled men and women, those whose preferred language was English or another language, and individuals of varying races and ethnicities in the proportions generally expected given the demographic characteristics of all individuals served by Project Liberty at sites participating in the study. Responses on the brief questionnaire and telephone interview regarding experiences after the World Trade Center attacks and satisfaction with Project Liberty services yielded similar responses, which suggested that the methods gathered comparable information ( Table 1 ).

This report describes the means used to evaluate anonymous mental health services as they were being developed and refined in response to the terrorist attacks of September 11, 2001, on the World Trade Center. Conclusions from this convenience sample must be taken as preliminary. Participation rates for the brief questionnaire and telephone interview were at best modest (less than 20 percent). However, among individuals who indicated a willingness to participate in a telephone interview, nearly three-quarters did so. Gender, racial and ethnic groups and reimbursement for time spent during the telephone interview were not associated with a greater likelihood of participating in either the telephone interview or brief questionnaire. Information obtained via each method was similar and demonstrated good response variability, and offering both methods did not adversely impact participation rates in either. These findings suggest that both methods represent feasible ways of obtaining feedback from anonymous service recipients who are willing to participate. That nearly two-thirds of participants chose to return their forms by mail underscores the utility of having multiple means of obtaining feedback.

Because responses to both methods were similar and individuals with different demographic characteristics did not seem to prefer one over the other, future investigators interested in obtaining satisfaction information may prefer the less expensive brief questionnaire, which cost $7 per returned survey. On the other hand, if investigators want additional information, such as ongoing symptoms and problems that individuals experienced after the attacks, they may need to conduct a more expensive telephone interview, which cost $190 per interview, including a $20 payment per participant in our evaluation. Given the challenge of obtaining participant evaluation while Project Liberty services were ongoing and service recipients were anonymous, we found the response rates disappointing but not surprising.

Although the low response rate was bad news for this evaluation, it served as an indicator that future evaluations need to incorporate methods to maximize response rates and to maintain provider adherence to administrative tasks critical to the evaluation. In addition, future providers of postdisaster services may consider the tradeoff between providing these services anonymously versus using unique identifiers that allow for more comprehensive service evaluation through time.

This evaluation was funded by grant FEMA-1391-DR-NY (titled "Project Liberty: Crisis Counseling Assistance and Training Program") to New York State from the Federal Emergency Management Agency. The Center for Mental Health Services of the Substance Abuse and Mental Health Services Administration administered the grant. The authors express their appreciation to Katherine Shear, M.D.

Dr. Covell and Dr. Essock are affiliated with the Department of Psychiatry, Division of Health Services Research, Mount Sinai School of Medicine, New York City. Dr. Essock is also with the Mental Illness Research, Education, and Clinical Center, James J. Peters Veterans Affairs Medical Center, the Bronx. Ms. Donahue and Mr. Felton are with the Center for Information Technology and Evaluation Research, New York State Office of Mental Health, Albany. Dr. Ulaszek and Ms. Dunakin are affiliated with the Department of Psychology, University of Connecticut, Storrs, and with the State of Connecticut Department of Mental Health and Addiction Services, Hartford. Send correspondence to Dr. Essock, Department of Psychiatry, Mount Sinai School of Medicine, 1 Gustave L. Levy Place, Box 1230, New York, NY 10029-6574 (e-mail: [email protected]). This report from Project Liberty is part of a special issue of Psychiatric Services commemorating the five-year anniversary of the September 11, 2001, attacks. Susan M. Essock, Ph.D., served as guest editor.