Relationship of Youth Satisfaction With Mental Health Services and Changes in Symptoms and Functioning

Managed care companies and public-sector agencies commonly use consumer satisfaction as an indicator of the quality of mental health care. Indeed, consumer satisfaction may be the most highly valued indicator of the effectiveness of services provided by behavioral health managed care organizations (1).

However, little is known about the meaning of consumer satisfaction, specifically the extent to which it is associated with changes in patients' clinical functioning—that is, symptom severity or functional impairment. The limited available research on youth services does not support a strong relationship between consumer—parent or youth—satisfaction and improvements in clinical functioning (1,2,3). Preliminary research on correlates of adolescents' satisfaction with mental health services suggests that consumers' attitudes and characteristics at service entry—for example, the severity of symptoms—may be stronger predictors of satisfaction than clinical change or characteristics of the services received (1,2).

Studies of consumer satisfaction are often criticized for methodologic weaknesses, including low participation rates and the associated sampling bias, use of positively biased outcome and satisfaction data with low variance, and use of psychometrically unsound measures of satisfaction (4,5). The study reported here addressed these weaknesses by including a random sample of youths from a large public system of care and using a psychometrically sound satisfaction measure. In addition, established measures of symptom severity and functional impairment were administered prospectively to multiple informants to assess clinical change. The goal of the study was to examine how youth-reported satisfaction with specialty mental health services was associated with clinical change on the basis of parent, youth, and interviewer assessment.

Methods

The study sample comprised 369 youths who were aged between 11 and 20 years at two-year follow-up, with a mean±SD age of 16.2±2.3 years, representing a subsample of the Patterns of Care (POC) study in San Diego, California (6). The POC study is a longitudinal study assessing the need for, use of, and outcomes of mental health services for high-risk youths recruited from public service sectors. The entire POC study sample included 1,715 youths aged six to 18 years who were randomly sampled from the following public service sectors: child welfare, juvenile justice, mental health, special education, and alcohol and drug abuse. The study reported here included only youths who reported receiving specialty mental health services during the two-year interval between baseline and follow-up interviews, those who spoke English, and those who were aged at least 11 years at follow-up.

In this subsample of 369 youths, 234 (63 percent) were male, and the racial or ethnic distribution was 45 percent Caucasian (165 youths), 25 percent Latino (92 youths), 16 percent African American (60 youths), 10 percent mixed or other (38 youths), and 4 percent Asian or Pacific Islander (14 youths). The median annual household income was $20,000 to $25,000. These youths received a variety of publicly funded specialty mental health services, primarily from community-based outpatient clinics or individual fee-for-service providers who accepted MediCal. The study was approved by the appropriate institutional human subjects protection committees. Further details of sampling design, interviewer training, and reliability checks have been published previously (6).

Youths and their caregivers were interviewed individually, usually in their homes, at baseline (between October 1997 and February 1999) and two years later. The symptom and functioning measures were administered at baseline and at follow-up, whereas the consumer satisfaction measure was administered at follow-up only. The following measures were used: the parent and youth report versions of the Child Behavior Checklist (CBCL) (7), the parent and youth report versions of the Columbia Impairment Scale (CIS) (8), the Children's Global Assessment Scale (C-GAS) (9), and the Multidimensional Adolescent Satisfaction Scale (MASS) (2,4).

The parent and youth report versions of the CBCL are well-established, comprehensive measures of children's emotional and behavioral problems. The reliability and validity of these measures are well documented (7). With the CIS, parents and youths each complete the 13-item self-report measure to assess functional impairment. This global impairment measure has strong psychometric characteristics (8). The C-GAS assesses global functional impairment. Interviewers assign C-GAS ratings after completion of interviews. Finally, the MASS is a 21-item scale assessing adolescents' satisfaction with mental health services. The test-retest and internal reliability of the total scale, as well as the empirically derived subscales have been established—for example, test-retest reliability for a two-week interval=.88, Chronbach's alpha=.91. The convergent, discriminate, and predictive validity of the scale are supported by previous studies (2,4). The subscales include counselor qualities (perceived competence of the clinician and quality of the relationship), meeting needs (perceived adequacy of quantity and type of service received), effectiveness (perceived benefit of service), and counselor conflict (experience of negative interactions with counselor).

To examine the association between consumer satisfaction and changes in symptoms, we performed separate multiple regression analyses for each symptom measure (parent and youth versions of the CBCL and CIS and the C-GAS). In these analyses, we treated the posttest score as the dependent variable and entered the pretest score, followed by satisfaction score, as independent variables (10). This procedure allowed us to determine whether satisfaction contributed significantly to the prediction of posttreatment scores beyond pretreatment score alone. We also examined simple correlations between difference scores on the clinical outcomes and satisfaction scores, and the pattern of results was very similar. However these results are not presented because of controversy over the use of difference scores (10).

Results

Youth-reported satisfaction with services was generally positive: the item mean±SD was 2.92±.57 on a scale of 1 to 4, with higher scores indicating greater satisfaction. This finding is consistent with those of previous studies of consumer satisfaction among adolescents (4). Youths' global satisfaction was significantly associated with change on all youth-reported symptom and functioning measures. Several satisfaction subscales were also associated with youth-reported change, as can be seen in Table 1. However, youth satisfaction was generally unrelated to change on parent- and interviewer-reported symptom and functioning measures, with one exception: scores on the "meeting needs" subscale were associated with parent-reported change in functional impairment. As can be seen in Table 1, the magnitude of effects were small (none above 4.2 percent). The table lists standardized betas for significant associations between satisfaction and symptom and functioning outcomes and change in R²—that is, the incremental variance explained by adding satisfaction as a predictor, beyond that accounted for by the pretreatment score alone.

Discussion and conclusions

We found that youths' satisfaction with the mental health services they received was minimally related to changes in mental health symptoms and functional impairment. Significant associations were generally limited to the youth's perspective and thus may have been due to shared informant variance. Youths' global satisfaction was not significantly related to any parent- or observer-reported change in symptoms or functioning. Our results are consistent with those of the only other study that examined these relationships among youths (1).

Overall, our findings suggest that consumer satisfaction should not be used as an indicator of effectiveness in reducing symptoms or improving functioning among youths. Of course, satisfaction with services may be associated with some other aspect of service use, engagement, or impact, or it may be a valuable outcome in its own right. Alternatively, as some research has suggested, consumer satisfaction may be largely predetermined at service entry by baseline characteristics—for example, severity of symptoms and expectations about services (2). The meaning—and thus the usefulness—of the satisfaction construct remains relatively unclear. As such, future investigations of predictors and correlates of consumer satisfaction are warranted.

All authors are affiliated with the Child and Adolescent Services Research Center of the Children's Hospital and Health Center in San Diego, California. Dr. Garland, Dr. Aarons, and Dr. Hough are also with the department of psychiatry at the University of California, San Diego. Dr. Hawley is also with the department of psychology at San Diego State University. Send correspondence to Dr. Garland at 3020 Children's Way, MC 5033, San Diego, California 92123 (e-mail, [email protected]).

Enlarge table