Reliability of the Services Assessment for Children and Adolescents
Abstract
OBJECTIVE: This study examined the test-retest reliability of a new instrument, the Services Assessment for Children and Adolescents (SACA), for children's use of mental health services. METHODS: A cross-sectional survey was undertaken at two sites. The St. Louis site used a volunteer sample recruited from mental health clinics and local schools. The Ventura County, California, site used a double-blind, community-based sample seeded with cases of service-using children. Participating families completed the SACA and were retested within four to 14 days. The reliability of service use items was calculated with use of the kappa statistic. RESULTS: The SACA- Parent Version had excellent test-retest reliability for both lifetime service use and previous 12-month use. The SACA also had good to excellent reliability when administered to children aged 11 and older for lifetime and 12-month use. Reliability figures for children aged nine and ten years were considerably lower for lifetime and 12-month use. The younger children's responses suggested that they were confused about some questions. CONCLUSIONS: This study demonstrates that parents and older children can reliably report use of mental health services by using the SACA. The SACA can be used to collect currently unavailable information about use of mental health services.
An estimated 14 to 25 percent of children in the United States meet criteria for a psychiatric diagnosis and are functionally impaired, yet many do not receive mental health services (1,2,3,4,5,6,7). Furthermore, little information is available about the few services that they do receive (8,9,10,11).
The literature on mental health services utilization by children has identified several issues that have yet to be thoroughly investigated. First, in order to examine the predictors of onset and cessation of service use, data must be collected in meaningful units (12). Episodes of care have been used for this purpose, and they are considered to be the boundaries for summing all inputs to the care process by specifying the beginning and ending points of treatment and the course of services used during treatment (13). The Health Insurance Experiment identified the potential importance of using episodes of care as the unit of measure because the investigators noted that different features were related to first visits for service than to subsequent visits (14,15,16). However, few studies have examined episodes of care or their applicability to children's mental health problems.
Second, despite the documented need for additional or more effective mental health services (17,18), exploration of the pathways into, between, and out of mental health services has been inadequate (9,11,19,20,21,22). Only the study by Farmer and colleagues (23) has directly assessed pathways.
Third, instruments used to measure service use need to address the intensity, content, and coordination of care. A number of studies have concluded that coordination of services is necessary but not sufficient for better outcomes (22,24,25,26,27,28). The results of these studies, along with those of studies that examined the effectiveness of protocol-driven services, point to the need to understand both the intensity and the content of care received (29,30).
Finally, few studies have examined how racial identity, acculturation, and cultural mistrust affect help-seeking patterns, yet these factors are important for the development of any type of measurement (31). Programs for children who have serious emotional disturbances need to address the unique cultural characteristics of these children, their families, and their communities. Several studies have found that ethnic minorities underuse particular services, even when service use barriers related to income and availability are reduced or eliminated (32,33,34,35,36,37,38).
We evaluated currently available instruments to determine whether they could be used to obtain key information about children's use of mental health services. We found that the service-use questions used in the Methods for the Epidemiology of Child and Adolescent Mental Disorders study, which was sponsored by the National Institute of Mental Health, could not identify episodes, provide any information on pathways, or differentiate the provider of care from the service setting (8).
The Child and Adolescent Service Assessment (CASA) has demonstrated good to excellent parent and child three-month test-retest reliability (39,40). For parent data, the kappa coefficient was .82 for any inpatient service use, .82 for any outpatient service use, and .62 for use of any school services. Its validity was similar, with a kappa of .84 for any service use. However, in the form that we reviewed, it did not examine the most recent episode of care, it could not distinguish the type of provider from the service setting, and it did not examine pathways into care.
The Services for Children and Adolescents-Parent Interview, which was developed for use in the Multisite Treatment Study for Attention Deficit Hyperactivity Disorder, has no youth version, and it has a three-month time frame. Therefore, it cannot address either lifetime or one-year use.
The Referral Sequence and Problems Interview details the process of entry into mental health services (41); however, it is not designed to distinguish the type of service from the provider and setting or to assess the duration, intensity, or content of services received.
Because we found no instrument that could assess all of the underinvestigated issues identified in the literature review, we constructed a new one by modifying the CASA and adding elements from other instruments. Our instrument, the Services Assessment for Children and Adolescents (SACA), was constructed in modules by service setting and structured such that, with longitudinal assessment over defined, consistent time periods, episodes of care could be identified.
The analyses reported here describe the formal reliability testing of the SACA. Data on its validity, parent-child agreement, and content of care have been published elsewhere (42,43).
Methods
In preparation for a large national study of children's mental health, we examined the reliability and validity of the SACA. Data were collected in St. Louis, Missouri, by investigators from the department of psychiatry at Washington University and in Ventura County, California, by investigators from the University of California, Los Angeles (UCLA), and Rand. The study took place in 1996.
Sample
St. Louis site. The clinical subjects for the study were recruited from a list of children aged 4 to 17 years who had been treated at the inpatient and outpatient clinics (N=88 and 443, respectively) of a local hospital within the previous 12 months. Children who had a diagnosis of mental retardation were not included in the study. Additional subjects, who were not service users, were recruited from a list of children who attended public schools and a list of children who attended a local day care facility (N=320).
A letter was sent to the parents of all children in both the clinical and nonclinical groups. The parents were invited to participate in the study by returning an enclosed response card that would allow a research assistant to contact them to explain the study. To meet our goal of approximately half of the study participants being African American, 26 families were recruited by asking African-American participants to identify friends and neighbors to contact.
A total of 146 parent-child dyads completed the test and retest interviews. Most of the parents were female (91 percent), and most were the child's biological mother (85 percent). Seventy-five parents (51 percent) were European American, and 70 (47 percent) were African American. Ninety-two parents (63 percent) had more than a high school education. By design, the children ranged in age from nine to 17 years.
UCLA site. A list of addresses of households with children aged 4 to 17 years who had used mental health services within the previous 12 months was obtained from the Ventura County Mental Health management information system. Members of the households were invited to participate and were screened for language and ethnic eligibility. For each of the eligible households, an attempt was made to randomly recruit from the surrounding neighborhood a household matched for ethnicity and children's age group in which no child had used mental health services during the previous 12 months.
A total of 470 households in which children had used services were invited to participate. Of these, 159 (34 percent) were eligible, and 94 (59 percent) agreed to participate. A total of 527 households in which children had not used services were invited to participate; of these, 108 (20 percent) were eligible, and 51 (47 percent) agreed to participate. A total of 106 parent-child dyads completed the test interview, and 91 dyads completed the retest interview. Like their St. Louis counterparts, participants were mostly female (90 percent) and were typically the child's biological mother (82 percent). Fifty-five (60 percent) were European American, and 25 (27 percent) were Hispanic; two were in the category "other," and seven were biracial. Just over half (47 participants, or 52 percent) had more than a high school education.
Procedure
At test and retest the SACA was administered separately to parents and to children in their homes by trained lay persons. Retests were conducted within one week, with a range of four to 14 days. Randomly selected interviews were audited during the interview process. Interviewers were blinded by having each test and retest conducted by a different interviewer who was not informed of any previous responses. The instrument took between two minutes and 45 minutes to administer. The initial module (see below) took two to 15 minutes.
Instrument
The SACA-Parent Version, in the form tested, begins with a module that gathers data on a child's lifetime and past-year use of 30 service settings that are grouped into three broad areas: inpatient, outpatient, and school. Following the initial module are 30 individual modules that ask more specific questions about each of the settings. The initial module can be used independently or in conjunction with the individual modules. Because the instrument was developed for use across the United States, general service setting terminology is used. To avoid an excessively long instrument, the amount of information requested varies; only specialty mental health settings and school counseling are questioned in detail.
Queries about reasons for seeking treatment for the child are designed to pinpoint specific problems rather than to elicit global descriptions (for example, depression) or life events (for example, a divorce). Questions about treatment are also specific and ask respondents about child-related interventions, such as therapy or counseling, medications, case management, and evaluation or testing, as well as family-oriented services. The temporal order in which the different service types were used in the past 12 months is then ascertained, and the modules are arranged in that order to facilitate the unfolding of the respondent's story.
The SACA also contains two specialized modules. The final services module is asked of everyone, regardless of whether they reported using services. For respondents who report the use of a service, a series of questions ascertains the most and least beneficial service, whether additional services were felt to be needed, and, if so, why any additional services were not used. If no service use is reported, respondents are asked whether they thought services were needed, the reason for needing services, and why services weren't used.
The other specialized module is the medication module, which was designed for parent report only. It elicits information about prescriptions for psychoactive medications, including dosage and duration of use, and insurance coverage for any psychoactive medication.
Analysis
All statistical analyses were conducted with SAS (version 6.12). After the data were carefully reviewed to identify any out-of-range values or inconsistencies, the test-retest analyses were undertaken. As is customary in epidemiologic investigations of mental health issues, kappa statistics were calculated to evaluate the agreement between categorical data and to correct for chance associations (44,45,46). Kappa values over .75 represent excellent agreement, values between .40 and .75 indicate good agreement, and values less than .4 indicate poor agreement (47,48). The reliability of continuous data was evaluated with the intraclass correlation coefficient with the same agreement categories (49,50).
Results
Of the 237 adult respondents (all test and retest pairs), 164 (69 percent) reported at least one occasion of service use at some time during the child's lifetime; 73 (31 percent) reported no service use. The majority of reported service contacts were in an outpatient setting. In the first test, 360 of the reported 591 service contacts (61 percent) were in outpatient settings, and of these, 240 were in a mental health clinic or a private practitioner's office (67 percent). In the retest, 342 of the reported 570 contacts were in outpatient settings (60 percent), with 231 (68 percent) taking place in a mental health clinic or a private practitioner's office. School-based services accounted for only 165 and 160 (28 percent) of the reported service settings in both tests.
The same pattern was seen in service use by setting, with outpatient settings accounting for the majority of service settings—232 of the 381 contacts reported during the first test (61 percent), and 226 of the 382 contacts reported during the retest (59 percent).
Reports of the type of service use settings did not vary significantly between the first test and the retest for either lifetime or 12 month. Furthermore, with the exception of four types of outpatient settings—mental health clinics, in-home services, private mental health practitioner's office, and medical doctor's office—and four school settings, the use of individual service settings for the previous 12 months was reported too infrequently for reliable calculation of the kappa statistic (51). Therefore, all analyses are reported combined across the two sites and in six superordinate categories: any use, any inpatient use, any specialty mental health inpatient use (psychiatric hospital, psychiatric unit in a general hospital, drug or alcohol unit, or residential treatment center), any outpatient use, any specialty mental health outpatient use (mental health clinic, day treatment, drug or alcohol clinic or psychologist, social worker, or counselor), and any school use.
Module reliability
Initial module. As shown in Table 1, for parents, both lifetime and 12-month test-retest reliability figures are excellent, with kappas for data combined across two sites ranging from .82 to .94 for services ever used and .75 to .86 for service use within the previous 12 months. Information about specific items in the initial module showed that the reliability of the question about overall pathways was moderate to excellent when the item asked about the person whom the respondent spoke to before using services but poor when the item asked about the advice given by that person (data not shown).
The children's test-retest reliability increased with age, with those aged 11 years and older demonstrating moderate to excellent reliability. For these children, the kappas for reported services ever used ranged from .64 to .96, and for self-reported services used within 12 months the kappas ranged from .63 to .77 (Table 2). The results for the younger children—those in our sample aged nine and ten years—were adequate but less encouraging, with kappas for lifetime use ranging from .41 to .64, and for 12-month use, a kappa of .64 (data not shown). These children had difficulty distinguishing between closely related service settings such as psychiatric hospitals and psychiatric units in general hospitals.
To determine whether memory for the SACA questions contributed to the excellent test-retest results, we compared the results of respondents who were retested within seven days with the results of respondents who were retested after more than seven days. For all categories, there were minimal or no differences in kappa scores between the two groups, with no consistent trend favoring scores obtained within seven days. In the ever-using-services category, for example, kappas ranged from .94 for any use to .84 for a school use for respondents who were retested within seven days and .94 for any use to .80 for a school use for those who were retested after more than seven days.
Individual service setting module. Parents' and older children's reports of reasons for use of services were highly reliable between initial testing and retesting, indicating that people who use services do so for well-developed reasons. Kappa values for parents' reliability were 1 for psychiatric hospitals, .89 for mental health clinics, and .87 for mental health professionals. Kappa values for older children were .70 for psychiatric hospitals, .80 for mental health clinics, and .75 for mental health professionals. Younger children reported reasons for seeking care at psychiatric hospitals or mental health clinics too infrequently for reliability figures to be obtained; their kappa value for seeking care from mental health professionals indicated only moderate reliability (kappa=.5).
Reliability for parents' reports of intensity of services, as measured by the number of inpatient admissions and the number of visits for the most recent outpatient mental health clinic use, was uneven. The number of hospital admissions showed moderate reliability (intraclass correlation coefficient=.44), but the number of clinic visits was unreliable (ICC=.03). The reliability of older children's reports was poor for both inpatient psychiatric hospital admissions (ICC=.27) and number of visits to mental health clinics (ICC=.24).
The reliability of parents' reports of the types of treatments received in any one setting were generally moderate to excellent for treatments offered by mental health clinics (kappas ranging from .57 to .8) and poor to moderate for treatments offered by mental health professionals (kappas ranging from .3 to .55).
The number of children reporting specific types of treatments in any one setting was generally too low to be used as an indicator of reliability, with the exception of treatment from a mental health professional. In this case, the reliability of older children's reports of child therapy, medications, evaluation or testing, and family therapy were low to moderate (kappas ranging from .38 to .54) (data not shown).
The only benefit question for which the sample was large enough for analysis was parents' assessments of the benefits of mental health clinic use. The reliability of this item was good (kappa=.77 for mental health clinic use). The number of reports of follow-up services and cultural sensitivity were too small for reliable analysis. The financial questions showed very poor reliability (ICC ranging from 0 to .25 for out-of-pocket costs) (data not shown).
Medications module. Reliability for questions about any use of medications within the past six months was generally very good among adult respondents (kappa=.96) (Table 3). Methylphenidate was the single most prevalent medication named—41 of 149 respondents (28 percent). Central nervous system stimulants (38 percent) and antidepressants (41 percent) were about equally represented. During retesting, parents were able to name the same drugs about 87 percent of the time.
The reliability of reported dosage and number of days that medications were used were analyzed separately for parents who reported the use of one medication (56 percent) and those who reported the use of more than one medication (30 percent). The percent agreement as to whether the child was still taking the medication was excellent. Agreement was also excellent for knowledge of whether health insurance paid for the medication (kappa was 1 for parents who reported use of one medication and .73 for parents who reported use of more than one medication), amount paid out of pocket (ICC=.96 and .96, respectively), the dose of medication given (ICC=.88 and .99), and the number of times per day the child took medication (ICC=.86 and .85). The reliability of the number of days in the previous month that the child took medication was poor (ICC=.41 and .06) (data not shown). Questions about dosage schedules, particularly the questions that attempted to capture "drug holidays" or varying weekly or daily schedules, proved problematic.
Discussion
This test-retest reliability study showed that the SACA-Parent Version had uniformly excellent reliability for both lifetime and 12-month service use. When we compared these data with the parent test-retest reliability data reported for the CASA, we found that 12-month reliability data for the SACA indicate reasonable recall over a longer period. We thought it was important to assess a 12-month period because one year is a common time frame for gathering diagnostic and payment information, both in provider settings and in services research.
The structure of the SACA—with its initial module—appears to prevent the common problem of attenuation of responses in instruments that have a repetitive format (52). Furthermore, an examination of test-retest results stratified by the length of recall period—one week or less compared with more than one week—indicated consistently high agreement over the two time spans. Although these findings do not definitively rule out memory as a reason for the high reliability, the finding that a longer time between test and retest did not affect the SACA's reliability provides evidence that the respondents were remembering the services used rather than just the questions asked.
The data for the children presented a more varied picture. For children aged 11 years and older, the kappas for lifetime and 12-month use were in the good to excellent range. These results are consistent with the findings from child reports for the CASA (39).
The reliability figures for children aged nine and ten years were in the moderate range and were considerably lower than those for the older children. Our examination of the younger children's responses suggested considerable confusion about several of the service use settings; thus, we question whether children who are younger than 11 are reliable reporters of their own use of mental health services. These results are similar to those found when using the Diagnostic Interview Schedule for Children with children aged nine to 11 years (53).
Types of treatments—with the exception of case management and other support services, such as help with rent and food—had excellent test-retest reliability for adults and children, which suggests that services that are clearly demarcated are reliably reported. The experience with the SACA suggests that researchers who are gathering information about case management should focus their inquiries on specific activities.
Similarly, assessments of benefit and follow-up were usually reported quite reliably by both parents and children. Parents and children answered questions about cultural sensitivity too infrequently to enable us to draw firm conclusions. This lack of information is unfortunate, because perceptions of the cultural responsiveness of services may predict how well clients maintain contact with their service providers (8).
Results for the medications module demonstrate that information about dosage, form, and frequency of use as well as insurance coverage can be reliably reported. Detailed information about dosages on different days was difficult to report reliably and time-consuming to collect. The question about the start date of medications appeared to have caused some confusion. Better information might be elicited if the instrument were revised to ask how long the child had been on medication in terms of months and years.
Conclusions
The SACA reliability study demonstrates that parents and children aged 11 years and older can report both lifetime and 12-month use of mental health services with good to excellent reliability. The SACA can be used to collect some information about mental health services that has been missing in previous research and thus greatly expand the knowledge base in children's mental health services.
Acknowledgments
This work is a product of the multisite study of Mental Health Services Use, Needs, Outcomes, and Costs in Child and Adolescent Populations supported by the National Institute of Mental Health; the Administration for Children, Youth, and Families; the Center for Mental Health Services; the National Institute of Child Health and Human Development; and the Department of Education.
Footnote
Dr. Horwitz is with the departments of epidemiology and public health and the Child Study Center at Yale University School of Medicine and the Institute for Social and Policy at Yale. Dr. Hoagwood is associate director for child and adolescent mental health services at the National Institute of Mental Health in Rockville, Maryland. Dr. Stiffman is with the George Warren Brown School of Social Work at Washington University in St. Louis. Dr. Summerfeld is with the department of psychology at Michigan State University in East Lansing. Dr. Weisz is with the department of psychology at the University of California, Los Angeles (UCLA). Ms. Bean is with Rand in Santa Monica, California, and with UCLA. Dr. Costello is with the department of psychiatry and behavioral sciences at Duke University Medical Center in Chapel Hill, North Carolina. Dr. Rost is with the department of family medicine at the University of Colorado in Denver. Dr. Cottler is with the department of psychiatry at Washington University School of Medicine. Dr. Leaf is with the department of mental hygiene at Johns Hopkins University in Baltimore. Ms. Roper is with the division of services and intervention research at the National Institute of Mental Health in Bethesda, Maryland, where Dr. Norquist is division director. Address correspondence to Dr. Horwitz at Yale University School of Medicine, Department of Epidemiology and Public Health 60 College Street, Room 310, New Haven, Connecticut 06520 (e-mail, [email protected]).
References
1.
Brandenburg NA, Friedman RM, Silver SE: The epidemiology of childhood psychiatric disorders: prevalence findings from recent studies. Journal of the American Academy of Child and Adolescent Psychiatry 29:76-83, 1990
2.
Anderson JB, Williams S, McGee R, et al: DSM-III disorders in pre-adolescent children. Archives of General Psychiatry 44:69-76, 1987
3.
Costello EJ: Developments in child psychiatric epidemiology. Journal of the American Academy of Child and Adolescent Psychiatry 28:836-841, 1989
4.
Offord DR, Boyle MH, Szatmari P, et al: Ontario Child Health Study: II. six-month prevalence of disorders and rates of service utilization. Archives of General Psychiatry 44:832-836, 1987
5.
Costello EJ, Angold A, Burns BJ, et al: The Great Smoky Mountains study of youth, goals, design, methods, and prevalence of DSM-III-R disorders. Archives of General Psychiatry 53:1129-1136, 1996
6.
Shaffer D, Fisher P, Dulcan MK, et al: The NIMH Diagnostic Interview Schedule for Children Version 2.3 (DISC-2.3): description, acceptability, prevalence rates and performance in the MECA study. Journal of the American Academy of Child and Adolescent Psychiatry 35:865-877, 1996
7.
Costello EJ, Messer SC, Reinherz HZ, et al: The prevalence of serious emotional disturbance: a re-analysis of community studies. Journal of Child and Family Studies 7:411-432, 1998
8.
Leaf PJ, Alegria M, Cohen P, et al: Mental health service use in the community and schools: results from the four-community MECA study. Journal of the American Academy of Child and Adolescent Psychiatry 35:889-897, 1996
9.
Stiffman AR, Chen YW, Elze D, et al: Adolescents' and providers' perspectives on the need for and use of mental health services. Journal of Adolescent Health 21:335-342, 1997
10.
Zahner GE, Pawelkiewicz W, DeFrancesco JJ, et al: Children's mental health services needs and utilization patterns in an urban community: an epidemiological assessment. Journal of the American Academy of Child and Adolescent Psychiatry 31:951-960, 1992
11.
Burns BJ, Costello EJ, Angold A, et al: Children's mental health service use across service sectors. Health Affairs 14(3):147-159, 1995
12.
Kessler LG, Steinwachs DM, Hankin JR: Episodes of psychiatric utilization. Medical Care 18:1219-1227, 1980
13.
Hornbrook MC, Berki SE: Practice mode and payment method: effects on use, costs, quality, and access. Medical Care 23:484-511, 1985
14.
Keeler EB, Manning WG, Wells KB: The demand for episodes of mental health services. Journal of Health Economics 7:369-392, 1988
15.
Keeler EB, Rolph JE: The demand for episodes of treatment in the health insurance experiment. Journal of Health Economics 7:337-367, 1988
16.
Wells KB, Keeler E, Manning GW: Patterns of outpatient mental health care over time: some implications for estimates of demand and for benefit design. Health Services Research 24:773-789, 1990
17.
Knitzer J: Unclaimed Children: The Failure of Public Responsibility to Children and Adolescents in Need of Mental Health Services. Washington, DC, Children's Defense Fund, 1982
18.
Stroul B, Friedman RM: A System of Care for Children and Youth With Severe Emotional Disturbances (rev). Washington, DC, CASSP Technical Assistance Center, Georgetown University Child Development Center, 1986
19.
Bird HR, Canino G, Rubio-Stipec M, et al: Estimates of the prevalence of childhood maladjustment in a community survey in Puerto Rico. Archives of General Psychiatry 45:1120-1128, 1988
20.
Costello EJ, Burns BJ, Costello AJ, et al: Service utilization and psychiatric diagnosis in pediatric primary care: the role of the gatekeeper. Pediatrics 82:415-424, 1988
21.
Weisz JR, Weiss B: Studying the referability of child clinical problems. Journal of Consulting and Clinical Psychology 59:266-273, 1991
22.
Services Integration: A Twenty-Year Retrospective. Washington, DC, US Department of Health and Human Services, Office of the Inspector General, 1991
23.
Farmer EMZ, Stangl DK, Burns BJ, et al: Persistence and intensity: patterns of care for children's mental health across one year. Community Mental Health Journal 35:31-46, 1999
24.
Bickman L: A continuum of care: more is not always better. American Psychologist 51:689-701, 1996
25.
Returning the Mentally Disabled to the Community: Government Needs to Do More. HRD-76-152. Washington, DC, US General Accounting Office, 1977
26.
Goldman HH, Morrissey JP, Ridgely MS: Evaluating the Robert Wood Johnson Foundation program in chronic mental illness. Milbank Quarterly 72:37-47, 1994
27.
Glisson C, Hemmelgarn A: The effects of organizational climate and interorganizational coordination on the quality and outcomes of children's service systems. Child Abuse and Neglect 22:1-21, 1998
28.
Lehman AF, Postrado LT, Roth D, et al: Continuity of care and client outcomes in the Robert Wood Johnson Foundation program in chronic mental illness. Milbank Quarterly 72:105-122, 1994
29.
Weisz JR, Weiss B, Donenberg GR: The lab versus the clinic: effects of child and adolescent psychotherapy. American Psychologist 47:1578-1585, 1992
30.
Weisz JR, Donenberg GR, Han SS, et al: Bridging the gap between laboratory and clinic in child and adolescent psychotherapy. Journal of the American Academy of Child and Adolescent Psychology 63:688-701, 1995
31.
Costello EJ, Farmer EMZ, Angold A, et al: Psychiatric disorders among American Indian and white youth in Appalachia: The Great Smoky Mountains Study. American Journal of Public Health 87:827-832, 1997
32.
Chung FK, Snowden LR: Community mental health and ethnic minority populations. Community Mental Health Journal 26:277-291, 1990
33.
Scheffler RM, Miller AB: Demand analysis of mental health service use among ethnic subpopulations. Inquiry 26:202-215, 1989
34.
Broman CL: Race differences in professional help seeking. American Journal of Community Psychology 15:473-489, 1987
35.
McMiller WP, Weisz JR: Help-seeking preceeding mental health clinic intake among African-American, Latino, and Caucasian youths. Journal of the American Academy of Child and Adolescent Psychiatry 35:1086-1094, 1996
36.
Takeuchi DT, Leaf PJ, Kuo HS: Ethnic differences in the perception of barriers to help-seeking. Social Psychiatry and Psychiatric Epidemiology 23:273-280, 1988
37.
Terrell F, Terrell SL: Race of counselor, client sex, cultural mistrust level, and premature termination from counseling among black clients. Journal of Counseling and Psychology 31:371-375, 1984
38.
Weisz JR, McMiller WP: Minority help-seeking. Journal of the American Academy of Child and Adolescent Psychiatry 36:444-445, 1997
39.
Farmer EMZ, Angold A, Burns BJ, et al: Reliability of self-reported service use: test-retest consistency of children's responses to the Child and Adolescent Services Assessment (CASA). Journal of Child and Family Studies 3:307-325, 1994
40.
Ascher BA, Farmer EMZ, Burns BJ, et al: The Child and Adolescent Services Assessment (CASA): description and psychometric. Journal of Emotional Behavioral Disease 4:12-20, 1996
41.
Weisz JR: Studying Clinic-Based Child Mental Health Care. Research in Progress. Los Angeles, University of California, 1996
42.
Hoagwood K, Horwitz SM, Stiffman AR et al: Concordance between parent reports of children's mental health services and services records. Journal of Child and Family Services, in press
43.
Stiffman AR, Horwitz SM, Hoagwood K, et al: Adult and child reports of mental health services in the Service Assessment for Children and Adolescents (SACA). Journal of the Academy of Child and Adolescent Psychiatry 39:1032-1039, 2000
44.
Schwab-Stone M, Fisher P, Piacentini J, et al: The Diagnostic Interview Schedule for Children-Revised Version (DISC-R): II. test-retest reliability. Journal of the American Academy of Child and Adolescent Psychiatry 3:651-657, 1993
45.
Edelbrock C, Crnic K, Bohnert A: Interviewing as communication: an alternative way of administering the Diagnostic Interview Schedule for Children. Journal of Abnormal Child Psychology 6:447-453, 1999
46.
Cicchetti D: Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment 4:284-290, 1994
47.
Cohen J: A coefficient of agreement of nominal scales. Educational and Psychological Measurement 20:37-46, 1960
48.
Fleiss JL: Statistical Methods for Rates and Proportions, 2nd ed. New York, Wiley, 1981
49.
Bartko JJ: The intraclass correlation coefficient as a measure of reliability. Psychological Reports 19:3-11, 1966
50.
Cicchetti DV, Sparrow SS: Developing criteria for establishing interrater reliability of specific items: applications to assessment of adaptive behavior. American Journal of Mental Deficiency 86:127-137, 1981
51.
Donner A, Eliasziw M: Sample size requirements for reliability studies. Statistics in Medicine 6:441-448, 1987
52.
Piacentini J, Roper M, Jensen P, et al: Informant-based determinants of symptom attenuation in structured child psychiatric interviews. Journal of Abnormal Child Psychology 6:417-428, 1999
53.
Schwab-Stone M: Do children aged 9 through 11 years understand the DISC version 2.25 questions? Journal of the American Academy of Child and Adolescent Psychiatry 34:954-956, 1995
Information & Authors
Information
Published In
History
Published online: 1 August 2001
Published in print: August 2001
Authors
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBGet Access
Login options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).