Reliable Self-Report of Health Service Use by Individuals With Serious Mental Illness
Abstract
The purpose of this study was to determine whether individuals with serious mental illness could consistently report their use of medical services. Twenty-nine individuals with schizophrenia completed a test-retest protocol using a modified version of the Medical Expenditure Panel Survey; there was a one-week interval between the two administrations of the instrument. Consistency of reporting was generally strong and was strongest for items indicating whether a service had been used in the past six months, which suggests that people with schizophrenia can provide reliable information about their use of health services. Continued research in this area will increase understanding of the types and quality of medical care received by persons with schizophrenia.
Medical morbidity and mortality rates are higher among persons who have serious mental illness than in the general population (1). In addition, recent research suggests that psychiatric populations have different patterns of health service use than nonpsychiatric populations (2,3). Because data on service use are commonly self-reported (4,5), research is needed to evaluate the reliability with which individuals who have serious mental illness report their use of health care services.
Previous research has suggested that reliable self-report among psychiatric populations, particularly self-assessment of symptoms, may be compromised by the cognitive deficits and lack of insight associated with serious mental illness (6). Whether this observation extends to the self-report of behavioral events such as use of services has been tested less frequently. We reviewed the literature and found only two studies that examined this question with a psychiatric population (7,8), and only one of them assessed test-retest reliability (7). Reporting test-retest correlation coefficients ranging from .87 to .89, Calsyn and colleagues (7) concluded that homeless individuals with mental illness can provide reliable information about their use of mental health services. However, no study has provided detailed information about the extent to which individuals who have serious mental illness reliably report their use of medical services with instruments developed for nonpsychiatric populations.
Using a modified version of a well-known national health services survey, we conducted this study to determine whether individuals with schizophrenia could consistently report their use of medical services. On the basis of the findings of Calsyn and colleagues (7), we hypothesized that this patient population would report service use consistently.
Methods
The participants in this study were part of a larger study of quality of care in schizophrenia. A modified version of the household component of the Medical Expenditure Panel Survey (MEPS), a scientifically valid and widely used survey of medical service use and expenditures developed by the Agency for Healthcare Research and Quality (AHRQ) (4), was administered to 171 outpatients who had been diagnosed as having schizophrenia. Our review of the literature and MEPS-related documentation from the AHRQ found that no test-retest results were available for the national survey.
Twenty-nine participants completed a one-week test-retest protocol using the modified MEPS. The instrument—read aloud by the interviewer—included items about medical services received in outpatient settings (for example, "During the past six months, did you go to a doctor's office, outpatient clinic, health center, or some other outpatient setting for any kind of medical care?"), hospital emergency departments (for example, "During the past six months, did you go to a hospital emergency room for medical care?"), and inpatient hospital settings (for example, "During the past five years, approximately how many times were you a patient in the hospital because of a medical condition or problem?"). There were also items related to dental, vision, and hearing services.
For dichotomous variables, we measured agreement across administrations by using a simple (unweighted) kappa statistic. On the basis of the standard error, a 95 percent confidence interval was also calculated as a measure of the precision of the estimate of agreement. Because low-prevalence phenomena can influence the kappa statistic, percent agreement scores were also calculated. For ordinal variables with three or more levels, a weighted kappa statistic was used to allow for maximum weighting of the more discrepant cases of disagreement. To increase the cell counts and improve statistical power, some response levels were combined. In accordance with the method of Landis and Koch (9), estimated kappa values below .40 were considered poor to fair; .41 to .60, moderate; .61 to .80, substantial; and .81 to 1.00, almost perfect.
Results
Twenty-seven (93 percent) of the study participants were men; 25 (86 percent) were African American, and four (14 percent) were Caucasian. The mean±SD age of the patients was 40±7.4 years (range, 26 to 56 years). Efforts were made to administer the two interviews as closely as possible to seven days apart. The mean period between administrations was 5±2 days.
Response distributions across 17 items for each administration of the survey, kappa values and related confidence intervals, a strength-of-consistency classification based on the criteria of Landis and Koch, and total agreement percentages are listed in Table 1. Consistency across items was generally strong: of the 16 variables for which kappa values could be classified, three were classified as almost perfect, six as substantial, six as moderate, and one as fair.
Consistency was strongest for items that required respondents to report whether they had used a particular service in the past six months—for example, for six-month use of an emergency department, kappa=.91; for six-month use of outpatient medical services, kappa=.86. Kappa values and percent agreement scores were generally somewhat lower for items that sought information about the number of visits—for example, for the number of outpatient visits in the past six months, kappa=.50; for the number of visits to an emergency department in the past five years, kappa=.30. Respondents were moderately to substantially consistent in reporting receipt of medications (kappa=.45) and use of dental services (.61 and .62). Substantial agreement was found for receipt of a vision test (.79) and receipt of a hearing test (.63).
Discussion and conclusions
The purpose of this study was to determine whether individuals with schizophrenia could consistently report their use of medical services. Overall, the kappa values and percent agreement scores indicated strong consistency. Respondents were most consistent about whether they had used a particular service and were somewhat less consistent in reporting their actual level of service use. These results both support and extend the findings of Calsyn and colleagues (7). Although previous research has shown that the cognitive deficits and lack of insight associated with schizophrenia may compromise the reliability of self-assessments of symptoms (6), our results suggest that people with schizophrenia can reliably report behavioral events such as use of medical services.
It is not known, however, how a longer interval between the two administrations of the survey would have affected consistency scores. Nor do we know how our results compare with those from other samples of individuals who responded to the MEPS, as we did not find reports of test-retest reliability for the instrument in our review of the literature and related MEPS documentation. Also, our findings do not address the validity of the study participants' reports. Although the results of recent research have suggested that the validity of self-report among individuals with serious mental illness may vary by the type of service use reported (7,8,10), the services assessed in those studies were primarily mental health services.
Thus future research also needs to assess the validity of self-reported use of medical services among individuals with serious mental illness. Once the reliability and validity of such information can be established, researchers can use national surveys of medical service utilization, such as the MEPS, to better understand the types and quality of medical care received by people with schizophrenia. Such information will also enable services researchers and policy makers to learn more about the integration of psychiatric and medical services for this vulnerable population.
Acknowledgments
Support for this project was provided by grant MH-57707 from the National Institute of Mental Health and from the Mental Illness Research, Education, and Clinical Center of the Veterans Affairs Capitol Health Care Network (VISN 5).
Footnote
The authors are affiliated with the School of Medicine at the University of Maryland at Baltimore. Send correspondence to Dr. Goldberg at the Center for Mental Health Services Research, University of Maryland at Baltimore, 685 West Baltimore Street, MSTF Building, Room 300, Baltimore, Maryland 21201 (e-mail, [email protected]). This paper is one of several in this issue focusing on the health and health care of persons with severe mental illness.
References
1.
Felker B, Yazel JJ, Short D: Mortality and medical comorbidity among psychiatric patients: a review. Psychiatric Services 12:1356-1363, 1996
2.
Berren MR, Santiago JM, Zent MR, et al: Health care utilization by persons with severe and persistent mental illness. Psychiatric Services 50:559-561, 1999
3.
Steiner JL, Hoff RA, Moffett C, et al: Preventive health care for mentally ill women. Psychiatric Services 49:696-698, 1998
4.
Cohen J: Design and Methods of the Medical Expenditure Panel Survey Household Component. MEPS methodology report no 1, AHCPR publication 97-0026. Rockville, Md, Agency for Health Care Policy and Research, 1997
5.
Narrow WE, Regier DA, Rae DS, et al: Use of services by persons with mental and addictive disorders: findings from the National Institute of Mental Health Epidemiologic Catchment Area Program. Archives of General Psychiatry 50:95-107, 1993
6.
Atkinson M, Zibin S, Chuang H: Characterizing quality of life among patients with chronic mental illness: a critical examination of the self-report methodology. American Journal of Psychiatry 154:99-105, 1997
7.
Calsyn RJ, Allen G, Morse GA, et al: Can you trust self-report data provided by homeless mentally ill individuals? Evaluation Review 17:353-366, 1993
8.
Calsyn RJ, Morse GA, Klinkenberg WD, et al: Reliability and validity of self-report data of homeless mentally ill individuals. Evaluation and Program Planning 20:47-54, 1997
9.
Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics 33:159-174, 1977
10.
Kashner TM, Suppes T, Rush AJ, et al: Measuring use of outpatient care among mentally ill individuals: a comparison of self-reports and provider records. Evaluation and Program Planning 22:31-39, 1999
Information & Authors
Information
Published In
History
Published online: 1 July 2002
Published in print: July 2002
Authors
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBGet Access
Login options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).