Agreement Between Seriously Mentally Ill Veterans and Their Clinicians About Medication Compliance
Abstract
Objectives: Levels of agreement about medication compliance in a large cohort of seriously mentally ill veterans and their clinicians were examined to determine whether agreement increased with exposure to enhanced treatment programs emphasizing compliance and whether compliance reports were associated with hospitalization. METHODS: A total of 1,369 seriously mentally ill patients and their treating clinicians at 14 Veterans Affairs medical centers rated medication compliance at enrollment in enhanced programs or comparison programs offering standard care. Patients and clinicians reassessed compliance one and two years after enrollment. Overall agreement, agreement about compliance and noncompliance, and kappa statistics were determined for concurrent assessments. RESULTS: Overall, patients rated themselves as significantly more compliant with medication than did clinicians at enrollment. Cohen's kappa at enrollment was .095, indicating little patient-clinician agreement beyond that expected by chance. Kappa values increased significantly at one and two years for patients in the enhanced programs but continued to indicate poor-to-modest levels of agreement. Patient-clinician pairs in enhanced programs did not differ from those in comparison programs in overall agreement. Reports of good compliance by both patients and clinicians were associated with significantly decreased odds of hospital admission in the 30 days after the report was made. CONCLUSIONS: Seriously mentally ill patients and their clinicians showed little agreement about medication use beyond that expected by chance. Intensive programming appeared to have little effect on agreement. Both patients' and clinicians' compliance assessments predicted hospitalization and thus can be used in research models that attempt to predict relapse and readmission.
Antipsychotic medication is known to be an efficacious treatment for schizophrenia, with the majority of patients showing symptomatic improvement (1). Fifty to 75 percent of stable patients who discontinue antipsychotic medications relapse within a year (2,3,4,5), and current practice guidelines recommend long-term maintenance treatment for most patients (6).
Medication noncompliance is common among seriously mentally ill patients (7) and is a predictor of rehospitalization (3,8,9,10). Noncompliance may result from patients' confusion about the prescribed regimen, disagreement about the need for treatment, medication side effects, or a focus on other life issues (11,12,13). Compliance has been found to decrease when the medication regimen is long term, patients are experiencing chronic rather than acute symptoms, and treatment is preventive—all characteristics of antipsychotic maintenance treatment (14,15). Concurrent substance abuse, a lack of awareness of the psychiatric illness, and thought disorganization are also associated with decreased compliance (8,16,17).
Studies of compliance among seriously mentally ill patients have typically used patients' self-reports or ratings by clinicians and family members as compliance measures (7,10,18,19). Other types of measures, such as systematic pill counts or determinations of plasma drug levels, have advantages, but they are expensive, difficult to employ in naturalistic settings, and still subject to error (12,20,21). Studies that have used pill counts or plasma levels as compliance measures have found approximately 25 to 65 percent of patients with schizophrenia to be noncompliant (13,22,23).
Researchers have attempted to improve the accuracy of compliance reports by obtaining ratings from multiple observers—patients, family members, and clinicians. However, few studies have commented on agreement between concurrent assessments, and those that have commented found both "no significant correlation" (17) and "adequate concordance" (16).
Investigators have used concurrent compliance reports in a variety of ways. They have combined reports into an overall compliance score (24), analyzed reports separately (17), or selectively used the reports, often selecting the one indicating poorer compliance as the study measure (9,19). Examining the relationship between reports by the patient, the clinician, and family members may help determine if the different assessments tap the same underlying behaviors and may also shed light on how to combine or otherwise use these measures.
Agreement between patients and clinicians on medication use has implications for clinical care as well as compliance research. Clinicians have been urged to engage seriously mentally ill patients in treatment partnerships (11,25) and to "elicit the patient's cooperation in monitoring participation in treatment" (25). Open disclosure about compliance has been thought to be "a product and a catalyst for practitioner-patient relationships of high quality" (26), and poor compliance has been viewed as a failure of the patient-clinician partnership.
In clinical partnerships characterized by accurate patient self-monitoring and open discussions about medication, a common understanding about medication use should logically emerge. Disparate reports of medication use by clinicians and patients would reflect ineffective communication and suggest that these partnerships have not yet developed.
In 1991 Congress appropriated special funds to establish enhanced treatment programs for seriously mentally ill veterans at 14 Veterans Affairs medical centers. The enhanced programs include day treatment programs, intensive community case management programs, and STAR II programs, which are intensive rehabilitative programs designed to assist patients in adjusting to community living (27). All enhanced programs were designed to decrease hospital admissions and lengths of stay for patients with high service utilization. Eleven of the 14 enhanced programs list "increasing compliance" as a major programmatic goal, and all programs include treatment components thought to increase compliance, such as increased contact with patients, increased continuity of care by a single treatment team, and coordinated educational programs on medication and symptom management.
In addition, all the enhanced treatment programs were supplied with the Brentwood modules (28), a structured educational intervention designed to help patients "become progressively more self-reliant in their use of antipsychotic medication." The Brentwood modules emphasize the development of skills for self-monitoring, medication management, and negotiation among seriously mentally ill patients (29).
We conducted a secondary analysis of data from the longitudinal cohort study of seriously mentally ill veterans enrolled either in enhanced treatment programs or in comparison programs of standard care. Our objectives were to determine agreement about compliance in a large cohort of seriously mentally ill patients and their clinicians, whether agreement increased with exposure to intensive programming that emphasized compliance, and whether compliance reports by patients or clinicians were associated with hospital admission.
Methods
Study population
Veterans were eligible for enrollment in enhanced treatment programs or comparison standard care programs if they qualified for VA hospital care, had a DSM-III-R diagnosis indicating psychosis (schizophrenia, schizoaffective disorder, bipolar disorder with psychosis, chronic organic psychosis, or other psychoses), and had either 150 or more documented days of hospitalization or five or more inpatient admissions in the year before enrollment.
A total of 1,258 eligible veterans were enrolled in the enhanced programs, and 342 were enrolled in the comparison programs, in a four-to-one ratio. Evaluation coordinators at each site assigned patients to the programs based on the enrollment ratio and clinical judgment. A total of 1,600 veterans were enrolled between January 9, 1991, and December 19, 1995, at 14 VA medical centers. Due to rolling enrollment and patient mortality, 1,516 veterans were eligible for a one-year follow-up, and 1,281 veterans were eligible for a two-year follow-up by February 4, 1997.
Assessments and measures
Veterans and treating clinicians were asked to rate medication compliance at the time of enrollment and at one and two years after enrollment. Clinicians' compliance reports were collected by surveying clinicians, and patients' compliance reports were collected through structured interviews conducted by evaluation coordinators, who were research staff located on site and not directly involved in patient care. Evaluation coordinators received two days of formal training in the use of the structured interview and the interview procedure manual, as well as annual retraining and regular consultation during monthly conference calls.
Members of clinical treatment teams, including psychiatrists, social workers, psychologists, and nurses, were trained in use of the clinician survey. Team members viewed a video of a patient interview, completed clinician surveys, and compared scores with other team members and an expert rater. Clinicians on the team with the closest relationship with the patient were responsible for completing the survey. The clinician's ratings of the patient were completed in close conjunction with the patient's visits. Patients were scheduled to see their clinicians when the surveys were due, unless a clinical visit was already scheduled at the appropriate time.
Both patients and clinicians rated medication compliance on a 5-point ordinal scale with anchors of taking medication none of the time, some of the time, quite a bit of the time, most of the time, or all of the time. Ordinal ratings were dichotomized. Clinicians' and patients' reports were considered to indicate compliance if the clinician or the patient reported that medication was taken all or most of the time. The reports were considered to indicate noncompliance if the clinician or patient reported that medication was taken quite a bit, some, or none of the time. This dichotomy was chosen to reflect a point at which a reasonable clinician might be concerned about adverse consequences arising from the reported level of medication use.
Patients assessed their compliance over the previous three months, whereas clinicians assessed compliance over the previous six months. Patient-clinician assessments with overlapping time frames were used in the analyses.
Data on hospital admissions were obtained from the centralized data files maintained by the Department of Veterans Affairs.
Statistical analyses
Analyses were conducted on data collected between January 9, 1991, and February 5, 1997. Differences in ordinal assessments of compliance by patients and clinicians were analyzed with the median sign test.
Agreement between patients and clinicians was determined with four measures: overall agreement; Cohen's kappa, a statistical test that corrects for agreement expected by chance; positive agreement, or agreement that patients were compliant; and negative agreement, or agreement that patients were noncompliant. Documentation of positive and negative agreement supplements information contained in the omnibus test of agreement, the kappa, and is useful in understanding low kappas coupled with high levels of observed agreement (30,31). Further information about these measures can be found elsewhere (30,31).
Associations between overall agreement and time in the program were analyzed with McNemar's test for paired data. The significance of differences in kappa scores at baseline, one, and two years was determined with the chi square test described by Fleiss (32).
Multivariable analyses were conducted using generalized estimating equations (GEEs). The first analysis looked at the relationship between agreement (yes or no) and the predictor variable (program type), with covariates of age and initial score on the Global Assessment of Functioning (GAF). The second analysis looked at the relationship of agreement to predictor variables of age, GAF scores, and scores on the Brief Psychiatric Rating Scale (BPRS). BPRS items were scored using a rating scale of 0 to 6; a score of 20 is equivalent to a score of 38 on the commonly used scale, on which ratings of 1 to 7 are given.
The final GEE analysis examined the relationship between hospitalization within the 30 days after assessment (yes or no) and clinicians' and patients' compliance reports.
All GEE analyses accounted for the fact that observations would be correlated for patients with more than one observation. Statistical analyses were completed using SAS software, version 6.12.
Results
Compliance data
Compliance reports with appropriate time overlaps were available for 1,369 of the 1,600 patient-clinician pairs at baseline (86 percent of eligible pairs), for 829 of the 1,001 pairs with one-year follow-up assessments (83 percent of the pairs with one-year data, and 55 percent of eligible pairs), and for 676 of the 793 pairs with two-year follow-up assessments (85 percent of pairs with two-year data, and 53 percent of eligible patients).
Patient characteristics
The mean±SD age of patients at study entry was 51.3±12.9 years, with a range of 21.3 to 86 years. Ninety-six percent of patients were male. Seventy-six percent had a qualifying diagnosis of schizophrenia; 13 percent, schizoaffective disorder; 4 percent, bipolar disorder; 5 percent, another mood disorder; and 2 percent, organic psychotic disorder. Patients were seriously impaired, with mean±SD baseline BPRS scores of 20±12 and mean GAF scores of 45±15.
Assessments of compliance
Table 1 summarizes data on assessments of medication compliance by clinicians and patients in enhanced programs at baseline and at one and two years after enrollment. Assessments differed significantly at baseline (sign test, p<.001), with patients considering themselves far more compliant than clinicians. At enrollment, 95 percent of patients in the enhanced programs stated that they were taking their medications all or most of the time, while only 71 percent of clinicians judged these patients to be taking medications all or most of the time. Significant differences between patients' and clinicians' compliance assessments remained at one and two years for patients in the enhanced and comparison programs.
Agreement
Figure 1 summarizes data on overall observed agreement and agreement about compliance and noncompliance between clinicians and patients in enhanced programs. Kappas for the patient-clinician pairs in enhanced programs at enrollment and at one and two years after enrollment are also illustrated.
Although relatively high levels of overall agreement at enrollment were found for the clinician-patient pairs in enhanced programs, there was little agreement beyond that expected by chance (Cohen's kappa=.095). This low kappa value was the result of a low level of agreement about noncompliance. On average, patients and clinicians agreed only 17 percent of the time when one rater indicated noncompliance. The majority of raters indicating noncompliance were clinicians. In the enhanced programs, when clinicians indicated noncompliance (N=322), patients agreed only 10 percent of the time, and when patients indicated noncompliance (N=51), clinicians agreed 61 percent of the time.
Overall agreement increased over time for patients in enhanced programs, as did agreement about both compliance and noncompliance. At two years, patients and clinicians agreed an average of 34 percent of the time when one rater indicated noncompliance.
Significant differences were found between kappa scores at baseline, one year, and two years for patients in enhanced programs (kappas=.095, .21, .29, respectively; χ2=13.1, df=2, p< .01). Kappas at one and two years were significantly greater than the baseline kappa (χ2=4.6, df=1, p<.05, and χ2=10.4, df=1, p<.01, respectively). However, even the largest kappa of .29 indicated only poor-to-modest agreement. No significant differences between kappa scores were found for patients in the comparison group at baseline, one year, and two years (kappas=.096, .08, and .21, respectively).
Although patient-clinician pairs in enhanced programs showed increased overall agreement with time, they did not differ significantly from patient-clinician pairs in the comparison program when the analyses adjusted for age and initial GAF scores (odds ratio=1.3, 95 percent confidence interval=1 to 1.6).
The above conclusions remained essentially unchanged when ordinal compliance reports for patients and clinicians in enhanced programs were analyzed with weighted kappas. Again, there were low levels of agreement that increased modestly with time.
Effects of patient attrition
Some evidence was found of differential patient attrition. Patients considered noncompliant by clinicians at baseline were less likely to complete the one-year follow-up assessment than those judged compliant at baseline (χ2=18.4, df=1, p<.001). Analyses using data only from patients with complete data from all assessments (N=569) continued to show significant increases in agreement at one and two years for patients in enhanced programs (McNemar's test, p<.02 at one year and p<.001 at two years) but not for patients in comparison programs.
Other predictors of agreement
Higher BPRS scores and lower GAF scores, indicating a higher level of symptoms and impairment, were associated with less patient-clinician agreement about compliance (GEE analysis for BPRS scores, z=-5.4, p< .001; for GAF scores, z=4.4, p<.001). Thus more impaired patients were less likely to agree with clinicians. Agreement increased significantly with patient age (z=3.1, p<.01).
Relationship with hospitalization
Assessments of medication compliance by both patients and clinicians were strongly associated with hospitalization in the 30 days after the assessment. Patients judged compliant by clinicians had half the odds of admission in that period as patients judged noncompliant (OR=.50, 95 percent CI=.37 to .68, p<.001). Patients who reported they were compliant also had approximately half the odds of admission as patients who reported they were noncompliant (OR= .54, 95 percent CI=.32 to .91, p<.05).
Discussion
Although this study found high levels of agreement between seriously mentally ill patients and clinicians about patients' medication compliance, there was little agreement beyond that expected by chance. Low levels of agreement resulted primarily from poor agreement about noncompliance, with patients seldom reporting noncompliance even when their clinicians considered them to be noncompliant. Agreement increased over time for patients and clinicians participating in enhanced programs; however, even in enhanced programs, agreement beyond that expected by chance remained in the poor-to-modest range (kappas<.4).
The low levels of agreement suggest that patients and clinicians may have difficulty achieving a common understanding about medication use. If patients and clinicians are engaged in working partnerships, with accurate patient self-monitoring and open discussions about medication, one would expect that clinicians' assessments of compliance would be influenced by credible reports about medication use from their patients, and that patients' compliance reports would be influenced by their clinicians' supportive but careful questioning about medication use. The fact that this common understanding did not emerge suggests that working partnerships that include effective communication about medication compliance may be difficult to establish.
In this study, poor agreement occurred primarily because patients reported higher levels of medication use than clinicians—a finding that can be interpreted in two ways. If patients' and clinicians' compliance reports are assumed to be equally valid, with neither report being considered more correct or more closely approximating actual compliance, then the observed discrepancies could be attributed either to patients' overestimation or to clinicians' underestimation of compliance. This position could be taken because the reports of both patients and clinicians predicted hospitalization. One would then conclude only that agreement about compliance is low and that it is affected little by intensive programming.
A second approach to the data would be to assume that clinicians' assessments of medication compliance are the more correct assessments—a quasi-gold standard. Discrepancies in compliance reports would then be interpreted as arising from patients' overestimation rather than clinicians' underestimation of compliance. This position might be taken for several reasons. First, the majority of published studies suggest that patients (21) as well as clinicians (26) generally overestimate medication compliance. Patients' overestimation may arise from a desire to give a socially acceptable response, inaccurate perceptions of their own behavior (33), problems with recall, or inadequate understanding of the prescribed regimen. Patients who have neurocognitive impairments (17) and whose medication consumption is more variable (20) may be particularly prone to overestimation.
Patients in this study assessed medication use during interviews with research staff, and they may have been reluctant to acknowledge low levels of use. They were also seriously impaired and may have had neurocognitive problems, prominent psychiatric symptoms, and sleep disruptions, resulting in increased variability in medication consumption and increased difficulties in recalling consumption. Only 5 percent of these seriously mentally ill patients assessed themselves as noncompliant at enrollment. This rate was below noncompliance rates in the literature for this group; however, clinicians' assessments of noncompliance—29 percent at enrollment—were within reported ranges (13,16,23).
If clinicians' assessments of compliance are assumed to be the more correct assessments, difficulties in reaching a common understanding can then be tentatively attributed to patients' difficulties in perceiving or disclosing noncompliance. Actual compliance is seen as increasing over time as clinicians increasingly report good compliance, even though agreement about compliance remains poor to modest. These conclusions, in turn, suggest that agreement might be increased by facilitating more accurate or open patient disclosure of medication use. These conclusions also suggest that reaching a common understanding about medication use may not be essential in increasing compliance and that other strategies, such as "eyes-on monitoring" or intensive case management, might increase compliance in the absence of high levels of patient-clinician agreement.
In this study both patients' and clinicians' compliance reports were associated with hospital admission in the 30 days after assessment, indicating the predictive validity of the reports. Because only modest agreement was found between the reports, information in the measures was not redundant—that is, patients' and clinicians' reports tapped somewhat different behaviors and attitudes, and both predicted hospitalization. Thus both patients' and clinicians' reports can be used in models attempting to predict relapse and readmission.
Several factors may limit the generalizability of these conclusions. First, the study population was predominantly male, older, and seriously impaired. As a result, study patients may have been less compliant, less accurate in their compliance reports, and less likely to agree with clinicians than persons with serious mental illness in general. Second, the comparison group was small, and assignment to groups was not random, which may have limited our ability to detect increased agreement resulting from specialized programming. Third, patients assessed their compliance over the previous three months in structured interviews with research staff, whereas clinicians assessed compliance over the previous six months in self-administered surveys. This difference in data collection methods may have resulted in systematic biases in compliance reports that were not corrected by use of overlapping time frames in the analyses. Finally, we were unable to determine if agreement was independently associated with hospital admission because increases in agreement were necessarily confounded by increases in assessments of good compliance.
Conclusions
Seriously mentally ill patients and their clinicians may have difficulty reaching a common understanding about medication use. Intensive programming appears to have little effect on agreement. If clinicians' assessments of compliance are considered to be more correct than patients' assessments, the results of this study suggest that patients seldom perceive or disclose noncompliance. Patients' and clinicians' reports are not redundant, and both can be used in research models that attempt to predict relapse and readmission.
Acknowledgments
Funding for this research was provided by the Mental Health Strategic Health Group of the Veterans Health Administration. The authors thank Myra Kim, Ph.D., for statistical assistance and Ray Bingham, Ph.D., Gregory Dalack, M.D., Tim Hofer, M.D., Steven Katz, M.D., and Alan Mellow, M.D., for reviewing earlier versions of the paper.
Figure 1. Agreement about medication compliance between seriously mentally ill patients in enhanced programs and their clinitians1
1 Patient-clinitian pairs in enhanced programs: N=1,112 at baseline, N=695 at one year, and N=558 at two years
2 Observed proportion of agreement
3 Observed agreement on compliance divided by the mean number of responses from rates indicating compliance
4 Observed agreement on noncompliance divided by the mean number of responses from raters indicating noncompliance
Footnote
The authors are affiliated with the Serious Mental Illness Treatment Research and Evaluation Center, Health Services Research and Development, Veterans Affairs Medical Center, P.O. Box 130170, Ann Arbor, Michigan 48113-0170 (e-mail, [email protected]). Dr. Valenstein, Dr. Barry, and Dr. Blow are also with the department of psychiatry at the University of Michigan in Ann Arbor.
References
1.
National Institute of Mental Health Psychopharmacology Service Center Collaborative Study Group: Phenothiazine treatment in acute schizophrenia. Archives of General Psychiatry 10:246-261, 1964
2.
Viguera AC, Baldessarini RJ, Hegarty JD, et al: Clinical risk following abrupt and gradual withdrawal of maintenance neuroleptic treatment. Archives of General Psychiatry 54:49-55, 1997
3.
Curson D, Barnes T, Bamber R, et al: Long-term depot maintenance of chronic schizophrenic out-patients: the seven year follow-up of the Medical Research Council fluphenazine/placebo trial. British Journal of Psychiatry 146:464-480, 1985
4.
Eklund K, Forsman A: Minimal effective dose and relapse: double-blind trial: haloperidol decanoate vs placebo. Clinical Neuropharmacology 14(suppl 2):S7-12, 1991
5.
Hogarty GE, Ulrich RF: Temporal effects of drug and placebo in delaying relapse in schizophrenic outpatients. Archives of General Psychiatry 34:297-301, 1977
6.
Practice Guideline for the Treatment of Patients With Schizophrenia. American Journal of Psychiatry 154(Apr suppl):1-63, 1997
7.
Kelly G, Scott J: Medication compliance and health education among outpatients with chronic mental disorders. Medical Care 28:1181-1197, 1990
8.
Bebbington PE: The content and context of compliance. International Clinical Psychopharmacology 9(suppl 5):41-50, 1995
9.
Sullivan G, Wells KB, Morgenstern H, et al: Identifying modifiable risk factors for rehospitalization: a case-control study of seriously mentally ill persons in Mississippi. American Journal of Psychiatry 152:1749-1756, 1995
10.
Haywood TW, Kravitz HM, Grossman LS, et al: Predicting the "revolving door" phenomenon among patients with schizophrenic, schizoaffective, and affective disorders. American Journal of Psychiatry 152:856-861, 1995
11.
Corrigan P, Liberman R, Engel J: From noncompliance to collaboration in the treatment of schizophrenia. Hospital and Community Psychiatry 41:1203-1211, 1990
12.
Becker MH: Patient adherence to prescribed therapies. Medical Care 23:539-555, 1985
13.
Buchanan A: A two-year prospective study of treatment compliance in patients with schizophrenia. Psychological Medicine 22:787-797, 1992
14.
Insull W: The problem of compliance to cholesterol altering therapy. Journal of Internal Medicine 241:317-325, 1997
15.
Rogers G, Bullman W: Prescription medication compliance: a review of the baseline of knowledge: a report of the National Council on Patient Information and Education, in Advancing Prescription Medicine Compliance: New Paradigms, New Practices. Edited by Fincham J. Binghamton, NY, Pharmaceutical Products Press, 1995
16.
Owen RR, Fischer EP, Booth BM, et al: Medication noncompliance and substance abuse among patients with schizophrenia. Psychiatric Services 47:853-858, 1996
17.
Cuffel BJ, Alford J, Fischer EP, et al: Awareness of illness in schizophrenia and outpatient treatment adherence. Journal of Nervous and Mental Disease 184:653-659, 1996
18.
Adams SG Jr, Howe JT: Predicting medication compliance in a psychotic population. Journal of Nervous and Mental Disease 181:558-560, 1993
19.
Weiden P, Rapkin B, Zygmunt A, et al: Postdischarge medication compliance of inpatients converted from an oral to a depot neuroleptic regimen. Psychiatric Services 46:1049-1054, 1995
20.
Dunbar J: Issues in assessment, in New Directions in Patient Compliance. Edited by Cohen S. Lexington, Mass, Lexington Books, 1979
21.
Melnikow J, Kiefe C: Patient compliance and medical research: issues in methodology. Journal of General Internal Medicine 9:96-105, 1994
22.
Scottish Schizophrenia Research Group: The Scottish first episode schizophrenia study: II. pimozide versus fluphenthixol. British Journal of Psychiatry 150:334-338, 1987
23.
Young JL, Zonana HV, Shepler L: Medication noncompliance in schizophrenia: codification and update. Bulletin of the American Academy of Psychiatry and Law 14:105-122, 1986
24.
Kemp R, Hayward P, Applewhaite G, et al: Compliance therapy in psychotic patients: a randomised controlled trial. British Medical Journal 312:345-349, 1996
25.
Dencker SJ, Liberman RP: From compliance to collaboration in the treatment of schizophrenia. International Clinical Psychopharmacology 9(suppl 5):75-78, 1995
26.
Dimatteo MR: Achieving Patient Compliance. New York, Pergamon, 1982
27.
Blow F, Ullman E, Barry K, et al: The Effectiveness of Specialized Treatment Programs for Veterans With Serious and Persistent Mental Illnesses:3-Year Follow-Up Study. White Paper Report to the Department of Veterans Affairs. Washington, DC, US Department of Veterans Affairs, May 6, 1997
28.
Liberman RP (ed): Medication-Management Module: Social and Independent Living Skills: Trainer's Manual. Los Angeles, Clinical Research Center for Schizophrenia and Psychiatric Rehabilitation, 1988
29.
Wallace CJ, Liberman RP, MacKain SJ, et al: Effectiveness and replicability of modules for teaching social and instrumental skills to the severely mentally ill. American Journal of Psychiatry 149:654-658, 1992
30.
Feinstein A, Cicchetti D: High agreement but low kappa: I. the problems of two paradoxes. Journal of Clinical Epidemiology 43:543-549, 1990
31.
Cicchetti D, Feinstein A: High agreement but low kappa: II. resolving the paradoxes. Journal of Clinical Epidemiology 43:551-558, 1990
32.
Fleiss J: The measure of interrater agreement, in Statistical Methods for Rates and Proportions. New York, Wiley, 1981
33.
Sackett D: Future applications and hypotheses from old research, in New Directions in Patient Compliance. Edited by Cohen S. Lexington, Mass, Lexington Books, 1979
Information & Authors
Information
Published In
History
Published online: 1 August 1998
Published in print: August 1998
Authors
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBGet Access
Login options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).