Clinician Factors Related to Outcome Differences Between Black and White Patients at CMHCs
Abstract
Objectives:
Data are limited on how clinicians contribute to outcome differences between black patients and white patients. Because the clinician-patient relationship is the foundation of mental health services, understanding clinicians' role in outcome differences may help identify evidence-based interventions that decrease disparities and capitalize on positive differences. Symptoms and functioning in a sample of black and white adults receiving outpatient services were examined to determine the effects of their primary clinician on those patterns.
Methods:
The study included 551 patients (25% black) with serious mental illness and 62 mental health professionals (21% black) identified as the patients' primary clinician. Treatment outcomes were measured at baseline and two follow-ups (two and four months) with the Behavior and Symptom Identification Scale, a measure of symptoms and functioning. Data were analyzed with hierarchical linear modeling. Clinicians' levels of multicultural competence, burnout, and education were analyzed.
Results:
Clinicians moderated the relationship between patient race and outcome differences. There was significant variability among clinicians: approximately 20% had black patients whose outcomes were worse than those of their white patients, and 40% had black patients with better outcomes than their white patients. The only clinician factor predicting these differences was clinician's general experiences and relationships with people from racial-ethnic and cultural groups other than their own.
Conclusions:
The occurrence of outcome differences varied across clinicians, with some clinicians magnifying outcome differences between black and white patients and others minimizing them. Factors other than clinicians' race, multicultural competence, education, and burnout may contribute to outcome differences between black and white patients. (Psychiatric Services 62:525–531, 2011)
The foundation for most mental health services is the relationship between the clinician providing services and the patient receiving services (1). There is an a priori theory that for black patients this relationship is less than optimal, mirroring the racial tensions in society and contributing to inferior treatment outcomes for black patients compared with white patients (2–7). This is based on an implicit assumption that black patients differ from white clinicians in their life experiences, which create subtle and complex personal constructions of racial identity that diverge significantly from those constructed by white clinicians (2–6).
Two clinician factors are frequently posited to either produce or ameliorate outcome differences for black patients: clinicians' level of multicultural counseling competency (MCC) and their race (3,8–10). In a content analysis covering 20 years of MCC research, Worthington and colleagues (11) found some evidence that such competence improves clinicians' skills with patients from nonwhite racial-ethnic and cultural minority groups. Two significant weaknesses can be found in these studies. First, 70% of the studies were descriptive field surveys. Second, 80% of the studies relied on “pseudo-patients,” leaving some question about the impact of MCC in naturalistic settings (11,12). Shin and colleagues (13) conducted a meta-analysis of research on racial matching and found that matched and unmatched clinician-patient dyads were similar in service retention and total number of sessions. In general, matching appears to be a weak approach to addressing disparities in mental health services, but no research has examined the impact of matching on patients' symptoms and functioning over time.
Levels of clinician burnout and education have been shown to affect treatment outcomes (14–16). For this study we hypothesized that these factors also influence outcome differences—higher levels of burnout by decreasing the overall functioning of clinicians and thereby affecting outcomes negatively for black patients, who may require an extra effort to ensure the appropriate provision of services, and higher levels of education through greater exposure to MCC via classroom and continuing education experiences (17). The few studies that have examined clinician factors that may affect treatment outcomes for black patients have neglected to answer the following basic questions that are addressed in this study. Are clinicians empirically linked to the occurrence of outcome differences between black and white patients? Does a clinician's race and levels of burnout, education, and MCC have an impact on the occurrence of outcome differences? What proportion of clinicians produce outcome differences between black and white patients?
Methods
Participants were drawn from 13 community mental health centers (CMHCs) (18–21) located in the Upper Mississippi River Basin (UMRB). The southern region of the UMRB, comprising southern Illinois, western Kentucky, eastern Missouri, and northern Arkansas, has one of the highest concentrations of rural blacks outside the deep South (22). The UMRB provides a unique opportunity to study racial differences in a predominantly rural context. The CMHCs are located in micropolitan statistical areas (micro areas) or core-based statistical areas (CBSA). Micro areas contain at least one urban center of between 10,000 and 50,000 people as well as “adjacent outlying counties” that are socially and economically integrated with the urban center through commuting (23). The CBSA is a more expansive category, containing a metropolitan statistical area and one or more adjacent micro areas that are integrated by close proximity.
Data were collected between 2007 and 2009. CMHC staff introduced interested patients to a research team member located on site who provided an overview of the study. After consent was obtained, baseline data were collected and arrangements were made for follow-up data collection. Up to two follow-ups were conducted at two-month intervals. Clinicians were invited to participate via mail and one group meeting held at each CMHC. All patients were in treatment at the time of the baseline measure. (There was no random or purposeful nonrandom assignment of patients to clinicians.) All methods were approved by the University of Illinois at Urbana-Champaign Institutional Review Board.
Instruments
Clinicians.
Clinicians completed three standardized instruments: the Universal Orientation Scale (UOS), Multicultural Counseling Inventory (MCI), and Maslach Burnout Inventory (MBI). The 20-item UOS (24) and the 43-item MCI (25) both capture aspects of MCC. The UOS was developed to measure nonprejudice and has moderate positive correlations with measures of empathy and adequate test-retest and internal consistency reliability coefficients (24). The MCI is used in a wide body of MCC research and consistently demonstrates adequate psychometric properties (25,26). The four MCI subscales measure multicultural skills, awareness, relationships, and knowledge. The MBI is a widely used measure of burnout that has been normed on mental health workers (27). It has three subscales: emotional exhaustion, depersonalization, and personal accomplishment. Clinicians' race and ethnicity and other demographic variables, such as gender and education, were self-reported.
Patients.
Treatment outcomes were measured with the Behavior and Symptom Identification Scale (BASIS-24), a self-report instrument that measures difficulty in several major symptom and functional domains (28–30). It is used across the diagnostic spectrum in outpatient settings to measure the outcome of interventions over time. It has been normed with black and white patients (28–30). Respondents are asked to indicate the degree of difficulty they have experienced during the past week. Patients' race and ethnicity were self-reported, along with gender, marital status, education, and employment status.
Administrative data
Patients' diagnosis, utilization of medication, number of months in service at the CMHC at the time of baseline measure, receipt of psychosocial rehabilitation services, and hours of service received during the 90 days before the final follow-up measure were gathered from administrative data. Patients were provided with a variety of service packages tailored to their needs, which may have included individual counseling, group therapy, psychosocial rehabilitation, or medication. The possible range of services available to any patient was similar across CMHCs. A variety of therapeutic models shaped these services; the most common modalities were cognitive-behavioral therapy and insight-oriented psychotherapy. Each patient's primary clinician was identified by use of administrative records.
Statistical analysis
The analysis was similar to analyses utilized in the therapist effects literature (31–36) and was conducted with HLM software, version 6.06 (37). Full maximum likelihood estimation was used, which permits model comparison tests of nonnested models (38). The model contained three levels: repeated measures of the BASIS-24 at level 1, patient demographic and service variables at level 2, and clinician variables at level 3. Conceptually, level 1 represents treatment outcomes measured over time for each patient. At level 2, each patient is described by his or her baseline score (intercept), treatment outcomes (slope), and demographic and service predictors. Level 3 represents the influence of clinicians on outcome differences between white and black clients.
Results
Clinicians
Table 1 presents data on clinician characteristics. Sixty-two clinicians who were treating at least one white and one black patient in the study were included in the analysis. The median number of patient participants per clinician was eight (mean±SD=8.89±5.24). The median number of black patient participants per clinician was two (mean±SD=2.26±2.06). Clinicians provided therapeutic sessions, coordinated care, made referrals, and monitored patient progress.
Patients
Table 2 presents data on patient characteristics. Of the 551 patients in the study, most relied on Medicaid and had received CMHC services for several years. A total of 340 patients (62%) completed the baseline interview and the two follow-up interviews. During the study, nine patients (one black) declined to continue participation. Black and white patients differed on a number of characteristics, including hours of services received during the study period; black patients received more service hours than white patients. Differences in demographic characteristics and service hours between black and white patients were included as controls in the final model.
Model comparisons
An unconditional growth curve model was fit to the data in the first step of the model-building process (Table 3, model 1). [Additional data from this analysis are presented in supplemental Tables 6 and 7 in an online appendix to this article at ps.psychiatryonline.org.] This model included no predictors of treatment outcomes other than time; however, it did control for the clustering of observations within patients as well as the clustering of patients within clinicians (38–40). Each subsequent model was compared with the unconditional model to determine which provided the better fit. This was accomplished using a chi square test of the deviance statistics with degrees of freedom equal to the difference in the number of parameters estimated (38). The variance component associated with treatment outcomes (Table 3) denotes how well each model fit the data, with higher numbers indicating greater amounts of random (unexplained) variance between the observed and predicted treatment outcomes (38). Subsequent models sought to further reduce the unexplained variance through the inclusion of patient and clinician variables.
The second model represents the effect of patient-level variables on treatment outcomes (Table 3). [Tables 8 and 9 in the appendix at psychyatryonline.org present additional data.] Although this model controlled for the clustering of patients within clinicians, no clinician predictors were included (37,38). Compared with the unconditional model, model 2 explained an additional 27% of variability in treatment outcomes (40). The deviance statistics showed that model 2 also fit significantly better than model 1 (χ2=19.61, df=8, one-tailed p<.01).
Model 3, the final model, included all of the patient-level predictors identified in model 2 plus the clinician-level predictors (Tables 4 and 5). Compared with the unconditional model (model 1), the final model explained an additional 37% of variability in treatment outcomes and provided a significantly better fit to the data (χ2=29.67, df=21, one-tailed p<.05). Model 3 did not fit the data significantly better than model 2.
Model specification
All patient-level variables (Table 2) were tested for inclusion in the final model. Months in service and receipt of psychosocial rehabilitation services were excluded because of multicollinearity with hours of service. Level of education, marital status, and living situation were excluded because they were not essential to the stability of the model and were not significantly associated with outcomes (41).
Of all the patient-level variables tested, only the effect of race significantly varied across clinicians; therefore, it was the only predictor treated as a random effect (39). Subsequent model building was dedicated to describing the random effect of race.
Level 1 model (repeated measures)
Time was coded to represent the measurement occasion (baseline=1, first follow-up=2, and second follow-up=3). The log-time model was selected because it provided a better fit than a linear model of time. The log transformation was necessary to meet the linearity assumption and merely changed the scale of the time variable (38) without altering other properties of the model.
Level 2 model (patient level)
Predictors of treatment outcomes at the patient level included a diagnosis of schizophrenia, male gender, use of psychotropic medications, being black, working in a paid or volunteer capacity, having public insurance, age, and the square root of hours of services (42). The original variable for service hours in the prior 90 days was highly skewed. Most of these predictors were coded and interpreted as dummy variables reflecting the presence or absence of that predictor; however, both age and root-hours of treatment were centered on the grand mean. For centered variables, each increment in a positive direction meant that the patient was one year older or received one root-hour more service than the “mean patient” across the entire sample.
The average patient BASIS-24 score at baseline was estimated at 1.49 (Table 4). The average treatment outcome was −.003, which did not differ significantly from zero. The average treatment outcome represented the expected change over time for a patient with values of 0 on all dummy variables and who was of average age and received an average amount of services. Patient-level parameters represent increments or decrements to the average treatment outcomes.
On average, the effect of patient race was not statistically significant; however, this effect was moderated by clinicians, with substantial variability among clinicians in the size and direction of the effect. The random effect of race had an acceptable reliability (Table 5). This finding answered the first research question by indicating that outcome differences did exist between white and black patients and that those differences were linked to clinicians.
Level 3 model (clinician level)
To answer the second research question, clinicians' level of education, MCC, and burnout as well as their race and gender were included at level 3 to determine whether these variables explained the observed difference between white and black patients' outcomes. MCI and UOS scores (MCC) were centered at the grand mean such that each unit increase or decrease in the parameter indicated a clinician's deviation above or below the mean among all clinicians. MBI (burnout) subscale scores were dummy coded, with 1 indicating a score that signified clinician burnout (27). Other dummy variables included male gender, having a graduate degree, and whether the therapist was black.
The only variable that significantly predicted differences in treatment outcomes was the relationship subscale of the MCI. Higher scores on that subscale predicted slightly but significantly decreased differences in treatment outcomes (−.094; Table 4), although substantial variability among clinicians remained unexplained.
To answer the third research question concerning the degree of variability in outcome differences across clinicians, we used the residuals from level 3 to calculate the predicted treatment outcomes for a hypothetical white patient and black patient. Both patients were in the reference category (coded as 0) or at the grand mean for all predictors (other than patient race). For each clinician, the difference between the predicted treatment outcomes for white and black patients was calculated. Negative values indicated that black patients' outcomes were worse than those of white patients (moderate disparities, −.25 to −.50 points on the BASIS-24 5-point scale; substantial disparities, −.51 or fewer points). Values near zero indicated the absence of differences between black and white patients (−.24 to .24 points), and positive values indicated outcome differences that favored black patients (moderate differences, .25 to .50 points; substantial differences, .51 or more points). The categories denote a difference of less than 5% in outcomes between black and white patients was considered clinically insignificant, between 5% and 10% moderately clinically significant, and more than 10% highly clinically significant.
Among the 62 clinicians, seven (11%) produced substantial outcome disparities for their black patients (between −.59 and −1.02 points on the BASIS-24 5-point scale), five (8%) produced moderate outcome disparities for their black patients (between −.26 and −.46 points), 26 (42%) had no or virtually no outcome differences between black and white patients (between −.24 and .24 points), nine (15%) produced moderately better outcomes for their black patients (between .31 and .43 points), and 15 (24%) produced substantially better outcomes for their black patients (between .82 and 1.34 points) (data not shown).
Discussion
This study found that treatment outcomes for black and white patients varied depending on the clinician providing the services. Within this sample of black and white clinicians providing services to both black and white patients, approximately 20% had black patients with worse treatment outcomes than their white patients, and 40% had black patients with better outcomes than their white patients. For the other 40% of these clinicians few or no differences in outcomes between black and white patients were found. The assumption that black patients treated by white clinicians receive less than optimal care does not reflect accurately this continuum of outcomes. This assumption oversimplifies the complex relationships between patients and clinicians and may hinder the development of evidence-based practices that decrease outcome disparities.
Of the clinician factors tested, only a single dimension of MCC (multicultural counseling competency) predicted outcome differences among black and white patients—the degree of positive experiences and relationships a clinician has with people from racial and ethnic groups other than his or her own. Outcome differences did not appear to be readily influenced by clinicians' self-reported race, education, or job burnout or their MCC skills, knowledge, or empathy, as we had hypothesized. Substantial evidence indicates that racial bias is shaped by past experiences and influences behavior in ways beyond an individual's conscious awareness (43). Prejudices as well as positive feelings by whites and blacks about each other are therefore connected to unconscious perceptions of race and ethnicity (44). Our findings support this line of reasoning and suggest that the traditional approach to reducing disparities in mental health services—that is, having all clinicians participate in multicultural competency training—may be less than effective at decreasing the racial bias of the clinicians associated with outcome disparities.
The most significant limitation of the study was the naturalistic setting. Considerable disagreement exists regarding the effects that naturalistic settings may have on outcomes. To examine clinicians' role in producing outcomes, naturalistic settings may be preferable because controlled settings purposely minimize clinician differences, such as those related to patient race (11). Some argue that naturalistic settings introduce serious confounds created by the lack of random assignment of patients to clinicians and the researchers' lack of control over treatment regimens (31,33). In our study, the lack of random assignment meant that the distribution of black patients was uneven; some clinicians saw only one or two black patients. Clinicians whose black and white patients had different outcomes could have been seeing only one or two black patients, which leaves some question as to whether the observed differences were related to patients' race. The race parameter had ample reliability and was statistically significant (Table 5), indicating that the relationship between clinicians and patient race was not an anomaly.
Our lack of control over treatment regimens meant that black patients received significantly more hours of services than white patients. The hours of services received did not vary systematically by clinician. Any differences in hours of services received by black patients were therefore not moderated by clinician factors and are irrelevant to the moderating role clinicians played in the outcome differences associated with patient race.
How variations in the services received and the number of black patients treated by a clinician are related to outcome differences warrants further study. However, insufficient knowledge of these relationships should not prevent further study of clinician functioning in settings such as CMHCs, where naturally occurring variations in services and patient mix are likely. In general, more investigation is needed on the implications of naturalistic settings and the role of modeling for validity (31,33).
Conclusions
Outcome differences experienced by black and white adults with serious mental illness who were receiving mental health services from CMHC clinicians did not consist exclusively of disparities in which black patients had worse outcomes than white patients. Instead, a continuum of differences was observed, from outcomes favoring white patients to those favoring black patients. Clinicians' role in outcome differences and the reasons for that role should be reexamined. Levels of education, burnout, and MCC and clinicians' race appear less important than general experiences and relationships with people from other racial and ethnic groups. Perhaps factors embedded in community dynamics or organizational functioning also play a role, along with experiences and relationships, in shaping clinicians' ability to work effectively with black and white patients (10,45).
Acknowledgments and disclosures
This work was supported by grant R34 MH074640-01A2 from the National Institute of Mental Health.
The authors report no competing interests.
Supplementary Material
File (Larrison_DS.pdf)
- View/Download
- 11.91 KB
References
1.
Elkin I: A major dilemma in psychotherapy outcome research: disentangling therapists from therapies. Clinical Psychology: Science and Practice 6:10–32, 1999
2.
Davis LE, Proctor EK: Race, Gender, and Class: Guidelines for Practice with Individuals, Families, and Groups. Englewood Cliffs, NJ, Prentice Hall, 1989
3.
Ortega AN, Rosenheck R: Hispanic patient-case manager matching: differences in outcomes and service use in a program for homeless persons with severe mental illness. Journal of Nervous and Mental Disease 190:315–325, 2002
4.
Ridley CR: Overcoming Unintentional Racism in Counseling and Therapy: A Practitioner's Guide to Intentional Intervention. Thousand Oaks, Calif, Sage, 2005
5.
Snowden LR: Bias in mental health assessment and intervention: theory and evidence. American Journal of Public Health 93:239–243, 2003
6.
Mental Health: Culture, Race and Ethnicity: A Supplement to Mental Health: A Report of the Surgeon General. Rockville, Md, US Department of Health and Human Services, US Public Health Service, 2001
7.
Hölzer CE, Goldsmith HF, Ciarlo JA: Effects of rural urban county type on the availability of health and mental health care providers; in Mental Health, United States. Edited by, Manderscheid RW, Henderson MJ Rockville, Md, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, 1998
8.
Bickman LA: Measurement Feedback System (MFS) is necessary to improve mental health outcomes. Journal of the American Academy of Child and Adolescent Psychiatry 47:1114–1119, 2008
9.
Institute of Medicine: Improving the Quality of Health Care for Mental and Substance-Use Conditions. Washington, DC, National Academies Press, 2005
10.
Hernandez M, Nesman T, Mowery D, et al.: Cultural competence: a literature review and conceptual model for mental health services. Psychiatric Services 60:1046–1050, 2009
11.
Worthington RL, Soth-McNett AM, Moreno MV: Multicultural counseling competencies research: a 20-year content analysis. Journal of Counseling Psychology 54:351–361, 2007
12.
Arredondo P, Rosen DC, Rice T, et al.: Multicultural counseling: a 10-year content analysis of the Journal of Counseling and Development. Journal of Counseling and Development 83:155–161, 2005
13.
Shin S-M, Chow C, Camacho-Gonsalves T, et al.: A meta-analytic review of racial-ethnic matching for black and Caucasian American patients and clinicians. Journal of Counseling Psychology 52:45–56, 2005
14.
Bednar SG: Elements of satisfying organizational climates in child welfare agencies. Families in Society 84:7–12, 2003
15.
Schulz R, Greenley JR, Brown R: Organization, management, and client effects on staff burnout. Journal of Health and Social Behavior 36:333–345, 1995
16.
Glisson C, Landsverk J, Schoenwald S, et al.: Assessing the Organizational Social Context (OSC) of mental health services: implications for research and practice. Journal of Administration and Policy in Mental Health and Mental Health Services Research 35:98–113, 2008
17.
Larrison CR, Schoppelrey SL, Hadley-Ives E, et al.: Organizational climate and treatment outcomes for African American clients receiving services at community mental health agencies. Administration in Social Work 32:111–138, 2008
18.
Cutler DL, Bevilacqua J, McFarland BH: Four decades of community mental health: a symphony in four movements. Community Mental Health Journal 39:381–398, 2000
19.
Tevis C: Rural health: new help for rural mental health needs. Successful Farming 101:54–, 2003
20.
Torrey EF: Surviving Schizophrenia: A Manual for Families, Patients, and Providers, 5th ed. New York, HarperCollins, 2006
21.
Mental Health: A Report of the Surgeon General. Rockville, Md, US Department of Health and Human Services, US Public Health Service, 1999
22.
Percent of Persons Who Are Black or African American Alone: 2000. Washington, DC, US Census Bureau, 2000
23.
Office of Management and Budget: Standards for defining metropolitan and micropolitan statistical areas: notice. Federal Register 65:82228–82238, 2000
24.
Phillips ST, Ziller RC: Toward a theory and measure of the nature of nonprejudice. Journal of Personality and Social Psychology 73:420–434, 1997
25.
Sodowsky GR, Taffe RC, Gutkin TB, et al.: Development of the Multicultural Counseling Inventory: a self-report measure of multicultural competencies. Journal of Counseling Psychology 41:137–148, 1994
26.
Leong FTL: Appendix C: Definitions and measurement of cross-cultural counseling competencies; in Community Support: Notes From a Roundtable on Conceptualizing and Measuring Cultural Competence. Rockville, Md, National Mental Health Information Center, 1998
27.
Maslach C, Jackson SE, Leiter MP: Maslach Burnout Inventory Manual, 3rd ed. Palo Alto, Calif, Consulting Psychologists Press, 1996
28.
Eisen S, Dill DL, Grob MC: Reliability and validity of a brief patient report instrument for psychiatric outcome evaluation. Hospital and Community Psychiatry 45:242–247, 1994
29.
Eisen SV, Gerena M, Ranganathan G, et al.: Reliability and validity of the BASIS-24 Mental Health Survey for whites, African-Americans, and Latinos. Journal of Behavioral Health Services and Research 33:304–323, 2006
30.
Eisen SV, Normand SLT, Belanger AJ, et al.: The revised Behavior and Symptom Identification Scale (BASIS-24): reliability and validity. Medical Care 42:1230–1241, 2004
31.
Elkin I, Falconnier L, Martinovich Z, et al.: Clinician effects in the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Psychotherapy Research 16:144–160, 2006
32.
Kim DM, Wampold BE, Bolt DM: Clinician effects in psychotherapy: a random-effects modeling of the National Institute of Mental Health Treatment of Depression Collaborative Research Program data. Psychotherapy Research 16:161–172, 2006
33.
Lutz W, Leon SC, Martinovich Z, et al.: Clinician effects in outpatient psychotherapy: a three-level growth curve approach. Journal of Counseling Psychology 54:32–39, 2007
34.
Collins JF, Elkins I: Randomization in the NIMH Treatment of Depression Collaborative Research Program. New Directions for Program Evaluation 28:27–37, 1985
34.
Crits-Christoph P, Mintz J: Implications of clinician effects for the design and analysis of comparative studies of psychotherapies. Journal of Consulting and Clinical Psychology 59:20–26, 1991
36.
Crits-Christoph P, Gallop R: Clinician effects in the TDCRP and other psychotherapy studies. Psychotherapy Research 16:178–181, 2006
37.
HLM 6.06. Lincolnwood, Ill, Scientific Software International, 2004
38.
Singer JD, Willett JB: Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence. New York, Oxford University Press, 2003
39.
Byrk AS, Raudenbush SW: Hierarchical Linear Models: Applications and Data Analysis Methods. Newbury Park, NJ, Sage, 2002
40.
Merlo J, Chaix B, Yang M, et al.: A brief conceptual tutorial of multilevel analysis in social epidemiology: linking the statistical concept of clustering to the idea of contextual phenomenon. Journal of Epidemiology and Community Health 59:443–449, 2005
41.
Little RJA: Modeling the dropout mechanism in repeated-measures studies. Journal of the American Statistical Association 90:1112–1121, 1995
42.
Bush PW, Drake RE, Xie H, et al.: The long-term impact of employment on mental health service use and cost for persons with severe mental illness. Psychiatric Services 60:1024–1031, 2009
43.
Greenwald AG, Banaji MR: Implicit social cognition: attitudes, self-esteem, and stereotypes. Psychological Review 102:4–27, 1995
44.
Plant EA, Devine PG: The antecedents and implications of interracial anxiety. Personality and Social Psychology Bulletin 29:790–801, 2003
45.
Hohmann AA, Parron DL: How the new NIH guidelines on inclusion of women and minorities apply: efficacy trials, effectiveness trials, and validity. Journal of Consulting and Clinical Psychology 64:851–855, 1996
Figures and Tables
Information & Authors
Information
Published In
Cover: The Lee Shore, by Edward Hopper, 1941. Oil on canvas, 28 × 43 inches. Private collection. Photo © Art Resource, New York.
History
Published online: 1 May 2011
Published in print: May 2011
Authors
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBLogin options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).