Lack of Representation in Psychiatric Research: A Data-Driven Example From Scientific Articles Published in 2019 and 2020 in the American Journal of Psychiatry

Given the lack of description of sociodemographic data of individual (human) samples, meta-analyses, animal studies, commentaries, editorials, reviews, and letters were excluded, resulting in the identification of 125 scientific articles. Two independent raters (S.L.P., P.P., C.E.V., and K.L.) reviewed all components of each article and supplementary material for information on sociodemographic variables (age, assigned sex at birth, and racial, ethnic, sexual, and gender identities) and description of these questions. Raters also searched the entire article for words (e.g., discrimination, disparities, inequities, race, sexual orientation, generalizability, representation) and topics (e.g., depression in Black adults) related to examination of health inequities and inclusion of lack of generalizability or representation as a limitation. The infrequent discrepancies (e.g., raters disagreed 2.4% [3/125] of the time about whether the article examined health inequity, and 19% [24/125] of the time about representation/generalizability being mentioned) were resolved by a third rater.

Raters coded the type (e.g., genetic, neuroimaging) and the location (e.g., United States, United Kingdom) of the study. Data are described separately for genetic and nongenetic studies, given previous research showing that genetic studies almost exclusively include participants of European ancestry (>78%) (11) and separately for studies conducted within and outside of the United States to allow for comparison with 2019 U.S. Census data population estimates (12). Studies conducted within the United States were assessed for over- or underrepresentation of participants with specific sociodemographic characteristics by comparing to local U.S. Census estimates for articles where a specific location or region was provided (N=19 for race/ethnicity; N=25 for sex/gender). Articles for which a location could not be determined were tested against national U.S. Census estimates. Z-tests of the difference of proportions using the prop.test function in R were run to assess whether the probability of the numbers of female individuals/women and people from specific racial and ethnic groups were higher or lower in the published articles compared to the U.S. Census data (13, 14). The z-tests provide a 95% confidence interval and chi-square with a p value.

We examined representation across a recent 2-year period (2019–2020) in a high-impact psychiatry journal. We found that minoritized racial and ethnic groups, especially Asian American and Latino/Hispanic participants, and female participants/women were underrepresented in these articles. Our results also showed that descriptions of sociodemographic characteristics were frequently missing, use of inaccurate labels was prevalent, and acknowledgment of lack of representation as a limitation was uncommon.

To increase diversity within research samples, investigators can connect with community-engaged researchers at their universities and read the existing literature (e.g., 15) to learn about best practices for partnering with minoritized populations. Researchers should also review the sociodemographic characteristics of participants in an ongoing fashion, like other enrollment criteria, and adjust as needed to ensure adequate representation based on local population estimates. Additionally, it is imperative to collect genetic data from ancestrally diverse populations and interpret these data in culturally informed ways. There are published recommendations to guide researchers (16).

Our results also found missing or inaccurate language about race and ethnicity in the articles. First, less than half of the articles provided the racial composition of the sample, limiting our ability to even evaluate representation. Further, within the studies that provided descriptors for racial identities, minoritized racial and ethnic groups may be being homogenized into an “other/nonwhite” category, as this group was overrepresented compared to U.S. Census data in 49% of the articles (and underrepresented in 5%). This is an example of “white centering” in research (17, 18), which assumes similarity of all minoritized racial and ethnic groups into a single group in relation to White participants. Additionally, “Caucasian,” a label with a racist history (19), was used as a racial category in 45% of articles.

Despite the Institute of Medicine stating that assessing and reporting sexual and gender identities is important (20), only one article reported the sexual identity of participants, no studies clarified if they were reporting on assigned sex or gender identity, and descriptions of samples defaulted to an assumed gender binary. Researchers should review how they are assessing sociodemographic characteristics to ensure that accurate and affirming questions are being used to collect information on sexual and gender identities (as well as racial and ethnic identities). For the 1.2 million people in the United States who have a minoritized gender identity (21), binary response options (e.g., only man or woman) are often inaccurate and hurtful. The PhenXToolkit (22) provides examples of how to assess sociodemographic characteristics.

Despite lack of representation or missing data related to sociodemographic characteristics, there is significant value in utilizing existing data for secondary data analyses. However, when interpreting results and writing manuscripts, directly acknowledging these limitations and framing conclusions accordingly is crucial. This will spur further innovative research and increase rigor and reproducibility. Authors are also called upon to be thoughtful about conducting and interpreting subgroup analyses. It is imperative to place results within a sociohistorical context that recognizes structural inequities experienced by minoritized individuals and communities and to define race, ethnicity, and gender as social constructs. Additionally, if scientists more clearly and accurately describe the sample characteristics and questions used, meta-analyses can be utilized to examine determinants of health inequities by pooling across samples that were not initially powered to examine processes within and across subpopulations.

In addition to individual investigators, scientific journals and funding agencies (e.g., the National Institutes of Health [NIH]) can foster investment in recruiting representative samples and describing sociodemographic characteristics. For example, the Journal states on its website, “The methods section should provide a comprehensive description of the nature of the study group.” However, in the articles reviewed here, this guideline was largely not followed, indicating that expanded guidelines and provision of concrete requirements for characterizing sample sociodemographic characteristics (e.g., sample description must include racial and ethnic identities, gender identities, and age) may be beneficial. Resources can also be added to journal and funding agency websites. For example, the American Psychological Association has published guidelines that are available (23, 24) for reducing bias in scientific language, and there are published recommendations on how to report racial and ethnic identities in medical and scientific journals (25). Journals and funding agencies could also include specialty reviewers to provide critiques focused on inaccurate language, interpretation of results, and representation, ensuring that conclusions are framed within this context. This will also educate scientists and improve future research.

The lack of research (<2% of articles) on health inequities/disparities in 2019 and 2020 in a top-tier psychiatry journal is another urgent issue that can be remedied by journals and funding agencies. Our findings dovetail with a recent review of over 26,000 articles published in top-tier psychology journals that found that around 5% of the articles reviewed focused on outcomes for minoritized racial groups (26). Individuals from minoritized identities are more likely to conduct health equity research, which is subject to publication bias (27), subsequently contributing to NIH funding inequities (28), further hampering the careers of scientists already facing discrimination (28). As NIH pushes to fund more health equity science and scientists call for NIH to address long-standing funding inequities (29), journals could set benchmarks (and track progress over time) for inclusion of health equity research.

While the present study utilized a large sample of 125 recently published articles, several limitations are worth noting. First, we focused on articles only from 2019 and 2020 and one journal. We chose these years because they include recent research that was likely accepted for publication prior to the murder of George Floyd, which occurred in May 2020. We examined one high-impact journal to allow for in-depth review of each article for multiple sociodemographic characteristics. This approach also allows our findings to serve as a “baseline” for the Journal to track progress over time and provides an example that other journals can follow, as these issues are not specific to this journal. We also did not know the specific location of all U.S. studies, which diminished the specificity of the representation results for these studies.

Despite these limitations, the present study significantly advances understanding about representation and scientific language in psychiatric research. Further research that integrates the identities of authors, reviewers, and editorial board members and examines change over time in comparison to this “baseline” is needed. We hope that these results and conclusions encourage action and further discussion across levels of the scientific community (e.g., researchers, community members, institutions, journals, funding agencies).

The authors acknowledge Denis McCarthy, Ph.D., for his feedback on the manuscript.