Twitter Article Mentions and Citations: An Exploratory Analysis of Publications in the American Journal of Psychiatry

To the Editor: Research impact is assessed partly by how the research has made demonstrable contributions to academia and society at large. Most funding agencies rely on journal metrics to judge research impact and guide funding decisions, yet these metrics can be a poor measure of individual article impact. While citation numbers of individual articles have emerged as a surrogate measure, these can take time to accrue. Consequently, article-level research impact is not known until well after publication. Web-based alternative metrics—commonly known as altmetrics—collect data on the frequency of online mentions of a publication from web-based services and can be applied to gauge almost immediate research impact. Various social media platforms provide the bulk of altmetric data, with Twitter yielding the richest data set. As research has yet to evaluate the role of Twitter in assessing research impact in psychiatry, we investigated this association in a 2-year sample of publications from the American Journal of Psychiatry.

Altmetric data scraping from altmetric.com was performed with R statistical software (version 3.1.1) using the “rAltmetric” package (http://CRAN.R-project.org/package=rAltmetric). On the same date (Oct. 20, 2015), Twitter mention counts for all Journal print articles published in 2013 and 2014 were retrieved (N=438), and corresponding article citation counts were collected from Web of Science^TM. Articles were then sorted chronologically based on the date they were first tracked by Altmetric. A previously developed test (1) was implemented to assess the relationship between altmetrics and citations, accounting for time bias (i.e., older articles have more citation opportunities). This test compares a given publication against the publications that chronologically appeared immediately before and after it. A “successful” test occurs when the number of Twitter mentions and citations for a given publication are both higher (or both lower) than the average of Twitter mentions and citations of the two adjacent articles (full analysis R script is available at http://github.com/dsquintana/ajp). Tests revealed 241 successful associations and 108 failures (equal score=89). A binomial test (two-tailed) revealed that the proportion of successful predictions (69.1%) was significantly greater than prediction failures (95% CI=63.9–73.9, p<0.001).

The data suggest that greater frequency of Twitter mentions is associated with more citations and with a higher rate of successful prediction compared with biomedical articles across all fields (57%) (1). While it is difficult to tease apart causality for the associations in the present data, prior research indicates that Twitter mentions precede publication downloads and subsequent citations (2). Thus, the present results provide preliminary evidence that Twitter mentions can be used as an early and accessible proxy of article-level research impact in psychiatry.