Online Reviews of Mental Health Treatment Facilities: Narrative Themes Associated With Positive and Negative Ratings
Abstract
Objective:
Previous studies indicate that patients’ satisfaction with mental health care is correlated with both treatment outcomes and quality of life. The aims of this study were to describe online reviews of mental health treatment facilities, including key themes in review content, and to evaluate the correlation between narrative review themes, facility characteristics, and review ratings.
Methods:
United States National Mental Health Services Survey (N-MHSS) facilities were linked to corresponding Yelp pages, created between March 2007 and September 2019. Correlations between review ratings and both machine learning–generated latent Dirichlet allocation topics and N-MHSS–reported facility characteristics were measured by using Spearman’s rank-order correlation coefficient. Significance was defined by a Bonferroni-adjusted p<0.001.
Results:
Of 10,191 unique mental health treatment facilities, 1,383 (13.6%) had relevant Yelp pages with 8,133 corresponding reviews. The number of newly reviewed facilities and the number of new reviews increased throughout the study period. Narrative topics positively correlated with review ratings included caring staff (Spearman’s ρ=0.39) and nonpharmacologic treatment (ρ=0.16). Topics negatively correlated with review ratings included rude staff (ρ=−0.14) and safety and abuse (ρ=−0.14). Of 126 N-MHSS survey items, 11 were positively correlated with review rating, including “outpatient mental health facility” (ρ=0.13), and 33 were negatively correlated with review rating, including accepting Medicare (ρ=−0.21).
Conclusions:
Narrative topics provide information beyond what is currently collected through the N-MHSS. Topics associated with positive and negative reviews, such as staff attitude toward patients, can guide improvement in patients’ satisfaction and engagement with mental health care.
HIGHLIGHTS
•
This study utilized natural language processing of 8,133 online patient reviews, in combination with review ratings and National Mental Health Services Survey data, to understand themes associated with positive and negative ratings of mental health care.
•
Themes associated with positive ratings included caring staff, nonpharmacologic treatment, and therapeutic alliance, while those associated with negative ratings included concerns about safety and abuse, pharmacotherapy, and poor communication with family.
•
The study’s limitations included that online reviews skew toward extreme opinions and that the only available national-level data for comparison are survey data that do not include patients’ ratings, perspectives, or outcomes.
In the United States, approximately one in five people experience a mental illness each year, and less than 50% receive treatment (1). Although most evidence points to increasing provision of treatment over time, studies also show an increasing mental health burden both nationally and internationally (2–4). Factors contributing to the treatment gap include not only underfunding—in 2015, the mental health burden in the United States was 2.7 times greater than the proportion of health funds allocated to mental health (3)—but also patients’ engagement in their own care. Patients’ engagement affects treatment retention and therapeutic outcomes (5). Although engagement depends on patients’ perceptions of the quality of care they receive, few studies have analyzed factors associated with positive or negative experiences of mental health treatment (6, 7). Patients’ satisfaction is associated with fewer psychiatric symptoms and better quality of life (8).
The implementation of quality measures in mental health care lags behind that in other fields of medical care (9). The majority of the quality measures that have been endorsed by the U.S. National Quality Forum and used in major quality reporting programs, such as the Centers for Medicare and Medicaid Services’ Hospital Compare, are related to screening and assessment rather than patient-centered outcomes (10). According to the Patient-Centered Outcomes Research Institute, end users of research, including patients, should be involved in defining outcomes that are “meaningful and important to patients and caregivers” (11). Psychiatric hospitals included in Hospital Compare and evaluated through the Inpatient Psychiatric Facility Quality Reporting Program do not report star ratings, and patients who “received psychiatric or rehabilitative services” are excluded from the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) because the “instrument is not designed to address . . . the behavioral health issues pertinent to psychiatric patients” (12, 13). The Substance Abuse and Mental Health Services Administration (SAMHSA) collects data on mental health treatment facilities through the National Mental Health Services Survey (N-MHSS), but the survey does not include quality measures or patients’ satisfaction measures (14).
Online review sites, such as Yelp, Google, and Facebook, provide platforms for word-of-mouth communication about health care, including mental health services (15–23). Risks with using online review data include fraudulent reviews and overrepresentation of extreme opinions. Online reviews do, however, offer advantages: they allow for narrative reporting, are public, and are available to both patients and their support networks (15, 24). Studies have demonstrated that review ratings and narrative themes tend to correlate with both existing national surveys, such as HCAHPS and the National Survey of Substance Abuse Treatment Services, and, in some cases, with health outcomes (16, 21, 25). Review themes also provide insight into drivers of satisfaction, as measured by review ratings, that are not captured by standard surveys (16, 18, 19, 21, 22, 25).
To identify factors associated with patients’ or their families’ positive or negative experiences with mental health care, we analyzed Yelp reviews of mental health facilities registered in SAMHSA’s Treatment Locator. As of March 31, 2020, Yelp had 35 million average monthly mobile app users and 211 million reviews (26). Given Yelp’s extensive usership and the lack of standardized patient quality and satisfaction measures in U.S. mental health care, patients may turn to the site when selecting or evaluating a mental health facility. Understanding the factors that influence user reviews of facilities could inform areas of focus for reducing unmet mental health needs in the United States. The purpose of this analysis was threefold: to understand the extent to which Yelp is used to review mental health treatment facilities, to describe narrative themes in Yelp reviews of mental health treatment, and to identify narrative themes and facility services correlated with review ratings.
Methods
This study was considered exempt by the University of Pennsylvania Institutional Review Board. The Yelp data used in the study represent an academic data set generated directly by Yelp for research and include all U.S. facilities tagged as “health” facilities according to Yelp’s developer documentation (27). The data set is updated daily and extends beyond content available through the Yelp application programming interface. SAMHSA facility data and N-MHSS data are freely available for download from the SAMHSA website.
Study Sample
To evaluate online reviews for a validated group of facilities, we sought to match SAMHSA mental health facilities to their corresponding Yelp pages. This matching afforded the added advantage of allowing for comparison between Yelp review ratings and the N-MHSS data associated with each SAMHSA facility.
We downloaded all SAMHSA data and Yelp data in September 2019. Facilities were added to Yelp between March 2007 and September 2019. Facility data included name, location, and reviews, including both review star ratings (one to five stars) and narrative review content.
From SAMHSA’s Treatment Locator, we identified 10,191 unique mental health facilities, with listings last updated between February 2017 and September 2019. Facilities are eligible for registration in the Treatment Locator if they are funded by a state mental health agency, administered by the U.S. Department of Veterans Affairs, or licensed by a state agency or national organization to provide mental health treatment. Eligible facilities are identified through surveys of state mental health authorities and medical organizations. All included facilities complete the N-MHSS, and, according to the 2018 N-MHSS, the “survey universe” of mental health treatment facilities was 14,159 (14).
Each SAMHSA facility was matched to a corresponding Yelp page according to a probabilistic string-matching algorithm with a sensitivity of 88.9% and a specificity of 95.1% (28, 29). The list of SAMHSA mental health facilities successfully matched to Yelp pages included medical centers with psychiatric units. In a majority of such cases, the corresponding Yelp page was for the parent organization, and most of the reviews pertained to general medical care rather than mental health care. We used a term search strategy to identify and remove facilities with reviews including high proportions of terms specific to general medical care (rather than mental health care) (19). This step was validated through a combination of hand coding and topic modeling. (Further details are available in an online supplement to this article.)
Review Analysis
To understand Yelp usage over time, we calculated the number of new reviews and the number of facilities receiving a first review by yearly quarter. To evaluate review content, we used a machine learning–based natural language processing model called latent Dirichlet allocation. Latent Dirichlet allocation operates under the assumption that documents, in this case Yelp reviews, can be described by a prespecified number of topics. Topics are defined by distributions of commonly co-occurring words, and each review is then defined by a distribution of topics. A topic most strongly defined by the words “staff,” “helpful,” and “friendly” might then then be highly representative of a review such as, “Great place!! Front desk staff were very helpful.” Two authors (D.C.S., H.M.) independently reviewed the 10 words and reviews most strongly associated with each of the 30 latent Dirichlet allocation topics and, where review content was consistent, assigned the topics a theme. Discrepancies were adjudicated by a third reviewer (R.M.M.). This process resulted in identification of 13 consistent topics. This approach, and the ratio of meaningful topics to all latent Dirichlet allocation topics, was consistent with prior Yelp studies (16, 19, 21).
Survey Analysis
Patients’ ratings of mental health treatment facilities may be correlated with existing objective facility metrics, as has proven true in hospitals, substance use treatment facilities, and nursing homes (16, 21, 25). In mental health care, for instance, evidence from qualitative analyses suggests that patients have concerns specific to pharmacologic and inpatient treatment (30, 31). The N-MHSS is an annual survey conducted by SAMHSA and is the only national survey of both public and private mental health treatment facilities (14). The survey includes 18 service categories divided into 126 binary service codes and covers such information as facility type, special programming, and availability of emergency mental health services. To identify how narrative analysis of review content may support or augment data currently collected by SAMHSA, we sought to measure the correlation between N-MHSS survey items and Yelp review ratings.
Statistical Analysis
To compare Yelp star ratings with both N-MHSS services and latent Dirichlet allocation topics, we used Spearman’s rank-order correlation coefficient, which is suited to comparisons of nonparametric data. Star ratings are ordinal, ranging from 1 to 5, while latent Dirichlet allocation probability distributions are generally nonnormally distributed continuous variables between 0 and 1. N-MHSS services are binary variables. Both the correlations between N-MHSS services and star ratings and between star ratings and latent Dirichlet allocation probabilities were performed at the level of the Yelp review. Significance was defined by a Bonferroni-corrected p<0.001, corresponding to a corrected α=0.0000072 (32). All statistical analyses were performed by using R, version 3.6.1 (33).
Results
Study Sample
Of the 10,191 unique SAMHSA facilities, 2,403 (23.6%) were matched to Yelp “health” facilities. Of these matches, 1,383 (13.6%) remained after filtering by general medical terms (a flow diagram is available in the online supplement ). As summarized in Table 1, the final sample differed meaningfully from SAMHSA mental health facilities without dedicated Yelp pages by geographic distribution, with overrepresentation of facilities located in western states, and on several survey items. Among service settings, hospital inpatient and partial hospitalization or day treatment were overrepresented, as was special programming for specific groups, such as young adults and veterans.
| Facilities with Yelp reviews | Facilities without Yelp | ||||
|---|---|---|---|---|---|
| (N=1,383) | reviews (N=8,808) | ||||
| Characteristic | N | % | N | % | p |
| Regionb | <.001 | ||||
| Northeast | 254 | 18.4 | 1,941 | 22.0 | |
| South | 358 | 25.9 | 2,682 | 30.4 | |
| Midwest | 303 | 21.9 | 2,303 | 26.1 | |
| West | 468 | 33.8 | 1,806 | 20.5 | |
| Service settingc | |||||
| Hospital inpatient | 396 | 28.6 | 1,233 | 14.0 | <.001 |
| Outpatient | 1,113 | 80.5 | 7,012 | 79.6 | .478 |
| Partial hospitalization or day treatment | 369 | 26.7 | 7,544 | 14.4 | <.001 |
| Residential | 218 | 15.8 | 1,237 | 14.0 | .097 |
| Telemedicine/telehealth | 513 | 37.1 | 3,358 | 38.1 | .481 |
| Special programming for specific groupsc | |||||
| Young adults | 356 | 25.7 | 1,907 | 21.7 | <.001 |
| Seniors | 460 | 33.3 | 2,300 | 26.1 | <.001 |
| Veterans | 291 | 21.0 | 1,436 | 16.3 | <.001 |
| Lesbian, gay, bisexual, or transgender clients | 426 | 30.8 | 1,838 | 20.9 | <.001 |
| Persons with co-occurring mental and substance use disorders | 827 | 59.8 | 4,430 | 50.3 | <.001 |
| Persons living with HIV/AIDS | 204 | 14.8 | 920 | 10.4 | <.001 |
| Persons who experienced trauma | 681 | 49.2 | 3,844 | 43.6 | <.001 |
| Children or adolescents with serious emotional disturbances | 492 | 35.6 | 3,268 | 37.1 | .287 |
| Persons living with serious mental illness | 721 | 52.1 | 4,327 | 49.1 | .040 |
| Emergency mental health servicesc | |||||
| Crisis intervention team | 643 | 46.5 | 4,481 | 50.9 | .003 |
| Psychiatric emergency walk-in services | 512 | 37.0 | 2,914 | 33.1 | .004 |
a
Proportions were compared by using chi-square tests.
b
Among facilities without Yelp reviews, 76 were excluded from region totals because they were in American Samoa, Puerto Rico, or the Virgin Islands.
c
Categories are not mutually exclusive, and, therefore, differences were tested individually.
The facilities had a total of 8,133 reviews, corresponding to a mean of 5.88 reviews per facility (range 1–107, median=3). The mean±SD word count per review was 170.2±164.8. The distribution of review ratings was bimodal, with 57.5% one-star (lowest possible) and 26.5% five-star (highest possible) reviews. As shown in Figure 1, both the number of reviews and the number of facilities receiving a first review increased from 2006 to 2019.
Review Analysis
Of the 13 latent Dirichlet allocation topics, four were positively correlated with review ratings, seven were negatively correlated with review ratings, and two were not significantly correlated with review ratings, according to a Bonferroni-corrected p<0.001. The topics, redacted sample reviews, and correlations with review ratings are presented in Table 2. The strength of correlations was generally greater for positive topics, for example, “caring staff” (Spearman’s ρ=0.39), than for negative topics, such as “rude ancillary staff” (ρ=−0.14).
| Topic | Example reviewa | Spearman’s ρ |
|---|---|---|
| Positively correlated with review rating | ||
| Caring staff | “Everyone here [is] so nice and so helpful . . . from the front staff all the way to the doctors! . . . I’m very happy with the service I have received and continue to receive. Thank you everyone.” | 0.39 |
| Life-saving treatment | “If you are ready to get better and you work the program you will succeed! Staff is caring and compassionate. . . . They will fight for you, which at times you will hate, but in the end, it will save your life.” | 0.28 |
| Nonpharmacologic treatment modalities | “I did outpatient individual therapy, dietitian support . . . and . . . group therapy all in one building. . . . [Every] provider I worked with really genuinely cared for me.” | 0.16 |
| Therapeutic alliance with primary provider | “Dr M is the best. He listens and he shows concern as well as empathy. . . . I would highly recommend Dr M.” | 0.11 |
| Negatively correlated with review rating | ||
| Rude ancillary staff | “For how amazing and incredible the residential staff is, the front desk staff is arrogant, rude, and comes across as not caring. Front desk is the first impression. . . . It’s a shame it’s being ruined by your front desk staff.” | –0.14 |
| Safety and abuse | “I left with more psychological problems than when I arrived. The staff are not trained. . . . I laughed when they tried to stop bullying since the staff were the ones bullying children with name calling and brutal physical assaults.” | –0.14 |
| Billing and insurance | “I was lied to about insurance coverage . . . told multiple times that my insurance would cover it all and then got billed. . . . They’ll only take your insurance if you work for certain companies.” | –0.11 |
| Scheduling | “My family member has called almost every week to schedule an appointment now that she did her intake, and each time she calls they tell her that she has not been assigned a counselor and they transfer her to . . . voicemail. None of her messages have been returned. . . . It shouldn’t take 3 months to get an appointment. . . . Every time we ask to speak to the director, we are told that they are not in.” | –0.09 |
| Treatment for specific diagnosis | “Brought a person here who needed help with drug dependency and depression. . . . Patient was treated as if they were a criminal being booked into a jail. Still in search of dignified and compassionate care!” | –0.08 |
| Communication with family | “No communication with the families. No support for the families. When we picked up our loved one we were given ZERO information. No info about options of what the next steps should be once the patient leaves.” | –0.08 |
| Pharmacotherapy | “My counselor was great . . . but the Doctor got me hooked on antidepressants and after 4 years tried to discharge me after I missed some counseling appts. I ended up . . . thanking my counselor and told her that I would not be back and that I would ween myself off the meds.” | –0.06 |
| Nonsignificantly correlated with review ratingb | ||
| Child and adolescent treatment | “I was 14 years old when I was at this place. I hated the way the staff would treat us kids, they are there to help us not destroy our happiness. . . . Continuously put residents down as if they were trash. I’m almost 30 now and I’m still scarred and scared of that place.” | 0.01 |
| Inpatient facility amenities | “Their food is horrible. . . . Beds are trash. Worst ‘hospital’ ever. Extremely limited food, outside activity, phone calls, visitation, recreational exercise. The nurses go in your room every 5 seconds slamming doors open and shut in your shared room.” | –0.05 |
a
Elements of the reviews have been redacted in order to preserve user anonymity.
b
Correlation is considered significant if p is below the Bonferroni-corrected alpha of 0.001 (p<(0.001/139)=7.2 × 10–6).
Reviews most represented by the generally positive topic “nonpharmacologic treatment modalities” (ρ=0.16) included mentions of “individual therapy,” “yoga,” “dietitian support,” “group therapy,” “dialectical behavior skills,” “family workshops,” and “trauma program(s).” Those for the generally negative topic “safety and abuse” (ρ=−0.14) mentioned specific concerns about “name calling” and being “violated . . . verbally,” “physical assault” by both patients and staff, “sexual assault,” “theft,” and “neglect.” Of the seven topics negatively correlated with review rating, two were related to clerical services: “billing and insurance” (ρ=−0.11) and “scheduling” (ρ=−0.09).
Survey Analysis
Facilities offered, on average, 39.3±10.9 of the 126 N-MHSS services. In total, 11 services were significantly correlated with five-star reviews, and 33 were significantly correlated with one-star reviews. Table 3 shows the 10 services with strongest positive and negative correlations to Yelp review ratings. Among facility types, “outpatient mental health facility” (ρ=0.13) and “residential treatment center for adults” (ρ=0.07) were positively correlated with review rating, whereas “psychiatric hospital” (ρ=−0.19) was negatively correlated with review rating.
| Service | Spearman’s ρ |
|---|---|
| Most positively correlated with review rating | |
| Facility type | |
| Outpatient mental health facility | .13 |
| Residential treatment center for adults | .07 |
| Facility operation (e.g., private, public): Veterans Affairs Medical Center | .05 |
| Exclusive services: serves veterans only | .05 |
| Service settings: outpatient | .06 |
| Special programs/groups offered | |
| Lesbian, gay, bisexual, or transgender clients | .10 |
| Persons with posttraumatic stress disorder | .06 |
| Ancillary services | |
| Housing services | .07 |
| Intensive case management | .06 |
| Psychosocial rehabilitation services | .05 |
| Most negatively correlated with review rating | |
| Payment, insurance, funding accepted | |
| Medicare | –.21 |
| Medicaid | –.18 |
| Military insurance | –.16 |
| State welfare or child and family services funds | –.12 |
| Facility type: psychiatric hospital or psychiatric unit of a general hospital | –.19 |
| Service settings: hospital inpatient | –.18 |
| Emergency mental health services: psychiatric emergency walk-in services | –.17 |
| Language services: services for the deaf and hard of hearing | –.16 |
| Treatment approaches: psychotropic medication | –.15 |
| Tobacco screening services: screening for tobacco use | –.13 |
a
N-MHSS, National Mental Health Services Survey; SAMHSA, Substance Abuse and Mental Health Services Administration. All p values are below the Bonferroni-corrected alpha of 0.001 (p<(0.001/139)=7.2 × 10–6).
As with the latent Dirichlet allocation topic “nonpharmacologic treatment modalities” (ρ=0.16), N-MHSS codes for special programs and for ancillary services—for example, programming for “lesbian, gay, bisexual or transgender clients” (ρ=0.10) and “housing services” (ρ=0.07)—were generally positively correlated with review rating. On the other hand, as with the latent Dirichlet allocation topic “billing and insurance,” N-MHSS codes for specific forms of payment and insurance, especially public insurance—for example, “Medicare” (ρ=−0.21)—were generally negatively correlated with review rating, as were both the latent Dirichlet allocation topic “pharmacotherapy” (ρ=−0.06) and the N-MHSS code “psychotropic medication” (ρ=−.15).
Discussion
This study had three main findings. First, individuals are increasingly reviewing mental health facilities online. The percentage of SAMHSA facilities identified on Yelp (13.6%) was between the percentage of hospitals (31.0%) and skilled nursing facilities (10.4%) identified on the site in prior studies (16, 25). The number of Yelp pages and reviews of mental health facilities is expected to continue to increase: in the 4 months between June and September 2019, 806 Yelp pages were added under the site’s “counseling and mental health” tag. It is noteworthy that facilities in western states and facilities offering inpatient treatment were overrepresented on Yelp. Yelp is based in California, which may explain the former, and large facilities may be both more likely to offer inpatient services and more likely to have Yelp pages. The bimodal distribution of online review ratings, with high proportions of one-star and five-star reviews, was consistent with distributions observed for other health facilities (16, 19). In the absence of formal outlets for patients to report their experiences of mental health care, we expect the volume of patients choosing to do so through more informal means, such as online reviews, to continue to grow.
Second, correlations between review ratings, narrative topics, and N-MHSS services were consistent with previous findings in both qualitative analyses of mental health care and analyses of online reviews. An analysis of online reviews of substance use treatment facilities in Pennsylvania identified the positive review theme “life-changing experiences” and the negative review theme “medication needs,” consistent with themes we identified as correlated positively (“life-saving treatment”) and negatively (“pharmacotherapy”) with online reviews of mental health facilities (21). In the study of substance use treatment facilities, the strength of the correlations reported between themes and review ratings was slightly larger than those reported here, perhaps because of a combination of fewer latent Dirichlet allocation topics and binary treatment of review ratings. Studies of inpatient psychiatric care point to patients’ concerns around safety, restrictions on freedom, communication, and stigma (34, 35). In our analysis, there were more negative comments associated with inpatient services than with outpatient services, and reviews with poor ratings were more likely to mention issues of “safety and abuse,” “communication with family,” and “rude ancillary staff.” Themes associated with more positive ratings, such as “therapeutic alliance with primary provider” and “caring staff,” also correspond to existing evidence: a study of 61 inpatients found that “feeling cared for” and “positive qualities of staff” were themes common to positive patient appraisals of care (34). In general, evidence suggests that patient satisfaction is perhaps most dependent on staff-patient communication (36). Communication with staff has been consistently identified as a theme in online reviews of health care, including in reviews for emergency departments, hospitals, and substance use treatment facilities (16, 18, 21). Among mental health care facility reviews, we found that topics related to communication, namely “caring staff” and “rude ancillary staff,” were those most strongly correlated with positive and negative review ratings, respectively.
Third, narrative topics identified in reviews include actionable items that may improve satisfaction with mental health care. Satisfaction is inherently important—how patients feel about the care they receive should matter to providers—but in mental health care in particular, it may also be associated with fewer psychiatric symptoms and better quality of life (8). Evidence regarding the relationship between satisfaction and health outcomes in medical care is mixed but generally supports positive correlations between patients’ evaluations of communication with both doctors and nurses and more objective measures of quality (25, 36, 37). Many negative reviews of mental health care facilities mentioned “rudeness” explicitly. While “rudeness” is not unique to mental health care, it may contribute to patients’ internalized stigma, which is associated with decreased help-seeking and disempowerment among those with mental illness (38, 39). Mental health care facilities could use the information from Yelp reviews to guide patient-centered interventions. For instance, negative reviews consistently mentioning rude staff could prompt clinic managers to organize targeted communication and antistigma training, with the potential to improve patient engagement in mental health care.
Limitations inherent to online reviews include that reviews skew toward extremes and that fake reviews may escape algorithmic detection. In addition, Yelp users are not representative of those needing mental health care. According to Yelp, U.S. users are almost equally distributed between the ages of 18–34 years, 35–54 years, and 55 years and older. Users skew heavily toward at least some college education (82%), and half of users have an income above $100,000 a year (26). According to a 2018 SAMHSA report, “any mental illness” was most common among those ages 18–49 years, and, among those ages 18 years and older, approximately 64% had at least some college education and 24% were living below the poverty line (1). Yelp is therefore at least overrepresentative of those with higher levels of education and income. Yelp notably does not report on the racial-ethnic demographics of its users, and several racial-ethnic groups in the United States, particularly Black and Latinx people, face persistent barriers to mental health care (40). In the future, dialogue with those who have lived experience of mental health treatment and those with specific barriers to care will be necessary to determine how well Yelp themes encompass factors contributing to patients’ satisfaction.
This study was further limited by a relatively low proportion of SAMHSA facilities with Yelp pages. Google and Facebook reviews are not accessible through application programming interfaces but may include facilities not found on Yelp (21). Subanalyses by facility type, for example, inpatient versus outpatient and adult versus child, were not conducted because of limitations in sample size. These analyses, and evaluations of emergency psychiatric care and of psychiatric telehealth communication, will be important as further online review data becomes available.
A limitation of natural language processing is that identified themes cannot always be easily addressed (23). For instance, despite a negative correlation between pharmacologic treatment and review ratings, in many cases, evidence supports the use of medication in treating mental illness. Knowledge of patients’ aversion to medication, however, can help set the foundation for therapeutic alliance.
Finally, although we were able to compare facility ratings to standard facility-level characteristics through N-MHSS data, there is no nationally reported facility rating system for mental health facilities in the United States (such as Hospital Compare for hospitals) against which to measure Yelp review ratings. Evidence from studies where such comparisons were possible point to correlations ranging from 0.09 to 0.50 (16, 20).
Conclusions
Online reviews are powerful in that they are unscripted, and, as such, the themes that arise from these review narratives are naturally patient centered. Our study’s findings regarding review ratings and the narrative content of reviews are consistent with both national survey data and the available literature. In the absence of the widespread adoption of validated patient-centered outcome measures in mental health, online reviews of mental health facilities have the potential to guide interventions in order to improve patient satisfaction.
Footnote
This study was supported by funding from the National Institutes of Health and the National Institute on Drug Abuse (grant R21 DA050761).
Supplementary Material
File (appi.ps.202000267.ds001.pdf)
- View/Download
- 1.82 MB
References
1.
Key Substance Use and Mental Health Indicators in the United States: Results From the 2018 National Survey on Drug Use and Health. Rockville, MD, Center for Behavioral Health Statistics and Quality, Substance Abuse and Mental Health Administration, 2019. www.samhsa.gov/data/release/2018-national-survey-drug-use-and-health-nsduh-releases. Accessed Dec 9, 2019
2.
Jorm AF, Patten SB, Brugha TS, et al: Has increased provision of treatment reduced the prevalence of common mental disorders? Review of the evidence from four countries. World Psychiatry 2017; 16:90–99
3.
Vigo DV, Kestel D, Pendakur K, et al: Disease burden and government spending on mental, neurological, and substance use disorders, and self-harm: cross-sectional, ecological study of health system response in the Americas. Lancet Public Health 2019; 4:e89–e96
4.
Kohn R, Saxena S, Levav I, et al: The treatment gap in mental health care. Bull World Health Organ 2004; 82:858–866
5.
Dixon LB, Holoshitz Y, Nossel I: Treatment engagement of individuals experiencing mental illness: review and update. World Psychiatry 2016; 15:13–20
6.
Tambuyzer E, Van Audenhove C: Is perceived patient involvement in mental health care associated with satisfaction and empowerment? Health Expect 2015; 18:516–526
7.
Ratner Y, Zendjidjian XY, Mendyk N, et al: Patients’ satisfaction with hospital health care: identifying indicators for people with severe mental disorder. Psychiatry Res 2018; 270:503–509
8.
Vermeulen JM, Schirmbeck NF, van Tricht MJ, et al: Satisfaction of psychotic patients with care and its value to predict outcomes. Eur Psychiatry 2018; 47:60–66
9.
Institute of Medicine Committee on Crossing the Quality Chasm: Improving the Quality of Health Care for Mental and Substance-Use Conditions. Washington, DC, National Academies Press, 2006. www.ncbi.nlm.nih.gov/books/NBK19830. Accessed Oct 17, 2019
10.
Kilbourne AM, Beck K, Spaeth-Rublee B, et al: Measuring and improving the quality of mental health care: a global perspective. World Psychiatry 2018; 17:30–38
11.
Frank L, Basch E, Selby JV: The PCORI perspective on patient-centered outcomes research. JAMA 2014; 312:1513–1514
12.
Inpatient Psychiatric Facility Quality Reporting Program Manual. Baltimore, Centers for Medicare & Medicaid Services, 2020. https://www.qualityreportingcenter.com/globalassets/ipf-tools-and-resources/ipfqr-program-manual/ipfqr_programmanualv5.1_20200102_vfinal508c.pdf
13.
CAHPS Hospital Survey (HCAHPS) Quality Assurance Guidelines. Baltimore, Center for Medicare & Medicaid Services, 2018. hcahpsonline.org/globalassets/hcahps/quality-assurance/2018_qag_v13.0.pdf
14.
National Mental Health Services Survey (N-MHSS): 2018 Data on Mental Health Treatment Facilities. Rockville, MD, Substance Abuse and Mental Health Services Administration, 2019. wwwdasis.samhsa.gov/dasis2/nmhss/NMHSS-2018-C.pdf
15.
Merchant RM, Volpp KG, Asch DA: Learning by listening-improving health care in the era of Yelp. JAMA 2016; 316:2483–2484
16.
Ranard BL, Werner RM, Antanavicius T, et al: Yelp reviews of hospital care can supplement and inform traditional surveys of the patient experience of care. Health Aff 2016; 35:697–705
17.
Bardach NS, Lyndon A, Asteria-Peñaloza R, et al: From the closest observers of patient care: a thematic analysis of online narrative reviews of hospitals. BMJ Qual Saf 2016; 25:889–897
18.
Kilaru AS, Meisel ZF, Paciotti B, et al: What do patients say about emergency departments in online reviews? A qualitative study. BMJ Qual Saf 2016; 25:14–24
19.
Agarwal AK, Mahoney K, Lanza AL, et al: Online ratings of the patient experience: emergency departments versus urgent care centers. Ann Emerg Med 2019; 73:631–638
20.
Johari K, Kellogg C, Vazquez K, et al: Ratings game: an analysis of Nursing Home Compare and Yelp ratings. BMJ Qual Saf 2018; 27:619–624
21.
Agarwal AK, Wong V, Pelullo AM, et al: Online reviews of specialized drug treatment facilities—identifying potential drivers of high and low patient satisfaction. J Gen Intern Med 2020; 35:1647–1653
22.
Graves RL, Goldshear J, Perrone J, et al: Patient narratives in Yelp reviews offer insight into opioid experiences and the challenges of pain management. Pain Manag 2018; 8:95–104
23.
Gibbons C, Greaves F: Lending a hand: could machine learning help hospital staff make better use of patient feedback? BMJ Qual Saf 2018; 27:93–95
24.
Cognetta-Rieke C, Guney S: Analytical insights from patient narratives: the next step for better patient experience. J Patient Exp 2014; 1:20–22
25.
Ryskina KL, Andy AU, Manges KA, et al: Association of online consumer reviews of skilled nursing facilities with patient rehospitalization rates. JAMA Netw Open 2020; 3:e204682–e204682
26.
Fast Facts. San Francisco, Yelp, 2020. www.yelp-press.com/company/fast-facts/default.aspx. Accessed July 16, 2020
27.
All Category List—Yelp Fusion. San Francisco, Yelp, 2020. www.yelp.com/developers/documentation/v3/all_category_list. Accessed Nov 14, 2019
28.
Van der Loo MP: The stringdist package for approximate string matching. R J 2014; 6:111–122
29.
McKenzie G, Janowicz K, Adams B: Weighted multi-attribute matching of user-generated points of interest. Proceedings of the 21st ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems. New York, Association for Computing Machinery, 2013
30.
Shields MC, Stewart MT, Delaney KR: Patient safety in inpatient psychiatry: a remaining frontier for health policy. Health Aff 2018; 37:1853–1861
31.
McHugh RK, Whitton SW, Peckham AD, et al: Patient preference for psychological vs pharmacologic treatment of psychiatric disorders: a meta-analytic review. J Clin Psychiatry 2013; 74:595–602
32.
Kern ML, Eichstaedt JC, Schwartz HA, et al: From “Sooo excited!!!” to “So proud”: using language to study development. Dev Psychol 2014; 50:178–188
33.
R: A Language and Environment for Statistical Computing. Vienna, R Foundation, 2019. www.R-project.org/
34.
Chevalier A, Ntala E, Fung C, et al: Exploring the initial experience of hospitalisation to an acute psychiatric ward. PLoS One 2018; 13:e0203457.
35.
Staniszewska S, Mockford C, Chadburn G, et al: Experiences of in-patient mental health services: systematic review. Br J Psychiatry 2019; 214:329–338
36.
Manary MP, Boulding W, Staelin R, et al: The patient experience and health outcomes. N Engl J Med 2013; 368:201–203
37.
Fenton JJ, Jerant AF, Bertakis KD, et al: The cost of satisfaction: a national study of patient satisfaction, health care utilization, expenditures, and mortality. Arch Intern Med 2012; 172:405–411
38.
Clement S, Schauman O, Graham T, et al: What is the impact of mental health-related stigma on help-seeking? A systematic review of quantitative and qualitative studies. Psychol Med 2015; 45:11–27
39.
Wang K, Link BG, Corrigan PW, et al: Perceived provider stigma as a predictor of mental health service users’ internalized stigma and disempowerment. Psychiatry Res 2018; 259:526–531
40.
Cook BL, Trinh N-H, Li Z, et al: Trends in racial-ethnic disparities in access to mental health care, 2004–2012. Psychiatr Serv 2017; 68:9–16
Information & Authors
Information
Published In
History
Received: 21 April 2020
Revision received: 1 August 2020
Accepted: 25 September 2020
Published online: 21 May 2021
Published in print: July 01, 2021
Keywords
Authors
Competing Interests
Dr. Beidas has received royalties from Oxford University Press and has consulted for Merck and the Camden Coalition of Healthcare Providers. She currently consults for United Behavioral Health and serves on the scientific advisory committee for Optum Behavioral Health. The other authors report no financial relationships with commercial interests.
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBLogin options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).