With a prevalence approaching 1%, schizophrenia is a leading contributor to overall world health burden (
1). Understanding of schizophrenia by medical students is suboptimal, which may contribute to the stigmatization of patients (
2–
8). In recent years the Internet has been used increasingly as a primary source of information regarding medical conditions (
3,
9). The video sharing Web site YouTube (
www.youtube.com) is the third most popular Web site on the Internet (
10), with over a billion users worldwide (
11). Although the Web site is used as a learning tool by the public, patients, and health care professionals (
12,
13), videos are uploaded by the online user community with no formal vetting of quality (
13).
A number of studies have examined the quality and educational utility of YouTube videos relating to a wide variety of health conditions, with most concluding that YouTube videos are not adequate for use as educational tools in isolation (
12–
24). There has been no similar analysis of the accuracy or quality of YouTube videos relating to mental health conditions. This is an important omission, given that portrayals of mental illness in the media influence public understanding (
25,
26) and may contribute to the damaging stigma surrounding psychiatric diagnoses (
8).
The purpose of this study was to analyze the accuracy of depictions of psychosis in the context of a diagnosis of schizophrenia (hereafter referred to as “acute schizophrenia”) on YouTube, with a specific focus on the depiction of the signs and symptoms of acute schizophreniform psychosis. An additional aim was to assess the educational utility of YouTube portrayals of acute schizophrenia, specifically for teaching medical students about the symptoms and signs of acute schizophrenia.
Methods
YouTube Search
YouTube was searched from March 1, 2015, to March 14, 2015, for all videos claiming to show footage of people exhibiting the signs and symptoms of acute schizophrenia. Search terms with good face validity were used to identify additional search terms by using Google Trends (
www.google.com/trends/) until no novel terms were generated. This search engine may be used to identify the most searched terms on the Internet. The final list of 21 search terms is as follows: “schizophrenia,” “schizophrenia experience,” “schizophrenia interview,” “schizophrenia patient,” “my schizophrenia,” “schizophrenia symptoms,” “untreated schizophrenia,” “schizophrenia case study,” “schizophrenia example,” “paranoid schizophrenia,” “catatonic schizophrenia,” “hebephrenic schizophrenia,” “undifferentiated schizophrenia,” “residual schizophrenia,” “simple schizophrenia,” “negative symptoms schizophrenia,” “positive symptoms schizophrenia,” “cognitive symptoms schizophrenia,” “thought disorder schizophrenia,” “hallucinations schizophrenia,” and “delusions schizophrenia.” YouTube searches were performed by using the default settings and were sorted by relevance. [A figure summarizing an outline of the study methodology is available in an
online supplement to this article.]
A total of 2,820,272 videos were returned from all of the YouTube searches. For each YouTube search, the first 200 videos (ten pages) were assessed against predefined eligibility criteria (N=4,200). The search was restricted to the first ten pages, similar to the methodology of comparable studies, because it was deemed unlikely that medical students would continue searching after that point (
14,
22).
Eligibility Criteria
The main inclusion criterion was that a video claimed to show a presentation of acute schizophrenia. Exclusion criteria were videos that did not claim to show footage of a person exhibiting the signs and symptoms of acute schizophrenia, duplicated videos, non–English language videos, videos of children, videos with <10 seconds (s) of relevant footage, and performing arts videos made by students for school projects.
Eligible videos were defined as those meeting the inclusion criterion and not meeting any of the exclusion criteria. Within the sample of eligible videos, unique presentations of people with acute schizophrenia were defined as single cases for further analysis. If a single video contained more than one unique presentation of acute schizophrenia, each presentation was analyzed as a separate case. If unduplicated content of a single person with acute schizophrenia was shown in multiple videos, the presentations were analyzed as a single case.
Case Rating
Cases meeting eligibility criteria were independently assessed by two consultant psychiatrists (O-MT and AB) for probable psychiatric diagnosis, psychopathological content, and educational utility. All videos were watched on full-screen mode, such that raters were blinded to the video title, description, viewer comments, upload date, viewer ratings, and video author. Cases were rerated by both raters after 22 days.
Adequacy criteria and psychiatric rating.
Eligible cases were rated as to whether they contained sufficient information to make a probable diagnosis and to allow a rating of psychopathological content.
Options for psychiatric diagnosis rating were schizophrenia (any ICD-10 subtype), unspecified psychosis, nonpsychotic behavioral disturbance, mania with psychotic symptoms, mania without psychotic symptoms, factitious disorder, depressive episode, acute anxiety, and schizophrenia-like psychotic disorder due to psychoactive substance use. Raters could also indicate whether there was insufficient information to make a probable diagnosis, in which case the case would be deemed inadequate for further analysis.
Psychopathological content was assessed on 13 symptom domains of particular relevance to acute psychosis (
Table 1). For each symptom domain, raters could indicate whether the symptom or sign was present or absent or whether there was insufficient information in the video to provide a rating. A symptom or sign was deemed to be present if it was rated as being present by at least one rater. The symptom domains assessed overlap with the seven symptoms and signs that constitute the positive scale of the Positive and Negative Syndrome Scale (PANSS) for schizophrenia (delusions, conceptual disorganization, hallucinatory behavior, excitement, grandiosity, suspiciousness, and hostility) (
27), which is a standard tool used in research for assessment of psychotic symptoms in schizophrenia.
Cases with insufficient information to make a probable diagnosis or to assess the presence or absence of symptoms for at least two of the 13 psychopathological domains were deemed to have inadequate content for further analysis (“inadequate cases”). Cases with sufficient information both to make a probable diagnosis and to assess the presence or absence of symptoms for at least two psychopathological domains were deemed to have adequate content for further analysis (“adequate cases”).
Educational utility.
Videos of adequate cases were further assessed for educational utility. The educational utility of the videos was defined pragmatically. For each adequate case, the raters answered a yes-or-no question, “Considering the quality and content of this video, would you consider using it in a medical student teaching session as an illustration of the signs and symptoms of acute psychosis in schizophrenia?” Videos receiving a “yes” response by both independent raters were considered to have good educational utility.
Original Video Features
Original video features used in further analyses included video duration, view counts, whether the video was made for medical education purposes (as stated in the video title or description), and the number of viewers who rated the video positively by using YouTube’s rating system (thumbs-up).
Statistical Analysis
For statistical analysis, adequate cases were grouped into those given a schizophrenia diagnosis by both raters or those given a diagnosis other than schizophrenia by both raters (cases that received a diagnosis of schizophrenia by only one rater were excluded from subgroup analysis). Differences in original video features and ratings by consultants between adequate and inadequate cases and between cases of schizophrenia and all other diagnoses were assessed by using independent-samples, two-tailed t tests (comparing means between groups) and Fisher’s exact test (comparing proportions between groups). Differences are deemed statistically significant for p values that are less than .05 after adjustment for the number of simultaneous statistical tests being performed on the sample (Bonferroni correction for multiple comparisons).
The comparison of original video features included three variables (view count, duration, and whether the video was originally made for medical education purposes), resulting in a significance threshold of p<.017. The comparison of consultant psychopathology ratings included 13 variables (persecutory delusions, grandiose delusions, nihilistic delusions, passivity phenomena, auditory-verbal hallucinations, visual hallucinations, formal thought disorder, flow-of-thought abnormality, mood disturbance [a domain encompassing one or more features, including anxiety, irritability, hostility, elation, and depression], inappropriate affect, bizarre behavior, negative symptoms, and cognitive symptoms), resulting in a significance threshold of p<.004.
Interrater and intrarater reliability was assessed by using Cohen’s kappa (κ) coefficient (
28). Kappa values were interpreted according to criteria defined by Landis and Koch, with values from .00 to .20 indicating slight agreement; .21 to .40, fair agreement; .41 to .60, moderate agreement; .61–.80, substantial agreement; and .81–1.00, almost perfect agreement (
28). Statistical analysis was conducted by using MatLab (MATLAB and Statistics Toolbox Release 2013b).
Results
Video Properties
Of the initial 4,200 videos assessed, 55 videos met eligibility criteria. Videos were excluded because they did not claim to show footage of a person exhibiting the signs and symptoms of acute schizophrenia (N=3,106, 74%) or were duplicated videos (N=464, 11%), non–English language videos (N=125, 3%), videos of children (N=169, 4%), videos with <10 s of relevant footage (N=82, 2%), and performing arts videos (N=253, 6%).
The 55 videos varied widely in their view count (mean±SD=127,222±271,637 views; range 54–1,243,550), positive reviews (239±584 thumbs-up ratings; range 0–3,481), and duration of relevant footage (184.3±187.8 s; range 10–759 s).
Four eligible videos contained more than one unique case, and eight eligible videos contained unduplicated content of a single case, resulting in 58 unique cases. Thirty-five cases (60%) were deemed by both raters to meet adequacy criteria (κ=.346). Adequate cases were significantly longer than inadequate cases (227.3±203.4 s versus 83.9±120.8 s, p=.004) and had more views (193,962±330,569 versus 15,168±26,901, p=.013). There was no significant difference in the proportion of adequate and inadequate cases that were sourced from videos originally made for educational purposes (31% [N=11] versus 52% [N=12]).
The psychopathological content of adequate cases was largely nonspecific
(Table 1). Thirty-seven percent (N=13) of the purported cases of acute schizophrenia did not portray even a single symptom or sign of the positive syndrome of schizophrenia (as defined by PANSS criteria [
27], with agreement by both raters).
Diagnostic Accuracy
Of the 35 adequate cases, 12 (34%) were rated to have a probable diagnosis of schizophrenia by both raters, and 13 (37%) received diagnoses other than schizophrenia from both raters (κ=.444). The remaining ten cases received a diagnosis of schizophrenia from only one rater. All subsequent analysis was performed on the cases that received either a diagnosis of schizophrenia (N=12) or a diagnosis other than schizophrenia (N=13) from both raters.
Taking both consultant ratings into account, the prevalence of probable schizophrenia subtype diagnoses among schizophrenia cases was as follows: paranoid schizophrenia (59%), undifferentiated schizophrenia (13%), hebephrenic schizophrenia (8%), residual schizophrenia (4%), and unspecified schizophrenia (17%). Prevalence of probable diagnoses in the cases with diagnoses other than schizophrenia was as follows: unspecified psychosis (54%), nonpsychotic behavioral disturbance (23%), mania with psychotic symptoms (8%), factitious disorder (8%), depressive episode (4%), and schizophrenia-like psychotic disorder due to psychoactive substance use (4%). In total, 26 cases (74%) were consistent with a psychotic presentation (schizophrenia, mania with psychotic symptoms, drug-induced psychosis, or unspecified psychosis) by both raters (κ=.533).
There were no significant differences between the subgroups (schizophrenia and diagnosis other than schizophrenia) in length of video (209±124.2 s versus 299±287.7 s) or view counts (290,048±381,563 versus 186,124±366,200). Although a greater proportion of schizophrenia cases versus cases of other diagnoses were sourced from videos explicitly intended for medical education purposes (50% [N=6] versus 8% [N=1], p=.030), this difference did not remain significant at the significance threshold adopted for multiple comparisons (p<.017).
Psychopathology
Compared with cases with a diagnosis other than schizophrenia, schizophrenia cases had a significantly higher prevalence of persecutory delusions (83% [N=10] versus 15% [N=2], p=.001), inappropriate affect (75% [N=9] versus 8% [N=1], p=.001), and negative symptoms (83% [N=10] versus 15% [N=2], p=.001)
(Table 1). Seventy-five percent (N=9) of schizophrenia cases presented at least one symptom or sign of the positive syndrome of schizophrenia, and 67% (N=8) presented multiple positive symptoms or signs (as defined by PANSS criteria [
27]). No single psychopathological feature was able to significantly predict classification of schizophrenia versus cases of other diagnoses in a univariate logistic regression analysis.
Educational Utility
Of all adequate cases, 16 (46%) were deemed by both raters to have good educational utility (κ=.407). There was a significant difference between the proportions of cases of schizophrenia versus other diagnoses that were deemed to have good educational utility (83% [N=10] versus 15% [N=2], p=.001).
Intrarater Reliability
Importantly, intrarater reliability (day 1 versus 22) was substantial for both educational-utility (κ=.77) and diagnosis (κ=.76) ratings for adequate cases (
28).
Discussion
This study is the first to systematically assess the accuracy, psychopathological content, and educational quality of videos purporting to show acute schizophrenia on the video-sharing Web site YouTube. Our main findings were that eligible videos were largely inaccurate, containing psychopathological features not specific to schizophrenia; that only 21% of eligible cases were deemed to accurately represent acute schizophrenia; and that in the subset of cases that accurately depicted acute schizophrenia, the disorder was portrayed as a condition of persecutory delusions, inappropriate affect, and negative symptoms. Forty percent of eligible cases were deemed to contain inadequate information to make an assessment of psychopathological content, and in the subgroup that did permit an assessment of psychopathology, less than half of the cases were deemed to have good educational utility.
These findings are important for psychiatrists, given that the Internet has a vast amount of medical information that is easily accessible to medical students (
9,
13). Medical schools currently place increasing emphasis on self-directed learning, and many students turn to Web sites like YouTube as an alternative to textbooks (
12). In addition to watching videos of clinical case presentations, students are likely to use YouTube for other educational resources, such as lectures and presentations. Analysis of these additional educational resources, as well as medical students’ Internet use patterns, is beyond the scope of this study and may warrant future investigation. Recent studies have warned that YouTube videos vary widely in their educational utility and medical accuracy in a number of clinical domains (
12–
24), yet no studies have assessed the quality of psychiatric content on the Web site.
Our study did not investigate how the quality and accuracy of YouTube videos affect understanding of schizophrenia among medical students or the public or how these effects may have a negative impact on patient experiences. Nevertheless, other work has highlighted that public sources of inaccurate information about psychiatric conditions may contribute to stigma and its negative effects on patients’ quality of life and treatment (
8) and that portrayals of schizophrenia in popular media can influence public understanding of the condition (
25,
26). Recently it has been argued that many videos on YouTube that purport to relate to psychiatry are highly critical of the specialty (
29). But more encouraging, it is also true that appropriately selected video footage of patients with schizophrenia may be helpful in improving understanding and reducing stigma among medical students (
30,
31).
A key strength of this study was the exhaustive nature of the YouTube search, resulting in 4,200 videos being considered for analysis against predefined inclusion and exclusion criteria. Moreover, videos were rated in a blinded manner by two independent expert raters on two occasions.
Our study had some key limitations. First, despite the exhaustive nature of our initial YouTube search, only a small number of videos met predefined inclusion and exclusion criteria, perhaps reflecting narrow eligibility criteria. The narrowness of the eligibility criteria, however, is a reflection of the fact that our study explicitly focused on videos depicting psychosis in the context of schizophrenia, rather than videos relating to schizophrenia more generally. Second, interrater agreement in the analysis of video adequacy and probable diagnosis was modest, likely due to the short duration of most case presentations (227.3 s and 83.9 s for adequate and inadequate cases, respectively). Third, although YouTube is the largest video-sharing Web site on the Internet, the fact is that whether our conclusions generalize to other similar Web sites remains an open question. Fourth, our study excluded videos of patients in remission. Consequently, it may be argued that the videos that we deemed eligible showed only the most severe cases and did not offer an optimistic or recovery-oriented picture of the disorder. Conversely, our exclusion criteria allowed us to exclude many irrelevant and inappropriately titled videos. Finally, our study did not examine to what extent YouTube videos affect the attitudes and understanding of viewers, although other studies provide evidence that similar videos can change attitudes about schizophrenia (
30,
31). The relationship between misrepresentations of mental illness on the Internet and public understanding of these conditions will be an important focus for future work.