Potential Benefits of Incorporating Peer-to-Peer Interactions Into Digital Interventions for Psychotic Disorders: A Systematic Review

This review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) (45). Only original, peer-reviewed research articles published in English were included in the review. We identified studies for inclusion through searching the electronic databases PubMed, EMBASE, PsycINFO, Ovid MEDLINE, Cochrane Central Register of Controlled Trials, and Health Technology Assessment Database.

Three sets of keyword search algorithms were used and linked with the Boolean operator AND. The first set was related to serious mental illness and psychotic disorders: “schizophr*” OR “schizoaffective” OR “psychosis” OR “psychotic” OR “psychoses” OR “severe mental illness” OR “serious mental illness” OR “severe mental health” OR “serious mental health” OR “SMI.” The second set was related to digital health interventions: “technolog*” OR “digital” OR “internet” OR “online” OR “Website” OR “Web-based” OR “smartphone” OR “text message” OR “SMS” OR “mHealth” OR “mobile” OR “eHealth” OR “computer.” The third set was related to peer-to-peer interactions: “peer” OR “peer-to-peer” OR “peer support” OR “social network*” OR “virtual communit*” OR “support groups.” Using these criteria, we first screened titles and abstracts of search results. Next, we assessed for eligibility the full text of included articles and screened references of included articles to identify any publications that the search terms had not identified. Finally, we discussed and resolved any disagreements until a consensus was reached about study inclusion.

A key aim of this review was to include all published studies of DIs for psychosis that allowed for peer-to-peer interactions. We defined a DI as Web- or mobile phone–based treatment that aims to improve psychological or behavioral outcomes. Studies were excluded if they did not provide quantitative data on feasibility, acceptability, or preliminary efficacy; if they investigated only participants’ interest in and willingness to receive DIs; if they used digital platforms only for medication adherence and appointment reminders; if they used digital technology for diagnostic assessments or symptom monitoring without providing feedback to participants; if they used digital technology only for telepsychiatry; and if they did not include participants with diagnoses confirmed by a clinician or through an initial assessment. We included studies that used digital health tools with mixed samples of caregivers and service users, although we report characteristics and outcomes only for the participants from the service user sample. We also included studies in which DIs were offered in conjunction with face-to-face therapy or treatment as usual.

From each study, we extracted data on retention, engagement, acceptability, and preliminary efficacy. Retention was defined as the proportion of participants who remained in the study for the entire duration and completed follow-up assessments (when applicable). Engagement was indexed by intervention use and completion rates, such as time spent using the digital health intervention or number of uses per day. Acceptability was assessed through the evaluation of data about satisfaction ratings and qualitative feedback. Preliminary efficacy was assessed through intervention-induced changes in measured outcomes.

We conducted full database searches in February 2017. [A flow diagram showing each stage of the search process is available in an online supplement to this article.] The search strategy returned 3,276 results, providing 1,015 unique citations after duplicates were removed. Of these, 54 full-text articles were reviewed for eligibility. We identified one study from reviewing the references of the included papers. The final number of included papers after screening the title, abstract, and full paper was eight. The eight eligible papers reported data from six independent trials and five interventions, which were reviewed in full. Table 1 summarizes information about the included interventions. [A table in the online supplement provides further details.]

Kaplan and colleagues (46) conducted a randomized controlled trial (RCT) to investigate the efficacy of peer support forums over a 12-month period with 300 participants with mental illness, of whom 22% had a diagnosis of a schizophrenia spectrum disorder. Participants in the active arms were assigned to a Listserv group (N=101) consisting of anonymous group e-mail communication or to a peer-support bulletin board group (N=99). The contents of the Listserv and the online bulletin board were entirely peer directed and not facilitated by staff. The research staff were available only for technical help. The control condition (N=100) consisted of a waiting list, where participants were asked to refrain from using Internet peer support for the duration of their participation. Study attrition was remarkably high: only 18% adhered to the Listserv, 10% to the bulletin board, and 12% to the waitlist. Participants were categorized into high and low participation levels. Those categorized in the “high participation” group at the 12-month, post-baseline point reported having read the messages at least weekly and sent at least five messages over the past year. Individuals in the high participation group showed significantly higher distress levels at four and 12 months postbaseline, whereas those in the low participation group reported less distress at 12 months than at baseline. Participants’ experiences with the ListServ and online bulletin board groups were assessed with the Online Group Questionnaire (47), which included questions with responses on a Likert scale (from 1, never, to 6, always) about how often users thought the discussion topic was relevant, whether they felt supported by the group members, and whether they were satisfied with the groups. The authors used follow-up data, averaged the responses to each of the items, and categorized participants into those with a positive online experience (score ≥3 [N=90]) and those with a less positive experience (score <3 [N=95]). Although the group with positive experiences (49%) was significantly more distressed than the group with less positive experiences, the former group also reported higher perceived social support, relevance, and overall satisfaction at 12 months.

Rotondi and colleagues (48) developed the Schizophrenia Online Access to Resources (SOAR), a Web site that includes psychoeducational forums moderated by clinicians with master’s degrees in social work and with doctoral degrees; access to project team members to ask questions; a library of educational articles; and a list of local community events. In the initial study, Rotondi and colleagues compared the efficacy of SOAR versus treatment as usual for 31 participants with schizophrenia. Throughout the 18 months of the intervention, all 16 of those assigned to SOAR (100%) engaged actively with the Web site (used the educational material on at least four visits and contributed to the forum on at least 13 visits). Participants in the active arm spent a median of 971 minutes (range 162–11,796 minutes) on the Web site. Baseline symptom severity was positively correlated with time using the Web site. On average, participants were active in the forum for 11±4 months, with an average of 3.2 hours per month. The educational material was accessed less frequently, with only 18% of the total visits dedicated to the questions and articles page. On average, participants viewed the educational articles for 8±4 months, with an average of 32±26 minutes per month. SOAR demonstrated high acceptability: 69% participants rated the ease of use as “very much” or “extremely,” and 94% rated the value of the Web site as “very much” or “extremely” (48,49). At the three-month follow-up, SOAR was associated with significant improvements in perceived social support, knowledge about the diagnosis of schizophrenia, and stress; at 12-month follow up an improvement in positive psychotic symptoms was noted (48,49).

Alvarez-Jimenez and colleagues (50) designed a psychosocial platform for FEP called HORYZONS, which includes peer-to-peer online social networking via a newsfeed, interactive tailored psychosocial modules, and expert moderation by health care providers. On the newsfeed, clients and moderators can post comments, upload pictures and videos, and “like” posted content. The newsfeed incorporates an automated system to categorize the discussion threads into relevant themes. By clicking on one of these themes, users can display the activity and posts of other peers and engage them in moderated peer-to-peer messaging. The self-guided psychosocial modules use a strength-based approach to target key factors related to psychotic relapse and early recovery, including psychoeducation, early warning signs of relapse, depression, social anxiety, and stress management. Users' activity in the online psychosocial modules is displayed in the social network. As users advance through the therapy modules, moderated social networking features, such as online problem-solving groups, are progressively unlocked. “Talking points” prompt users to discuss themes through structured phases, following an evidence-based problem-solving framework. The therapeutic objectives of the modules are bolstered through discussion with peers in the social networking environment under guidance from the moderator. Finally, users' experiences and personal coping resources for stressful events are stored and categorized in a database serving as a “wiki” for users.

The initial four-week, pilot, single-arm study evaluated the use, acceptability, and safety for 20 FEP participants (50). The study retention rate was 100%, with 70% using HORYZONS for at least three weeks. Among the participants, 15% completed all seven psychosocial modules, and 60% completed three or more; 95% used the social network, with a total of 371 postings and 4,170 social page views or actions. Regarding acceptability, 75% of participants had a positive experience, 90% would recommend it to others, and 70% found HORYZONS to be useful. All participants (100%) “agreed” or “strongly agreed” that the social networking component was safe and confidential because of its anonymous nature, restricted access, and expert moderation. Of the participants, 90% felt that moderation of peer-support specifically contributed to the safety (51). The inclusion of peer-to-peer communication was particularly appreciated. Regarding preliminary efficacy, the authors found reductions in depression ratings and no significant improvements in anxiety after one month of using HORYZONS. In addition, 60% of participants reported increased perceived social connectedness, and 55% felt empowerment in their recovery process.

Schlosser and colleagues (52) created and evaluated the feasibility of a Personalized Real-Time Intervention for Motivation Enhancement (PRIME). PRIME was designed to target reward-processing impairments, enhance motivation, and improve quality of life in recent-onset schizophrenia. The intervention integrates SMS (short message service)-based motivational coaching from trained therapists, individualized goal setting within important psychosocial domains (health and wellness, creativity, sociality, and productivity), and social networking via direct peer-to-peer messaging and a community “moments feed” to capture and reinforce rewarding experiences and goal achievements. In PRIME, users can set up their profile, describe their interests, and freely select from prepopulated goal categories. Personalized tailored daily challenges associated with such goals are then presented as suggestions. Users can select the amount of time spent engaging in these challenges as well as set reminders. Once users complete their challenges, they have the option to post the goal achievement to the community or to their own private feed. Users can also post “moments” in the form of pictures or text. The daily challenge and moments are displayed on the PRIME community feed and the participants’ personal feed. In the PRIME community feed, participants can view their peers’ profiles and message with them directly. In addition, they can decide to “like” or to comment on moments posted to the community feed by other peers. Finally, each user is assigned to a coach, who provides SMS-based motivational coaching, is available daily, and offers guidance on creating and accomplishing challenges and goals. Coaches can also reach out to the user, with a frequency that depended on the user’s preference, clinical issues, and degree of overall progress toward goal achievement.

The 12-week, single-arm, pilot study showed high feasibility: of the 20 participants diagnosed as having schizophrenia, schizophreniform disorder, or schizoaffective disorder, 100% completed the study (52). In terms of engagement, the average times active per week was 7.52±6.05 and the average challenge completion rate was 85%±16.85%. The most popular psychosocial domain of goals set were health and wellness challenges, with approximately one challenge completed per week, followed by creativity challenges, social challenges, and productivity challenges. Over the course of the 12 weeks, the average number of user-initiated peer interactions was 12.4±15.65, and the average number of user-initiated coach interactions was 56.6±64.35. Finally, PRIME showed high acceptability with an overall satisfaction rate of 8 out of 10.

Biagianti and colleagues (53) developed an integrated platform called Creating Live Interactions to Mitigate Barriers (CLIMB) to improve social functioning among individuals with chronic psychotic disorders. CLIMB integrates computerized, neuroscience-informed, social cognition training; remote weekly group therapy sessions that are held via videoconferencing; and moderated peer-to-peer multimodal messaging. The training exercises treat social cognition deficits by targeting the impaired brain systems underlying social cognition. Briefly, the computerized exercises harness the principles of brain plasticity, employing speeded, accurate, and increasingly more challenging discriminations of socially relevant information (for example, eye gazes, emotional faces, prosody, and social situations). The group teletherapy sessions are led weekly by a master’s-level clinician and based on principles of the recovery model (54) and SMART goals (55). For every session, participants set a SMART goal and report on the goals that they attempted during the previous week. This is followed by psychoeducation and discussions of shared experiences (for example, social isolation, stigma, motivational deficits, and relapse detection). Next, users learn and practice mindfulness skills and cognitive-behavioral techniques. Peer-to-peer multimodal messaging is used between video calls to optimize group teletherapy. The clinician and a moderator use the group text chat to send links and articles about information and topics discussed during the video calls, remind the group of scheduled sessions, and coach and assist participants.

The feasibility of CLIMB was investigated in a six-week, single-arm, pilot study with 27 participants who had a diagnosis of schizophrenia, schizoaffective disorder, or bipolar disorder with psychotic features and who participated in the intervention remotely (53). Study retention was 78%. On average, participants attended 84% of the group therapy sessions, completed a median of 9.5 hours of social cognition training, and posted a median of 5.2 messages per week on the group text chat. In terms of acceptability, participants rated CLIMB in the medium range for enjoyment, ease of use, ease of fit into daily schedule, and perceived benefits. Finally, an exploratory analysis of outcome improvements indicated significant improvements in emotion identification abilities for prosodic happiness, prosodic happiness intensity, and facial anger, with a large within-group effect size. Trend-level improvements were also observed for the psychosocial and motivation and energy aspects of quality of life, whereas no significant improvements were observed for symptoms.

Given the broad definition of DI, we found substantial heterogeneity of aims, form, content, and duration among reviewed studies. In addition, peer-to-peer communication was implemented in various ways, partially because of the technology solutions that were available when studies were conducted. In the study by Kaplan and colleagues (46), peer-to-peer communication was limited (e-mail or bulletin board) and offered as a stand-alone feature. Furthermore, interactions occurred via anonymous accounts and were not supervised by clinicians or coaches. In SOAR, HORYZONS, PRIME, and CLIMB, peer-to-peer communication is one feature of multicomponent DIs, and mental health professionals are available to support, moderate, or facilitate peer-to-peer interactions. In the SOAR peer discussion forums, therapists lead discussions focused on problem solving, alleviating stress, and interacting with peers to develop a supportive forum where users can work together to address problems (48). However, the integration of these forums with the library of educational materials and questions is limited.

Compared with SOAR, the peer-to-peer communication technology used in HORYZONS, PRIME, and CLIMB more closely resembles social media platforms, in that it is more customizable, personalized, interactive, and multimedia. This allows for a high degree of integration with other treatment components. For example, in HORYZONS, users progressing through the online psychosocial modules can use talking points to directly access the social network and put learned skills into practice with other peers (50). In PRIME, users have the ability to share goal achievements and rewarding experiences by using the community “moments feed,” which in turn promote support and encouragement via peer-to-peer interactions (52). In CLIMB, peer-to-peer multimedia messaging is used between video calls to consolidate rapport, create opportunities for social engagement, and encourage participants to share personal content through links, videos, pictures, drawings, poems, and quotes (53). Taken together, these findings suggest that social media technology can be used in creative and flexible manners not only to promote social interactions and support among peers but also to enhance engagement with other treatment components.

All studies reported on retention over the trial period, which ranged from four weeks to 18 months. On average, 78.4% of participants completed the trials, although the heterogeneity between studies was high. When peer-to-peer interactions were not supervised by mental health professionals and not integrated into a multicomponent DI, the retention rate was very low (14%) (46). For the remaining studies, the average retention was very high (94.5%) and similar to the one found in a recent meta-analysis of smartphone apps for schizophrenia (56).

Because intervention use, module completion rates, and other variables of engagement were reported in different ways across studies, data could not be analyzed with meta-analytic methods. Summary statistics were calculated for engagement variables that were consistent across studies.

Two studies reported number of log-ins per week (50,52): on average, users logged in 3.74 times per week during the study period. Four articles reported on rates of social networking use, broadly defined as number of messages, posts, and “likes.” Over the course of 12 months in the study by Kaplan and colleagues (46), participants assigned to the bulletin board posted 2,141 messages in total, whereas Listserv members sent a total of 8,964 messages. The HORYZONS social networking features were used by 95% of the sample (50). The mean number of messages, posts, and “likes” per participant over the one-month pilot study was 208.50. On average, a PRIME participant posted 3.24 messages, comments, or “likes” per week (52). Finally, an average of 10.34 messages were posted per participant per week on the CLIMB messaging platform (53). Therefore, the average rate of social networking use was 16.66 social media actions per participant per week.

Completion rates for the non–social networking features (amount of completed tasks divided by the amount of tasks to be completed or offered) were reported for three DIs. The HORYZONS average psychosocial module completion rate calculated across the seven available modules was 45% (50). The average challenge completion rate in PRIME was 85% (52). CLIMB participants completed on average of 52% of the requested amount of social cognition training. Therefore, the average task completion rate was 47% (53).

Among the reviewed multicomponent DIs, peer-to-peer communication features were the most engaging. In fact, Rotondi and colleagues (48) found that the time spent on forums was significantly higher compared with the time spent on educational activities. Alvarez-Jimenez and colleagues (50) found a higher level of online social networking use (with 52.13 social media actions per participant per week), compared with use of interactive psychoeducational modules (with 17.38 page hits per participant per week). Biagianti and colleagues (53) found that the attendance rate for group therapy sessions was higher than for social cognition training.

Four articles provided data on participants’ characteristics that were associated with engagement patterns. In the study by Kaplan and colleagues (46,49), participants who engaged more in Internet peer support reported higher levels of distress 12 months later, compared with those who had less or no participation. Seven of the ten HORYZONS participants who had treatment-induced reductions in depression scores were regular users of the online platform (50). SOAR participants with more positive symptoms at baseline spent more time on the SOAR Web site and accessed it more frequently. Conversely, CLIMB participants with less severe negative symptoms and a shorter duration of illness were more active on the peer-to-peer messaging platform (53).

We note that these results rely on data about mean intervention use and do not analyze engagement patterns over time. Although we extrapolated results regarding long-term retention from the two 12-month trials (46,49), the three most recent DIs reviewed here had intervention periods ranging from four to 12 weeks, which precluded a comprehensive analysis of long-term engagement. The available data suggest a marked variability in engagement with peer-to-peer communication over time, with some participants showing regular use and others engaging only in the initial phase or dropping out during the follow-up (46,49).

Two studies measured acceptability through the proportion of participants who were satisfied with the ease of use and perceived helpfulness of the DIs, and were willing to recommend the intervention to others. HORYZONS’s acceptability was high, with 75% participants reporting a positive experience, and 90% willing to recommend the DI to others (50). Similarly, 69% to 81% of participants found SOAR moderately to extremely easy to use, and 94% would have liked to be involved in SOAR after study conclusion (49). Three studies measured the actual acceptability through ratings for overall satisfaction, ease of use, and perceived helpfulness on Likert scales. Although ratings for PRIME were on the upper end of the Likert scale, with a mean overall satisfaction of 8 of 10 (52), the average rates found in the study by Kaplan and colleagues (46) and in CLIMB (53) were 5.1 of 10 and 6.7 of 10, respectively. All reviewed studies included qualitative analyses of participants’ views on the acceptability of the DI. Key qualitative themes emerged, including appreciation of the inclusion of peer-to-peer communication, the importance of an engaging intervention delivery format, technical issues as potential barriers to acceptability, and the impact of psychological state and other activities on motivation to adequately engage.

All interventions proved feasible and acceptable for a broad spectrum of uses, including improvements in perceived social connectedness and empowerment (50), perceived social support and stress (48,49), quality of life (53), social cognition (53), and knowledge about the diagnosis of schizophrenia (48,49) and reductions in positive (48) and depressive symptoms (50). Importantly, all four interventions that reported quantitative outcomes demonstrated improvements in perceived social support, suggesting that peer-based interventions can address social isolation.

However, there was significant variation in the outcomes evaluated, and no findings were replicated across two or more studies. Furthermore, three interventions were evaluated with uncontrolled designs, providing only preliminary data about the acceptability, feasibility, safety, and effects (50,52,53). Only two interventions were evaluated with RCT designs (46,48,49), but trial reporting was generally poor. In addition, in both RCTs, individuals in the control groups received care as usual, a nontechnological intervention, which limits the generalizability of the findings. Finally, all studies except the study by Kaplan and colleagues (46) were underpowered to detect moderate treatment effects.

In this review, we systematically examined the feasibility, acceptability, and preliminary efficacy of DIs for people with psychosis that included online peer-to-peer interactions. Our results suggest that DIs offering peer-to-peer communication overall are associated with good retention rates. In addition, including mental health providers, who facilitate positive peer-to-peer interactions and provide support and safety, improved engagement with the DI. This finding is consistent with evidence from research on in-person peer support groups for psychosis, in which minimal guidance from nurses or other mental health professionals is often necessary to overcome the cognitive and social difficulties that individuals with psychosis encounter when participating in group sessions (57). Currently, the number of DIs offering peer-to-peer communication is too small to determine whether the presence of a moderator-facilitator during peer-to-peer communication predicts retention rates. However, this hypothesis is consistent with recent findings showing that completion rates and intervention usage of DIs for bipolar disorder were significantly higher among participants who were offered remote coaching and support than among those who were not supported (44).

Regarding engagement, we found that, in the context of multicomponent DIs, peer-to-peer communication was highly engaging. On average, users logged in every other day and shared content on social media more than twice a day. Conversely, completion rates for the non–social networking features were suboptimal, with less than half of requested actions or tasks completed during the intervention. Marked heterogeneity was also observed in how distinct users engaged with different components of the DI (for example, those who engaged in social networking versus those who read the material about therapy). Therefore, integrating online social networking into evidence-based therapies (50,52,53) is more likely to cater to various users' needs and preferences, thus producing more acceptable online environments (28). In addition, active engagement with peer-to-peer messaging can be leveraged to improve task completion for the non–social networking features. For example, enhancing the moderation of peer-to-peer interactions could provide users with more opportunities and reinforcement for using skills learned in other DI components and for receiving the appropriate rewards. More research is needed to optimize methods of peer moderation such that the time of coaches and clinicians is best utilized.

Results about predictors of engagement were somewhat inconclusive, because studies did not consistently report on whether baseline patient attributes were associated with specific components of the DI. Users with more positive (48) or fewer negative (53) symptoms engaged more with the DIs. In turn, more active engagement was associated with worse (46) or better (50) clinical outcomes. It is intuitive that patients’ attributes, such as symptoms, cognitive abilities, level of functioning, level of intrinsic motivation, insight, and age, all have important effects on the take-up of DIs offering peer-to-peer communication. However, characteristics of the DIs (for example, various features and their integration, number of available modules, duration, automated versus human-supported systems, type of human moderation, design, and interface), and other factors (for example, financial incentives for high completion rates and technical issues with treatment delivery) are all potential variables that should be carefully analyzed to tease out their potential impact on engagement and, similarly, on effectiveness (58,59).

In light of these considerations, we believe that researchers who are interested in implementing peer-to-peer communication in DIs should first and foremost follow theory-driven models of social engagement conceptualized specifically for psychotic disorders (15,60–62). Second, during the design phase, researchers should follow guidelines from digital health design models for people with serious mental illness, such as the Flat Explicit Design Model, which aims to reduce the cognitive effort required to effectively use the intervention by using explicit and brief text, minimizing layers or hierarchy in content organization, employing navigational memory aids, and grouping hyperlinks in one area (63). Finally, once the development is completed, hypotheses about potential variables that influence engagement with each treatment component should be tested.

Findings about acceptability confirmed not only that most participants perceived DIs as helpful and useful but also that peer-to-peer communication was the most appreciated feature. Findings also emphasize the reasons to integrate communication technology that is moderated and secure. Because perceived safety is known to influence the acceptability of online interactions (51), the digital environments should be secure (64), and online interactions should be regularly monitored and moderated (51,65). Clear guidelines for online interactions and safety as well as emergency response protocols should be made available to all users, and automated monitoring of self-harm–related terms should be implemented in the back-end infrastructure. Above and beyond safety and moderation, acceptability of peer-to-peer communication could be enhanced by incorporating innovative sharing technologies, in particular when working with youths, because their preference for media such as videos and teletherapy has been demonstrated (25,53,66).

Of interest, we found that studies involving service users at some point in the design process (49,50,52) resulted in higher rates of acceptability than those that did not (46,53). Engaging key stakeholders (patients, clinicians, research experts, and designers) to participate in user testing, interviews, simulations, and mock-ups during iterative, human-centered design (HCD) processes, in which results from initial stages are used to guide next iterations of the DI, is critical to building clinically relevant technology-based tools that can be implemented and disseminated successfully on a large scale (67,68). For example, the employment of a stepwise HCD process in PRIME led to a two- to threefold increase in the number of peer-to-peer and peer-to-coach interactions (52). In sum, we believe that to ensure acceptability, future studies should continuously involve service users throughout all aspects of intervention development. In addition, studies should examine user-, intervention-, and study-specific factors that in addition to influencing intervention engagement can similarly affect acceptability.

Finally, the reviewed DIs showed promise in improving several outcomes, including social connectedness and support, socialization, perceived stress and empowerment, psychotic symptoms, depression, and quality of life. However, study methodology and trial reporting were generally poor. Validated assessments and appropriate clinical trial procedures that encompass follow-up periods need to be consistently implemented across studies. Traditional RCTs may not be the most effective way to test the efficacy of DIs, given that the rapid developments in the available technology may make the original product obsolete and uncompetitive by the time the trial is over. Instead, well-conducted, properly powered clinical trials that incorporate novel analytical methods and designs—including sequential, multiple assignment, randomized trial (SMART) methods that focus on multiple assessments on an individual level (69,70) or stepped-wedge research designs (71,72)—may be more useful in determining and comparing the efficacy of DIs and their components.

In addition to the methodological shortcomings identified in the existing literature and summarized above, several limitations should be taken into account when interpreting our findings. First, despite our search strategy and broad inclusion criteria, it is also possible that other studies are still in progress or unpublished. Second, use of English-language studies may have biased our results toward Western countries. Analysis of studies produced in languages other than English may have yielded different findings. Most studies originated in the United States, and it is unclear whether the same findings would be reflected in other countries or cultures. Third, we did not assess for the methodological quality of the included studies, given the small samples in four studies and lack of control groups in three. Finally, we note that a publication bias is likely to exist in this area of research. Apart from the fact that positive results are more likely to be published than negative results, we suspect that many DIs for mental health have not been scientifically investigated or that research studies testing them were conducted by private-sector companies and thus are unavailable for review.