Skip to main content

Abstract

With more than 10,000 mental health apps available, consumers and clinicians who want to adopt such tools can be overwhelmed by the multitude of options and lack of clear evaluative standards. Despite the increasing prevalence of curated lists, or app guides, challenges remain. Organizations providing mental health services to consumers have an opportunity to address these challenges by producing guides that meet relevant standards of quality and are tailored to local needs. This column summarizes an example of the collaborative process of app guide development in a publicly funded mental health service context and highlights opportunities and barriers identified through the process.

HIGHLIGHTS

Although app marketplaces contain many mental health apps, little curation occurs, impeding adoption and posing potential harm to consumers.
Curated lists of mental health apps, or app guides, that meet relevant standards of quality can and should be tailored to the needs of various communities.
A collaborative project focused on the creation of such a guide is summarized to illustrate an example of the iterative process and the resulting product of app guide development in a publicly funded context, and the opportunities and barriers that emerged are outlined.

More than 10,000 mobile mental health apps are currently available to the public (1). Sifting through this multitude of apps often leaves consumers and clinicians overwhelmed, unsatisfied, and in many cases without a useful or effective resource (2). Although a number of frameworks exist to guide people through the process of app evaluation (3), these frameworks leave consumers and clinicians without specific app recommendations. An emerging solution to these challenges is the advent of curated lists, or app guides, that provide recommendations for a limited number of apps, usually with a specific target, such as depression (4). These guides range from relatively basic top 10 lists to expertly researched guides evaluating aspects of apps such as evidence of their effectiveness, user experience, adequacy of data security, and privacy (5). Often, these guides have been curated for specific audiences, such as veterans or employers (6).
Despite the proliferation of these guides, challenges remain. Many challenges mirror those posed by the apps themselves, including consumer and clinician awareness, unknown quality, limited relevance, and lack of direct consumer input. Mental health service organizations have an opportunity to address these challenges through a collaborative process of app guide development (4). By involving community stakeholders, subject matter experts, clinicians, and end users in this process, organizations can produce app guides that meet explicitly stated standards of quality and are tailored to local needs. Here, we present an example from a California county mental health program to illustrate the iterative process and resulting product of app guide development, and we outline opportunities and barriers that emerged.

The Help@Hand Project

We present output from a publicly funded innovation project in California, Help@Hand. In 2005, California enacted Proposition 63, the Mental Health Services Act (Assembly Bill 488), which authorized earmarked taxes for behavioral health services (7). Each county allocates the revenue generated by this tax to a combination of several specified areas, including innovation projects. Innovation projects develop or test “new, unproven mental health models” with the goal of increasing the quality of mental health services and access to those services by underserved groups (8). Help@Hand is a multiyear innovation project representing a collaboration among 14 California city and county behavioral health departments to explore the use of technology to improve the reach and impact of mental health services. A formative evaluation of this project is being conducted by a team with expertise in program evaluation, health services research, mental health app evaluation, and human-computer interaction.

Peers’ Guide to Behavioral Health Apps

As part of the Help@Hand project, Kern Behavioral Health and Recovery Services (KernBHRS) created a curated list of apps, “The Peers’ Guide to Behavioral Health Apps” (Kern App Guide), to inform consumers about various apps that could help support people’s mental health. A Spanish version, “La Guía de Compañeros Para Aplicaciones Sobre la Salud Conductual,” was also produced to reach the large proportion of Spanish speakers in the community.
The development of the Kern App Guide was an iterative process that is well suited to analysis using the replicating effective programs (REP) implementation framework (9), which has been widely used to prepare and package community-based programs for dissemination (4). The REP framework stages are as follows: preconditions (identifying local needs and implementation barriers), preimplementation (gathering stakeholder feedback and testing the intervention), implementation (intervention training, delivery, evaluation, and refinement), and maintenance and evolution (establishing organizational changes to sustain practices). We describe the Kern App Guide development process according to these four REP stages.

Preconditions.

In the context of the Kern App Guide development, the preconditions stage comprised several activities, including identification of a need for a new app guide, recruitment of a committee representing the various stakeholders affected by the project, and drafting of an initial list of apps for consideration. The idea for an app guide was first suggested by the director of KernBHRS in spring 2018 as an opportunity to create a practical product in collaboration with Help@Hand. App guide development was initiated in summer 2018 by a KernBHRS project manager, who assembled a committee of stakeholders. The App Guide Committee included 12 peers—community members with lived experience of mental health (or co-occurring) issues, experience with recovery, and training to use these experiences—working in a volunteer or employed capacity in the Recovery Services Division of KernBHRS. By November 2018, this committee had produced an initial outline of the app guide, consisting of 15 apps selected on the basis of familiarity and personal experience as a starting point for the formal evaluation process.

Preimplementation.

The preimplementation stage involved the committee meeting to determine the evaluation criteria for refinement of the list, as well as the evaluation and selection process itself. Beginning in November 2018, committee members initiated a formal review process. Several factors were chosen for app evaluation and selection on the basis of the needs of the community, including zero cost, availability in Spanish, accessibility on both iPhone and Android platforms, privacy and data protection, simplicity, and ease of use. (The rubric used for app evaluation is available as an online supplement to this column.) Using these criteria, committee members searched for new apps to add to the original list of 15 apps for formal review. Several dozen apps were identified, each of which was subsequently reviewed independently by at least three committee members. These reviews were discussed by the committee in weekly 1-hour meetings, and final inclusion was decided by majority vote. By the end of December 2018, a pilot app guide was published for internal review and distribution. The committee also solicited feedback on the pilot guide and candidate apps from the evaluation partners, who provided recommendations regarding app evaluation and selection criteria as well as design and dissemination considerations, which were incorporated before the release of the guide.

Implementation.

With the help of the marketing staff, the first edition of the Kern App Guide was published in April 2019 (all versions are available at https://www.kernbhrs.org/appguide). The 12-page first edition included 30 apps organized into six categories: behavioral health and wellness, the mind, the body, recovery, veterans, and sleep. Short descriptions, including the app’s purpose and an overview of its contents, were provided for each recommended app, as well as an indication of which apps were available in Spanish. The guide was disseminated, both digitally on the KernBHRS website and via physical handouts circulated within KernBHRS, from leadership to service delivery teams. At least 6,500 copies have been printed and distributed within Kern County in both English and Spanish. Outside of KernBHRS, the guide was shared with other county mental health service organizations, including the California Mental Health Services Authority conference of Help@Hand participants and the California Behavioral Health Directors Association.

Maintenance and evolution.

After the publication of the first edition, the committee continued meeting weekly to incorporate stakeholder feedback and select and review new apps. These meetings typically included five to nine members and involved brainstorming, discussion, and voting to add or remove apps from the guide. The evaluation partner also provided detailed recommendations regarding app evaluation and selection criteria. One example of a change made in regard to stakeholder feedback was to avoid apps that felt “classlike,” with extensive homework, reading, or lectures. A second edition was released in December 2019. To support the sustainable implementation of behavioral health apps into the existing system of care, KernBHRS created a plan to revise the guide semiannually and devoted considerable marketing efforts into distributing the guide, including promoting the guide via local media channels as an available resource during the outbreak of the COVID-19 pandemic.

Preliminary Outcomes and Learning

The Kern App Guide development process provides an example of the time, effort, and collaboration required to create a tailored app guide that meets the specific needs of local community members. A team of approximately a dozen people at KernBHRS developed the guide over a period of approximately 1 year, with each member dedicating on average several hours per week to the project. The skills and resources required to produce the Kern App Guide included knowledge of digital mental health products, community needs, and services, in addition to design and dissemination skills, and development of the guide involved community stakeholders, subject matter experts from Help@Hand, members of the marketing team, and many other KernBHRS peers and staff. The guide has been widely disseminated in Kern County, both physically and digitally, and versions of the guide were adapted for five other counties: Inyo, Mono, Modoc, San Bernardino, and Santa Barbara.

Discussion

The integration of behavioral health apps into the services provided by KernBHRS has provided lessons in both the opportunities of incorporating multiple perspectives into innovation projects and the barriers to implementing such collaboration.

Opportunities.

The inclusion of multiple perspectives, especially those of community stakeholders, was one of the key opportunities identified. The use of technology to provide mental health services often circumvents traditional delivery pathways. This may be useful for patient outreach, creating direct-to-consumer supportive care, but it can pose challenges to identifying those helped by such products, making it difficult to incorporate their feedback. This lack of feedback can be especially problematic given that low community involvement is a key weakness of many existing app guides (3). In this case, the potential disconnect that can accompany technological solutions was addressed by adopting a stakeholder-driven process of evaluation that allowed for the refinement and tailoring of recommendations (e.g., through the guide) to improve their relevance to the actual needs and expectations of the community they were meant to serve. Specifically, the involvement of community stakeholders allowed for the prioritization of features relevant to Kern County community members, such as the availability of apps in Spanish (and the production of a Spanish version of the app guide).

Barriers.

Nevertheless, this multistep process is not simple and illustrates some barriers to doing this work. Recruiting and organizing team members with a wide variety of experiences and different areas of expertise can be challenging. Once the team has been assembled, negotiating among the perspectives, preferences, and needs of different stakeholders requires care and patience. It can be difficult to arrive at a consensus regarding the many decisions required of the process, and some trade-offs may be necessary. For instance, some evaluation criteria were not weighted as heavily, such as the evidence base for app effectiveness, because much of the evidence created for these products was not cocreated with relevant communities and therefore might not be relevant (10).
Ways to address these barriers would include the use of structured and iterative processes, such as plan-do-study-act frameworks, or the creation of tool kits that identify core processes and roles and might facilitate adaptation. With this in mind, the Kern App Guide was presented to the community as just one version of the guide that could and would be updated. The challenge of conducting and updating app evaluations has been demonstrated elsewhere (11), and another solution might be increased transparency regarding the limitations present in any guide based on such evaluations.

Conclusions

Generating an app guide is a multifaceted process that requires diverse knowledge and resources. Here, we provide an example of how a publicly funded mental health service organization, in collaboration with different partners, created a tailored app guide to increase the community’s awareness of mental health apps that could serve as relevant mental health resources. The Kern App Guide, as part of the Help@Hand project, provides a concrete example of the implementation process of health app evaluation, upon which other service settings can draw. Such guides are not one-size-fits-all solutions, but the process represents a template for how app guides can be created by considering the unique needs of a community and leveraging various areas of expertise. Many organizations are considering using apps to expand mental health service delivery systems. If they choose not to develop their own apps, they must learn how to select and recommend existing apps. App guides such as the one presented here can be a useful tool to support consumers and clinicians in navigating the complexities of the digital mental health landscape.

Footnote

This work was funded by Help@Hand (contract number 417-ITSUCI- 2019), a project overseen by the California Mental Health Service Authority (CalMHSA). CalMHSA reviewed the manuscript for confidentiality. The information or content and conclusions presented here are those of the authors and should not be construed as official position or policy of, nor should any endorsements be inferred by, the participating Help@Hand counties and/or CalMHSA.

Supplementary Material

File (appi.ps.202000803.ds001.pdf)

References

1.
Torous J, Roberts LW: Needed innovation in digital health and smartphone applications for mental health: transparency and trust. JAMA Psychiatry 2017; 74:437–438
2.
Larsen ME, Huckvale K, Nicholas J, et al: Using science to sell apps: evaluation of mental health app store quality claims. NPJ Digit Med 2019; 2:18
3.
Torous JB, Chan SR, Gipson SYT, et al: A hierarchical framework for evaluation and informed decision making regarding smartphone apps for clinical care. Psychiatr Serv 2018; 69:498–500
4.
Camacho E, Lagan S, Rodriguez-Villa E, et al: Technology Evaluation and Assessment Criteria for Health apps (TEACH-apps). J Med Internet Res 2020; 22:e18346
5.
Neary M, Schueller SM: State of the field of mental health apps. Cognit Behav Pract 2018; 25:531–537
6.
Cunningham-Hill M, Dodge-Rice Z, Wilson-Meyers C, et al: Digital Tools and Solutions for Mental Health: An Employer’s Guide. New York, Northeast Business Group on Health and One Mind PsyberGuide, 2020
7.
Report to the Governor and Legislature. Sacramento, CA, Mental Health Services Oversight and Accountability Commission, 2013. http://archive.mhsoac.ca.gov/MHSOAC_Publications/docs/Publications/LegReport_FINAL_121813.pdf
8.
Purtle J, Stadnick NA: Earmarked taxes as a policy strategy to increase funding for behavioral health services. Psychiatr Serv 2020; 71:100–104
9.
Kilbourne AM, Neumann MS, Pincus HA, et al: Implementing evidence-based interventions in health care: application of the replicating effective programs framework. Implement Sci 2007; 2:42
10.
Rycroft-Malone J, Seers K, Titchen A, et al: What counts as evidence in evidence-based practice? J Adv Nurs 2004; 47:81–90
11.
Carlo AD, Hosseini Ghomi R, Renn BN, et al: By the numbers: ratings and utilization of behavioral health mobile applications. NPJ Digit Med 2019; 2:54

Information & Authors

Information

Published In

Go to Psychiatric Services
Go to Psychiatric Services
Psychiatric Services
Pages: 1229 - 1232
PubMed: 34030454

History

Received: 2 November 2020
Revision received: 18 December 2020
Accepted: 8 January 2021
Published online: 25 May 2021
Published in print: October 01, 2021

Keywords

  1. Community mental health services
  2. Digital Mental Health
  3. Smartphones
  4. Dissemination

Authors

Affiliations

Robert M. Montgomery, M.A.
Department of Psychology, California State University, Northridge (Montgomery); Kern County Behavioral Health and Recovery Services, Bakersfield, California (Brandysky); Department of Psychological Science (Neary, Schueller), Department of Informatics (Mark, Zheng, Schueller), Department of Population Health and Disease Prevention (Schneider), and Department of Medicine (Mukamel, Sorkin), University of California, Irvine; Department of Family Medicine and Public Health (Eikey), and Department of Psychiatry (Stadnick), University of California, San Diego. Dror Ben-Zeev, Ph.D., is editor of this column.
Lamar Brandysky, L.M.F.T.
Department of Psychology, California State University, Northridge (Montgomery); Kern County Behavioral Health and Recovery Services, Bakersfield, California (Brandysky); Department of Psychological Science (Neary, Schueller), Department of Informatics (Mark, Zheng, Schueller), Department of Population Health and Disease Prevention (Schneider), and Department of Medicine (Mukamel, Sorkin), University of California, Irvine; Department of Family Medicine and Public Health (Eikey), and Department of Psychiatry (Stadnick), University of California, San Diego. Dror Ben-Zeev, Ph.D., is editor of this column.
Martha Neary, M.Sc.
Department of Psychology, California State University, Northridge (Montgomery); Kern County Behavioral Health and Recovery Services, Bakersfield, California (Brandysky); Department of Psychological Science (Neary, Schueller), Department of Informatics (Mark, Zheng, Schueller), Department of Population Health and Disease Prevention (Schneider), and Department of Medicine (Mukamel, Sorkin), University of California, Irvine; Department of Family Medicine and Public Health (Eikey), and Department of Psychiatry (Stadnick), University of California, San Diego. Dror Ben-Zeev, Ph.D., is editor of this column.
Elizabeth Eikey, Ph.D.
Department of Psychology, California State University, Northridge (Montgomery); Kern County Behavioral Health and Recovery Services, Bakersfield, California (Brandysky); Department of Psychological Science (Neary, Schueller), Department of Informatics (Mark, Zheng, Schueller), Department of Population Health and Disease Prevention (Schneider), and Department of Medicine (Mukamel, Sorkin), University of California, Irvine; Department of Family Medicine and Public Health (Eikey), and Department of Psychiatry (Stadnick), University of California, San Diego. Dror Ben-Zeev, Ph.D., is editor of this column.
Gloria Mark, Ph.D.
Department of Psychology, California State University, Northridge (Montgomery); Kern County Behavioral Health and Recovery Services, Bakersfield, California (Brandysky); Department of Psychological Science (Neary, Schueller), Department of Informatics (Mark, Zheng, Schueller), Department of Population Health and Disease Prevention (Schneider), and Department of Medicine (Mukamel, Sorkin), University of California, Irvine; Department of Family Medicine and Public Health (Eikey), and Department of Psychiatry (Stadnick), University of California, San Diego. Dror Ben-Zeev, Ph.D., is editor of this column.
Margaret Schneider, Ph.D.
Department of Psychology, California State University, Northridge (Montgomery); Kern County Behavioral Health and Recovery Services, Bakersfield, California (Brandysky); Department of Psychological Science (Neary, Schueller), Department of Informatics (Mark, Zheng, Schueller), Department of Population Health and Disease Prevention (Schneider), and Department of Medicine (Mukamel, Sorkin), University of California, Irvine; Department of Family Medicine and Public Health (Eikey), and Department of Psychiatry (Stadnick), University of California, San Diego. Dror Ben-Zeev, Ph.D., is editor of this column.
Nicole A. Stadnick, Ph.D., M.P.H.
Department of Psychology, California State University, Northridge (Montgomery); Kern County Behavioral Health and Recovery Services, Bakersfield, California (Brandysky); Department of Psychological Science (Neary, Schueller), Department of Informatics (Mark, Zheng, Schueller), Department of Population Health and Disease Prevention (Schneider), and Department of Medicine (Mukamel, Sorkin), University of California, Irvine; Department of Family Medicine and Public Health (Eikey), and Department of Psychiatry (Stadnick), University of California, San Diego. Dror Ben-Zeev, Ph.D., is editor of this column.
Kai Zheng, Ph.D.
Department of Psychology, California State University, Northridge (Montgomery); Kern County Behavioral Health and Recovery Services, Bakersfield, California (Brandysky); Department of Psychological Science (Neary, Schueller), Department of Informatics (Mark, Zheng, Schueller), Department of Population Health and Disease Prevention (Schneider), and Department of Medicine (Mukamel, Sorkin), University of California, Irvine; Department of Family Medicine and Public Health (Eikey), and Department of Psychiatry (Stadnick), University of California, San Diego. Dror Ben-Zeev, Ph.D., is editor of this column.
Dana B. Mukamel, Ph.D.
Department of Psychology, California State University, Northridge (Montgomery); Kern County Behavioral Health and Recovery Services, Bakersfield, California (Brandysky); Department of Psychological Science (Neary, Schueller), Department of Informatics (Mark, Zheng, Schueller), Department of Population Health and Disease Prevention (Schneider), and Department of Medicine (Mukamel, Sorkin), University of California, Irvine; Department of Family Medicine and Public Health (Eikey), and Department of Psychiatry (Stadnick), University of California, San Diego. Dror Ben-Zeev, Ph.D., is editor of this column.
Dara H. Sorkin, Ph.D.
Department of Psychology, California State University, Northridge (Montgomery); Kern County Behavioral Health and Recovery Services, Bakersfield, California (Brandysky); Department of Psychological Science (Neary, Schueller), Department of Informatics (Mark, Zheng, Schueller), Department of Population Health and Disease Prevention (Schneider), and Department of Medicine (Mukamel, Sorkin), University of California, Irvine; Department of Family Medicine and Public Health (Eikey), and Department of Psychiatry (Stadnick), University of California, San Diego. Dror Ben-Zeev, Ph.D., is editor of this column.
Stephen M. Schueller, Ph.D. [email protected]
Department of Psychology, California State University, Northridge (Montgomery); Kern County Behavioral Health and Recovery Services, Bakersfield, California (Brandysky); Department of Psychological Science (Neary, Schueller), Department of Informatics (Mark, Zheng, Schueller), Department of Population Health and Disease Prevention (Schneider), and Department of Medicine (Mukamel, Sorkin), University of California, Irvine; Department of Family Medicine and Public Health (Eikey), and Department of Psychiatry (Stadnick), University of California, San Diego. Dror Ben-Zeev, Ph.D., is editor of this column.

Notes

Send correspondence to Dr. Schueller ([email protected]).

Competing Interests

Mr. Montgomery is an employee of Happify Health. Mr. Brandysky was employed as contract staff to manage the Help@Hand project for Kern Behavioral Health and Recovery Services during the writing of this article. Dr. Schueller has received consulting payments from Otsuka Pharmaceuticals. The other authors report no financial relationships with commercial interests.

Metrics & Citations

Metrics

Citations

Export Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

For more information or tips please see 'Downloading to a citation manager' in the Help menu.

Format
Citation style
Style
Copy to clipboard

There are no citations for this item

View Options

View options

PDF/ePub

View PDF/ePub

Get Access

Login options

Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.

Personal login Institutional Login Open Athens login
Purchase Options

Purchase this article to access the full text.

PPV Articles - Psychiatric Services

PPV Articles - Psychiatric Services

Not a subscriber?

Subscribe Now / Learn More

PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.

Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).

Media

Figures

Other

Tables

Share

Share

Share article link

Share