Curating the Digital Mental Health Landscape With a Guide to Behavioral Health Apps: A County-Driven Resource

In the context of the Kern App Guide development, the preconditions stage comprised several activities, including identification of a need for a new app guide, recruitment of a committee representing the various stakeholders affected by the project, and drafting of an initial list of apps for consideration. The idea for an app guide was first suggested by the director of KernBHRS in spring 2018 as an opportunity to create a practical product in collaboration with Help@Hand. App guide development was initiated in summer 2018 by a KernBHRS project manager, who assembled a committee of stakeholders. The App Guide Committee included 12 peers—community members with lived experience of mental health (or co-occurring) issues, experience with recovery, and training to use these experiences—working in a volunteer or employed capacity in the Recovery Services Division of KernBHRS. By November 2018, this committee had produced an initial outline of the app guide, consisting of 15 apps selected on the basis of familiarity and personal experience as a starting point for the formal evaluation process.

The preimplementation stage involved the committee meeting to determine the evaluation criteria for refinement of the list, as well as the evaluation and selection process itself. Beginning in November 2018, committee members initiated a formal review process. Several factors were chosen for app evaluation and selection on the basis of the needs of the community, including zero cost, availability in Spanish, accessibility on both iPhone and Android platforms, privacy and data protection, simplicity, and ease of use. (The rubric used for app evaluation is available as an online supplement to this column.) Using these criteria, committee members searched for new apps to add to the original list of 15 apps for formal review. Several dozen apps were identified, each of which was subsequently reviewed independently by at least three committee members. These reviews were discussed by the committee in weekly 1-hour meetings, and final inclusion was decided by majority vote. By the end of December 2018, a pilot app guide was published for internal review and distribution. The committee also solicited feedback on the pilot guide and candidate apps from the evaluation partners, who provided recommendations regarding app evaluation and selection criteria as well as design and dissemination considerations, which were incorporated before the release of the guide.

With the help of the marketing staff, the first edition of the Kern App Guide was published in April 2019 (all versions are available at https://www.kernbhrs.org/appguide). The 12-page first edition included 30 apps organized into six categories: behavioral health and wellness, the mind, the body, recovery, veterans, and sleep. Short descriptions, including the app’s purpose and an overview of its contents, were provided for each recommended app, as well as an indication of which apps were available in Spanish. The guide was disseminated, both digitally on the KernBHRS website and via physical handouts circulated within KernBHRS, from leadership to service delivery teams. At least 6,500 copies have been printed and distributed within Kern County in both English and Spanish. Outside of KernBHRS, the guide was shared with other county mental health service organizations, including the California Mental Health Services Authority conference of Help@Hand participants and the California Behavioral Health Directors Association.

After the publication of the first edition, the committee continued meeting weekly to incorporate stakeholder feedback and select and review new apps. These meetings typically included five to nine members and involved brainstorming, discussion, and voting to add or remove apps from the guide. The evaluation partner also provided detailed recommendations regarding app evaluation and selection criteria. One example of a change made in regard to stakeholder feedback was to avoid apps that felt “classlike,” with extensive homework, reading, or lectures. A second edition was released in December 2019. To support the sustainable implementation of behavioral health apps into the existing system of care, KernBHRS created a plan to revise the guide semiannually and devoted considerable marketing efforts into distributing the guide, including promoting the guide via local media channels as an available resource during the outbreak of the COVID-19 pandemic.

The inclusion of multiple perspectives, especially those of community stakeholders, was one of the key opportunities identified. The use of technology to provide mental health services often circumvents traditional delivery pathways. This may be useful for patient outreach, creating direct-to-consumer supportive care, but it can pose challenges to identifying those helped by such products, making it difficult to incorporate their feedback. This lack of feedback can be especially problematic given that low community involvement is a key weakness of many existing app guides (3). In this case, the potential disconnect that can accompany technological solutions was addressed by adopting a stakeholder-driven process of evaluation that allowed for the refinement and tailoring of recommendations (e.g., through the guide) to improve their relevance to the actual needs and expectations of the community they were meant to serve. Specifically, the involvement of community stakeholders allowed for the prioritization of features relevant to Kern County community members, such as the availability of apps in Spanish (and the production of a Spanish version of the app guide).

Nevertheless, this multistep process is not simple and illustrates some barriers to doing this work. Recruiting and organizing team members with a wide variety of experiences and different areas of expertise can be challenging. Once the team has been assembled, negotiating among the perspectives, preferences, and needs of different stakeholders requires care and patience. It can be difficult to arrive at a consensus regarding the many decisions required of the process, and some trade-offs may be necessary. For instance, some evaluation criteria were not weighted as heavily, such as the evidence base for app effectiveness, because much of the evidence created for these products was not cocreated with relevant communities and therefore might not be relevant (10).

Ways to address these barriers would include the use of structured and iterative processes, such as plan-do-study-act frameworks, or the creation of tool kits that identify core processes and roles and might facilitate adaptation. With this in mind, the Kern App Guide was presented to the community as just one version of the guide that could and would be updated. The challenge of conducting and updating app evaluations has been demonstrated elsewhere (11), and another solution might be increased transparency regarding the limitations present in any guide based on such evaluations.