Mental Health App Evaluation: Updating the American Psychiatric Association’s Framework Through a Stakeholder-Engaged Workshop

The levels in the original APA app evaluation framework included background information, risk-privacy and security, evidence, ease of use, and interoperability. A series of questions enabled an individual to consider various components related to each level. The framework was structured in a hierarchical fashion so that the evaluator could choose to stop the inquiry if he or she noted concerns about an app early in the evaluation process (3). For example, if an app lacked a privacy policy and did not specify data usage, limiting its use in clinical settings, there would be no need to continue the evaluation and consider facets of evidence and usability for a clinician or consumer concerned with maintaining a high standard of privacy and security. Before meeting in person, each of the panelists was asked to watch a three-part series available online through the APA, which provided the foundation for the framework and app evaluation process in its original format. This framework was created by harmonizing 961 questions from 45 different app evaluation frameworks (3, 4).

On the first day of the in-person meeting, the group convened and reviewed the APA app evaluation framework as it existed in its previous form. The group reviewed the most recent evidence on mobile apps for mental health and noted that limited evidence of benefit in high-quality studies and mounting privacy concerns warranted a cautious approach. The full framework was scrutinized through structured discussion among the panelists to ensure that the evaluation tool was effective in providing a thorough inquiry into the factors that clinicians and patients should consider when using a mobile app. Although the conversation typically focused on the use of apps within the psychiatric clinical practice setting, the group felt it was important to create a tool that could be used broadly to evaluate apps intended for use by an individual or clinician regardless of specialty.

To make decisions regarding the app evaluation framework, the panel used a modified Delphi procedure, with iterative discussion through each level until a consensus was reached. Three defined rounds of discussion took place. Although voting scores were not recorded, a consensus was reached after each round before proceeding to the next. First, the panel discussed whether to retain the initial 2018 framework with edits or to restructure it entirely. Although the panel decided to keep the hierarchical pyramid structure of the framework, in the second round of discussion, the five levels were slightly altered to better reflect their constituent questions. The first level became “accessibility and background” instead of “background information,” because the panel emphasized a renewed focus on access. If an individual is unable to access an app because of lack of online connectivity or incompatibility with smartphone accessibility features, the app is not usable, and consideration of the framework’s upper levels is unnecessary. Access is thus a foundational component of evaluation, as reflected by the new questions in the first level.

The second level of the pyramid, which addresses privacy and security, remained largely the same, whereas the third level became “clinical foundation” instead of “evidence” to emphasize not only the app’s specific evidence but also the body of clinical research underpinning its various use cases. The group debated what level of evidence is necessary for an app and realized that this question is complex, especially because the lack of published research is not necessarily indicative of low quality of newer tools that may not have undergone more longitudinal testing. The fourth level remained largely the same, although the panel chose to use the label “usability” instead of “ease of use” to better incorporate the various features that users may be drawn to in an app. In the third round of discussion, efforts were made to offer more concrete questions to guide assessment of this very personalized and unique facet of app evaluation. Finally, the fifth level maintained its focus on data integration but was termed, more descriptively, “data integration toward therapeutic goal” instead of “interoperability.” In the third round of discussion, the original list of 27 questions was clarified and expanded to 36 questions (see online supplement).

Another core objective for the first day of the meeting was to determine specific questions that providers and patients could use to complete a quick and simple screening process for an app intended for their use. All members agreed that although a more thorough assessment is always beneficial, it may be possible to offer users value with a brief screening tool that can be quickly applied. Thus, each level within the APA app evaluation framework was discussed and evaluated to determine which questions were key to providing a brief but thorough screening of mobile apps. The 36 questions from the full framework were reduced to eight key questions across the five broader levels (see online supplement). This “screener” can be completed more quickly than the full framework but retains the spirit and priorities of the full version. The screener version was not designed to replace the full version but rather to offer a tool more applicable to a busy clinical setting. Although it accomplishes the goal of providing a condensed, easily deployable tool, the screener does not prompt users to consider as many dimensions of the app as the full framework.

The second day of the in-person meeting tasked the panel with putting the APA app evaluation framework into practice. The day commenced with a group evaluation of a chatbot-based app that provided online cognitive-behavioral therapy. Together, the panel went through the process of evaluating this chatbot by using the updated framework. On the basis of this group exercise, minor adjustments were made to the wording of questions. After this group evaluation, each panelist evaluated an app of their choosing and presented the findings to the group. These evaluations were conducted in pairs to identify any confusing questions that may have needed further refinement. The framework proved to be a feasible tool for a wide variety of apps, ranging from child-focused apps, to apps with corresponding wearable devices, to popular artificial intelligence therapy apps. Each of these diverse app types was evaluated by two panelists, with the framework evaluations demonstrating strong reliability between pair members.

The evaluation begins at the foundational level, with a consideration of background and accessibility information. At this level, we addressed questions pertaining to various facets of access, including cost, offline functionality, and stability (reflected by an update within the past 180 days). The evaluation then proceeded through privacy and security, clinical foundation, engagement style and usability, and data integration toward therapeutic goal (Figure 1). Tutorials for evaluating 11 apps with this framework are now publicly available on the APA website (https://www.psychiatry.org/psychiatrists/practice/mental-health-apps), and 11 tutorials apply the new framework to diverse mental health–related apps.