Attributes of Provider Referrals for Digital Mental Health Applications in an Integrated Health System, 2019–2021
Abstract
Objective:
This article describes trends and attributes associated with digital mental health application (DMHA) referrals from December 2019 through December 2021.
Methods:
In total, 43,842 DMHA referrals for 25,213 unique patients were extracted from the electronic health record of a large, diverse, integrated health system. DMHAs were aggregated by type (cognitive-behavioral therapy [CBT] or mindfulness and meditation [MM]). Monthly referral patterns were described and categorized into mutually exclusive clusters (MM, CBT, or MM and CBT). Multinomial logistic regression and post hoc predicted probabilities were used to profile patient, clinical, and encounter attributes among referral clusters.
Results:
DMHA referrals increased, reached equilibrium, and then began to decline over the 25-month observation period. Compared with the referral cluster average, MM-alone referrals were more likely to occur for patients who were ages ≥65, who were Hispanic or Asian, whose reason for visit concerned mental health, and who had a primary diagnosis of other anxiety disorders. CBT-alone referrals were more likely to occur for patients with a primary diagnosis of depression and less likely to occur for Hispanic patients. Combined MM and CBT referrals were more likely to occur for patients who were ages 18–30, whose reason for visit was “other,” and who had a primary diagnosis of depression and were less likely to occur for Hispanic patients and those ages ≥65.
Conclusions:
Although this study demonstrates readiness to integrate DMHA referral into clinical workflows, observed variations in attributes of referral clusters support the need to further investigate provider decision making and whether referral patterns are optimal and sustainable.
HIGHLIGHTS
•
After an initial modest increase, digital mental health application (DMHA) referrals rapidly accelerated, reached equilibrium, and then began to decline over the 25-month observation period from December 2019 through December 2021.
•
Significant variations in age, race-ethnicity, reason for visit, primary diagnosis, presence of depression screening on the date of encounter, provider location, and visit type among DMHA referral clusters were observed.
•
Ongoing provider training and guidance for initiating DMHA referrals in clinical practice may be necessary to reduce variations in DMHA referral patterns and promote sustainable use of DMHAs.
Anxiety and depressive disorders are the most common mental health conditions in the United States, with lifetime risks of 29% and 20%, respectively (1, 2). These conditions incur disability, high health care utilization, and low quality of life (1, 3–6). Social and financial stress experienced during the COVID-19 pandemic contributed to increases in anxiety and depressive disorders (7, 8). Increased incidence of mental disorders, pervasive provider shortages, and stigma associated with mental health treatment may have negatively affected patient access and increased unmet need (9–11).
Digital mental health applications (DMHAs) often incorporate mindfulness and meditation (MM) and cognitive-behavioral therapy (CBT) as evidence-based approaches for managing anxiety and depression and can help alleviate barriers to care (12, 13). MM strategies focus on reducing stress and anxiety through relaxation, whereas CBT strategies require active patient engagement to address the problem source.
More than 10,000 DMHAs are available online, but provider guidance may assist patients with selecting an appropriate app (14). By including DMHA referral into clinical workflows, providers can help patients select the correct app for their need, mitigate app cost, increase patient engagement, track referrals, and optimize patient app use (15). Although DMHAs should not substitute for in-person care, they may provide novel, complementary solutions that can be broadly disseminated, address subclinical mental health problems prior to clinical intervention, and supplement a care plan. Clinical trial evidence suggests that DMHAs may reduce anxiety and depression symptoms, support emotional well-being by reducing stress, and improve resiliency (14, 16–23), yet little is known about their integration and effectiveness in clinical practice.
Integrated health care systems such as the Veterans Health Administration and Kaiser Permanente (KP) have been at the forefront of integrating DMHAs into clinical care (24–27). In December 2019, the rollout of a new initiative within the KP Mid-Atlantic States (KPMAS) region (District of Columbia, Maryland, and Virginia) allowed KP’s behavioral health providers to refer patients for any of six DMHAs that focused on MM (Calm, Headspace, Whil) or CBT (myStrength, Thrive, SilverCloud) with no patient registration cost. Primary care providers in the health system were limited to referrals for Calm and myStrength.
Improved understanding of how and for which patients providers make referrals for DMHAs in clinical practice is important for identifying unmet need, informing future provider training, and establishing expectations for health systems considering implementation of DMHA referrals into clinical workflows. Experiences from integrated systems such as KPMAS can inform the implementation of DMHA referrals in other systems. Therefore, this study sought to describe trends in referrals for DMHAs over a 25-month observation period and to profile patient, clinical, and encounter attributes associated with DMHA referral clusters.
Methods
Design, Sample, and Setting
A retrospective, cross-sectional design was approved by the KPMAS Institutional Review Board. All clinical encounters for KPMAS members ≥18 years old with an initial referral for a unique DMHA and associated patient, clinical, and encounter referral attributes were extracted from the electronic health record. Inclusive dates were December 1, 2019, through December 31, 2021. As a large integrated health system, KPMAS serves a diverse group of more than 800,000 active members. The system’s coverage area includes three primary regions (Baltimore, District of Columbia and southern Maryland [DCSM], and Northern Virginia [NoVA]). By linking with a common electronic health record, the KPMAS system provides comprehensive coordination across the care continuum.
Measurements
Referrals were initially grouped into two categories (MM or CBT) to be consistent with national initiative guidance (26). Referrals were further organized into three mutually exclusive referral clusters—MM alone, CBT alone, or MM and CBT—if a DMHA from each group was ordered for the same patient during the observation period (see the online supplement to this article).
Referral clusters were described by patient, clinical, and encounter attributes by using the referral as the unit of analysis. Demographic characteristics included patients’ age at referral, gender, and self-reported race-ethnicity.
Clinical attributes associated with each referral included four mutually exclusive groups of patient-indicated reasons for visit (i.e., chief complaint): mental health (anxiety, depression, stress), learning (wellness coaching, education, counseling), annual care, and “other” (online supplement ). Provider-assigned primary diagnosis for each encounter was categorized into four mutually exclusive groups: depression (single episode or recurrent) (ICD-10 codes F32 and F33), unspecified mood disorder (ICD-10 code F39), other anxiety disorders (ICD-10 code F41), or “other” (online supplement ). Categories for reason for visit and primary diagnosis were chosen to be consistent with prompts from the DMHA workflow within the electronic health record and to represent patient and provider perspectives. Because Patient Health Questionnaire (PHQ) or Generalized Anxiety Disorder assessment (GAD) screening may inform DMHA referral, presence of depression or anxiety screening on the date of referral encounter was included in primary analyses.
Referrals were described by encounter type (office, telephone, or videoconferencing). Provider attributes for each referral were characterized by training (physician or nonphysician), assigned specialty affiliation (primary care, specialty care, or behavioral health care), and location within the KPMAS region, given that referral patterns may vary by these characteristics.
Analysis
Patient, clinical, and encounter attributes of referrals were aggregated and reported overall and by referral cluster. Positive PHQ-2 and GAD-2 screening scores of ≥3 on the same day of referral were stratified by DMHA referral cluster but were not included in primary multivariable analyses, because only a subset of the sample completed the screenings. Monthly referral frequency trends for the 25-month observation period were reported overall, by app category, and by individual DMHA.
Multivariable, multinomial logistic regression models were used to compare patient, clinical, and encounter attributes across DMHA referral clusters. Models included patient demographic characteristics (age, race-ethnicity, gender), clinical characteristics (presence of PHQ screening, reason for visit, primary diagnosis), and encounter attributes (visit type, provider training, and provider location within KPMAS region). Provider specialty and GAD screening were described but not included in final models because of associations with other model variables. Referrals were clustered by patient to control for within-patient correlation. Post hoc margins tests were conducted to estimate the overall probability of patients being in each referral cluster and the probabilities of various patient, clinical, and encounter attributes occurring within each referral cluster. Margins tests were calculated by keeping all variables constant at their mean to estimate the sample average. To determine attribute variations within referral clusters, the difference between the overall sample average probability and the individual probability of each attribute occurring within the referral cluster was calculated and multiplied by 100 to reflect a percentage point difference. Differences for each attribute within a referral cluster were profiled in forest plots, with a positive difference indicating a higher probability of a given attribute being in the referral cluster and a negative difference indicating a lower probability of being in the referral cluster. The unit of analysis was the referral, and statistical significance was set to α=0.05 for all analyses.
Results
Overall Referral Attributes
A total of 43,842 initial DMHA referrals were made for 25,213 unique patients; 69% of patients had ≥2 referrals (Table 1). Of patients receiving ≥2 referrals, 91% (N=27,773 of 30,342) received all referrals on the same day. The combination of MM and CBT DMHAs was most often referred (54%), followed by MM-alone DMHAs (39%) and CBT-alone DMHAs (8%). Overall, the largest proportions of patients in the DMHA referral clusters were White (39%), in the 31–50-year age range (41%), and female (72%).
| | MM alone (N=17,010) | CBT alone (N=3,344) | MM and CBT (N=23,488) | Total (N=43,842) | ||||
|---|---|---|---|---|---|---|---|---|
| Attribute | N | % | N | % | N | % | N | % |
| DMHA | | | | | | | | |
| Calm | 11,621 | 68.3 | 0 | — | 7,309 | 31.1 | 18,930 | 43.2 |
| Headspace | 3,631 | 21.3 | 0 | — | 3,071 | 13.1 | 6,702 | 15.3 |
| Whil | 1,758 | 10.3 | 0 | — | 1,662 | 7.1 | 3,420 | 7.8 |
| myStrength | 0 | — | 2,361 | 70.6 | 7,485 | 31.9 | 9,846 | 22.5 |
| SilverCloud | 0 | — | 548 | 16.4 | 1,923 | 8.2 | 2,471 | 5.6 |
| Thrive | 0 | — | 435 | 13.0 | 2,038 | 8.7 | 2,473 | 5.6 |
| Age category | | | | | | | | |
| 18–30 years | 5,943 | 34.9 | 1,245 | 37.2 | 9,248 | 39.4 | 16,436 | 37.5 |
| 31–50 years | 6,954 | 40.9 | 1,340 | 40.1 | 9,728 | 41.4 | 18,022 | 41.1 |
| 51–64 years | 2,741 | 16.1 | 529 | 15.8 | 3,313 | 14.1 | 6,583 | 15.0 |
| ≥65 years | 1,372 | 8.1 | 230 | 6.9 | 1,199 | 5.1 | 2,801 | 6.4 |
| Race-ethnicity | | | | | | | | |
| Asian | 1,588 | 9.3 | 254 | 7.6 | 1,828 | 7.8 | 3,670 | 8.4 |
| Black | 5,653 | 33.2 | 1,294 | 38.7 | 9,726 | 41.4 | 16,673 | 38.0 |
| Hispanic | 1,914 | 11.3 | 249 | 7.5 | 2,151 | 9.2 | 4,314 | 9.8 |
| White | 7,162 | 42.1 | 1,415 | 42.3 | 8,714 | 37.1 | 17,291 | 39.4 |
| Other | 306 | 1.8 | 50 | 1.5 | 432 | 1.8 | 788 | 1.8 |
| Missing | 387 | 2.3 | 82 | 2.5 | 637 | 2.7 | 1,106 | 2.5 |
| Gender | | | | | | | | |
| Male | 4,805 | 28.2 | 1,025 | 30.7 | 6,312 | 26.9 | 12,142 | 27.7 |
| Female | 12,205 | 71.8 | 2,319 | 69.3 | 17,176 | 73.1 | 31,700 | 72.3 |
| PHQ screen on date of encounterb | | | | | | | | |
| Not completed/missing | 5,557 | 32.7 | 805 | 24.1 | 6,271 | 26.7 | 12,633 | 28.8 |
| Completed | 11,453 | 67.3 | 2,539 | 75.9 | 17,217 | 73.3 | 31,209 | 71.2 |
| GAD screen on date of encounterc | | | | | | | | |
| Not completed/missing | 6,050 | 35.6 | 961 | 28.7 | 6,977 | 29.7 | 13,988 | 31.9 |
| Completed | 10,960 | 64.4 | 2,383 | 71.3 | 16,511 | 70.3 | 29,854 | 68.1 |
| Number of DMHAs referred | | | | | | | | |
| 1 | 10,717 | 63.0 | 2,783 | 83.2 | 0 | — | 13,500 | 30.8 |
| 2 | 4,154 | 24.4 | 492 | 14.7 | 10,812 | 46.0 | 15,458 | 35.3 |
| 3 | 2,139 | 12.6 | 69 | 2.1 | 4,311 | 18.4 | 6,519 | 14.9 |
| 4 | 0 | — | 0 | — | 4,176 | 17.8 | 4,176 | 9.5 |
| 5 | 0 | — | 0 | — | 2,065 | 8.8 | 2,065 | 4.7 |
| 6 | 0 | — | 0 | — | 2,124 | 9.0 | 2,124 | 4.8 |
| Reason for visit | | | | | | | | |
| Anxiety, depression, stress | 6,520 | 38.3 | 990 | 29.6 | 6,699 | 28.5 | 14,209 | 32.4 |
| Wellness coaching, education, counseling | 266 | 1.6 | 58 | 1.7 | 256 | 1.1 | 580 | 1.3 |
| Annual care | 2,397 | 14.1 | 477 | 14.3 | 2,890 | 12.3 | 5,764 | 13.1 |
| Other | 6,335 | 37.2 | 1,499 | 44.8 | 12,523 | 53.3 | 20,357 | 46.4 |
| Missing | 1,492 | 8.8 | 320 | 9.6 | 1,120 | 4.8 | 2,932 | 6.7 |
| Primary diagnosis | | | | | | | | |
| Depression (single episode or recurrent) | 2,625 | 15.4 | 826 | 24.7 | 4,664 | 19.9 | 8,115 | 18.5 |
| Unspecified mood disorder | 389 | 2.3 | 112 | 3.3 | 686 | 2.9 | 1,187 | 2.7 |
| Other anxiety disorders | 6,037 | 35.5 | 843 | 25.2 | 6,602 | 28.1 | 13,482 | 30.8 |
| Other | 7,933 | 46.6 | 1,556 | 46.5 | 11,509 | 49.0 | 20,998 | 47.9 |
| Missing | 26 | .2 | 7 | .2 | 27 | .1 | 60 | .1 |
| Provider training | | | | | | | | |
| Physician | 5,573 | 32.8 | 1,024 | 30.6 | 6,385 | 27.2 | 12,982 | 29.6 |
| Nonphysician | 11,437 | 67.2 | 2,320 | 69.4 | 17,101 | 72.8 | 30,858 | 70.4 |
| Missing | 0 | — | 0 | — | 2 | <.1 | 2 | <.1 |
| Provider specialty | | | | | | | | |
| Behavioral health care | 15,026 | 88.3 | 2,814 | 84.2 | 20,736 | 88.3 | 38,576 | 88.0 |
| Primary care | 1,908 | 11.2 | 522 | 15.6 | 2,722 | 11.6 | 5,152 | 11.8 |
| Specialty care | 69 | .4 | 8 | .2 | 15 | .1 | 92 | .2 |
| Missing | 7 | <.1 | 0 | — | 15 | .1 | 22 | <.1 |
| Visit type | | | | | | | | |
| Office | 3,307 | 19.4 | 549 | 16.4 | 2,626 | 11.2 | 6,482 | 14.8 |
| Telephone | 1,666 | 9.8 | 314 | 9.4 | 2,477 | 10.5 | 4,457 | 10.2 |
| Videoconferencing | 12,037 | 70.8 | 2,481 | 74.2 | 18,385 | 78.3 | 32,903 | 75.0 |
| Provider location | | | | | | | | |
| Baltimore | 3,487 | 20.5 | 675 | 20.2 | 3,139 | 13.4 | 7,301 | 16.7 |
| District of Columbia, southern Maryland | 6,909 | 40.6 | 1,444 | 43.2 | 12,291 | 52.3 | 20,644 | 47.1 |
| Northern Virginia | 6,608 | 38.8 | 1,224 | 36.6 | 8,042 | 34.2 | 15,874 | 36.2 |
| Missing | 6 | <.1 | 1 | <.1 | 16 | .1 | 23 | <.1 |
a
MM, mindfulness and meditation; CBT, cognitive-behavioral therapy; PHQ, Patient Health Questionnaire; GAD, Generalized Anxiety Disorder assessment.
b
Either the PHQ-2 or the first two questions of the PHQ-9.
c
Either the GAD-2 or the first two questions of the GAD-7.
Fifty-two percent of referrals had a primary psychiatric diagnosis. Same-day PHQ screening occurred for 71% of referrals, whereas GAD screening was performed concomitantly for 68% of referrals. The percentage of referrals with a positive PHQ-2 score on the same day was highest for CBT alone (47%, N=1,202 of 2,537), followed by MM and CBT (46%, N=7,848 of 17,197) and MM alone (41%, N=4,729 of 11,441) (χ2=62.88, df=2, p<0.001). In contrast, the percentage of referrals with a positive GAD-2 score on the same day was highest for MM and CBT (64%, N=10,585 of 16,494), followed by MM alone (63%, N=6,883 of 10,948) and CBT alone (60%, N=1,436 of 2,380) (χ2=15.22, df=2, p<0.001). Within the category of “other” reason for visit, MM-alone referrals were most commonly made during psychotherapy (34%, N=2,142 of 6,335), videoconferencing visits (13%, N=792 of 6,335), and medication management encounters (11%, N=719 of 6,335). CBT-alone referrals were most commonly made during psychosocial assessment (40%, N=603 of 1,499), psychotherapy (19%, N=279 of 1,499), and videoconferencing visits (6%, N=84 of 1,499). MM and CBT referrals were most commonly made during psychosocial assessment (42%, N=5,221 of 12,523), medication management encounters (21%, N=2,631 of 12,523), and psychotherapy (13%, N=1,615 of 12,523) (online supplement ). Within the category of “other” primary diagnosis, MM-alone (60%, N=4,725 of 7,933), MM and CBT (60%, N=6,960 of 11,509), and CBT-alone (50%, N=778 of 1,556) referrals most commonly and consistently included a diagnosis of reaction to severe stress (online supplement ).
Referrals were mostly made by nonphysicians (70%), including clinical social workers (76%, N=23,604 of 30,858), professional counselors (22%, N=6,906 of 30,858), psychiatric nurse specialists (0.9%, N=276 of 30,858), and psychologists (0.2%, N=72 of 30,858). Referrals were most common at videoconferencing visits (75%), within the DCSM location (47%), and from behavioral health care providers (88%). Mean referral rates over the 25-month study period were 84.9, 100.2, and 104.4 per referring provider for the Baltimore, DCSM, and NoVA locations, respectively.
Referral Trends
Total referrals fluctuated throughout the observation period (Figure 1A). Four months after initiative rollout, total monthly referrals accelerated and peaked at 7 months postrollout. Referrals then decelerated and reached equilibrium at 10 months postrollout, until another substantial increase was observed at 16 months postrollout. Thereafter, monthly referrals began a sustained decline until the end of the observation period at 25 months postrollout. Referral trends for CBT and MM DMHAs mirrored the total trends (Figure 1A). Calm was the most referred DMHA throughout the observation period; Headspace and myStrength alternated as the second and third most commonly referred DMHAs for most of the observation period (Figure 1B).
FIGURE 1. Referral counts by referral cluster (type of digital mental health application) and by individual app, December 2019–December 2021a
aA: referral counts are aggregated by month and type of app. Mindfulness and meditation apps include Calm, Headspace, and Whil. Cognitive-behavioral therapy apps include Thrive, myStrength, and SilverCloud. Notable events during the observation period are annotated. HP, health plan; CME, continuing medical education. B: referral counts are aggregated by month and individual app.
Predicted Probabilities of Attributes Being Within Each Referral Cluster
After adjustment for patient, clinical, and encounter attributes, overall predicted percentages of referrals were 55.2% for MM and CBT apps, 37.5% for MM alone, and 7.2% for CBT alone. Results for each type of attribute are described below.
Patient Attributes
MM and CBT cluster.
Compared with the referral cluster average, predicted percentage points of referrals were significantly higher for patients ages 18–30 years (2.45) and for Black patients (3.14). In contrast, predicted percentage points of referrals were significantly lower for patients who were ages 51–64 years (−3.27), ages ≥65 years (−9.62), Asian (−3.57), Hispanic (−4.93), and male (−1.57) (Figure 2).
FIGURE 2. Differences in probability of referral for both a cognitive-behavioral therapy and a mindfulness and meditation digital mental health application, by patient, clinical, and provider attributes and compared with the sample averagea
aPredicted probabilities, derived from the multinomial logistic regression and post hoc margins testing, represent the likelihood of an attribute being in the referral cluster of mindfulness and meditation and cognitive-behavioral therapy, given that the stated characteristic is present (e.g., male gender or physician provider training), with all other covariates held constant at their means. To obtain the difference, the average predicted probability for the sample is subtracted from the individual predicted probability for each characteristic. Predicted percentage point is calculated by multiplying the difference by 100. Values in parentheses and error bars represent 95% CIs. The reason-for-visit category of mental health included anxiety, depression, or stress; the reason-for-visit category of learning included wellness coaching, education, or counseling. PHQ, Patient Health Questionnaire screening on the date of encounter (either the PHQ-2 or the first two questions of the PHQ-9); DCSM, District of Columbia and southern Maryland; NoVA, Northern Virginia.
MM-alone cluster.
Predicted percentage points of referrals were significantly higher for patients who were ages 51–64 years (2.84), ages ≥65 years (9.04), Asian (4.34), and Hispanic (6.79), compared with the referral cluster average. However, predicted percentage points were significantly lower for patients who were ages 18–30 years (−2.38) and Black (−3.45) (Figure 3).
FIGURE 3. Differences in probability of referral for a mindfulness and meditation digital mental health application, by patient, clinical, and provider attributes and compared with the sample averagea
aPredicted probabilities, derived from the multinomial logistic regression and post hoc margins testing, represent the likelihood of an attribute being in the referral cluster of mindfulness and meditation, given that the stated characteristic is present (e.g., male gender or physician provider training), with all other covariates held constant at their means. To obtain the difference, the average predicted probability for the sample is subtracted from the individual predicted probability for each characteristic. Predicted percentage point is calculated by multiplying the difference by 100. Values in parentheses and error bars represent 95% CIs. The reason-for-visit category of mental health included anxiety, depression, or stress; the reason-for-visit category of learning included wellness coaching, education, or counseling. PHQ, Patient Health Questionnaire screening on the date of encounter (either the PHQ-2 or the first two questions of the PHQ-9); DCSM, District of Columbia and southern Maryland; NoVA, Northern Virginia.
CBT-alone cluster.
Compared with the referral cluster average, predicted percentage points of referrals were significantly higher for male patients (0.92) and significantly lower for Hispanic patients (−1.86) (Figure 4).
FIGURE 4. Differences in probability of referral for a cognitive-behavioral therapy digital mental health application, by patient, clinical, and provider attributes and compared with the sample averagea
aPredicted probabilities, derived from the multinomial logistic regression and post hoc margins testing, represent the likelihood of an attribute being in the referral cluster of cognitive-behavioral therapy, given that the stated characteristic is present (e.g., male gender or physician provider training), with all other covariates held constant at their means. To obtain the difference, the average predicted probability for the sample is subtracted from the individual predicted probability for each characteristic. Predicted percentage point is calculated by multiplying the difference by 100. Values in parentheses and error bars represent 95% CIs. The reason-for-visit category of mental health included anxiety, depression, or stress; the reason-for-visit category of learning included wellness coaching, education, or counseling. PHQ, Patient Health Questionnaire screening on the date of encounter (either the PHQ-2 or the first two questions of the PHQ-9); DCSM, District of Columbia and southern Maryland; NoVA, Northern Virginia.
Clinical Attributes
MM and CBT cluster.
Predicted percentage points of referrals were significantly higher than the referral cluster average for patients reporting a reason for visit of “other” (6.25) and having a primary diagnosis of depression (single episode or recurrent) (3.66). Predicted percentage points of referrals were significantly lower for patients not completing a PHQ screen on the encounter date (−1.56), reporting a reason for visit of mental health (−8.89), and having a primary diagnosis of other anxiety disorders (−3.41) (Figure 2).
MM-alone cluster.
Compared with the referral cluster average, predicted percentage points of referrals were significantly higher for patients not completing a PHQ screen on the encounter date (3.25), reporting a reason for visit of mental health (9.40), and having a primary diagnosis of other anxiety disorders (4.71). In contrast, predicted percentage points of referrals were significantly lower for patients reporting a reason for visit of “other” (−6.29) and having a diagnosis of depression (single episode or recurrent; −6.34) or unspecified mood disorder (−4.98) (Figure 3).
CBT-alone cluster.
Predicted percentage points of referrals were significantly higher for patients with a primary diagnosis of depression (single episode or recurrent; 2.68) and completing a PHQ screen on the encounter date (0.74), compared with the referral cluster average. However, patients with a primary diagnosis of other anxiety disorders (−1.30), with a reason for visit of mental health (−0.51), and not completing a PHQ screen on the encounter date (−1.69) had significantly lower predicted percentage points of referrals when compared with the referral cluster average (Figure 4).
Encounter Attributes
MM and CBT cluster.
Predicted percentage points of referrals were significantly higher than the referral cluster average for patients who received care in the DCSM area (6.75) and had a videoconferencing visit (1.84) and were significantly lower for patients who received care in the Baltimore (−10.31) and NoVA (−4.16) areas and had office-based visits (−11.66) (Figure 2).
MM-alone cluster.
Compared with the referral cluster average, predicted percentage points of referrals were significantly higher for patients who received care in the Baltimore (8.30) and NoVA (3.82) areas and had office-based visits (9.79). In contrast, predicted percentage points of referrals were significantly lower for patients who received care in the DCSM area (−5.79) and had telephone (−2.29) and videoconferencing visits (−1.50) (Figure 3).
CBT-alone cluster.
Predicted percentage points of referrals were significantly higher than the referral cluster average for patients who received care in the Baltimore area (2.00) and had office-based visits (1.87) but were significantly lower for patients who received care in the DCSM area (−0.97) and from physician providers (−1.10) (Figure 4).
Discussion
Our results provide initial insight into providers’ DMHA referral patterns and describe variations that may be expected when integrating DMHA referrals into clinical workflows on a large scale. DMHA referral attributes differed from the general KPMAS (Baltimore, DCSM, and NoVA) overall membership estimates from 2020; participants in our study were more likely to be White (39% vs. 26%) and female (72% vs. 53%) and less likely to be ages ≥65 (6% vs. 15%) and Hispanic (10% vs. 14%). Providers readily engaged in DMHA referral after the initial initiative rollout. After 4 months, referrals accelerated and then reached equilibrium at 6–7 months. Referrals steadily declined from month 16 to the end of the 25-month observation period. Last, clear variations in the attributes of each DMHA referral cluster resulted in unique profiles.
Observed increases in DMHA referrals during the first 4 months of the COVID-19 pandemic in the United States, with some stabilization thereafter, were consistent with existing research (28). Given that the estimated overall number of visits during a 4-week period at KPMAS locations between May 3 and June 20, 2020, was 222,000 (29) and that approximately 2,000 DMHA referrals were made per month, on average, during the study period, referrals occurred at <1% of visits. Annual new member enrollment in the health plan that typically coincides with the beginning of the calendar year increases the number of members who are eligible for referral and likely contributed to observed increases during early periods of each calendar year. Stand-alone continuing medical education programs had mixed effects on referral patterns, suggesting the need for ongoing provider training. The COVID-19 pandemic, a seminal disruptive event, appears to have initially accelerated overall DMHA referrals. However, the decline in referrals during the later months of the observation period suggests the need for future research to assess the sustainability and clinical impact of DMHA referrals.
Providers frequently initiated multiple DMHA referrals on the same encounter date. Simultaneous DMHA referrals may be efficient and flexible and may support patient access and choice, but they also may overwhelm and confuse patients in the absence of ongoing follow-up. Multiple referrals at the same time may also reflect provider uncertainty, underscoring the need for continuing education about best practices for implementing DHMA referrals in a comprehensive care plan. Evaluation of a more stepped approach to DMHA referral that includes periodic assessments may be warranted, given the potential confusion about DMHA use in clinical practice.
Referral patterns reflected a range of patient demographic characteristics, clinical characteristics, and encounter attributes, and variations among DMHA referral clusters emerged. For example, MM-alone referrals were more likely to be given to patients who were ages 51 or older, were Hispanic or Asian, reported a reason for visit of mental health, or had a primary diagnosis of other anxiety disorders. CBT-alone referrals were more likely to be given to patients with a primary diagnosis of depression (single episode or recurrent) and less likely to be made for Hispanic patients. MM and CBT referrals were more likely to be ordered for patients who were ages 18–30, reported a reason for visit of “other,” or had a primary diagnosis of depression (single episode or recurrent), and referrals were less likely to be given to Hispanic patients or to patients ages ≥65.
Observed variations in referral clusters may be explained by perceived technological savvy (e.g., ability to easily understand and use technology), cultural receptivity, and clinical need assessed at the clinical encounter. Historically, the “digital divide” between those with and without adequate access to contemporary technology has often made the process of obtaining care more difficult for certain subgroups of the general population (30). Although organizational commitment to a digital mental health initiative (26), as observed in the KPMAS system, likely removed some of the common access barriers (e.g., registration fees, lack of formal clinical pathways), additional effort is clearly needed to ensure equitable access to and use of digital tools (15). Thought leaders have proposed provider education in five areas of competency to enhance best practices: evidence, integration, security and privacy, ethics, and cultural considerations (31). Future research should further evaluate sociodemographic and cultural differences in referral patterns.
Providers appear to consider a patient’s condition when making a DMHA referral. For example, the primary diagnosis of other anxiety disorders was more common within the MM-alone referral cluster. Moreover, the primary diagnosis of depression (single episode or recurrent) was more common in both the MM and CBT and CBT-alone referral clusters, suggesting that a diagnosis of depression may encourage the likelihood of referral for a CBT DMHA (with or without an MM DMHA). In addition, the frequencies of primary diagnoses within the category of “other” varied among referral clusters; for example, a sleep disorder diagnosis was more common in the CBT-alone referral cluster than in the MM-alone and MM and CBT referral clusters. Reaction to severe stress was less common in the CBT-alone referral cluster than in the MM and CBT and MM-alone referral clusters, which also suggests that providers are making referrals with the primary diagnosis in mind. Bivariate analyses suggest that a relationship may exist between a positive PHQ-2 or GAD-2 screen and DMHA referral cluster, although more detailed investigation of this relationship was beyond the scope of this study and requires more comprehensive analysis to understand the implications.
Although MM and CBT have some crossover in their strategies for treating depression and anxiety, our findings suggest that there may be some degree of alignment between clinical need and compatible therapeutic strategy. Consistent with existing literature, MM can address the hyperarousal that is characteristic of anxiety disorders, such as increased heart rate, panic, and restlessness. CBT can integrate active engagement and the challenging of negative cognitive ruminations to treat depression (32–34). However, when the primary diagnosis is categorized as “other” (online supplement ), which may reflect a broad set of needs, providers more commonly refer MM and CBT simultaneously, underscoring potential uncertainty about which type of DMHA to recommend. Collectively, these findings suggest the need to further evaluate the use and outcomes of DMHAs in specific conditions, given that there may be shared experience of some comorbid conditions (e.g., primary anxiety that contributes to secondary depression over time).
Compared with the referral cluster average, providers at DCSM locations were more likely to refer MM and CBT apps, those at NoVA locations were more likely to refer MM apps, and providers in Baltimore were more likely to refer CBT apps. Regional variations suggest that referral patterns may differ even within one organization and may reflect diverse needs among patients within regions. Further exploration is required to understand potential sources of these variations.
Finally, within the MM and CBT cluster, there was a higher probability for referral (vs. the sample average) during videoconferencing visits. This finding is consistent with observed increases in weekly visit trends for both behavioral health and videoconferencing visits within the KPMAS system during the early phases of the COVID-19 pandemic (26). In contrast, within the MM-alone and CBT-alone clusters, the probability of referral was higher (vs. the sample average) during office-based visits. These findings may suggest that providers place more focused and targeted attention on the nuances of a particular DMHA when they see patients in person, whereas the higher likelihood of a combined MM and CBT DMHA referral during videoconferencing visits may reflect that a provider perceives the patient to have greater comfort with technology and therefore may engage in a more general referral strategy.
This exploratory observational study was limited to information in an electronic health record. Our study described only referrals to DMHAs and did not evaluate DMHA enrollment, patient experience, or the clinical consequences of DMHA use. In addition, we were unable to explore the rationale and sequencing of the referrals as well as the patient-provider dialogue that may have encouraged use of DMHAs. However, the findings raise important questions for future studies. Generalizability may be limited to settings where the cost of DMHA subscriptions can be covered by a health system or insurance plan.
Conclusions
Referrals to DMHAs included an array of patient, clinical, and encounter attributes. Monthly referrals rapidly accelerated during the early months of the COVID-19 pandemic, with a subsequent plateau and then a decline during the extended pandemic period. Observed patient, clinical, and encounter attribute variations by referral cluster support the need to further evaluate provider decision making and whether referral patterns lead to optimal patient use. Future work is needed to further explore the end-user DMHA experience, including adherence and persistence over time, and DMHAs’ subsequent clinical impact.
Supplementary Material
File (appi.ps.20220401.ds001.pdf)
- View/Download
- 137.27 KB
References
1.
Kessler RC, Berglund P, Demler O, et al: Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry 2005; 62:593–602
2.
Hasin DS, Sarvet AL, Meyers JL, et al: Epidemiology of adult DSM-5 major depressive disorder and its specifiers in the United States. JAMA Psychiatry 2018; 75:336–346
3.
Murray CJL, Lopez AD: The Global Burden of Disease: A Comprehensive Assessment of Mortality and Disability From Diseases, Injuries, and Risk Factors in 1990 and Projected to 2020: Summary. Boston, Harvard School of Public Health and World Health Organization, 1996. https://apps.who.int/iris/bitstream/handle/10665/41864/0965546608_eng.pdf?sequence=1&isAllowed=y
4.
Porensky EK, Dew MA, Karp JF, et al: The burden of late-life generalized anxiety disorder: effects on disability, health-related quality of life, and healthcare utilization. Am J Geriatr Psychiatry 2009; 17:473–482
5.
Lenze EJ, Rogers JC, Martire LM, et al: The association of late-life depression and anxiety with physical disability: a review of the literature and prospectus for future research. Am J Geriatr Psychiatry 2001; 9:113–135
6.
Brenes GA: Anxiety, depression, and quality of life in primary care patients. Prim Care Companion J Clin Psychiatry 2007; 9:437–443
7.
Vahratian A, Blumberg SJ, Terlizzi EP, et al: Symptoms of anxiety or depressive disorder and use of mental health care among adults during the COVID-19 pandemic—United States, August 2020–February 2021. MMWR Morb Mortal Wkly Rep 2021; 70:490–494
8.
Hawes MT, Szenczy AK, Klein DN, et al: Increases in depression and anxiety symptoms in adolescents and young adults during the COVID-19 pandemic. Psychol Med 2022; 52:3222–3230
9.
Olfson M: Building the mental health workforce capacity needed to treat adults with serious mental illnesses. Health Aff 2016; 35:983–990
10.
Thomas KC, Ellis AR, Konrad TR, et al: County-level estimates of mental health professional shortage in the United States. Psychiatr Serv 2009; 60:1323–1328
11.
Roll JM, Kennedy J, Tran M, et al: Disparities in unmet need for mental health services in the United States, 1997–2010. Psychiatr Serv 2013; 64:80–82
12.
Hofmann SG, Sawyer AT, Witt AA, et al: The effect of mindfulness-based therapy on anxiety and depression: a meta-analytic review. J Consult Clin Psychol 2010; 78:169–183
13.
Hollon SD, Stewart MO, Strunk D: Enduring effects for cognitive behavior therapy in the treatment of depression and anxiety. Annu Rev Psychol 2006; 57:285–315
14.
Carlo AD, Hosseini Ghomi R, Renn BN, et al: By the numbers: ratings and utilization of behavioral health mobile applications. NPJ Digit Med 2019; 2:54
15.
Satre DD, Meacham MC, Asarnow LD, et al: Opportunities to integrate mobile app–based interventions into mental health and substance use disorder treatment services in the wake of COVID-19. Am J Health Promot 2021; 35:1178–1183
16.
Proudfoot J, Clarke J, Birch M-R, et al: Impact of a mobile phone and Web program on symptom and functional outcomes for people with mild-to-moderate depression, anxiety and stress: a randomised controlled trial. BMC Psychiatry 2013; 13:312
17.
Huberty J, Eckert R, Larkey L, et al: Smartphone-based meditation for myeloproliferative neoplasm patients: feasibility study to inform future trials. JMIR Form Res 2019; 3:e12662
18.
Huberty J, Green J, Glissmann C, et al: Efficacy of the mindfulness meditation mobile app “Calm” to reduce stress among college students: randomized controlled trial. JMIR mHealth uHealth 2019; 7:e14273
19.
Mani M, Kavanagh DJ, Hides L, et al: Review and evaluation of mindfulness-based iPhone apps. JMIR mHealth uHealth 2015; 3:e82
20.
Hirsch A, Luellen J, Holder JM, et al: Managing depressive symptoms in the workplace using a Web-based self-care tool: a pilot randomized controlled trial. JMIR Res Protoc 2017; 6:e51
21.
Richards D, Enrique A, Eilert N, et al: A pragmatic randomized waitlist-controlled effectiveness and cost-effectiveness trial of digital interventions for depression and anxiety. NPJ Digit Med 2020; 3:85
22.
Schure MB, Lindow JC, Greist JH, et al: Use of a fully automated Internet-based cognitive behavior therapy intervention in a community population of adults with depression symptoms: randomized controlled trial. J Med Internet Res 2019; 21:e14754
23.
Firth J, Torous J, Nicholas J, et al: The efficacy of smartphone-based mental health interventions for depressive symptoms: a meta-analysis of randomized controlled trials. World Psychiatry 2017; 16:287–298
24.
Lagan S, Emerson MR, King D, et al: Mental health app evaluation: updating the American Psychiatric Association’s framework through a stakeholder-engaged workshop. Psychiatr Serv 2021; 72:1095–1098
25.
McGee-Vincent P, Mackintosh MA, Jamison AL, et al: Training staff across the Veterans Affairs health care system to use mobile mental health apps: a national quality improvement project. JMIR Ment Health 2023; 10:e41773
26.
Mordecai D, Histon T, Neuwirth E, et al: How Kaiser Permanente created a mental health and wellness digital ecosystem. NEJM Catal Innov Care Deliv 2020; 2:0295
27.
Wasil AR, Gillespie S, Shingleton R, et al: Examining the reach of smartphone apps for depression and anxiety [letter]. Am J Psychiatry 2020; 177:464–465
28.
Sorkin DH, Janio EA, Eikey EV, et al: Rise in use of digital mental health tools and technologies in the United States during the COVID-19 pandemic: survey study. J Med Internet Res 2021; 23:e26994
29.
Miller MJ, Watson ES, Horberg MA, et al: Patient experience after modifying visit delivery during the COVID-19 pandemic. Am J Manag Care 2021; 27:e54–e63
30.
Ramsetty A, Adams C: Impact of the digital divide in the age of COVID-19. J Am Med Inform Assoc 2020; 27:1147–1148
31.
Schueller SM, Armstrong CM, Neary M, et al: An introduction to core competencies for the use of mobile apps in cognitive and behavioral practice. Cogn Behav Pract 2022; 29:69–80
32.
Vøllestad J, Nielsen MB, Nielsen GH: Mindfulness‐ and acceptance‐based interventions for anxiety disorders: a systematic review and meta‐analysis. Br J Clin Psychol 2012; 51:239–260
33.
Zhou X, Guo J, Lu G, et al: Effects of mindfulness-based stress reduction on anxiety symptoms in young people: a systematic review and meta-analysis. Psychiatry Res 2020; 289:113002
34.
Ekers D, Richards D, Gilbody S: A meta-analysis of randomized trials of behavioural treatment of depression. Psychol Med 2008; 38:611–623
Information & Authors
Information
Published In
History
Received: 4 August 2022
Revision received: 11 November 2022
Revision received: 28 February 2023
Accepted: 17 April 2023
Published online: 26 July 2023
Published in print: January 01, 2024
Keywords
Authors
Competing Interests
Dr. McGuire has received research support from the Tourette Association of America, the American Academy of Neurology, the Brain Research Foundation, the American Psychological Foundation, the American Psychological Association, the Hilda & Preston Davis Foundation, and the Misophonia Research Fund. He has received royalties from Elsevier and an honorarium from Springer for editorial responsibilities, and he serves as a consultant for Syneos Health. Dr. Miller receives an honorarium for his role as a senior associate editor of the American Journal of Health-System Pharmacy. The other authors report no financial relationships with commercial interests.
Funding Information
Funding for the project was provided in part by the Kaiser Permanente Mid-Atlantic States Community Benefits Program and the Johns Hopkins Clinical and Translational Service Award grant from the National Center for Advancing Translational Sciences, NIH (5UL1TR003098).The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBLogin options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).