Process Evaluation of an Early-Intervention Program for Mood and Anxiety Disorders Among Older Adolescents and Young Adults

The First-Episode Mood and Anxiety Program (FEMAP) was designed to reduce barriers for youths seeking help for significant mood or anxiety concerns. A formative evaluation of the causal and logic models of FEMAP was published previously (8). [Modified versions of the models are available as an online supplement to this report.] This report relates to steps 2–4 of phase I of the causal model; the community outreach component (step 1) of phase I was described previously and has remained unchanged (8).

FEMAP was designed for youths already experiencing symptoms severe enough to warrant specialty services. In that sense, “first episode” is euphemistic. Phase I contains several unique aspects, including location in a youth-friendly setting; an option for self-referral; detailed assessment to determine placement in the correct treatment setting and to build rapport between youths and clinical staff; use of clinical case conferences to determine optimal treatment; and rapid enrollment in FEMAP, facilitation of referral to alternative services—if indicated—or reassurance if no specialized treatment appears warranted.

Initial assessment was conducted during in-person, customary clinical interviews by a licensed, master’s-level social worker (CS or CF) or psychologist (ER) with diagnostic, symptom, situational, and functional assessments. At the conclusion of the interview, participants completed research questionnaires. Most participants completed the questionnaire in person, but, occasionally, a questionnaire was completed by phone or e-mail. A participant’s diagnostic category was determined by these clinical interviews and questionnaires. A clinical case conference ensued to establish optimal treatment for each patient and the urgency of treatment.

Youths ages 16 to 26 were enrolled. All participants either were referred by a physician or contacted FEMAP without a physician’s referral. Youths with lifetime medication treatment lasting longer than 18 months were excluded because using medication for that length of time is an indicator of having received significant care from a previous psychiatric provider or service. Youths with developmental delays, major general medical problems, history of head injury resulting in a significant period of unconsciousness, and lengthy medication use for treatment of attention-deficit hyperactivity disorder (ADHD) were excluded and referred to alternative services. Youths who endorsed psychiatric symptoms that began only after significant alcohol or illicit drug use were referred to community addiction services and excluded. Youths evaluated to be in imminent danger to self or others were immediately referred to emergency services until they were more stable. All youths who presented to FEMAP between October 2009 and November 2012 were asked to enroll. Willing participants signed written consent to participate, as approved by the Human Research Ethics Board for the University of Western Ontario.

Measures included the Beck Depression Inventory–II (BDI); the State Trait Anxiety Inventory (STAI); the Sheehan Disability Scale (SDS); the Structured Clinical Interview for DSM-IV screening questionnaire for substance use, anxiety, and eating disorders; the Emotion Regulation Questionnaire (ERQ); the National Institute on Drug Abuse Alcohol, Smoking and Substance Involvement Screening Test (NIDA ASSIST)–Prescreen V1.0; the Adult ADHD Self-Report Scale, version 1.1, part A (ASRS); and the Youth Risk Behavior Survey (YRBS). A demographic screening questionnaire developed by the investigators recorded participants’ emotional concerns, psychiatric treatment history, and demographic characteristics (8).

Three months or longer after initial contact with FEMAP, participants were contacted for follow-up, regardless of acceptance to or treatment at FEMAP. Three attempts were made to contact each participant at scheduled clinical appointments or by phone or e-mail. The follow-up consisted of a questionnaire about their original mental health concern or concerns, interval changes to mental health and addiction diagnoses and treatments, and their experiences with their mental health treatment. In addition, the SDS was readministered to evaluate functional impairment. The follow-up was conducted to assess patient follow-through with recommendations during phase I of the causal model and not to evaluate the effectiveness of treatment per se.

All statistical analyses used SPSS, version 21. Missing data were omitted from individual analyses.

Of 830 youths who presented for services, 560 (67%) youths agreed to participate in research; the remaining youths were treated identically but did not participate in data collection; there was no incentive for participation. Results are reported for 548 youths (12 youths had extensive missing data).

Of the 548 youths, 399 (73%) were accepted to FEMAP, and 149 (28%) were referred to more appropriate services or, in rare cases, were reassured only. In both subgroups (accepted and not accepted), approximately two-thirds were female (N=243, 61%, and N=98, 66%, respectively), and most came without a physician’s referral (N=241, 60%, and N=91, 61%, respectively), had received prior treatment (N=253, 63%, and N=89, 60%, respectively), and had received medication (N=206 and N=77, respectively, 52% each). [Tables summarizing demographic and clinical characteristics, including referral sources, of the youths are available in the online supplement.] No between-group differences in demographic and clinical characteristics were found, although a trend indicated that accepted youths were more likely to complete the follow-up questionnaires.

Accepted participants were more likely than those who were not accepted to have a presumptive diagnosis in a mood or anxiety disorder category (p<.001), indicating that the assessment model correctly identified appropriate youths. Youths with presumed diagnoses that fell in categories outside the mood and anxiety categories were less likely to be accepted. Diagnosis of a primary substance use disorder was low, as expected, given that having such was a criterion for exclusion. Nevertheless, over 50% of all participants were at moderate or high risk of a substance use problem. [A table summarizing the participants’ presumptive diagnostic categories is available in the supplement.]

The severity of depression symptoms at intake was high, both for youths who were accepted by FEMAP and youths who were not accepted (Table 1). Anxiety was moderate for both subgroups. SDS scores indicated clinical concern for both groups. Compared with youths who were not accepted by FEMAP, accepted youths had higher ratings for depression (F=5.4, df=1 and 527, p=.02); worse scores on the ERQ, indicating poorer emotional regulation (F=15.1, df=1 and 529, p<.001); and greater functional impairment on the SDS (F=5.7, df=1 and 519, p=.018) but did not differ on other measures. Youths who were not referred by a physician had higher scores on the STAI (F=8.5 df=1, p=.004), indicating greater anxiety, and the NIDA ASSIST (F=3.8, df=1, p=.05), indicating higher risk of drug and alcohol use, compared with youths who were referred by a physician. There was no difference in severity of functional impairment on the SDS between the physician-referred youths and the youths who came without a physician referral.

Among youths who completed follow-up questionnaires, FEMAP-accepted youths were more likely than youths who were not accepted to FEMAP to follow through with recommendations made at initial intake (χ²=30.6, df=1, p<.001). About half of all youths (N=282, 51.5%) completed follow-up questionnaires, consistent with follow-up rates in other adolescent research (9). Sixteen independent variables were hypothesized to relate to follow-up loss. After removing redundant (highly correlated) and noncontributory (Spearman correlation) variables, eight remained. Binary logistic regression was performed to assess the impact of these remaining variables, measured at study entry, on absence of follow-up. Included were gender, final presumptive diagnosis, lifetime marijuana use, physician referral or no physician referral, accepted versus referred or reassured, parental marital status, ASRS score, and ERQ score.

The full model was statistically significant (χ²=49.3, df=19, p<.001) and correctly classified the follow-up status of 63% of cases. Three variables were statistically significant: lifetime use of marijuana, driven primarily by the 164 youths who endorsed using marijuana on 100 or more occasions (odds ratio [OR]=1.88, 95% confidence interval [CI]=1.13–3.13, compared with no use); having unmarried or non-cohabiting parents (OR=1.56, CI=1.06–2.30, compared with married parents); and entry into FEMAP without a physician’s referral (OR=.67, CI=.46–.99, compared with physician’s referral), suggesting that self-referral was protective against loss to follow-up.

Preliminary evaluation of clinical improvement was conducted by comparing SDS scores at intake and follow-up for all youths who completed the SDS at both time points. A repeated-measures, general linear model with one within-subjects factor (time) and one between-subjects factor (accepted versus referred or reassured) was used to assess improvement associated with three related dependent variables (SDS subscales for school or work, social, and family). Multivariate results showed a main effect for time (F=23.1, df=3, p<.001) and no other main effects or interactions. Univariate results showed improvement on school or work (F=62.3, df=1, p<.001), social (F=30.4, df=1, p<.001), and family (F=24.1, df=1, p<.001) subscales.

This process evaluation assessed implementation and coverage of the first phase of a novel early-intervention model for transition-age youths with mood or anxiety concerns. This model offered a unique approach by virtue of being located in the community and being easily and independently accessible to youths. It included a detailed, in-person intake assessment with a skilled clinician and a case conference to optimize treatment initiation. Recommendations and assistance related to accessing the most appropriate treatment—FEMAP or other programs—were provided.

The model did not demand training of clinicians beyond the standard required by their disciplines for mental health care specialists. It preserved the services of the most highly paid professionals (psychiatrists) until phase II of the model, after needs were fully determined in phase I. A self-referral option and careful initial assessment facilitated quicker access to optimal services, including triage for immediacy of need. If alternative services were needed, FEMAP provided facilitation. This practice has been referred to as “open door” access, with an expectation that “every door is the right door” (10). Because of this policy, youths did not have to ask for help multiple times from multiple agencies, which delays treatment and engenders frustration and extra costs.

One potential unintended consequence of a self-referral option is overcoverage (11), the inclusion of patients who do not require specialized services, resulting in increased wait times for more severely ill patients and increased cost by overutilizing high-cost professionals. Our data showed that at least 60% of youths in both subgroups had already received some form of treatment and over 50% had already received medication. Yet symptom severity and functional impairment remained high, implying that previous treatment had not been successful. This was true regardless of whether youths received a physician’s referral. Of note, acceptance rates were identical regardless of whether youths were referred by a physician, suggesting that nonphysicians (including youths themselves) were no worse than physicians at identifying FEMAP as the correct treatment setting. Thus we found no evidence to support that allowing self-referrals leads to overcoverage.

Detailed, in-person clinical assessment ensured that youths who were better served elsewhere (27%) were identified and referred or, rarely, were reassured that specialty services were not indicated. FEMAP-accepted youths were more likely than youths who received referrals or reassurance to have had symptoms consistent with mood or anxiety disorders and higher depression scores, poorer emotion-regulation styles, and more functional impairment. Again, this pattern suggests that coverage was appropriate, given that the model captured youths in need of specialty services for mood and anxiety. FEMAP-accepted youths were somewhat more likely to complete follow-up and were significantly more likely to follow through with recommendations if they completed follow-up, potentially because of greater commitment. Differences in clinical characteristics preclude definitive explanation for these differences.

Many individuals who came directly to the program may have eventually found their way to FEMAP through a doctor’s referral. However, requiring a referral utilizes more physician time and ensures a longer wait until needed mental health treatment is provided. By removing added delay and physician contact, allowing self-referrals may facilitate overall efficiency at no additional, or even at reduced, health care cost. A probable prerequisite for savings, however, is careful clinical evaluation at intake to determine the proper services for patients’ needs before a subspecialty physician is engaged.

Loss to follow-up was substantial, a finding that was closely representative of other research with adolescents (9). There were several predictors of follow-up loss. Most significant was lifetime use of marijuana, defined as use on more than 100 occasions, which was endorsed by a substantial subgroup of participants. Thus, although substance use, in general, appeared to increase the likelihood that youths entered FEMAP without a physician’s referral, lifetime marijuana use increased the likelihood that they would not complete follow-up. Nevertheless, seeking services without a physician’s referral was more likely to protect against loss to follow-up overall. This finding may reflect additional “buy in” among youths who had chosen to contact FEMAP without a physician’s referral or may be related to other factors.

Among youths who completed the SDS at both intake and follow-up, functioning improved. Because there was no control group, it is impossible to attribute this improvement to FEMAP.

One limitation of this research is related to follow-up, given that only half of the youths provided follow-up information, perhaps because the study did not provide incentives for the participants to do so. Another potential limitation was some undercoverage bias associated with the reality that eligible youths may not contact FEMAP or may contact FEMAP but decline to participate. Youths often heard about the service through school, doctors, or word of mouth. Individuals who had little contact with such sources would be less likely to find the service, although Internet usage could have directed them to it.

Demands for clinically effective and cost-appropriate models of early intervention for youths with moderate to severe mental illness are growing. This process evaluation of the initial phase of one such model illustrates that the mental health care needs of youths with moderate to severe mental illness can be identified effectively. The model provides an approach to assessing mood and anxiety disorders associated with clinically significant symptoms and functional impairment before the symptoms become severe and chronic. The results showed that the model achieved its desired goals without excessive overutilization of specialty services. Future evaluation of the therapeutic effectiveness and cost-effectiveness of this model, as described in phase 2 of the causal model, is warranted.