Enhanced Adoption of Measurement-Based Care in a Psychiatry Outpatient Clinic After High-Reliability Process Changes

This study was reviewed by the University of Michigan’s Institutional Review Board and was determined to be exempt from review. We selected 10 consecutive calendar months before and 10 months after the COVID-19 pandemic began in the United States in March 2020, and we designated our sampling times as prepandemic (May 1, 2019–February 28, 2020) and pandemic (May 1, 2020–February 26, 2021). Questionnaire completion data were abstracted from the medical records of all adult patients (18 years of age or older) who had been assigned questionnaires to complete and returned for a follow-up appointment. Other abstracted data included date of visit, age, race, and sex. Patients were assigned questionnaires to complete at return visits; questionnaires were administered no more than every 2 weeks.

During the prepandemic sampling period, patients had the option of completing questionnaires by accessing their patient portal via the Epic electronic medical record platform. Patients who were enrolled in the patient portal were prompted to complete the questionnaires during the electronic check-in process and were sent e-mail reminders to complete the questionnaires before their appointment. Patients who did not complete the questionnaires before arriving at the clinic were, upon check-in at the front desk, provided a tablet on which to complete the questionnaires.

On March 25, 2020, in compliance with the State of Michigan’s Executive Order 2020-21 (COVID-19) mandating that only emergency health care be provided in person, nearly all outpatient appointments in the Department of Psychiatry were converted to virtual visits. A University of Michigan Health System–wide initiative ensured that as many patients as possible were enrolled in the patient portal to facilitate virtual care. Reprogramming of the MBC questionnaire administration process was required to accommodate this shift to virtual care. After a brief reprogramming hiatus from late March through April 2020, patients were presented with their questionnaires when they checked in for their virtual visit. With the new programming, when patients started the electronic check-in at least 15 minutes before their visit start time, they had to complete the entire electronic check-in process, including all assigned questionnaires, in order to start the virtual visit. This resulted in a near-seamless integration of the questionnaires into the check-in process. By enrolling all patients into the portal, leveraging the electronic check-in process, and ensuring patients could not go on to the next step (i.e., start the virtual visit) until questionnaires were completed, these process changes incorporated high-reliability principles (8). However, to avoid delaying visits and disrupting schedules, patients who began electronic check-in fewer than 15 minutes before their visit start time were not presented with questionnaires and could thereby bypass completion of the measures. A linear mixed-effects logistic regression model tested whether questionnaire completion rates differed by time (i.e., prepandemic vs. pandemic). Two variables were dichotomized for inclusion in the model: race (White vs. non-White) and age (younger adults [<65 years of age] vs. older adults [≥65 years of age]). The model included fixed effects for time, race, age, and sex and interaction effects for time × race, time × age, and time × sex. The model contained one random intercept per patient. Post hoc comparisons tested group differences for significant interactions.

There were 44,305 distinct appointments that were completed by patients and met criteria for inclusion in the current study; 23,031 (52.0%) were completed during the prepandemic period, and 21,274 (48.0%) were completed during the pandemic period. Patients ranged in age from 18 to 89 (10.0%, N=4,451, were 65 years or older), and a majority of visits were completed by White patients (84.7%) and by non-Hispanic patients (93.4%). Patient demographic characteristics are shown in Table 1. None of the visits were completed virtually during the prepandemic sampling period, whereas nearly all (97.1%, N=20,653) of the visits were completed virtually during the pandemic sampling period.

During the prepandemic period, MBC questionnaires were completed in 6,991 (30.4%) visits. In contrast, during the pandemic period, questionnaires were completed in 11,562 visits (54.3%). This increase of 78.6% was significant (β=3.39, p≤0.001). There was a main effect for sex (β=1.27, p≤0.001) but no interaction with time, indicating that at both time points women were more likely to complete questionnaires relative to men. There was a significant time × race interaction (β=1.18, p=0.025), with post hoc comparisons indicating that questionnaire completion rates did not differ by race in the prepandemic study period; however, White patients were more likely than non-White patients to complete questionnaires after the process changes were implemented (t=3.92, df=44,293, p≤0.001). There was a significant time × age interaction (β=0.81, p=0.021), with post hoc comparisons indicating that rates of questionnaire completion did not differ by age group in the prepandemic study period; however, older adults were more likely than younger adults to complete questionnaires after the process changes were implemented (t=4.05, df=44,293, p≤0.001).

In the current study, we examined the impact of high-reliability changes to the MBC administration process on questionnaire completion rates. We also examined demographic factors as they related to completion of questionnaires. Our findings show that the MBC process changes were associated with a 78.6% increase in completion of MBC questionnaires at return visits, which compares favorably with other reports (9). This dramatic change can be attributed to the full integration of questionnaires into the electronic check-in process for virtual visits, enrollment of all patients into the portal, and the robust increase in virtual visit volume. However, we note that even with this significant increase in completion rates, MBC questionnaires were completed at less than 60% (54.3%, N=11,562) of follow-up visits. In our clinic’s MBC process, patients are not presented questionnaires if they check in fewer than 15 minutes before their visit start time, and this factor was likely a major contributor to the suboptimal completion rates observed in our study. Further work to refine administration processes, grounded in high-reliability principles, is needed to enhance completion rates.

For both the prepandemic and pandemic periods, women were more likely to complete questionnaires relative to men, which suggests that a factor other than the MBC administration process accounts for sex differences in completion rates. Although the current study cannot determine the reasons for the absence of a time × sex interaction, personality differences—women are generally higher in conscientiousness than are men—may translate to a greater likelihood of complying with the request to complete questionnaires (10). Hypothetically, providers may also be more likely to engage with female patients about questionnaire completion and results. Qualitative approaches (e.g., focus groups with providers and with male patients) may help to illuminate reasons for this sex disparity.

Older adults were more likely to complete questionnaires relative to younger adults after the process changes. Compared with younger adults, it is possible that older adults are more likely to be retired from work and to have fewer fixed responsibilities, which may be advantageous to starting the check-in process earlier.

Non-White patients were less likely to complete measures relative to White patients only after the process changes occurred, which indicates that something inherent in these changes may have been a significant contributing factor to this disparity. Although more work is needed to determine the reasons for this difference, racial disparities may contribute to poorer-quality Internet access (11) and the inability to check in more than 15 minutes before the appointment start time because of unstable and less predictable work schedules (12). If check-in time is a contributing factor, then a potential solution may be to allow patients to complete questionnaires regardless of when they check in for a visit. Future work is critical to identify determinants of this racial disparity, because we know that completion of measures enhances treatment outcomes.

Even when we integrated MBC questionnaires with the electronic medical record before the COVID-19 pandemic, such that patients could complete questionnaires during the electronic check-in process, completion rates were not very high (30.4%, N=6,991 visits) when patient care was occurring in person. A major barrier to completion in this configuration was that a proportion of patients were not enrolled in the patient portal; therefore, they were not able to check in electronically and did not receive reminders to complete the questionnaires. We observed a significant increase in completion rates only after our clinic enrolled all patients in the portal, which was necessary to facilitate virtual visits within the electronic medical record, and then administered the measures directly to patients as part of the electronic check-in process and required patients to complete the questionnaires before starting their virtual visit (unless check-in occurred fewer than 15 minutes before the visit start time). Although these changes were prompted by the transition to telepsychiatry in our clinic, they may be implemented regardless of treatment modality. We believe our experience may help to inform other psychiatric services about ways to enhance patient completion of MBC questionnaires.

Strengths of the current study include a large sample size (N=44,305 appointments) and examination of novel aspects of MBC. However, confounding variables and collider bias are limitations of the pre-post study design. The COVID-19 pandemic may represent a confounding variable, because patients were possibly more willing to complete questionnaires as a result of increased distress related to the pandemic. Finally, self-report questionnaires such as those used in MBC may not accurately reflect clinical change relative to patient perception (13).

Our findings show that high-reliability process changes in the administration of MBC are associated with a significant increase in patients’ questionnaire completion rates. However, it is important to note that even with this increase, our clinic’s questionnaire completion rates were still well below 60%, and more work is needed to identify process enhancements that would further bolster questionnaire completion by patients. Psychiatry clinics that use MBC may wish to leverage technology to support implementation of high-reliability processes. Further, clinics that use MBC should be aware that factors such as patient race, age, and sex may influence completion of questionnaires and consider ways to ensure more equitable use by diverse populations.