A Comparison of Assertive Community Treatment Fidelity Measures and Patient-Centered Medical Home Standards

In 2011, the NCQA revised PCMH standards and scoring guidelines that are used to determine whether programs qualify as medical homes. They are divided into six standards subdivided into 28 major elements comprising 152 individual factors (26). Each element is assigned from 2 to 6 points (weighted), for a maximum score of 100. Practices scoring between 35 and 59 are recognized as a level 1 medical home; between 60 and 84, level 2, and 85 or higher, level 3. Additionally, each standard has one “must-pass” element. A practice must satisfy 50% of the factors within that element to pass.

There are two ACT team fidelity measures and one set of program standards. The Dartmouth Assertive Community Treatment Scale (DACTS) is the original national standard measure of fidelity of ACT (18). ACT teams are scored on 28 items by using a 5-point behaviorally anchored scale (12,29). Similarly, the Tool for Measurement of Assertive Community Treatment (TMACT) is a newer fidelity measure comprising 47 items, each also scored on a similar 5-point scale (13). The 2003 National Program Standards for ACT Teams (14) were developed on the basis of the ACT start-up manual (30) and provide the foundation for the evidence-based practices tool kit for ACT developed by the Substance Abuse and Mental Health Services Administration (15,31). The standards include examples of standard program operation. These three indices are the measures employed most commonly to assess ACT teams’ adherence to the ACT model (18).

Two authors (JMC and ERV) independently applied the NCQA medical home standards to the three ACT measures to determine whether the ACT measures met requirements for each NCQA factor. Explicit satisfaction of a NCQA factor was defined a priori as a direct mention in the fidelity measure or program standards. Implicit satisfaction was defined as meeting the spirit of the factor. An example of a factor that meets implicit but not explicit criteria is the NCQA requirement that practices document the provision of general medical advice in the medical record. The TMACT does not require teams to record such data in charts as part of their daily routine, but teams undergoing a TMACT fidelity review must document the occurrence of necessary clinical contacts, thereby meeting the NCQA criterion implicitly. For purposes of analysis, the authors agreed that the process of fidelity review could be considered to implicitly satisfy NCQA standards regarding quality improvement (32). The authors were blinded to the point value and must-pass status of each NCQA element throughout the review process. Items unrelated to ACT teams, including items related to pediatric practice only (N=3), depression screening (N=1), and phenotypic syndrome surveillance (N=1) were discarded, leaving a total of 147 NCQA factors for comparison.

The two authors reconciled differences in determination of factor satisfaction through discussion and mutual agreement. After further comparison, they scored the TMACT again by using the complete protocol rather than the summary scale alone, given that TMACT fidelity audits are usually performed utilizing the entire protocol: the summary scale is only part of the fidelity review process. The authors differed on a maximum of only 26 NCQA factors for any ACT measure. Of factors with disagreement, 64% were up-coded after discussion to indicate the NCQA factor criteria had been satisfied. The two authors had no relationship to the development of ACT or NCQA measures. One author (MMDV) contributed additional expertise regarding proper interpretation of the TMACT once scoring was completed.

The number of NCQA factors with explicit satisfaction by the ACT measures was 23 (16%) for the DACTS, 51 (35%) for the TMACT, and 61 (41%) for the program standards. NCQA factors with implicit satisfaction numbered 47 (32%) for the DACTS, 75 (51%) for the TMACT, and 76 (52%) for the program standards (Table 1). There was a qualitative trend for the NCQA factors to overlap more consistently with the program standards and the TMACT than with the DACTS.

At least one factor in each NCQA standard was satisfied implicitly by all ACT measures. NCQA standards 1 (enhance access and continuity), 3 (plan and manage care), and 4 (provide self-care support and community resources) consistently demonstrated stronger overlap across ACT measures compared with standards 2 (identify and manage patient populations), 5 (track and coordinate care), and 6 (measure and improve performance) (Table 1). NCQA standard 6, pertaining to quality improvement, overlapped more with the TMACT and the program standards than with the DACTS. Because ACT teams hold daily meetings to discuss all patients, all ACT measures had significant overlap for elements regarding the identification of high-risk clients. ACT measures shared much in common with NCQA requirements pertaining to client care management, the provision of referrals to community resources, and coordination with families and the management of care transitions.

ACT measures met none or only one of the factors in elements requiring the use of an electronic medical record (EMR) or other electronic components—such as electronic prescribing, patient identification or management, or other communication—automatically disqualifying 50 NCQA factors. A majority of NCQA standard-2 factors (21 of 34) and one in three standard-5 factors (eight of 25) require the use of an EMR. These standards were among the three with the smallest overlap with NCQA factors, in part because of the significant reliance on EMR. Although there was considerable overlap between ACT measures and the comprehensive health assessment element in standard 2, some factors were not satisfied. No ACT measure mentions advanced care planning—for example, a living will and medical advance directives—although the TMACT supports the implementation of wellness management strategies, such as Wellness Recovery Action Plans and psychiatric advance directives. Because wellness management strategies are inherently nonmedical, they were not counted explicitly or implicitly. No ACT measure explicitly details the processes necessary for testing (laboratory or imaging) or referral coordination and follow-up.

Total scores for explicit satisfaction of NCQA factors ranged from 16.75 (DACTS) to 49 (program standards), and scores for implicit satisfaction ranged from 36.25 (DACTS) to 61 (program standards) (Figure 1). Only the DACTS, when explicit scoring criteria were applied, lacked the requisite 35 points to achieve level 1 medical home status. By using a comprehensive EMR for data management, tracking of medical tests, and follow-up and monitoring of chronic disease outcomes, ACT teams could obtain an extra 23 possible points. Adding these points to baseline scores would move all ACT measures into the scoring range for a level 1 medical home and move the TMACT into the level 3 scoring range.

High-fidelity ACT teams that adhere to the program standards and the implicit TMACT criteria met the essential must-pass elements of standards 1, 3, 4, and 6 (Table 1). Must-pass elements of standards 2 and 5 had little overlap with ACT measures. All three measures failed to explicitly satisfy the must-pass element in standard 5, although teams adhering to the intent of full TMACT fidelity would likely pass with a score of 100%. This element pertains to the tracking and coordination of referrals.

The analysis demonstrated that high-fidelity ACT teams are equipped with the infrastructure necessary to support the requirements of a medical home. We found a marked overlap between NCQA medical home criteria and the ACT program standards, the TMACT fidelity criteria, and, to a lesser degree, the DACTS. Were it not for scores for two “must-pass” elements related to referral coordination and identification and management of three prevalent chronic disease states, high-fidelity ACT teams would qualify as level 1 or 2 medical homes.

Standards 1, 3, and 4 had the most overlap with ACT fidelity measures and program standards. These standards include factors such as “the practice provides same-day appointments,” “the practice conducts previsit preparations,” and “the practice counsels at least 50% of patients to adopt healthy behaviors.” The domains of enhanced access and continuity, care management, and supported community living are original tenets of ACT (33).

The least amount of overlap was found for NCQA standards 2, 5, and 6, which specify how a practice coordinates care and provides quality improvement for chronic general medical illnesses. ACT teams are uniformly required to coordinate care with emergency departments and admitting facilities and, concordantly, scored highly on standard 5. However, ACT measures lack specific criteria for coordinating consultation for the results of laboratory tests or imaging. As an example, the must-pass element of standard 5 requires that practices assume responsibility for care coordination by establishing communication with other providers, tracking the status of referrals, and documenting referral results in the medical record. The TMACT is the only measure that details this process. Additionally, standard 6 relies heavily on quality improvement processes and self-evaluation. Only the TMACT (in full-fidelity review) and the program standards detailed the processes necessary to demonstrate satisfaction of NCQA quality improvement criterion.

The only overlap of NCQA criteria for standard 2 (identify and manage patient populations) and the ACT measures was for the comprehensive health assessment. The TMACT and the program standards require ACT teams to perform a comprehensive health assessment and charge ACT nurses with the identification and management of comorbid general medical illnesses or appropriate referral of ACT clients with these illnesses. The companion manual for ACT program start-up that supports the program standards includes a worksheet for health assessment that must be performed within 72 hours of enrollment (30). The DACTS does not mention general medical health assessment and, therefore, could satisfy only a fraction of the factors in this element.

ACT measures lacked overlap in the other three elements of NCQA standard 2, which describe systematic management of clinical data. The must-pass element requires practices to proactively identify common chronic diseases and systematically engage in the management of at least three. One of those chronic conditions must be a psychiatric or substance use disorder. Because no ACT measure requires teams to systematically review their population, identify highly prevalent disease states, and manage persons at a population level, none were able to satisfy this must-pass element.

NCQA criteria are intended for primary care practices, with the assumption that the care delivered is medical. As a result, the bulk of NCQA factors refer primarily to the processes of care. These factors are similar to items in the fidelity instruments in their processes of care. ACT teams following the program standards or undergoing TMACT fidelity review could have the necessary infrastructure to serve as a medical home if they were properly equipped to provide direct general medical care.

Although ACT teams may not be equipped to provide clinical advice on urgent general medical conditions around the clock, they are frequently the first point of medical contact for many clients and often are instrumental in triaging general medical concerns and providing transportation to hospitals or clinics. Many ACT providers already support a broad provision of medical services by their teams. Some ACT teams have had success providing general medical care by using peer support and nursing (24). Others have argued that ACT may be underutilized as a means for providing general medical care (21,34). No studies have systematically examined the role of ACT teams in the general medical care of their clients. It remains to be seen whether ACT teams are willing to adapt their services more formally and whether the care they deliver could be translated into improved health outcomes for clients. The exact prevalence of chronic diseases among ACT clients is unknown. However, the prevalence of chronic general medical conditions among patients with serious and persistent mental illness is high, and because all ACT clients have serious mental illnesses, it can be inferred that many ACT clients have comorbid vascular disease risk factors such as hyperlipidemia and hypertension that easily satisfy the NCQA requirements for chronic illnesses (19,35,36).

In a nationwide analysis using the 2008 NCQA PCMH criteria, Hollingsworth and others (37) concluded that 46% of primary care practices would not qualify for PCMH recognition. In results similar to those of our analysis, most deficiencies among solo primary care practices were related to electronic prescribing, tracking of test results, and quality improvement. Primary care systems and ACT teams face similar challenges to PCMH accreditation.

There were several limitations to this analysis. We attempted to limit coding bias among the authors who applied the standards by blinding them to the NCQA scoring algorithm and must-pass elements. This analysis did not take into account the practices of ACT teams in vivo. As mentioned previously, many ACT-like programs may lack full implementation of the ACT model. However, a 2010 survey of state mental health program directors demonstrated that 38 of 42 states with ACT programs require monitoring of services with a fidelity instrument (18). Generally, high fidelity is tied to better outcomes across many evidence-based practices, including ACT (11). This analysis is most applicable to programs that adhere closely to the program standards and fidelity measures. However, it may underestimate the degree to which ACT teams that already engage in general medical care could reach medical home accreditation (21,24,38).