Outcome Measures Are Foundation of Revised Accreditation System

The so-called “Next Accreditation System” envisioned by the Accreditation Council for Graduate Medical Education (ACGME) is one that will move accreditation of residency training to an “outcome-based” system providing continuous program assessment.

In an address to educators at the annual meeting of the American Association of Directors of Psychiatric Residency Training (AADPRT), Chris Thomas, M.D., the chair of the ACGME’s Psychiatry Residency Review Committee, outlined major features of the Next Accreditation System, as the ACGME calls it. The system, articulated by Thomas Nasca, M.D., in a 2012 article in the New England Journal of Medicine, is a comprehensive restructuring of accreditation based on the assessment of residents’ achievement of specific educational outcomes—codified in “milestones” developed by each specialty—that are linked to the six core physician competencies the ACGME formulated in 1999.

The new system also involves a transformation of program assessment, replacing site visits with “continuous observation” (see box).

At the AADPRT meeting, Thomas said the system will be a major departure from a longstanding system of periodic site visits and what he called a “process-based” system of resident assessment that was not linked to acquisition of specific physician skills.

“Under our previous system, trainees were required to do so many months of internal medicine, so many months of neurology, with the expectation that just by attending and showing up, the trainee would somehow become a competent physician,” he said.

Key to this transformation is the introduction of the Milestone Project, in which psychiatry and all other specialties are developing specific goals, or milestones, for assessing resident progression toward competency. A set of 22 milestones for psychiatry in the six domains of physician competency—medical knowledge, patient care, practice-based learning and improvement, systems-based practice, professionalism, and interpersonal and communication skills—were finalized last year and will be implemented for residency programs in July (Psychiatric News, August 28, 2013).

“The milestones move accreditation from structure- and process-based to outcomes-based assessment,” Thomas said. “They are intended to follow the trainee’s progression from entrance into residency to practice as an independent clinician. They should help to articulate shared expectations across residency programs, establish aspirational goals of excellence, and provide a framework and a language for common discussion.”

Thomas said residency review committees will receive only aggregate program data, not individual reports on individual residents. Individual residents will be assessed by a Clinical Competency Committee within each program; the task of the committee is to review all resident evaluations semiannually, prepare milestone ratings of each resident, and advise the program director on resident promotion, remediation, probation, or dismissal.

The project is a major change in the way programs operate, and at the AADPRT meeting there was at least some degree of concern—ranging from uncertainty about how it would work to skepticism that it would work at all—but Thomas urged program directors “not to panic.”

In a separate presentation at the AADPRT meeting by program directors from other specialties, Conrad Clemens, M.D., M.P.H., associate dean for graduate medical education at the University of Arizona, backed Thomas’s call for calm in the face of the changes. Clemens said using the milestones at his own institution was easier—or not as difficult—as many had expected. “Some faculty took me aside and said, ‘You know, it’s really not as bad as we thought it would be,’ ” he said.

And Thomas urged directors not to discard the assessment tools they are familiar with and have been using. In fact, the milestones are not, he said, an assessment tool per se; rather, they are a map charting developmental goals in the progression toward independent practice—the project preserves the independence of programs to use their own methods for assessing the achievement of those goals.

He said programs can use the milestones to provide feedback to residents, benchmark residents to the mean overall performance of the program, identify program strengths and opportunities for improvement, and help benchmark each program against the performance of other programs around the country. Residents, in turn, can use the milestones to garner feedback about specific tasks and skills, determine their own strengths and training goals, and better learn the objectives of each rotation.

The first reports on milestones for general psychiatry programs are expected to be filed between November 1 and December 31. Milestones for psychiatry subspecialties are expected to be completed this December for use in July 2015. ■