Level 3: Transfer from knowledge to practice: The current frontier
Kirkpatrick’s (
17) level 3 (Figure 1) is to assess whether knowledge is passing into action or being applied in clinical practice, and this is a very hot topic in CME today (
15). Psychopharmacology CME is not about knowledge for knowledge’s sake but about learning that leads to improvement in clinical practice. It is quite difficult to show when or whether learning something about treatment, diagnosis, and laboratory testing leads to transfer of this information into use in clinical practice (
1–
8). Methods for measuring whether such transfer has occurred have many problems and limitations. Thus, surveys (small number returned), focus groups (nonrandom and probably not representative samples), prescription audits (extremely expensive and structured for commercial applications, not best practices analyses), and chart audits (expensive and confidentiality issues) are all examples of flawed methods in the attempt to show that learning in CME can transfer into behavioral changes in practice (
1–8, 22, 2
3). It is not surprising that there is currently significant questioning as to whether transfer can be measured. This leads in turn to the controversy over whether CME activities are even capable of facilitating such a transfer because it has arguably never been adequately demonstrated.
A more realistic goal than to attempt to measure actual transfer of knowledge into clinical practice may be to show that the information presented and learned in a CME activity is actually applied later in that same CME activity through an exercise emulating clinical practice or as reflected in answers to questions about attitudes, levels of confidence, or intention to change clinical practice (
23,
24). At best, such outcomes are merely proxies for what really happens in a clinical practice setting, yet these proxies can be measured as part of any CME activity, and, thus, a determination can be made as to whether the knowledge is being transferred at least to model cases by using exercises in which participants “practice by doing” (
21,
25) Rarely, however, are such proxies developed, measured, or evaluated. More research is clearly needed to determine whether clinical case exercises and monitoring of attitudes and intentions are genuinely predictive of changes in clinical practice, but, for now, transfer of knowledge to case exercises and documentation of attitude changes are measurable, attainable, and intuitively attractive outcomes that can begin to address Kirkpatrick’s (
17) level 3 evaluations of transfer of knowledge to use in clinical practice.
For example, an audience of 297 psychiatrists and other mental health professionals attending the Neuroscience Education Institute Psychopharmacology Academy in San Francisco in 2006 was asked before the course about their preferred treatment for patients with attention deficit hyperactivity disorder (ADHD) who have predominantly inattentive symptoms. Their responses are shown in Figure 3A. Although there is no completely correct or incorrect answer here, the results agree with previous needs analyses (Neuroscience Education Institute, unpublished data) showing that most practitioners feel that stimulants are greatly preferred for such patients. However, many psychiatrists treating adults are not willing to treat anyone with stimulants and thus decline to treat ADHD in adults at all. To encourage more treatment of ADHD in adults, a CME activity was developed to outline alternatives to treating inattentive symptoms with stimulants only, and participants were asked after this activity to again indicate what their preference was for a patient with ADHD with predominantly inattentive symptoms (Figure 3B). The posttest in Figure 3B utilizes a very short and simple case vignette (
24,
26) to capture treatment preferences for this patient type and documents a shift away from stimulants to other alternatives.
Another technique to begin the process of measuring Kirkpatrick (
17) level 3 transfer of knowledge into action (Figure 1) is to ask about intent to change. Such shifts in attitudes may precede actual behavioral changes (
27,
28), and questioning participants about their attitudes could potentially show whether there is at least an intent to transfer knowledge into practice. Although research is still clarifying to what extent intent to change actually presages actual behavior changes (
27,
28), it seems highly unlikely that any behavior changes will occur if a CME participant does not even intend to change behavior.
An example of documentation of intent to change is shown in Figure 4. Before hearing lectures on monitoring metabolic changes in psychiatry, the 297 participants at the Neuroscience Education Institute Psychopharmacology Academy in San Francisco in 2006 were asked how many of their patients currently have their blood pressures, weights, and body mass indices measured and recorded regularly. A very low number was indicated. After the lectures on monitoring metabolic status in psychiatry, the audience was asked how many patients they think should have these items measured and recorded regularly, and the results showed a huge shift upward. Whether these participants will actually attain these goals is unknown, but this attitude shift would seem to be a necessary precursor to any actual behavior change.