Implementation of Evidence-Based Psychotherapies for Posttraumatic Stress Disorder in VA Specialty Clinics
Abstract
Objective
The U.S. Department of Veterans Affairs (VA) has engaged in substantial efforts to promote the use of evidence-based psychotherapies for posttraumatic stress disorder (PTSD). The authors evaluated the effectiveness of these efforts.
Methods
This study used a cross-sectional, mixed-methods evaluation of treatment provided by the VA at specialty PTSD clinics in New England during the first six months of fiscal year 2010. Natural language processing algorithms were applied to clinical notes to determine utilization of evidence-based psychotherapy (prolonged exposure therapy and cognitive-processing therapy) among patients who were newly diagnosed as having PTSD. Data regarding efforts to implement evidence-based psychotherapy and other clinic characteristics were obtained through qualitative interviews with clinical and administrative staff (N=30), and the Promoting Action on Research Implementation in Health Services framework was used to identify clinic factors associated with use of evidence-based psychotherapy.
Results
Six percent of patients (N=1,924) received any sessions of an evidence-based psychotherapy for PTSD (median=five sessions). Several clinic factors were associated with an increased rate of implementation, including prior experience with use of the treatments, customization of training, and prolonged contact with the implementation and training team. Facilitation with broad training goals and clinics with highly organized systems of care were negatively associated with implementation.
Conclusions
Few patients with PTSD received evidence-based psychotherapy for PTSD during their first six months of treatment at a VA specialty PTSD clinic. The implementation framework poorly predicted factors associated with uptake of evidence-based psychotherapy. These results suggest that additional research is needed to understand implementation of evidence-based therapy in mental health settings.
Posttraumatic stress disorder (PTSD) is a common and often debilitating condition (1,2). The disorder occurs in the aftermath of an event or events involving actual or threatened death or serious injury or threat to physical integrity (3). The treatment of PTSD is a major objective at U.S. Department of Veterans Affairs (VA) medical facilities. A large percentage of veterans returning from the wars in Iraq and Afghanistan suffer from PTSD, and many veterans from prior conflicts still seek treatment for PTSD (4,5).
Over the past 20 years, multiple specific treatments for PTSD have been developed. These treatments, including various types of both medication and psychotherapy, have shown effectiveness in rigorous clinical trials (6,7). Convergent recommendations from clinical practice guidelines and other published reviews suggest that medications, including selective serotonin reuptake inhibitors and noradrenergic antidepressants, and psychotherapies, including several forms of cognitive-behavioral therapy, are effective treatments for PTSD (8–10). Evidence suggests that cognitive-behavioral therapies, including prolonged exposure (PE) (11) and cognitive-processing therapy (CPT) (12), are efficacious for treatment of PTSD among veterans and can be implemented in routine practice (13,14). The VA has advocated the use of a range of evidence-based treatments for PTSD, including specific types of cognitive-behavioral therapy, such as PE and CPT. The VA has ensured that evidence-based medications and psychotherapy are widely available to veterans with PTSD.
In addition to generally advocating the use of PE and CPT, the VA has taken active steps to promote the spread and use of these treatments, including requiring each VA facility to provide the treatments, hiring additional psychotherapy staff, and training psychotherapists to perform the treatments (15–17). Between 2005 and 2010, the VA hired more than 7,000 new mental health professionals, bringing the total mental health staff to more than 20,000 (17). The VA has developed clear clinical guidelines and administrative policies that advocate and even require that veterans have access to specific evidence-based treatments for PTSD (10,16). An administrative mentoring program assists clinical leaders in implementing these treatments (18). In addition, a large-scale and well-resourced national clinical training program trains VA therapists in the use of PE and CPT. As of 2010, 2,300 therapists had been trained in CPT and 1,100 in PE (15,19). The training includes attendance at a several-day course conducted by experts in the treatments and an ongoing program of mentorship and supervision of treatment. Clinicians who complete the course and program may obtain certification in the use of the treatments.
Although the VA’s efforts to develop policies that promote the use of evidence-based psychotherapy for PTSD have been described, little is known about the effectiveness of those interventions. Moreover, the subject of promoting the use of evidence-based treatments in mental health settings is not well understood (20–23). As an example, it is unclear if training staff to perform evidence-based therapy for PTSD actually leads to increased use of these treatments. Furthermore, there is no clear understanding of the steps necessary to promote use of evidence-based practices in mental health settings. These questions apply both to training and to other attempts to promote use of evidence-based treatments for PTSD, such as increasing staffing and adopting policies that improve access to the treatments. Overall, two problems have limited exploration of these questions. First, previous examinations were not able to distinguish evidence-based psychotherapies from psychotherapy in general (24,25). Second, examination of implementation of mental health efforts has been limited because implementation frameworks have been developed largely for use in general medical settings (26,27). As such, their applicability in mental health settings has not been rigorously tested.
The Promoting Action on Research Implementation in Health Services (PARiHS) framework may have particular relevance in understanding VA efforts to increase the use of PE and CPT. The PARiHS framework was designed to aid those attempting to implement a research finding into clinical practice. It examines three independent and important domains for the implementation process—evidence for the practice, clinical context, and the facilitation efforts aimed at promoting use (28,29). Notably, the PARiHS framework has been used both as a road map to guide implementation efforts and as a tool to understand and contextualize the results of improvement efforts that were planned without a formal framework (20–22).
The overall goal of this study was to examine the effectiveness of VA efforts to promote use of evidence-based psychotherapies to treat veterans with PTSD. We determined the use of these treatments in one region and, through staff interviews, explored factors and approaches that facilitated or impeded the use—and spread—of these treatments. We used the PARiHS framework to develop a rubric for evaluating how aspects of implementation affected use of evidence-based psychotherapies.
Methods
The study and protocol were reviewed and approved by the Dartmouth College Committee for the Protection of Human Subjects.
Data source
We used data obtained from the VA New England Data Warehouse. The data warehouse contains clinical and administrative data for all inpatient and outpatient care delivered in the New England VA. We analyzed data from six specialized VA PTSD clinics, three located at large, academically affiliated VA medical centers; two at medium-sized, academically affiliated VA medical centers; and one at a large VA medical center not closely affiliated with a medical school.
Current use of PE and CPT
Our method for determining use of PE and CPT consisted of four steps: identification of patients, automated coding of PE and CPT notes, validation, and determination of the rate of use. The method is described in greater detail elsewhere (30).
Identification of patients.
We identified patients who were presenting for the first time to one of the clinics during the first half of fiscal year (FY) 2010. We obtained administrative data and progress notes for these patients from the data warehouse. We used the administrative data to identify patients who received a new diagnosis of PTSD and who were seen at least once in a specialized PTSD clinic. We selected this patient population, given that they are the ideal candidates for evidence-based psychotherapy for PTSD. This diagnosis of PTSD was based on clinical examination by PTSD clinic staff. Overall, 16,121 patients who were seen in VA New England Health Care facilities during this time period received a primary diagnosis of PTSD during one or more outpatient encounters. During that period, the specialized outpatient PTSD clinics saw 5,379 outpatients.
Automated coding.
Validation.
Manual raters performed blinded coding of a 10% sample of the notes. When the agreement among reviewers on the manual rating team and the ARC was greater than .8, we considered the automated ratings valid. The remaining 90% of notes were coded only by ARC. Agreement between manual ratings and ARC was calculated by using information retrieval measures called recall (the proportion of relevant notes identified by the ARC), precision (the proportion of notes identified by the ARC that were relevant), and “f measure” (a balanced measure of the recall and precision) (28). The recall was .9; the precision, 1; and the f measure, .95.
Rates of PE and CPT use.
Using the coded notes, we calculated the rate of use of CPT and PE at each of the six specialized PTSD clinics. The rate represents the percentage of new patients receiving at least one session of PE or CPT in the six months following clinic enrollment.
Qualitative assessment of implementation
Interview development.
We developed an interview guide to conduct semistructured interviews by telephone with staff members at each clinic. We used the PARiHS framework to develop overarching questions about the implementation of CPT and PE (29) (Table 1). These overarching questions guided the development of the more specific questions used in the interview guide. Phone interviews were conducted with the mental health service line chief, the PTSD clinic leader, and PTSD therapists.
| Domain and element | Question |
|---|---|
| Evidence | |
| Awareness of research evidence | What is strength of evidence for the effectiveness of evidence-based psychotherapy for PTSD, and how aware are providers? |
| Valuation of research evidence | How strong do the providers believe the evidence of effectiveness is for evidence-based psychotherapy for PTSD? |
| Observed effectiveness from clinical experience | How effective has the evidence-based psychotherapy for PTSD been in clinicians’ own clinical experience? |
| Amount of clinical experience | How much clinical exposure have clinicians had to evidence-based psychotherapy for PTSD? |
| Solicitation of patient treatment preferences | Are patient treatment preferences solicited in an organized way? |
| Valuation of patient treatment preferences | Are patient’s preferences regarding treatment seen as an important determinant of treatment? |
| Context | |
| Teamwork culture | Do clinicians see themselves as an organized team providing care or more as individuals? |
| Learning organization | Does the PTSD clinic value change and improvement? |
| Leadership style | Are mental health leaders seen as democratic and transformative or dogmatic? |
| Leadership: role clarity and resource use | Are clinicians’ roles clearly defined, and are resources available for them to complete [the] task? |
| Leadership: system of care | Is there a clearly organized system of care in the PTSD clinics, or is care more case by case? |
| Evaluation use | Does the PTSD team use a systematic process to track patient outcomes? |
| Facilitation | |
| Purpose of facilitation | Does the facilitation promote holistic growth in the trainees or is [it] more focused on a specific technique? |
| Sustained contact with facilitation | What is the length and frequency of connection between [the] facilitation team and trainees? |
| Customization in facilitation | Is the facilitation adapted to the baseline knowledge of the trainees? |
| Knowledge of facilitation team | Are the facilitation leaders seen as highly skilled and knowledgeable in evidence-based psychotherapy for PTSD? |
| Connectivity with facilitation team | Are facilitation leaders able to connect interpersonally with those being trained? |
a
PARiHS, Promoting Action on Research Implementation in Health Services
The phone interviews (N=30) were recorded and transcribed. The transcriptions were reviewed, and the contents were coded by domain and element of the PARiHS framework. Evaluators were blinded to the use of treatments at the site.
Interview scoring.
We developed a scoring rubric that transformed qualitative data for each element in the PARiHS framework into a numeric value (Table 1). First, the interview transcripts from each staff member at each site were reviewed, and each PARiHS element was assigned a score from 1 to 10, indicating a range of possible responses. Using this method, we developed a site score for each element. As an example, if all staff members interviewed described highly organized and systematic care and referral processes, we scored the clinic as having a highly organized system of care.
Data analysis
The goal of the data transformation was to develop a continuous measure for each domain and element (in contrast to the use of dichotomous scoring [present or absent]) (33,34). We then conducted a Poisson linear regression equation that used element scores from each facility as independent variables and the percentage of patients at each site receiving any evidence-based therapy as the dependent variable.
Results
Rate of psychotherapy use
The review of administrative data revealed that 1,924 newly diagnosed patients with PTSD were seen at one of the VA New England specialty PTSD clinics in the first six months of FY 2010. The use of evidence-based psychotherapy for PTSD varied by site (Table 2). Overall, there was greater variation in the use of CPT (1%–13%) than in the use of PE (0%–3%). The overall use of either therapy among the sites varied from 4% to 14%. Patients who received at least one session of either therapy received a median of five sessions. Again, there was considerable variation among sites in the median number of sessions (range 2–9).
| PEa | CPTb | PE or CPT | N ofsessions(median) | |||||
|---|---|---|---|---|---|---|---|---|
| Clinic | N | N | % | N | % | N | % | |
| 1 | 242 | 8 | 3 | 18 | 7 | 26 | 11 | 9 |
| 2 | 245 | 1 | 0 | 32 | 13 | 33 | 14 | 5 |
| 3 | 401 | 9 | 2 | 6 | 1 | 15 | 4 | 2 |
| 4 | 341 | 7 | 2 | 14 | 4 | 19 | 6 | 3 |
| 5 | 356 | 0 | — | 15 | 4 | 15 | 4 | 7 |
| 6 | 339 | 4 | 1 | 9 | 3 | 13 | 4 | 3 |
| All | 1,924 | 29 | 2 | 93 | 5 | 121 | 6 | 5 |
a
Prolonged exposure
b
Cognitive-processing therapy
Psychotherapy and implementation domains
Next we used a Poisson linear regression to correlate the proportion of patients receiving at least one session of evidence-based psychotherapy at each specialty PTSD clinic and the scores on each element of the PARiHS implementation framework. Five of the 18 elements showed a statistically significant association with use of at least one of the evidence-based psychotherapies for PTSD (Table 3). The strongest predictor in terms of both the z score of the association and the r2 of the correlation was prior clinical experience having used an evidence-based therapy for PTSD. This suggests that attempts to use these treatments as part of training and other experiences strongly influence eventual adoption of the treatment.
| Domain and element | z | r2 | p |
|---|---|---|---|
| Evidence | |||
| Awareness of research evidence | –1.62 | .06 | .104 |
| Valuation of research evidence | –.57 | .01 | .571 |
| Observed effectiveness in clinical experience | 1.40 | .06 | .162 |
| Amount of clinical experience | 3.26 | .30 | .001 |
| Solicitation of patient treatment preferences | –1.08 | .03 | .260 |
| Valuation of patient treatment preferences | –1.63 | .06 | .103 |
| Context | |||
| Teamwork culture | –1.76 | .07 | .078 |
| Learning organization | .59 | .01 | .555 |
| Leadership style | 1.55 | .06 | .122 |
| Leadership: role clarity and resources use | –1.28 | .04 | .201 |
| Leadership: system of care | –1.97 | .08 | .049 |
| Evaluation use | –1.82 | .04 | .069 |
| Facilitation | |||
| Purpose of facilitation | –2.46 | .16 | .014 |
| Sustained contact with facilitation | 3.61 | .25 | <.001 |
| Active learning techniques used in facilitation | .71 | .01 | .477 |
| Customization in facilitation | 2.09 | .11 | .036 |
| Knowledge of facilitation team | –1.25 | .03 | .211 |
| Connectivity with facilitation team | .71 | .01 | .477 |
a
PARiHS, Promoting Action on Research Implementation in Health Services
The next strongest predictor was a sustained involvement with the implementation team. Those sites with a more sustained connection with the facilitation team were more likely to use evidence-based therapy for PTSD. Similarly, the therapy was more likely to be used clinically if training was customized to the therapist.
Two elements were negatively correlated with use of evidence-based therapy for PTSD. Contrary to the prediction of the conceptual framework, training that focused on overall clinician development resulted in less use of evidence-based therapy for PTSD than training that focused narrowly on the use of the treatments (purpose of facilitation). Last, organized and highly systematic mental health systems with clearly defined processes were slightly less likely than less organized systems to use evidence-based therapy for PTSD (leadership: system of care). This finding also contradicted the predictions of the PARiHS framework.
Discussion
Summary of findings
During the period of study, delivery of evidence-based psychotherapy for PTSD among newly diagnosed patients of specialty PTSD clinics was quite low. From 4% to 14% of new patients with PTSD received an evidence-based psychotherapy for PTSD in their first six months of enrollment in specialized PTSD clinics.
Most patients who received any evidence-based psychotherapy for PTSD received a modest number of sessions (median=5). Typically, in clinical trials, a full course of CPT was 12 sessions, and a full course of PE ranged from eight to 18 sessions. Although a full course may not be necessary for all patients, it is unlikely that fewer than five sessions of either psychotherapy would constitute an adequate dose. Even when the analysis used a minimal threshold of eight sessions to indicate an adequate dose, the results suggest that 2% of veterans newly presenting with PTSD received an adequate trial of either PE or CPT over their initial six months of care.
This suggests that the VA efforts to promote use of the treatments were not fully successful. Our work examining the factors that promoted use of psychotherapy suggests that, overall, the implementation framework that we applied may need to be modified for implementations in mental health settings or, perhaps more specifically, for implementations of psychotherapy interventions. Even in our small sample, a number of elements of the implementation framework robustly predicted the use of evidence-based therapy for PTSD. Predictive elements included previous experience with and use of the treatments, sustained contact with the facilitation team, and customization of training experiences.
Systems of care that were highly structured and organized and facilitation that focused on overall professional growth rather than on the specific treatment protocols were actually negatively correlated with use of evidence-based therapy for PTSD. These findings contradict the PARiHS framework, which advocates highly organized clinics and facilitation efforts with broad goals. Therefore, we must consider that organized clinics and broad facilitation goals may not be useful to promote mental health implementation efforts. Moreover, the findings of negative associations have some face validity. It is easy to understand how more focused and specific implementation efforts may have more effect than broad ones. Similarly, one can imagine that highly structured clinics may also be more difficult to change.
Limitations
There were a substantial number of limitations to our work. First, we examined only six clinics in a narrow geographic region. It is possible that the use of evidence-based therapy for PTSD and the features associated with implementation are unique to these clinics or this region. More important, our limited sample size likely diminished the study’s power to detect other elements of the framework that were important and valid predictors of use. Thus, in a larger sample, other elements could emerge as significant. Second, we examined treatment use during a narrow, cross-sectional sample of time. A longitudinal approach to examine trends in use would also be desirable. In addition, this study did not evaluate more recent strategies by the VA to implement evidence-based therapy. For example, the VA recently augmented an initiative for hiring mental health providers through the addition of piloted metrics that encourage delivery of evidence-based therapy for treatment of PTSD among veterans returning from Iraq and Afghanistan. Our interview strategy and structured questions were targeted toward a single implementation model. Ideally, we would have considered multiple competing frameworks simultaneously in this work. Last, we did not consider the important effect of patients’ preferences on the use of these treatments.
Implications
The overall results of this study suggest that the uptake and use of evidence-based therapy for PTSD were limited, despite a thoughtful, well-resourced, and highly motivated implementation effort. The scope of the VA efforts to promote use of these treatments was likely larger and more developed than most implementation efforts. Based on our work, we believe that a number of aspects of the facilitation efforts could be further emphasized. Our results suggest that during training, providing therapists with an opportunity to participate in trial treatment is essential. This step also requires that patients are willing and available to participate. Clinical experience with the treatments was the most important predictor of the delivery of evidence-based therapy for PTSD. Designing training to provide a more prolonged contact with the training and facilitation team seems warranted. In this study, sites that reported more than a year of contact were more likely to use evidence-based therapy than sites that had less than a year of contact. We suggest that training activities focus relatively specifically on the desired treatments. Other aspects of professional growth and development seem to detract from the effect. Last, customizing the training to therapists is important. Applying a single approach to all therapists may, for example, bore those with experience with other forms of CBT and overwhelm those with no CBT exposure.
Guidance based on our findings can be of immediate help for those undertaking similar efforts to implement evidence-based therapy in mental health settings. However, the overall lack of effectiveness of these implementation strategies suggests that additional investment in researching implementation in mental health settings is warranted. Efforts to determine optimal methods to promote the use of new types of mental health services deserve similar investments. An important first step would be a detailed description of successful implementation in mental health settings. These successes may provide important clues about the steps needed for effective spread. Ultimately, implementation and dissemination methods need testing in rigorous trials. Unsuccessful practices should be eliminated, and promising approaches should be compared directly. Last, the relative ease and difficulty of spreading different types of mental health interventions deserve more study. Understanding which types of intervention are easier and harder to spread could foster greater investment in approaches that are easier to disseminate.
This study was unable to validate most of the domains of the PARiHS framework, and further research regarding validation of this framework is warranted. Other implementation frameworks, such as the Combined Framework for Implementation Research, show promise in behavioral health implementation research and may have revealed more about factors predictive of implementation (35). More work is needed to determine the applicability of this framework and other implementation models to other mental health settings.
Conclusions
Our work suggests that a major implementation effort aimed at promoting use of evidence-based psychotherapies for PTSD had a modest effect in one region. Nevertheless, some factors predicted the extent to which clinics used the treatments. Overall, it seems that the implementation framework did not predict the clinics’ use of evidence-based therapy for PTSD. Further work is needed to determine the factors that might drive use of these treatments in clinics. Patient preferences were also not explored in this study and could be a focus of future work.
Acknowledgments and disclosures
This work was funded by the VA Quality Enhancement Research Initiative (RRP11-432) and by the VA New England Healthcare Network (V1CDA2010-03). The authors thank Joan Cooke, Ph.D., and Mark McGovern, Ph.D., for their critical feedback on the manuscript.
The authors report no competing interests.
References
1.
Kessler RC, Berglund P, Demler O, et al.: Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Archives of General Psychiatry 62:593–602, 2005
2.
Kessler RC, Sonnega A, Bromet E, et al.: Posttraumatic stress disorder in the National Comorbidity Survey. Archives of General Psychiatry 52:1048–1060, 1995
3.
Diagnostic and Statistical Manual of Mental Disorders, 4th ed, text revision. Washington, DC, American Psychiatric Association, 2004
4.
Hermes ED, Rosenheck RA, Desai R, et al.: Recent trends in the treatment of posttraumatic stress disorder and other mental disorders in the VHA. Psychiatric Services 63:471–476, 2012
5.
Rosenheck RA, Fontana AF: Recent trends in VA treatment of post-traumatic stress disorder and other mental disorders. Health Affairs 26:1720–1727, 2007
6.
Bisson J, Andrew M: Psychological treatment of post-traumatic stress disorder (PTSD). Cochrane Database of Systematic Reviews 2:CD003388, 2005
7.
Stein DJ, Ipser JC, Seedat S: Pharmacotherapy for posttraumatic stress disorder (PTSD). Cochrane Database of Systematic Reviews 1:CD002795, 2006
8.
Post-traumatic Stress Disorder (PTSD): The Management of PTSD in Adults and Children. Clinical Guideline 26. London, National Institute for Clinical Excellence, 2005. Available at www.nice.org.uk/nicemedia/pdf/CG026NICEguideline.pdf
9.
Foa EB, Keane TM, Friedman MJ, et al.: Effective Treatments for PTSD: Practice Guidelines From the International Society for Traumatic Stress Studies. New York, Guilford, 2009
10.
Management of Post-traumatic Stress (clinical practice guideline). Arlington, Va, US Department of Defense, 2010. Available at www.healthquality.va.gov/PTSD-FULL-2010c.pdf
11.
Schnurr PP, Friedman MJ, Engel CC, et al.: Cognitive behavioral therapy for posttraumatic stress disorder in women: a randomized controlled trial. JAMA 297:820–830, 2007
12.
Monson CM, Schnurr PP, Resick PA, et al.: Cognitive processing therapy for veterans with military-related posttraumatic stress disorder. Journal of Consulting and Clinical Psychology 74:898–907, 2006
13.
Tuerk PW, Yoder M, Grubaugh A, et al.: Prolonged exposure therapy for combat-related posttraumatic stress disorder: an examination of treatment effectiveness for veterans of the wars in Afghanistan and Iraq. Journal of Anxiety Disorders 25:397–403, 2011
14.
Chard KM, Schumm JA, Owens GP, et al.: A comparison of OEF and OIF veterans and Vietnam veterans receiving cognitive processing therapy. Journal of Traumatic Stress 23:25–32, 2010
15.
Karlin BE, Ruzek JI, Chard KM, et al.: Dissemination of evidence-based psychological treatments for posttraumatic stress disorder in the Veterans Health Administration. Journal of Traumatic Stress 23:663–673, 2010
16.
Uniform mental health services in VA medical centers and clinics; in VHA Handbook 1160.01. Washington, DC, US Department of Veterans Affairs, Veterans Health Administration, 2008. Available at www1.va.gov/vhapublications/ViewPublication.asp?pub_ID=1762
17.
Wagner TH, Sinnott P, Siroka AM: Mental health and substance use disorder spending in the Department of Veterans Affairs, fiscal years 2000–2007. Psychiatric Services 62:389–395, 2011
18.
Rosen CS, Nguyen T, Bernardy NC, et al.: Evaluation of a mentoring program for PTSD clinic managers in the US Department of Veterans Affairs. Psychiatric Services 63:1047–1050, 2012
19.
Karlin BE: Bridging the gap in delivery of psychological treatments for posttraumatic stress disorder. Journal of Rehabilitation Research and Development 49:xiii–xvi, 2012
20.
Hunt JB, Curran G, Kramer T, et al.: Partnership for implementation of evidence-based mental health practices in rural federally qualified health centers: theory and methods. Progress in Community Health Partnerships: Research, Education, and Action6:389–398, 2012
21.
Chang ET, Rose DE, Yano EM, et al.: Determinants of readiness for primary care-mental health integration (PC-MHI) in the VA health care system. Journal of General Internal Medicine 28:353–362, 2013
22.
van Dijk MK, Verbraak MJ, Oosterbaan DB, et al.: Implementing practice guidelines for anxiety disorders in secondary mental health care: a case study. International Journal of Mental Health Systems 6:20, 2012
23.
Karlin BE, Brown GK, Trockel M, et al.: National dissemination of cognitive behavioral therapy for depression in the Department of Veterans Affairs health care system: therapist and patient-level outcomes. Journal of Consulting and Clinical Psychology 80:707–718, 2012
24.
Harris AH, Reeder RN, Ellerbe L, et al.: Are VHA administrative location codes valid indicators of specialty substance use disorder treatment? Journal of Rehabilitation Research and Development 47:699–708, 2010
25.
Shiner B, D’Avolio LW, Nguyen TM, et al.: Automated classification of psychotherapy note text: implications for quality assessment in PTSD care. Journal of Evaluation in Clinical Practice 18:698–701, 2012
26.
Arean PA, Raue PJ, Sirey JA, et al.: Implementing evidence-based psychotherapies in settings serving older adults: challenges and solutions. Psychiatric Services 63:605–607, 2012
27.
Salvador-Carulla L, Costa-Font J, Cabases J, et al.: Evaluating mental health care and policy in Spain. Journal of Mental Health Policy and Economics 13:73–86, 2010
28.
Hripcsak G, Wilcox A: Reference standards, judges, and comparison subjects: roles for experts in evaluating system performance. Journal of the American Medical Informatics Association 9:1–15, 2002
29.
Helfrich CD, Damschroder LJ, Hagedorn HJ, et al.: A critical synthesis of literature on the Promoting Action on Research implementation in Health Services (PARiHS) framework. Implementation Science 5:82, 2010
30.
Shiner B, D’Avolio LW, Nguyen TM, et al.: Measuring use of evidence based psychotherapy for posttraumatic stress disorder. Administration and Policy in Mental Health and Mental Health Services Research 40:311–318, 2013
31.
D’Avolio LW, Nguyen TM, Farwell WR, et al.: Evaluation of a generalizable approach to clinical information retrieval using the automated retrieval console (ARC). Journal of the American Medical Informatics Association 17:375–382, 2010
32.
D’Avolio LW, Nguyen TM, Goryachev S, et al.: Automated concept-level information extraction to reduce the need for custom software and rules development. Journal of the American Medical Informatics Association 18:607–613, 2011
33.
Kitson AL, Rycroft-Malone J, Harvey G, et al.: Evaluating the successful implementation of evidence into practice using the PARiHS framework: theoretical and practical challenges. Implementation Science 3:1, 2008
34.
Stetler CB, Damschroder LJ, Helfrich CD, et al.: A guide for applying a revised version of the PARiHS framework for implementation. Implementation Science 6:99, 2011
35.
Cook JM, O’Donnell C, Dinnen S, et al.: Measurement of a model of implementation for health care: toward a testable theory. Implementation Science 7:59, 2012
Information & Authors
Information
Published In
Cover: Gisele, by Elizabeth Shippen Green Elliott, published in Harper's magazine, 1908. Watercolor and charcoal drawing. Library of Congress Prints and Photographs Division, Washington, D.C.
History
Published in print: May 2014
Published online: 30 July 2014
Authors
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBLogin options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).