Bridging the Gap: A Hybrid Model to Link Efficacy and Effectiveness Research in Substance Abuse Treatment
Abstract
Many behavioral and pharmacologic treatments for which there is strong empirical support are rarely used in clinical practice in the treatment of substance dependence. In an effort to facilitate greater emphasis on issues such as utility, practicality, and cost earlier in the evaluation of promising therapies, the authors propose a hybrid model to link efficacy and effectiveness research. A hybrid model may foster broader use of empirically validated treatments in substance abuse treatment programs and enhance the scientific yield of effectiveness research. The hybrid model retains essential features of efficacy research (randomization, use of control conditions, independent assessment of outcome, and monitoring of treatment delivery) while expanding the research questions to also address issues of importance in effectiveness studies. Such issues include diversity in settings, clinicians, and patients; cost-effectiveness of treatment; training issues; and patient and clinician satisfaction.
A major issue facing virtually all areas of clinical psychiatry is the gap between research and practice and the associated call for clinical research to place a greater emphasis on effectiveness research and the evaluation of clinical utility of treatments (1,2,3). Disparities between research and practice are particularly apparent in drug abuse treatment (4). For example, despite overwhelming empirical support for the efficacy and cost-effectiveness of methadone maintenance (5,6,7), access to this form of treatment remains highly restricted in many areas of the United States (8,9,10). In locations in which methadone maintenance is available, this treatment is often administered in inadequate doses or with insufficient counseling and ancillary services, both of which make it much less effective (11,12). Similarly, behavioral treatments for which there is strong empirical support have rarely been implemented in clinical settings (13,14). Conversely, many treatments are widely used in clinical practice that have not undergone any evaluation of efficacy—for example, 12-step methods and auricular acupuncture (15,16).
Need for new research strategies
A recent landmark report by the Institute of Medicine (4) called for a greater emphasis on effectiveness research as a strategy for bridging the gap between research and practice. However, although methods and strategies for efficacy evaluation through randomized controlled trials have been well defined for both behavioral and pharmacologic therapies (17,18,19), the nature of what constitutes sound effectiveness research is much less clear. Traditional strategies—for example, large-scale demonstration projects—have too frequently had a negligible impact on the clinical community, policy makers, or third-party payers. This lack of impact may be due to several factors, such as failure to target critical questions about the treatment's utility in standard clinical practice, to methodologic flaws that have undermined internal validity, or to failure to address critical questions relevant to providers and policy makers.
The stage model of behavioral therapies research, developed by the National Institute on Drug Abuse (17), is one such attempt to conceptualize the transition from initial development of a new treatment intervention to ultimate community utilization. This model is innovative in the arena of behavioral therapies research, because it articulates three progressive stages that roughly parallel those for the development of pharmacologic therapies, as shown in Table 1 (18). Stage I consists of initial development and pilot or feasibility testing of new and untested treatments. Stage II consists principally of randomized controlled clinical trials to evaluate efficacy of treatments that have shown promise or efficacy in initial studies. Stage III, which corresponds to phase IV research for pharmacologic trials, is intended to address issues of transportability of treatments whose efficacy has been demonstrated in at least two stage II trials. A limitation of this model is the currently underdeveloped stage III, which is the focus of this article.
To illustrate the need to address practice-related issues in efficacy research, we present two examples of pharmacologic and behavioral treatments that have demonstrated efficacy and that have largely failed to be adopted in clinical practice. We then propose a hybrid strategy for effectiveness research that retains critical features of efficacy trials but that extends these features to address issues of interest to clinicians and policy makers.
Behavioral treatments
Cocaine dependence became a major public health problem in the 1980s and was met with a multitude of evaluations of novel behavioral and pharmacologic approaches. Despite many trials that have evaluated a wide range of pharmacologic agents, none has consistently been found to be effective in treating general groups of cocaine users (12). Thus, perhaps the most exciting findings on the effectiveness of treatment for cocaine dependence have been the reports (20,21) on contingency management.
However, the initial demonstrations of the impressive effects of contingency management (21) were conducted in a comparatively homogeneous population of Caucasian, middle-class cocaine users, which raised concerns about generalizability to more diverse populations. However, numerous clinical trials have since demonstrated the efficacy of voucher-based contingency management in a broad range of settings and populations. These include methadone maintenance settings for cocaine-dependent substance users in large urban areas (22), homeless substance users with significant levels of psychopathology (23), and freebase cocaine users (24).
Contingency management has also been shown to be flexible in that several other behaviors can also be targeted—for example, medication compliance (25,26). The consistency and strength of the empirical data supporting contingency management have been impressive; few negative results of trials have been reported, and the estimated effect size of .25 in methadone maintenance settings (27) is notable. Nevertheless, ten years after the initial report on their efficacy, contingency management approaches are virtually nonexistent outside research settings.
Several reasons have been cited for the failure of contingency management approaches to be adopted in community settings. First, the cost of this approach—in the original Higgins system, patients may receive more than $1,000 in vouchers—has been seen as unrealistic. However, lower-cost contingency management approaches, with an average cost of $200 per patient, have recently been demonstrated to be effective in community settings (28,29).
Second, ideological issues have limited the acceptance of contingency management procedures outside research settings. These approaches have been perceived as tantamount to "paying patients to stay clean." Moreover, it is not clear that heretofore limited efforts to disseminate contingency management have emphasized the basic behavioral principles underlying this approach, such as the value of positive reinforcement in changing behavior.
Third, several aspects of contingency management may be difficult for clinicians to implement without programmatic or payer support. For example, verification of abstinence requires much more frequent urine screening than is typical in many clinical programs (30). Successful implementation of contingency management also requires rapid feedback of urine test results, which is often unavailable in community-based settings.
Pharmacotherapies
In contrast with behavioral therapies, pharmacotherapies typically command vigorous, sophisticated treatment dissemination efforts funded by private pharmaceutical firms that stand to profit directly from widespread use. Despite such considerations, only two agonist agents—methadone for opioid dependence and nicotine replacement treatments for tobacco dependence—can be characterized as playing a major role in routine substance abuse treatment. Other agents that have been approved by the Food and Drug Administration are used among less than 5 percent of the target population of substance abusers who are in treatment (31,32,33). Factors that impede the transfer of new pharmacotherapies from research settings to routine clinical use are exemplified by the largely unsuccessful U.S. marketing efforts for naltrexone treatment for alcohol dependence in the two years after naltrexone received FDA approval in late 1994.
Many indicators suggested the promise of naltrexone as a treatment for alcohol dependence. In two well-designed randomized clinical trials, naltrexone demonstrated strong superiority over placebo for preventing relapse and maintaining abstinence in recently detoxified alcoholics in outpatient treatment (34,35,36). A third open-label study of more than 500 patients had similar success rates and affirmed safety and tolerability of naltrexone for alcoholics with and without comorbid psychopathology (37).
Alcohol dependence is the most frequently diagnosed mental disorder in large-scale community surveys (38,39,40), affecting 4 to 6 percent of adults at any given time. Naltrexone has only one pharmacological competitor—disulfiram—which received FDA approval more than 40 years ago and which has been infrequently used because of low patient acceptance and limited efficacy results (41). Nevertheless, and despite an extensive marketing campaign, naltrexone failed to become a mainstream treatment for alcohol dependence. Its manufacturer, DuPont Pharma, ceased U.S. marketing efforts after two years of prescription rates that fell far short of initial projections.
Rigorously designed clinical trials indicated naltrexone's strengths but failed to address what turned out to be major impediments to its dissemination. One major factor is the decentralization and diversity of alcoholism treatment and the limited role of physicians in specialized substance abuse programs. Hence traditional educational efforts that targeted physicians failed to reach the nonmedical counselors and clinicians who were responsible for day-to-day treatment. In addition, the highly influential Alcoholics Anonymous philosophy encourages patients to avoid reliance on medications to maintain sobriety (42). Thus, even if physicians were enthusiastic about naltrexone, lack of support by counseling staff tended to prevent referrals and to undermine compliance.
Second, naltrexone cannot be considered a stand-alone treatment, because its efficacy has been tested predominantly when the agent was combined with specialized behavioral treatment programs (34,35). No data are available on naltrexone's effectiveness in combination with more generally available treatments, such as counseling, self-help groups, and medically oriented clinical management. This requirement for nontraditional behavioral treatment has raised doubts about naltrexone's use in the context of standard treatment programs. Similarly, naltrexone's efficacy has been evaluated only among patients who have already achieved seven to 30 days of abstinence from alcohol, raising questions about its potential for the many patients who have difficulty achieving sustained abstinence.
Third, the use of naltrexone entails some risks and requires medical monitoring (43). The relatively low risk of hepatotoxicity requires that liver function blood tests be performed before treatment and periodically throughout treatment. Highly elevated liver function levels, a frequent finding in alcoholism, is a relative contraindication for naltrexone treatment. In addition, the use of naltrexone entails complicated considerations in the treatment of pain. Because naltrexone is a powerful opioid antagonist, it can cause withdrawal symptoms among patients who are currently using opioid medications and precludes routine use of opioid analgesics, such as codeine or morphine, if patients develop a painful condition of sudden onset—for example, renal stones or trauma. These medical complications tended to limit naltrexone prescribing to physicians with a special interest in substance abuse treatment and tended to put off generalist psychiatrists and primary care physicians.
Finally, the fact that there are only two FDA-approved medications for alcoholism has had unanticipated drawbacks. Although naltrexone has only one weak competitor—disulfiram—correspondingly few patients and clinicians are accustomed to thinking about pharmacotherapy as a major part of alcoholism treatment after detoxification. Finally, many patients and clinicians have expressed concerns about the cost of naltrexone at $5 to $6 a day, naltrexone remains unlisted in many prescription plans.
Limitations of existing efficacy models
Efficacy trials typically involve tightly controlled settings and more narrowly defined, homogeneous samples than those seen in clinical practice, which raises questions about their generalizability (44,45). The higher costs of treatment delivery and training relative to the potential benefits of novel approaches have not been addressed. The feasibility and acceptability of the approaches to clinicians, payers, and patients are rarely systematically evaluated in efficacy trials.
A hybrid model
The challenges of fostering wider use of empirically validated treatments in clinical settings revolve around a number of critical issues: generalizability (Will this treatment work with different practitioners, patients, and settings?), implementation (What kinds of training and what kinds of trainers are necessary for what kinds of clinicians to learn a new technique?), cost-effectiveness (Compared with the costs of learning and implementing this treatment, what are the savings, particularly in comparison to existing methods?), and marketing (How acceptable is a new treatment to both clinicians, patients, and payers outside research settings?)
Efficacy and effectiveness research have often been regarded as being at opposite ends of the internal-external validity continuum (46). Efficacy trials are frequently criticized as having limited generalizability to real-world issues and problems, whereas effectiveness studies that place a high premium on ecological validity often lack design features required to definitively answer basic issues regarding treatment effectiveness or to rule out alternative explanations of findings. Below we propose a "hybrid" model that shares critical components of efficacy research but also includes features intended to reduce barriers to implementation of these treatments in clinical settings (Table 2).
Elements of efficacy trials to be retained
In a hybrid model, it is critical that scientific rigor be preserved through the use of design features that protect crucial aspects of internal validity. These features include random assignment of patients to treatments, blind delivery of treatments, blind assessment of outcomes, intention-to-treat analyses, use of objective outcome measures, definition and monitoring of treatments delivered, specialized training of providers in delivering study treatments, and evaluation of the integrity of study treatments (for example, assessment of compliance in medication trials and evaluation of treatment fidelity in behavioral trials).
Although these features add to the cost and complexity of clinical trials, the disadvantages of omitting them are numerous and consequential. For example, lack of randomization to treatment conditions means that a study loses the ability to address fundamental questions, such as whether the experimental treatment was more effective than standard treatment and whether the level of change associated with the experimental treatment was meaningful. Most important, however, is that studies that do not randomly assign participants to a condition lose the ability to rule out numerous alternative explanations for findings, such as maturation, history, and measurement effects. Borkovec and colleagues (47) have noted that failure to protect these vital aspects of internal validity renders generalizability issues moot.
Moreover, for most empirically validated treatments for substance abuse, only a handful of supporting clinical trials may exist. Thus, even as effectiveness studies are undertaken, it will be essential to continue to conduct randomized controlled comparisons, because there are few true reference conditions in substance abuse treatment against which to evaluate the efficacy of novel approaches. For example, in the treatment of opioid dependence, methadone maintenance most resembles a standard "reference" treatment. However, largely due to variability in the clinical context in which it is delivered, outcomes for methadone maintenance are highly variable (11,48).
In addition, control conditions in efficacy trials are generally designed to evaluate highly focused research questions—for example, whether a given medication is more effective than a placebo that controls for expectations of improvement or whether a given behavioral therapy is more effective than a minimal discussion condition. Thus control conditions that are typically used in efficacy research rarely resemble treatment as delivered in clinical practice and thus do not address issues of interest to clinicians and policy makers—for example, whether adding a novel treatment enhances outcomes compared with treatment as usual.
Effectiveness elements to be retained
To expand the range of issues addressed in efficacy research, we propose a hybrid model in which the above design elements are retained while additional effectiveness components are added.
Enhanced diversity in patients and settings. Few single-site trials are designed or sufficiently powered to allow meaningful analysis of sources of variability in outcome due to variables such as ethnicity, gender, severity, or psychiatric comorbidity. Diversity may be enhanced by using a number of strategies, such as conducting clinical trials in community-based programs and multiple geographic settings, using less restrictive inclusion and exclusion criteria, reducing barriers to participation (for example, not restricting protocols to English-speaking participants or not requiring participants to undergo very lengthy assessment batteries), and allowing some variation in treatment implementation across sites.
Such strategies will greatly enhance our knowledge about the robustness of various treatments. For example, it will be extremely useful to know if treatment A is effective among persons with both alcohol and cocaine abuse but not those with both depression and cocaine abuse. Similarly, it is of great importance to know whether an innovative treatment is effective when implemented by a broad range of clinicians, as opposed to just those with advanced degrees.
Enhancing diversity in study populations typically requires multiple single-site replication studies or larger multisite trials. However, Klein and Smith (49) have noted that even community-based replications or multisite trials may have limited generalizability. That is, simply because a study is conducted in a community-based setting does not mean that its results are generalizable to all other community settings.
Attention to training issues. If clinicians are to implement empirically validated treatments, effective training is essential. Although pharmaceutical companies widely disseminate information about new pharmacologic approaches to physicians, many community-based treatment programs have no or few affiliated medical personnel to implement these treatments (50).
Similarly, although methods for training clinicians in manual-guided therapies for clinical efficacy trials are well established (51,52,53), such methods have largely been accepted on the basis of face validity (54,55,56). It is not known whether standard methods of training therapists will be feasible or effective when applied to real-world clinicians. Training in clinical efficacy trials is usually geared toward highly trained and experienced clinicians. Thus trainers may assume basic familiarity with underlying principles of the treatment. However, clinicians who work in community-based drug abuse settings have varied educational backgrounds (4). Thus standard training methods for clinical trials—which encompass brief review of a treatment manual, watching a few videotaped vignettes, and participating in a few role-playing exercises—may be insufficient for many drug abuse counselors.
If we are to learn what kind of training is needed by various types of clinicians to effectively implement scientifically validated treatments, empirical studies of training methods are needed. Focused evaluations of training methods may be integrated into large stage III hybrid studies in which, for example, the pilot or startup phase might be seen as providing a vehicle for systematically evaluating the effectiveness of various training methods with a variety of types of clinicians and settings. That is, clinicians could be randomly assigned to different training methods, such as low-intensity versus high-intensity training, and their adherence and competence in implementing treatments could be evaluated.
Evaluation of cost-effectiveness. Almost all novel therapies come with some added cost, either the cost of the treatment itself or training costs. Because the added cost of empirically validated treatments has typically not been a component of efficacy studies, researchers have had little basis on which to convince program leaders or policy makers that the efforts and costs entailed in introducing new treatments are justified.
The value of integrating cost-effectiveness analyses into clinical trials—even those that occur early during the course of a treatment's evaluation—is suggested by several recent studies that have demonstrated cost savings and benefits associated with adding enhancements to standard approaches to substance abuse treatment (57,58,59,60,61). When cost savings for innovative treatments are demonstrated, providers and third-party payers are likely to be much more interested in supporting the adoption of these treatments into standard care.
Assessment of patient and provider satisfaction. In recent years, increasing emphasis has been placed on client preference and patient satisfaction as indicators of a treatment's utility and value (62,63,64,65). Indicators of patient satisfaction are important in determining whether a new approach would add value to a program by making it more attractive to patients. Inclusion of satisfaction measures that address patients' global judgments of satisfaction and improvement, supplemented by more specific feedback on their reactions to individual components of innovative treatments—for example, content, duration, intensity, and therapist (66)—will be invaluable in evaluating the success of novel treatments in community settings. Greater satisfaction of clinicians with the nature of the interventions they provide to their patients may be important in reducing staff turnover in community programs, in which the rate of turnover is as high as 15 to 50 percent annually (13,67).
Acknowledgments
Preparation of this article was supported by grants U10-DA-13038, P50-DA0924, and K05-DA00457 from the National Institute on Drug Abuse.
Footnote
The authors are affiliated with the department of psychiatry at Yale University School of Medicine and with the Department of Veterans Affairs VISN 1 Mental Illness, Research, and Clinical Center of the VA Connecticut Healthcare System in West Haven, Connecticut. Send correspondence to Dr. Carrol at the Division of Substance Abuse, VA Connecticut Healthcare System (151D), West Haven, Connecticut 06516 (e-mail, [email protected]).
References
1.
Beutler LE, Howard KI: Clinical utility research: an introduction. Journal of Clinical Psychology 54:297-301, 1998
2.
Norquist G, Lebowitz B, Hyman S: Expanding the frontier of treatment research. Prevention and Treatment 2, 1999. Available at www.journals.apa.org/prevention/volume2/pre0020001a.html
3.
Wells KB: Treatment research at the crossroads: the scientific interface of clinical trials and effectiveness research. American Journal of Psychiatry 156:5-10, 1999
4.
Institute of Medicine: Bridging the Gap Between Practice and Research: Forging Partnerships With Community-Based Drug and Alcohol Treatment. Washington, DC, National Academy Press, 1998
5.
Sees KL, Delucchi KL, Masson C, et al: Methadone maintenance vs 180-day psychosocially enriched detoxification for treatment of opioid dependence: a randomized controlled trial. JAMA 283:1303-1310, 2000
6.
Barnett PG: The cost-effectiveness of methadone maintenance as a health care intervention. Addiction 94:479-488, 1999
7.
Barnett PG, Hui SS: The cost-effectiveness of methadone maintenance. Mount Sinai Journal of Medicine 5:365-374, 2000
8.
National Consensus Development Panel on Effective Medical Treatment of Opiate Addiction: Effective medical treatment of opiate addiction. JAMA 280:1936-1943, 1998
9.
Rounsaville BJ, Kosten TR: Treatment for opioid dependence: quality and access. JAMA 283:1337-1339, 2000
10.
McCarty D, Frank RG, Denmead GC: Methadone maintenance and state Medicaid managed care programs. Milbank Quarterly 77:341-362, 1999
11.
Ball JC, Ross A: The Effectiveness of Methadone Maintenance Treatment. New York, Springer-Verlag, 1991
12.
O'Brien CP: A range of research-based pharmacotherapies for addiction. Science 278:66-70, 1997
13.
McLellan AT, McKay JR: The treatment of addiction: what can research offer practice? in Bridging the Gap Between Practice and Research: Forging Partnerships With Community Based Drug and Alcohol Treatment. Edited by Lamb S, Greenlick MR, McCarty D. Washington, DC, National Academy Press, 1998
14.
Miller WR, Brown JM, Simpson TL, et al: What works? A methodological analysis of the alcohol treatment literature, in Handbook of Alcoholism Treatment Approaches: Effective Alternatives. Edited by Hester RK, Miller WR. Boston, Mass, Allyn & Bacon, 1995
15.
McLellan AT, Grossman DS, Blaine JD, et al: Acupuncture treatment for drug abuse: a technical review. Journal of Substance Abuse Treatment 10:569-576, 1993
16.
Horgan CM, Levine HJ: The substance abuse treatment system: what does it look like and whom does it serve? in Bridging the Gap Between Practice and Research: Forging Partnerships With Community-Based Drug and Alcohol Treatment. Edited by Lamb S, Greenlick MR, McCarty D. Washington, DC, National Academy Press, 1999
17.
Rounsaville BJ, Carroll KM, Onken LS: A stage model of behavioral therapies research: getting started and moving on from stage I. Clinical Psychology: Science and Practice 8:133-142, 2001
18.
Prien RF, Robinson DS: Clinical Evaluation of Psychotropic Drugs: Principles and Guidelines. New York, Raven, 1994
19.
Institute of Medicine: Small Clinical Trials: Issues and Challenges. Washington, DC, National Academy Press, 2001
20.
Budney AJ, Higgins ST: A Community Reinforcement Plus Vouchers Approach: Treating Cocaine Addiction. Rockville, Md, National Institute on Drug Abuse, 1998
21.
Higgins ST, Delany DD, Budney AJ, et al: A behavioral approach to achieving initial cocaine abstinence. American Journal of Psychiatry 148:1218-1224, 1991
22.
Silverman K, Higgins ST, Brooner RK, et al: Sustained cocaine abstinence in methadone maintenance patients through voucher-based reinforcement therapy. Archives of General Psychiatry 53:409-415, 1996
23.
Milby JB, Schumacher JE, Raczynski JM, et al: Sufficient conditions for effective treatment of substance abusing homeless persons. Drug and Alcohol Dependence 43:39-47, 1996
24.
Kirby KC, Marlowe DB, Festinger DS, et al: Schedule of voucher delivery influences initiation of cocaine abstinence. Journal of Consulting and Clinical Psychology 66:761-767, 1998
25.
Carroll KM, Ball SA, Nich C, et al: Targeting behavioral therapies to enhance naltrexone treatment of opioid dependence: efficacy of contingency management and significant other involvement. Archives of General Psychiatry 58:755-761, 2001
26.
Preston KL, Silverman K, Umbricht A, et al: Improvement in naltrexone treatment compliance with contingency management. Drug and Alcohol Dependence 54:127-135, 1999
27.
Griffith JD, Rowan-Szal GA, Roark RR, et al: Contingency management in outpatient methadone treatment: a meta-analysis. Drug and Alcohol Dependence 58:55-66, 2000
28.
Petry NM, Martin B, Cooney JL, et al: Give them prizes and they will come: contingency management treatment of alcohol dependence. Journal of Consulting and Clinical Psychology 68:250-257, 2000
29.
Petry NM, Martin B: Low-cost contingency management for treating cocaine- and opioid abusing methadone patients. Journal of Consulting and Clinical Psychology 70:398-405, 2002
30.
Saxon AJ, Calsyn DA, Haver VM, et al: A nationwide survey of urinalysis practices of methadone maintenance clinics: utilization of laboratory services. Archives of Pathology and Laboratory Medicine 114:94-100, 1990
31.
Kessler RC, Zhao S, Katz SJ, et al: Past-year use of outpatient services for psychiatric problems in the National Comorbidity Study. American Journal of Psychiatry 156:115-123, 1999
32.
Regier DA, Narrow WE, Rae DS, et al: The defacto US mental health and addictive disorders service system: Epidemiological Catchment Area prospective one-year prevalence rates of disorders and services. Archives of General Psychiatry 50:85-91, 1993
33.
Norquist G, Regier DA: The epidemiology of psychiatric disorders and the de facto mental health care system. Annual Review of Medicine 47:473-479, 1996
34.
O'Malley SS, Jaffe AJ, Chang G, et al: Naltrexone and coping skills therapy for alcohol dependence: a controlled study. Archives of General Psychiatry 49:881-887, 1992
35.
Volpicelli JR, Alterman AI, Hayashida M, et al: Naltrexone in the treatment of alcohol dependence. Archives of General Psychiatry 49:876-880, 1992
36.
O'Malley SS, Croop RS, Wroblewski JM, et al: Naltrexone in the treatment of alcohol dependence: a combined analysis of two trials. Psychiatric Annals 25:681-688, 1995
37.
Croop RS, Falkner EB, Lagriola DF, et al: The safety profile of naltrexone in the treatment of alcoholism: results from a multicenter usage study. Archives of General Psychiatry 54:1130-1135, 1997
38.
Regier DA, Farmer ME, Rae DS, et al: Comorbidity of mental disorders with alcohol and other drug abuse: results from the Epidemiologic Catchment Area (ECA) study. JAMA 264:2511-2518, 1990
39.
Kessler RC, Crum RM, Warner LA, et al: Lifetime co-occurence of DSM-III-R alcohol abuse and dependence with other psychiatric disorders in the National Comorbidity Study. Archives of General Psychiatry 54:313-321, 1997
40.
Kessler RC, McGonagle KA, Zhao S, et al: Lifetime and 12-month prevalence of DSM-III-R psychiatric disorders in the United States: results from the National Comorbidity Study. Archives of General Psychiatry 51:8-19, 1994
41.
Fuller RK, Branchey L, Brightwell DR, et al: Disulfiram treatment of alcoholism: a Veterans Administration cooperative study. JAMA 256:1449-1455, 1986
42.
Rychtarik RG, Connors GJ, Derman KH, et al: Alcoholics Anonymous and the use of medications to prevent relapse: an anonymous survey of member attitudes. Journal of Studies on Alcohol 61:134-138, 2000
43.
Gonzalez JP, Brogden RN: Naltrexone: a review of its pharmacodynamic and pharmacokinetic properties and therapeutic efficacy in the management of opioid dependence. Drugs 35:192-213, 1988
44.
Humphreys K, Weisner C: Use of exclusion criteria in selecting research subjects and its effect on the generalizability of alcohol treatment outcome studies. American Journal of Psychiatry 157:588-594, 2000
45.
Carroll KM, Nich C, McLellan AT, et al: Research versus real world patients: representativeness of subjects participating in clinical trials of treatments for cocaine dependence. Drug and Alcohol Dependence 54:171-177, 1997
46.
Donenberg GR, Lyons JS, Howard KI: Clinical trials versus mental health services research: contributions and connections. Journal of Clinical Psychology 55:1135-1145, 1999
47.
Borkovec TD, Echemendia RJ, Ragusea SA, et al: The Pennsylvania Practice Research Network and future possibilities for clinically meaningful and scientifically rigorous psychotherapy effectiveness research. Clinical Psychology: Science and Practice 8:155-167, 2001
48.
McLellan AT, Arndt IO, Metzger D, et al: The effects of psychosocial services in substance abuse treatment. JAMA 269:1953-1959, 1993
49.
Klein DF, Smith LB: Organizational requirements for effective clinical effectiveness studies. Prevention and Treatment 2,1999. Available at www.journals.apa.org/prevention/volume2/pre0020002a.html
50.
Carise D, McLellan AT, Gifford LS, et al: Developing a national addiction treatment information system: an introduction to the Drug Evaluation Network System. Journal of Substance Abuse Treatment 17:67-77, 1999
51.
Carroll KM, Connors GJ, Cooney NL, et al: Internal validity of Project MATCH treatments: discriminability and integrity. Journal of Consulting and Clinical Psychology 66:290-303, 1998
52.
Crits-Christoph P, Siqueland L, Chittams J, et al: Training in cognitive, supportive-expressive, and drug counseling therapies for cocaine dependence. Journal of Consulting and Clinical Psychology 66:484-492, 1998
53.
Rounsaville BJ, Chevron E, Weissman MM, et al: Training therapists to perform interpersonal psychotherapy in clinical trials. Comprehensive Psychiatry 27:364-371, 1986
54.
Crits-Christoph P, Baranackie K, Kurcias J, et al: Meta-analysis of therapist effects in psychotherapy outcome studies. Psychotherapy Research 1:81-91, 1991
55.
DeRubeis RJ, Hollon SD, Evans MD, et al: Can psychotherapies for depression be discriminated? A systematic evaluation of cognitive therapy and interpersonal psychotherapy. Journal of Consulting and Clinical Psychology 50:744-756, 1982
56.
Hill CE, O'Grady KE, Elkin I: Applying the Collaborative Study Psychotherapy Rating Scale to rate therapist adherence in cognitive-behavioral therapy, interpersonal therapy, and clinical management. Journal of Consulting and Clinical Psychology 60:73-79, 1992
57.
Kraft MK, Bothbard AB, Hadley TR, et al: Are supplementary services provided during methadone maintenance really cost-effective? American Journal of Psychiatry 154:1214-1219, 1997
58.
Avants SK, Margolin A, Sindelar JL, et al: Day treatment versus enhanced standard methadone services for opioid-dependent patients: a comparison of clinical efficacy and cost. American Journal of Psychiatry 156:27-33, 1999
59.
Rosenheck RA, Kosten TR: Buprenorphine for opiate addiction: potential economic impact. Drug and Alcohol Dependence 63:253-262, 2001
60.
Holder HD, Cisler RA, Longabaugh R, et al: Alcoholism treatment and medical care costs for Project MATCH. Addiction 95:999-1013, 2000
61.
Svikis DS, Golden AS, Guggins GR, et al: Cost-effectiveness of treatment for drug-abusing pregnant women. Drug and Alcohol Dependence 45:105-113, 1997
62.
Kazdin AE: Comparative outcome studies of psychotherapy: methodological issues and strategies. Journal of Consulting and Clinical Psychology 54:95-105, 1986
63.
McLellan AT, Hunkeler E: Patient satisfaction and outcomes in alcohol and drug abuse treatment. Psychiatric Services 49:573-575, 1998
64.
Holcomb WR, Parker JC, Leong GB, et al: Customer satisfaction and self-reported treatment outcomes among psychiatric inpatients. Psychiatric Services 49:1622-1623, 1998
65.
Rohland BM, Langbehn DR, Rohrer JE: Relationship between service effectiveness and satisfaction among persons receiving Medicaid mental health services. Psychiatric Services 51:248-250, 2000
66.
Donovan DM, Kadden RM, DiClemente CC, et al: Client satisfaction with three therapies in the treatment of alcohol dependence: results from Project MATCH. American Journal on Addictions 11:291-307, 2002
67.
McLellan AT, Belding MA, McKay JR, et al: Can the outcomes research literature inform the search for quality indicators in substance abuse treatment? in Managing Managed Care: Quality Improvement in Behavioral Health. Edited by Edmunds M, Frank RG, Hogan M, et al. Washington, DC, National Academy Press, 1996
Information & Authors
Information
Published In
History
Published online: 1 March 2003
Published in print: March 2003
Authors
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBGet Access
Login options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).