Association Between Quality Measures and Perceptions of Care Among Patients With Substance Use Disorders
Abstract
Objective:
This study evaluated whether eight quality measures assessing care for patients with a substance use disorder were associated with patient perceptions of their care, including perceived improvement and global rating of behavioral health care.
Methods:
Secondary data analyses were conducted of administrative and patient survey data collected as part of a national evaluation of Veterans Health Administration (VHA) mental health and substance use services. Data for patients who received care for substance use disorders during October 2006–September 2007 paid for by the VHA and who participated in a telephone interview about their care (N=2,074) were included. Measures of patient perceptions of care included perceived improvement and global rating of behavioral health care. Eight quality measures based on administrative data assessed initiation and engagement in substance use disorder care, receipt of psychotherapy or psychosocial treatment, and follow-up after hospitalization. Regression models were conducted in which each quality measure predicted each outcome, with analyses adjusting for patient characteristics and functioning.
Results:
Treatment engagement, two measures of psychotherapy receipt, and psychosocial treatment were significantly associated with perceived improvement, whereas treatment initiation and follow-up after hospitalization (seven and 30 days) were not. Psychotherapy receipt and follow-up after hospitalization (seven and 30 days) were significantly associated with global rating of behavioral health care.
Conclusions:
Some quality measures assessing care for substance use disorders were significantly associated with patient perceptions of care. Results provide additional support for these quality measures and suggest that patient perceptions of care are an important outcome in assessing care.
Quality measures can be a valuable tool to assess and improve quality of care for substance use disorders, yet there are few validated substance use disorder quality measures (1–4). Furthermore, very few measures endorsed by the National Quality Forum (NQF) assess care for substance use disorders (4). Nonetheless, there is increased emphasis on assessing quality and using quality measures in new payment and incentive models, and greater access to substance use disorder treatment could follow from health care reform legislation. Together these developments may stimulate interest in validated substance use disorder quality measures. Recently, in fact, quality measures for substance use disorder treatment were highlighted as a key priority for new measure development (5–9).
When process-based behavioral health quality measures are developed and tested, they are typically evaluated in terms of their ability to predict clinical outcomes (4,10). Demonstrating this process-outcome link helps to ensure that resources targeted toward improving quality of care are also likely to result in improved patient outcomes. Yet, despite several important efforts, few process-based quality measures of substance use disorder treatment have demonstrated a process-outcome link (11–21). There remains a need for validated quality measures to assess care for substance use disorder.
An alternative approach to examining the association between process-based quality measures and clinical outcomes is to assess whether quality measures are associated with patient perceptions of the care they received, given that patient perceptions of care are also an important outcome. Measures of patient perceptions of care provide critical information for understanding whether care delivered is patient centered (22), a dimension of health care quality highlighted over a decade ago (23). It has been argued that delivering patient-centered care is an important goal, even independent of its relationship with health outcomes (24). Beyond simple patient satisfaction, patient perceptions of care include a variety of dimensions, including timely access to care, communication (for example, feeling respected and listened to by providers), shared decision making, and the helpfulness of treatment (25,26). Additionally, prior work suggests that perceptions of care may be related to treatment adherence and outcomes. Specifically, better patient care experiences are associated with higher levels of adherence to recommended prevention and treatment processes, better clinical outcomes, better patient safety within hospitals, and less health care utilization (27). Furthermore, data suggest that patients with substance use disorders may have lower perceptions of care compared with patients receiving care for psychiatric diagnoses (28,29). Taken together, improvement in patient perceptions of care may be an important outcome on its own.
There have been few attempts to evaluate quality measures in terms of their associations with patient perceptions of care, but preliminary work has been promising (30). To address the paucity of literature in this area, we sought to evaluate whether eight quality measures that assess care for patients with a substance use disorder were associated with patient perceptions of care, including perceived improvement and global rating of behavioral health care.
Methods
We conducted secondary data analyses of administrative and patient telephone survey data collected as part of a national evaluation of mental health and substance use services provided by the Veterans Health Administration (VHA) (31,32), which examined care for substance use disorders, schizophrenia, bipolar disorder, posttraumatic stress disorder (PTSD), and major depressive disorder. Administrative data were obtained from the VHA Medical SAS data sets and included demographic information, claims, diagnoses, dates and types of services, admissions, and discharges. The study population included patients who received care from or paid for by the VHA in fiscal year (FY) 2007 and who had at least one inpatient episode or two outpatient encounters of which at least one was for a primary or secondary diagnosis of schizophrenia, bipolar disorder, PTSD, or depression. Patients were assigned to one of the four psychiatric diagnostic cohorts based on the modal frequency of episodes of care, with ties resolved by using the following rank order: schizophrenia, bipolar, PTSD, and depression. In addition, patients with substance use disorder diagnoses were identified as a separate cohort. Patients with co-occurring substance use and psychiatric disorders were assigned to both the substance use disorder cohort and a psychiatric diagnostic cohort.
We drew a random sample, stratifying the population by geographic region, psychiatric diagnostic cohort, and substance use disorder cohort. Given concerns with recall bias in asking respondents to recall the care they received in FY 2007, survey questions focused on mental health care during the 12 months prior to the interview date. Therefore, we restricted the study population to patients with any VHA health care utilization in FY 2008 to minimize the number of survey respondents who would report having received no care from the VHA in the prior 12 months at the time they were contacted for the survey. A total of 6,190 patients participated in the telephone interview (response rate of 67% for all patients and 62% for patients in the substance use disorder cohort), which was fielded from November 2008 to August 2009.
These analyses focus on telephone survey respondents who had a substance use disorder diagnosis when selected for a cohort and had available patient perceptions data (N=2,074). Specifically, approximately one-third (N=736) of the sample had only a substance use disorder, and two-thirds (N=1,338) had a substance use disorder and a co-occurring psychiatric diagnosis (depression, bipolar, PTSD, or schizophrenia). Additional detail on the survey methods is available elsewhere (29,33). The RAND Human Subjects Protection Committee and institutional review boards of the Central Arkansas Veterans Healthcare Center and the University of Arkansas for Medical Sciences approved all procedures. Informed consent was obtained prior to the telephone interview.
Measures
Patient Characteristics
Age, gender, and disability service connection were drawn from administrative data, and race-ethnicity, marital status, education, employment status, income, mental and physical health functioning, and service utilization (inpatient nights and outpatient visits) were drawn from survey data. Functioning was assessed by using the Veterans RAND 12-Item Health Survey (34). The two composite scores reflecting mental and physical health functioning range from 0 to 100, with higher scores indicating better functioning. Both scores are norm-based and can be interpreted in relation to the distribution of scores in the 1990 adult U.S. population (mean±SD=50±10).
Process-of-Care Measures
We examined eight quality measures that are based on administrative data, including initiation and engagement in substance use disorder care (35), receipt of any psychotherapy or any psychosocial treatment, and follow-up after hospitalization. Four of these measures are endorsed by the NQF (initiation, engagement, and follow-up after hospitalization within seven days and 30 days). The remaining four measures were developed in a prior evaluation of VHA care (31). Some measures assess care received in the four months following initiation of a new treatment episode for a substance use disorder, and some assess care for the entire FY 2007 observation period.
A new treatment episode was defined as any inpatient hospitalization for or an outpatient visit with a substance use disorder diagnosis after a break in care of 150 days or more. A break in care was defined as no outpatient visits with a substance use disorder diagnosis or substance use disorder–related medication, which differs from the 60 days used in the NQF specifications for initiation and engagement, with the goal of increasing the likelihood of including only patients who were not in treatment. The psychotherapy measure was restricted to receiving various types of psychotherapy, whereas the psychosocial treatment indicator included both psychotherapy and other psychosocial interventions (mental health intensive case management, family psychoeducation, and supported employment). Detailed technical specifications of each quality measure are available elsewhere (32).
Outcome Measures
We assessed patients’ perceived improvement by computing a mean of four items from the Experience of Care and Health Outcomes (ECHO) survey (36,37), which assesses improvement in ability to deal with daily problems and social situations, ability to accomplish goals, and symptoms in the past 12 months. Respondents are asked to rate each target area on a 5-point scale, from 1 (much worse) to 5 (much better), compared with 12 months ago. Internal consistency reliability of these items was .88, providing support for computing a mean score. Respondents were required to have responses for at least three of the four items. We used person-mean imputation for 21 respondents who were missing one item. For patients who endorsed receiving care from the VHA in the past year (N=1,575), we assessed their overall perception of their VHA behavioral health care by using one item from the ECHO. Patients were asked to rate all their “counseling or treatment” in the past 12 months from 0 to 10, where 0 was the “worst counseling or treatment possible” and 10 was the “best counseling or treatment possible.” The format of this item maps to the Consumer Assessment of Healthcare Providers and Systems (CAHPS) global rating item, which has been implemented across a range of health care settings (25). Because responses to this item tend to be highly skewed, we analyzed this item as a dichotomous measure (ratings of 0–8 versus 9–10). The correlation between perceived improvement and global rating of care was .34, suggesting that these measures reflect related but still unique aspects of patient perceptions.
Statistical Analyses
We present demographic and other treatment characteristics and performance on each quality measure, weighted to reflect the population of patients who received care associated with a substance use disorder diagnosis. We then present results from multiple regression models of the association between perceived improvement and each quality measure and logistic regression models of the association between the global rating of care and each quality measure. These models adjusted for patient characteristics (age, race-ethnicity, gender, marital status, service connection, and rural/urban residence status) and mental and physical health functioning. Sampling weights were used to adjust for the stratified sampling design (38). A nonresponse weight reflecting the inverse probability of each sampled patient completing the survey was derived by using logistic regression of an indicator of survey completion on veteran characteristics (39). To reflect the population of patients who received care associated with a substance use disorder diagnosis, all estimates were weighted by a final analysis weight that was the product of the sampling and nonresponse weights. The sandwich estimator was used to obtain robust standard error estimates for regression coefficients (40). Given that multiple statistical tests were conducted, we accounted for multiple comparisons by holding the false discovery rate to 5% (41).
Results
Sample Characteristics
Weighted data on the demographic and treatment characteristics of the sample are provided in Table 1. The average age was approximately 54 years, over 90% were men, and about 60% were non-Hispanic white. A majority of patients completed at least some college (53%), were out of the labor force (68%), and reported annual incomes of $30,000 or less (74%). Only about a third of patients (37%) had a service-connected disability for any condition, meaning a mental or general medical condition that was incurred or aggravated during active duty. Respondents’ mental and physical functioning scores were approximately 1.0 and 1.5 standard deviations below the general population, respectively. Among patients with at least one outpatient visit or one inpatient night in the past year, patients reported a median of six VHA outpatient visits and 14 VHA inpatient nights, respectively.
| Characteristic | N | Weighted % | SE |
|---|---|---|---|
| Age | |||
| 18–34 | 103 | 5 | .6 |
| 35–44 | 249 | 11 | .8 |
| 45–54 | 783 | 33 | 1.2 |
| 55–64 | 769 | 41 | 1.3 |
| ≥65 | 170 | 10 | .8 |
| Male | 1,958 | 96 | .5 |
| Race-ethnicity | |||
| Non-Hispanic black | 522 | 26 | 1.1 |
| Hispanic | 140 | 8 | .7 |
| Non-Hispanic white | 1,257 | 59 | 1.3 |
| Othera | 155 | 7 | .7 |
| Married or living as married | 590 | 31 | 1.2 |
| Education | |||
| Did not complete high school | 143 | 8 | .7 |
| High school graduate or GED | 812 | 40 | 1.3 |
| Some college | 708 | 33 | 1.2 |
| College graduate or beyond | 411 | 20 | 1.0 |
| Employment status | |||
| Employed | 388 | 20 | 1.0 |
| Unemployed | 240 | 12 | .8 |
| Out of labor force | 1,442 | 68 | 1.2 |
| Data missing | 4 | 0 | .1 |
| Annual income | |||
| ≤$15,000 | 996 | 46 | 1.3 |
| $15,001–$30,000 | 562 | 28 | 1.2 |
| $30,001–$60,000 | 333 | 16 | 1.0 |
| >$60,000 | 96 | 5 | .6 |
| Data missing | 87 | 5 | .6 |
| Rural residence | 394 | 18 | 1.0 |
| Service-connected disabilityb | 759 | 37 | 1.3 |
| Diagnostic cohort | |||
| Bipolar disorder | 418 | 7 | .4 |
| Major depressive disorder | 307 | 11 | .6 |
| PTSD | 324 | 26 | 1.3 |
| Schizophrenia | 289 | 8 | .5 |
| Substance use disorder only | 736 | 49 | 1.3 |
| Global rating of care ≥9c | 587 | 36 | 1 |
| M | SE | Median | |
| Functioningd | |||
| Mental health | 40.1 | .4 | 39.6 |
| Physical health | 35.7 | .3 | 34.9 |
| Past year utilizatione | |||
| N of VHA outpatient visits | 19.6 | 1.4 | 5.7 |
| N of VHA inpatient nights | 35.9 | 3.8 | 13.9 |
| Perceived improvementf | 3.2 | .02 | 3.0 |
a
Includes Asian, Native Hawaiian/other Pacific Islander, American Indian/Alaska Native, multiracial, none of these races, and refused/don’t know
b
Defined as having a psychiatric or general medical condition that was incurred or aggravated during active duty
c
Possible response options range from 0 to 10, with higher ratings indicating a more positive rating of care.
d
Possible scores range from 0 to 100, with higher scores indicating better functioning.
e
Among patients with at least one visit or inpatient night
f
Mean score for four items from the Experience of Care and Health Outcomes survey, which assesses improvement in ability to deal with daily problems and social situations, ability to accomplish goals, and symptoms in the past 12 months. Possible scores range from 1, much worse, to 5, much better.
Quality Measure Performance
The performance across the eight quality measures varied widely, ranging from 11% for treatment engagement to 82% for outpatient follow-up within 30 days after an inpatient discharge (Table 2).
| Quality measure | Eligible Na | N | % |
|---|---|---|---|
| Treatment within 14 days of an inpatient or outpatient substance use disorder new treatment episode (treatment initiation) | 1,382 | 243 | 16 |
| Treatment initiation, plus ≥2 encounters within 30 days of the substance use disorder new treatment episode (treatment engagement) | 1,382 | 149 | 11 |
| Any psychotherapy within 4 months of new treatment episode | 645 | 300 | 35 |
| Any psychotherapy | 2,074 | 1,012 | 45 |
| Any psychosocial treatment within 4 months of new treatment episode | 645 | 482 | 54 |
| Any psychosocial treatment | 2,074 | 1,609 | 72 |
| Follow-up within 7 days of inpatient discharge | 304 | 156 | 52 |
| Follow-up within 30 days of inpatient discharge | 304 | 251 | 82 |
a
Only patients with a new treatment episode for a substance use disorder were eligible for assessment of measures that pertained to new treatment episodes.
Multivariate Models
Of the eight quality measures, four were significantly associated with perceived improvement (Table 3). Specifically, high performance on engagement in substance use disorder treatment, receiving any psychotherapy within four months of beginning a new treatment episode, receiving any psychotherapy, and receiving any psychosocial treatment were each positively associated with perceived improvement. An additional quality measure, receiving any psychosocial treatment within four months of beginning a new treatment episode, was significantly associated with perceived improvement, but only before the analyses were adjusted for multiple comparisons.
| Perceived improvementa | Global ratinga,b | |||||||
|---|---|---|---|---|---|---|---|---|
| Quality measure | N | Coeffc | SE | pd | N | OR | 95% CI | pd |
| Treatment within 14 days of an inpatient or outpatient substance use disorder new treatment episode (treatment initiation) | 1,375 | .1 | .07 | .246 | 986 | .99 | .67–1.47 | .956 |
| Treatment initiation, plus ≥2 encounters within 30 days of the substance use disorder new treatment episode (treatment engagement) | 1,375 | .25 | .08 | .006 | 986 | 1.10 | .69–1.77 | .725 |
| Any psychotherapy within 4 months of new treatment episode | 642 | .19 | .07 | .025 | 516 | 1.86 | 1.20–2.89 | .016 |
| Any psychotherapy | 2,063 | .28 | .04 | <.001 | 1,568 | 1.16 | .90–1.50 | .358 |
| Any psychosocial treatment within 4 months of new treatment episode | 642 | .17 | .08 | .087 | 516 | 1.45 | .84–2.50 | .291 |
| Any psychosocial treatment | 2,063 | .37 | .05 | <.001 | 1,568 | .91 | .63–1.31 | .708 |
| Follow-up within 7 days of inpatient discharge | 300 | .12 | .11 | .369 | 277 | 2.89 | 1.52–5.50 | .005 |
| Follow-up within 30 days of inpatient discharge | 300 | –.12 | .16 | .568 | 277 | 5.26 | 2.17–12.74 | .002 |
a
Only patients with a new treatment episode for a substance use disorder were eligible for assessment of measures that pertained to new treatment episodes. Ns are reduced from those reported in Table 2, given that models included only patients with patient perceptions-of-care responses.
b
Only patients who endorsed receiving care from the VA in the past year were asked to provide a global rating of care.
c
Positive coefficients indicate higher patient perceptions of care.
d
All p values are adjusted for a false discovery rate of 5%.
Of the eight quality measures, three measures were significantly associated with the global rating of care, although they varied from those that were significantly associated with perceived improvement (Table 3). Specifically, receiving any psychotherapy within four months of beginning a new treatment episode, receiving a follow-up outpatient visit within seven days of psychiatric inpatient discharge, and receiving a follow-up outpatient visit within 30 days of psychiatric inpatient discharge each were significantly associated with higher global ratings of care.
Discussion
Our results suggest that several process-based quality measures were associated with higher perceptions of care, including higher perceived improvement and higher global rating of care, among patients with substance use disorders, although the two perceptions-of-care measures were not associated with the same quality measures. This relationship suggests that both these process-based quality measures and patient perception of care are valid measures of quality. Global rating of care has been assessed more frequently across health care, including ratings of health plans, providers, and hospitals, given that it is included as an item in all CAHPS measures (25). Given that assessments of patient perceptions have emphasized a global rating of care, we anticipated that quality measures would be associated with the global rating in this population. Yet quality measures were more frequently associated with patient perceptions of improvement. These results suggest that perceived improvement may be an additional patient perception of care outcome that could be useful in assessing and improving the quality of care for patients with substance use disorders.
Four of the eight process-based quality measures evaluated are endorsed by the NQF (an endorsement that signals the rigor of the measures and increases the likelihood of broad implementation), and of those, three were significantly associated with one of the two patient perception measures. Treatment engagement was significantly associated with perceived improvement, but not global rating of care. Although treatment initiation was not associated with either outcome, initiation is often considered to be best understood when assessed alongside treatment engagement because initiating treatment would not be expected to indicate sufficient treatment. Both measures of timely follow-up after inpatient discharge were significantly associated with global rating of care but not with perceived improvement. These results provide additional support for the use of these NQF-endorsed measures and suggest that adherence to these measures may be associated with increased delivery of patient-centered care.
Our study had limitations. Identification of the cohort of patients with substance use disorders included in these analyses and operationalization of the quality measures relied on administrative claims data. Thus substance use disorder diagnoses recorded in the medical record may not accurately represent diagnoses assigned through standardized clinical assessments. Furthermore, the quality measures predominantly focus on the number and timing of visits rather than on more detailed information about care that would be documented in patient medical records.
There were limitations associated with patient perceptions of care, given that these measures have limitations similar to those of other data gathered through patient self-report (42). Patients may have difficulty recalling the care they received, differentially weight the value of positive and negative experiences, differentially weight recent versus distal experiences, or base their perceptions on varying expectations for their care. Patients were asked to report on their behavioral health care, likely leading some patients to include care received for co-occurring psychiatric diagnoses. Furthermore, perceived improvement is a retrospective measure, so it likely differs from clinician-rated improvement measures or patient self-report measures collected during the course of treatment. Patient perspectives of care were assessed after the process of care assessed by the quality measure occurred, and this time lag was lengthy for some patients. This delay may have hampered recall of their care, or other care received more recently could have affected their perceptions. Therefore, it would be important to assess these relationships in a new sample that assesses patient perceptions more proximal to the care being assessed. It is notable, however, that several significant associations were found between quality of care and patient perceptions despite this gap.
Finally, these correlational analyses found significant associations between quality measures and patient perceptions of care, but evidence of this relationship should not be interpreted as a unidimensional causal link (in other words, that increased quality causes increased patient perceptions of care). For example, it is also possible that patients who were satisfied with their care or who perceived their treatment as more helpful were more conscientious about their care, continued their care more consistently, and accessed further resources. Simply put, if patients perceive a benefit of treatment, they may be more likely to receive additional treatment. It is possible that there is a bidirectional relationship between treatment quality and patient perceptions of care.
Conclusions
Some quality measures assessing care for patients with substance use disorders were associated with patient perceptions of care. Results provide additional support for these quality measures and suggest that patient perceptions of care are an important outcome in assessing care.
Acknowledgments
The authors thank Tiffany Hruby for her assistance in preparation of the manuscript.
Footnote
The opinions expressed here are the authors’ and do not necessarily represent the views of the U.S. Department of Veterans Affairs or any other entity of the U.S. government.
References
1.
Institute of Medicine: Psychosocial Interventions for Mental and Substance Use Disorders: A Framework for Establishing Evidence-Based Standards. Washington, DC, National Academies Press, 2015
2.
Harris AH: The primitive state of quality measures in addiction treatment and their application. Addiction 111:195–196, 2016
3.
Patel MM, Brown JD, Croake S, et al: The current state of behavioral health quality measures: where are the gaps? Psychiatric Services 66:865–871, 2015
4.
Watkins KE, Farmer CM, De Vries D, et al: The Affordable Care Act: an opportunity for improving care for substance use disorders? Psychiatric Services 66:310–312, 2015
5.
Pincus HA, Scholle SH, Spaeth-Rublee B, et al: Quality measures for mental health and substance use: gaps, opportunities, and challenges. Health Affairs 35:1000–1008, 2016
6.
Garnick DW, Horgan CM, Acevedo A, et al: Performance measures for substance use disorders—what research is needed? Addiction Science and Clinical Practice 7:18, 2012
7.
Kilbourne AM, Fullerton C, Dausey D, et al: A framework for measuring quality and promoting accountability across silos: the case of mental disorders and co-occurring conditions. Quality and Safety in Health Care 19:113–116, 2010
8.
Kilbourne AM, Keyser D, Pincus HA: Challenges and opportunities in measuring the quality of mental health care. Canadian Journal of Psychiatry 55:549–557, 2010
9.
Thomas CP, Garnick DW, Horgan CM, et al: Advancing performance measures for use of medications in substance abuse treatment. Journal of Substance Abuse Treatment 40:35–43, 2011
10.
Pincus HA, Spaeth-Rublee B, Watkins KE: Analysis and commentary: the case for measuring quality in mental health and substance abuse care. Health Affairs 30:730–736, 2011
11.
Garnick DW, Horgan CM, Acevedo A, et al: Criminal justice outcomes after engagement in outpatient substance abuse treatment. Journal of Substance Abuse Treatment 46:295–305, 2014
12.
Garnick DW, Lee MT, O’Brien PL, et al: The Washington Circle engagement performance measures’ association with adolescent treatment outcomes. Drug and Alcohol Dependence 124:250–258, 2012
13.
Harris AH, Ellerbe L, Phelps TE, et al: Examining the specification validity of the HEDIS quality measures for substance use disorders. Journal of Substance Abuse Treatment 53:16–21, 2015
14.
Harris AH, Gupta S, Bowe T, et al: Predictive validity of two process-of-care quality measures for residential substance use disorder treatment. Addiction Science and Clinical Practice 10:22, 2015
15.
Harris AH, Humphreys K, Bowe T, et al: Measuring the quality of substance use disorder treatment: evaluating the validity of the Department of Veterans Affairs continuity of care performance measure. Journal of Substance Abuse Treatment 36:294–305, 2009
16.
Harris AH, Humphreys K, Bowe T, et al: Does meeting the HEDIS substance abuse treatment engagement criterion predict patient outcomes? Journal of Behavioral Health Services and Research 37:25–39, 2010
17.
Harris AH, Humphreys K, Finney JW: Veterans Affairs facility performance on Washington Circle indicators and case mix–adjusted effectiveness. Journal of Substance Abuse Treatment 33:333–339, 2007
18.
Lee MT, Horgan CM, Garnick DW, et al: A performance measure for continuity of care after detoxification: relationship with outcomes. Journal of Substance Abuse Treatment 47:130–139, 2014
19.
Schmidt EM, Gupta S, Bowe T, et al: Predictive validity of a quality measure for intensive substance use disorder treatment. Substance Abuse (Epub ahead of print, July 19, 2016)
20.
Strickler GK, Reif S, Horgan CM, et al: The relationship between substance abuse performance measures and mutual help group participation after treatment. Alcoholism Treatment Quarterly 30:190–210, 2012
21.
Dunigan R, Acevedo A, Campbell K, et al: Engagement in outpatient substance abuse treatment and employment outcomes. Journal of Behavioral Health Services and Research 41:20–36, 2014
22.
Cella D, Hahn EA, Jensen SE, et al: Patient-Reported Outcomes in Performance Measurement. Research Triangle Park, NC, Research Triangle Institute, 2015
23.
Institute of Medicine: Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC, National Academies Press, 2001
24.
Epstein RM, Street RL Jr: The values and value of patient-centered care. Annals of Family Medicine 9:100–103, 2011
25.
The CAHPS Program. Rockville, MD, Agency for Healthcare Research and Quality, 2016
26.
Beebe TJ, Harrison PA, McRae JA Jr, et al: Evaluating behavioral health services in Minnesota’s Medicaid population using the Experience of Care and Health Outcomes (ECHO) Survey. Journal of Health Care for the Poor and Underserved 14:608–621, 2003
27.
Anhang Price R, Elliott MN, Zaslavsky AM, et al: Examining the role of patient experience surveys in measuring health care quality. Medical Care Research and Review 71:522–554, 2014
28.
Blonigen DM, Bui L, Harris AH, et al: Perceptions of behavioral health care among veterans with substance use disorders: results from a national evaluation of mental health services in the Veterans Health Administration. Journal of Substance Abuse Treatment 47:122–129, 2014
29.
Hepner KA, Paddock SM, Watkins KE, et al: Veterans’ perceptions of behavioral health care in the Veterans Health Administration: a national survey. Psychiatric Services 65:988–996, 2014
30.
Etingen B, Miskevics S, LaVela SL: Assessing the associations of patient-reported perceptions of patient-centered care as supplemental measures of health care quality in VA. Journal of General Internal Medicine 31(suppl 1):10–20, 2016
31.
Watkins KE, Pincus HA, Paddock S, et al: Care for veterans with mental and substance use disorders: good performance, but room to improve on many measures. Health Affairs 30:2194–2203, 2011
32.
Watkins KE, Pincus HA: Veterans Health Administration Mental Health Program Evaluation: Capstone Report. Santa Monica, CA, RAND, 2011
33.
Hepner KA, Paddock SM, Watkins KE, et al: Program Evaluation of VHA Mental Health Services: Client Survey Report. Alexandria, VA, Altarum Institute and RAND-University of Pittsburgh Health Institute, 2010
34.
Iqbal SU, Rogers W, Selim A, et al: The Veterans RAND 12-Item Health Survey (VR-12): What It Is and How It Is Used. Bedford, MA, Veterans Administration Medical Center, Center for Health Quality, Outcomes, and Boston University School of Public Health, Economic Research, and Center for the Assessment of Pharmaceutical Practices, 2007
35.
Garnick DW, Lee MT, Horgan CM, et al: Adapting Washington Circle performance measures for public sector substance abuse treatment systems. Journal of Substance Abuse Treatment 36:265–277, 2009
36.
Daniels AS, Shaul JA, Greenberg P, et al: The experience of care and health outcomes survey (ECHO): a consumer survey to collect ratings of behavioral health care treatment, outcomes and plans; in The Use of Psychological Testing for Treatment Planning and Outcomes Assessment Instruments for Adults, 3rd ed. Edited by Maruish ME. Fairfax, VA, Erlbaum, 2004
37.
Read About the ECHO Survey. Rockville, MD, Agency for Healthcare Research and Quality, 2016. https://www.ahrq.gov/cahps/surveys-guidance/echo/about/index.html. Accessed March 22, 2017
38.
Cochran W (ed): Sampling Techniques, 3rd ed. London, Wiley, 1977
39.
Little RJA: Survey nonresponse adjustments for estimates of means. International Statistical Review 54:139–157, 1986
40.
White H: A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica 48:817–838, 1980
41.
Benjamini Y, Hochberg Y: Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society Series B 57:289–300, 1995
42.
Kupfer JM, Bond EU: Patient satisfaction and patient-centered care: necessary but not equal. JAMA 308:139–140, 2012
Information & Authors
Information
Published In
Cover: Still Life of Fruit, anonymous, circa 1865. Gift of Edgar William and Bernice Chrysler Garbisch, National Gallery of Art, Washington, D.C.
History
Received: 24 October 2016
Revision received: 27 March 2017
Accepted: 28 April 2017
Published online: 3 July 2017
Published in print: November 01, 2017
Keywords
Authors
Competing Interests
The authors report no financial relationships with commercial interests.
Funding Information
National Institute on Drug Abuse10.13039/100000026: R01DA033953
This project was funded by the National Institute on Drug Abuse (R01DA033953).Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBGet Access
Login options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).