Development, Testing, and Dissemination of a Public-Facing Tool to Help Consumers Find Higher-Quality Addiction Treatment
Abstract
Objective:
The authors aimed to describe the development and testing of quality measures included in a public-facing addiction treatment facility search engine.
Methods:
An addiction treatment facility survey was created that queried providers in six U.S. states about whether they offered the services and used the processes identified by federal agencies and nonprofit organizations as signs of higher-quality addiction treatment. Four insurance claims–based quality measures were created to capture the percentage of a provider’s patients with opioid use disorder receiving opioid use disorder medications, who filled prescriptions for such medication for at least 180 days, who received follow-up care after treatment for substance use disorder in inpatient or residential settings, or who had a substance use disorder–related hospitalization or emergency department visit. A patient experience-of-care survey captured patients’ perceptions of the quality of the addiction treatment. The project was undertaken from November 2018 through July 2020.
Results:
The authors tested the measures by using 1,245 facility surveys, 7,970 patients’ experience-of-care surveys, and four claims-based measures submitted by 129, 136, 283, and 408 addiction treatment providers. Statistical testing demonstrated that the quality measures were reliable and valid. The quality measure scores varied among providers, capturing a wide performance range. A public website containing quality measures launched in July 2020 in the six states and has been accessed by thousands of consumers.
Conclusions:
This study developed valid, reliable, and useful addiction treatment quality measures. Dissemination of these measures may help consumers select among providers and help providers, policy makers, and payers improve quality.
HIGHLIGHTS
•
The authors describe the development and testing of a public-facing addiction treatment provider search engine with quality metrics that reflect the signs of higher-quality addiction treatment as described by the National Institutes of Health.
•
They found that the quality measures were reliable and valid and varied significantly among providers.
•
A website containing the quality measures was publicly launched in July 2020 in six U.S. states and may help improve the quality of addiction treatment.
Individuals with substance use disorders and their families are often unsure how to find high-quality addiction treatment, resulting in patients receiving inappropriate, low-quality care or giving up trying to find addiction treatment altogether (1–3). The federal government hosts online search engines to help consumers find high-quality treatment for most provider types, including hospitals, nursing homes, psychiatric hospitals, inpatient rehabilitation facilities, home health care, hospice care, long-term-care hospitals, dialysis centers, and physicians (4). For example, consumers can use the Centers for Medicare and Medicaid Services (CMS) Medicare Compare website (at https://www.medicare.gov/care-compare) to identify nursing homes in their region and to compare their quality along various dimensions, such as staff hours per resident, rates of health citations and complaints, and rehospitalization rates. The federal government has not implemented a similar system for addiction treatment.
Although the Substance Abuse and Mental Health Services Administration (SAMHSA) and the National Institute on Alcohol Abuse and Alcoholism (NIAAA) make available online treatment locators, their websites give consumers only general criteria for identifying higher-quality addiction treatment (5, 6). They do not indicate whether providers included in the search engine meet these criteria. This is a missed opportunity. Studies have shown that public reporting of provider quality measures stimulates higher quality by allowing consumers to shop for quality dimensions, encouraging providers to compete on quality, and stimulating providers to learn from each other about improving quality (7–10).
In this article, we describe the development and testing of quality metrics for a public-facing search engine for addiction treatment. The system was developed and launched in six U.S. states (Massachusetts, New York, Delaware, West Virginia, North Carolina, and Louisiana).
Methods
We developed addiction quality measures by using three data sources: treatment facility surveys, insurance claims, and patient experience-of-care surveys. The project began in November 2018. The quality measures were made available to participating states, health plans, providers, and the public in July 2020. The project was reviewed by the RTI Institutional Review Board and determined not to be human subjects research.
Treatment Facility Survey
An addiction treatment facility survey was fielded that queried 2,441 providers in the six states about whether they offered the services and used processes identified by the National Institute on Drug Abuse, NIAAA, SAMHSA, and nonprofit organizations as signs of higher-quality treatment (11–15). These signs include rapid access to treatment when needed, use of evidence-based behavioral health therapies, access to medications for addiction, accreditation and appropriately credentialed staff, attention to mental health and general medical health needs, a personalized treatment plan for each patient, engagement in treatment for a period long enough to be effective, and the provision of recovery support services.
Addiction providers were identified with SAMHSA’s Inventory of Behavioral Health Services and input from state substance use disorder agencies (16). Data were collected from October 14, 2019, through January 31, 2020, via a Web-based survey. Across the six pilot states, 1,245 (out of 2,441; 51% participation rate) specialty substance use disorder treatment facilities completed the survey. Participation rates varied from 74% in New York to 28% in North Carolina.
We implemented validity checks to enhance the accuracy of survey responses. Facilities were required to upload documentation supporting their responses to specific questions, such as a copy of the facility’s license and a screenshot of an electronic medical record. Overall, 85% of providers uploaded a copy of the facility license, and 84% provided screenshots of their electronic medical record. A senior leader at the facility was required to sign off on the accuracy of the survey responses (96% attested to their accuracy). Inconsistent results triggered additional verification processes, including direct outreach to a treatment facility to confirm data accuracy and cross-checking against other data sources. For a random 5% of facilities, we compared responses to our survey with those on the SAMHSA National Survey of Specialty Substance Abuse Treatment Systems. Specifically, we compared responses to the questions about providing medications and recovery support services. Responses matched 94% of the time for methadone, 77% for buprenorphine, and 72% for naltrexone. The SAMHSA treatment locator data were collected approximately 1 year before our survey. Facilities may have added buprenorphine and naltrexone to their treatment array over the time between the two surveys.
Insurance Claims Data
As of 2018, the National Quality Forum (NQF) had endorsed the following quality measures for substance use disorder treatment for use in Medicaid programs and health plans: patients with opioid use disorder receiving medications for opioid use disorder (NQF 3400), continuity of medication use for opioid use disorder among patients with opioid use disorder (NQF 3175), and follow-up within 7 and 30 days after treatment for substance use disorder in inpatient or residential settings (NQF 3453). We respecified these measures to apply at the provider level rather than the Medicaid or health plan level. We also created a new provider-level claims-based measure that captured the percentage of a facility’s patients with a substance use disorder diagnosis who had a substance use disorder–related hospitalization or emergency department (ED) visit within 14, 30, 90, or 180 days from their second encounter over a 14-day period with the provider or after discharge from a 14-day inpatient stay.
We worked with one state Medicaid program and one commercial health plan to test and refine the methods for calculating the claims-based measures. We then provided final programming specifications to the other participating state Medicaid programs and commercial insurers. Four state Medicaid programs (Massachusetts, New York, Delaware, and West Virginia) and four private health insurance plans submitted the provider-level claims-based measures.
After states and commercial insurers submitted their metrics, we performed quality checks, such as ensuring that exclusion criteria were applied correctly and that final counts of beneficiaries, discharges, and medication episodes were consistent with the programming procedures. We then conducted reliability and validity testing by using the approaches recommended by NQF and CMS (17).
Patient Experience-of-Care Survey
The patient experience-of-care survey consisted of seven close-ended questions drawn from previously validated surveys. Most questions were sourced from the Consumer Assessment of Healthcare Providers and Systems Experience of Care and Health Outcomes Survey. We also included one open-ended question that asked for feedback on what the facility was doing right and where it could improve.
Because of heightened concerns about patient privacy in the context of addiction treatment, we developed a specific patient survey approach that involved no exchange of patient identifiers. We disseminated a letter unique to each addiction treatment facility and asked each provider to distribute it to all patients. The letter invited patients to go to a secure website and fill out the patient experience-of-care survey. Each letter had a personal identification number specific to that facility, which ensured that patient responses were linked to the correct facility.
Between October 14, 2019, and February 28, 2020, a total of 7,970 patients responded to the survey. Of the facilities invited (N=2,441), 16% (N=389) had at least one patient who completed an experience-of-care survey, and the number of facilities with at least one response to each item on the survey ranged from 370 to 386. Five percent (N=122) had ≥20 patient survey responses. Reliability testing of pilot data indicated that a minimum of 20 patient survey responses was needed for reliable comparison of facilities on the basis of patient experience-of-care survey responses.
Results
The addiction treatment quality metrics described above were made available in a public-facing search engine (https://treatmentatlas.org). Table 1 shows the variation in some treatment facility characteristics identified as signaling higher-quality treatment. We measured whether facilities offered rapid treatment access when needed by asking them to report on their hours of operation and whether they offered same-day access. Research has shown that same-day access can substantially reduce wait times (18–21). Across the six states, 77% of facilities said that they offered same-day access. Another sign of higher quality is a personalized treatment plan for each patient. We captured this in the facility survey by asking whether the facility conducted a comprehensive biopsychosocial assessment at intake as indicated by whether they asked patients about the six biopsychosocial dimensions of care defined by the American Society of Addiction Medicine (ASAM). Among the facilities responding, 81% said they conduct comprehensive biopsychosocial assessments.
| Service | N | % |
|---|---|---|
| Same-day access to outpatient care | 719b | 77 |
| Comprehensive biopsychosocial assessment at intake | 1,005 | 81 |
| Opioid use disorder medications onsite | 877 | 70 |
| Alcohol use disorder medications onsite | 750 | 60 |
| Availability of any identified evidence-based psychosocial treatments | 1,176 | 94 |
| Cognitive-behavioral therapy | 1,110 | 89 |
| Contingency management | 402 | 32 |
| Primary health care onsite | 247 | 20 |
| Mental health care onsite | 848 | 68 |
| Psychiatric medications onsite | 872 | 70 |
| Assistance in obtaining social services | 961 | 77 |
| Case management | 944 | 76 |
| Legal aid | 157 | 13 |
| Child care for patients’ children | 109 | 9 |
a
The states were Massachusetts, New York, Delaware, West Virginia, North Carolina, and Louisiana.
b
Restricted to facilities that offered outpatient services (N=937).
Other indicators of higher care quality are offering evidence-based behavioral health therapy and addiction medications. Overall, 70% of facilities said they offered opioid use disorder medications, and 60% said they offered alcohol use disorder medication (Table 1). NIAAA lists several evidence-based, effective behavioral health therapies, and we asked whether the facilities offered any of these (22). Among the facilities surveyed, 94% offered at least one evidence-based behavioral intervention. Cognitive-behavioral therapy was offered by 89% of the facilities, and contingency management was offered by 32%.
The federal agencies highlight that higher-quality facilities also attend to patients’ general medical and mental health needs. Most facilities reported being able to provide mental health care onsite (68%) and to offer psychiatric medications onsite (70%). In contrast, only 20% of facilities offered primary care onsite, which might have limited these facilities’ abilities to screen for and manage medical comorbid conditions that co-occur with substance misuse. Finally, federal agencies emphasize the importance of providing recovery supports. The survey asked whether the facility offered different types of recovery supports. The most frequently offered recovery supports were assistance obtaining social services (77%) and case management (76%). The least common were legal aid and child care.
Claims-Based Quality Measures
We conducted reliability tests to determine whether the claims measures could distinguish well-performing providers from poorly performing providers—that is, whether the differences in the quality measures among providers were due to provider-specific differences and not due to random noise. We conducted three types of reliability tests: signal-to-noise ratio and effect size tests, equality-of-distribution tests, and Adams’s rho. All tests indicated that the measures were reliable. The face validity of the measures was determined by a literature review (e.g., of research on the association between postdischarge follow-up and positive outcomes). We also tested the measures’ convergent validity by calculating the correlation between related quality concepts. We found that higher provider rates of opioid use disorder prescriptions among patients with opioid use disorder were associated with lower provider-level rates of substance use disorder hospitalizations, fewer ED visits, and higher follow-up visit rates.
Figure 1 shows the median and range for the four claims-based measures for the Medicaid population. Only providers with at least 10 patients who met the measures’ denominator requirements were included to ensure patient privacy and measure reliability. In the median (“typical”) facility, 29% (N=408 facilities) of patients with an opioid use disorder diagnosis filled a prescription for an opioid use disorder medication within 30 days of visiting that provider. The bottom and top quartile providers had rates of ≤18% and ≥67%, respectively. The 6-month (180-day) adherence rate for the median provider was 23% (N=283 facilities), meaning that about a quarter of that provider’s patients stayed on the opioid use disorder medication for at least 6 months. This metric ranged from ≤18% in the lowest quartile to ≥67% in the highest quartile.
FIGURE 1. Box plots indicating median and range on four claims-based measures for specialty substance use disorder providers, calculated with Medicaid claims dataa
aED, emergency department; MOUD, medication for opioid use disorder; SUD, substance use disorder.
Among the inpatient or residential specialty substance use disorder providers, the median facility follow-up rate was 40% (N=129 facilities), with a rate of ≤30% in the lowest quartile and ≥47% in the highest quartile. In terms of hospitalizations and ED visits for substance use disorders or overdoses, the median facility had 8% of patients admitted within 30 days of their first encounter (N=136 facilities). This metric ranged from ≤5% to ≥15% in the lowest and highest quartiles, respectively.
Patient Experience-of-Care Measures
Figure 2 shows the median and range for the patient experience-of-care measures. The survey responses that allowed for four response possibilities (i.e., never, sometimes, usually, always) were top coded, indicating the percentage of patients at the facility who gave the best possible answer (i.e., always).
The ratings showed significant variation among facilities. In the median facility, 100% (N=371 facilities) of patients said they were given information about different treatment options (interquartile range [IQR]=11%). In the median facility, 90% (N=374 facilities) of patients reported that staff at the program discussed including family or friends in treatment (IQR=27%), 89% (N=386 facilities) said that they had been helped a lot by the facility (IQR=25%), 86% (N=372 facilities) reported that the staff always showed respect for what they had to say (IQR=35%), and 64% (N=370 facilities) indicated that they always saw staff from the treatment program as soon as they wanted (IQR=39%).
In terms of day-to-day functioning, in the median facility, 70% (N=376 facilities) of patients said their ability to deal with their daily problems was much better than before treatment (IQR=43%). In the median facility, 60% (N=382 facilities) of patients rated the facility as 9 or 10 on a scale from 1 to 10 (with 1 indicating worst and 10 indicating best; IQR=50%).
Discussion
The results of this quality measurement development project indicate that it is feasible to collect and publicly report reliable and valid addiction treatment provider quality measures. Over 5 months, survey data were received from 1,245 addiction treatment facilities and 7,970 patients across six states. Four state Medicaid programs submitted the claims-based quality measures. The quality measures varied significantly among providers, indicating that these measures could help consumers select providers.
Our results revealed several lessons for future efforts aimed at measuring addiction treatment quality. A limitation of the project was that the rate of participation of facilities in the facility survey varied from 74% to 28% among the six states. An adequate number of patient experience-of-care surveys (i.e., 20 surveys) were received for 5% of facilities. Participation in quality measurement systems usually grows over time as providers become more comfortable with the reporting requirements and realize the importance of participation to their reputation.
Federal quality reporting systems have the advantage over privately sponsored systems in that they can compel provider participation as a condition for receiving Medicare or Medicaid payments or contracts. The federal government may also have more resources to keep the information collected up to date. Moreover, in contrast to this project, federal and state governments have greater latitude to disseminate patient experience-of-care surveys directly to patients by using contact information collected as part of Medicaid and Medicare, which can facilitate more responses from patients.
This study established the feasibility of creating valid and reliable provider-level claims measures. When we began the project, the only NQF-endorsed, claims-based addiction quality measures were for use at the health plan or state Medicaid program levels. Our provider-level measures could help Medicaid programs uncover the root causes of performance lags. For example, hospitals and residential providers with low discharge follow-up rates may need training in discharge planning and assistance in identifying appropriate and available outpatient programs. The claims-based measures were limited to only four domains of quality of care. Ideally, the claims measures would expand to capture additional evidence-based interventions such as the delivery of contingency management and cognitive-behavioral therapy (assuming procedure codes exist to capture these interventions). Moreover, another limitation was that the measure of substance use disorder–related hospitalizations and ED visits was not risk adjusted. Risk adjustment could be applied in future iterations.
The federal government’s sponsorship and support of public-facing quality measures began more than two decades ago with Nursing Home Compare in 1998. Over time, CMS has added new providers and measures, refined existing measures, and improved how the data are displayed and communicated. Their effort has resulted in meaningful improvements in the quality of care.
Conclusions
This article describes a successful effort in six U.S. states to develop and disseminate addiction treatment provider quality metrics. Despite this success, we remain concerned by the lack of federal leadership in the dissemination of addiction treatment quality metrics. The federal government invests tens of millions of dollars each year in quality measurement targeted to most health care settings. It disseminates public-facing, consumer-friendly quality metrics on thousands of providers. The quality of addiction treatment would benefit from being incorporated into federal quality measurement programs.
References
1.
Grassley at Hearing on the One-Year Anniversary of Landmark Anti-Opioid Legislation. Washington, DC, US Senate Committee on Finance; 2019. www.finance.senate.gov/chairmans-news/grassley-at-hearing-on-one-year-anniversary-of-landmark-anti-opioid-legislation. Accessed Sep 2, 2020
2.
Armstrong D, Allen E: Desperate for addiction treatment, patients are pawns in lucrative insurance fraud scheme. Boston Globe. July 7, 2017. www.statnews.com/2017/07/07/opioid-insurance-fraud. Accessed Sep 2, 2020
3.
Park-Lee E, Lipari RN, Hedden SL, et al: Receipt of Services for Substance Use and Mental Health Issues Among Adults: Results From the 2016 National Survey on Drug Use and Health. Rockville, MD, Substance Abuse and Mental Health Services Administration, 2017. www.samhsa.gov/data/report/receipt-services-substance-use-and-mental-health-issues-among-adults-results-2016-national. Accessed Sep 2, 2020
4.
Find and Compare Nursing Homes, Hospitals, and Other Providers Near You. Baltimore, Centers for Medicare and Medicaid Services, n.d. www.medicare.gov/care-compare/#search. Accessed Aug 1, 2020
5.
Searching for Treatment Options. Rockville, MD, Substance Abuse and Mental Health Services Administration, n.d. https://findtreatment.gov/results. Accessed Sept 13, 2020
6.
Find Your Way to Quality Alcohol Treatment. Bethesda, MD, National Institute on Alcohol Abuse and Alcoholism, n.d. https://alcoholtreatment.niaaa.nih.gov. Accessed Sept 13, 2020
7.
Castle NG, Engberg J, Liu D: Have Nursing Home Compare quality measure scores changed over time in response to competition? Qual Saf Health Care 2007; 16:185–191
8.
Ivers N, Jamtvedt G, Flottorp S, et al: Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012; 6:CD000259
9.
Werner R, Stuart E, Polsky D: Public reporting drove quality gains at nursing homes. Health Aff 2010; 29:1706–1713
10.
2018 National Impact Assessment of the Centers for Medicare & Medicaid Services (CMS) Quality Measures Report. Baltimore, Centers for Medicare & Medicaid Services, 2018. www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/QualityMeasures/Downloads/2018-Impact-Assessment-Report.pdf
11.
Shatterproof National Principles of Care. Norwalk, CT, Shatterproof, n.d. www.shatterproof.org/shatterproof-national-principles-care. Accessed Aug 31, 2020
12.
Principles of Drug Addiction Treatment: A Research-Based Guide, 3rd ed. Bethesda, MD, National Institute on Drug Abuse, 2018
13.
Finding Quality Treatment for Substance Use Disorders. Rockville, MD, Substance Abuse and Mental Health Services Administration, 2018. store.samhsa.gov/product/Finding-Quality-Treatment-for-Substance-Use-Disorders/PEP18-TREATMENT-LOC. Accessed December 1, 2019
14.
Step 3—Choose Quality Care. Bethesda, MD, National Institute on Alcohol Abuse and Alcoholism, 2018. alcoholtreatment.niaaa.nih.gov/how-to-find-alcohol-treatment/how-to-spot-quality-treatment. Accessed May 1, 2020
15.
Mark TL, Dowd WN, Council CL. Tracking the Quality of Addiction Treatment Over Time and Across States: Using the Federal Government’s “Signs” of Higher Quality. Research Triangle Park, NC, RTI Press; 2020. https://www.rti.org/rti-press-publication/addiction-treatment-quality. Accessed June 7, 2021
16.
Inventory of Behavioral Health Services (I-BHS) Online State User Manual 5.0. Arlington, VA, Eagle Technologies, Inc., 2017. wwwdasis.samhsa.gov/dasis2/manuals/IBHS_5.0_State_User_Manual_final.pdf
17.
Guidance for Measure Testing and Evaluating Scientific Acceptability of Measure Properties. Washington, DC, National Quality Forum, 2011. www.qualityforum.org/Publications/2011/01/Measure_Testing_Task_Force.aspx. Accessed Aug 2020
18.
Festinger DS, Lamb RJ, Kirby KC, et al: The accelerated intake: a method for increasing initial attendance to outpatient cocaine treatment. J Appl Behav Anal 1996; 29:387–389
19.
Sigmon SC, Ochalek TA, Meyer AC, et al: Interim buprenorphine vs waiting list for opioid dependence. N Engl J Med 2016; 375:2504–2505
20.
Hoffman KA, Ford JH, Tillotson CJ, et al: Days to treatment and early retention among patients in treatment for alcohol and drug disorders. Addict Behav 2011; 36:643–647
21.
Madden LM, Farnum SO, Eggert KF, et al: An investigation of an open-access model for scaling up methadone maintenance treatment. Addiction 2018; 113:1450–1458
22.
Searching for Alcohol Treatment: What Evidence-Based Behavioral Treatments Are Available for Treating AUD. Bethesda, MD, National Institute on Alcohol Abuse and Alcoholism, n.d. https://alcoholtreatment.niaaa.nih.gov/FAQs-searching-alcohol-treatment#topic-what-behavior-treatments-for-available-for-aud
Information & Authors
Information
Published In
History
Received: 4 November 2020
Revision received: 11 April 2021
Accepted: 7 May 2021
Published online: 20 July 2021
Published in print: March 01, 2022
Keywords
Authors
Competing Interests
The authors report no financial relationships with commercial interests.
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBLogin options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).