A Non‐Randomized Trial of In‐Person Versus Text/Telephone Screening, Brief Intervention and Referral to Treatment for Pregnant and Postpartum Women
Publication: Psychiatric Research and Clinical Practice
Abstract
Background
Systems of care that improve mental health and substance use disorder Screening, Brief Intervention and Referral to Treatment (SBIRT) for pregnant and postpartum women are needed.
Aims
The aim of this study is to determine if women receiving prenatal care from January 2020 to April 2021 are more likely to be screened, screen positive, be referred for treatment and attend treatment with technology facilitated SBIRT, compared to women receiving prenatal care and in‐person SBIRT January 2017 to December 2019.
Materials & Methods
Technology facilitated SBIRT, designated Listening to Women (LTW), includes text message‐based screening, phone‐based brief intervention, and referral to treatment by a remote care coordinator. A total of 3535 pregnant and postpartum women were included in the quasi‐experimental study and data were collected via text message and Electronic Health Record.
Results
In‐person SBIRT was completed by 65.2% (1947/2988) of women while 98.9% (547/553) of women approached agreed to take part in LTW and 71.9% (393/547) completed SBIRT via LTW. After controlling for potentially confounding variables, women enrolled in LTW were significantly more likely to be screened (relative risk [RR]: 1.10, 95% CI 1.03–1.16), screen positive (RR 1.91, 95% CI 1.72–2.10), referred to treatment (RR 1.55, 95% CI 1.43–1.69) and receive treatment (RR 4.95, 95% CI 3.93–6.23), compared to women receiving in‐person SBIRT. Black women enrolled in LTW were significantly more likely to screen positive (RR 1.65, 95% CI 1.35–2.01), be referred to treatment (RR 1.54, 95% CI 1.35–1.76) and attend treatment (RR 5.49, 95% CI 3.69–8.17), compared to Black women receiving in‐person SBIRT.
Discussion
LTW appears to increase the proportion of pregnant and postpartum women receiving key elements of SBIRT.
Highlights
•
Technology‐based solutions have the potential to overcome many of the key patient, provider and system‐level barriers that inhibit widespread adoption of evidence‐based screening and referral recommendations for Perinatal Mood and Anxiety Disorders (PMADs), Perinatal Substance Use Disorders (PSUDs), and Intimate Partner Violence (IPV).
•
Listening to Women (LTW), a text message‐based screening, phone‐based brief intervention, and referral to treatment by a remote care coordinator, appears to increase the proportion of pregnant and postpartum women being screened, screening positive, referred to treatment and attending treatment, compared to in‐person Screening, Brief Intervention and Referral to Treatment (SBIRT) during prenatal care.
•
Racial disparities in attendance to treatment identified with in‐person SBIRT were not present with LTW; Black women were more likely to attend treatment with LTW compared to in‐person SBIRT and Black and White women were equally as likely to attend treatment with LTW.
Perinatal Mood and Anxiety Disorders (PMADs), Perinatal Substance Use Disorders (PSUDs) and Intimate Partner Violence (IPV) are common during pregnancy and the postpartum year (1, 2, 3), and have been strongly associated with significant morbidity and mortality for both women and their children (4, 5). Multiple professional organizations have endorsed screening for PMADs, PSUDs and IPV during pregnancy and the postpartum period using standardized, validated screening tools, followed by assessment and referral to treatment (6). For those screening positive for substance use, a brief intervention using in‐person motivational interviewing to support behavioral change and referral to treatment (e.g., Screening, Brief Intervention and Referral to Treatment [SBIRT]) is recommended (7, 8). Despite strong evidence supporting this approach to screening and intervention among patients in primary and obstetrics care, particularly for alcohol abuse, other substance use disorders, depression, and other mental health conditions (9, 10, 11), it has yet to be fully implemented and utilized in obstetric settings: only one in five pregnant and one in eight postpartum women are screened for PMADs (12) and PSUDs is often undetected (10). Among those detected, fewer than 20% of women attend treatment for PMADs (9) or PSUDs (13), despite the established efficacy of treatments for these diseases (14, 15). Similarly, only 10–20% of peripartum women are screened for IPV (16). Unlike PMADs and PSUDs, however there is still a large knowledge gap in optimal interventions for IPV following a positive screen (17).
Key patient, provider and system‐level barriers exist that inhibit widespread adoption of evidence‐based screening and referral recommendations during pregnancy and the postpartum year (18, 19). The breadth of recommended screenings, and the depth of knowledge needed to adequately assess and then appropriately connect women with treatment resources is, in general, beyond the capability of our current health system (8). Insufficient time, unfamiliarity with screening tools, lack of knowledge about PMADs, PSUDs and IPV, combined with the limited availability of accessible treatment services and/or resources are reasons cited for not adopting or fully utilizing screening and referral practices during prenatal care (18, 19). System or structural barriers can include lack of reimbursement for SBIRT efforts and/or integration of SBIRT in the Electronic Health Record (EHR). In addition, individual patient factors such as lack of knowledge and stigma concerns are potent barriers to mental health and/or substance use disorder treatment (18, 19).
Innovations that can overcome barriers to universal adoption and effective utilization of currently available evidence‐based practices for screening and referral to treatment for PMADs, PSUDs and IPV (8) are critically needed. Text‐messaging or computer‐based assessments can result in a greater proportion of individuals endorsing IPV (16), or psychological distress compared to face‐to‐face assessments (20, 21). In addition, a remote care coordinator with a master's in clinical social work may be more adept at employing a brief intervention for PMADs, PSUDs or IPV compared to prenatal care staff, resulting in more women attending treatment. The aim of this study is to determine if a program called Listening to Women (LTW), a text message‐based screening, phone‐based brief intervention, and referral to treatment by a remote care coordinator with a master's in clinical social work increases the proportion of women being screened, screening positive, referred to treatment and attending treatment, compared to in‐person SBIRT. Given the significant racial disparities in maternal mental health and substance use disorder treatment, with Black women significantly less likely to receive treatment for these disorders as compared to White women (22, 23), we examined differences in screening, screening positive, referred to treatment and attending treatment by race within each screening method (LTW or in‐person SBIRT) and by screening method (LTW vs. in‐person SBIRT) to determine if racial disparities exist within a given screening method, and/or between screening methods, respectively.
METHODS
Participants and Setting
This quasi‐experimental study compared two cohorts of pregnant and postpartum women who received SBIRT in the same prenatal care clinic. Cohorts differed in the timing and method by which they received SBIRT. The first cohort received in‐person SBIRT during prenatal care January 1, 2017 to December 31, 2019. The second cohort received LTW during prenatal care January 1, 2020 to April 5, 2021. The prenatal care took place in a large outpatient obstetrics and gynecology clinic within a large academic medical center located in the southeast region of the United States. This outpatient clinic provides prenatal care for approximately 1200–1500 pregnant patients per year and is one of five clinics affiliated within a single hospital system. All five clinics are in the same county and are less than 30 min apart from each other. The clinic where the study took place reflects a very similar patient demographic and type of practice as all obstetric providers rotate working at each of the five clinics. Women eligible for the study were those receiving prenatal care in this clinic January 2017 to April 2021. Demographic information such as age, source of classification for race and relationship status was extracted from the EHR.
Outcomes
Primary outcomes for this study include the proportion of women completing a screen (defined as completing at least 1 screening question), screening positive (defined as answering “Yes” to at least one screening question), referred to treatment (defined as a referral to a mental health or substance use disorder professional e.g., psychiatrist, psychologist, Licensed Professional Counselor, or Licensed Independent Social Worker), and attending at least one appointment with one of these professionals within 3 months following a referral to treatment. We examined differences in screening, screening positive, referred to treatment and attending treatment by race within each screening method (LTW or in‐person SBIRT) and by screening method (LTW vs. in‐person SBIRT) to determine if racial disparities were present within a screening method, and/or between screening methods, respectively.
To determine if screening positive, referral to treatment and attending treatment varied by problem area, we compared the rates of screening positive, referral to treatment and attending treatment by group (LTW vs. In‐Person SBIRT) for specific screening questions relating to PMADs (i.e., “Over the past few weeks has worry, anxiety, depression or sadness made it difficult for you to do your work, get along with people or take care of things at home?”), PSUDs (i.e., “In the past, have you had difficulties in your life due to alcohol or other drugs including prescription medications?” or “In the past month, have you drunk any alcohol or used other drugs?”) and IPV (i.e., “Are you feeling at all Insafe in any way in your relationship with your current partner?”).
In‐Person SBIRT
All pregnant and postpartum women receiving prenatal care in the clinic described above were screened for mental health, substance use, and IPV using standardized screening questions and their responses were recorded in the EHR. The standardized screening questions and SBIRT trainings were developed and widely disseminated by a statewide perinatal collaborative (23) to improve detection and treatment of PMADs, PSUDs and IPV in pregnancy and up to 12 months postpartum. The screening is completed once during the first prenatal care appointment. The state Centers for Medicaid and Medicare and other major health insurers provide reimbursement for one screening and two brief interventions per fiscal year for pregnant and postpartum women.
Screening Assessment
The state supported SBIRT includes eight screening questions verbally asked by a nurse during routine prenatal intake to identify potential problems with mental health, substance use or IPV. Four of the eight screening questions include the 4Ps (24), a measure that asks if your Parents, Partner or Peers have a problem with alcohol or drug use and if “In the past month, have you drunk any alcohol or used other drugs?”. The other four questions focus on difficulties with alcohol or drugs including prescription medications (“In the past, have you had difficulties in your life due to alcohol or other drugs, including prescription medications?”), cigarette use in the past 3 months (“Have you smoked any cigarettes in the past three months?”), IPV (“Are you feeling at all unsafe in any way in your relationship with your current partner?”), and the presence of worry, anxiety, depression, or sadness that interferes with functioning (i.e., “Over the last few weeks, has worry, anxiety, depression or sadness made it difficult for you to do your work, get along with people or take care of things at home?”). Screening questions are verbally administered, and response (Yes/No) are entered in the EHR by the clinic nursing staff and reviewed by the obstetric provider (e.g., Obstetrician and Gynecologist [OB/GYN] or Certified Nurse Midwife [CNM]). If the screening is not completed, screening question responses are left blank in the EHR and can be completed at a subsequent clinic visit. For women screening positive, defined as answering “Yes” to any of the screening questions, the OB/GYN or CNM has a 5–10 min conversation with the patient using motivational interviewing techniques including: i) providing feedback and education regarding the screening results and associated risks to the woman and baby; ii) listening to the patient and eliciting internal motivation for change and providing support to promote healthy choices; and iii) providing guidance, support, and referral to mental health or substance use disorder treatment services. The obstetric provider records the brief intervention and referral information in the EHR. Patients are referred to the mental health providers located within the outpatient OB/GYN practice. Responses to the in‐person screening questions, referral to treatment if indicated, and attendance to a mental health treatment appointment are captured within the EHR. A third party, designated by the Institutional Review Board, provided investigators with de‐identified data from the EHR for all women receiving prenatal care for the designated clinic beginning January 1, 2017 through December 31, 2019. De‐identified data included SBIRT screening and referral information and completed appointments for mental health treatment. Attendance to treatment was recorded in the EHR and defined as attending an appointment with a mental health professional within 3 months after the date of in‐person SBIRT and referral to treatment.
Listening to Women
From January 1, 2020 to April 5, 2021, women receiving prenatal care in the same large outpatient clinic described above were enrolled by clinic nursing staff in an alternative SBIRT system, LTW. LTW employs the same eight SBIRT screening questions and delivers a brief intervention leveraging motivational interviewing techniques and referral to resources and treatment identical to those described for in‐person SBIRT. LTW differs in the delivery of screening questions in that women answer the eight screening questions via text message. In addition, if women answer Yes to the screening question about their mood and anxiety they also complete the Edinburgh Postnatal Depression Scale (EPDS) (25) via their phone. If they answer yes to any of the questions related to substance use, they complete the National Institute on Drug Abuse (NIDA) Modified Assist (26) via their phone. Immediately after completing the screenings, all women receive an automated text message with feedback about their screening results, contact information for the care coordinator and letting them know if a care coordinator will be contacting them. In addition, the automated text message includes other resources for urgent mental health problems and resources for national hotlines for suicide prevention and domestic violence. LTW also differs from in‐person SBIRT in that any indicated brief intervention is completed via phone by a care coordinator with a master's degree in clinical social work. The care coordinator assesses mental health and/or substance use disorders and IPV. The care coordinator identifies any needed resources (e.g., housing, food etc.) and makes referrals to an appropriate level of care and/or resources. Responses to the text‐message screening questions, phone‐based assessments, and referrals are recorded in REDCap, an on‐line data capture system and a summary of this information is automatically generated. The care coordinator makes any necessary edits to this automatically generated summary and “copies and pastes” this information into a progress note in the EHR. Attendance at a mental health treatment appointment is captured within the EHR. De‐identified data from the EHR including attendance to a mental health treatment appointment for women enrolled in LTW were also provided by a third party (as above) to study investigators.
Nursing staff were introduced to the LTW program by the study staff. The study staff provided a 45‐min in‐service training on the program and a written script explaining how to introduce the program to women along with a brief checklist for enrolling women into the program (see Supporting Information S1). The time required to introduce and enroll women in the LTW system was approximately equivalent to completing the eight in‐person SBIRT screening questions. The script included information about the purpose of the program, who would review the screening information and that a care coordinator would be calling women in response to a positive screen. Women were also told the program was voluntary and if they wanted, they could delete the text messages and their responses at any time. Instead of completing in‐person SBIRT during routine clinic workflow, nurses began enrolling women into LTW. Study staff demonstrated for nursing staff how to introduce and enroll at least two women receiving care in the clinic into the program and then observed nursing staff enrolling women into the program twice per week for two consecutive weeks. Study staff was available in‐person during this time to answer any questions and provide feedback to nursing staff about the program introduction and enrollment process. After the 2 weeks, study staff provided nursing staff with their contact information and encouraged the nursing staff to call with any follow‐up questions or concerns. Participants declining participation in the LTW program were asked to complete a brief online survey sent via text message to indicate the reason for declining participation.
As part of this pragmatic quasi‐experimental clinical trial there was no additional training provided to nurses or the care coordinator in motivational interviewing techniques. Their experience with motivational interviewing is based on their prior education and training including a master's in clinical social work (MSW) for the care coordinator, and participation in a state supported SBIRT training for nurses including a half day in‐person or online training in motivational interviewing. A master's degree in clinical social work is typically a 1 to 2‐year program. Relevant common courses to all MSW programs include mental health and substance use assessment, diagnosis, and treatment including motivational interviewing. Upon completion of MSW a clinical social worker can provide direct clinical services for individuals, families, couples, and groups with mental health and/or substance use disorders. A single care coordinator provided the phone‐based brief assessment and motivational interviewing throughout the entire LTW intervention. Because the LTW system was delivering the standard of care for SBIRT in this clinic, and only de‐identified EHR data was provided to investigators through an honest broker, the Medical University of South Carolina Institutional Review Board approved a waiver of written informed consent (Protocol # 00106246).
Sample Size
Based on the sample achieved we have performed a post‐hoc power analyses using a two‐sided Z‐Test with unpooled variance with a significance level of 0.05 for the four primary outcomes. Screened: With a sample size of 547 for LTW and 2988 for In‐person SBIRT, we achieved 88.00% power to detect a difference between group proportions of 0.0669. Screened Positive: With a sample size of 393 for LTW and 1947 for In‐person SBIRT, we achieved 99.00% power to detect a difference between group proportions of 0.1160. Referred to Treatment: With a sample size of 257 for LTW and 649 for In‐person SBIRT, we achieved 99.00% power to detect a difference between group proportions of 0.1484. Attendance to Treatment: With a sample size of 229 for LTW and 374 for In‐person SBIRT, we achieved 99.00% power to detect a difference between group proportions of 0.1549.
Data Analysis
Descriptive statistics were calculated for each screening method, including counts and percentages for categorical variables and mean and standard deviations for continuous variables.
Unadjusted differences between women in LTW and women receiving in‐person SBIRT were calculated using chi‐square tests. The Wilcoxon‐Mann Whitney test was used to calculate unadjusted age differences. All reported outcomes were dichotomous. Relative risk (RR) ratios were estimated using Poisson regression models with robust error variance in PROC GENMOD (27). Adjusted analyses controlled for potentially confounding variables (e.g., age, race, in a committed relationship). Outcomes are reported as adjusted RR as recommended for cohort studies where the outcome is not a rare event (28). To account for multiple comparisons within secondary outcome analyses within each question (i.e., screened positive, referred to treatment, attended treatment), we applied a Bonferroni correction with an adjusted alpha = 0.0177 (0.05/3). All tests were two‐sided with a type 1 error set at α < 0.05 and all analyses were performed using SAS statistical software version 9.4 (Cary, NC).
RESULTS
In‐person SBIRT was completed in 65.2% (1947/2988) of the women receiving prenatal care from January 1, 2017 to December 31, 2019. Conversely, 98.9% (547/553) of women receiving prenatal care from January 1, 2020 to April 5, 2021, were approached and agreed to take part in LTW. Of these, 71.9% (393/547) completed the text message‐based screening and 68.5% (375/547) of women were contact by phone by the care coordinator. Reasons for declining participation in LTW included not feeling comfortable answering questions about behavioral health via mobile phone (n = 1), and “other” (n = 5). Explanations in the “other’” category included: I am not interested in this care (n = 2); as an employee (at the academic institution), I do not feel comfortable taking part in this program (n = 3) (Figure 1). Demographic variables such as age, race and in a committed relationship are compared between groups for those that are eligible for SBIRT, completed SBIRT and did not complete SBIRT (Table 1).
| Demographics | Screening method | p | |||
|---|---|---|---|---|---|
| LTW (N = 547) | In‐person SBIRT (N = 2988) | ||||
| Mean | (SD) | Mean | (SD) | ||
| Age (years), total | 29.85 | (5.91) | 28.88 | (5.93) | 0.0017 |
| Screened | 30.18 | (5.88) | 28.84 | (5.90) | 0.0001 |
| Not screened | 29.01 | (5.90) | 28.95 | (5.97) | 0.7722 |
| N | (%) | N | (%) | ||
|---|---|---|---|---|---|
| Pregnant | 357 | (65.25) | 1984 | (66.42) | 0.3941 |
| Race, total | <0.0001 | ||||
| Black | 165 | (30.16) | 1276 | (42.70) | ‐ |
| White | 291 | (53.20) | 1486 | (49.73) | ‐ |
| Other | 91 | (16.64) | 226 | (7.56) | ‐ |
| Screened | <0.0001 | ||||
| Black | 109 | (27.74) | 841 | (43.19) | ‐ |
| White | 218 | (55.47) | 949 | (48.74) | ‐ |
| Other | 66 | (16.79) | 157 | (8.06) | ‐ |
| Not screened | 0.0002 | ||||
| Black | 56 | (36.36) | 435 | (41.79) | ‐ |
| White | 73 | (47.40) | 537 | (51.59) | ‐ |
| Other | 25 | (16.23) | 69 | (6.63) | ‐ |
| Marital status, total | <0.0001 | ||||
| Committed relationship | 241 | (44.06) | 1389 | (46.49) | ‐ |
| Divorced/separated | 6 | (1.10) | 41 | (1.37) | ‐ |
| Single | 287 | (52.47) | 1555 | (52.04) | ‐ |
| Widowed | 0 | (0.00) | 2 | (0.07) | ‐ |
| Unknown | 10 | (1.83) | 1 | (0.03) | ‐ |
| Missing | 3 | (0.55) | 0 | (0.00) | ‐ |
| Screened | <0.0001 | ||||
| Committed relationship | 195 | (49.62) | 877 | (45.04) | ‐ |
| Divorced/separated | 2 | (0.51) | 25 | (1.28) | ‐ |
| Single | 186 | (47.33) | 1042 | (53.52) | ‐ |
| Widowed | 0 | (0.00) | 2 | (0.10) | ‐ |
| Unknown | 8 | (2.04) | 1 | (0.05) | ‐ |
| Missing | 2 | (0.51) | 0 | (0.00) | ‐ |
| Not screened | <0.0001 | ||||
| Committed relationship | 46 | (29.87) | 512 | (49.18) | ‐ |
| Divorced/separated | 4 | (2.60) | 16 | (1.54) | ‐ |
| Single | 101 | (65.58) | 513 | (49.28) | ‐ |
| Unknown | 2 | (1.30) | 0 | (0.00) | ‐ |
| Missing | 1 | (0.65) | 0 | (0.00) | ‐ |
| In a committed relationship, total | 241 | (44.06) | 1389 | (46.49) | 0.2951 |
| Screened | 195 | (49.62) | 877 | (45.04) | 0.0969 |
| Not screened | 46 | (29.87) | 512 | (49.18) | <0.0001 |
Abbreviations: LTW, Listening to Women; SBIRT, Screening, Brief Intervention and Referral to Treatment.
In unadjusted analyses, a significantly greater proportion of women in LTW were screened, screened positive, referred for treatment, and received treatment, compared to women receiving in‐person SBIRT (Figure 2). After controlling for potentially confounding variables (i.e., age, race, in a committed relationship), women enrolled in LTW were significantly 10% more likely to be screened, 91% more likely to screen positive, 55% more likely to be referred to treatment, and 395% more likely to receive treatment, compared to in‐person SBIRT (Table 2). The rate of positive responses to each screening question for in‐person SBIRT and LTW are included in Figure S1.
| Outcome | LTW (N = 547) | In‐person SBIRT (N = 2988) | Unadjusted results | Adjusted resultsa | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| N | (%) | N | (%) | RR | (95% CI) | p | RR | (95% CI) | P | |
| Screened | 393 | (71.85) | 1947 | (65.16) | 1.30 | (1.10–1.55) | 0.0024 | 1.10 | (1.03–1.16) | 0.0027 |
| Positive screen | 257 | (65.39) | 649 | (33.33) | 2.99 | (2.47–3.62) | <0.0001 | 1.91 | (1.72–2.10) | <0.0001 |
| Referred treatment | 229 | (89.11) | 374 | (57.63) | 4.11 | (2.85–5.93) | <0.0001 | 1.55 | (1.43–1.69) | <0.0001 |
| Received treatment | 195 | (85.15) | 64 | (17.11) | 7.62 | (5.49–10.56) | <0.0001 | 4.95 | (3.93–6.23) | <0.0001 |
Abbreviations: LTW, Listening to Women; RR, Relative risk; SBIRT, Screening, Brief Intervention and Referral to Treatment.
a
Adjusted for age, race and in a committed relationship.
Table 3 includes unadjusted and adjusted analyses examining racial disparities within each screening method (Table 3, Panel A) and between screening methods (Table 3, Panel B). In adjusted analyses within the in‐person SBIRT group, controlling for age, and in a committed relationship, Black women were 36% less likely to screen positive, and 46% less likely to attend mental health treatment, compared to White women; however, no differences were found between White and Black women in likelihood of being screened or referred to treatment with in‐person SBIRT. In adjusted analyses within the LTW group, controlling for age, and in a committed relationship, Black women were 20% less likely to screen positive, compared to White women; however, no differences were found between White and Black women in likelihood of being screened, referred to treatment and attending treatment with LTW. In adjusted analyses, the comparison between screening programs (i.e., in‐person SBIRT vs. LTW) among Black women and controlling for age, and in a committed relationship, Black women enrolled in LTW were 35% more likely to screen positive, 46% more likely to be referred to treatment and 449% more likely to attend mental health treatment, compared to Black women receiving in‐person SBIRT.
| Panel A. Within screening group racial disparities (Black vs. White) | ||||||||
|---|---|---|---|---|---|---|---|---|
| Unadjusted results | Adjusted resultsa | |||||||
| Black | White | |||||||
| N | (%) | N | (%) | p | RR | (95% CI) | P | |
| Within LTW group | ||||||||
| Screened | 109 | (66.06) | 218 | (74.91) | 0.0437 | 0.98 | (0.85–1.13) | 0.7846 |
| Not screened | 56 | (33.94) | 73 | (25.09) | ‐ | ‐ | ‐ | ‐ |
| Screened positive | 59 | (54.13) | 149 | (68.35) | 0.0118 | 0.80 | (0.65–0.99) | 0.0422 |
| Screened negative | 50 | (45.87) | 69 | (31.65) | ‐ | ‐ | ‐ | ‐ |
| Referred to treatment | 53 | (89.83) | 128 | (85.91) | 0.4478 | 1.06 | (0.93–1.21) | 0.3537 |
| Not referred to treatment | 6 | (10.17) | 21 | (14.09) | ‐ | ‐ | ‐ | ‐ |
| Received treatment | 44 | (83.02) | 110 | (85.94) | 0.6160 | 1.03 | (0.89–1.20) | 0.6896 |
| Did not receive treatment | 9 | (16.98) | 18 | (14.06) | ‐ | ‐ | ‐ | ‐ |
| Within in‐person SBIRT group | ||||||||
| Screened | 841 | (65.91) | 949 | (63.86) | 0.2616 | 0.99 | (0.92–1.06) | 0.7097 |
| Not screened | 435 | (34.09) | 537 | (36.14) | ‐ | ‐ | ‐ | ‐ |
| Screened positive | 277 | (32.94) | 323 | (34.04) | 0.6231 | 0.64 | (0.56–0.75) | <0.0001 |
| Screened negative | 564 | (67.06) | 626 | (65.96) | ‐ | ‐ | ‐ | ‐ |
| Referred to treatment | 164 | (59.21) | 182 | (56.35) | 0.4798 | 1.01 | (0.86–1.18) | 0.8926 |
| Not referred to treatment | 113 | (40.79) | 141 | (43.65) | ‐ | ‐ | ‐ | ‐ |
| Received treatment | 24 | (14.63) | 39 | (21.43) | 0.1020 | 0.54 | (0.32–0.92) | 0.0236 |
| Did not receive treatment | 140 | (85.37) | 143 | (78.57) | ‐ | ‐ | ‐ | ‐ |
| Panel B: Within race group screening disparities (LTW vs. in‐person SBIRT) | ||||||||
|---|---|---|---|---|---|---|---|---|
| Unadjusted results | Adjusted resultsa | |||||||
| LTW | In‐person SBIRT | |||||||
| N | (%) | N | (%) | p | RR | (95% CI) | p | |
| Within Black group | ||||||||
| Screened | 109 | (66.06) | 841 | (65.91) | 0.9692 | 1.00 | (0.89–1.13) | 0.9610 |
| Not screened | 56 | (33.94) | 435 | (34.09) | ‐ | ‐ | ‐ | ‐ |
| Screened positive | 59 | (54.13) | 277 | (32.94) | <0.0001 | 1.65 | (1.35–2.01) | <0.0001 |
| Screened negative | 50 | (45.87) | 564 | (67.06) | ‐ | ‐ | ‐ | ‐ |
| Referred to treatment | 53 | (89.83) | 164 | (59.21) | <0.0001 | 1.54 | (1.35–1.76) | <0.0001 |
| Not referred to treatment | 6 | (10.17) | 113 | (40.79) | ‐ | ‐ | ‐ | ‐ |
| Received treatment | 44 | (83.02) | 24 | (14.63) | <0.0001 | 5.49 | (3.69–8.17) | <0.0001 |
| Did not receive treatment | 9 | (16.98) | 140 | (85.37) | ‐ | ‐ | ‐ | ‐ |
| Within White group | ||||||||
| Screened | 218 | (74.91) | 949 | (63.86) | 0.0003 | 1.18 | (1.09–1.27) | <0.0001 |
| Not screened | 73 | (25.09) | 537 | (36.14) | ‐ | ‐ | ‐ | ‐ |
| Screened positive | 149 | (68.35) | 323 | (34.04) | <0.0001 | 1.91 | (1.68–2.18) | <0.0001 |
| Screened negative | 69 | (31.65) | 626 | (65.96) | ‐ | ‐ | ‐ | ‐ |
| Referred to treatment | 128 | (85.91) | 182 | (56.35) | <0.0001 | 1.53 | (1.36–1.72) | <0.0001 |
| Not referred to treatment | 21 | (14.09) | 141 | (43.65) | ‐ | ‐ | ‐ | ‐ |
| Received treatment | 110 | (85.94) | 39 | (21.43) | <0.0001 | 4.05 | (3.05–5.38) | <0.0001 |
| Did not receive treatment | 18 | (14.06) | 143 | (78.57) | ‐ | ‐ | ‐ | ‐ |
Abbreviations: LTW, Listening to Women; RR, Relative risk; SBIRT, Screening, Brief Intervention and Referral to Treatment.
a
Adjusted for age and in a committed relationship.
Subgroup analyses comparing rates of screening positive, referral to treatment and attendance to treatment by group and specific questions approximating PMADs, PSUDs, and IPV are included in Table S1. For PMADs and PSUDs, LTW significantly increased the proportion of women screening positive, being referred to treatment and attending treatment compared to In‐Person SBIRT (p < 0.0177). Similarly, for IPV, LTW increased the proportion of women screening positive, being referred to treatment and attending treatment compared to In‐Person SBIRT, but differences were not statistically significant (p = 0.3068) (see Table S1).
DISCUSSION
The study findings demonstrate that obstetric nurses enrolling pregnant and postpartum women into a text message‐based mental health, substance use and IPV screening program with a remote care coordinator is feasible and advantageous to introduce into a large volume, obstetrical care clinic. A greater proportion of women enrolled in the LTW program were successfully screened, screened positive, referred to treatment and attended treatment, compared to in‐person SBIRT screening. Racial disparities identified with in‐person SBIRT, where Black women are less likely to attend treatment compared to White women, were not present with the LTW program.
Our findings are consistent with a single previous study employing text message screening for postpartum depression in an obstetric clinic that also demonstrates this method of screening is feasible and acceptable to patients (29). Our study significantly extends this line of research by including a comparison group as well as screening for substance use and IPV. Furthermore, our study demonstrates that with minimal training, implementation of the LTW system into a busy obstetrical care clinic can be accomplished using the current obstetrical care nursing staff, as opposed to study staff (29). The LTW technology‐based approach was found to be highly feasible and successful with efficiently screening a significantly higher percentage of women. Among the women available to be screened during prenatal care visits, nurses were more likely to screen women with the LTW system, compared to in‐person SBIRT. Similarly, pregnant and postpartum women were overwhelmingly accepting of a text‐message screening system with fewer than 2% of approached women declining participation in the LTW program and overall, a greater proportion of women completing screenings, compared to in‐person SBIRT which had been the established standard of care. The integration of the program into routine care by clinic nurses, combined with a care coordinator with no other SBIRT training, aside from the usual training and clinical experience for an individual with a master's degree in clinical social work, and high end‐user participation rates suggest that the LTW program can be easily integrated into routine prenatal care practices. In doing so, the LTW system may help address some of the practice and provider level barriers to maternal mental health, substance use and IPV screening (18, 19).
From a clinical benefit perspective, the LTW screening system results in a greater proportion of women screening positive and attending treatment for mental health and substance use disorders, compared to in‐person SBIRT. These findings are important given the known tendency of pregnant and postpartum women to underreport mental health and/or substance use problems and IPV (30, 31). It also suggests that LTW will result in more opportunities for health care providers to help facilitate needed support and life‐saving treatments (4, 5). Mental illness and substance use disorders are among the most stigmatized of all health conditions and such stigma is a major barrier to seeking and receiving effective care for these illnesses (18, 19). This is likely to be particularly true for pregnant and postpartum women who might feel that health care providers could be judgmental about the consequences of mental health or substance‐related issues for their pregnancy and infant. As such, it is possible that text‐messaging provides a greater sense of confidentiality and lower likelihood of feeling judged as compared to a face‐to‐face clinical interview with a health care provider. These findings are consistent with others demonstrating more positive responses with computer‐based assessments, compared to face‐to‐face assessments, for IPV (16) and other “stigmatized behaviors” (e.g., intravenous drug use, high risk sex, high risk HIV transmission behaviors), and “psychological distress” (e.g., hopeless, worried, depressed, suicidal ideation), compared to “neutral behaviors” (e.g., prior abscess, prior treatment) for those with substance use disorders and/or HIV (20, 21).
The increase in treatment attendance seen with LTW, compared to in‐person SBIRT is promising, especially given the complexity involved in engaging pregnant and postpartum women in mental health and substance use disorder treatment. The many challenges encountered in facilitating referrals to mental health and substance use disorder treatment (18, 19) may be better navigated by an experienced social worker than by a busy obstetrical provider. Overall, these findings are consistent with a large systematic review examining depression screening and referral interventions in outpatient perinatal care settings that demonstrates an increase in utilization of depression care with screenings and referral interventions that are more intensive and target multiple patient, provider, and practice‐level barriers to screening and treatment (9). While the range of study designs included in this systematic review are heterogenous, the results support a dose‐response relationship indicating that more intense strategies targeting multiple barriers to depression care are associated with enhanced mental health treatment and may be required to improve current screening and referral practices (9).
There is a robust literature demonstrating the benefit of in‐person SBIRT for peripartum substance use (10, 11, 32); therefore, for current study, it is important to employ in‐person SBIRT as the comparison group. Despite the efficacy of SBIRT in controlled studies, the effectiveness of SBIRT in clinical practice is not as reproducible (33). One explanation, suggested by Ino and Cho (2013), for the dilution of effectiveness may be the lack of motivational interviewing training and experience of providers tasked with employing brief interventions (33). It is possible that the greater proportion of women attending treatment with LTW, compared to in‐person SBIRT, may be explained by the involvement of a care coordinator with a mental health and social work background who potentially is more skilled at assessing these conditions and/or has greater experience and training in motivational interviewing, compared to obstetric health care providers. It is also possible that the care coordinator associated with the LTW intervention has more information (i.e., completed EPDS and/or NIDA modified assist) at the time of the brief intervention and/or have more dedicated time to talk with women, compared to obstetric providers with limited time and competing priorities during routine visits.
Computer‐assisted SBIRT for pregnant women identified with substance use or misuse has shown to be more effective than enhanced usual care (i.e., educational pamphlet plus existing treatment resources) in reducing substance use, but comparable to the effectiveness of in‐person SBIRT (13). Of note, providers delivering in‐person SBIRT in this study (13) received a 15‐h SBIRT workshop, followed by practice cases and feedback, coaching and monthly group supervision for the duration of the trial. These findings support the idea that fidelity to motivational interviewing is important and can potentially be achieved with computer‐based algorithms. Computer‐assisted SBIRT programs and phone‐based SBIRT programs employing care coordinators may potentially prove cost‐effective by alleviating the time needed to train obstetric staff in motivational interviewing practices as well as retrain employees due to staff turnover, not even considering the clinical health and resource saving benefits of more effective mental health and substance use disorders screening and treatment (34). Furthermore, these models used centralized personnel resources and, as such, do not require dedicated SBIRT personnel in each obstetric clinic and is therefore a less costly alternative to in‐person SBIRT (34). This same model could be applied to LTW. Furthermore, Medicaid and other health insurers provide reimbursement for SBRIT (i.e., $24 per annual screen, and $48 for each brief intervention reimbursed twice per year), suggesting that the cost of the care coordinator's time could be supported through SBIRT reimbursement.
Comparatively, there are fewer studies examining screening and referral interventions for IPV for pregnant and postpartum women, although, app‐based screening programs are currently being piloted (35). IPV is one of the leading causes of pregnancy‐related deaths (4), so improvements in detection and interventions for IPV could be lifesaving. The current study found that, although not statistically significant, women were twice as likely to endorse feeling unsafe with their partner via text message compared to in‐person screening. This is consistent with the fact that abusive partners frequently control their partners interactions with health care providers, potentially leaving a brief text‐messaging approach (where text‐messages can be quickly deleted) as one potential way to share confidential concerns of IPV. As such, further exploration of technology‐based applications to improve detection and access to support for women experiencing IPV is warranted.
Rates of screening positive for mental health, substance use or IPV were greater for Black women enrolled in LTW, compared to Black women enrolled in in‐person SBIRT. However, Black women enrolled in LTW were less likely to screen positive compared to White women enrolled in LTW. A research study including qualitative interviews with pregnant and postpartum Black women is currently underway to understand and mitigate this racial disparity (NIDA R34 DA046730). However, the racial disparities reported with in‐person SBIRT, with Black women less likely to attend mental health treatment compared to White women, was not seen in the LTW program. In fact, Black women were 5 times more likely to receive mental health treatment with the LTW intervention, compared to Black women receiving in‐person SBIRT. These findings are exciting given that peripartum Black women are significantly less likely to receive treatment, compared to peripartum White women for substance use disorders and postpartum depression (22, 23).
Strengths of this study are its large sample size and conduct of the study in a large obstetrical clinic using front‐line providers. For the purposes of the study and data collection, having all screening, referral and treatment services captured in the EHR is a strength. A limitation of the study is the inclusion of only one large prenatal care practice in an academic health system. It is unclear if the results of the study generalize to community‐based practices, although it is believed that implementation in a large practice might be more problematic than in a smaller facility. A second limitation of the study is that fidelity to motivational interviewing was not measured. Future studies will include measurement of fidelity to motivational interviewing to better understand the potential mechanism by which LTW potentially improves attendance to treatment. Third, there were fewer women enrolled in LTW, compared to In‐Person SBIRT. This was expected given that in‐Person SBIRT data were extracted from the EHR and LTW data were collected via active recruitment. Furthermore, LTW recruitment occurred for less time, compared to in‐person SBIRT (16 vs. 24 months) and most importantly during the LTW study collection period, there was a decrease in the volume of women presenting for prenatal care due to the COVID‐19 pandemic and a temporary change in workflow with some visits occurring remotely. These changes may have accounted for the number of women recruited for participation in LTW, compared to in‐Person SBIRT. Importantly, in the LTW group, only 22 women or 4.02% of women identified as Hispanic and similarly only 123 women or 4.12% of women in the SBIRT group identified as Hispanic. Given the relatively small group of Hispanic women, this precluded us from estimating effects especially for outcomes that do not include the entire group, that is, screening positive, referred to treatment, attending treatment. While the proportion of Hispanic women in our sample is consistent with the ethnicity of the overall clinic and state's population, lack of representation of Hispanic women it is a limitation of the study. Lastly, the fact that the in‐person SBIRT group includes individuals screened from 2017 to 2019 while data on the LTW group was collected in 2020–2021 is another potential study limitation. External factors such as COVID‐19 might have exerted a differential influence on the two groups as early reports suggest that IPV and mental health problems may have increased for pregnant and postpartum women during the pandemic, compared to pre‐pandemic. However, in a sample of 959 pregnant women completing an app‐based voluntary IPV screening did not demonstrate a greater incidence of IPV during COVID‐19 compared to prior to COVID‐19 (35), and a recent meta‐analysis of eight studies including 7750 pregnant or postpartum women did not demonstrate significantly higher rates of depressive symptoms, but moderately higher levels of anxiety symptoms (36) during COVID‐19, compared to pre‐COVID‐19. Additionally, the use of telemedicine during the pandemic could have made attending treatment more convenient for women, however telemedicine was a primary modality of service delivery for the study site prior to the pandemic. Nonetheless, these findings suggest that circumstances of COVID‐19 may have had a differential effect on screening and attendance to treatment for among women in LTW, compared to in‐person SBIRT. However, it is unlikely that COVID‐19 alone accounted for all study findings as findings are robust, intriguing and the study design supports the feasibility of implementing a larger cluster randomized controlled trial with the inclusion of community‐based OB/GYN practices which would provide greater scientific rigor to evaluating the generalizability of the LTW intervention.
CONCLUSION
Suicide and drug overdose combine to constitute the leading cause of maternal mortality during pregnancy and the postnatal period (4, 5), with PMADs, PSUDs and IPV increasing this risk. Additionally, each of these disorders contribute to significant maternal and child morbidity (6, 7). Despite strong recommendations from virtually all professional societies to implement effective screening and referral processes, the uptake of these recommendations has been limited and many screening and referral practices are largely ineffective. Technology facilitated SBIRT, that is, LTW, that incorporates all recommended screenings by professional societies into one program and can be delivered by current obstetric staff and care coordinators with minimal training and with high end‐user participation appears to be a feasible and promising approach to improving SBIRT effectiveness. High end‐user participation in the program and a greater number of positive screens creates more opportunities for health care providers to discuss and normalize problems with mental health, substance use or IPV and may help to destigmatize these conditions making it easier for women to reach out for help when needed. A large randomized controlled trial is needed to establish the efficacy and impact of the LTW intervention on maternal and child health.
Footnote
Preliminary data was presented virtually at the North American Society of Psychological Obstetrics and Gynecology (September 2020) and the Association for Psychosomatic Medicine (November 2020). The efforts of Drs. Guille, Maldonado, Simpson, King, and Ford were funded by Health Resources and Services Administration (HRSA) of the U.S. Department of Health and Human Services (HHS) as part of the National Telehealth Center of Excellence Award (U66 RH31458). Additionally, the effort of Dr. Guille was funded by the National Institute on Drug Abuse (NIDA) (R34 DA046730) and Duke Endowment (6563‐SP). Drs. Brady and Guille were funded by NIDA/Office of Research on Women's Health (U54DA016511) and NIDA Clinical Trails Network (5UG1DA013727). Dr. Brady was funded by the National Center on Clinical and Translational Science (NCATS) (UL1TR001450). The contents are those of the authors and do not necessarily represent the official views of, nor an endorsement, by NIDA, NCATS. HRSA, or the U.S. Government or Duke Endowment. All funding sources did not have a role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. Drs. Guille, Maldonado and Simpson had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. The authors would like to acknowledge the women, health care providers, obstetric practice, and South Carolina Birth Outcomes Initiative. Without these individuals and organizations, this work would not be possible. They would also like to acknowledge the MUSC Biomedical Informatics Center who programmed the LTW intervention. Dr. Guille receives advisory board and speaker's bureau honoraria from Sage Therapeutics. No other authors report any conflicts of interest.
Supplementary Material
Supporting Information S1
- Download
- 16.30 KB
Table S1
- Download
- 20.21 KB
Figure S1
- View/Download
- 23.98 KB
REFERENCES
1.
Martin SL, Mackie L, Kupper LL, et al: Physical abuse of women before, during, and after pregnancy. J Am Med Assoc 2001;285(12):1581–1584
2.
Luca DL, Margiotta C, Staatz C, et al: Financial toll of untreated perinatal mood and anxiety disorders among 2017 births in the United States. Am J Public Health 2020;110(6):888–896
3.
U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Behavioral Health Statistics and Quality . National survey on drug use and health 2019 (NSDUH), 2020
4.
Trost S, Beauregard JL, Smoots AN, et al: Meta‐analysis of antenatal depression and adverse Birth outcomes in US populations, 2010–20. Health Aff 2021;40(10):1560–1565
5.
Simonovich S, Nidey NL, Gavin AR, et al: Preventing pregnancy‐related mental health deaths: insights from 14 US Maternal Mortality Review Committees, 2008–17. Health Aff 2021;40(10):1551–1559
6.
Byatt N, Masters GA, Bergman AL, et al: Screening for mental health and substance use disorders in obstetric settings. Curr Psychiatry Rep 2020;22(11):62
7.
Byatt N, Levin LL, Ziedonis D, et al: Enhancing participation in depression care in outpatient perinatal care settings a systematic review. Obstet Gynecol 2015;126(5):1048–1058
8.
Krist AH, Davidson KW, Mangione CM, et al: Screening for unhealthy drug use: US Preventive Services Task Force recommendation statement. J Am Med Assoc 2020;323(22):2301–2309.
9.
Moyer V. Screening and behavioral counseling interventions in primary care to reduce alcohol misuse: US Preventive Services Task Force recommendation statement. Ann Intern Med 2013;159(3):210–218
10.
Humeniuk R, Ali R, Babor T, et al: A randomized controlled trial of a brief intervention for illicit drugs linked to the Alcohol, Smoking and Substance Involvement Screening Test (ASSIST) in clients recruited from primary health‐care settings in four countries. Addiction 2012;107(5):957–966
11.
Dwinnells R. SBIRT as a vital sign for behavioral health identification, diagnosis, and referral in community health care. Ann Fam Med 2015;13(3):261–263
12.
Bauman BL, Ko JY, Cox S, et al: Vital signs: postpartum depressive symptoms and provider discussions about perinatal depression—United States, 2018. Morb Mortal Wkly Rep 2020;69(19):575–581
13.
Martino S, Ondersma SJ, Forray A, et al: A randomized controlled trial of screening and brief interventions for substance misuse in reproductive health. Am J Obstet Gynecol 2018;218(3):322.e1–322.e12
14.
Kozhimannil KB, Dowd WN, Ali MM, et al: Substance use disorder treatment admissions and state‐level prenatal substance use policies: evidence from a national treatment database. Addict Behav 2019;90:272–277
15.
Byatt N, Simas TAM, Lundquist RS, et al: Strategies for improving perinatal depression treatment in North American outpatient obstetric settings. J Psychosom Obstet Gynecol 2012;33(4):143–161
16.
Alvarez C, Fedock G, Grace KT, et al: Provider screening and counseling for intimate partner violence: a systematic review of practices and influencing factors. Trauma Violence Abuse 2017;18(5):479–495
17.
ChisholmBullock CAL, Ferguson JE. Intimate partner violence and pregnancy: screening and intervention. Am J Obstet Gynecol 2017;217(2):145–149
18.
Frazer Z, McConnell K, Jansson LM. Treatment for substance use disorders in pregnant women: motivators and barriers. Drug Alcohol Depend 2019;205:107652
19.
Byatt N, Biebel K, Lundquist RS, et al: Patient, provider, and system‐level barriers and facilitators to addressing perinatal depression. J Reprod Infant Psychol 2012;30(5):436–449
20.
Newman JC, Des Jarlais DC, Turner CF, et al: The differential effects of face‐to‐face and computer interview modes. Am J Public Health 2002;92(2):294–297
21.
Perlis TE, Des Jarlais DC, Friedman SR, et al: Audio‐computerized self‐interviewing versus face‐to‐face interviewing for research data collection at drug abuse treatment programs. Addiction 2004;99(7):885–896
22.
Patel I, Kalkat S, Kwerteng E, et al: Racial disparities in treatment of pregnant women with drug use, abuse and dependence. Value Health 2017;20(9): A900
23.
Kozhimannil KB, Trinacty CM, Busch AB, et al: Racial and ethnic disparities in postpartum depression care among low‐income women. Psychiatr Serv 2011;62(6):619–625
24.
Chasnoff IJ, Wells AM, McGourty RF, et al: Validation of the 4P's Plus© screen for substance use in pregnancy validation of the 4P's Plus. J Perinatol 2007;27(12):744–748
25.
Cox JL, Chapman G, Murray D, et al: Validation of the Edinburgh Postnatal Depression Scale (EPDS) in non‐postnatal women. J Affect Disord 1996;39(3):185–189
26.
Ondersma SJ, Chang G, Blake‐Lamb T, et al: Accuracy of five self‐report screening instruments for substance use in pregnancy. Addiction 2019;114(9):1683–1693
27.
Lindquist K. How can I estimate relative risk in SAS using PROC GENMOD for common outcomes in cohort studies? | SAS FAQ. Accessed Aug 12, 2021. Available from: https://stats.idre.ucla.edu/sas/faq/how‐can‐i‐estimate‐relative‐risk‐in‐sas‐using‐proc‐genmod‐for‐common‐outcomes‐in‐cohort‐studies/
28.
Lemeshow S, Sturdivant RX, Hosmer DW. Applied Logistic Regression. New York, NY, Wiley, 2013
29.
Lawson A, Dalfen A, Murphy K, et al: Use of text messaging for postpartum depression screening and information provision. Psychiatr Serv 2019;70(5):389–395
30.
Mazza M, Caroppo E, Marano G, et al: Caring for mothers: a narrative review on interpersonal violence and peripartum mental health. Int J Environ Res Publ Health 2021;18(10):5281
31.
Yonkers K, Howell H, Gotman N, et al: Self‐report of illicit substance use versus urine toxicology results from at‐risk pregnant women. J Subst Use 2010;16(5):372–380
32.
Committee on Obstetric Practice . Committee opinion No. 711: opioid use and opioid use disorder in pregnancy. Obstet Gynecol 2017;130(2):e81–e94
33.
Ino A, Cho T. [Significance of SBIRT and countermeasures for its dissemination]. Nihon Arukoru Yakubutsu Igakkai Zasshi 2013;48(2):105–117
34.
Olmstead TA, Yonkers KA, Ondersma SJ, et al: Cost‐effectiveness of electronic‐ and clinician‐delivered screening, brief intervention and referral to treatment for women in reproductive health centers. Addiction 2019;114(9):1659–1669
35.
Krishnamurti T, Davis AL, Quinn B, et al: Mobile remote monitoring of intimate partner violence among pregnant patients during the COVID‐19 shelter‐in‐place Order: quality improvement pilot study. J Med Internet Res 2021;23(2):e22790
36.
Hessami K, Romanelli C, Chiurazzi M, et al: COVID‐19 pandemic and maternal mental health: a systematic review and meta‐analysis. J Matern Fetal Neonatal Med 2020:1–8. https://doi.org/10.1080/14767058.2020.1843155
Information & Authors
Information
Published In
History
Received: 21 August 2021
Revision received: 13 October 2021
Accepted: 15 October 2021
Published online: 18 November 2021
Published in print: Winter 2021
Authors
Funding Information
Health Resources and Services Administration: U66 RH31458
Office of Research on Women's Health: U54DA016511
Duke Endowment: 6563‐SP
National Center for Advancing Translational Sciences: UL1TR001450
National Institute on Drug Abuse: 5UG1DA013727, R34 DA046730
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBLogin options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).