Revalidating the Brief Jail Mental Health Screen to Increase Accuracy for Women

The BJMHS was revised by reanalyzing the data from the previous study and conducting an extensive review of existing mental health screens. In the previous study ( 11 ) it was found that the modal Structured Clinical Interview for DSM-IV (SCID) diagnosis for missed cases (both male and female) was major depression. The existing SCID interview data were analyzed using the all-subsets-regression method to find the best predictive model for a diagnosis of major depression. Three symptoms of depression were identified that did not have a corresponding question in the original BJMHS or in its parent screen, the Referral Decision Scale ( 6 ). With slightly modified format consistent with prior questions, these were added to the BJMHS (the first two were adapted from the RAND Depression Screener and one was adapted from the Mini-International Neuropsychiatric Interview).

It was felt that part of the reason that the original BJMHS performed relatively poorly with females was that the BJMHS did not measure the symptoms of anxiety associated with the high incidence of PTSD experienced by female detainees. Thus a fourth question, relating to PTSD, was added to the screen. These four new questions were placed immediately before the original eight questions. The revised instrument is referred to herein as the BJMHS-R. As with the previous study, the SCID was used as the gold standard for clinical validation of the BJMHS-R.

Participants included jail detainees admitted to one of four county jails—two in Maryland and two in New York. Data were collected between November 2005 and June 2006. At three of the jails (Rennselaer County, New York; Montgomery County, Maryland; and Prince George's County, Maryland) all the participants were screened upon admission to the jail. At the fourth jail (Monroe County, Rochester, New York), because of the heavy volume of detainees, participants were screened after their initial court hearing, which was generally within 24 hours of their initial booking.

The screening data were used to identify an equal size subsample of detainees who scored positive and negative for referral on the BJMHS-R for the SCID validation.

The BJMHS-R is organized into two sections. The first section includes the ten symptom items that ask whether someone is currently experiencing the occurrence of mental health symptoms. The original six questions can be found in the prior report ( 11 ). The new questions asked whether one is feeling sad, empty, or depressed nearly every day; whether one has lost interest in things like work, hobbies, and other things usually enjoyed; whether one is currently having trouble sleeping nearly every night; or whether one has ever had any experience that was so frightening, horrible, or upsetting that it led to recent nightmares or being easily startled.

The second section of the BJMHS-R includes two items addressing whether a detainee was ever hospitalized for emotional or mental health problems and whether he or she is currently taking psychotropic medication.

For the purpose of the validation study, serious mental illness was defined as the presence of one or more of the following SCID diagnoses: major depressive disorder, depressive disorder not otherwise specified, bipolar disorder (I and II and not otherwise specified), schizophrenia disorder, schizoaffective disorder, schizophreniform disorder, brief psychotic disorder, delusional disorder, and psychotic disorder not otherwise specified.

Correctional classification officers in all four jails received training on administration of the BJMHS-R. This training, which took place in the jails, included a brief video that provided a description of the research project and instructions on completing the BJMHS-R during the booking process.

Sixteen clinical research interviewers participated in a two-day training session on administration of the SCID by a clinically trained SCID instructor. In addition, all of the clinical research interviewers conducted practice interviews with acquaintances and with psychiatric patients who volunteered. In order to ensure interrater reliability, all the interviewers rated two SCID reliability tapes and were observed conducting interviews at the jail by the SCID trainer and other project staff.

As soon as detainees were classified into the referral and nonreferral groups, the clinical research interviewers, who were blind to the detainees' sampling group statuses, approached the detainees on their list of potential participants and completed SCID interviews with those who consented to participate in the study. All interviews occurred within 72 hours of a detainee's admission to the jail.

Participation in this study was voluntary. Informed consent forms approved by the institutional review board were required and obtained for all SCID interview participants. Participants were informed that their decision to participate would not affect their stay in the jail and were administered a brief quiz to assess their competency to consent. The overall refusal rate was 31%. This refusal rate, while high, is identical to the 31% in our prior research ( 11 ) and is most likely due to the lack of compensation and scheduling difficulty within the jail constraints.

Participants included 10,562 male and female jail detainees admitted to one of four county jails from November 2005 to June 2006. The participants were predominately male (8,864 detainees, or 84%), and the mean±SD age was 32±11 years. Data were available for race and ethnicity for 9,495 detainees. Over half were African American (5,857 detainees, or 62%), 1,965 (21%) were white, 1,321 (14%) were Hispanic, and 352 (4%) were of another race or ethnicity.

Detainees were considered to be referred for further evaluation if they endorsed at least two of the ten symptoms from section 1 of the BJMHS-R or either of the two medical history questions from section 2 of the BJMHS-R. Of the 10,255 detainees with valid BJMHS-R screening data, 2,235 (22%) were classified as needing a referral for further mental health assessment. Female detainees (655 of 1,614 women, or 41%) were more than twice as likely as men (1,580 of 8,607 men, or 18%) to be classified as needing a referral.

There was a clear gender difference among correction officers doing the screens. Detainees who were screened by a female officer were twice as likely to be referred as those who were screened by a male officer (1,098 of 3,501 detainees screened by a female officer, or 31%, compared with 960 of 6,071 detainees, or 16%). The pattern was similar for both male and female detainees. Of the male detainees screened by male officers, 15% (816 of 5,589 detainees) were referred using the screen, compared with 26% (636 of 2,445 detainees) of the males screened by a female officer. For female detainees, the proportions were 31% (142 of 460 detainees) and 44% (461 of 1,046 detainees), respectively. At the Monroe County jail, nearly all the female detainees were screened by female officers. If this jail is excluded from the analysis, the discrepancy for females is considerably less (136 of 446 detainees screened by a male officer, or 30%, compared with 254 of 698 detainees screened by a female officer, or 36%), but the general pattern holds.

The screening data were used to identify a subsample of 945 detainees who were selected for a detailed clinical assessment. Approximately 26% of those selected (N=248) were released before the interview or were unavailable for a variety of reasons, such as illnesses. An additional 219 detainees (23%) refused to participate in the study, and 14 (1%) had incomplete interviews. Participants in the final SCID subsample were 464 detainees that included 175 referrals (68 males and 107 females) and 289 nonreferrals (138 males and 151 females).

Table 1 presents the clinical cross-validation results of the 12-item BJMHS-R and the eight-item BJMHS for the subsample of 464 detainees who received the SCID. For comparison purposes, the table also shows the cross-validation results of the eight-item BJMHS from the original study (N=357) ( 11 ). Clearly, the four added questions in the BJMHS-R did not improve the performance of the original instrument. Also, as measured by difference of means test for the eight-item BJMHS, the study presented here showed a significant improvement in the percentage of females correctly classified, compared with the original study (z=–2.13, p=.033). For sensitivity, specificity, false-negative rates, and false-positive rates there is no gold standard for this type of instrument. In practice, it is a policy call as to what are acceptable rates given real-world practices, procedures, and potential liability. Nonetheless, the key point here is that the original eight-item BJMHS seems preferable in balancing the competing interests of accurately identifying people in need of services while controlling system burden from the number of referrals for more intensive mental illness assessment.

Despite making a number of what seemed to be clinically informed changes to the BJMHS, when new data were collected at three of the four original study sites plus a fourth new site, the original BJMHS turned out to be more powerful. These results reinforce the value of this screen for men and suggest that the earlier reservations about its use for women were overstated.

It's unclear why the 35% false-negative rate in the original study was reduced to 14% in the study presented here for the eight-item BJMHS. It would seem that, in effect, the confidence interval for false negatives for womenfor the eight-item BJMHS is between 14% and 37%, depending on the particular jails in which it is being used. It is quite possible that the results for this screening tool vary not only by sample, but they also may be influenced by other factors, such as the base rate of mental illness in the jail.

In the study presented here, female booking officers were slightly more likely than male booking officers to elicit responses that resulted in referral when using the 12-item BJMHS-R, compared with the eight-item BJMHS; however, these gender differences were statistically significant for both versions of the BJMHS. It is important to note that the overall accuracy results from the eight-item BJMHS do not significantly differ from female booking staff to male booking staff (76% versus 78%, respectively). This is explained by the fact that although female officers tended to refer more individuals and had a lower false-negative rate, they also had a higher false-positive rate, resulting in an overall equivalency in accuracy. We do not believe that these observations mean that the BJMHS should be done only by female booking staff because it is simply impractical in most jails and the overall accuracy rate is almost identical with female and male correction officers.

At this time, the BJMHS, at very low expense, provides very accurate identification of detainees with current symptoms of mental illness. It produces a reasonable proportion of those screened (between 11% and 16%) who should be referred for more intensive assessment by medical staff. Within the constraints of time, staff, and money in U.S. jails, this instrument can be administered by regular corrections staff in two to three minutes and offers the strongest empirical basis for regular use as a practical tool of any instruments currently available. With these new data, there is even stronger support for the conclusion of the recent National Institute of Justice report on the two gender-specific Correctional Mental Health Screens and the BJMHS, "Based on their successful validation results, it is anticipated that these tools will be disseminated nationwide for use in all correctional facilities" ( 10 ). To this we would add that such instruments should be used for universal screening in all U.S. jails in keeping with the American Psychiatric Association's Guidelines for Psychiatric Services in Jails and Prisons ( 5 ).

This project was supported by grant 2005-IJ-CX-0004 from the U.S. Department of Justice, National Institute of Justice, Office of Justice Programs. The authors are grateful to their recently deceased colleague Steven M. Banks, Ph.D., for his statistical advice and interpretations on this and so many other projects. They are indebted to Tara Agnese, M.A., who coordinated the two Maryland sites. They are also grateful to the jails and their administrators who participated in the research: Col. Robert R. Loveridge at Rensselaer County (New York), Maj. John Caceci at Monroe County (New York), Arthur Wallenstein at Montgomery County (Maryland), and Barry Stanton at Prince George's County (Maryland). The authors also acknowledge the contributions of the New York data collectors: Elizabeth Stewart, M.S., Robert Hawkes, M.A., Corrine Kindzierski, Ph.D., and Christine Sergent, M.S. (Monroe County); Janet Poole, B.S., Lucas Jacobs, M.A., Steven Hallock, B.A., and Beth Kellerman, Ph.D. (Rensselaer County); and Jim Blucher, M.A., Michael Sklar, M.A., Alana Henninger, M.A., Jenny Daffin, M.A., and Tiffanie Sim, B.S. (Maryland). The project would not have been successful without the commitment to field interviewing exhibited by these interviewers and the SCID training and observation provided by Jennifer Norbeck, Ph.D. Points of view in this article are those of the authors and do not necessarily represent the official position or policies of the U.S. Department of Justice.

The authors report no competing interests.

Dr. Steadman, Ms. Robbins, and Mr. Islam are affiliated with Policy Research Associates, Inc., 345 Delaware Ave., Delmar, NY 12054 (e-mail: [email protected]). Dr. Osher is with the Council of State Governments, Bethesda, Maryland.