Understanding the Research on Extreme Risk Protection Orders: Varying Results, Same Message

More than a dozen states have enacted risk-based temporary firearm removal laws—typically called extreme risk protection orders (ERPOs)—and numerous other states are considering similar legislation (1, 2). Emerging research on these policies’ fairness and effectiveness thus takes on new import. Do ERPOs save lives and also respect the rights of law-abiding gun owners who pose no threat? Studies by Swanson and colleagues (3, 4) and by Kivisto and Phalen (5) independently examined the impact of firearm removal authorized by civil courts in Connecticut and Indiana, the first two states to enact such laws. These studies converge in their main conclusion: two different states’ laws enforced for over a decade to temporarily separate firearms from people considered at high risk of harming themselves or others were effective in preventing firearm-related suicides. But the studies’ findings differed substantially in their estimates of the magnitude of both the laws’ benefits and potential adverse consequences. This commentary compares the two sets of studies and discusses possible reasons for their varying results. The discrepancies, rooted in methodological differences, do not undermine the relevant policy message from both studies: ERPOs save lives. Still, further research is needed in other jurisdictions, using a range of empirical methods—rigorous, complementary approaches that can both illuminate the underlying legal processes and evaluate the relevant outcomes of ERPOs.

Kivisto and Phalen (5) compared trends in population-level suicide rates in Connecticut and Indiana with trends in “synthetic control” states, which lack gun removal laws. Using a statistically sophisticated quasi-experimental analysis, the researchers estimated that Indiana’s gun removal law was associated with a 7.5% reduction in gun suicides over 10 years and that Connecticut’s law was associated with a 13.7% reduction in the “post–Virginia Tech period” (5). However, the researchers concluded that the laws’ benefits were somewhat offset by increased nonfirearm suicides. Although the gun removal laws prevented an estimated 128 firearm suicides in Connecticut and 383 firearm suicides in Indiana, the laws contributed to an estimated 140 nonfirearm suicides in Connecticut and 44 nonfirearm suicides in Indiana, according to Kivisto and Phalen (5).

Taking a more direct research approach, Swanson and colleagues (3, 4) analyzed individual-level data on firearm removal cases matched to death records in Connecticut and Indiana and concluded that the gun removal laws prevented approximately 72 firearm suicides in Connecticut and 39 firearm suicides in Indiana—much smaller numbers than Kivisto and Phalen’s estimates. Also in contrast to the population-level results, Swanson and colleagues’ study identified 15 nonfirearm suicides in Connecticut (3) and seven nonfirearm suicides in Indiana (4) in the death records of persons subjected to firearm removal. Why did these studies produce such different results?

Swanson and colleagues acknowledged that their study possibly underestimated the number of prevented suicides (4). Their analysis focused on attempted suicide to estimate the reduction in fatalities attributable solely to the shift from firearms to less lethal means of self-injury. But their calculation did not include averted suicides among those who did not attempt suicide by any means after gun removal, possibly because they were able to access timely mental health treatment that effectively mitigated suicidality. Given Swanson and colleagues’ finding from Connecticut (3) that the proportion of people subjected to ERPOs who received treatment doubled in the year following firearm removal (from about 12% to about 24%), it is at least plausible that the law produced some indirect benefit by reducing primary suicidal behavior in response to newly accessed treatment. Kivisto and Phalen’s study theoretically could have captured this broader effect.

However, there are perhaps stronger reasons to suspect that Kivisto and Phalen’s analysis overestimated both the number of firearm suicides prevented and the number of nonfirearm suicides precipitated by the gun removal laws in Connecticut and Indiana. Whereas the Swanson study had access to individual-level data on suicide outcomes for the people who experienced firearm removal, Kivisto and Phalen’s study did not. Arguably, the known number of people whose guns were removed in the two states is too small to account for the magnitude of effect that Kivisto and Phalen’s method calculated.

Kivisto and Phalen’s estimate of 383 prevented firearm suicides in Indiana (5) is almost as high as the total number of Indiana gun removal cases that Swanson and colleagues (3) studied: 395 individuals, of whom 14 eventually died by suicide. Kivisto and Phalen’s assessment thus invites the problematic assumption that, but for the law, virtually every gun removal subject would have died of suicide. Similarly, Kivisto and Phalen’s approximation of the number of nonfirearm suicides attributable to the gun removal laws far exceeds the number of nonfirearm suicides documented in the death records of persons whose firearms were removed under these laws: 140 versus 15 in Connecticut and 44 versus seven in Indiana. It is difficult to construe a causal mechanism by which gun removal laws could have contributed substantially to nonfirearm suicides among people who were not subjected to firearm removal.

All empirical research studies must attempt to minimize threats to validity by using the most appropriate study designs, methods, and analysis. On one hand, “ecological” studies that compare population-level suicide rates in jurisdictions with different laws—valuable inquiries, as far as they go—are susceptible to causal misinterpretation; rates and trends pertaining to causally unrelated variables can sometimes coincide due to the common influence of other unobserved variables. Kivisto and Phalen (5) acknowledged this limitation in their study. On the other hand, studies of individuals directly exposed to different legal policies may produce more reliable information about specific outcomes in these individuals, but such studies tend to be limited by small samples, a lack of appropriate comparison groups, and difficulty generalizing to the population level.

Over time, it is important not only to replicate policy research findings but also to build a broad and robust base of evidence using alternative approaches in which one study’s methodological strengths might compensate for another study’s limitations. The scientific evidence base for ERPOs is very promising but is still in its infancy. As more states gain practical experience with ERPOs, it is crucial that additional studies be conducted carefully to examine the underlying legal processes as well as the outcomes of these news laws, in as many jurisdictions as possible. Evidence-based policy improvements and policy-informed evaluation research should proceed in a mutually reinforcing, iterative process.

The relevant question is no longer, Do ERPOs work? (They do.) Rather, the emerging questions are: How do ERPOs work best? For whom do they work and under what conditions? What public resources are necessary to implement ERPOs and optimize their benefits? What are the potential adverse consequences that must be guarded against and minimized? What are the appropriate roles of family members, judges, police, attorneys, and clinicians in the process of using ERPOs? What are the ideal features of an ERPO statute? How do ERPOs interact with other laws and policies? If ERPOs are optimally implemented, how large is their impact likely to be? These questions form an agenda for the next generation of research studies—an important next step in bringing this life-saving legal policy to scale.