Integrating Stakeholder Engagement With Claims-Based Research on Health Insurance Design and Bipolar Disorder

The project benefited from fast friendship between the academic and advocacy coleads on the interview study and community engagement (J.M.M. and P.M.F.). Extensive telephone, e-mail, and face-to-face communications fostered intensive colearning, whereby each colead developed an understanding of the other’s professional world, norms, and language. For example, the DBSA coinvestigator received formal training in the protection of human subjects, and we shared thoughts on how institutional review board requirements for health-related research contrast with the freedom DBSA typically enjoys as an advocacy organization gathering and reporting information about its audience. We discussed other relevant projects under way, eventually forging new collaborations. Early in the current project, the academic colead urged recruiting interviewees from a defined population—DBSA chapters—while expressing skepticism of web-based platforms that the DBSA colead felt would accelerate recruitment. The academic colead felt that a single recruitment avenue offered clarity in later reporting and confidence that interviewees had “authentic” roots in DBSA. We adhered to that decision, but, over time, the authenticity of the DBSA Facebook audience became increasingly obvious, and slow chapter-based recruitment became a concern. On best specific strategies for outreach to chapters and Facebook readers, DBSA consistently charted the course. Outreaches were crafted, and their resulting feedback was analyzed jointly by the two coleads.

The academic colead acted as liaison between qualitative and quantitative project activities because it was not feasible for all investigators to fully immerse themselves in both. Nevertheless, all investigators attended monthly project meetings, contributing to protocol refinements for both studies. Our DBSA coinvestigator brought contemporary issues into these discussions, such as the growing role of paid peer counselors and the distinction between “getting to wellness” and merely eliminating clinical symptoms. Although the term “patient” is prominent in publications about health care and its costs and research engagement, many DBSA peers consider the word too narrow and disempowering, so our investigators learned to seek alternatives. The DBSA coinvestigator, in turn, reviewed our progress with DBSA’s peer-led national headquarters throughout the project period.

Quarterly meetings brought together the stakeholder advisors and the full research team. Investigators reviewed recent study results and community feedback, upcoming plans, and any problems or pending decisions. The meetings were opportunities to take stock of the project and receive fresh perspectives and ideas. All attendees had substantial content expertise, and the moderate size of the group permitted discussions that were simultaneously high-level, informal, and generally satisfying. Nevertheless, after a year, in response to a peer representative’s concerns, we revised our meeting format. The academic researchers had come to panel meetings having greater familiarity with both the project and the other attendees but having little experience with engaged research. In their diligence, researchers sometimes inadvertently dominated discussions, and some peer representatives felt unheard. Recognition of this problem was slow because most attendance was by telephone. To rebalance the meetings and elevate peer voices, we adjusted our procedures as follows: helping external advisors feel readier to contribute by distributing slide decks further in advance, with slides explicitly labeled “questions for our advisors,” and establishing a hierarchical speaking order for discussion periods. We invited comments first from peer representatives, next from other external advisors, and then from researchers. Participants agreed that the discussion quality improved as a result of these changes.

Most research team members were unable to extensively examine interview transcripts or comments gathered through social media, so quarterly meetings were their primary engagement experience. Peer representatives on the advisor panel offered interpretations of study results through the lens of personal experience, suggested which results would be useful for peer audiences, and highlighted health care system dysfunctions deserving closer study. The liaison investigator encouraged follow-up comments and conversations after these meetings, which were frequent and fruitful and strengthened feelings of trust.

The in-depth interviews had high value within the project both as a formal research aim and activity and as a way to incorporate more peer voice and experience into the project generally. Our claims data came from a single large insurer, and our quantitative study cohorts were highly specified. By contrast, interview respondents reported a range of coverage situations—all were employer sponsored, but with different years’ duration, levels of generosity, and benefit structures. Respondents shared details of their living circumstances and journeys toward recovery and provided fully dimensional views of their experience navigating systems of care for bipolar disorder. They heightened our awareness of issues such as the U.S. psychiatrist shortage, narrow provider networks, self-coordination between primary and behavioral care, disruptions in care following routine changes in employer benefits, mental illness stigma, complications of psychotropic treatment, and strategies for achieving wellness and reducing costs. Insights from the interviews guided analyses of the claims data. For example, stronger awareness of network-related problems led us to examine regional differences more closely and develop plans for detecting evidence of care that is obtained entirely outside of coverage.

Our claims data analyses detected only modest effects on care utilization following the shift into HDHPs, despite substantial increases in cost-sharing. Specifically, we observed evidence of cost-driven reductions in psychotherapy visits but not in psychiatrist visits or medications. These modest findings were more easily understood in light of our interview findings. Respondents reported great reluctance to tinker with their mental health care if it was working; many explained that when money was tight, they usually tried to cut back in other areas, to the point of genuine hardship, but not their care for bipolar disorder (33). Conducting interviews thus augmented our project by informing and improving empirical analyses and extending beyond them with distinct findings. Having a mixed-method research design can hedge against the disappointment of “null” statistical results. Interviews seldom produce null results, because their purpose is to unearth what people experience and say, and they almost always yield some insight or finding. Moreover, by illustrating the trade-offs and hardships in daily life that often stem from high costs of care, the interviews revealed consequences of high costs that were not detectable in claims data. This strengthened our net findings and increased the likelihood that policy makers consuming the research might consider reforms to protect patients from such harms.

The interviews were also deeply moving: people told vivid, intimate stories with granularity and humor. They humanized our research questions in ways that claims data cannot, sharpening our sense of the project’s potential impact. The interviews effectively created another community of clients for our research, reminding us of its importance and the need to do it well.

We periodically reflected on what aspects of the project would be clarified with more real-world examples or would be augmented by fresh perspectives; we formulated these into questions for DBSA’s audience. These outreach-and-feedback activities overlapped in purpose with our interview study in that we received many additional stories and insights gained through lived experience, adding to our research ideas, understanding, and motivation.

However, in the research realm, engagement feedback is classified very differently from study data (i.e., interviews and claims data). We could learn from these comments and analyze and summarize them but could never publish studies from them; they remained for internal use only and were not monitored under standard human subject protections. Also, feedback comments were not collected with the same rigor as the study data. Questions could be changed at the last minute on Facebook and the blog website, and anyone on the Internet could respond. Small sample sizes were not cause for concern. For researchers, this flexibility felt liberating. We floated open-ended questions and structured capsule surveys of varying lengths and topics (Table 1). Over time, we stopped targeting the questions solely to people with employer-sponsored insurance. Instead, we asked audience members to indicate their coverage situation and then examined whether their responses were consistent across different coverage types. Also, by the middle of the project, all social media outreaches used a capsule survey design. Each outreach presented about a dozen questions with a mixed format—typically, a multiple-choice item, followed by an invitation to tell us more in an open-response item. The multiple-choice format allowed audience members to respond quickly by using a cell phone if that was their preference. Although the feedback did not offer formal empirical validation of our study data, it provided us with “gut checks.” Community comments also generated hypotheses for potential exploration in future research (e.g., on the positive and negative aspects of employment for wellness or on cost conversations with providers).

Outreaches to the DBSA Peer Council or DBSA Advocacy Network were an excellent complement to DBSA’s social media presence, which is estimated to reach millions of individuals. Compared with the social media activities, these outreach efforts reached smaller numbers of individuals (about 300 and 2,500, respectively), identifiable to and having enduring relationships with DBSA. They included a higher proportion of people with professional experience of bipolar disorder (e.g., clinicians). Because they received multiple e-mails over time about our project, most people in these two groups had longitudinal familiarity with it. Peer Council and Advocacy Network outreaches thus fell somewhere on a continuum between the queries for the vast DBSA Facebook audience and for our intimate stakeholder advisor panel. Because we used e-mail to contact these two groups, we tried to limit the burden (i.e., frequency and length) of contacts and selected questions taking advantage of the deeper DBSA involvement of these groups. That said, even for Facebook- and blog-based outreaches, we felt that four per year was a prudent maximum to avoid audience fatigue with our project and interference with other DBSA priorities for community interaction.

Response tallies ranged from zero to several hundred per outreach, likely reflecting several factors, including which platform was used, level of interest in specific topics, whether people felt they could comment productively, occasional technical glitches, and fluctuations in competing demands for audience attention. We struggled more than anticipated to generate feedback about our project or its results. Response volume was greater when we asked for people’s own experiences. Negative reactions were rare (e.g., someone pointing to the role of academic experts in creating prominent problems in U.S. health care). Instead, we were encouraged by frequent, unsolicited “cheerleading” comments about the importance of research that involves peers and shines light on issues like health care affordability.

Dissemination of and engagement around our actual findings are ongoing; plans include joint appearances at scientific conferences and posting links to articles and lay synopses on DBSA platforms. Solid stakeholder involvement throughout a project ensures that there will be multiple partners at the project’s end who are excited to spread its messages and help identify potential opportunities for translation into practice.