Postmarket Surveillance for Effective Regulation of Digital Mental Health Treatments

Randomized controlled trials (RCTs) consistently show that digital mental health treatments (DMHTxs), which offer patients mobile or Web apps supported by low-intensity coaching, can be highly effective in managing common mental health problems such as depression and anxiety. On the basis of these data, the Banbury Forum, representing health care systems, payers, employers, researchers, and others, recommended that coached DMHTxs be available to all Americans with these mental health conditions and that reimbursement mechanisms necessary for broad DMHTx implementation be established (1).

Efforts are under way to integrate DMHTxs into the U.S. health care system in value-based care systems and some states (2). Bills that direct the Centers for Medicaid and Medicare Services (CMS) to reimburse digital therapeutics, defined as tools having U.S. Food and Drug Administration (FDA) certification, have been introduced into the U.S. Senate (S723) and House (H.R.1458). CMS has begun to issue Healthcare Common Procedure Coding System codes for billing for some DMHTxs. Thus, the outlines of reimbursement and regulatory processes for DMHTxs are emerging.

At the same time, suspicion of DMHTxs’ effectiveness and safety has grown. Coverage in the popular press increasingly highlights problems and abuses, questioning the value of DMHTxs (3). DMHTxs have had challenges in the research-to-practice transition similar to those of many other treatments. DMHTxs commonly have lower engagement and adherence when deployed in routine clinical settings rather than in RCTs, and thousands of DMHTxs exist that have no evidence of clinical effectiveness whatsoever. Of note, some DMHTxs, under certain circumstances, can produce adverse events. A large implementation trial found that a coached digital program teaching evidence-based self-management skills to reduce the risk for suicidal behaviors resulted in significant increases in fatal and nonfatal self-harm relative to care management and treatment as usual (4). Although the DMHTx showed good engagement of patients in early research, it had very low rates of use in real-world deployment. It is unclear whether the DMHTx itself or the implementation (e.g., receipt of a program that users perceived as unsuitable for managing a severe condition) increased self-harm. Regardless of the cause, implementation of the DMHTx in treatment settings resulted in far less engagement than previously seen in research settings, was not effective, and increased risk for fatal and nonfatal self-harm.

The FDA is being thrust into the role of regulating DMHTxs. Because most DMHTxs are designed primarily to support behavior change to treat, manage, or prevent the targeted mental health condition, they are currently classified as low risk for adverse events. This places DMHTxs under “enforcement discretion,” meaning that the FDA elects not to require review and certification. However, current legislation defines digital therapeutics as having FDA clearance or approval, placing the FDA front and center for review and certification of these therapeutics. If DMHTxs require FDA clearance for reimbursement, the FDA will be reviewing these products, even if they are categorized as low risk for adverse events. The FDA’s role in DMHTx clearance raises the question, How effective are the FDA’s methods for ensuring the safety and effectiveness of DMHTxs?

FDA review can be de novo, entailing review of RCT data, or via the 510(k) premarket notification pathway, in which the applicant must demonstrate similarity of the treatment to an existing predicate or precursor supported by RCT data. Accordingly, implicitly or explicitly, the FDA relies on RCT data. RCT findings can have little relationship to the effectiveness or use of these products in routine clinical practice settings. Some of this lack of generalizability from RCTs to real-world deployment is due to factors also seen in trials of pharmaceuticals and devices, such as narrow participant inclusion criteria, consenting and assessment processes that select for patients with greater motivation, and ongoing involvement of research staff that enhances adherence to treatment. But DMHTxs also have unique factors that are likely to further increase sampling biases and decrease generalizability. The ease with which participants can be enrolled in trials facilitates selection from much larger groups. For example, in our center, although we do not have consistent records of the number of people who see trial recruitment advertisements, only 2.6% of those who go to online eligibility screeners eventually become enrolled. Even though this “denominator inflation” is rarely reported, it is likely substantial. In one of the few studies that did report this factor, only 406 participants (<0.01%) were recruited from 8.6 million insurance members (5). In addition, DMHTxs may have properties that heighten the effects of selection biases. Adherence to medications or devices is already challenging for patients, and DMHTxs require even greater levels of effort and motivation. Typically, patients must remain engaged with the DMHTx and implement psychological and behavioral strategies in their lives for many weeks to gain benefits from the intervention.

RCTs provide necessary evidence of efficacy of an intervention under research conditions; however, RCTs are not sufficient to establish effectiveness of DMHTxs in routine clinical practice. Low generalizability of RCTs to deployment of a treatment in various settings may manifest in several ways: the DMHTx may be ineffective and unengaging for most people, produce harmful effects for some people because most DMHTx RCTs are too small to observe infrequent adverse events, or be inadequately implemented (e.g., have low coaching quality or poor consideration of how and when in the care continuum the DMHTx is offered), rendering a DMHTx ineffective or harmful.

Postmarket surveillance is currently used by the FDA to monitor harms and effectiveness of a treatment after approval or clearance. Initially, this surveillance approach relied on reports to the FDA. More recently, the FDA has created the Sentinel product safety surveillance system, which monitors postmarket effectiveness and safety of drugs and devices by using health care data from distributed insurance claims and electronic health record (EHR) databases containing data from 240 million people, along with patient reports. Although these data may be helpful in identifying some adverse events, such as self-harm, they would not capture other problems such as lack of engagement. DMHTxs offer an additional and fortuitous stream of potentially accessible app usage data as well as in-app assessments. These data, if combined with available EHR and claims data, would provide a comprehensive picture of app use, symptom change, occurrence of adverse events, and linkage of those outcomes to DMHTx use.

Postmarket surveillance platforms for DMHTxs would contribute to the safety, effectiveness, and viability of DMHTx services in several ways. Postmarket surveillance would identify ineffective DMHTxs for which RCT findings of effectiveness are not generalizable; identify harms that are too infrequent to be detected in RCTs; provide information on implementation failures, such as changes to coaching protocols or misplacement of the treatment in the care continuum; monitor the effect of feature and interface changes to DMHTxs; assist in disaggregating DMHTx product failure from implementation failures through examination of variability in outcomes across care systems; provide information to health care systems and payers that supports decision making about which products to purchase, retain, or eliminate from services (6); and enable value-based contracting (7).

DMHTxs have the potential to greatly improve access to mental health care and to use mental health providers more efficiently, given their low number and capacity. As clinicians and administrators move toward reimbursement and integration of DMHTxs into the U.S. health care system, it is in the interest of all parties to ensure that these services are safe and effective. Integration of DMHTx postmarket surveillance into the FDA Sentinel system would provide a critical regulatory layer that would strengthen the FDA clearance and approval processes. Postmarket surveillance tools deployed through health care systems to provide information on the use, benefits, and harms of DMHTxs could support decision making. Postmarket surveillance to ensure that DMHTxs are safe, effective, useable, and properly deployed will also help instill public confidence and protect the interests of all parties, including patients, health care providers, payers, and, in the long run, the industry for digital interventions.