Clinician's Guide to Evaluating and Developing eHealth Interventions for Mental Health

Intervention mapping (37) is a framework designed to guide development of health promotion interventions. It was developed as a response to the lack of structure in health promotion program development. It seeks to explicitly connect theory and prior findings to the intervention development process. It does so through development of an initial model of the problem, followed by a detailed logic model of how change will be effected, using theory‐based intervention approaches that are organized into a coherent whole that will maximize implementation and scalability. Intervention mapping is an influential and commonly used approach that promotes rigorous connections between the identified problem and intervention design, with thorough integration of theory and the available literature. For example, Brendryen and colleagues (38) used intervention mapping to develop an online alcohol intervention. The goal of the intervention was to help clients drink less by developing self‐regulation skills to maintain change over time and adjust their coping during moments of risk. The authors used strategies such as screening with brief personalized feedback, content “tunneling” (i.e., delivering content in a fixed sequence, thereby reducing user burden), goal setting (i.e., having clients determine drinking goals ahead of time and report on those goals), and just‐in‐time therapy via prerecorded audio dialogue. The authors suggested that the clear program blueprint makes the intervention more coherent and will help clarify results from future program evaluations.

BIT is a relatively recent model that, unlike intervention mapping, was designed specifically to inform development of eHealth interventions (39). The BIT model combines technological elements and principles of behavior change and integrates the evaluation approaches typically taken by software developers with those of behavioral science. Broadly, the BIT model asks developers to specify why (specific aims such as reducing depression), conceptually how (specific behavior change strategies, such as motivation enhancement), what (specific technical elements, such as tailored messaging), technically how (e.g., technical platform, design), and when (frequency and timing of the intervention). This framework is then integrated with a technological framework, referred to as BIT‐tech, designed to guide software developers. BIT‐tech is composed of four components: the profiler (who collects data to define the user and the environment), the intervention planner (who specifies the exact intervention element for each timepoint), the intervention repository (which stores all available intervention components), and the user interface (the front end that presents a specific intervention element to the user at a particular time).

Ranney and colleagues (40, 41) used the BIT model to develop an 8‐week texting intervention to reduce violence and depression among adolescents being discharged from the emergency department. The tailored text messages reinforced cognitive reappraisal, emotion self‐regulation, and the use of self‐efficacy skills. On the basis of prior work, the developers created three message groups: one for girls with low‐violence behavior, one for boys with low‐violence behavior, and one for adolescents of both genders with high violence behavior. Participants received daily messages that were based on their group and their current mood rating. They could request additional messages on demand. The three groups received similar message content that differed in terms of behavioral activation activities, types of stressors, and language used in the messages.

An alternative approach to eHealth intervention development was offered by Yardley and colleagues (42). They called this approach “person centered” to emphasize the perspectives of the end users in the development of the intervention. In the PBA, a wide range of consumers provide perspectives on what parts of the program will be used, by whom, and in what context. The PBA focuses on two key steps: in‐depth qualitative research with representative end users that begins early in the development and continues throughout all stages, including implementation of the intervention; and identification of guiding principles that inform the selection of behavior change techniques as well as choices about how the technology will embody those techniques. The PBA is a sequential and practical model that emphasizes continuing input from end users to inform development and implementation. For instance, Yardley and colleagues (42) provide an example of a common intervention feature that was rated poorly by a group of end users. Mobile health interventions sometimes use a person's location to send tailored messages. However, end user focus groups tended to be skeptical about this strategy, believing that the location‐sensing was sometimes inaccurate or misconstrued and that location‐triggered messages might sometimes have an iatrogenic affect (e.g., by reminding the person that he or she is at risk).

The MOST approach defines three key phases of intervention development: preparation, optimization, and evaluation (43, 44). In the preparation phase, the MOST approach uses activities that are similar to those in intervention mapping and BIT, for instance clarifying the aims of the intervention, gathering relevant theoretical and empirical information from the literature, and developing a theoretical model. However, the MOST approach goes beyond other models, applying engineering approaches to evaluate components of the intervention that were identified during the preparation stage, typically by using one or more fractional factorial trials. The logic for these factorial trials is that engineering and design comprise too many options to be able to test out every possible combination. Thus, the developer identifies key components and tests the different options, often by using lower scientific standards for group assignment (e.g., convenience assignment), size (e.g., fewer subjects), and statistical analysis (e.g., effect sizes rather than statistical significance). The goal is to generate the most effective prototype without testing every possible combination. Collins and colleagues (43) give an example of using the MOST approach to develop a smoking cessation program that contained theoretically derived units, such as outcome expectation messages, efficacy expectation messages, message framing, testimonials, exposure schedule, and message sourcing. The design process would help to decide which components to focus on and the amount of content in each unit. This optimization stage is followed by an evaluation—in a traditional randomized trial—of an intervention consisting of the strongest subset of components. The MOST approach is unique from the models described thus far, in that it incorporates empirical evidence (again, largely gathered through factorial trials) as a key step between intervention design and testing in a randomized trial.

Although not specific to the design of eHealth interventions, the user‐centered design (UCD) approach (45) is a frequent part of the broader intervention design models reviewed above. UCD puts user needs at the center of the software design process, from its earliest stages and throughout development. UCD maintains a consistent focus on who will be using the intended software, for what reasons, and in what context to ensure that the final product fits the context and meets the needs of its target audience. eHealth intervention apps that have incorporated UCD from the beginning may be more likely to be seen by patients as relevant, easy to use, and intuitive, in part because of careful analysis of the users' needs and the contexts in which they might use the technology.

Similarly, although not a design approach in itself, the behavior change technique taxonomy (46) has become an important resource for many eHealth intervention designers. This taxonomy was designed to create a common language for describing behavioral intervention techniques, in part through input from a broad range of experts who sought to define and cluster all known approaches. The current taxonomy consists of 93 discrete behavior change techniques arranged under 16 clusters. For example, cluster 1, scheduled consequences, includes techniques such as differential reinforcement and shaping, and cluster 8, feedback and monitoring, includes techniques such as biofeedback and self‐monitoring of behavior.

The models discussed above are well known in the field, and thus it is important to be aware of them when evaluating or developing eHealth interventions. However, four points should be highlighted with respect to these models. First, although all of them can provide important guidance in intervention development, they vary widely in terms of level of guidance and detail. Second, these approaches offer only a framework; the difficult work of designing the best possible intervention, with incomplete guidance from an imperfect literature, remains. Third, none of these models has been tested with the level of rigor needed to conclude that their use will lead to a more effective intervention. In fact, only the MOST approach includes empirical evaluation as part of the development process. Fourth, although these approaches include a great deal of expert advice regarding intervention development, there is certainly additional general advice to be found. For example, Michie and colleagues (47) synthesized the consensus of eHealth intervention experts regarding the development and evaluation of eHealth interventions. Many of their recommendations (e.g., adopt engineering methodologies, such as factorial trials; use person‐centered and qualitative approaches throughout the development process; and link theory to behavior change techniques) are reflected in the above models. However, many other of Michie et al.'s recommended best practices (e.g., use Bayesian approaches to improve predictive modeling capabilities, evaluate cost‐effectiveness, ensure compliance with national standards) can be characterized as general advice that should be considered regardless of the development framework being used.

Single‐case research designs (also called single‐subject, small‐N, or N‐of‐one designs) are dramatically underused (48), despite having multiple advantages—particularly during the early stages of eHealth intervention design (49). Although they do not allow the causal inference afforded by randomized trials, these approaches can strongly suggest causality and as such are included as evidence in some evaluations of whether an intervention meets criteria for being evidence‐based (e.g., the Federal What Works Clearinghouse) (50). In their simplest form, single‐case designs involve systematic provision and removal of an intervention along with continuous assessment in an attempt to identify associated fluctuations in an outcome. For example, if a study participant reliably shows higher fruit and vegetable intake on days when she receives multiple text‐message reminders compared with days on which those reminders are not provided, the efficacy of those reminders is supported. This evidence is stronger if it remains true across multiple on‐off trials with irregular schedules (e.g., 4 days on, 2 days off; 2 days on, 1 day off). Similarly, if the pattern holds true with four additional participants, the efficacy of the reminders would be further supported.

Multiple baseline designs are an alternate single‐case approach. In this approach, two or more behaviors are targeted (e.g., depression, smoking cessation), but an intervention is introduced for one behavior at a time to see if the change in each outcome will coincide with introduction of the intervention targeting that behavior. Multiple baseline designs are powerful ways to provide evidence of causality but are most appropriate when multiple discrete behavioral outcomes can be addressed sequentially and when a stable baseline can be demonstrated prior to the introduction of the eHealth intervention. Regardless, this and other single‐case or small‐N designs are valid approaches that should not be overlooked. See Dallery et al. (49, 51) for a full discussion of how single‐case designs can be used to evaluate eHealth interventions.

Extending the single‐case or small‐N approach to larger samples can also yield powerful evidence in support of eHealth interventions, even without the use of a control condition. Interrupted‐time‐series approaches seek to imply causality through showing a disruption in a stable baseline that coincides with introduction of an intervention (52). Thus, a simple single‐group pre‐post design (AB, in which A is the pretest measure and B is the posttest measure) is extended to include multiple pre‐ and postintervention observations (e.g., AAAAABBBBB). If an otherwise stable baseline value suddenly changes after introduction of the intervention, and if that change is sustained throughout multiple postintervention observations, then the results strongly suggest that changes in the outcome measure can be attributed to the intervention. The stepped‐wedge design (53, 54) is an extension of this approach in which individual interrupted‐time‐series studies are conducted separately within multiple discrete sites (such as outpatient clinics or schools). If an eHealth intervention is introduced at those sites in a staggered way, with order being randomly selected, and if a change in the outcome of interest is consistently demonstrated following introduction of the intervention (but not before), then there is strong evidence for causality.

As with other interventions, randomized trials remain the gold standard for evaluation of the efficacy and effectiveness of eHealth interventions. However, as is increasingly being highlighted (55, 56), the rigor of randomized trials is highly variable. Important factors include preregistration of trials, consistency of primary outcomes and analytic methods with those outlined during preregistration, use of intent‐to‐treat analyses, adequate sample size, control for multiple comparisons, use of reliable and valid measures, blinding of evaluators, and evaluation of potential bias. A rigorous trial should also use appropriate methods to address attrition, such as multiple imputation or full information maximum likelihood techniques as opposed to listwise deletion (simply dropping cases with missing outcome data), last observation carried forward (using data from the last available observation), or presuming failure among those lost to attrition (57, 58).

Choice of control and/or comparison group is also a key design decision, with each option carrying a unique set of pros and cons. Disagreement regarding this choice, in particularly as applied to eHealth interventions, has been significant enough that the National Institutes of Health (59) formed an expert panel to develop a framework for selection of comparators in trials of health‐related behavioral interventions. That panel developed the pragmatic model for comparator selection in health‐related behavioral trials, which outlines that selection of comparators should take into account the purpose of the trial (i.e., whether the goal is to identify whether the intervention has an effect of any kind, how that effect compares with the current standard of care, or how and/or why the intervention works); the developmental stage of the intervention (e.g., early or late); and the research context (e.g., the availability of alternative services in a given area) (59). For example, although an initial study of an eHealth intervention may use a very minimal comparator, such as a waitlist control or other inactive control condition, later studies may require a more formidable comparator, such as an intervention delivered by a therapist.

Randomized trials of eHealth interventions carry other unique considerations as well. For example, Mohr and colleagues (60, 61) have highlighted some of the reasons the rapidly changing landscape of computer‐based intervention technologies is not suited to the typical model of conducting multiple years‐long trials of a particular intervention. In a traditional model, the intervention being tested is a medication or a therapeutic approach that changes little or not at all, either in its own makeup or in how it is altered to fit a particular setting or sample. In contrast, part of the advantage of technology is that it evolves rapidly as new information or needs are identified. Mohr and colleagues (62) suggested a focus on hybrid implementation‐efficacy trials that are designed specifically to address the needs of a particular setting (30) and testing of intervention principles rather than a specific, locked‐down intervention.