Generalizability in the Family-to-Family Education Program Randomized Waitlist-Control Trial

Individuals were eligible to participate in the primary study if they were between ages 21 and 80, desired enrollment in the next FTF class regarding a family member or significant other, and spoke English. A total of 1,532 individuals who expressed interest in FTF were screened for study participation; 1,168 were found to be eligible. From this group, 318 individuals consented to participate in the randomized portion (RCT) of the overall study; 160 were randomly assigned to FTF, and 158 were assigned to the waitlist. An additional sample of 124 individuals from the 850 who had declined enrollment in the RCT and who were planning to take the class enrolled in the nonrandomized portion of the overall study. This decliner group was recruited approximately midway through the RCT when the need to address the modest consent rate was recognized; of the persons deemed eligible for the RCT, those who consecutively refused to participate in the RCT were offered enrollment as decliners until we achieved our target enrollment. Participants in both the RCT and decliner (nonrandomized) portions of the study completed identical baseline and follow-up interviews. We refer to three groups: decliners, RCT FTF participants, and RCT waitlist participants.

The institutional review board (IRB) at the University of Maryland approved all study activities; because interviews were conducted over the telephone, the IRB permitted consent to be obtained over the telephone after complete description of the study to the participants. Participants were recruited between March 2006 and September 2009.

This study considered three sets of variables. The first set includes all of the variables that were obtained in the participant interview. [This set is available online as a data supplement to this article.] Each variable was tested for inclusion in the propensity score analyses.

The second set of variables, described below, included those that differed between the decliner and RCT samples and therefore were used to generate the propensity scores. These included consumer race-ethnicity, consumer gender, living siblings of the consumer, family member income, family member marital status, consumer hospitalization in the past six months, and information about objective illness burden and required supervision obtained from the Family Experience Interview Survey (4).

Also in the second variable set, the Family Empowerment Scale provided measures of family, community, and service system empowerment (5); the Experience of Caregiving Inventory provided measures of positive aspects of the relationship, need of backup, problem with service system, stigma, and total positive and total negative subscales (6,7); and the NAMI Family Member Questionnaire provided measures of empowerment, coping with consumer’s illness, subjective burden and worry, and understanding of the mental health system (8). We also used the physical composite score of the 12-Item Short Form Health Survey (9), the Global Severity Index of the Brief Symptom Inventory–18 (BSI-18) (10,11), the percentage correct on the FTF mental illness knowledge test, and whether the family member self-reported having ever attended any formal NAMI educational programs (8).

The third relevant set of variables consisted of outcomes that improved with FTF for participation in the RCT. Knowledge was measured with a 20-item true-false test of factual information covering material drawn from the FTF curriculum that tapped general knowledge about mental illnesses (3). The five-item anxiety subscale of the BSI-18 measured psychological distress. It is designed for use primarily in nonclinical, community populations and has well-established reliability and validity (10,11). Family functioning was measured with the five-item problem-solving subscale of the Family Assessment Device, which evaluates family functioning and family relations. It is widely used in studies of family response to general medical illness and has well-established reliability and validity (12). The four-item acceptance dimension of the COPE measures emotion-focused coping (13) and family, service system, and community empowerment as described above. The analyses used both baseline and three-month measures of FTF outcomes.

Although this FTF study began with a traditional randomization process, the addition of the decliners transformed the combined RCT plus the decliner nonrandomized portion into a PRN study, a hybrid of randomized and observational study that is used in effectiveness analyses (14). Observational studies often suffer from selection bias; that is, people who receive treatment may differ systematically from those who do not. One approach is to match those in the treatment group with those in the control group, so that treatment effects can be attributed to the treatment rather than to baseline differences between the treated and untreated participants.

Propensity score matching is a useful statistical tool for adjusting for many covariates simultaneously (15). Matching participants on the unidimensional propensity score between those who receive or do not receive treatment has been shown formally to be statistically comparable to matching separately on each of the multiple covariates used to create the propensity score, but the former is preferable because separate matching becomes infeasible when there are more than a few covariates.

The propensity score is defined as the probability that a participant received the treatment versus the control condition, which is contingent on a set of potential measured confounders. Commonly, the propensity score is estimated with logistic regression to model the propensity of an individual to receive the treatment versus the control condition (16). However, to assess the generalizability of the treatment effect for the decliners, we chose to evaluate the propensity to be in the RCT versus in the decliner sample, in order to clarify characteristics that were associated with being a decliner. This approach has been shown to be mathematically equivalent to the more common method of estimating propensity to receive the treatment versus the control condition but provides more useful information for this study design (14).

First, propensity scores for each person were estimated with logistic regression. Variables were selected for the regression model by using bivariate statistics (chi square and t tests) to compare the RCT sample with the decliner sample for all baseline measures we collected in the RCT, including, for example, consumer and family demographic variables, family member–reported objective and subjective burden, coping, empowerment, family functioning, and other supports (3). Variables showing significant differences between the RCT and decliner samples (23 variables, p<.2) were then entered into a logistic regression model comparing the RCT sample and the decliner sample. Missing data were handled by using missing-data indicators (17). Propensity scores were calculated from this model. Participants with higher propensity scores had profiles more closely resembling RCT enrollees. All participants were placed into quintiles according to their propensity score (18). We then examined the distribution of participants by propensity score quintile for the randomized FTF, randomized waitlist, and decliner samples.

A sample of covariates by quintile and group is listed in Table 1. Analyses of variance (ANOVAs) and chi square tests were used to assess heterogeneity across quintiles, with respect to each selected covariate.

Our primary goal was to determine whether the estimate of the effect of FTF versus waitlist observed in the RCT generalized to the decliner sample. We planned to derive estimates of FTF’s impact on the outcomes of knowledge, family problem solving, empowerment, acceptance aspects of coping, anxiety, and subjective burden (worry) for the decliner sample and compare these with the estimates of benefits of FTF observed in the published RCT (3). Our approach was to build on the internally valid estimates of benefit derived from the RCT and enhance external validity by weighting the estimates of effectiveness observed in the RCT to fit the distribution of the propensity score quintiles for the baseline decliner population.

Although similar to age-adjusted estimates that are commonly used in life tables, propensity score matching enabled us to adjust for many covariates simultaneously. Estimates of the effectiveness of RCT FTF versus RCT waitlist were calculated for decliners. [Calculations and corresponding standard errors are available online as a data supplement to this article.] This approach provided an estimate of how individuals similar to the decliners would do if they received FTF versus how they would do if they could also be observed after assignment to (hypothetically) the waitlist. We next used these estimates and standard errors to calculate 95% confidence intervals and corresponding effect sizes regarding FTF versus waitlist effectiveness for the decliner sample. These confidence intervals were then compared with the RCT estimates.

Of the total sample of 409 consumers whose race was reported by family member participants, 260 (64%) were white, 11 (3%) were Asian, 105 (26%) were black, seven (2%) were Hispanic, and 26 (6%) were other. When compared with the decliners, participants in the RCT were significantly more likely to report having family income greater than $50,000 per year (χ²=4.39, df=1, p<.036), to report that the consumer required more assistance in daily living (t=2.13, df=435, p<.033), and to report that the family member had a psychiatric hospitalization in the past six months (χ²=11.35, df=1, p<.001). With respect to study outcomes, decliners had less knowledge about mental illness (t=2.43, df=435, p<.016) and less community empowerment (t=2.51, df=434, p=.012) at baseline.

Figure 1 provides an example of how the propensity score approach allowed the decliner sample to be matched with the RCT sample when one of the variables contributed to the propensity score. An important aspect of matching groups via propensity score quintiles was to examine the percentage of participants within each quintile by sample. Specifically, the critical question was whether there was a comparable percentage of each sample (decliner, RCT waitlist, or RCT FTF) for any particular variable represented in each quintile.

The figure shows that the variable, percentage of participants whose family member experienced a psychiatric hospitalization in the past six months, decreased from quintile 1 (most like the decliners) to quintile 5 (least like the decliners); in other words, compared with the RCT participants, a higher percentage of the decliners tended to have a family member who had been hospitalized, consistent with the bivariate analysis presented above. However, within each quintile, the percentage was generally similar across the three samples. It is important to note that matching does not require perfect balance. Other covariates were also effectively balanced by propensity score quintile matching. Table 1 gives baseline covariate and outcome data by group (RCT waitlist, RCT FTF, or decliner) and propensity score quintile. ANOVAs and chi squares across quintiles of all covariates demonstrated significant heterogeneity.

Table 2 provides mean outcome levels by group and propensity score quintile for three-month outcomes. The three groups appeared to be comparably distributed within each quintile, as shown in Figure 2 for the knowledge test. Table 3 presents the effectiveness estimates with confidence intervals and effect sizes derived from the RCT for comparison of individuals receiving FTF with individuals on the waitlist. It also provides estimates of the RCT FTF versus RCT waitlist effect for the decliner sample—estimates that reflect the capacity of the propensity scoring process and assignment of quintiles to predict what the effects of FTF versus waitlist would have been for the decliners. We note that the effect sizes were remarkably similar despite the selection differences for being in the decliner population. For example, with respect to knowledge, the effect size observed in the RCT was .31. The estimated effect size for the decliner population was .29. Also, the confidence intervals had a high degree of overlap.