Skip to main content

Abstract

Objective

Randomized controlled trials (RCTs) may have limited generalizability for the community when a high proportion of individuals refuse randomization or otherwise do not participate—a not uncommon phenomenon. A randomized waitlist-control trial of the Family-to-Family (FTF) education program, a 12-week course offered by the National Alliance on Mental Illness for family members of adults with mental illness, was previously reported. This study assessed whether the RCT-derived estimates of effectiveness of FTF were generalizable to individuals who participated in FTF but declined participation in the RCT.

Methods

Propensity score matching was used to create five quintiles, each containing scores for individuals in FTF or waitlist conditions and for decliners; scores were matched on multiple baseline characteristics (N=442) within each quintile. Effectiveness estimates, with standard errors, were derived for the decliner population on the basis of effectiveness estimates derived from participants in the RCT; estimates were weighted to the baseline distribution of quintiles for the decliners.

Results

For each outcome, estimates of the effect sizes observed in the RCT were very similar to the effect sizes observed for the decliner population; confidence intervals also had a high degree of overlap.

Conclusions

This study suggests that the benefits of FTF observed in the RCT are generalizable to the group of individuals who declined RCT participation, providing further evidence of FTF’s effectiveness. Propensity score matching was a useful statistical tool for addressing selection bias resulting from high rates of nonconsent in randomized waitlist-control trials.
Randomized controlled trials (RCTs) are considered the most rigorous test of an intervention’s effectiveness. The internal validity of RCTs gives confidence that study findings can be attributed to the differences between the experimental and control conditions. However, RCTs may have limited external validity (generalizability) for the community of potential users of the program being tested if a high proportion of individuals refuse randomization or otherwise do not participate—a not uncommon phenomenon (1).
Individuals may decline to consent to random assignment to a treatment if it differs greatly from those currently received or familiar (medication versus psychotherapy, for example) (1). A similar situation arises when an RCT control group is placed on a waitlist for an experimental intervention; some people may withhold consent for random assignment if they are unwilling to wait for the experimental treatment. In prior work, we proposed the parallel randomized and nonrandomized (PRN) clinical trial design (also known as the partially randomized preference design) as a solution for this problem (1,2). Most RCTs exclude individuals who do not consent to randomization. However, in the PRN trial design, those who consent to randomization are randomly assigned, and those who do not are assigned to their treatment of choice and are followed in a manner similar to those in the RCT. This design can enhance generalizability by enabling the estimation of effectiveness for those who decline randomization (1).
We reported the results of an RCT that tested the effectiveness of the National Alliance on Mental Illness (NAMI) Family-to-Family (FTF) program, a 12-week course for family members of adults with mental illness (3). In this study, 318 consenting participants in five Maryland counties and Baltimore City were randomly assigned to participate in FTF immediately or to wait at least three months for the next available class and to freely use in the meantime any other NAMI, community, or professional supports. We found that FTF participants had significantly greater improvements in coping, family problem solving, knowledge and distress. However, less than one-third of the potential sample was willing to consider study participation. The most common reason for declining was unwillingness to undergo random assignment because of the potential delay in FTF participation (3).
The study’s consent rate created a concern that the RCT participant sample was not representative of individuals who generally participate in FTF. To address this, we offered nonrandomized study participation to a cohort of 124 individuals who refused to enroll in the RCT and who were planning to take the class immediately. We evaluated these individuals (called the “decliner” sample) according to the same schedule as the participants in the RCT. The aims of the study were to apply innovative statistical methods to determine whether the findings of the RCT could generalize to the sample of decliners and therefore potentially to the population of individuals who enroll in FTF through usual NAMI programming.

Methods

Participants

Individuals were eligible to participate in the primary study if they were between ages 21 and 80, desired enrollment in the next FTF class regarding a family member or significant other, and spoke English. A total of 1,532 individuals who expressed interest in FTF were screened for study participation; 1,168 were found to be eligible. From this group, 318 individuals consented to participate in the randomized portion (RCT) of the overall study; 160 were randomly assigned to FTF, and 158 were assigned to the waitlist. An additional sample of 124 individuals from the 850 who had declined enrollment in the RCT and who were planning to take the class enrolled in the nonrandomized portion of the overall study. This decliner group was recruited approximately midway through the RCT when the need to address the modest consent rate was recognized; of the persons deemed eligible for the RCT, those who consecutively refused to participate in the RCT were offered enrollment as decliners until we achieved our target enrollment. Participants in both the RCT and decliner (nonrandomized) portions of the study completed identical baseline and follow-up interviews. We refer to three groups: decliners, RCT FTF participants, and RCT waitlist participants.
The institutional review board (IRB) at the University of Maryland approved all study activities; because interviews were conducted over the telephone, the IRB permitted consent to be obtained over the telephone after complete description of the study to the participants. Participants were recruited between March 2006 and September 2009.

Variables

This study considered three sets of variables. The first set includes all of the variables that were obtained in the participant interview. [This set is available online as a data supplement to this article.] Each variable was tested for inclusion in the propensity score analyses.
The second set of variables, described below, included those that differed between the decliner and RCT samples and therefore were used to generate the propensity scores. These included consumer race-ethnicity, consumer gender, living siblings of the consumer, family member income, family member marital status, consumer hospitalization in the past six months, and information about objective illness burden and required supervision obtained from the Family Experience Interview Survey (4).
Also in the second variable set, the Family Empowerment Scale provided measures of family, community, and service system empowerment (5); the Experience of Caregiving Inventory provided measures of positive aspects of the relationship, need of backup, problem with service system, stigma, and total positive and total negative subscales (6,7); and the NAMI Family Member Questionnaire provided measures of empowerment, coping with consumer’s illness, subjective burden and worry, and understanding of the mental health system (8). We also used the physical composite score of the 12-Item Short Form Health Survey (9), the Global Severity Index of the Brief Symptom Inventory–18 (BSI-18) (10,11), the percentage correct on the FTF mental illness knowledge test, and whether the family member self-reported having ever attended any formal NAMI educational programs (8).
The third relevant set of variables consisted of outcomes that improved with FTF for participation in the RCT. Knowledge was measured with a 20-item true-false test of factual information covering material drawn from the FTF curriculum that tapped general knowledge about mental illnesses (3). The five-item anxiety subscale of the BSI-18 measured psychological distress. It is designed for use primarily in nonclinical, community populations and has well-established reliability and validity (10,11). Family functioning was measured with the five-item problem-solving subscale of the Family Assessment Device, which evaluates family functioning and family relations. It is widely used in studies of family response to general medical illness and has well-established reliability and validity (12). The four-item acceptance dimension of the COPE measures emotion-focused coping (13) and family, service system, and community empowerment as described above. The analyses used both baseline and three-month measures of FTF outcomes.

Statistical approach

Although this FTF study began with a traditional randomization process, the addition of the decliners transformed the combined RCT plus the decliner nonrandomized portion into a PRN study, a hybrid of randomized and observational study that is used in effectiveness analyses (14). Observational studies often suffer from selection bias; that is, people who receive treatment may differ systematically from those who do not. One approach is to match those in the treatment group with those in the control group, so that treatment effects can be attributed to the treatment rather than to baseline differences between the treated and untreated participants.
Propensity score matching is a useful statistical tool for adjusting for many covariates simultaneously (15). Matching participants on the unidimensional propensity score between those who receive or do not receive treatment has been shown formally to be statistically comparable to matching separately on each of the multiple covariates used to create the propensity score, but the former is preferable because separate matching becomes infeasible when there are more than a few covariates.
The propensity score is defined as the probability that a participant received the treatment versus the control condition, which is contingent on a set of potential measured confounders. Commonly, the propensity score is estimated with logistic regression to model the propensity of an individual to receive the treatment versus the control condition (16). However, to assess the generalizability of the treatment effect for the decliners, we chose to evaluate the propensity to be in the RCT versus in the decliner sample, in order to clarify characteristics that were associated with being a decliner. This approach has been shown to be mathematically equivalent to the more common method of estimating propensity to receive the treatment versus the control condition but provides more useful information for this study design (14).
First, propensity scores for each person were estimated with logistic regression. Variables were selected for the regression model by using bivariate statistics (chi square and t tests) to compare the RCT sample with the decliner sample for all baseline measures we collected in the RCT, including, for example, consumer and family demographic variables, family member–reported objective and subjective burden, coping, empowerment, family functioning, and other supports (3). Variables showing significant differences between the RCT and decliner samples (23 variables, p<.2) were then entered into a logistic regression model comparing the RCT sample and the decliner sample. Missing data were handled by using missing-data indicators (17). Propensity scores were calculated from this model. Participants with higher propensity scores had profiles more closely resembling RCT enrollees. All participants were placed into quintiles according to their propensity score (18). We then examined the distribution of participants by propensity score quintile for the randomized FTF, randomized waitlist, and decliner samples.
A sample of covariates by quintile and group is listed in Table 1. Analyses of variance (ANOVAs) and chi square tests were used to assess heterogeneity across quintiles, with respect to each selected covariate.
Table 1 Baseline covariates of participants in a randomized controlled trial (RCT) of the National Alliance on Mental Illness (NAMI) Family-to-Family psychoeducation, by propensity score quintiles
 OverallQuintile 1Quintile 2Quintile 3Quintile 4Quintile 5
Covariate and groupTotal NN%Total NN%Total NN%Total NN%Total NN%Total NN%
RCT waitlist for Family-to-Family                  
 White consumers1408863181372261350312684301860351851
 Married or living as if married1408359181372261765311755302067351646
 Family income >$50,0001409568181372262181312271302170351851
 Objective daily living assistance rating (M±SD)a1401.2±1.0 181.3±1.0 261.6±1.1 311.4±1.0 301.0±.9 35.9±.9 
 Attended any formal NAMI educational programs14033241884426415316193082735720
 Male consumers1407251181161261558311858301447351440
 Any psychiatric hospitalization in past 6 months1404230181056261558319293031035514
 Knowledge (M±SD % correct)14057.3±17.3 1843.9±20.6 2655.3±14.9 3160.4±17.7 3060.5±12.1 3560.1±18.0 
 Problem solving (M±SD)b13613.2±2.9 1813.8±4.2 2613.0±2.5 3012.8±2.3 3013.1±2.4 3213.3±3.3 
 Anxiety (M±SD)c14053.2±10.1 1852.7±8.5 2649.8±9.2 3152.3±9.5 3054.1±10.7 3556.1±11.0 
 Global Severity Index (M±SD)d14053.2±9.8 1851.8±6.2 2649.6±8.5 3151.9±10.0 3052.6±9.1 3558.3±11.0 
 Empowerment in aspect of family (M±SD)e1403.4±.7 182.9±.7 263.4±.7 313.3±.6 303.3±.6 353.5±.7 
 Empowerment in aspect of service (M±SD)e1403.1±.9 182.6±.8 263.3±1.1 313.0±.9 302.9±.8 353.5±.9 
 Empowerment in aspect of community (M±SD)e1402.4±.8 181.9±.5 262.4±.8 312.2±.6 302.3±.5 352.9±.9 
 Acceptance (M±SD)f13912.4±2.6 1811.7±3.1 2612.8±2.3 3012.5±2.2 3012.5±2.3 3512.5±2.9 
 Depression (M±SD)g13910.1±8.7 178.5±6.5 268.5±6.3 318.4±7.0 309.5±8.1 3514.0±11.8 
 Worry (M±SD)h1401.8±.5 181.6±.5 261.8±.5 311.7±.4 301.9±.5 351.8±.5 
 Subjective burden (M±SD)h1402.6±.5 182.4±.5 262.5±.5 312.7±.5 302.7±.5 352.7±.5 
RCT Family-to-Family participants                  
 White consumers1529059181478211676342368412151381642
 Married or living as if married1529764181583211676342676412254381847
 Family income >$50,00015210468181478211990342162412971382155
 Objective daily living assistance rating (M±SD)a1521.1±.9 181.2±1.0 211.4±.9 341.0±.9 41.9±.9 381.0±1.0 
 Attended any formal NAMI educational programs152181218317211534618415123838
 Male consumers1528455181478211571341956412151381539
 Any psychiatric hospitalization in past 6 months152513418105621104834123541143438513
 Knowledge (M±SD % correct)15258.3±17.8 1844.4±18.2 2161.0±12.4 3455.9±18.5 4161.7±18.6 3861.9±15.7 
 Problem solving (M±SD)b15112.8±2.9 1814.0±3.3 2113.2±3.0 3412.4±2.8 4012.3±2.7 3812.8±2.9 
 Anxiety (M±SD)c15252.6±9.2 1851.6±10.1 2148.7±9.9 3454.4±10.2 4151.7±8.1 3854.5±8.0 
 Global Severity Index (M±SD)d15251.9±9.3 1850.3±11.6 2148.6±8.9 3453.6±9.1 4150.7±7.9 3854.4±9.3 
 Empowerment in aspect of family (M±SD)e1523.5±.6 183.0±.7 213.4±.7 343.4±.6 413.6±.5 383.6±.5 
 Empowerment in aspect of service (M±SD)e1523.2±.9 182.7±1.0 213.0±1.0 343.3±.9 413.3±.7 383.4±.8 
 Empowerment in aspect of community (M±SD)e1522.6±.8 182.0±.9 212.5±.6 342.3±.7 412.8±.7 382.8±.8 
 Acceptance (M±SD)f15112.9±2.3 1812.5±2.5 2113.8±1.4 3312.4±2.7 4112.9±2.3 3813.0±2.2 
 Depression (M±SD)g1498.6±7.2 188.7±6.2 216.1±6.9 339.2±6.0 398.5±8.2 389.3±7.7 
 Worry (M±SD)h1521.8±.5 181.7±.4 211.8±.4 341.8±.5 411.9±.6 381.8±.5 
 Subjective burden (M±SD)h1522.6±.5 182.5±.3 212.6±.5 342.7±.5 412.7±.4 382.6±.5 
Decliners (no RCT random assignment)                  
 White consumers1178270463474352983171271115458225
 Married or living as if married117827046357635267417953116558675
 Family income >$50,0001179178464291352777171376117648225
 Objective daily living assistance rating (M±SD)a1171.3±1.0 461.6±1.0 351.4±1.1 171.2±.9 11.7±.6 8.6±.7 
 Attended any formal NAMI educational programs117302646183935617174241121880
 Male consumers1177463463372352160171059115458563
 Any psychiatric hospitalization in past 6 months117585046367835164617529111980
 Knowledge (M±SD % correct)11754.0±17.5 4650.2±18.6 3557.3±17.5 1760.8±11.7 1152.4±18.8 848.8±14.8 
 Problem solving (M±SD)b10913.1±2.4 4312.7±2.2 3113.2±2.2 1713.2±3.1 1014.6±2.5 812.9±2.6 
 Anxiety (M±SD)c11751.6±10.2 4651.7±11.4 3549.7±9.3 1752.9±6.8 1155.3±8.9 851.5±14.3 
 Global Severity Index (M±SD)d11750.7±10.7 4650.2±12.4 3549.4±9.2 1751.3±7.3 1155.3±10.3 851.8±13.6 
 Empowerment in aspect of family (M±SD)e1173.3±.6 463.3±.5 353.3±.6 173.4±.7 113.2±.6 83.6±.7 
 Empowerment in aspect of service (M±SD)e1173.1±.8 463.2±.7 353.1±.8 173.1±.7 112.9±.7 83.7±1.0 
 Empowerment in aspect of community (M±SD)e1172.3±.7 462.1±.6 352.2±.7 172.4±.7 112.8±.8 82.5±1.0 
 Acceptance (M±SD)f11612.6±2.2 4512.6±2.3 3512.9±1.9 1713.4±1.5 1111.4±2.7 811.6±3.5 
 Depression (M±SD)g1178.8±7.0 469.0±8.0 357.6±5.0 178.3±6.1 1111.2±9.0 810.8±6.9 
 Worry (M±SD)h1171.7±.4 461.8±.4 351.8±.4 171.7±.4 111.7±.5 82.0±.5 
 Subjective burden (M±SD)h1172.6±.5 462.5±.5 352.7±.4 172.6±.5 112.3±.6 82.9±.5 
a
From the Family Experience Interview Schedule. Possible scores range from 0 to 4, with higher scores indicating more frequent assistance in daily life.
b
From the Family Assessment Device. Possible scores range from 6 to 24, with higher scores indicating worse problem solving.
c
T scores range from 38 to 81, with higher scores indicating more anxiety symptoms.
d
T scores range from 33 to 81, with higher scores indicating more global symptoms.
e
Family Empowerment Scale. Possible scores range from 1 to 5, with higher scores indicating more empowerment.
f
From the COPE. Possible scores range from 4 to 16, with higher scores indicating better coping.
g
As measured on the Center for Epidemiological Studies Depression Scale. Possible scores ranges from 0 to 42, with higher scores indicating more severe depression symptoms.
h
Family Member Questionnaire. Possible scores range from 1 to 4, with higher scores indicating less worry and fewer burdens.

Effect size estimates for the decliner sample

Our primary goal was to determine whether the estimate of the effect of FTF versus waitlist observed in the RCT generalized to the decliner sample. We planned to derive estimates of FTF’s impact on the outcomes of knowledge, family problem solving, empowerment, acceptance aspects of coping, anxiety, and subjective burden (worry) for the decliner sample and compare these with the estimates of benefits of FTF observed in the published RCT (3). Our approach was to build on the internally valid estimates of benefit derived from the RCT and enhance external validity by weighting the estimates of effectiveness observed in the RCT to fit the distribution of the propensity score quintiles for the baseline decliner population.
Although similar to age-adjusted estimates that are commonly used in life tables, propensity score matching enabled us to adjust for many covariates simultaneously. Estimates of the effectiveness of RCT FTF versus RCT waitlist were calculated for decliners. [Calculations and corresponding standard errors are available online as a data supplement to this article.] This approach provided an estimate of how individuals similar to the decliners would do if they received FTF versus how they would do if they could also be observed after assignment to (hypothetically) the waitlist. We next used these estimates and standard errors to calculate 95% confidence intervals and corresponding effect sizes regarding FTF versus waitlist effectiveness for the decliner sample. These confidence intervals were then compared with the RCT estimates.

Results

Baseline differences

Of the total sample of 409 consumers whose race was reported by family member participants, 260 (64%) were white, 11 (3%) were Asian, 105 (26%) were black, seven (2%) were Hispanic, and 26 (6%) were other. When compared with the decliners, participants in the RCT were significantly more likely to report having family income greater than $50,000 per year (χ2=4.39, df=1, p<.036), to report that the consumer required more assistance in daily living (t=2.13, df=435, p<.033), and to report that the family member had a psychiatric hospitalization in the past six months (χ2=11.35, df=1, p<.001). With respect to study outcomes, decliners had less knowledge about mental illness (t=2.43, df=435, p<.016) and less community empowerment (t=2.51, df=434, p=.012) at baseline.
Figure 1 provides an example of how the propensity score approach allowed the decliner sample to be matched with the RCT sample when one of the variables contributed to the propensity score. An important aspect of matching groups via propensity score quintiles was to examine the percentage of participants within each quintile by sample. Specifically, the critical question was whether there was a comparable percentage of each sample (decliner, RCT waitlist, or RCT FTF) for any particular variable represented in each quintile.
Figure 1 Percentage of participants reporting that a family member had a psychiatric hospitalization in the past 6 months, by propensity score quintilea
a Respondents received or were waitlisted to receive the 12-week Family-to-Family (FTF) psychoeducation program in a randomized controlled trial (RCT). Persons who declined random assignment (decliners) received FTF but did not participate in the RCT.
The figure shows that the variable, percentage of participants whose family member experienced a psychiatric hospitalization in the past six months, decreased from quintile 1 (most like the decliners) to quintile 5 (least like the decliners); in other words, compared with the RCT participants, a higher percentage of the decliners tended to have a family member who had been hospitalized, consistent with the bivariate analysis presented above. However, within each quintile, the percentage was generally similar across the three samples. It is important to note that matching does not require perfect balance. Other covariates were also effectively balanced by propensity score quintile matching. Table 1 gives baseline covariate and outcome data by group (RCT waitlist, RCT FTF, or decliner) and propensity score quintile. ANOVAs and chi squares across quintiles of all covariates demonstrated significant heterogeneity.

Generalizability estimates

Table 2 provides mean outcome levels by group and propensity score quintile for three-month outcomes. The three groups appeared to be comparably distributed within each quintile, as shown in Figure 2 for the knowledge test. Table 3 presents the effectiveness estimates with confidence intervals and effect sizes derived from the RCT for comparison of individuals receiving FTF with individuals on the waitlist. It also provides estimates of the RCT FTF versus RCT waitlist effect for the decliner sample—estimates that reflect the capacity of the propensity scoring process and assignment of quintiles to predict what the effects of FTF versus waitlist would have been for the decliners. We note that the effect sizes were remarkably similar despite the selection differences for being in the decliner population. For example, with respect to knowledge, the effect size observed in the RCT was .31. The estimated effect size for the decliner population was .29. Also, the confidence intervals had a high degree of overlap.
Table 2 Three-month outcomes in a randomized controlled trial (RCT) of participants receiving or waiting to receive Family-to-Family (FTF) psychoeducation, by propensity score quintile
 OverallQuintile 1Quintile 2Quintile 3Quintile 4Quintile 5
Outcome and groupTotal NMSDTotal NMSDTotal NMSDTotal NMSDTotal NMSDTotal NMSD
RCT FTF waitlist                  
 Knowledge (% correct)11458.817.41558.713.82260.617.52559.616.82856.616.52458.721.6
 Problem solvinga11312.92.91513.23.92212.52.42412.51.82813.53.12412.83.2
 Anxietyb11452.49.41552.17.42249.36.82551.310.42854.610.12453.810.6
 Global Severity Indexc11451.99.41550.55.72249.09.12551.18.72853.510.22454.510.9
 Empowerment in aspect of familyd1143.5.6153.2.7223.5.6253.5.5283.4.8243.6.6
 Empowerment in aspect of serviced1143.1.9152.8.8223.2.8253.2.9282.9.9243.2.9
 Empowerment in aspect of communityd1142.5.8152.1.6222.3.6252.3.6282.5.8242.91.0
 Acceptancee11412.72.51512.02.42213.42.22512.62.62812.42.72412.92.3
 Depressionf1148.56.9157.04.6228.05.9256.35.1289.88.12410.78.6
 Worryg1141.9.5151.8.5221.8.4252.0.4282.1.6242.0.5
 Subjective burdeng1142.7.5152.5.5222.5.4252.9.6282.7.5242.8.5
RCT FTF participants                  
 Knowledge (% correct)12965.216.81658.018.72170.513.22667.616.93664.318.53064.115.1
 Problem solvinga12612.12.61613.12.62111.62.52612.02.53411.62.42912.63.0
 Anxietyb12950.68.11648.17.62148.07.42653.67.23650.67.83051.29.4
 Global Severity Indexc12950.39.01646.99.22147.07.82653.27.23651.27.93050.911.1
 Empowerment in aspect of familyd1293.7.6163.4.9213.7.7263.7.6363.8.5303.7.6
 Empowerment in aspect of serviced1293.4.8163.21.1213.5.8263.4.8363.5.7303.5.8
 Empowerment in aspect of communityd1292.9.8162.4.8212.8.7262.7.8363.1.7303.2.7
 Acceptancee12813.62.01613.02.52114.01.82614.12.03513.71.83013.12.1
 Depressionf1297.57.0166.46.1214.64.3269.46.9366.95.9309.39.4
 Worryg1292.0.5161.7.7211.9.4262.0.4362.1.6302.0.5
 Subjective burdeng1292.7.5162.6.4212.6.4262.8.5362.9.4302.7.5
Decliners (no RCT random assignment)                  
 Knowledge (% correct)9163.818.93865.219.82765.820.11461.916.3661.910.0653.220.7
 Problem solvinga8412.62.63412.12.12612.92.51313.22.3615.23.4510.43.8
 Anxietyb8950.18.13749.37.72652.27.71450.06.4648.07.8648.714.9
 Global Severity Indexc8949.48.63747.88.82650.58.01451.16.2648.010.0651.512.8
 Empowerment in aspect of familyd903.5.6373.6.5273.5.5143.4.763.8.663.7.7
 Empowerment in aspect of serviced903.4.7373.4.7273.3.7143.3.763.5.963.51.1
 Empowerment in aspect of communityd902.7.7372.6.6272.7.8142.9.863.2.962.8.9
 Acceptancee9113.32.23813.42.02713.72.31413.02.5611.82.8612.72.0
 Depressionf897.86.6375.76.0269.35.9148.46.7610.76.4610.210.0
 Worryg902.0.4371.9.4272.0.4141.9.562.0.462.2.8
 Subjective burdeng902.7.5372.7.4272.8.5142.7.562.7.563.0.9
a
From the Family Assessment Device. Possible scores range from 6 to 24, with higher scores indicating worse problem solving.
b
From the Brief Symptom Inventory. Anxiety T scores range from 38 to 81, with higher scores indicating more anxiety symptoms.
c
T scores range from 33 to 81, with higher scores indicating more global symptoms.
d
From the Family Empowerment Scale. Possible scores range from 1 to 5, with higher scores indicating more empowerment.
e
From the COPE. Possible scores range from 4 to 16, with higher scores indicating better coping.
f
As measured on the Center for Epidemiological Studies Depression Scale. Possible scores ranges from 0 to 42, with higher scores indicating more severe depression symptoms.
g
From the Family Member Questionnaire. Possible scores range from 1 to 4, with higher scores indicating less worry and fewer burdens.
Figure 2 Knowledge scores at 3 months, by propensity score quintilea
a Respondents received or were waitlisted to receive the 12-week Family-to-Family (FTF) psychoeducation program in a randomized controlled trial (RCT). Persons who declined random assignment (decliners) received FTF but did not participate in the RCT.
Table 3 Effectiveness of Family-to-Family program versus waitlist and estimated generalizability for persons declining random assignment in the randomized controlled trial (RCT)
MeasureGeneralizability estimatea95% CIEffect size
Knowledge   
 RCT5.283.33 to 7.23.31
 Decliners4.942.09 to 7.79.29
Problem solving   
 RCT–.70–.99 to –.41–.23
 Decliners–.56–1.09 to –.03–.19
Anxiety   
 RCT−1.95–2.89 to –1.01–.21
 Decliners−2.00–3.23 to –.77–.22
Global Severity Index   
 RCT−1.62–2.53 to –.71–.18
 Decliners−2.17–3.56 to –.78–.24
Family empowerment   
 RCT.14.08 to .20.23
 Decliners.21.08 to .34.35
Service system empowerment   
 RCT.23.15 to .31.26
 Decliners.35.19 to .51.39
Community empowerment   
 RCT.26.19 to .33.37
 Decliners.40.28 to .52.58
Acceptance   
 RCT.74.48 to 1.00.32
 Decliners.93.52 to 1.34.40
Depression   
 RCT−1.23–2.18 to –.28–.16
 Decliners−1.17–2.13 to –.21–.15
Worry   
 RCT.04–.04 to .12.08
 Decliners–.01–.11 to .09–.02
Subjective burden   
 RCT–.10–.09 to .07–.20
 Decliners.07–.01 to .15.13
a
Comparisons were as follows: RCT, estimate from the RCT, excluding decliners (3). For decliners, the estimate is of Family-to-Family recipients versus the waitlist effect for the decliner group.

Discussion

RCTs are vulnerable to selection bias that can reduce the external validity of study findings. Without external validity, the overall value of RCTs for informing care delivery and policy is critically limited. Programs that are widely available prior to effectiveness evaluation may face special challenges in avoiding significant selection bias when attempting to conduct RCTs. This creates difficulty in amassing high-quality practice-based evidence sufficient to merit the program’s determination as an evidenced-based practice. This study’s overall significance derives from its development and application of methods to meet that challenge.
Our RCT of NAMI’s FTF education program appeared to be vulnerable to selection bias because individuals could access FTF without participating in our study. We were able to empirically evaluate this threat to our analysis by recruiting a sample of persons (decliners) who declined to participate in the randomization process. We showed that the estimated RCT FTF versus RCT waitlist effect sizes for the decliner sample were quite similar to the effect sizes observed in the RCT in which the individuals randomly assigned to FTF were compared with a waitlisted group. We thus conclude that FTF may indeed be effective for a target population that includes people similar to the decliners as well as those similar to the RCT participants.
This study therefore reinforced our previous findings that the NAMI FTF program is a valuable resource to family members of individuals with mental illness. FTF has been found to increase knowledge about mental illness, improve self-reported family problem-solving skills, and reduce distress. The RCT also demonstrated that FTF improves family members’ coping skills and empowerment (3). The positive and generalizable impact of FTF observed in this study further reinforces the value of this program as an evidence-based practice and the imperative for mental health providers and clinicians to consider it a resource for struggling family members. These findings also underscore the unique contributions of peer-based support programs in the service array for persons with mental illnesses and their relatives (19,20).
The differences between RCT decliners and RCT participants may suggest some unique sampling vulnerabilities for RCTs with waitlisted control groups. As could be expected, individuals with higher indicators of need (greater objective burden and greater likelihood of a consumer’s recent hospitalization) were less willing to take the chance of random assignment to the waitlist condition. In addition, individuals with greater income were more likely to refuse RCT enrollment. Such patterns could have plausibly produced estimates suggesting that FTF would not have been effective with the decliners. The analyses presented therefore underscore the importance of adopting a systematic, empirical approach to evaluating external validity.
This use of propensity scores to evaluate such potential biases in nonrandomized samples was limited by the fact that the estimates for the decliners were unbiased only when we could adjust for all confounders. Thus it is important for confounders to be considered during the design phase, so they can be measured. This involves collecting information about characteristics that might be related to being a decliner, as well as characteristics that are thought to influence the primary outcomes of a study. Collecting more covariates can add cost and complexity, but it is important not to overlook those necessary to determine whether results are convincing. Qualitative methods can be helpful in identifying additional confounders, particularly for areas in which there is not much existing research.
Our approach can be modified to other situations that commonly occur in psychiatric services research, for example, when people do not consent to randomization processes because they have strong preferences about treatment (1,14). Studies comparing medication to psychotherapy, two different medications, or two different psychotherapies exemplify this circumstance. In these situations, it is essential to document reasons for nonconsent and to subsequently collect outcome data when possible.

Conclusions

This study used innovative statistical methods to assess whether the benefits observed in a RCT of NAMI’s FTF education program could be extended to a majority of eligible individuals who declined to participate in the RCT. By including a cohort of decliners and evaluating their status before and after participating in FTF, the analyses suggest that FTF versus waitlist benefits of improving knowledge, reducing distress, and improving family problem solving generalize to the larger group. The significance of this study rests not only in the demonstration of benefits of FTF but also provides an important example of how RCTs of interventions available in the community can address the problem of external validity for the valid designation of programs as evidence-based practices.

Acknowledgments and disclosures

This project was supported by grant 1R01-MH72667-01A1 from the National Institute of Mental Health, by grant P20 MH085983 from the Center for Collaborative Inner-City Child Mental Health Services Research, and by grant P30 MH090322-01 A1 from the Advanced Center on Implementation–Dissemination Science in States for Children and Families.
The authors report no competing interests.

Supplementary Material

Supplemental Material (754_ds001.pdf)

References

1.
Marcus SM: Assessing non-consent bias with parallel randomized and nonrandomized clinical trials. Journal of Clinical Epidemiology 50:823–828, 1997
2.
Paradise JL, Bluestone CD, Bachman RZ, et al.: Efficacy of tonsillectomy for recurrent throat infection in severely affected children: results of parallel randomized and nonrandomized clinical trials. New England Journal of Medicine 310:674–683, 1984
3.
Dixon LB, Lucksted A, Medoff DR, et al.: Outcomes of a randomized study of a peer-taught Family-to-Family Education Program for mental illness. Psychiatric Services 62:591–597, 2011
4.
Tessler R, Gamache G: Family Experiences Interview Schedule (FEIS); in the Toolkit on Evaluating Family Experiences With Severe Mental Illness. Cambridge, Mass, Human Services Research Institute, Evaluation Center, 1995. Available at www.hsri.org
5.
Koren P, DeChillo N, Friesen B: Measuring empowerment in families whose children have emotional disorders: a brief questionnaire. Rehabilitation Psychology 37:305–321, 1992
6.
Szmukler GI, Burgess P, Herrman H, et al.: Caring for relatives with serious mental illness: the development of the Experience of Caregiving Inventory. Social Psychiatry and Psychiatric Epidemiology 31:137–148, 1996
7.
Joyce JL, Leese M, Szmukler G: The Experience of Caregiving Inventory: further evidence. Social Psychiatry and Psychiatric Epidemiology 35:185–189, 2000
8.
Dixon L, Lucksted A, Stewart B, et al.: Outcomes of the peer-taught 12-week family-to-family education program for severe mental illness. Acta Psychiatrica Scandinavica 109:207–215, 2004
9.
Ware J, Kosinski M, Keller SD: A 12-Item Short-Form Health Survey: construction of scales and preliminary tests of reliability and validity. Medical Care 34:220–233, 1996
10.
Derogatis LR, Melisaratos N: The Brief Symptom Inventory: an introductory report. Psychological Medicine 13:595–605, 1983
11.
Derogatis LR: BSI-18: Administration, Scoring and Procedures Manual. New York, Pearson, 2001
12.
Sawin KJ, Harrigan MP: Measures of Family Functioning for Research and Practice. New York, Springer, 1995
13.
Carver CS, Scheier MF, Weintraub JK: Assessing coping strategies: a theoretically based approach. Journal of Personality and Social Psychology 56:267–283, 1989
14.
Marcus SM, Stuart EA, Wang P, et al.: Estimating the causal effect of randomization versus treatment preference in a doubly randomized preference trial. Psychological Methods 17:244–254, 2012
15.
Rosenbaum PR, Rubin DB: The central role of the propensity score in observational studies for causal effects. Biometrika 70:41–55, 1983
16.
Stuart EA, Marcus SM, Horvitz-Lennon MV, et al.: Using non-experimental data to estimate treatment effects. Psychiatric Annals 39:41451, 2009
17.
Haviland AM, Nagin DS, Rosenbaum PR: Combining propensity score matching and group-based trajectory analysis in an observational study. Psychological Methods 12:247–267, 2007
18.
Rosenbaum PR, Rubin DB: Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association 79:516–524, 1984
19.
Segal SP, Silverman CJ, Temkin TL: Self-help and community mental health agency outcomes: a recovery-focused randomized controlled trial. Psychiatric Services 61:905–910, 2010
20.
Brown LD: Consumer-Run Mental Health: Framework for Recovery. New York, Springer, 2012

Information & Authors

Information

Published In

Go to Psychiatric Services
Go to Psychiatric Services

Cover: Summer Afternoon, by William Dean Fausett, 1943. Oil and tempera on masonite, 30 × 38 inches. Collection of the San Antonio Art League and Museum, San Antonio, Texas.

Psychiatric Services
Pages: 754 - 763
PubMed: 23633161

History

Published in print: August 2013
Published online: 15 October 2014

Authors

Details

Sue M. Marcus, Ph.D.
Dr. Marcus and Dr. Duan are affiliated with the Departments of Psychiatry and Biostatistics in the Division of Biostatistics at New York State Psychiatric Institute (NYSPI), Columbia University, New York City. Dr. Dixon is affiliated with the Department of Psychiatry, Columbia University, and with NYSPI, New York City. Dr. Medoff, Ms. Fang, and Dr. Lucksted are with the Department of Psychiatry, University of Maryland School of Medicine, Baltimore. Mr. Weaver is with the Research Foundation for Mental Health, New York City. Send correspondence to Dr. Dixon, Department of Psychiatry, Columbia University/NYSPI, 1051 Riverside Dr., New York, NY 10032 (e-mail: [email protected]).
Deborah Medoff, Ph.D.
Dr. Marcus and Dr. Duan are affiliated with the Departments of Psychiatry and Biostatistics in the Division of Biostatistics at New York State Psychiatric Institute (NYSPI), Columbia University, New York City. Dr. Dixon is affiliated with the Department of Psychiatry, Columbia University, and with NYSPI, New York City. Dr. Medoff, Ms. Fang, and Dr. Lucksted are with the Department of Psychiatry, University of Maryland School of Medicine, Baltimore. Mr. Weaver is with the Research Foundation for Mental Health, New York City. Send correspondence to Dr. Dixon, Department of Psychiatry, Columbia University/NYSPI, 1051 Riverside Dr., New York, NY 10032 (e-mail: [email protected]).
Li Juan Fang, M.S.
Dr. Marcus and Dr. Duan are affiliated with the Departments of Psychiatry and Biostatistics in the Division of Biostatistics at New York State Psychiatric Institute (NYSPI), Columbia University, New York City. Dr. Dixon is affiliated with the Department of Psychiatry, Columbia University, and with NYSPI, New York City. Dr. Medoff, Ms. Fang, and Dr. Lucksted are with the Department of Psychiatry, University of Maryland School of Medicine, Baltimore. Mr. Weaver is with the Research Foundation for Mental Health, New York City. Send correspondence to Dr. Dixon, Department of Psychiatry, Columbia University/NYSPI, 1051 Riverside Dr., New York, NY 10032 (e-mail: [email protected]).
James Weaver, M.P.H.
Dr. Marcus and Dr. Duan are affiliated with the Departments of Psychiatry and Biostatistics in the Division of Biostatistics at New York State Psychiatric Institute (NYSPI), Columbia University, New York City. Dr. Dixon is affiliated with the Department of Psychiatry, Columbia University, and with NYSPI, New York City. Dr. Medoff, Ms. Fang, and Dr. Lucksted are with the Department of Psychiatry, University of Maryland School of Medicine, Baltimore. Mr. Weaver is with the Research Foundation for Mental Health, New York City. Send correspondence to Dr. Dixon, Department of Psychiatry, Columbia University/NYSPI, 1051 Riverside Dr., New York, NY 10032 (e-mail: [email protected]).
Naihua Duan, Ph.D.
Dr. Marcus and Dr. Duan are affiliated with the Departments of Psychiatry and Biostatistics in the Division of Biostatistics at New York State Psychiatric Institute (NYSPI), Columbia University, New York City. Dr. Dixon is affiliated with the Department of Psychiatry, Columbia University, and with NYSPI, New York City. Dr. Medoff, Ms. Fang, and Dr. Lucksted are with the Department of Psychiatry, University of Maryland School of Medicine, Baltimore. Mr. Weaver is with the Research Foundation for Mental Health, New York City. Send correspondence to Dr. Dixon, Department of Psychiatry, Columbia University/NYSPI, 1051 Riverside Dr., New York, NY 10032 (e-mail: [email protected]).
Alicia Lucksted, Ph.D.
Dr. Marcus and Dr. Duan are affiliated with the Departments of Psychiatry and Biostatistics in the Division of Biostatistics at New York State Psychiatric Institute (NYSPI), Columbia University, New York City. Dr. Dixon is affiliated with the Department of Psychiatry, Columbia University, and with NYSPI, New York City. Dr. Medoff, Ms. Fang, and Dr. Lucksted are with the Department of Psychiatry, University of Maryland School of Medicine, Baltimore. Mr. Weaver is with the Research Foundation for Mental Health, New York City. Send correspondence to Dr. Dixon, Department of Psychiatry, Columbia University/NYSPI, 1051 Riverside Dr., New York, NY 10032 (e-mail: [email protected]).
Lisa B. Dixon, M.D., M.P.H.
Dr. Marcus and Dr. Duan are affiliated with the Departments of Psychiatry and Biostatistics in the Division of Biostatistics at New York State Psychiatric Institute (NYSPI), Columbia University, New York City. Dr. Dixon is affiliated with the Department of Psychiatry, Columbia University, and with NYSPI, New York City. Dr. Medoff, Ms. Fang, and Dr. Lucksted are with the Department of Psychiatry, University of Maryland School of Medicine, Baltimore. Mr. Weaver is with the Research Foundation for Mental Health, New York City. Send correspondence to Dr. Dixon, Department of Psychiatry, Columbia University/NYSPI, 1051 Riverside Dr., New York, NY 10032 (e-mail: [email protected]).

Metrics & Citations

Metrics

Citations

Export Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

For more information or tips please see 'Downloading to a citation manager' in the Help menu.

Format
Citation style
Style
Copy to clipboard

View Options

View options

PDF/EPUB

View PDF/EPUB

Login options

Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.

Personal login Institutional Login Open Athens login
Purchase Options

Purchase this article to access the full text.

PPV Articles - Psychiatric Services

PPV Articles - Psychiatric Services

Not a subscriber?

Subscribe Now / Learn More

PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.

Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).

Media

Figures

Other

Tables

Share

Share

Share article link

Share