Does One Size Fit All? What We Can and Can't Learn From a Meta-analysis of Housing Models for Persons With Mental Illness

Our review of housing intervention evaluations indicated that the community settings where persons with mental illness reside have been referred to by various names. We propose that these settings can be classified into four groups that we describe below: three unique housing models and nonmodel housing. The different models emphasize different outcomes and in some cases serve different subpopulations of persons with serious mental illness. The fourth housing type—nonmodel housing—is for housing not categorizable according to the models above. "Socially complex services" ( 12 ) such as housing models can be classified in multiple ways, and any one scheme inevitably has elements of arbitrariness. We offer this scheme as a useful starting point.

Residential care and treatment model housing. The earliest models, developed in response to deinstitutionalization, provided room and board and supervision, and in some cases treatment services, to manage or reduce symptoms related to mental illness and in some cases co-occurring substance abuse. In the literature, these settings are referred to by names such as board-and-care homes, group homes, community residences, halfway houses, and cooperative apartments. For the purposes of this analysis, we classified these housing interventions together as residential care and treatment housing models.

Ideally, some residents in residential care and treatment model housing would progress to housing with less intensive services, but this was not a primary goal of programming in the evolution of this model. Services, typically support services but occasionally treatment, were routinely provided on site by "housing staff," who sometimes lived in the program's housing. House rules were often made and implemented solely by housing staff, although residents might play a role in establishing rules. Residential care and treatment housing programs, with the exception of board-and-care homes, are often "high demand" environments, where privileges and rules of conduct are well defined, participation in services (some delivered in the housing) is required, and abstinence from alcohol or other drugs is a prerequisite ( 13 ). As such, they tend to select residents who need continued treatment but are "ready" to follow house rules, and the programs may also be thought of as high-demand-high-readiness housing models.

Residential continuum model housing. As community care progressed and the community support movement emerged, it became apparent to stakeholders that persons with serious mental illness had rehabilitation goals—and, more recently, recovery goals—such as employment, illness self-management, and avoidance of adverse events, such as involvement with the criminal justice system ( 2 ). One mental health system response to this realization was to design housing continua that added to residential treatment models certain housing interventions intended to foster consumer functioning and independence ( 9 ). These interventions were viewed as more "normalizing," in that services are not routinely provided in the housing and staff presence in the housing is limited. Residents are expected to leave their housing during the day and attend treatment, day activities, or work. Embodied in the concept of a continuum is the idea that residents move from one housing model to another as they progress in their rehabilitation and recovery ( 8 ).

Residential continuum models overlap in their component programs with residential care and treatment models and permanent supported housing models (described below). However, these programs are unique because they have components of a continuum consisting of different housing models that residents are expected to move between depending on their course of recovery and rehabilitation. Residential continuum models may also be thought of as high-demand-high-readiness housing models.

Permanent supported housing model. Most recently, experience and the emergence of the recovery movement have suggested that residential continua are difficult to implement and perhaps counterproductive from the perspective of rehabilitation and recovery ( 9 ). Requiring residents to change their housing as they improve or regress in functioning is viewed as not "normalizing" and possibly harmful. Having to give up familiar living arrangements and in some cases important relationships because of improvements in functioning and advances in recovery is viewed as a disincentive to progress toward these goals. Moving persons out of autonomous settings and into more supervised settings when problems arise is also seen as unnecessarily adding to stress. Therefore, housing interventions emerged that assume that persons with serious mental illness can be provided with permanent housing into which staff support and treatment can be introduced as needed (wraparound services), allowing residents to remain in familiar settings and maintain relationships, regardless of their needs for support and treatment. We classified housing interventions of this type as permanent supported housing models.

Permanent supported housing models generally provide low-demand or least restrictive environments. They have few rules of conduct, encourage service use in nonhousing settings, and view stable housing as a prerequisite for global lifestyle changes (the housing-first approach) ( 13 ). One type of permanent supported housing—housing ready—may require participation in services and may consider abstinence from alcohol or other drugs a prerequisite for program entry. Another type—housing first—does not require participation in services and follows harm reduction approaches to alcohol and other substance use. Given their less restrictive environments, permanent supported housing programs include persons who are in the early stages of readiness or motivation to make global lifestyle changes.

Nonmodel housing. In a subset of studies, housing outcomes are described for people living on the street, using shelter beds, or residing in housing that is described simply as a part of "treatment as usual." For housing described as treatment as usual, no reference is made to any service provided specifically to find housing for individuals or to support them in it. Some of these studies merely describe outcomes under these living arrangements. However, treatment as usual is often compared with model housing, strongly implying the nonmodel nature of such housing. We refer to these living arrangements as naturally occurring nonmodel housing.

In discussing our meta-analysis, we review findings in the context of the logic and ethics of intervention testing developed for evaluating other medical and psychosocial interventions. Under this logic, the efficacy or effectiveness of an intervention is evaluated by comparing it with that of a placebo or of no treatment. If an intervention is identified that is more efficacious or effective than placebo or no treatment, then it is ethical to conduct studies that compare other interventions with that intervention to determine relative efficacy or effectiveness ( 14 ). This leads to the question: What constitutes an ethical and meaningful comparison of housing models?

Strayhorn ( 15 ) discussed the ethical and practical problems of approximating placebo comparisons in psychosocial intervention research. He proposed three alternatives to placebo and no-treatment controls that are relevant for this meta-analysis: the "minimal control," the "dismantling control," and the "alternative intervention control." The minimal and dismantling controls consist of interventions that contain some but not all elements of the experimental intervention. We propose that naturally occurring nonmodel housing alternatives constitute minimal or dismantling controls for evaluating the effectiveness of model housing interventions. Therefore, for each outcome discussed below, we first compared model housing interventions to nonmodel housing to establish effectiveness. Then we compared alternative housing models to consider relative effectiveness.

The outcome most proximate to the intervention that model housing provides is stability in housing ( 9 ). Our review of previous housing studies found that other, more distal outcomes are also commonly investigated by use of diverse measures of similar underlying constructs and varying follow-up periods. The inclusion of these measures may reflect the theory that housing models enhance the effectiveness of other services by providing the preconditions for adherence to treatment plans. The outcome constructs that we attempted to code in our meta-analyses were stability in housing, psychiatric symptoms, psychiatric hospitalizations (hospital admissions and emergency room use), alcohol use, drug use, incarceration, employment, and satisfaction.

Housing intervention studies commonly report a number of sociodemographic and clinical characteristics, although they do not always use them in subgroup or covariance analyses. The sociodemographic characteristics that we sought to code for subgroup analyses were age (mean age of sample), gender (percentage female), race (percentage nonwhite), ethnicity (percentage Hispanic), and veteran status (percentage of veterans). The clinical variables we sought were diagnosis (percentage of persons with schizophrenia and bipolar disorder), level of functioning (Global Assessment of Functioning score <50 or an equivalent rating), and co-occurring disorders (percentage of persons with a mental illness and a substance use condition).

Newman ( 6 ) has stressed the importance of studying the effects of specific housing attributes, such as the number of occupants in housing, neighborhood characteristics, and urban or rural setting. Our preliminary examination, like Newman's, showed that few studies provided data on these characteristics. Therefore, we did not extract data on housing attributes.

Evaluations of housing models, like evaluations of other psychosocial interventions, differ in their design quality. However, evaluating housing models poses more complicated challenges than evaluating many treatments ( 6 ). These challenges include the problem of defining meaningful comparison groups given that everyone resides somewhere, the ethical issues involved in random assignment of persons to housing that is perceived to be less than adequate, variability in how ostensibly similar housing interventions are implemented, and identification of resident- and housing-related covariates to control for in quasi-experimental designs.

Most studies that we were able to locate for use in our meta-analysis were single-group, pre-post, or post-only designs. Therefore, we used study arm as the unit of analysis in our meta-analysis, which prevented us from rating study quality in terms of the usual convention: randomized trial or observational study. However, in response to Newman's recommendations ( 6 ), we tried to code certain other indicators of data quality related to the psychometric properties of outcome measures and the way in which data were collected for use in exploring the correlates of data quality. The data quality indicators were whether measures were identified before the study, reliability and validity of measures were reported or cited, fidelity measures were used for housing models, data collectors were blind to the intervention received, attrition or missing data were accounted for in analyses, and anomalous findings, if reported, were recognized and discussed.

Typically, the evaluations we reviewed were not theory guided. This lack of theory in housing research was noted by Newman ( 6 ) in 2001 and continues to be true. However, most studies reflected at least the implicit and general theory that housing outcomes are a function of type of housing or housing alternative (usually the independent variable), resident characteristics (usually moderating variables), and housing attributes (also usually moderating variables). We elaborated this theory with the following hypotheses, which we tested.

Hypothesis 1. Housing interventions are first and foremost intended to provide persons with acceptable places to continuously reside. Housing interventions will be most likely to achieve proximate outcomes reflective of stable housing.

Hypothesis 2. Housing interventions also can establish the preconditions for more distal outcomes, such as skill building, engaging in symptom treatment, exercising wellness behaviors, enacting normal role functioning, and developing self-esteem and other positive self-perceptions. Housing interventions should achieve these more distal outcomes to some degree but not to the degree that they achieve stable housing ( 6, 9, 14 ).

Hypothesis 3. Housing models will be differentially effective for different outcomes. Housing models with in-house services (residential care and treatment and residential continuum) will have stronger associations with reductions in psychiatric symptoms, hospitalizations, drug and alcohol use, and incarcerations than permanent supported housing. Permanent supported housing, because it is more normalized and flexible, will be associated with higher levels of employment and satisfaction ( 8, 9, 11 ).

Hypothesis 4. Housing interventions will be more effective when there is person-environment fit between the characteristics of residents (viewed as proxy variables for their specific needs and preferences) and the characteristics of the interventions ( 5 ). Certain types of housing interventions are likely to be more effective than others for achieving certain outcomes with certain types of residents. Thus the question of the effectiveness of a housing intervention is a variation of the question posed for psychotherapy: What types of interventions work in what ways for whom?

For this study, we searched MEDLINE and PsycINFO, combining the search terms "adults with severe and persistent mental illness" with key words related to housing models and homelessness, such as "housing," "community residences," "half-way houses," "group homes," "permanent supported housing," and "homelessness." Use of these search terms yielded a total of 3,986 unique studies. Attempts were made to obtain abstracts for all 3,986 studies. We also checked five narrative reviews ( 6, 8, 9, 10, 11 ) to identify any studies not discovered by our electronic search and queried housing experts cited in the literature, federal project officers for multisite evaluations conducted by the Substance Abuse and Mental Health Services Administration, and principal investigators for those evaluations about unpublished articles and reports that we might include in our analyses.

Abstracts and summaries were then reviewed to eliminate multiple reports of the same studies, narrative and qualitative reviews, studies that did not address adults with serious mental illness, studies conducted outside the United States ( 16, 17, 18, 19, 20, 21, 22 ), and studies that were judged as likely to lack necessary data. Each abstract was screened systematically by two raters using an electronic protocol. Intercoder reliability was assessed and found acceptable ( κ =.99). Using the criteria listed above, we identified and were able to obtain 227 research and evaluation reports that met criteria for full review.

Once reports of all the studies identified were obtained, we rereviewed the studies before coding to ensure that they met the criteria (for example, a unique study, use of housing as an intervention, a majority of study participants with mental illness, and a high probability of synthesizable data). On the basis of these precoding reviews, we eliminated another 167 studies, leaving 60 studies for coding. Table 1 shows the reasons for exclusion and the numbers of studies excluded.

Actual coding indicated that 30 studies were either kin studies or contained insufficient data for synthesis. As a result. the final number of studies analyzed was 30; the studies examined 44 unique housing interventions, including nonmodel housing.

The studies identified for the meta-analysis were coded according to a coding guide available from the authors. Coding results were entered into an Excel spreadsheet. Information was extracted about study design, housing model or models, outcomes, and clinical and sociodemographic variables for the groups studied.

To evaluate the reliability of our coding, a subset of studies was coded by two coders. The average intercoder agreement was approximately 87%, suggesting acceptable reliability. In addition, all studies coded were checked by a third person (one of the authors), who made final judgments in situations in which the coders disagreed.

Most of the studies we coded employed single-arm, pre-post, or post-only designs to evaluate the effectiveness of particular housing models. A small number of studies compared outcomes for persons in multiple housing models or compared outcomes for persons in housing models with outcomes for persons living on the street, using shelter beds, or residing in housing not identified as model housing. Given the predominance of single-arm designs, we made housing intervention the unit of analysis and, as study reports permitted, we collected study information separately for each of the housing interventions in multigroup studies. Given the nature of the studies available, this method provided a reasonable alternative to a narrative summary ( 23, 24, 25, 26 ). Because housing intervention was the unit of analysis for this investigation, all outcomes and resident characteristics are expressed as means, rates, or percentages for the housing interventions.

Studies measured outcomes in a variety of ways, using measures taken from diverse protocols or surveys applied over varying time periods. Follow-up periods, for example, ranged from six months to five years. This heterogeneity in the way outcome measures are reported in the mental health literature provides a unique challenge to meta-analysis.

To synthesize diverse measures of similar constructs for each housing intervention, we converted outcome scores to effect size measures appropriate for the data. In some cases this meant computing standardized change scores based on means. In others it required converting percentages to effect sizes. Typically, effect sizes from proportions are combined separately from those calculated from means. However, the two types of effect sizes can be combined if they are based on similar constructs ( 24, 26, 27 ).

Effect size scores were then combined, with each intervention weighted by number of participants and the inverse of variance to create random-effect estimates for all outcomes studied for each housing intervention. In studies that reported data for more than one follow-up, data for the last follow-up period was used for analysis. We then compared housing model effect sizes with the effect sizes estimated for persons in nonmodel housing, computing the standardized paired-difference effect size for each comparison. We used "comprehensive meta-analysis" (Biostat, version 2.3) to assist in the calculation. In this way, we were able to test whether persons in housing models had different outcomes from persons in nonmodel housing and to compare the effectiveness of different housing models.

When there were sufficient data, we also sought to determine for each outcome the housing interventions that worked best for specific subgroups (according to sociodemographic, clinical, and data quality variables) by categorizing outcomes as above or below the median for subgroup samples within the housing interventions. We then followed the meta-analytic procedure described above for each outcome, comparing effect sizes for housing intervention evaluations that differed in resident characteristics and data quality, when available data permitted.

Table 2 summarizes the data that we were able to extract for each of the housing interventions. Table footnotes include citations of the studies from which data were extracted ( 13, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 56, 57 ). We found data for a total of 44 housing interventions. The largest proportion (41%) were permanent supported housing. The next most frequent type of housing model for which we were able to find data was residential care and treatment (29%). The number of interventions by decade suggests that evaluation studies paralleled the emergence of housing models: evaluations of residential care and treatment housing appeared most frequently in the first two decades, and residential continuum and permanent supported housing interventions appeared most frequently in the most recent decade.

The evaluations represent data collected for 13,436 persons. The number of participants in the interventions studied ranged from 18 persons to more than 3,000 persons, which emphasizes the importance of weighting findings to reflect these differences.

A very small proportion of the intervention evaluations involved random assignment of participants. Because we did not compute effect sizes on the basis of within-study differences, randomization for our purposes does not necessarily imply that there were no differences between intervention groups. However, randomization may indicate a greater concern for avoiding study biases of all types. From this perspective it is heartening to see that the number of randomized studies increased by decade.

Possible data quality scores ranged from 0 to 13. We were able to rate data quality for only half of the evaluations. We were able to rate a higher proportion of the nonmodel housing interventions and the residential care and treatment interventions; these studies tended to have higher data quality scores.

Table 2 shows that the most frequently addressed outcomes were psychiatric symptoms, housing stability, and hospitalizations, in that order. Relatively few evaluations addressed individual outcomes. The distribution of outcomes also reduced the extent to which we could compare interventions for specific outcomes.

Table 2 also presents data on sociodemographic and clinical variables. One variable that we attempted to measure (persons with low functioning) is not shown in the table because of insufficient data. The data suggest relative similarity among housing types for the variables shown. Subgroup differences for outcome variables and interventions taken in combination are discussed below.

Table 3 presents the results of outcome comparisons for the four housing interventions. It shows the number of interventions synthesized and the random effect obtained for each housing intervention and outcome combination. For each outcome, if an effect size for a housing model differed significantly from the effect size for nonmodel housing, the significance level is shown. If the effect sizes for any pair of housing models differed, these significance levels are also shown. The table does not include data on incarceration or employment because the number of evaluations that addressed these outcomes were too few for analysis.

Table 3 shows that all three housing models achieved significantly greater housing stability than nonmodel housing. This effect size was greatest for permanent supported housing (.63, p<.05). However, none of the differences between the various housing models were statistically significant. It was suggested during the review process that we include findings from a specific review ( 49 ) that we had rejected because our inclusion criteria excluded reviews. We added the findings from the two relevant studies that were reported in that review (we did not have direct access to either study) to the studies shown in Table 3 and performed a post hoc analysis. This increased or did not change housing model effect size estimates for housing stability. In addition, it did not change the findings that all model housing options performed significantly better than nonmodel housing and did not differ significantly from each other.

With respect to psychiatric symptoms, only residential care and treatment differed significantly from nonmodel housing (effect size=.65, p<.05). However, this outcome was addressed by only one residential continuum housing evaluation and one permanent supported housing evaluation. The residential continuum evaluation yielded an effect size similar to that of residential care and treatment housing. The permanent supported housing evaluation yielded an effect size substantially lower.

For reduction in hospitalization, Table 3 shows that both residential care and treatment and permanent supported housing differed significantly from nonmodel housing. Hospitalization reduction was not addressed by any evaluations of residential continuum interventions. Although the effect size for permanent supported housing was more than double that of residential care and treatment, this difference was not statistically significant, which suggests high variability within the groups.

With respect to alcohol abuse, none of the housing models differed from nonmodel housing. The effect size for residential care and treatment housing was high (.87) and many times greater than the effect sizes for the other two interventions, but these differences were not statistically significant, which again suggests high intramodel variability. Effect sizes for reduction in drug abuse were smaller than for other outcomes, and there were no significant differences between any of the interventions.

Permanent supported housing achieved the highest effect size for satisfaction (.73), and this effect size differed significantly from the ones achieved by nonmodel housing (p<.001) and residential care and treatment (p<.05), which were substantially lower.

We were able to rate data quality for only half of the intervention evaluations ( 36, 38, 39, 40, 43, 44, 46, 50, 51, 52, 53 ). The distribution of these rated evaluations across housing types and outcome variables was such that there was only one outcome variable on which we could compare interventions on the basis of whether the evaluation data were rated as low or high quality. This was the case even when we grouped studies by outcome and ignored housing intervention. Thus we could examine only the association between data quality and effect sizes for supported housing interventions and housing stability outcomes. For two of the supported housing evaluations, the data quality scores were high ( 51, 54 ), and for another two the scores were low ( 52, 53 ). Housing stability increased in these studies regardless of data quality. However, the increase was much less in the higher-quality studies (effect size of .22 compared with .79).

When we subdivided interventions into subgroups on the basis of sociodemographic and clinical variables, we also found that the distribution of interventions was such that analyses for combinations of outcomes and interventions were not possible. However, when we examined the association of sociodemographic and clinical variables with outcomes without regard to housing intervention, we found a number of outcomes on which we could compare two or more interventions by sociodemographic and clinical subgroups. Table 4 presents the results of these subgroup analyses.

As shown in Table 4, subgroups with more females uniformly experienced better outcomes than groups with fewer females in terms of housing stability, hospitalization reduction, alcohol abuse reduction, drug abuse reduction, and satisfaction, although none of the individual differences were statistically significant.

Table 4 shows inconsistent findings for comparisons of certain subgroups. In comparisons of the pre-post effect sizes, we noted inconsistencies across all housing models on the five outcome areas for which we had sufficient data. For example, in studies reporting a majority of nonwhites, drug abuse reduction was greater than in studies reporting a majority of whites (.46 and .09, p<.001), as was hospitalization reduction (.32 and .22, not significant) and alcohol abuse reduction (.56 and .06, not significant). However, in studies with a majority of whites, housing stability was greater than in studies with a majority of nonwhites (.54 and .37, not significant) and satisfaction was greater (.19 and .09, not significant). We found a similar pattern of inconsistency in comparisons of the pre-post effect sizes of studies with a majority of Hispanics and studies with a majority of non-Hispanics. In the studies with a majority of non-Hispanics, drug abuse reduction (.47 and .09, p<.001) and alcohol abuse reduction (.33 and .06, not significant) was greater, but in studies with a majority of Hispanics, housing stability was greater in this group (.42 and .23, not significant).

Finally, participants without co-occurring substance use disorders experienced greater reductions in alcohol and drug abuse than those with co-occurring disorders (effect sizes of .40 and .06, respectively, for both alcohol and drug abuse, not significant) and greater satisfaction (.22 and -.34, p<.001). Among the studies included in the analysis, only two were unpublished. Omitting them from the analysis did not alter the results.