Validating the Measurement of Real-World Functional Outcomes: Phase I Results of the VALERO Study

The demographic and clinical characteristics of the patients are summarized in Table 1. There were differences between the sites in age, education level, race, ethnicity, and type of informant. All of the informants at the Atlanta VA Medical Center were friends or relatives. There were also differences in residential status, with the majority of the San Diego and Atlanta VA patients living in the community, whereas the Skyland Trail patients were more residentially disabled. There were no differences in employment status across the sites, and total scores on the Positive and Negative Syndrome Scale were essentially identical across sites. The scores on the three performance-based measures were similar across the sites as well, although the Atlanta VA patients had somewhat higher UPSA-B scores. Thus, the demographic differences between the samples and sites did not correspond to major differences in the performance-based measures.

Table 2 presents the overall fit of the complete baseline model with the ability latent trait based on three performance-based indicators and examiner-generated total scores for the six real-world functioning scales. All three performance-based variables were significantly related to the ability latent trait, and only one of the six rating scales was significantly related to the ability latent trait—the Specific Levels of Functioning Assessment in the baseline analyses. Nonetheless, the overall fit of the model still would be considered acceptable, given the small RMSEA and a p value close to nonsignificant. However, this model clearly had room for improvement, in that most of the rating scales were not independently related to the performance-based measures.

Next, we sequentially omitted real-world functioning scales, as presented in Figure 1. The sequence was determined by rank-ordering the nonsignificant p values for the factor loadings between the real-world rating measure and the ability latent trait. Using these criteria, the order of deletion was Social Behavior Schedule, Independent Living Skills Survey, Quality of Life Scale, and Social Functioning Scale. In each of the resulting models, there was still room for improvement in the fit of the model, although some began to show evidence of fitting the data acceptably (e.g., the model in which Social Behavior Schedule and Independent Living Skills Survey were deleted). When four of the six scales had been deleted, the model judged as best fitting was revealed, which suggested that the Life Skills Profile and the Specific Levels of Functioning Assessment were the two best scales in combination for the prediction of the ability latent trait.

The Life Skills Profile factor coefficient was not statistically significant in the last and best-fitting model, which suggests that it did not add any information to that provided by the Specific Levels of Functioning Assessment. For this reason, a simultaneous regression analysis was performed, with these two examiner-rated total scores entered as predictors of the ability latent trait. The analysis found that the Specific Levels of Functioning Assessment total score was significantly related to the ability latent trait (t=3.09, df=192, p=0.002), while the Life Skills Profile total score was not. In a forced-entry regression analysis, when the Specific Levels of Functioning Assessment was entered first, the results were significant (t=4.52, df=193, p<0.001), while the Life Skills Profile total score did not enter. When the Life Skills Profile total score was forced into the regression analysis first, it still did not enter the equation, while the Specific Levels of Functioning Assessment still contributed variance to the ability latent trait above and beyond the nonsignificant contribution of the Life Skills Profile (t=4.21, df=192, p<0.001). When we examined the total variance accounted for between the ability latent trait and the Specific Levels of Functioning Assessment total scores, we found that the Specific Levels of Functioning Assessment ratings accounted for 24% of the variance in the ability latent trait. Thus, reducing the number of real-world functional rating scales from six to one leads to a reduction in reliable variance of 17% but allows for the collection of functional information with a single questionnaire.

The results of this study indicate that real-world functional outcomes in people with schizophrenia, rated with an array of preselected rating scales and using information from the patient and an informant and condensed into a judgment rating by the examiner, are globally related to performance-based assessments of ability. This is an important finding, because several studies (reviewed above) reported modest to negligible correlations between cognitive and functional abilities and real-world outcomes, raising questions about the well-accepted relationship between cognitive impairment and disability in people with schizophrenia. Our data suggest that this relationship is actually as strong as suggested by Green et al. (7) but also indicate that both the rating scale and the methods for rating real-world functioning may influence the strength of the association.

Our results also indicate that many rating scales that address real-world functioning, even when using multiple sources of information and a systematic approach to ratings in optimal research conditions, are not strongly related to the indices of functional abilities that are employed as state-of-the-art outcome measures in treatment studies. Finally, it is possible to tentatively endorse a functional outcome scale—the Specific Levels of Functioning Assessment—that measures social, vocational, and everyday living outcomes and is related to performance on measures of everyday functional ability at a level that provides substantial information about everyday functioning.

In future studies using these performance-based measures as outcome measures, it appears that the Specific Levels of Functioning Assessment could be a suitable baseline measure to index ability-relevant real-world functioning or as an outcome measure in a long-term study, as well as for use in clinical assessment. Most real-world functional outcome scales seem to be largely redundant with each other when utilized simultaneously, and our results suggest that none of them has a stronger relationship to functional abilities than the Specific Levels of Functioning Assessment. However, this finding does not prove that other rating scales would not be suitable for use with similar rating methods. In future analyses, we intend to examine whether the subscales of these instruments are useful in augmenting the Specific Levels of Functioning Assessment for assessment of ability-relevant real-world functioning and whether any of the informant ratings could be substituted for the comprehensive interviewer judgments.

A number of limitations must be considered when evaluating these results. As noted, it is possible that items or subscales of the scales we examined are more closely related to ability measures than the total scores. Also, our analyses did not address whether self-reports or informant reports of functioning might be superior to interviewer judgments and whether either could reliably be used on their own with the scales used in this study; this question will be addressed in detail in subsequent analyses. Most of the available real-world functional status scales that we did not examine here were excluded either in the scale screening stage or by the RAND panel in the early phases of this study because not enough data were available to evaluate their suitability. Some scales have been developed since that time, such as the Schizophrenia Outcomes Functioning Interview (31), or have had several new studies published, such as the Personal and Social Performance Scale (32). Finally, since we did not use the MCCB social cognition measure, we could not compute the composite score with the MCCB normative sample scoring program. This program compares composite scores of individuals with schizophrenia with those of healthy individuals, producing composite overall scores in impaired populations that are lower than the average of the individual item scores. For example, in a recently published clinical trial (33), the screening composite T score on the MCCB was 24.7, but the average of the nine tests administered was 34.2. This score is similar to the average score of 38.6 on the same tests in our sample, indicating that this is not a relatively unimpaired sample.

As previous studies by our team and others have shown, other factors may contribute to the prediction of real-world outcomes, and some of them have occasionally been found to have a greater impact on real-world functioning than ability (e.g., disability compensation, depression, and negative symptoms). However, even if other factors also predict outcome, treatments aimed at disability reduction try to improve ability, not to change environmental, cultural, or emotional variables that influence disability. Later phases of the VALERO study will examine the usefulness of different informants in generating real-world functional ratings on the Specific Levels of Functioning Assessment, and later analyses of the present data set will address issues of site and informant differences in the validity of real-world functional ratings. The analyses reported here were aimed at identifying the most broadly useful real-world outcome rating scale, when administered to the patient and an informant and followed by a best-estimate rating by the examiner. Such a rating procedure has been successfully implemented in previous treatment studies (34), suggesting that it is a feasible procedure.

Received Dec. 14, 2010; revisions received Feb. 4 and Feb. 22, 2011; accepted March 3, 2011.