A New Approach to Integrating Data From Multiple Informants in Psychiatric Assessment and Research: Mixing and Matching Contexts and Perspectives

In both clinical and research applications, one informant may be selected as optimal, yet the criteria for choosing the optimal informant for each characteristic are not clear. To treat one source of information arbitrarily as the ideal informant increases the risk of obtaining the right answer to the wrong question. For example, the Infant Health and Development Program (19), a multisite, randomized clinical trial of a behavioral intervention to reduce health risks associated with low birth weight and prematurity, assessed children’s health status on the basis of retrospective maternal reports of illnesses. Such reports proved, however, to be highly correlated with the mothers’ vocabulary, education, race, and age. On the basis of the maternal reports, it appeared that lower family socioeconomic status was associated with better child health (19), a relationship that most investigators found untenable. Consequently, when it was shown that the reported health of the child was significantly poorer in the treatment group than in the control group, it was not clear whether the effect of the treatment was to improve the mother’s knowledge of and sensitivity to her child’s health (since one component of the treatment was parent education) or to impair the health of the child. If the treatment actually impaired the health of the child, one would question the clinical utility of the treatment, but if it improved the mother’s understanding of and sensitivity to her child’s health, the treatment would be encouraged.

In clinical applications, a clinician’s decision about whether to treat or how to treat remains unclear when multiple informants disagree on treatment indications. In research applications, on the other hand, a common strategy is to use data from all informants, however discrepant, separately and simultaneously. Thus, in the Multimodal Treatment of Attention Deficit Hyperactivity Disorder Study, an eight-site clinical trial of the effectiveness of a treatment for children with attention deficit hyperactivity disorder (ADHD), the various parent and teacher reports were separately reported as outcomes (20). With no adjustment for multiple testing, this strategy increases the risk of type I errors, i.e., greater likelihood of false positive results, because a separate risk of a false positive result exists for each outcome. With adjustment, the risk of type II error increases the likelihood of false negative results. When the conclusions of a randomized, controlled trial are incongruous because they are based on different measures of the same child characteristic, the trial’s message to clinicians and policy makers is ambiguous, undermining the pragmatic value of the trial.

Finally, in clinical and research applications, one might logically consider aggregation strategies, but, again, the selection of such strategies is often arbitrary. Recognizing the problem of multiple outcomes in the Multimodal Treatment of Attention Deficit Hyperactivity Disorder Study discussed in the previous section (20), the investigators made two post hoc attempts to aggregate across measures to adjust for the effects of multiple informants. One attempt was based on clinical judgment of whether symptoms reported by all informants were sufficiently few to consider the outcome a clinical success (21). The other was based on psychometric evaluation of the outcomes and was accomplished by entering the reports of multiple informants into a factor analysis, finding one factor related to mothers’ reports and another related to teachers’ reports, and averaging the effects of the two factors (22). Either aggregation strategy, had it been used a priori, would have led to clearer, more powerful, and less ambiguous results but would not necessarily have led to the same inferences from a clinical or policy view.

As of 2001, approximately 70 studies examining multi-informant variability in the assessment of child psychopathology had been published, yet there remains a paucity of theory describing processes or mechanisms that are capable of explaining and resolving multi-informant discrepancies. Most published empirical results have been descriptive (e.g., levels of variability by type of symptom, correlates of variability), leaving unresolved the major question about the processes required to obtain a more valid measure of the child’s characteristic.

Our goal was to develop a general procedure for derivation of a measure based on the reports of multiple informers that was closer to a population gold standard measure than are any of the conventional multiple individual informant strategies.

Figure 1 provides a graphic depiction of the concept underlying this general procedure, one very similar to the principles of the Global Positioning System. Suppose that the gold standard (T) resides at some location in a two-dimensional field and that the task is to locate the gold standard as accurately as possible. A gold standard measure is one that would point directly at the location of the gold standard, albeit perhaps with some imprecision (or unreliability), but no gold standard measure is available. Instead, multiple informants are available, as represented by the positions on the perimeter of the field (analogous to the satellites in various orbital positions), each of which can point with relative precision (i.e., reliably) in the general direction of the gold standard, but none of which can locate it accurately (i.e., validly). If one elected to use only the data from informant 1, the search for the gold standard would be narrowed substantially to a single line in the field, but not resolved. If one then elected to add the data from another informant, the least advantageous choice would be informant 2 or 3, or any other informant that shares the same limitations as informant 1. That is, one would not choose another informant whose perspective was “collinear” (or highly correlated) with that of informant 1. Instead, one would choose a second informant whose data could in some manner correct the deficiencies of informant 1’s data. In this case, the second informant would be informant 4 or 5, whose perspective is orthogonal to (or independent of) that of informant 1. In a two-dimensional space (east-west and north-south), one would minimally need two reliable informants for triangulation to find T, and the triangulated position would be a gold standard measure because it locates T both accurately and at least as precisely as the data from the multiple informants could separately.

Similarly, if the gold standard were located in a three-dimensional space (east-west, north-south, distance above ground level), one would minimally require three reliable informants to triangulate the position of the gold standard. Once again, optimal selection of the informants would avoid collinearity and maximize orthogonality. As the dimensionality of the problem increases, so also does the minimal number of informants, but the principle remains the same: it is not only the number of informants that matters; it is also how informants are selected. Moreover the lack of correlation (orthogonality) between informants, to date considered problematic, becomes precisely the phenomenon that facilitates a more valid measure. An infinite number of collinear informants cannot solve the problem, but in an M-dimensional space, M carefully selected informants may. The first challenge is to attempt a definition of dimensionality that clarifies what the dimensions are in the particular problem and hence what “collinear,” “orthogonal,” and “independent” mean in the selection of multiple informants. We consider this issue in the next section.

The preceding discussion, as well as past research, has led to the following mathematical representation. We propose that the response of an informant about the subject’s characteristic over a relevant span of time (I) is represented as follows:

I=T+C+P+E

where T, C, and P are three orthogonal dimensions and E is random error. In this equation, I is the informant’s report.

T is the trait dimension, i.e., the characteristic that differs from subject to subject in the population of interest but for each subject is constant over the designated span of time (perhaps the subject’s true mean over that span of time). The label “trait” for the characteristic of interest should not be confounded with the general usage of “trait” as an enduring or unchangeable characteristic of the subject. In this model, the designated span of time can be as short or long as is required for the issue under investigation—for example, in asking informants about a subject’s symptoms, the span of time may be 2 weeks, 6 months, or 3 years, depending on the focus of the research (e.g., if the study seeks to explore whether the duration of symptoms meets diagnostic criteria). Thus T, the “trait” of interest, may actually be a transient “state” rather than an enduring and unchangeable characteristic of the individual.

C is the context dimension, i.e., the factors related to place and circumstance that influence the subject’s expression of T and contribute to test-retest unreliability.

P is the perspective dimension, i.e., the characteristics of the informant that influence his or her assessment of the trait at any point in time and can constitute the major source of interobserver unreliability.

E is the error of measurement, i.e., the effects of random factors that are not related to subject, context, or perspective but that are perhaps related to the method by which the information is obtained. These effects constitute a major contribution to intraobserver unreliability.

Drawing on fundamental psychometric principles, we know that:

It has been known for almost a century (23, 24) that to increase the reliability of a measure under this model (i.e., reduce the percentage of variance due to E), one obtains M replicate (same T, P, and C) but independent measures (different E) and averages them. This is the rationale that underlies the standard practice of obtaining assays in triplicate and averaging the results, which disattenuates the reliability of a measure. At the same time, two additional actions are needed:

The practical challenge is to do both actions simultaneously and to combine the results because in theory that strategy would remove variance due to C and P and increase the proportion of variance due to T (i.e., increase the validity for T). This is not easy to do, however, since, for the most readily available informants, C and P are often confounded. For example, the mother (P) tends to see her child largely in the home (C), whereas the teacher (P) tends to see the child largely in school (C). Moreover, the instruments used by different informants are often insufficiently parallel to each other, and simply averaging scores measured on possibly different scales may be a mistake. To get the most valid measure of T possible from multiple informants, i.e., to remove extraneous variance due to P, C, and E, we need a better approach to put these principles into operation with multiple informants.

The first step is to identify the characteristic (T) of interest, the relevant span of time, and the population to which the results are to be applied. What is valid for clinic-referred children, for example, may not be valid for children in the community. A measurement approach valid at age 1 year may no longer be valid at age 5 years. A valid instrument for measuring a transient condition may not be valid for measuring an enduring characteristic over 6 months.

Next, the investigators would consider the totality of contexts and perspectives for which there are informants who are likely to provide reliable and to some degree valid data for the target characteristic. Then the challenge is to divide the totality of contexts envisioned into two or more broad categories that are 1) mutually exclusive, 2) broad enough to span contexts, and 3) likely to influence the expression of T as differently as possible, on the basis of one’s knowledge of the field. For a young child, these categories might be home versus nonhome. For a teenager, it might be more appropriate to contrast home versus school versus other setting, in the expectation that the expression of the characteristic in settings away from both home and school may be informative. In the same manner, the investigator would divide the totality of perspectives into two or more broad perspectives that are likely to influence informants’ reports as distinctively as possible. For a young child, such perspectives might include self, parent, and nonparent; for an older child, they may include self, parent, adult nonparent, and peer. Then the mix-and-match criteria would require contrasting informants with the same perspectives in different contexts and those in the same context with different perspectives.

In general, it can be shown that if there are c contexts and p perspectives (hence cp cells), the minimal number of informants to fill the cells is cp, but the minimal number of informants to satisfy the mix-and-match criterion is c+p–1. Thus, with two contexts and two perspectives (hence four cells), the minimal number of informants is three. With three contexts and two perspectives (hence six cells), the minimal number of informants is four. With three contexts and three perspectives (hence nine cells), the minimal number of informants is five. The difficulties of finding many reliable and valid informants militate against too complex a conceptualization of context and perspective.

This process creates a context-by-perspective grid, within which each possible, reliable, and at least minimally valid informant fits in one or more cells. Informants should be selected to fill at least three cells of the grid in such a way that each pair of contexts is seen in some one perspective, and each pair of perspectives is seen in some one context. By using this mix-and-match strategy, each of the informants is asked for an evaluation of the target characteristic for each child in the sample.

For example, suppose two contexts (home and nonhome) and two perspectives (self and other) were selected for a given research question as shown in the grid in Figure 2.

There are three possible informants in this example: child, teacher, and mother. Note that, by virtue of all cells being filled, the self perspective and other perspective are covered in the nonhome context as well as the home context, and the contrast of nonhome versus home is seen both from the self perspective and from the other perspective. Thus, this choice of informants (even when the two child reports were combined to avoid correlated errors) satisfies the mix-and-match requirement.

Sometimes self versus other, important as that conceptualization of perspective might be, is simply not possible (e.g., with infant subjects). Then we would have to limit perspective to the subspace of other. We might reconceptualize perspective as family versus nonfamily, with “family” including mother, father, siblings, and other relatives, and “nonfamily” including teachers, clinicians, peers, etc. The minimal number of informants necessary would still be three. However, no combination of mother, father, teacher, and clinician would now satisfy the mix-and-match criterion. Yet the most common combinations of informants in child assessments include these perspectives, as seen in the grid shown in Figure 3.

Could we, having mother, father, and teacher readily available, somehow reason backwards and find some conceptualization for which these informants would satisfy the mix-and-match criterion? For example, if the teachers are predominantly women, perhaps we could conceptualize perspective as male versus female. Then mother and father would match on context and contrast on perspective. Mother and teacher would match on perspective and contrast on context. If the conceptualization ill suits the situation (such as the conceptualization of male versus female, which ill suits the current example), the approach will not work (as we will show), but of course serendipity might favor such an attempt.

Clearly there are many possible ways to conceptualize context and perspective, some of which will, and some of which will not, satisfy the mix-and-match criterion. Some conceptualizations will succeed better than others because they reflect a better understanding or knowledge of the field. For example, as shown in Figure 1, one could choose any pair of noncollinear informants to implement triangulation. However, the accuracy of that determination reflects the precision of each informant and the degree of orthogonality between the informants. Thus, a poor choice of informants will result in failed triangulation.

Once the multiple informants have been selected, subjects are selected from the population of interest, and each informant is asked to report independently on the characteristic of interest for each selected subject. This set of observations is then subjected to principal-component analysis. Principal-component analysis is a mathematical model based on the assumption that the multiple informants’ reports are each linear combinations of orthogonal latent variables, as many such latent variables as there are informants per subject. These combinations of variables are labeled in descending order of the percentage of variance explained as the first, second, etc., principal components. Since every informant is selected to provide reliable and at least minimally valid data for T, and since this criterion is the only construct putatively common to all the informants, if the selection of multiple informants is well done, the first principal component should weight each informant’s data in the same direction, although not necessarily equally. The factor score on this first factor is a multi-informant estimate of T (which we denote as T*), which is largely free of the effects of C and P and is less affected by E than is any single informant’s measure. The second and third factors should correspond to contrasts in context (C*) and perspective (P*) that were built into the choice of informants. Since T*, C*, and P* are linear combinations of data from reliable informants, they should each be reliable. However, T* should be valid for T. Since C* and P* are designed to measure the influence of context and perspective on the expression or reporting of the trait that is independent of the trait itself, C* and P* are constructed to be invalid for T.

It should be emphasized that C* and P* may be interesting variables in their own right. For example, if perspective is structured as self versus other, P* would indicate how discrepant the child’s view of him/herself is from the view of others observing the child (e.g., mother and teacher). Such a measure itself may be an indication of a lack of self-awareness in the child or a lack of sensitivity in the adults. That we are here trying to remove the influence of C and P in order to focus on T reflects the goal of this effort to obtain a gold standard measurement of T and does not gainsay the importance of C and P. In fact, an understanding of how both context and perspective influence informants’ responses is necessary for successful application of these principles.

The final step of any process of developing a measure is, of course, validation, i.e., demonstration in independent samples from similar populations of the reliability and validity of the proposed multi-informant measure. In the following section, we present some evidence of validation when the process is correctly applied, as well as evidence from the validation attempt that would indicate the process has failed.

To test the viability and potential utility of the proposed approach to integrating multi-informant data, we used data from three different research projects: the Wisconsin Study of Family and Work (25), the MacArthur Three-City Outcome Study (26), and the Schoolchildren and Their Families Project (27). The three studies shared major features, specifically a focus on developmental psychopathology in middle childhood (ages 4–8 years) and the use of multiple methods (questionnaires, observations, experimentation) and multiple informants to garner reports on children’s symptoms, impairment, and health status. The studies differed, however, in overall design and in the characteristics of the samples. Specifically, the Schoolchildren and Their Families Project and the Wisconsin Study of Family and Work were prospective, longitudinal investigations of children’s development that utilized unselected community samples (25, 27). The MacArthur Three-City Outcome Study, by contrast, was a cross-sectional, case-control study in which half the sample was recruited from the community and half from mental health clinics or hospitals. This study was completed to test the reliability and discriminant validity of measures used in the Wisconsin Study of Family and Work. In addition to the references describing the studies, Table 1 provides details about the studies, particularly the range of children’s ages at the assessment(s) and the measures used to collect information from multiple informants.

Using the data for first-grade schoolchildren from the Wisconsin Study of Family and Work, we conducted multi-informant principal-component analyses with the mothers’, teachers’, and children’s reports (I) about three different childhood characteristics (T): 1) internalizing symptoms (depression, general anxiety, and separation anxiety), 2) externalizing symptoms (conduct problems, oppositionality, and overt aggression), and 3) academic functioning (academic competence and school engagement). As we discussed earlier, this combination of informants was one that might meet the theory’s mix-and-match criterion. The results of these analyses are presented in Table 2.

For each childhood characteristic, the data from all three informants loaded substantially and positively on the first factor, T*, which we interpret as a measure of T. This interpretation is supported by the facts that 1) the second factor (P*) contrasted the mothers’ and the teachers’ responses with those of the children, and 2) the third factor (C*) contrasted the mothers’ and the teachers’ responses, with the children’s responses (which incorporate both contexts) intermediate between the two and having a near-zero weight. These results capture the mixes and matches that motivated this choice of informants.

With data from the Wisconsin Study of Family and Work and the Schoolchildren and Their Families Project, Table 3 provides the results from a situation in which it is not clear that the three informants have been selected appropriately. Here, mothers’, fathers’, and teachers’ observations of children’s internalizing symptoms are used, but the children’s self-reports are not.

In the analysis of data from the Wisconsin Study of Family and Work shown in Table 3, the first factor weights each informant in the same direction and thus might conceivably be T*. However, the second factor, which appears to contrast teachers’ versus mothers’ and fathers’ responses, might be either C (home versus nonhome) or P (family versus nonfamily). The third factor contrasts fathers’ and teachers’ responses versus mothers’ responses, which seems inexplicable in terms of either context or perspective. In this case, there is a lack of empirical indication that C and P have been removed and thus that T is a more valid measure. This result is consistent with the theory.

In the analysis of data from the Schoolchildren and Their Families Project shown in Table 3, mothers’, teachers’, and laboratory-based observations of children’s internalizing behavior problems (T) are used to demonstrate further the effects of questionably selected informants. In this example, the third informant is a clinically trained research assistant who provided global ratings of children’s behaviors after a 2-hour laboratory visit and assessment. Here, none of the factors in the principal-component analysis is immediately identifiable as a measure of T*, C*, or P*. The first factor does not weight the data from all informants in the same direction, and neither the second nor the third factors indicate any relationship to context and perspective. Again, this failure is consistent with the theory.

The examples in Table 2 suggest (but do not prove) that the panel of informants consisting of mothers, teachers, and children will enable us to compute a measure of T that 1) combines each informant’s observation of that trait and 2) is relatively free from variance attributable to C and P. On the basis of the theory, it is predicted that T*, C*, and P* should all be reliable measures and that T* should be valid for T while C* and P* should not be valid for T. How well do these measures fulfill those predictions? For this purpose, data from the MacArthur Three-City Outcome Study were used, and T*, C*, and P* were calculated as in the Wisconsin Study of Family and Work described earlier.

Table 4 presents reliability coefficients for T*, C*, and P* separately in the clinical and community samples in the MacArthur Three-City Outcome Study. Despite the limitation in the range of values for the clinical and community samples (which usually depresses reliability), T*, C*, and P* all had satisfactory test-retest reliability, as did the separate scores of the three types of informants.

Table 4 also presents the effect size (standardized mean difference between the means for the clinical and community samples) indicating the discriminant validity for the presence/absence of clinical problems. It can be seen that T* is valid (an effect size of 0.8 is typically considered large) (30) but that P* and C*, as predicted, are not (an effect size of 0.2 is typically considered small). For internalizing symptoms, the effect size of T* is larger than the effect sizes for any of the categories of informant. For externalizing symptoms and ADHD, the effect size for T* is not the largest, but it must be remembered that referral of children, and thus their inclusion in the clinical sample, is typically based on the mother’s or teacher’s reports.

In a final set of analyses, data from the Schoolchildren and Their Families Project and the Wisconsin Study of Family and Work were used to give additional insights into the validity of T*. We examined three issues. First, are the results replicated from one study or site to another in community samples (i.e., are the results generalizable across communities)? Second, does the proportion of variance attributable to T change as a function of children’s age? The age ranges in these studies were very narrow, but in general one would expect that as the child matures, obtaining a gold standard measure would become easier and the percent of variance attributable to T would thus increase. Finally, how does each informant’s relative contribution (loadings) to the multi-informant measures of T, C, and P change as a function of children’s age? As the child matures, gaining in introspection and communication skills, one would expect that the weight placed on the child’s report would increase and the weight placed on the mother’s and teacher’s reports would decrease.

Table 5 presents the percent of variance accounted for by each multi-informant factor at four different grade levels during middle childhood: prekindergarten, kindergarten, and first grade in the Schoolchildren and Their Families Project, and first and third grades in the Wisconsin Study of Family and Work. For generalizability, the crucial comparison is between the first-grade samples at both sites. In both studies, the expected increase is seen in the percentage of total variance accounted for by T. Table 6 presents a closer look at changes in the mothers’, teachers’, and children’s weights in T* at each of these same time points. Again, the crucial comparison between the two first-grade samples indicates remarkable agreement between the two studies. Moreover, in most cases (reports on externalizing symptoms in the Schoolchildren and Their Families Project is the exception), the weight on the children’s reports tends to increase with the age of the child, reflecting the expected developmental growth in valid self-reports.