Mixed-Methods Designs in Mental Health Services Research: A Review

We conducted a literature review of mental health services research publications over a five-year period (January 2005 to September 2009), using the PubMed Central database and the following search terms: mental health services, mixed methods, and qualitative methods. Data were taken from the full text of each research article. Articles identified as potential candidates for inclusion had to report empirical research and meet one of the following selection criteria: a study specifically identified as a mixed-methods study in the title or abstract or through keywords; a qualitative study conducted as part of a larger project, including a randomized controlled trial, that also included use of quantitative methods; or a study that “quantitized” qualitative data (16) or “qualitized” quantitative data (17). On the basis of criteria used by McKibbon and Gadd (18) and Cresswell and Plano Clark (5), the analysis had to be fairly substantial; for example, a simple descriptive analysis of baseline demographic characteristics of participants was not sufficient to be included as a mixed-methods study. Further, qualitative studies that were not clearly linked to quantitative studies or methods were excluded from our review.

Using the same criteria and search terms, we also reviewed the NIH CRISP database (Computer Retrieval of Information on Scientific Projects) of projects funded over the same five-year period. Projects were limited to R series (independent research awards), F series (predissertation research awards), and K series (career development awards) grants. Data were taken from only the project descriptions provided by the applicant and contained in the database.

Using typologies employed in other fields of inquiry (5–7,9), we next assessed the use of mixed methods in each study to determine the study aims, rationale, structure, function, and process. Study aims referred to the objectives of the overall project that included both quantitative and qualitative studies or methods. The rationale for using mixed methods included conceptual reasons, such as exploration and confirmation (5), breadth and depth of understanding (19), and inductive and deductive theoretical drive (20). Pragmatic reasons for using mixed methods, such as addressing the weaknesses of one method by use of the other, and suitability to address research questions were also examined. Assessment of the structure of the research design was based on Morse's (7) taxonomy, which gives emphasis to timing (for example, using methods in sequence [represented by a → symbol] versus using them simultaneously [represented by a + symbol]) and to weighting (for example, primary method [represented by capital letters such as QUAN] versus secondary method [represented in lowercase letters such as qual]).

Assessment of the function of mixed methods was based on whether the two methods were being used to answer the same question or to answer related questions and whether they were used to achieve convergence, complementarity, expansion, development, or sampling (9). Finally, the process or strategies for combining qualitative and quantitative methods were assessed with the typology proposed by Cresswell and Plano Clark (5): merging or converging the two methods by actually bringing them together in the analysis or interpretation phase, connecting the two methods by having one build upon the results obtained by the other, or embedding one data set within the other so that one type of method provides a supportive role for the other method.

Our search identified 50 articles and 67 NIH-funded research projects published or funded between 2005 and 2009 that met our criteria for analysis. Seven of the NIH projects were excluded from further review because of missing data on the use of mixed methods. Three of the publications were based on one of the NIH-funded projects, and two other publications were based on one funded project each. Any redundant aims or strategies for combining qualitative and quantitative methods identified in linked publications and projects were counted only once in our analysis.

A list of the 26 journals in which the articles were published and the journals' impact factors (IFs) is presented in Table 1. One-fifth of the articles were published in Psychiatric Services. The 2008 IFs of the journals for which information was available ranged from .74 (Psychiatric Rehabilitation Journal) to 4.84 (Journal of the American Academy of Child and Adolescent Psychiatry). Twenty-one of the 50 articles (42%) had an IF of 2.0 or greater. Of the funded grants, three were predissertation research grants (F31s), 28 were career development awards (K01, K08, K23, K24, and K99), and 29 were independent research awards (R01, R03, R18, R21, R24, and R34).

Table 2 presents the year of publication for the 50 articles and the start date of the 60 funded projects. Sixteen of the projects funded during this period had a start date before 2005. The smaller numbers of publications and of projects in 2009 reflect the shorter period of observation (nine months) for that year. There was an exponential increase in the number of publications between 2005 and 2008, and the number of grants from 2005 to 2009 was more than twice that of the previous five-year period (2000–2004).

Table 3 summarizes for comparison the use of mixed-methods designs on the basis of study aims. Our analyses revealed the use of mixed methods to accomplish five distinct types of study aims and three categories of rationale. We further identified seven structural arrangements, five uses or functions of mixed methods, and three ways of linking quantitative and qualitative data together. Some papers and projects included more than one objective, structure, or function; hence the raw numbers may occasionally sum to more than the total number of studies examined. Twelve of the 50 articles presented qualitative data only but were part of larger studies that included the use of quantitative measures. Further, we identified four commonly used designs, with each design associated with a specific aim or set of aims (Figure 1).

The results of our analysis indicate that there has been substantial progress in using mixed-methods designs in mental health services research in response to efforts by NIMH (2,3) and other funding agencies to promote their use. Evidence for this progress is found in the increasing number of research projects that use mixed methods. The number of projects with mixed-methods designs funded over the five-year study period was more than twice the number that began in the previous five-year period (2000–2004). Furthermore, a majority (52%) of these funded projects were predissertation or career development awards used by junior and midlevel investigators to acquire expertise in mixed-methods research.

We also observed a notable increase in the number of studies based on mixed-methods designs published each year during this five-year period. The number of published mental health services research studies with mixed-methods designs increased by 67% between 2005 and 2006, by 80% between 2006 and 2007, and by 155% between 2007 and 2008. Furthermore, 21 of the 50 published studies (42%) that we reviewed appeared in journals with 2008 IFs of 2.0 or higher, including ten articles published in Psychiatric Services; four articles appeared in a journal with an IF of 4.0 or higher. In contrast, McKibbon and Gadd (18) reported that only 11 of 37 (30%) mixed-methods studies of health services appeared in a journal with an IF of 2.0 or higher in the year 2000.

Despite this progress, however, our review also suggests that there is room for improvement in use of mixed-methods designs. Most studies did not make explicit or provide support for the reasons for choosing a mixed-methods design; rather, we were forced to infer the rationale based on statements explaining what the methods were used for. Researchers may have felt that such explicit statements were as unnecessary as statements explaining the rationale for using certain quantitative methods, such as analysis of variance or survival analysis. However, the absence of an explicit rationale may also reflect a lack of understanding or appreciation of mixed-methods designs or a decision to use them without necessarily integrating or “mixing” them (5,6).

Most studies failed to provide explicit descriptions of the design structure or function that used terminology found in the mixed-methods literature; use of such terminology is consistent with the general standards for high-quality mixed-methods research recommended by Cresswell and Plano Clark (5). Further, three-fourths of the 50 published studies reviewed assigned priority to the use of quantitative methods, seven of the studies performed qualitative analyses of one or two open-ended questions attached to a survey, and 17 of the studies provided no references justifying their procedures for qualitative data collection or analysis. This may reflect an underappreciation of qualitative methods, as Robins and colleagues (1) have argued, or it may reflect a greater need for quantitative methods at the present time.

Although it was beyond the scope of this review to determine whether each study used mixed methods in effective ways, we note that each study was subjected to rigorous peer review before being published or funded, and each was judged by this process to make a valuable contribution to the field of mental health services research. These studies also provide evidence of meaningful and sensible variations in mixed-methods approaches to achieving various kinds of study aims and offer some guidance for integrating quantitative and qualitative methods in mental health services research. For instance, the choice of a mixed-methods design appears to be dictated by the nature of the questions being asked by mental health services researchers. Qualitative methods were used to explore a phenomenon when there was little or no previous research or to examine that phenomenon in depth, whereas quantitative methods were used to confirm hypotheses or examine the generalizability of the phenomenon and its associated predictors.

A majority of studies aiming to develop new practices or adapt existing practices to new populations had the same structure (beginning with a small qualitative study before developing or adapting the practice that was to be evaluated by using quantitative methods, which was found in 84% of the studies and projects) and the same process (connecting the findings of one set of methods with those of another set, which was found in 90% of the studies and projects). These studies reflect a growing awareness of the need to incorporate the preferences and perspectives of both service consumers and providers to ensure that new practices will be acceptable as well as feasible (32,36–39).

Studies of existing services also tended to be sequential in structure, with qualitative methods used to elaborate or explain the findings of quantitative studies. In the majority of these studies, the process of mixing methods involved either merging two sets of data to achieve convergence or connecting them to achieve expansion (5). A similar pattern was observed in studies that aimed to explore issues related to the needs for mental health services or provide more depth to our understanding of those needs. Such studies also appeared more likely to transform or “quantitize” qualitative data (24,35).

Randomized controlled trials and studies of implementation also shared similar patterns in use of mixed methods, including simultaneous use of both methods to achieve complementarity by embedding a qualitative or qualitative-quantitative study within a larger quantitative study, such as a randomized controlled trial. In the randomized controlled trials, qualitative methods were usually used to evaluate the process of providing the practice or intervention, whereas quantitative methods were used to evaluate the outcomes (25,40). In implementation research studies, qualitative methods were used to explore or provide depth to understanding barriers and facilitators of intervention implementation, whereas quantitative methods were used to confirm hypotheses and provide breadth to understanding by assessing the generalizability of findings (41,42).

The choice of mixed-methods designs was also dictated by how the individual questions being addressed by each method were related to one another. Studies that used different types of data to answer the same question reflected the function of convergence in a simultaneous structure, where data were merged for the purpose of triangulation, or a sequential structure, where qualitative data were transformed into quantitative data. Studies that used different types of data to answer related questions reflected the function of complementarity, in which quantitative methods were used to measure outcomes, describe content (for example, fidelity of services used and the nature of the mental health problem), and provide breadth (generalizability) of understanding, whereas qualitative methods were used to evaluate the process of service delivery (43–45), describe context (for example, setting) (26,34,46), describe consumer values or attitudes (35,42,47), and provide depth (meaning) of understanding (28,48) in a simultaneous structure and embedded data process. Expansion, development, and sampling were also used to provide answers to related questions that could not be answered by one method alone, usually in a sequential structure in which data sets were merged or connected together (24,30,37).

Finally, the choice of design appears to be based on the strengths of one method relative to the weaknesses of the other. For instance, expansion was used to explain findings based on quantitative data with qualitative data because explanation was not possible with the quantitative methods alone (25,27,40). In convergence, both sets of methods were used to confirm or validate one another, especially in instances where limited samples precluded testing of hypotheses with sufficient statistical power (30,49) and where limitations to qualitative data collection raised concerns about objectivity and transferability of results. In studies developing new methods, conceptual models, and interventions, qualitative methods also served to enhance quantitative analysis by laying the groundwork essential for more valid measurement and theory and more effective, usable, and sustainable interventions (37). Sampling also worked to enhance validity by using qualitative methods to enhance quantitative methods by developing targeted comparisons or by using quantitative methods to enhance qualitative methods by establishing criteria for purposeful sampling (36).

In summary, the choice of a mixed-methods design appears to be associated with three considerations: the nature of the question being asked (inductive-exploratory or deductive-confirmatory), how the questions being addressed by each method are related to one another, and the strengths of each method relative to the weaknesses of the other.

Caution should be exercised in interpreting these findings given limitations in our study design and analysis. Despite our efforts to be comprehensive in the search process and to select studies and projects on the basis of criteria with face validity, we undoubtedly excluded several articles or projects that used mixed methods. For example, we may have excluded mixed-methods projects listed in the CRISP database that did not specify use of qualitative or mixed methods in the abstracts. We may have also excluded published articles with qualitative data that were part of larger, primarily quantitative studies if the articles did not reference the larger studies, or we may have excluded articles not listed in PubMed Central. In the absence of explicit information, we were often forced to infer the structure, rationale, and function of the design based on statements contained in the available material. Similarly, the CRISP abstracts describe only what the investigators proposed to do with mixed methods and do not indicate what was actually done. Our use of existing typologies of structure, function, and process were intended to serve as a starting point in our analysis rather than an attempt to “pigeon-hole” each study into a specific typology. Our assessment of the progress made in the application of mixed-methods designs in response to calls for their use by funding agencies did not include indicators of whether these efforts had produced more useful, incisive, or insightful knowledge for the purpose of addressing mental health services questions and problems. Such an assessment would require comparisons with the products of studies based on monomethod designs, which was beyond the scope of this study.

Finally, it should be noted that the typology of mixed-methods use does not represent a set of standards for using mixed methods per se but is an important first step toward the development of such standards. Typologies by themselves do not explain why a particular method should be used and how to use a method appropriately. However, as Teddlie and Tashakkori (6) observed, there are five reasons or benefits to developing such a typology: typologies help to provide the field with an organizational structure, they provide examples of research designs that are clearly distinct from either qualitative or quantitative research designs, they help to establish a common language for the field, they help researchers decide how to proceed when designing their studies, and they are useful as a pedagogical tool. A consensus conference or workshop bringing together experts in mixed methods and mental health services research to evaluate the empirically generated typology found in current patterns of mixed-methods use would appear to be the next logical step in developing a set of standards. Such standards would also be required to adhere to the epistemological foundations of each method when used separately (for example, whether appropriate considerations are made to ensure the generalizability of quantitative results or theoretical saturation of qualitative data and whether each method is appropriately matched to the inductive or deductive theoretical drive of the study) and when combined (for example, whether the knowledge gained when using the two methods together is more insightful and of greater value than the knowledge gained when using them separately).

Despite the limitations described above, the findings suggest an increasing use of mixed-methods designs to address changing priorities in mental health services research and a consensus as to how such methods should be applied. The lack of explicit statements explaining the rationale for using mixed methods and the evident priority assigned to quantitative methods suggest that there is room for improvement. However, these studies appear to utilize a common set of designs and provide guidance for using mixed methods, with varying approaches based on the nature of the question being asked (exploratory or confirmatory), how questions being addressed by each method are related to one another, and the strengths of each method relative to the weaknesses of the other.

This study was funded through NIMH grant P50-MH50313-07.

The authors report no competing interests.