Measuring Patients’ Acquisition of Therapy Skills in Psychotherapy for Depression: Assessing the CCTS-SR and the IPSS-SR

The sample consisted of 202 patients with depression who met the following criteria: a DSM-IV diagnosis of major depressive disorder, age ≥18 years, no use of antidepressants or antidepressant use unchanged for at least 3 months prior to study entry, sufficient knowledge of the Dutch language, and a score >19 on the Beck Depression Inventory–II (BDI-II). Most patients were women (62%) (N=127), and the mean age was 38.24 years (SD=12.20). A total of 10% (N=20) had finished no education or lower education (no former education or special lower education, primary school or practical training school), 52% (N=105) had completed middle level education (lower or higher general secondary education or intermediate vocational education), and 38% (N=77) had completed higher education (higher vocational education, pre-university education, or university). The patient group was part of a treatment study described elsewhere (25). Participants completed the measures online while assigned to a waiting list to receive psychotherapy for depression in one of the participating Dutch specialized mental health organizations.

All procedures contributing to this work complied with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975 and its most recent revision. Informed consent was obtained from all participants.

A detailed description of statistical procedures is presented in Appendix A of the online supplement. First, we conducted a CFA on the Dutch version of the CCTS-SR. Second, we used an EFA to investigate the factor structure of the IPSS-SR, followed by a CFA to test the factor solution that resulted from the EFA and to compare this factor solution to the theory-based factor solution targeted to five themes. Third, we computed Cronbach’s alpha as an indicator of internal consistency for both questionnaires. Fourth, using the final versions of the CCTS-SR and IPSS-SR, we tested the correlations between the skills learned in therapy and the patients’ demographic characteristics (age, education level, and sex), dysfunctional thinking, behavioral activation practices, and depressive symptoms.

For the CCTS-SR, the initial CFA showed a poor fit for the one-factor solution (see Table 2, model 1). Subsequently, we investigated whether model fit could be improved by deleting items with factor loadings <0.3 and inspecting the modification indices. No items had factor loadings <0.3. Modification indices pointed to a high error covariance between items 2 and 3, 13 and 14, and 17 and 18, suggesting that these pairs of items shared a common variance unrelated to the one-factor model. Inspection of the items suggested that the similarity between items 13 and 14 and items 17 and 18 might be responsible for the high error covariance, and we therefore decided to delete items 13 and 18. Items 2 and 3 were retained as distinct items. For the chi-square test, deleting items 13 and 18 resulted in a better model fit for the patients with depression, but fit on the Comparative Fit Index (CFI), Tucker-Lewis Index (TLI), and root mean square error of approximation (RMSEA) remained outside the acceptable range (see Table 2, model 2). The internal consistency of the CCTS-SR was acceptable. Cronbach’s alpha was 0.91 for both the original version of the questionnaire and for the version without items 13 and 18. Tucker’s coefficient of congruence was 0.97, indicating that the one-factor solution in our study was highly similar to the one-factor solution reported by Strunk and colleagues (17).

Factor loadings for each model and for the English and Dutch versions of the CCTS-SR can be found in Appendixes B–D of the online supplement. Mean and standard deviations for both the 29-item and 27-item versions of the questionnaire can be found in Table 3.

The results of the EFA of the IPSS-SR showed a Kaiser-Meyer-Olkin (KMO) score of 0.86, indicating that the data matrix was valid for the factor analysis (31). Seven eigenvalues >1 were identified, and the parallel analysis pointed to six factors. The scree plot pointed to the presence of three to seven factors. These initial solutions can be found in Appendixes E–G of the online supplement. On the basis of the values in the parallel analysis (four eigenvalues >1, with the remaining eigenvalues <1 and closer to the randomly created values) and the scree test, we decided to retain four factors in the final model and to test it by using a CFA. We conducted a second EFA with all 31 items forced into four factors (see Appendix H of the online supplement for the factor loadings). We deleted items 11 and 21 because their factor loadings were <0.3 for the factors in the pattern matrix. No items had cross-loadings less than a 0.15 difference from an item’s highest factor loading in the pattern matrix. We conducted a third EFA without items 11 and 21 and by forcing the model to retain four factors (see Appendix I of the online supplement for the factor loadings). Subsequently, we tested this four-factor structure by using a CFA and compared it with the original five-factor solution that was expected based on the development of the IPSS-SR. The four-factor model resulted in mediocre model fit, but the theoretically based five-factor solution showed even worse fit (Table 2, models 3 and 4). Therefore, we decided to additionally explore and test three-, six-, and seven-factor models and a higher-order solution model (i.e., in this type of model, one higher order factor is modeled to explain the correlations between the theoretically based five factors). Before testing the solutions by using a CFA for each potential factor solution, the number of factors was forced into the EFA to identify items that should be deleted because of low factor loadings (<0.3) or cross-loadings with less than a 0.15 difference from an item’s highest factor loading in the pattern matrix. The higher-order structure and seven-factor solutions (both models including all items) could not be estimated, the three-factor model (in this model, items 11 and 19 had been deleted due to a factor loading <0.3) showed no better fit, but the six-factor structure (this model included all items) showed better fit compared with the initial four-factor model (Table 2, models 5–8). However, output of the six-factor model indicated a correlation >1 between the first and sixth factor, and we therefore decided to merge these factors into the first factor. Face validity of the five remaining factors was inspected and considered low for one of the factors (factor 5, items 10, 16, 17, 22, 24 and 25) (for the test of this model, see Table 2, model 9). Scaling these items under factors they seemed to belong to (factor 3, items 10, 16, 24; factor 2, items 17, 22, and 25) increased face validity but did not improve fit (Table 2, model 10), and we therefore decided to delete these items. Deleting the items led to better fit (Table 3, model 11), although mediocre results were shown on the CFI, TLI, and RMSEA. The remaining four factors were “communication skills and social support” (factor 1), “understanding my own feelings” (factor 2), “coping with grief and major life change” (factor 3), and “understanding feelings of others” (factor 4).

Cronbach’s alpha was 0.88 for the final 25-item version of the IPSS-SR. The final factor loadings and the original and final versions of the IPSS-SR can be found in Appendixes J–M of the online supplement. Means and standard deviations for the final IPSS-SR total- and subscale scores are shown in Table 3. Pearson correlations between the IPSS-SR total and subscales scores are shown in Table 4.

We found a moderate and positive correlation between the CCTS-SR and the IPSS-SR (r=0.41); patients with depression who had higher scores on the CTSS-SR tended to also have higher scores on the IPSS-SR. As shown in Table 5, the CCTS-SR and IPSS-SR scores were not correlated with age, but higher education level was related to higher IPSS-SR scores. Higher scores on both the CCTS-SR and the IPSS-SR total, and on the first, third, and fourth factor scores were related to lower scores on the BDI-II and to higher scores on the BADS. Lower scores on the CCL were related to higher total scores on the IPSS-SR and to higher scores on the first, third, and fourth factors of the IPSS-SR.

First, future psychometric studies should investigate the factor structure of the IPSS-SR and CCTS-SR and test whether the structure of the scales can be strengthened by developing and adding new items to measure the different elements of the scales or by allowing items to load on multiple factors (if theoretically plausible). Also, the structure of the IPSS-SR should be tested in other samples to see whether the results from this sample translate to samples with more variation in and less severe depression. Second, construct validity should be further evaluated by investigating whether the scores on the measures are related to scores on similar measures (e.g., scores on the CCTS-SR should be compared with scores on observational instruments that measure CBT skills to measure convergent validity; scores on the IPSS-SR should be compared with instruments that measure other established psychological constructs, such as mindfulness, reflective functioning, or interpersonal functioning, to measure discriminant validity). In addition, follow-up studies should determine whether the IPSS-SR factor “understanding my own feelings” is a relevant part of the IPSS-SR or whether it relates more closely to other psychological constructs (e.g., reflective functioning). Third, future studies should investigate how the CCTS-SR and IPSS-SR perform psychometrically in the context of a randomized controlled trial focused on mechanisms of change and whether these instruments can play a role in explaining change or outcomes in psychotherapy for depression. Fourth, future studies should show that improvement of skills learned in therapy is not merely a reflection of symptom change or increased knowledge about the constructs. Fifth, the test-retest reliability of both measures should be investigated to ensure that changes over time in therapy skills are not just a result of measurement error.

The CCTS-SR and IPSS-SR should be included in future treatment studies to investigate whether patients’ therapy skills change over time or predict relapse of depression. Moreover, showing that scores on the CCTS-SR improve during CBT more than during IPT, or the reverse, would provide evidence for the specific roles of CBT and IPT skills. Studies should further investigate how acquisition of CBT skills may translate to change in behavioral activation and dysfunctional thinking. Future research should test whether therapy skills act as a mediator between treatment and outcome while also evaluating whether scores on the CCTS-SR and IPSS-SR change before depressive symptom changes occur. Probably the strongest test of the role of therapy skills would be to examine the effect of therapy skills in an experimental setting in which a therapy skill acquisition procedure is isolated and the direct effects of the procedure on the CCTS-SR and IPSS-SR scores and depressive outcome can be tested. Such an experimental study might also provide the opportunity to find out whether and how the acquisition of CBT skills may translate to less dysfunctional thinking and more behavioral activation.

Because mechanisms research has been scarce in the field of IPT for depression, a strength of the present study is that the development of the IPSS-SR might serve as a next step toward investigating what change mechanisms may be involved in the effects of IPT. Also, this study was the first to translate and investigate the use of the CCTS-SR in the Dutch population. A major limitation of the analysis was that only baseline data were available. In addition, the methods were limited because the factor solution found in the EFA was not cross-validated by using another sample during the CFA. Because fit of the factor structures was outside the acceptable range, interpretation of scores from the CCTS-SR, and IPT skills as represented by the four subscales, should be made with caution. Future studies are needed to investigate the factor structure of these measures in samples with broader variance in depression severity and in relation to pre- to posttreatment change in psychotherapy for depression.