The Road to Cognitive Skill Acquisition: Psychometric Evaluation of the Competencies of Cognitive Therapy Scale
Publication: American Journal of Psychotherapy
Abstract
Objective:
Cognitive therapy (CT) skills are an index of treatment progress. They predict changes in patients’ acute depressive symptoms and symptom relapses. However, the psychometric properties of the various measures of CT skills are poorly understood. This study aimed to investigate the factor structure of the Competencies of Cognitive Therapy Scale–Self Report (CCTS-SR) and assess its concurrent validity.
Methods:
The psychometric properties of the CCTS-SR were explored by using data from a panel of online respondents (N=410). The fit of a one-factor solution was explored by using a confirmatory factor analysis. Exploratory bifactor analyses (EBFA) were then conducted to determine other possible factor structures.
Results:
The one-factor solution did not fit the data well. Results of the EBFA suggested that the factor structure of the CCTS-SR may be characterized by a single underlying dimension capturing the general use of CT skills as well as by more specific factors the authors labeled “behavioral activation” and “CT comprehension.” The variance captured by the factor initially labeled as CT comprehension was correlated with measures of depression and emotional dysregulation, suggesting that these items do not capture CT comprehension and should be removed from the scale.
Conclusions:
The CCTS-SR seems to be characterized by more than a single factor, and items that seemingly compose CT comprehension (i.e., items 13 and 14) may need to be removed. Although the CCTS-SR may be a valid index of therapy progress, more attention needs to be paid to its psychometric properties.
Highlights
•
The Competencies of Cognitive Therapy Skills–Self Report is a low-cost assessment of cognitive therapy (CT) skills.
•
A one-factor structure of the measure did not achieve acceptable fit.
•
The analyses suggested the presence of a “behavioral activation”–specific factor.
•
The removal of items composing a “CT comprehension”–specific factor should be further evaluated.
Cognitive therapy (CT) is an efficacious psychosocial intervention for many common mental disorders (1–3). In particular, CT has been studied as a treatment for depression, exhibiting acute treatment results similar to those of antidepressants and superior long-term outcomes (4, 5). Although there is some debate regarding the mechanisms of change in CT (6–8), the therapy’s prophylactic effects may be driven by instilled skills that patients can use beyond the therapy room (9, 10). These skills can include the ability to detect automatic negative thoughts and generate realistic alternatives as well as behavioral skills to decrease avoidance and increase rewarding activities. Existing data support the hypothesis that CT skills predict lower relapse after CT for depression (9, 11). Thus, one way to identify individuals at risk for depressive relapse is to measure CT skills throughout treatment.
The various measures for evaluating CT skills include the Ways of Responding Questionnaire (WOR) (12), the Performance of Cognitive Therapy Skills Scale (PCTS) (9), and the Skills of Cognitive Therapy (13). Although data support their validity, these measures are burdensome to administer. For example, the WOR requires the use of trained raters to code participant responses to hypothetical scenarios, and the PCTS requires session reviews by trained therapists. To address these issues, Strunk and colleagues (10) developed the Competencies of Cognitive Therapy Scale–Self Report (CCTS-SR).
The CCTS-SR is a 29-item self-report measure that assesses a patient’s CT skill acquisition. Although the current literature assumes that the scale has a one-factor solution, few studies have explored its structure. Strunk and colleagues’ (10) initial exploratory factor analysis of the original scale, which had 30 items, was conducted with data from a sample of 67 treatment-seeking patients with depression and suggested a one-factor solution. During the analysis, Strunk removed one item because of a low factor loading, leading to a 29-item scale. In Strunk and colleagues’ study (10), the CCTS-SR showed concurrent validity, having significant positive correlations with the WOR and negative correlations with depressive symptoms pre- and postintervention. In another study, Bruijniks and colleagues (14) tested the one-factor solution, by using a Dutch-language CCTS-SR scale with data from a sample of 202 treatment-seeking individuals. Bruijniks et al. did not find adequate fit from a one-factor confirmatory factor analysis (CFA), even after deleting two items that fit poorly within the scale.
Given the limited research on the structure of the CCTS-SR, we explored its dimensionality with data from an online sample. Although previous analyses were unable to find adequate fit for a one-factor solution, the scale has primarily been assumed to be unidimensional (14–17). In the current study, we sought to explore the fit of the unidimensional model in a sample larger than that of prior studies. Additionally, to explore the concurrent validity of the dimensions of the scale, we correlated the CCTS-SR and its dimensions to established, conceptually related constructs: cognitive reappraisal (CR), the basic emotion-regulation process core of CT (6, 7); emotion dysregulation, which should be negatively related to CT skills (18); and depressed mood, which CT skills should protect against (10).
Methods
Participants
A total of 471 participants were recruited via a panel on the Qualtrics website. Approval from the Indiana University Institutional Review Board was obtained on April 13, 2020, and data were collected April 16–20, 2020. Participants were deemed eligible if they were age 18 or older, passed a reCAPTCHA check, committed to giving correct answers, and spent at least 5 minutes on the survey. Because of the relatively large sample size, the data were deleted listwise (i.e., participants were not analyzed unless they answered all questions), leading to a final sample of 410. The data set generated and analyzed for the current study is available online (https://osf.io/zhj5k).
Measures
CCTS-SR.
The CCTS-SR (10) was developed to assess patients’ use of CT skills. For the current study, we used the 29-item scale developed by Strunk and colleagues, although one study (14) removed items 13 and 18 because of poor fit. Each item was rated on a 7-point scale (1, not at all, to 7, completely), producing scores ranging from 29 to 203, with higher scores indicating greater command of CT skills. The mean±SD CCTS-SR score was 120.98±37.74, indicating moderate use of CT skills. Previous analyses of populations with clinical conditions have shown CCTS-SR scores ranging from 80 to 97. In the current sample, Cronbach’s alpha for the scale was 0.97. (For the contents of the CCTS-SR items see the online supplement to this article).
Patient-Reported Outcomes Measurement Information System—Emotional Distress, Depression (PROMIS-D) Short Form.
The PROMIS-D Short Form measure (19) is an 8-item self-report measure assessing depression severity over the past 7 days. Items are rated on a 5-point scale (1, never, to 5, always), with higher scores indicating higher symptom severity. The scale consists of short, simple statements (e.g., “I felt worthless”), and possible raw scores range from 8 to 40. The mean score for the current sample was 19.73±9.17, indicating mild depression. Cronbach’s alpha for the scale was 0.96.
Difficulties in Emotion Regulation Scale (DERS-18).
The DERS-18 (20) is an 18-item self-report measure that assesses an individual’s level of emotional dysregulation (e.g., “When I’m upset, I lose control of my behaviors”). Items are rated on a 5-point Likert scale (1, almost never, to 5, almost always). Scores range from 18 to 90, with higher scores indicating greater emotional dysregulation. The mean DERS-18 score for the current sample was 43.39±15.15, indicating moderately developed emotion regulation skills. Cronbach’s alpha for the scale was 0.92.
Emotion Regulation Questionnaire (ERQ).
The ERQ (21) is a 10-item self-report measure that assesses individual differences in habitual use of two differing emotion regulation strategies: CR (“When I want to feel less negative emotion, I change the way I’m thinking about the situation”) and expressive suppression (“I keep my emotions to myself”). Items are rated on a 7-point Likert scale (1, strongly disagree, to 7, strongly agree). Only the CR subscale was used in this study. Scores range from 6 to 42, with higher scores indicating more frequent use of CR. The mean ERQ-CR score was 29.36±6.83 for the current sample. Cronbach’s alpha for the ERQ-CR was 0.84.
Data Analysis
All analyses were conducted by using the R programming language. An initial CFA fitting a one-factor solution was conducted by using the Lavaan package (22). Goodness of fit for CFA models was evaluated by using the comparative fit index (CFI) (23), the Tucker-Lewis Index (TLI) (24), the standardized root mean square residual (SRMR) (25), and the root mean square error of approximation (RMSEA) (26) at its 90% confidence interval (CI). Criteria for acceptable model fit were defined as follows: CFI and TLI >0.9, SRMR <0.9, and RMSEA <0.08 (27).
Given the lack of good fit for the unidimensional model, we conducted an exploratory factor analysis (EFA) (28) to inform the factor structure. Prior to conducting the EFA, a Kaiser-Meyer-Olkin (KMO) (29) score of 0.97 was obtained, indicating the data were adequate for factor analysis. EFA was performed on the data set by using a parallel analysis with the GPArotation package (30) and a nonorthogonal oblimin rotation, because of high correlations among the factors. The number of specified factors were determined by using a scree plot (28), parallel analysis, and the Guttman-Kaiser Criterion (31, 32). The data set appeared to have a large ratio between the first and second eigenvalues, indicating the possibility of a bifactor structure (33). Thus, an exploratory bifactor analysis (EBFA) was performed on the data by using the bifactor rotation specified by Jennrich and Bentler (34). A minimum residuals factor estimation method and a factor loading cutoff score >0.30 were used. Although it may seem unorthodox to switch between CFA, EFA, and EBFA, there has recently been a resurgence of support for EBFA (35), in part because of growing dissatisfaction with the results of strict CFA approaches and trends toward more flexible forms of modeling (36–38). Finally, we correlated factor scores of the final CCTS model to measures of depression (PROMIS-D Short Form), emotion dysregulation (DERS-18), and CR (ERQ-CR).
Results
Participants
Most participants identified as male (55%, N=225) or female (44%, N=182), with a few respondents identifying as transgender or gender nonbinary (1%, N=3). Mean age was 46.4±16.03 (range 18–81). Most participants were Caucasian (74%, N=302; African American, 12%, N=49; Hispanic/Latino, 10%, N=43; Other, 4%, N=16). About half of the participants had an associate degree (52%, N=213); most had a GED equivalent (96%, N=394). The average participant maintained an annual household income of $50,000–$74,999 (20%, N=84), only a small percentage had an income greater than $200,000 (5%, N=19), and a significant percentage had an income less than $15,000 (14%, N=57). About half of the participants were identified as dating or married (55%, N=227), and just under half as not in a relationship (i.e., single, widowed, or divorced; 45%, N=183). A minority identified as members of the LGBT community (12%, N=49).
Factor Structure
The one-factor structure for the CCTS-SR did not fit the data well (χ2=1610.9, df=350, RMSEA=0.094 [90% CI=0.09, 0.10], CFI=0.85, TLI=0.84). We evaluated whether the fit could be improved by removing items with low factor loadings, but all items had factor loadings >0.30, and no items accounted for a significantly low proportion of the total variance. We explored the reliability of individual items by calculating Cronbach’s alpha with the items deleted. There was no evidence that reliability improved with the removal of items. Additionally, assessments were performed on the CCTS-SR to find problematic items (see online supplement). All total and corrected item correlations ranged from 0.60 to 0.81, indicating excellent discrimination.
We performed an EFA to help inform whether another factor model would provide a better fit. The parallel analysis suggested four factors. The scree plot showed an indistinct elbow, indicating two to four factors. The Guttman-Kaiser Criterion suggested a two-factor solution. The first two eigenvalues of the solution were 15.16 and 0.99. Because of the arbitrariness of the criterion (39), we assumed that the eigenvalues indicated a possible second factor. We generated two-, three-, and four-factor models. The results of these models were difficult to interpret and contained multiple cross-loadings (see online supplement).
The results of the EFA showed that the ratio between the eigenvalues of the first and second factors was 15.3 to 1. This result, along with the difficulty interpreting the factor solutions, suggested a higher-order general factor. Thus, we conducted an EBFA to investigate the scale’s dimensionality. On the basis of our EFA results, we specified bifactor models with two, three, and four specific factors (see online supplement). Only the bifactor model with two specific factors was interpretable and did not contain extraneous factors or cross-loadings. We labeled the two specific factors as “behavioral activation” (e.g., “I made an effort to engage in enjoyable activities”) and “CT comprehension” (e.g., “I have been noticing negative thoughts maintain my depression”). Table 1 shows the loadings of the model, and Figure 1 provides a visual depiction.
| Item | CCTS-G | BA | CC |
|---|---|---|---|
| 1 | .75 | ||
| 2 | .78 | ||
| 3 | .71 | ||
| 4 | .79 | ||
| 5 | .78 | ||
| 6 | .77 | ||
| 7 | .83 | ||
| 8 | .70 | ||
| 9 | .83 | ||
| 10 | .63 | ||
| 11 | .75 | ||
| 12 | .72 | ||
| 13 | .64 | .35 | |
| 14 | .68 | .46 | |
| 15 | .71 | ||
| 16 | .69 | .43 | |
| 17 | .64 | .59 | |
| 18 | .62 | .58 | |
| 19 | .68 | .43 | |
| 20 | .77 | ||
| 21 | .58 | ||
| 22 | .66 | ||
| 23 | .74 | ||
| 24 | .73 | ||
| 25 | .73 | ||
| 26 | .67 | ||
| 27 | .72 | ||
| 28 | .75 | ||
| 29 | .73 | ||
| Proportion of variance | .52 | .05 | .03 |
| Cumulative variance | .52 | .57 | .60 |
a
BA, behavioral activation factor; CC, cognitive therapy comprehension factor; CCTS-G, Competencies of Cognitive Therapy Scale–Self Report general factor. Only factor loadings <.30 are shown.
FIGURE 1. Bifactor model assessing the CCTS-SR with two specific factorsa
a BA, behavioral activation factor; CC, cognitive therapy comprehension factor; G, general factor. Error variances are not shown. Gray numbers represent factor loadings. Dashed arrows indicate fixed loadings. Cognitive Therapy Scale–Self Report (CCTS-SR) item numbers are indicated by a C and the proceeding item number.
Indices of fit generated during the EBFA indicated acceptable fit for the two specific factor bifactor model (TLI=0.92 and RMSEA=0.62, 90% CI=0.06, 0.07). Cronbach’s alpha did not increase by dropping items in either specific factor, indicating that all items positively contributed to scale reliability (see online supplement).
Concurrent Validity
We correlated the general CCTS factor and the specific factors with measures of depression, emotional dysregulation, and CR (Table 2). The general CCTS factor displayed moderate positive correlations with CR (ERQ-CR, r=0.53, 95% CI=0.46, 0.60, p<0.001), but nonsignificant correlations with the measures of depression and the DERS-18. The behavioral activation specific factor showed small-to-medium negative associations with PROMIS-D Short Form (r=–0.19, 95% CI=–0.28, –0.09, p<0.001), CR (r=0.18, 95% CI=0.08, 0.27, p<0.001) and DERS-18 (r=–0.27, 95% CI=–0.35, –0.17, p<0.001). Surprisingly, the CT comprehension factor displayed moderate positive associations with depression (r=0.38, 95% CI=0.30, 0.46, p<0.001) and emotion dysregulation (r=0.38, 95% CI=0.30, 0.46, p<0.001) and a small negative relationship with CR (r=–0.16, 95% CI=–0.26, –0.06, p<0.001). These results may be taken to suggest that the two items that make up the specific CT comprehension factor may not actually measure processes related to CR but may instead covary with negative mood and emotion dysregulation. We repeated these analyses by using data from a subsample of individuals (N=165) with at least moderate symptoms of depression (>22 on the PROMIS-D Short Form). In general, the results were replicated between the full sample and the subsample with moderate depression (see online supplement).
| Scale | CCTS-G | BA | CC | PROMIS-D | DERS-18 |
|---|---|---|---|---|---|
| PROMIS-D | .05 | –.19** | .38** | ||
| DERS-18 | .08 | –.27** | .38** | .66** | |
| ERQ-CR | .53** | .18** | –.16* | –.06 | –.14* |
a
BA, behavioral activation specific factor from the 2-factor bifactor model; CC, cognitive therapy comprehension factor from the 2-factor bifactor model; CCTS-G, Competencies of Cognitive Therapy Scale–Self Report general factor from the 2-factor bifactor model; DERS-18, Difficulties in Emotion Regulation Scale; ERQ-CR, Emotion Regulation Questionnaire–Cognitive Reappraisal Scale; PROMIS-D, Patient-Reported Outcomes Measurement Information System—Emotional Distress, Depression Short Form.
*
p<.01,
**
p<.001.
Discussion
This study was the first, to our knowledge, to evaluate the psychometric properties of the CCTS-SR with a non–treatment-seeking sample. The CCTS-SR has demonstrated predictive validity in relation to future depressive symptom change (15) and relapse among populations with clinically elevated symptoms (10). Given our difficulty fitting a unidimensional model, we explored the possibility of a multidimensional structure by using an EBFA. Although this method has been used for more than 70 years, use of EBFA has only recently gained traction within the research community (40). Its use before performing confirmatory analyses has been recommended by psychometric researchers (40) for its advantages in interpreting polytomous data and in representing hierarchical models of factor structures (33, 34).
To our knowledge, since the first publication of the CCTS-SR, only one published study (10) has explored the structure of the CCTS-SR via an EFA. This previous study used a relatively small sample of 67 patients, as opposed to the current sample of 410, and potentially lacked the sensitivity to detect the factors reported (41, 42). Another study used a Dutch-translated version of the CCTS-SR with a Dutch sample and explored a one-factor model that did not fit the data well (14). Our analyses suggested the presence of a general dimension capturing CT skills and a specific factor capturing the use of behavioral activation skills. Researchers should be mindful of the relative contribution of the behavioral activation–specific factor when exploring mechanisms of change in therapy. A second specific factor, CT comprehension, was composed of two items and initially seemed to capture an understanding of the relationship between negative thinking and mood (e.g., “I have noticed inaccurate, negative thoughts help maintain my depression”). However, there were reasons to doubt the validity of this specific factor. First, only two items loaded onto the CT comprehension factor. Moreover, this specific factor had moderately positive correlations with depression and emotion dysregulation and a negative association with CR. Most of the CCTS-SR items avoid referring to the person’s mood (e.g., item 21: “I often caught myself thinking in an irrational way, and I actively worked to develop more rational views”) or refer to a person coping with how their mood changes. However, the wording on items 13 and 14 may be taken to imply that the respondent experiences depression and other negative moods (e.g., “I have been recognizing that inaccurate, negative thoughts and judgments help to maintain my depression”). Thus, it is possible that individual differences in this specific factor are attributable to whether people are likely to report more depression and other negative emotions.
The concurrent validity of the CCTS-SR’s general factor with the current sample was supported by our correlational analyses. As predicted, measures of CR (ERQ-CR) were moderately and significantly correlated with the CCTS-SR’s general factor. Although other literature points to the CCTS-SR’s predictive validity for reductions in depressive symptoms (10), the current data did not show a cross-sectional relationship between the CCTS-SR and symptoms of depression. Interestingly, one prior study (15) found no cross-sectional association between the CCTS-SR and depressive symptoms, although changes in CCTS-SR predicted changes in depression.
Strengths and Limitations
Our study had several limitations. First, the sample was nonclinical; thus, the conclusions drawn could not be assumed to apply to populations with clinically elevated symptoms. Nonetheless, our sensitivity analyses, conducted with data from a subsample of individuals with at least moderate depression showed results similar to our main analyses. Second, several items in the CCTS-SR, including items from the CT comprehension factor, may assess participant insight into a person’s own moods and cognitions. Because there was limited psychoeducation on the relationship between moods and cognitions, an item measuring insight may have been difficult for participants to rate.
The current study had several strengths that differentiate it from previous literature. The analyses were conducted with data from a sample of 410 participants, a larger sample size than prior studies had. The lack of an adequate sample size (41, 42), in conjunction with a dominant first factor, may explain why the previous EFA (10) was unable to detect a multifactor structure. Moreover, the analyses in the current study used indices of emotion regulation, allowing for assertions about the scale’s concurrent validity. Finally, the use of data from a general population sample provided a baseline for normative levels of CT skill use among the general population, without the exclusion criteria common in depression trials (43, 44).
Future Research
Currently, valid measures of CT skills require time to administer and score. A self-administered measure lowers barriers for assessing CT skills in clinical practice and research. More research is needed on the psychometric properties of the CCTS-SR. Thus, we encourage replication studies and studies exploring the scale’s factor structure. Additionally, the EBFA solution could appear to be better simply because of its added complexity. CFAs should be used with new samples to evaluate the goodness of fit for the bifactor solution, and items 13 and 14 should be reevaluated for removal from the CCTS-SR. Furthermore, the validity and utility of the model’s multidimensionality should be scrutinized because the presence of specific factors does not always warrant forming specific subscales (45). In other words, just because a measurement model with specific factors fits the data well does not mean that it will be the most clinically useful way of scoring the measure.
Conclusions
Prior research (10) suggests that the CCTS-SR may be a useful predictor of patient progress in CT. However, on the basis of the current analysis, the measure does not appear to be unidimensional. Several researchers have pointed out how psychological measurement is often a neglected area in psychology and that this deficit has had deleterious effects on the progression of science (e.g., difficulty with replication) (46). Future studies should attempt to confirm the factor structure of the CCTS-SR.
Supplementary Material
File (appi.psychotherapy.20210002.ds001.pdf)
- View/Download
- 195.61 KB
References
1.
Dobson KS: A meta-analysis of the efficacy of cognitive therapy for depression. J Consult Clin Psychol 1989; 57:414–419
2.
Hofmann SG, Asnaani A, Vonk IJ, et al: The efficacy of cognitive behavioral therapy: a review of meta-analyses. Cognit Ther Res 2012; 36:427–440
3.
Lorenzo-Luaces L: The evidence for cognitive behavioral therapy. JAMA 2018; 319:831–832
4.
Gloaguen V, Cottraux J, Cucherat M, et al: A meta-analysis of the effects of cognitive therapy in depressed patients. J Affect Disord 1998; 49:59–72
5.
Vittengl JR, Clark LA, Dunn TW, et al: Reducing relapse and recurrence in unipolar depression: a comparative meta-analysis of cognitive-behavioral therapy’s effects. J Consult Clin Psychol 2007; 75:475–488
6.
Lorenzo-Luaces L, German RE, DeRubeis RJ: It’s complicated: the relation between cognitive change procedures, cognitive change, and symptom change in cognitive therapy for depression. Clin Psychol Rev 2015; 41:3–15
7.
Lorenzo-Luaces L, Keefe JR, DeRubeis RJ: Cognitive-behavioral therapy: nature and relation to non-cognitive behavioral therapy. Behav Ther 2016; 47:785–803
8.
Lorenzo-Luaces L, DeRubeis RJ: Miles to go before we sleep: advancing the understanding of psychotherapy by modeling complex processes. Cognit Ther Res 2018; 42:212–217
9.
Strunk DR, DeRubeis RJ, Chiu AW, et al: Patients’ competence in and performance of cognitive therapy skills: relation to the reduction of relapse risk following treatment for depression. J Consult Clin Psychol 2007; 75:523–530
10.
Strunk DR, Hollars SN, Adler AD, et al: Assessing patients’ cognitive therapy skills: initial evaluation of the Competencies of Cognitive Therapy Scale. Cognit Ther Res 2014; 38:559–569
11.
Schmidt ID, Pfeifer BJ, Strunk DR: Putting the “cognitive” back in cognitive therapy: sustained cognitive change as a mediator of in-session insights and depressive symptom improvement. J Consult Clin Psychol 2019; 87:446–456
12.
Barber, JP: What Is Learned in Cognitive Therapy? An Initial Validation of The Ways of Responding Questionnaire and a Test of the Compensatory Model of Change. Ann Arbor, MI, University Microfilms International, 1989
13.
Jarrett RB, Vittengl JR, Clark LA, et al: Skills of Cognitive Therapy (SoCT): a new measure of patients’ comprehension and use. Psychol Assess 2011; 23:578–586
14.
Bruijniks SJE, Peeters FPML, Strunk DR, et al: Measuring patients’ acquisition of therapy skills in psychotherapy for depression: assessing the CCTS-SR and the IPSS-SR. Am J Psychother 2019; 72:67–74
15.
Forand NR, Barnett JG, Strunk DR, et al: Efficacy of guided iCBT for depression and mediation of change by cognitive skill acquisition. Behav Ther 2018; 49:295–307
16.
Halaj A, Yekutiel N, Strauss AY, et al: Utilization of learned skills in cognitive behavioural therapy for panic disorder. Behav Cogn Psychother 2019; 47:645–658
17.
Ezawa ID, Goldstein LA, Strunk DR: Improving positive life event predictions through cognitive behavioral therapy. Cognit Ther Res 2020; 44:1034–1041
18.
Hallion LS, Steinman SA, Tolin DF, et al: Psychometric properties of the Difficulties in Emotion Regulation Scale (DERS) and its short forms in adults with emotional disorders. Front Psychol 2018; 9:539
19.
Pilkonis PA, Choi SW, Reise SP, et al: Item banks for measuring emotional distress from the Patient-Reported Outcomes Measurement Information System (PROMIS): depression, anxiety, and anger. Assessment 2011; 18:263–283
20.
Victor SE, Klonsky ED: Validation of a brief version of the difficulties in emotion regulation scale (DERS-18) in five samples. J Psychopathol Behav Assess 2016; 38:582–589
21.
Gross JJ, John OP: Individual differences in two emotion regulation processes: implications for affect, relationships, and well-being. J Pers Soc Psychol 2003; 85:348–362
22.
Rosseel Y: Lavaan: An R package for structural equation modeling and more. Version 0.5–12 (BETA). J Stat Softw 2012; 48:1–36
23.
Bentler PM: Comparative fit indexes in structural models. Psychol Bull 1990; 107:238–246
24.
Tucker LR, Lewis C: A reliability coefficient for maximum likelihood factor analysis. Psychometrika 1973; 38:1–10
25.
Bentler PM: EQS Structural Equations Program Manual, vol 6. Encino, CA, Multivariate Software, 1995
26.
Rigdon EE: CFI versus RMSEA: a comparison of two fit indexes for structural equation modeling. Struct Equ Modeling 1996; 3:369–379
27.
Hu L-T, Bentler PM: Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Modeling 1999; 6:1–55
28.
Cattell RB: The scree test for the number of factors. Multivariate Behav Res 1966; 1:245–276
29.
Dziuban CD, Shirkey EC: When is a correlation matrix appropriate for factor analysis? Some decision rules. Psychol Bull 1974; 81:358
30.
Bernaards, CA, Jennrich RI: Gradient projection algorithms and software for arbitrary rotation criteria in factor analysis. Educ Psychol Meas 2005; 65:676–696
31.
Guttman L: Some necessary conditions for common-factor analysis. Psychometrika 1954; 19:149–161
32.
Kaiser HF: The application of electronic computers to factor analysis. Educ Psychol Meas 1960; 20:141–151
33.
Reise SP, Moore TM, Haviland MG: Bifactor models and rotations: exploring the extent to which multidimensional data yield univocal scale scores. J Pers Assess 2010; 92:544–559
34.
Jennrich RI, Bentler PM: Exploratory bi-factor analysis. Psychometrika 2011; 76:537–549
35.
Giordano C, Waller NG: Recovering bifactor models: a comparison of seven methods. Psychol Methods 2020; 25:143–156
36.
Morin AJ, Arens AK, Marsh HW: A bifactor exploratory structural equation modeling framework for the identification of distinct sources of construct-relevant psychometric multidimensionality. Struct Equ Modeling 2016; 23:116–139
37.
Ferrando PJ, Lorenzo-Seva U: Program FACTOR at 10: origins, development and future directions. Psicothema 2017; 29:236–240
38.
Ferrando PJ, Lorenzo-Seva U: Assessing the quality and appropriateness of factor solutions and factor score estimates in exploratory item factor analysis. Educ Psychol Meas 2018; 78:762–780
39.
Kanyongo GY: The influence of reliability on four rules for determining the number of components to retain. J Mod Appl Stat Methods 2005; 5:7
40.
Reise SP: Invited paper: the rediscovery of bifactor measurement models. Multivariate Behav Res 2012; 47:667–696
41.
Yong AG, Pearce S: A beginner’s guide to factor analysis: focusing on exploratory factor analysis. Tutor Quant Methods Psychol 2013; 9:79–94
42.
Rouquette A, Falissard B: Sample size requirements for the internal validation of psychiatric scales. Int J Methods Psychiatr Res 2011; 20:235–249
43.
Lorenzo-Luaces L, Johns E, Keefe JR: The generalizability of randomized controlled trials of self-guided Internet-based cognitive behavioral therapy for depressive symptoms: systematic review and meta-regression analysis. J Med Internet Res 2018; 20:e10113
44.
Lorenzo-Luaces L, Zimmerman M, Cuijpers P: Are studies of psychotherapies for depression more or less generalizable than studies of antidepressants? J Affect Disord 2018; 234:8–13
45.
Reise SP, Morizot J, Hays RD: The role of the bifactor model in resolving dimensionality issues in health outcomes measures. Qual Life Res 2007; 16(suppl 1):19–31
46.
Fried EI, Flake JK: Measurement matters. APS Obs 2018; 31
Information & Authors
Information
Published In
History
Received: 9 January 2021
Revision received: 29 May 2021
Revision received: 12 August 2021
Accepted: 25 August 2021
Published online: 26 October 2021
Published in print: June 15, 2022
Keywords
Authors
Funding Information
The authors report no financial relationships with commercial interests.
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBLogin options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).