Advancing Recovery Science: Reliability and Validity Properties of the Recovery Assessment Scale
Abstract
Objective
The promotion of recovery is the driving philosophy underlying national, state, and local mental health systems. Although numerous recovery-oriented measures have been developed in response, the scientific assessment of recovery measures has lagged behind. The purpose of this literature review was to review the psychometric properties of the Recovery Assessment Scale (RAS), which is arguably the most commonly used measure of recovery in the published literature. Such information is critical for advancing recovery science.
Methods
A thorough literature search using the search term “Recovery Assessment Scale” was conducted in August 2012, yielding a total of 222 articles published from around the world. A total of 77 articles that included psychometric data on the RAS were used in this review.
Results
Means and standard deviations across studies were fairly consistent. Overall, the studies indicate very good results for internal consistency, test-retest reliability, and interrater reliability. A number of studies also reported consistent factor structures for the measure. The RAS was found to have positive associations with other related constructs and negative associations with constructs such as symptoms. Finally, the RAS appears to be sensitive to change over time.
Conclusions
The review found significant evidence to support the use of the RAS in recovery science as a means to measure recovery and to include it in mental health research.
Recovery is the driving philosophy underlying the development of national, state, and local mental health policy and services and promises to remain a guiding influence for decades to come. This recovery era requires a new generation of measures to be added to the current mental health measurement toolbox. Current recovery-related measures that have been discussed in various studies and compendia (1–5) fall into three broad categories. First, measures of the subjective experience of recovery are those that focus on the various ways in which people experience recovery and the recovery process. These measures typically assess the consumer-oriented perspective on recovery rather than taking a clinical perspective that focuses on reduced symptoms, increased functioning, and so on as indicators of recovery (6). A consumer-oriented perspective includes the extent to which people are living a satisfying, fulfilling, and hopeful life and developing meaning and purpose regardless of the presence of mental health issues.
Second are recovery-oriented attitudes, beliefs, and knowledge measures. These measures examine attitudes about the extent to which respondents (consumers, providers, and others) are favorable toward the idea that achieving recovery is possible. These measures also assess knowledge and beliefs about recovery and recovery-oriented practices. Third are measures of respondents’ perceptions of recovery-promoting environments, which include their opinions about the extent to which recovery-oriented policies, programs, and practices are in place.
Recovery-oriented research is still at a relatively early stage, especially compared with research examining other constructs, such as symptoms or cognitive functioning. And just like what happened in the early days of clinical outcome studies, thorough development and examination of evidence will be important to ensure that these measures meet rigorous measurement standards. This scrutiny will help researchers select credible recovery instruments for use in future studies, assist policy makers and providers who are also using various types of recovery measures to evaluate the effectiveness of policies and programs associated with recovery-oriented transformations, and heighten confidence in the results of these studies in order to advance recovery science.
This study examined the psychometric properties of what is plausibly the most commonly used recovery measure in the published literature, the Recovery Assessment Scale (RAS). The RAS was developed by mental health consumers in the mid-1990s, relatively early in the recovery era, through an analysis of recovery stories that resulted in the identification of 39 themes of the subjective experience of recovery. These themes were used to create the original 41-item measure (7). Using data from more than 1,800 respondents, a factor analysis of the original 41-item version identified five factors: personal confidence and hope, willingness to ask for help, goal and success orientation, reliance on others, and not being dominated by symptoms (8).
Several narrative reviews of recovery measures have been highly favorable toward the RAS. In a study by Law and colleagues (9) the RAS and five other measures met three criteria out of 25 recovery-oriented measures they examined: scrutinized in a peer-reviewed journal; offers a self-report questionnaire that yields quantitative results, thus enhancing its clinical utility; and aims to measure factors relating to what the researchers called “personal recovery.” The RAS was described as the most acceptable and valid measure available, although authors stated that no gold-standard measure of recovery had yet been developed. Burgess and colleagues (3,10) identified the RAS as the most suitable of four scales for measuring recovery concepts in Australia because it meets all nine criteria that they established. Cavelti and colleagues (4) concluded their review by describing the RAS as possibly the top measure of recovery available. And the RAS is listed in an assessment guide as the only highly recommended “subjective appraisal” recovery scale based on its adequate test-retest reliability, construct validity, treatment sensitivity, and clinical utility, with good internal consistency and content validity and excellent validity generalization (11). Finally, another review concluded that the RAS can facilitate dialogue between consumers and clinicians and is relevant to consumers’ lives. The review also noted that consumers expressed willingness to complete the assessment again and that clinicians thought that the instrument was better than other recovery scales at capturing illness management (5).
This study is the first to offer an in-depth review of the quantitative properties of the RAS to complement the aforementioned narrative reviews. The specific questions we set out to answer are as follows: Is there substantial evidence to conclude that the RAS has relatively consistent means and standard deviations and reliability across populations? What is the nature of the evidence pertaining to construct validity of the RAS? Is the RAS sensitive to change in intervention research?
Methods
A search was conducted in August 2012 on Google Scholar and PubMed with the search term “Recovery Assessment Scale.” This search resulted in the identification of 222 documents. From that list, we excluded articles for the following reasons: duplicates or translations (N=26); articles could not be translated from other languages by the authors (N=5); there is no discussion of the RAS or the article refers to a neurological recovery measure (N=66); the article does not present results or was not published in peer-reviewed journals, dissertations, book chapters, books, or compendia (N=19); the article reviews the concept and measurement of recovery but presents no data (N=16); and the article mentions the RAS, but the study used another measure of recovery for data collection (N=13). The remaining 77 articles that report use of the RAS in data collection and present results are included in this review.
These 77 studies came from 11 countries: United States, Australia, United Kingdom, Japan, Canada, Portugal, Spain, Israel, Sweden, Switzerland, and Taiwan, with most studies being from the United States and Australia. The sample sizes varied greatly across the studies, from a minimum of ten to a maximum of 1,827. Most studies used the RAS-41 or RAS-24 versions, but some used variations, such as the RAS-20, RAS-22, RAS-42, and RAS-50. Most studies used a 1–5 Likert scale, where 1 is strongly disagree and 5 is strongly agree. Some studies calculated scores as a sum across all items, whereas others presented means.
Because studies used different versions of the RAS, different response scales for the items, and different ways of computing composite scores, we needed to convert the results to allow for comparisons across studies. Summed results were converted to average scores by dividing the means and standard deviations of the reported summed scores by the number of items on the version of the RAS that was used. For instance, a RAS-24 sum score of 84 would be equivalent to an average score of 84/24=3.5. Also, if studies used a 0–4 Likert scale versus a 1–5 scale, we added 1 to the mean score; standard deviations remained the same, as shifting by a constant does not affect variance. In addition, there were several studies that did not report mean scores for the entire sample but reported them separately for different groups (experimental group and control group, for example). In those instances, we calculated weighted overall means, using sample size of groups as weights, and pooled standard deviations for the entire sample.
Results
Means
Twenty-eight of the 77 studies reported RAS means and standard deviations, the RAS version, and the response scale used for the items. The means in the 28 studies ranged between 3.14 and 4.12, according to the relatively standard scoring system of 1 (low) to 5 (high), and the interquartile range (IQR) was 3.72–3.90. The average of all reported mean scores was 3.78±.19. Figure 1 shows the distribution of the means and standard deviations.
Figure 1 Distribution of Recovery Assessment Scale scores in 28 studiesa
a Possible scores were standardized to a scale from 1 to 5, with higher scores indicating better recovery.
Reliability
Nineteen of the 77 studies reported on the internal consistency of the RAS (7,12–29). Cronbach’s alphas ranged between .76 and .97. Four studies reported test-retest reliability ranging from .65 to .88 (7,13,21,30). Two studies examined the inter-interviewer reliability of the RAS and found intraclass correlation coefficients to be .94 and .98 (31,32).
Factor structure
Corrigan and colleagues (8) were the first to subject the RAS to a factor analysis. Using data from 1,824 individuals with serious mental illnesses, they found a five-factor solution that incorporated 24 of the 41 items on the scale. The five factors were personal confidence and hope, willingness to ask for help, goal and success orientation, reliance on others, and no domination by symptoms. The five other studies that used American, Australian, and Japanese samples to conduct exploratory factor analysis or principal-components analysis (13,16,21,27,33) identified the same or highly similar factors or principal components. In addition, multiple other studies used these factors as subscales without rechecking the factor structure. A dozen studies (8,16,24,28,29,33–39) reported acceptable or good internal consistency (Cronbach’s α >.7) for most of these subscales, although some of the subscales in these and other studies had alphas near the lower limit of acceptability (α=.58–.70) (16,24,28,29,34,39,40).
Validity
A total of 49 studies examined point-in-time associations between RAS and other constructs; 47 of the 49 reported simple bivariate associations (Pearson or Spearman correlations, t tests, or one-way analyses of variance), and 14 used the RAS in a regression analysis (12 as a dependent variable and two as a predictor). Table 1 summarizes these studies.
| Study | RAS basis of scoring | Version | Likert scalea | Scorea | N | Sample characteristics | RAS associationsb |
|---|---|---|---|---|---|---|---|
| Corrigan et al., 1999 (7) | Total | RAS-41 | 1–5 | NS | 35 | 35% female; mean±SD age=33.1±9.2; 57% black, 37% white, and 6% other, including Asian American; all with serious mental illness | Psychosocial well-being: +; symptoms, distress, poor health: –; social functioning and support: + |
| Corrigan et al., 2003 (61) | Total | RAS-41 | 1–5 | Sum | 1,824 | 60% female; mean±SD age=41.8±10.4; 24% black, 75% white, 1% Asian, 18% Latino/Hispanic; 3% Native American; all with schizophrenia, bipolar disorder, or major depression and significant functional limitation | Religiosity and religious support: + |
| Ritsher et al., 2003 (15) | Total | RAS-41 | NS | NS | 127 | 6% female; mean±SD age=49.5±8.7; 62% white, 26% black, 9% Hispanic, 1% Asian/Pacific Islander, 5% Native American, 3% other race-ethnicity; 35% with schizophrenia, 21% paranoid psychosis, 27% affective psychosis, 82% depression, 39% PTSD, 34% anxiety disorder, 39% personality disorder, 76% substance use disorder, 68% alcohol abuse; all veterans | Stigma: – |
| Corrigan and Phelan, 2004 (53) | Subscales | RAS-41 | 1–5 | Sum | 176 | 62% female; mean±SD age=41.3±10.5; 84% white, 14% black, 5% Hispanic, 16% Native American, 1% Asian, 11% other race-ethnicity; all with serious mental illness and significant functional limitation | Social functioning and support: + |
| Flinn, 2004 (16)c | Total and subscales | RAS-24 | 0–4 | Sum | 355 | 38% female; 18% ages 18–39, 32% ages 40–49, 29% ages 50–59; 21% ages ≥60; 58% white, 40% black, 3% Hispanic; 61% with schizophrenia, 18% mood disorders, 3% anxiety disorders, 10% multiple diagnoses, 7% multiple diagnoses and substance abuse | Participation: +; other (stages of change): + |
| Andresen et al., 2006 (45) | Total | RAS-41 | NS | NS | 94 | 52% female; mean±SD age=44.1±12.6; Australia residents; 48% schizophrenia diagnosis, 43% self-reported schizophrenia, 2% schizoaffective disorder, 1% bipolar disorder, 5% other psychotic disorders | Other recovery measures: + |
| Deane and Andresen, 2006 (68) | Total | RAS-40 | 1–5 | Sum | 27 | 52% female; mean±SD age=46.2±10.4; Australia residents; 52% with schizophrenia, 22% with bipolar disorder, 22% with other diagnoses | Other (being partnered with community volunteers versus treatment as usual): no association |
| Lloyd et al., 2007 (58) | Total and subscales | RAS-24 | 1–5 | Sum | 44 | 32% female; mean age=33; Australia residents; all individuals with serious mental illness | Participation: + |
| McNaught et al., 2007 (33) | Subscales | RAS-24 | 1–5 | NS | 168 | 37% female; mean±SD age=39.0±12.1; all with psychotic disorder; Australia residents | Other recovery measures: +; symptoms, distress, poor health: – |
| Salyers et al., 2007 (30) | Total | NS | NS | Mean | 59 | 34% female; mean±SD age=43.5±10; 41% white, 51% black, 2% Native American, 7% Hispanic/Latino; all with psychiatric disabilities | Other recovery measures: +; symptoms, distress, poor health: – |
| Walby, 2007 (42)c | Subscales | RAS-41 | NS | Mean | 350 | 69% female; mean±SD age=41.1±11.3; 50% with serious mental illness, 50% with mild to borderline severe mental illness being treated in an outpatient setting | Psychosocial well-being: +; stigma: –; social functioning and support: + |
| Potokar, 2008 (18)c,d | Total | RAS-41 | 1–5 | Mean | 65 | 17% female; mean±SD age=51±9.8; 86% white; 52% with schizophrenia, 34% schizoaffective disorder, 14% bipolar disorder | Psychosocial well-being: +; other (loss due to mental illness): – |
| Clarke et al., 2009 (65) | Subscales | RAS-41 | 0–4 | Sum | 71 | 56% female; mean±SD age=40.7±11.3; Australia residents; 69% with schizophrenia, 14% major depressive disorder with psychotic features, 10% schizoaffective disorder, 7% bipolar disorder | Other (goal attainment): + |
| Hedden, 2009 (19)c,d | Total | RAS-41 | 1–5 | Mean | 81 | 51% female, 4% unknown; mean±SD age=44±13.1; 6% with schizophrenia, 10% schizoaffective disorder, 16% bipolar disorder, 40% major depression, 28% with multiple diagnoses | Religiosity and religious support: – |
| Hendryx et al., 2009 (59)d | Total | NS | NS | Sum | 153 | 49% female; mean±SD age=48.8±14.8; 94% white; 40% with schizophrenia | Participation: + |
| Pernice-Duca and Onaga, 2009 (13) | Total | RAS-41 | 1–5 | Mean | 221 | 53% female; mean±SD age=43.5±9.9; 10% black, 82% white, 5% multiracial, 3% Latino, Native American, Arab American, or other race-ethnicity; 53% with schizophrenia and related disorders, 33% major affective disorders, 14% other axis 1 diagnoses | Social functioning and support: + |
| Townley et al., 2009 (14) | Total | RAS-20 | 1–5 | NS | 40 | 55% female; mean age=46; 53% white, 45% black, 2% biracial; >50% with schizophrenia, all others with major depression or bipolar disorder | Participation: +; community inclusion: + |
| Andresen et al., 2010 (46) | Total and subscales | RAS-41 | 0–4 | Sum | 281 | 42% female; mean±SD age=39.71±11.84; Australia residents; 72% with schizophrenia, 8% schizoaffective disorder, 11% bipolar disorder, 9% depressive psychosis | Other recovery measures: + |
| Buckley-Walker et al., 2010 (47) | Total | RAS-24 | 0–4 | NS | 40 | 40% female; mean±SD age=40.9±10.5; Australia residents; all with serious mental illness | Other recovery measures: + |
| Chiba et al., 2010 (21) | Total | RAS-24 | NS | Sum | 209 | 41% female; mean±SD age=48.3±15.7; Japan residents; 61% with schizophrenia, 13% bipolar disorder, 10% depression, 16% other or unknown diagnoses; 45% living in community, 55% inpatients | Psychosocial well-being: +; symptoms, distress, poor health: – |
| Chiba et al., 2010 (48) | Total | RAS-24 | 1–5 | Sum | 223 | 41% female; mean±SD age=47.6±15.5; Japan residents; 60% with schizophrenia, 14% bipolar disorder, 11% depression, 16% other or unknown | Other recovery measures: + |
| Fuller, 2010 (66)d | Total and subscales | RAS-41 | 1–5 | NS | 100 | 50% female; mean±SD age=40.6±12.4; 93% white, 6% black, 1% Asian; 35% with schizophrenia, 37% with bipolar disorder, 32% with major depression, 29% with anxiety disorder, 20% with substance dependence | Other (disorder: substance use versus serious mental illness or co–occurring disorders): +; other (peer support): no association |
| Green et al., 2010 (63) | Total and subscales | NS | NS | Sum | 170 | 52% female; mean±SD age=49.2±14.5; 94% white, 6% black, 3% Native American/Alaska Native, 2% Asian/Pacific Islander, 5% mixed heritage, 1% Hispanic; 43% with schizophrenia spectrum disorder, 47% bipolar disorder, 11% affective psychosis | Other (behavior activation): + |
| Leung, 2010 (57)c | Total | NS | 1–5 | Sum | 50 | 50% female, 2% no gender information; ages ≥18; 90% white, 2% Hispanic, 2% Native American, 6% non-Hispanic or other; 42% with schizophrenia, 58% bipolar disorder or affective psychosis | Positive outlook: +; symptoms, distress, poor health: – |
| Lloyd et al., 2010 (41) | Total | RAS-24 | 1–5 | Sum | 161 | 49% female; mean±SD age=41.0±12.8; Australia residents; 34% with schizophrenia, 30% depression, 24% bipolar disorder, 8% anxiety disorder, 3% schizoaffective disorder, 1% personality disorder | Psychosocial well-being: +; community inclusion: +; other (unmet needs): –; other (paid employment): +; other (diagnosis: bipolar disorder versus schizophrenia or depression): + |
| Pernice-Duca, 2010 (54)d | Total | RAS-41 | 1–5 | Mean | 169 | Slightly fewer men than women; ages 30–45; mostly white; 41% with schizophrenia or a related disorder, 58% major affective disorder | Social functioning and support: + |
| Rogers et al., 2010 (44) | Total | RAS-41 | NS | NS | 1,827 | 60% female; mean±SD age=43.0±10.2; 57% white, 17% black, 26% Hispanic, other race-ethnicity, or multiracial; 50% with schizophrenia, schizoaffective disorder, or other psychotic disorders, 22% major depression or affective disorder, 18% bipolar disorder, 10% other disorders | Psychosocial well-being: + |
| Wolstencroft et al., 2010 (50) | Total | RAS-23 | NS | NS | 18 | 67% female; mean±SD age=46.50±11.07; Australia residents; 17% bipolar disorder, 11% schizophrenia, 72% depression, 17% PTSD, 6% obsessive-compulsive disorder, 6% anxiety, 11% schizoaffective disorder, 6% dissociative disorder | Other recovery measures: + |
| Chiba et al., 2011 (60) | Total | RAS-24 | 1–5 | Sum | 120 | 36% female; mean±SD age=41.2±12.2; Japan residents; 53% with schizophrenia, 18% bipolar disorder, 12% depression, 18% other or unknown disorder | Positive outlook: + |
| Connell et al., 2011 (12) | Total | RAS-41 | 0–4 | Sum | 234 | 35% female, 21% no gender information; mean±SD age=39.7±11.8; Australia residents; 65% schizophrenia, 11% bipolar disorder, 9% schizoaffective disorder, 9% depressive psychosis, 4% both schizophrenia and depression, 3% schizophrenia and comorbid conditions | Other (employment): no association |
| Conrad-Garrisi, 2011 (34)c | Total | RAS-41 | 1–5 | Sum for total; means for subscales | 143 | 46% female; mean age=47.1; 76% white, 17% black, 4% Arabic, 1% Latino, 1% Asian, 1% Native American; all clubhouse members with schizophrenia, mood disorders, anxiety disorder, and other mental illnesses | Psychosocial well-being: +; symptoms, distress, poor health: –; community inclusion: +; other (stages of change): +; other (financial deprivation): – |
| Copic et al., 2011 (36) | Subscales | RAS-24 | 0–4 | Mean | 77 | 41% female; mean age=43; Australia residents; 61% with schizophrenia, 18% schizoaffective disorder, 11% bipolar disorder, 8% depressive psychosis, 3% no available diagnosis information | Other recovery measures: +; psychosocial well-being: +; positive outlook: +; other (treatment involvement): + |
| de Pina Gonçalves de Sousa, 2011 (39)c | Subscales | RAS-24 | 1–5 | Mean | 17 | 12% female; mean±SD age=38.1±8.7; Portugal residents; all with schizophrenia | Stigma: – |
| Depp et al., 2011 (62)d | Total | RAS-41 | 1–5 | Sum | 73 | 47% female; mean±SD age=50.3±6.3; 45% white, non-Hispanic, 26% black non-Hispanic, 15% Hispanic, 8% Asian or Native American; all with schizophrenia | Other (behavior activation): + |
| Färdig et al., 2011 (51) | Total | RAS-41 | 1–5 | Sum | 107 | 38% female; mean±SD age=43±13.7; Sweden residents, 65% born in Sweden, 35% immigrants; 71% with schizophrenia, 29% with schizoaffective disorder | Other recovery measures: + |
| Liu, 2011 (55)e | Total and subscales | NS | NS | NS | 310 | All mental health consumers who were members of rehabilitation centers or half-way houses; Taiwan residents | Social functioning and support: + |
| Mukolo, 2011 (37)f | Total | RAS-41 | 1–5 | Sum for total; means for subscales | 110 | 61% female; mean±SD age=44.1±12.2; 72% white, 16% black, 12% other races; 16% with schizophrenia, 15% bipolar disorder, 9% comorbid bipolar and schizophrenia, 14% depression, 43% other or unspecified, 5% undisclosed diagnoses | Psychosocial well-being: + |
| Muñoz et al., 2011 (23) | Total | RAS-41 | 1–5 | NS | 108 | 26% female in low–self-stigma group, 50% female in high–self-stigma group; mean±SD age=36.8±8.5; Spain residents; 74% with schizophrenia or other psychotic disorders, 11% personality disorder, 4% schizophrenia and personality disorder, 5% bipolar disorder, 3% obsessive-compulsive disorder, 2% organic mental disorder | Stigma: –; social functioning and support: + |
| Roe et al., 2011 (40)d | Total | RAS-20 (analysis of 12 items) | NS | NS | 159 | 33% female; mean±SD age=43.2±10.7; Israel residents fluent in Hebrew; all with schizophrenia or schizoaffective disorder | Symptoms, distress, poor health: – |
| Siqueira, 2011 (64)c | Subscales | RAS-24 | 0–4 | NS | 41 | 63% female; mean±SD age=42.3±12.8; Australia residents; all with a chronic mental illness for past ≥12 months | Other (psychological acceptance): + |
| Webb et al., 2011 (25)d | Total | RAS-41 | 1–5 | Sum | 81 | 53% female, 3% gender not reported; mean±SD age=43±12; 84% white, 1% black, 12% Hispanic (12%), 1% Asian; 15% with schizoaffective disorder or schizophrenia, 41% depression, 19% bipolar disorder, 22% multiple diagnoses, 4% did not provide diagnosis information | Social functioning and support: +; religiosity and religious support: + |
| Weeks et al., 2011 (49) | Total | RAS-50 | 1–5 | Sum | 50 | 34% female; mean±SD age=32.4±12.1; U.K. residents; 58% white, 30% black, 6% Asian, 6% other race-ethnicity; 42% with schizophrenia, 22% other psychotic disorders, 12% mental or substance use disorder, 10% bipolar disorder, 6% depression, 4% obsessional disorder, 2% schizoaffective disorder, 2% emotionally unstable personality disorder | Other recovery measures: + |
| Bottonari et al., 2012 (67)g | Total | RAS-42 | 1–5 | Sum | 192 | 8% female; mean±SD age=52±10; 72% white, 29% black, 1% Asian, 3% American Indian, 10% Hispanic/Latino; 64% with mood disorder, 51% PTSD or other anxiety disorder, 28% psychotic disorder, 64% substance use disorder; all veterans | Other (peer support): no association |
| Brusilovskiy and Salzer, 2012 (69)e | Total | RAS-20 | 1–5 | Sum | 378 | 59% female; mean±SD age=48.4±9.8; 63% black, 37% white; 61% with schizophrenia spectrum disorder, 39% major depression | Other (community crime): +; other (community socioeconomic status): + |
| Clarke et al., 2012 (52) | Total and subscales | RAS-24 | 0–4 | NS | 144 | 48% female; mean±SD age=39.3±11.7; Australia residents; 69% with schizophrenia, 13% bipolar disorder, 12% schizoaffective disorder, 6% major depressive disorder with psychotic features | Other recovery measures: + |
| Cook et al., 2012 (28) | Total | RAS-41 | NS | Sum | 428 | 56% female; mean±SD age=42.8±10.9; 54% white, 34% black, 4% Hispanic/Latino, <1% Asian/Pacific Islander, 6% American Indian/Alaska Native, 2% other race-ethnicity; 15% with schizophrenia, 5% schizoaffective disorder, 40% bipolar disorder, 18% depression, 9% other | Psychosocial well-being: + |
| Hicks et al., 2012 (38) | Total | RAS-24 | 0–4 | Mean | 61 | 38% female; mean±SD age=45.6±10.9; Australia residents; 80% with schizophrenia or schizoaffective disorder, 20% bipolar disorder or major depression | Psychosocial well-being: +; other (provider relationship): + |
| Kaplan et al., 2012 (43)d | Total | RAS-20 | 1–5 | Sum | 1,827 | 60% female; 233 ages 17–30, 1,594 age ≥31; 54% white only, 16% black only, 29% all other race-ethnicities; 47% with schizophrenia, 40% bipolar disorder or depression, 14% all other mental illnesses | Psychosocial well-being: + |
| Norman et al., 2013 (56)d | Subscales | RAS-24 | NS | NS | 84 | 31% female; mean±SD age=28.0±7.4; Canada residents; 63% with schizophrenia, 13% schizoaffective disorder, 6% schizophreniform disorder, 6% substance-induced disorder, 8% psychosis NOS, 2% delusional disorder, 1% affective psychosis | Symptoms, distress, poor health: –; social functioning and support: + |
a
NS, not stated
b
Plus sign indicates positive association; minus sign indicates negative association.
c
Dissertation
d
Dependent variable and associations
e
Dependent variable only
f
Independent variable only
g
Independent variable and associations
Characteristics examined
Psychological well-being (12 studies).
Other recovery measures (11 studies).
Social functioning and support (nine studies).
Psychiatric symptoms, distress, and poor health (eight studies).
Stigma (four studies).
Community participation (four studies).
Perceived community inclusion (three studies).
Positive coping (three studies).
Religiosity and religious support (three studies).
Findings were mixed when it came to religiosity and religious support. In one study, RAS scores were found to be higher for those who considered themselves to be spiritual and religious (61), and in another study there was a positive association between RAS and perceived support from the congregation and God (25). However, a third study showed that lower RAS scores corresponded to greater religious support and religiosity (19).
RAS and other constructs (15 studies).
Studies have shown that the RAS total score or RAS subscale scores are positively associated with the relationship between consumers and their mental health providers (38); active involvement in treatment (36); acknowledgment of the need for change, as assessed by the Stages of Change Questionnaire (16,34); behavior activation (62,63); use of psychological acceptance, defined as “actively contacting psychological experiences while behaving effectively” (64); and progress toward goal attainment (65). On the other hand, loss attributed to mental illness (defined as a reduction in tangible or intangible resources in which a person has a significant emotional investment) (18), financial deprivation (34), and unmet needs (41) were negatively associated with the RAS.
Inconsistent results were found between RAS scores and employment status: one study showed that consumers engaged in paid employment had higher RAS scores than those receiving Social Security benefits only (41); however, another article reported no significant difference in total RAS scores between workers and nonworkers (12). Two studies found a relationship between the RAS and diagnosis: one found that persons with substance dependence had higher RAS scores than those with a serious mental illness or co-occurring mental and substance use disorders (66), and another showed that persons with bipolar disorder had higher RAS scores than those with schizophrenia or depression (41). Furthermore, individuals receiving peer support (66,67) or partnered with community volunteers (68) were not shown to have higher RAS scores than those receiving treatment as usual. Finally, one study (69) found a marginally significant positive association between RAS scores and neighborhood socioeconomic status.
RAS and change over time
Six randomized controlled studies looked at changes in the RAS over time to examine the effectiveness of an intervention that aimed to improve ratings of subjective recovery. In three of the six studies, individuals in the experimental group compared with those in the control group had a greater improvement in RAS total or subscale scores. These interventions were Building Recovery of Individual Dreams and Goals through Education and Support (BRIDGES) (28), Wellness Recovery Action Planning (29), and the Recovery Workbook (70). There were no significant differences in RAS scores between the experimental and control groups in the other three studies, which examined the effectiveness of Illness Management and Recovery (IMR) (71), Internet peer support (72), and participation in treatment that combines traditional services with a religious and spiritual assessment (73).
RAS total and subscale scores were used as outcome variables in 12 pretest-posttest longitudinal studies. Eight of the 12 studies found an increase in RAS scores over time that was associated with achieving goals (65); participating in a peer-led BRIDGES education intervention (74); participating in Compeer, a program that matches people with a serious mental illness to community volunteers in order to increase social support (75); participating in the adapted version of group positive psychotherapy (“positive living”) for people with schizophrenia (26); participating in Australian Salvation Army Alcohol Rehabilitation programs for people with co-occurring disorders or with a substance use disorder only (24); participating in assertive community treatment (76,77); and participating in IMR (77,78). On the other hand, participating in a program that uses psychoeducation through the combination of sociodrama and e-learning (39), being in a peer education and support program for veterans (20), and completing homework as part of therapy (17,22) were not associated with changes in RAS scores over time.
Other quantitative analyses
Not discussed here are studies that used the RAS for other types of analyses, including structural equation modeling and path analysis (23,34,65,79–81), Rasch analysis (32,82), and an attempt to find a cutoff score to indicate achievement of recovery (83). These studies were not relevant to the key issues we reviewed.
Discussion
An impressive number of quantitative studies using the RAS have been published around the world since its emergence in the late 1990s, leading us to the following answers to our research questions. First, there is substantial evidence to conclude that the RAS has solid and consistent psychometric properties, as indicated in findings pertaining to internal consistency, test-retest and interrater reliability, and the mean±SD results presented in Figure 1.
Second, there is solid evidence supporting the construct validity of the RAS. RAS scores were positively associated with other recovery measures, psychological well-being, positive illness outlook, higher levels of social participation and support, sense of belonging and community participation, and a whole range of other constructs, including positive relationships with mental health providers and active involvement in treatment. RAS scores were negatively correlated with perceived and self-stigma, presence of psychiatric symptoms, distress, and poor general medical and emotional health, as well as other constructs such as financial deprivation, the presence of unmet needs, and loss attributed to mental illness.
Third, the RAS appears to be sensitive to change in intervention research. Three out of six RCTs showed changes on the RAS, and eight of 12 pretest-posttest intervention studies also showed statistically significant changes over time. Of course, such results beg the question of why changes were not found in all studies. This certainly could reflect a lack of sensitivity of the measure but also could be the result of the ineffectiveness of the intervention under study in producing changes on the measure. The sensitivity of the RAS could also be affected by plausible ceiling effects that may be evident in our examination of the cross-study means reported in Figure 1. Over the 28 studies that reported mean scores, the subjective experience of recovery appeared fairly high on average, although there was certainly room for change. For use as an outcome measure, it might be advisable for intervention studies to enroll individuals whose score falls below a certain baseline threshold on the RAS to increase RAS sensitivity in detecting statistically significant changes over time. The studies that did not find changes on the RAS may have enrolled people who scored relatively high in their recovery at baseline.
Significant effort was made to include all studies that reported RAS quantitative results. The fact that we eliminated nearly two-thirds of all articles identified with our search term suggests that we cast a broad net in our search. Nonetheless, some relevant articles may have been missed. We discovered that many versions of the RAS exist and are being used in research. Although we do not believe that this fact affected our conclusions, more work is needed to establish a standard version of the measure. Good psychometric properties were demonstrated with the RAS-20 and RAS-24 versions of the RAS; thus these shorter versions of the scale may be the most parsimonious and may best uphold confidence in the scale’s psychometric properties.
Conclusions
The RAS is likely the most commonly used recovery measure in research. The thorough review of the data reported here, along with the favorable comments about the measure made by others in narrative reviews, afford confidence that the RAS can be justifiably used in future research, evaluations, and clinical assessments to advance recovery science. Such a conclusion is important as mental health research increasingly seeks to examine the impact of interventions on the subjective experience of recovery. Researchers and policy makers have greater assurance that recovery can be reliably and validly measured, which will further increase research in this critically important area. On the basis of our findings, we recommend that researchers who use this measure in the future be more consistent in reporting psychometric information; the greater use of standardized approaches to the numbers of items used, scaling, and scoring will further ensure that the RAS is a sound measure.
The results of this review also support the use of the RAS in clinical settings, especially to assess one’s current state of recovery and improvement over time. Although we were unable to find reports about the use of the RAS in assessment, service planning, and individual-level evaluation in the peer-reviewed literature, such research could be useful to providers and consumers in making decisions about recovery-oriented outcomes that they would like to target. For example, the RAS subscales could be used to determine whether targeted efforts are needed to address personal confidence and hope, support people in seeking help from others, and enhance coping strategies to reduce the effects of symptoms on their quality of life. Overall, the RAS appears to be a valuable tool for enhancing recovery science and practice.
Acknowledgments and disclosures
This review was supported by grant H133B100037 to Temple University (Dr. Salzer, principal investigator) from the U.S. Department of Education, National Institute on Disability and Rehabilitation Research (NIDRR). The contents of this article do not necessarily represent the policy of the U.S. Department of Education or reflect the endorsement of the federal government. NIDRR had no direct involvement in any aspect of conducting, reporting, or disseminating this research.
The authors report no competing interests.
References
1.
Campbell-Orde T, Chamberlin J, Carpenter J, et al.: Measuring the Promise of Recovery: A Compendium of Recovery Measures. Cambridge, Mass, Human Services Research Institute, Evaluation Center, 2005
2.
Ralph RO, Kidder K, Phillips D: Can We Measure Recovery? A Compendium of Recovery-Related Instruments. Cambridge, Mass, Human Services Research Institute, Evaluation Center, 2000
3.
Burgess P, Pirkis J, Coombs T, et al.: Assessing the value of existing recovery measures for routine use in Australian mental health services. Australian and New Zealand Journal of Psychiatry 45:267–280, 2011
4.
Cavelti M, Kvrgic S, Beck EM, et al.: Assessing recovery from schizophrenia as an individual process: a review of self-report instruments. European Psychiatry 27:19–32, 2012
5.
Gregg S, Scholte A, White A: Recovery Outcome Measures Project. Master of Science (OT) Program Symposium. Hamilton, Ontario, Canada, McMaster University, 2006
6.
Bellack AS: Scientific and consumer models of recovery in schizophrenia: concordance, contrasts, and implications. Schizophrenia Bulletin 32:432–442, 2006
7.
Corrigan PW, Giffort D, Rashid F, et al.: Recovery as a psychological construct. Community Mental Health Journal 35:231–239, 1999
8.
Corrigan PW, Salzer MS, Ralph RO, et al.: Examining the factor structure of the Recovery Assessment Scale. Schizophrenia Bulletin 30:1035–1041, 2004
9.
Law H, Morrison A, Byrne R, et al.: Recovery from psychosis: a user informed review of self-report instruments for measuring recovery. Journal of Mental Health 21:192–207, 2012
10.
Burgess P, Pirkis J, Coombs T, et al: Australian Mental Health Outcomes and Classification Network “Sharing Information to Improve Outcomes.” Parramatta, New South Wales, Australia, National Mental Health Strategy, 2010. Available at www.recoverydevon.co.uk/download/Review_of_Recovery_Measures.pdf
11.
Mueser KT, Glynn SM: Schizophrenia; in A Guide to Assessments That Work. Edited by, Mash JHEJ. New York, Oxford University Press, 2008
12.
Connell M, King R, Crowe T: Can employment positively affect the recovery of people with psychiatric disabilities? Psychiatric Rehabilitation Journal 35:59–63, 2011
13.
Pernice-Duca F, Onaga E: Examining the contribution of social network support to the recovery process among clubhouse members. American Journal of Psychiatric Rehabilitation 12:1–30, 2009
14.
Townley G, Kloos B, Wright PA: Understanding the experience of place: expanding methods to conceptualize and measure community integration of persons with serious mental illness. Health and Place 15:520–531, 2009
15.
Ritsher JB, Otilingam PG, Grajales M: Internalized stigma of mental illness: psychometric properties of a new measure. Psychiatry Research 121:31–49, 2003
16.
Flinn SR: Reliability and validity of the Recovery Assessment Scale for consumers with severe mental illness living in group home settings. Doctoral dissertation. Kent, Ohio, Kent State University, Graduate School of Education, 2004
17.
Kelly PJ: Does homework improve outcomes for individuals diagnosed with severe mental illness? Doctoral dissertation. Wollongong, New South Wales, Australia, University of Wollongong, School of Psychology, 2007
18.
Potokar DN: Living with serious mental illness: the role of personal loss in recovery and quality of life. Doctoral dissertation. Bowling Green, Ohio, Bowling Green State University, Department of Psychology, 2008
19.
Hedden K: Public and private religiosity, religious social support, and recovery from severe mental illness. Doctoral dissertation. Seattle, Wash, Seattle Pacific University, Department of Clinical Psychology, 2009
20.
Resnick SG, Rosenheck RA: Integrating peer-provided services: a quasi-experimental study of recovery orientation, confidence, and empowerment. Psychiatric Services 59:1307–1314, 2008
21.
Chiba R, Miyamoto Y, Kawakami N: Reliability and validity of the Japanese version of the Recovery Assessment Scale (RAS) for people with chronic mental illness: scale development. International Journal of Nursing Studies 47:314–322, 2010
22.
Kelly PJ, Deane FP: Relationship between therapeutic homework and clinical outcomes for individuals with severe mental illness. Australian and New Zealand Journal of Psychiatry 43:968–975, 2009
23.
Muñoz M, Sanz M, Pérez-Santos E, et al.: Proposal of a socio-cognitive-behavioral structural equation model of internalized stigma in people with severe and persistent mental illness. Psychiatry Research 186:402–408, 2011
24.
Cridland E, Deane F, Hsu C-I, et al.: A comparison of treatment outcomes for individuals with substance use disorder alone and individuals with probable dual diagnosis. International Journal of Mental Health and Addiction 10:670–683, 2012
25.
Webb M, Charbonneau AM, McCann RA, et al.: Struggling and enduring with God, religious support, and recovery from severe mental illness. Journal of Clinical Psychology 67:1161–1176, 2011
26.
Meyer PS, Johnson DP, Parks A, et al.: Positive living: a pilot study of group positive psychotherapy for people with schizophrenia. Journal of Positive Psychology 7:239–248, 2012
27.
Fukui S, Shimizu Y, Rapp CA: A cross-cultural study of recovery for people with psychiatric disabilities between US and Japan. Community Mental Health Journal 48:804–812, 2012
28.
Cook JA, Steigman P, Pickett S, et al.: Randomized controlled trial of peer-led recovery education using Building Recovery of Individual Dreams and Goals through Education and Support (BRIDGES). Schizophrenia Research 136:36–42, 2012
29.
Cook JA, Copeland ME, Floyd CB, et al.: A randomized controlled trial of effects of Wellness Recovery Action Planning on depression, anxiety, and recovery. Psychiatric Services 63:541–547, 2012
30.
Salyers MP, Godfrey JL, Mueser KT, et al.: Measuring illness management outcomes: a psychometric study of clinician and consumer rating scales for illness self management and recovery. Community Mental Health Journal 43:459–480, 2007
31.
Barbic S: The Effectiveness of the Recovery Workbook as a Psychoeducation Intervention for Facilitating Recovery in Persons With Serious Mental Illness. Kingston, Ontario, Canada, Queen’s University Press, 2007
32.
Hancock N, Bundy A, Tamsett S, et al.: Participation of mental health consumers in research: training addressed and reliability assessed. Australian Occupational Therapy Journal 59:218–224, 2012
33.
McNaught M, Caputi P, Oades LG, et al.: Testing the validity of the Recovery Assessment Scale using an Australian sample. Australian and New Zealand Journal of Psychiatry 41:450–457, 2007
34.
Conrad-Garrisi DL: Inter/intrapersonal variables and readiness for change on achieving recovery. Doctoral dissertation. Detroit, Mich, Wayne State University, Department of Educational Psychology, 2011
35.
Pickett S, Phillips H, Kraus D: Recovery International Group Meeting Evaluation Final Report. Chicago, University of Illinois at Chicago, Department of Psychiatry, Center on Mental Health Services, 2011
36.
Copic V, Deane FP, Crowe TP, et al.: Hope, meaning and responsibility across stages of recovery for individuals living with an enduring mental illness. Australian Journal of Rehabilitation Counselling 17:61–73, 2011
37.
Mukolo A, Heflinger C, Baxter J: Recovery and self-esteem: concurrent validity of the Recovery Assessment Scale. International Journal of Psychosocial Rehabilitation 15:41–68, 2011
38.
Hicks AL, Deane FP, Crowe TP: Change in working alliance and recovery in severe mental illness: an exploratory study. Journal of Mental Health 21:127–134, 2012
39.
de Pina Goncalves de Sousa S: Self-stigma on severe mental illness: development of an intervention program using the sociodrama and e-learning. Doctoral dissertation. Porto, Portugal, University of Porto, Department of Psychology and Educational Sciences, 2011
40.
Roe D, Mashiach-Eizenberg M, Lysaker PH: The relation between objective and subjective domains of recovery among persons with schizophrenia-related disorders. Schizophrenia Research 131:133–138, 2011
41.
Lloyd C, King R, Moore L: Subjective and objective indicators of recovery in severe mental illness: a cross-sectional study. International Journal of Social Psychiatry 56:220–229, 2010
42.
Walby GW: Associations between individual, social, and service factors, recovery expectations and recovery strategies for individuals with mental illness. Doctoral dissertation. Tampa, University of South Florida, Department of Community and Family Health, 2007
43.
Kaplan K, Salzer MS, Brusilovskiy E: Community participation as a predictor of recovery-oriented outcomes among emerging and mature adults with mental illnesses. Psychiatric Rehabilitation Journal 35:219–229, 2012
44.
Rogers ES, Ralph RO, Salzer MS: Validating the Empowerment Scale with a multisite sample of consumers of mental health services. Psychiatric Services 61:933–936, 2010
45.
Andresen R, Caputi P, Oades L: Stages of Recovery Instrument: development of a measure of recovery from serious mental illness. Australian and New Zealand Journal of Psychiatry 40:972–980, 2006
46.
Andresen R, Caputi P, Oades LG: Do clinical outcome measures assess consumer-defined recovery? Psychiatry Research 177:309–317, 2010
47.
Buckley-Walker K, Crowe T, Caputi P: Exploring identity within the recovery process of people with serious mental illnesses. Psychiatric Rehabilitation Journal 33:219–227, 2010
48.
Chiba R, Kawakami N, Miyamoto Y, et al.: Reliability and validity of the Japanese version of the Self-Identified Stage of Recovery for people with long term mental illness. International Journal of Mental Health Nursing 19:195–202, 2010
49.
Weeks G, Slade M, Hayward M: A UK validation of the Stages of Recovery Instrument. International Journal of Social Psychiatry 57:446–454, 2011
50.
Wolstencroft K, Oades L, Caputi P, et al.: Development of a structured interview schedule to assess stage of psychological recovery from enduring mental illness. International Journal of Psychiatry in Clinical Practice 14:182–189, 2010
51.
Färdig R, Lewander T, Fredriksson A, et al.: Evaluation of the Illness Management and Recovery Scale in schizophrenia and schizoaffective disorder. Schizophrenia Research 132:157–164, 2011
52.
Clarke S, Oades LG, Crowe TP: Recovery in mental health: a movement towards well-being and meaning in contrast to an avoidance of symptoms. Psychiatric Rehabilitation Journal 35:297–304, 2012
53.
Corrigan PW, Phelan SM: Social support and recovery in people with serious mental illnesses. Community Mental Health Journal 40:513–523, 2004
54.
Pernice-Duca F: Family network support and mental health recovery. Journal of Marital and Family Therapy 36:13–27, 2010
55.
Liu S: A correlation study of social functioning and recovery on community mental health patients: examples of community mental health centers and halfway houses. National Tawiwan University Social Work Journal 23:137–180, 2011
56.
Norman RMG, Windell D, Lynch J, et al.: Correlates of subjective recovery in an early intervention program for psychoses. Early Intervention in Psychiatry 7:278–284, 2013
57.
Leung S: Explanatory style and recovery from serious mental illness. Doctoral dissertation. La Mirada, Calif, Biola University, School of Psychology, 2010
58.
Lloyd C, King R, McCarthy M, et al.: The association between leisure motivation and recovery: a pilot study. Australian Occupational Therapy Journal 54:33–41, 2007
59.
Hendryx M, Green CA, Perrin NA: Social support, activities, and recovery from serious mental illness: STARS study findings. Journal of Behavioral Health Services and Research 36:320–329, 2009
60.
Chiba R, Kawakami N, Miyamoto Y: Quantitative relationship between recovery and benefit-finding among persons with chronic mental illness in Japan. Nursing and Health Sciences 13:126–132, 2011
61.
Corrigan P, McCorkle B, Schell B, et al.: Religion and spirituality in the lives of people with serious mental illness. Community Mental Health Journal 39:487–499, 2003
62.
Depp CA, Cardenas V, Harris S, et al.: Psychopathological and functional correlates of behavioral activation and avoidance in schizophrenia. Journal of Nervous and Mental Disease 199:861–865, 2011
63.
Green CA, Perrin NA, Polen MR, et al.: Development of the Patient Activation Measure for mental health. Administration and Policy in Mental Health and Mental Health Services Research 37:327–333, 2010
64.
Siqueira VR: Experiential avoidance and psychological acceptance in the psychological recovery from enduring mental illnesses. Doctoral dissertation. Wollongong, New South Wales, Australia, University of Wollongong, School of Psychology, 2011
65.
Clarke SP, Oades LG, Crowe TP, et al.: The role of symptom distress and goal attainment in promoting aspects of psychological recovery for consumers with enduring mental illness. Journal of Mental Health 18:389–397, 2009
66.
Fuller LS: The impact of peer support programs on the process of recovery from mental illness: a comparative analysis. Doctoral dissertation. Minneapolis, Minn, Capella University, School of Psychology, 2010
67.
Bottonari KSM, Resnick S, Mueller L, et al.: Peer-led vs clinician-led recovery-oriented groups: what predicts attendance by veterans? International Journal of Psychosocial Rehabilitation 16:88–105, 2012
68.
Deane FP, Andresen R: Evolution and sustainability of the helping hands volunteer program: consumer recovery and mental health comparisons six years on. Australian Journal of Rehabilitation Counselling 12:88–103, 2006
69.
Brusilovskiy E, Salzer MS: A study of environmental influences on the well-being of individuals with psychiatric disabilities in Philadelphia, PA. Social Science and Medicine 74:1591–1601, 2012
70.
Barbic S, Krupa T, Armstrong I: A randomized controlled trial of the effectiveness of a modified recovery workbook program: preliminary findings. Psychiatric Services 60:491–497, 2009
71.
Färdig R, Lewander T, Melin L, et al.: A randomized controlled trial of the illness management and recovery program for persons with schizophrenia. Psychiatric Services 62:606–612, 2011
72.
Kaplan K, Salzer MS, Solomon P, et al.: Internet peer support for individuals with psychiatric disabilities: a randomized controlled trial. Social Science and Medicine 72:54–62, 2011
73.
Huguelet P, Mohr S, Betrisey C, et al.: A randomized trial of spiritual assessment of outpatients with schizophrenia: patients’ and clinicians’ experience. Psychiatric Services 62:79–86, 2011
74.
Pickett SA, Diehl S, Steigman PJ, et al.: Early outcomes and lessons learned from a study of the Building Recovery of Individual Dreams and Goals through Education and Support (BRIDGES) program in Tennessee. Psychiatric Rehabilitation Journal 34:96–103, 2010
75.
McCorkle BH, Rogers ES, Dunn EC, et al.: Increasing social support for individuals with serious mental illness: evaluating the Compeer model of intentional friendship. Community Mental Health Journal 44:359–366, 2008
76.
Huguelet P, Koellner V, Boulguy S, et al.: Effects of an assertive community program in patients with severe mental disorders and impact on their families. Psychiatry and Clinical Neurosciences 66:328–336, 2012
77.
Salyers MP, Hicks LJ, McGuire AB, et al.: A pilot to enhance the recovery orientation of assertive community treatment through peer-provided illness management and recovery. American Journal of Psychiatric Rehabilitation 12:191–204, 2009
78.
Mueser KT, Meyer PS, Penn DL, et al.: The Illness Management and Recovery program: rationale, development, and preliminary findings. Schizophrenia Bulletin 32(suppl 1):S32–S43, 2006
79.
Pernice-Duca FM: The relationship between social network supports and recovery from mental illness. Doctoral dissertation. East Lansing, Michigan State University, Department of Family and Child Ecology, 2005
80.
Muñoz M, Sanz M, Perez-Santos E: The state of knowledge about internalized stigma on the basis of the socio-cognitive-behavioral model. Annuary of Clinical and Health Psychology 7:41–50, 2011
81.
Green CA, Polen MR, Janoff SL, et al.: Understanding how clinician-patient relationships and relational continuity of care affect recovery from serious mental illness: STARS study results. Psychiatric Rehabilitation Journal 32:9–22, 2008
82.
Hancock N, Bundy A, Honey A, et al.: Improving measurement properties of the Recovery Assessment Scale with Rasch analysis. American Journal of Occupational Therapy 65:E77–E85, 2011
83.
Chadwick SL: Investigating recovery in psychosis: a personal construct repertory grid study. Doctoral dissertation. Hertfordshire, United Kingdom, University of Hertfordshire, Department of Psychology, 2011
Information & Authors
Information
Published In
Cover: Marooned, by Howard Pyle, 1909. Oil on canvas. Delaware Art Museum, Museum Purchase, 1912.
History
Published in print: April 2014
Published online: 15 October 2014
Authors
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBLogin options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).