Reward/Punishment Reversal Learning in Older Suicide Attempters

Between May 2006 and October 2008, we recruited 65 participants age 60 and older: 15 depressed suicide attempters, 12 depressed suicide ideators, 24 nonsuicidal depressed elderly, and 14 nondepressed, nonsuicidal elderly comparison subjects. Suicide attempters, suicide ideators, and nonsuicidal depressed elderly were diagnosed as having major depression without psychotic features by the Structured Clinical Interview for DSM-IV Axis I Disorders (SCID; 21, 22). To exclude individuals with clinical dementia and to ensure that participants could engage in computerized assessments, all were required to have a score ≥24 on the Mini-Mental State Examination (23). We excluded elderly persons with sensory disorders that precluded cognitive testing, mental retardation, delirium, neurologic disorders, bipolar disorder, schizophrenia, schizoaffective disorder, and exposure to ECT in the previous 6 months.

All participants provided written informed consent. The University of Pittsburgh Institutional Review Board approved the study.

Suicide attempters. Participants in the suicide attempters group (N=15) had made a suicide attempt; nine of them made their first suicide attempt after age 60; all were inpatients. These participants displayed a high level of suicidal intent during their attempts and expressed severe suicidal ideation; five had made repeat attempts, six had made an attempt within a month of assessment, and nine within the previous year. Suicide attempt history was verified by a psychiatrist, using the interview, medical records, information from the treatment team, and information from family members or friends. We excluded participants with significant discrepancies between these sources. Although none of the suicide attempters had experienced head injuries directly related to attempt, we assessed potential anoxic-ischemic or toxic brain injury, based on the Beck Lethality Scale (24), medical records, and the clinical interview. A study psychiatrist (A.Y.D.) assessed participants for any attempts with a score ≥4 on the Beck Lethality Scale and any history of systemic hypotension lasting >5 minutes or asphyxia or neurotoxic ingestion. One participant met these criteria, and for two participants it was impossible to rule out brain injury during past attempts; we excluded these three participants from sensitivity analyses.

Suicide ideators. Participants in the suicide ideators group (N=12) had thoughts of suicide with specific plan, serious enough to precipitate an inpatient admission (N=10) or an increase in the level of outpatient care (N=2), and no lifetime history of suicide attempt. Suicidal ideation was also very severe in this group.

Nonsuicidal depressed elderly. Nonsuicidal depressed elderly (N=24) were included in the study to detect an association between decision making and suicidal behavior above and beyond cognitive effects of depression. These participants had no current or lifetime history of suicide attempts or suicidal ideation as established by clinical interview, review of medical records, the SCID, and the Scale for Suicidal Ideation (lifetime). Two participants in this group were inpatients. Participants were excluded from this group if they had indirect self-destructive behaviors.

Nondepressed comparison subjects. Nondepressed comparison subjects (N=14) were included as the benchmark group. They had no lifetime history of any psychiatric disorder as determined by the SCID.

The study was conducted at a university psychogeriatric inpatient unit and a specialty outpatient clinic for late-life depression. Since we aimed to capture decision making in a state similar to a suicidal crisis, we assessed participants with major depression during an acute depressive episode. Participants were tested within 2 weeks of inpatient admission or at the beginning of outpatient treatment. Depressed participants continued to receive psychotropic medications as clinically indicated. We ensured that none were intoxicated or had withdrawal symptoms at the time of testing. Neuropsychological testing took place in one to two sessions over 1–3 days. Assessors were blind to clinical history and ratings.

Cognitive and clinical characterization. Current global cognitive function was assessed with the Dementia Rating Scale (25). Depression severity was measured with the 17-item Hamilton Depression Rating Scale (HAM-D) (26). Burden of physical illness was assessed with the Cumulative Illness Rating Scale adapted for geriatrics (27). We obtained medication lists from pharmacy records. We measured the intensity of pharmacotherapy for the current episode of depression with the cumulative strength score from the Antidepressant Treatment History Form (28), based on antidepressant trial duration, dose, and use of augmenting agents. To capture exposure to psychotropic medications not included in this score, we assessed exposure to sedative-hypnotics, anticholinergic drugs, and opioid analgesics. Intraclass correlation coefficients measuring interrater reliability among our assessors were 0.95 for the HAM-D, 0.97 for the Cumulative Illness Rating Scale, and 0.99 for the Dementia Rating Scale.

Probabilistic reversal learning. The probabilistic reversal learning task requires participants to learn to choose one of two colored rectangles in each of 80 trials. In an initial 40-trial acquisition stage, on 80% of the trials the participant is rewarded for selecting stimulus 1 and punished for selecting stimulus 2; on 20% of the trials, false feedback is delivered such that the participant is punished for selecting stimulus 1 (noncontingent or "probabilistic" punishment) or rewarded for selecting stimulus 2. In the second 40-trial reversal stage, the probabilities are reversed (Figure 1). Our version of the task used symbolic reward (green frame around the stimulus, green "CORRECT" display, high-frequency tone) and punishment (red frame around the stimulus, red "WRONG" display, low-frequency tone). Previous studies of probabilistic reversal learning have found similar behavioral and neural responses to symbolic reward and punishment (29), monetary reward and loss (17), and angry and happy faces (30). Participants were considered to "pass" a stage if they reached a predetermined learning criterion of eight consecutive correct trials. It is difficult to achieve this learning criterion by chance, without mastering the task, since at least one or two trials are likely to include noncontingent punishment. In addition, we used the number of trials before achieving the learning criterion in each stage as a parametric measure of performance.

To perform well on this task, one needs to trade off "staying" (i.e., choosing the previously reinforced stimulus despite occasional noncontingent punishment) and "switching" (i.e., reversing the preferred choice after punishment is received). One strategy is to integrate the reinforcement across a number of trials. The tendency to stay while ignoring punishment feedback leads to perseverative errors, which result from selecting the previously correct stimulus after the reversal of probabilities. Conversely, the tendency to switch too easily results in a large number of probabilistic switch errors when one switches away from the preferentially reinforced stimulus in response to noncontingent punishment.

Control measures: forward planning and spatial working memory. To determine whether suicidal elderly demonstrate deficits that are specific to decision making and can be dissociated from other aspects of cognition, we employed a test of forward planning and spatial working memory, the Stockings of Cambridge test, a modification of the Tower of London task (31).

We used SPSS 15.0 (SPSS Inc., Chicago) and MATLAB 7.6 (MathWorks, Natick, Mass.). All tests were two-sided. We first compared groups on demographic and clinical characteristics using F tests and chi-square tests. For these and all subsequent F tests, we examined post hoc contrasts using the Tukey honestly significant difference test. We then used the chi-square test to compare proportions of participants in each group passing acquisition and reversal stages of the probabilistic reversal learning task. In our follow-up contrasts, we compared each of the two suicidal groups to comparison subjects and to nonsuicidal depressed participants. We then examined the performance pattern in the reversal stage by group using multinomial regression with group as dependent variable and two alternative markers of poor performance—perseverative errors and probabilistic switch errors—as covariates. Our sensitivity analyses used binary logistic regression with passing/failing the reversal stage as the dependent variable. The predictors included group status (dummy-coded) and potential confounding factors: global cognitive functioning, years of education, lifetime or current substance use disorders, and gender.

To test our hypothesis that poor decision making in suicide attempters is explained by abnormally high discounting of past rewards and punishments, we employed a computational modeling procedure in which the degree to which participants incorporate the past history into their decisions ("memory") is estimated on the basis of their responses. We adapted a reinforcement learning model common in studies of reinforcement/punishment-based learning (for examples, see references 17, 32, and 33). The model (described in detail in the data supplement that accompanies the online edition of this article) used a Rescorla-Wagner learning rule to choose parameters that afforded the strongest match between choices produced by the model and participants' actual choices, based on the reinforcement history experienced by each participant. The four free parameters were memory, learning rate from rewards, learning rate from punishments, and exploration. Memory reflected the effect of the prior reinforcement history on the choice. Learning rate from rewards and learning rate from punishments reflected the impact of reward or punishment on the last trial on the participant's subsequent choice. Exploration reflected whether choice was random or determined by feedback.

The groups were similar in their demographic characteristics (Table 1). Global cognitive functioning as measured by the Dementia Rating Scale was lower in the three depressed groups relative to the nondepressed comparison group. Suicide attempters had a higher lifetime prevalence of substance use disorders compared to nonsuicidal depressed participants (χ²=1,615, df=1, p=0.001); six of the suicide attempters and none of the participants in the other groups had current substance use disorders. Psychotropic exposure was similar across the three depressed groups.

Groups did not differ in their performance in the acquisition stage, suggesting similar overall learning ability. On the other hand, significant differences in performance emerged in the reversal stage (Figure 2). After the reinforcement contingencies were reversed, suicide attempters were less likely than nondepressed comparison subjects (χ²=12.5, df=1, p<0.001) and nonsuicidal depressed participants (χ²=6.5, df=1, p=0.011) to achieve a learning criterion. Suicide ideators did not differ from nondepressed comparison subjects and nonsuicidal depressed participants. These results were mirrored on trials needed to achieve the learning criterion in the acquisition (F=1.4, df=3, 61, p=0.24) and reversal stages (F=5.3, df=3, 61, p=0.003).

Measures of excessive switching (probabilistic switch errors) and perseveration in the reversal stage differentiated the groups (Table 2). Examination of suicide attempters' performance revealed two patterns. First, most attempters made multiple probabilistic switch errors, switching away from the newly reinforced stimulus after noncontingent punishment. Second, a smaller subgroup of suicide attempters made multiple perseverative errors. Suicide ideators did not differ from other groups.

Suicide attempters displayed lower memory relative to nondepressed comparison subjects (omnibus analysis of variance, F=2.77, df=3, 61, p=0.049; Tukey honestly significant difference, post hoc: suicide attempters < comparison subjects, p=0.039; Figure 3A). That is, in their choice, attempters relied less on their previous reinforcement history and more on feedback on the last trial. As expected, participants with lower memory made more total switches in their choice (Figure 3B) and more probabilistic switches (Figure 3C). While the three depressed groups, particularly suicide ideators, tended to have a lower learning rate from punishments, group differences were not significant (F=2.52, df=3, 61, p=0.066; suicide ideators < controls, p=0.087; Figure 3D). This seemingly surprising trend was due to perseveration in the depressed groups (Table 2): participants who were less likely to switch their choice following punishment made more perseverative errors (Figure 3E). The learning rate from punishments was also positively correlated with the proportion of probabilistic switches among all switches (Figure 3F). The learning rate from rewards and exploration did not differ between groups (see the online data supplement).

On the Stockings of Cambridge test, groups did not differ in the number of problems solved in minimum moves, the number of moves used to solve the problems (adjusting for level of difficulty), or deliberation times (see the online data supplement).

The groups still differed in achieving the learning criterion in the reversal stage (Wald statistic=5.8, df=1, p=0.015) after controlling for global cognitive functioning as measured by the Dementia Rating Scale, years of education, and the effects of lifetime history of substance use disorders. After all participants with lifetime substance use disorders were excluded, suicide attempters were still the least likely to pass the reversal stage (χ²=11.6, p=0.009, N=52; suicide attempters versus nonsuicidal depressed: χ²=5.6, p=0.018, N=29). The same was true after excluding participants with current substance use disorders and after limiting the attempter group to the nine participants with a first attempt after age 60 (see the online data supplement). Similarly, the difference between suicide attempters and nonsuicidal depressed participants remained after excluding the one suicide attempter with potential anoxic brain injury from the suicide attempt and the two for whom brain injury could not be ruled out (χ²=10.2, p=0.001). Gender was not related to performance in the reversal stage.

Our results extend earlier findings of impaired decision making in younger suicide attempters with affective disorders (11, 34) to a group of depressed elders with a history of suicide attempt. Suicide attempters in our study showed unstable decision making, which has been described in midlife depression (15, 16) but was evident to a more extreme degree in our study participants. Furthermore, decreased reliance on past history was dissociated from abnormal sensitivity to rewards or punishments. Thus, in counterpoint to the prevailing view that suicidal individuals' representations of reality are distorted in the valence domain (negative cognitive biases; see reference 35, for example), our findings indicate distortions in the time domain. This notion is supported by early empirical findings of altered time perception (36–39) and by self-report evidence that future orientation is weakened in older suicide ideators and attempters (40). A similar process could have contributed to the decaying performance of suicide attempters on the Iowa Gambling Task (11); that is, they may have failed to integrate reinforcement history from a card deck over a number of trials. Assuming that these laboratory observations are representative of individuals' lives, they suggest that people vulnerable to suicidal behavior, faced with change and uncertainty, fail to access their past experiences, making decisions based largely on their present state.

Human lesion (13, 41–43) and imaging (17, 29) studies implicate the ventral prefrontal cortex in reward/punishment-based learning. Thus, our findings converge with evidence of ventral prefrontal pathology from postmortem studies (reviewed in reference 44) and imaging studies of attempted suicide in younger persons (45–47). Furthermore, the suicide attempters' intact performance on the Stockings of Cambridge test could indicate that forward planning and working memory associated with the dorsolateral prefrontal cortex (48) are relatively preserved.

Detailed clinical characterization, the inclusion of suicide ideators, and the use of a control task add confidence in our findings. Many cognitive studies of suicide are difficult to interpret, since they often combine individuals with major depression, bipolar disorder (11), and schizophrenia (9). Unipolar depression is the most common antecedent of late-life suicide (49–51), and thus our study focused on elders with major depression. It is unclear whether our findings can be generalized to other psychiatric disorders, although in midlife the association between impaired decision making and attempted suicide appears to transcend diagnostic boundaries (34). Furthermore, we cannot assume that impaired reward/punishment-based probabilistic learning is unique to attempted suicide. While in our small sample it was not related to substance use, similar impairments have been described in pathological states characterized by impulsivity, such as pathological gambling (52) and behavioral toxicity of dopamine agonists (53, 54). Small group sizes and case-control design comprise the main limitations of this study. The lack of effect of depression on forward planning and working memory raises questions about the sensitivity of the Stockings of Cambridge test as a control test—a concern partly mitigated by the presence of robust test difficulty effects. Furthermore, suicide attempters paid less attention to their reinforcement history than did comparison subjects but did not differ significantly from depressed elders. This observation cannot rule out an alternative hypothesis that distortions in the time domain are associated with depression and not specifically with suicidal behavior. Finally, our reinforcement learning model is limited in the extent to which it captures human decision making. More complex formalisms, such as hidden Markov models (17) or models based on production rule learning (55), may be needed to characterize decision making in suicidal behavior.