In Vitro Evidence of Serotonergic Disruption in Dementia
Neurochemical and neuropathological disruptions in the serotonergic system have been established in AD. Loss of neurons in the raphe nuclei has been demonstrated,
21,22 as has loss of serotonergic nerve terminals in the neocortex.
15 Decreased concentrations of 5-HT and its major metabolite 5-hydroxyindoleacetic acid (5-HIAA) have been demonstrated in the central nervous system by use of postmortem brain studies, particularly in the temporal cortex
16,17,19,23–27 and in the cerebrospinal fluid (CSF).
28 Although indirect, taken as a whole these data provide evidence that there are changes in the actual functional status of central serotonergic transmission. The presence of expected adaptive consequences due to serotonergic lesions, such as supersensitivity, tolerance, trophic changes, or other changes in 5-HT-modulated central functions,
29 remains to be elucidated.
The actions of 5-HT are mediated through at least seven major receptor classes that have differing placements in the synapse, utilize different second messenger systems, and have different locations in the brain.
12,29 5-HT
1A receptors are located both presynaptically (somatodendritic in the raphe nuclei) and postsynaptically (terminals in the limbic system, hypothalamus, hippocampus, frontal cortex, and brainstem) and can have contradictory actions on intracellular signaling.
12,29 This receptor subtype is thought to be involved with anxiety, depression, sexual behavior, and aggression
12 (
Table 1), as well as appetite. The functions influenced by 5-HT
1Dα, 5-HT
1Dβ, 5-HT
1E, and 5-HT
1F are still being elucidated, and functions identified so far are not related to behavior.
12 Deficits involving 5-HT
1 receptors in cortex, hippocampus, and amygdala have been shown to exist in AD by ligand binding studies.
15,23,25,30–32 However, loss of 5-HT
1 receptors is also age-related in people with AD.
23 The 5-HT
1A selective agonist 8-hydroxy-2-(di-
n-propylamino) tetralin (abbreviated 8-OH-DPAT) has been used to suggest that 5-HT
1 decreases are in the 5-HT
1A receptor subtype.
31,32 Given the actions of 5-HT
1A, alterations in this subtype may be highly relevant to BPSD in AD.
There are three subtypes of 5-HT
2 receptors, all of which have some involvement in behaviors (
Table 1). 5-HT
2A is located in the cortex and caudate and is involved in anxiety (as well as vasoconstriction and migraine); 5-HT
2B is a postsynaptic receptor found in the limbic system and hypothalamus that has been associated with depression, sleep, and hallucinations; and 5-HT
2C is a postsynaptic receptor that has been associated with anxiety, depression, learning, and psychosis (as well as appetite and aversion).
12 Deficits involving 5-HT
2 receptors in AD frontal and temporal cortex, cingulate, hippocampus, and amygdala have been demonstrated with ligand binding studies,
23,30,33,34 autoradiographic studies,
35 and positron emission tomography (PET) studies.
36 Loss of 5-HT
2 receptors is much greater than loss of 5-HT
1 receptors in AD
23,34,37 and is not seen in other dementing illnesses.
11,34 One study, which found 5-HT
2 receptor binding was unchanged in the frontal cortex and hippocampus, may have been confounded by use of psychotropic medications at the time of death.
38 Given the magnitude of the deficit in 5-HT
2 receptors and the involvement of these receptors in behavior, this serotonergic system may also be important in BPSD associated with AD. Polymorphic variations have been identified for 5-HT
2A and 5-HT
2B receptors that may be risk factors for BPSD such as visual hallucinations.
39Relationships between the remaining 5-HT receptors and behavioral disorders seen in AD are not as well supported. The 5-HT
3 receptor is postsynaptic and found in the cortex, midbrain, limbic system, hippocampus, and vomiting centers.
12 It has very little homology with other 5-HT receptors and uses a difference second messenger system.
12,29 Behaviors modulated by 5-HT
3 are thought to include anxiety and psychosis,
12 and thus this receptor subtype has a potential role to play. However, loss of 5-HT
3 receptors was not demonstrated in the amygdala and hippocampus of AD patients compared with age-matched control subjects.
40 The 5-HT
4 receptor is found postsynaptically in the hippocampus, frontal cortex, limbic system, superior colliculi, and basal ganglia and has been implicated in learning, cognition, emotion, anxiety and sleep.
12 Alterations in this subtype have not yet been identified in AD. The functions and locations of 5-HT
5, 5-HT
6, and 5-HT
7 remain unclear, and their dysregulation in AD and related dementia has not been studied (
Table 1). Receptor-specific ligands are currently being developed for 5-HT
5,6,7.
29Preliminary studies show that AD may also be associated with changes in 5-HT signal transduction. Alterations in 5-HT
1A receptor coupling to guanine nucleotide binding (G) proteins have been demonstrated in the superior frontal cortex,
41 although another group found coupling to be preserved in the cerebral cortex.
42 Uncoupling of phosphoinositide (PPI) hydrolysis from 5-HT stimulation in frontal cortex tissue of AD brains compared with age-matched controls has also been demonstrated.
43 Study of second messenger systems is important because uncoupling implies that neurotransmitter replacement strategies would be ineffective.
Disruptions in serotonergic neurotransmission have also been studied in other dementing illnesses. Serotonin binding was reduced by 50% in the putamen of patients with the multi-infarct type of vascular dementia (VaD).
44 Serotonin deficits were also found in a non– multi-infarct category of VaD in cortical and subcortical gray matter.
45 However, a more recent study was unable to demonstrate any serotonin deficits.
46 Radioligand binding showed an intact brain serotonin system both presynaptically (uptake with [
3H]paroxetine) and postsynaptically (5-HT
1A with [
3H]8-OH-DPAT and 5-HT
2 with [
3H]ketanserin) in the frontal cortex, temporal cortex, and caudate nucleus (presynaptic only) in vascular dementia.
46 As expected, serotonin deficits may depend upon the subtype of VaD and the extent and location of lesions. There is also preliminary evidence for serotonergic changes in Pick's disease
47 and Lewy body dementia.
48,49In Vivo Studies Examining Linkage With BPSD
Neuroendocrine studies have been performed in AD patients that support the role of 5-HT in behavior control. In one study using the serotonin agonist
m-chlorophenylpiperazine (m-CPP), patients with AD (
n=12) showed increased behavioral responsivity (psychomotor activation, restlessness, and perceptual abnormalities) compared with 10 age-matched control subjects.
50,51 This may reflect damage to the serotonergic system with subsequent up-regulation of the remaining postsynaptic receptors.
11 Unexpectedly, the neuroendocrine response to m-CPP did not differ between the two groups. However, since m-CPP binds to 5-HT
1A, 5-HT
2, and 5-HT
3, as well as α1-, α2-, and β-adrenergic sites, dopaminergic and cholinergic sites, discrepancies between the neuroendocrine response and the behavioral response of m-CPP may be due to actions on nonserotonergic neuronal systems. Although this study indicates a possible link between 5-HT and behavior, unfortunately there was no assessment for presence of BPSD in these patients. McLoughlin et al.
52 performed a neuroendocrine study that confirmed the presence of a 5-HT hyperresponsivity in AD. Unfortunately, no attempt was made to relate this to the presence or absence of behavioral disturbances.
The relationship between decreased serotonin and specific behaviors in patients with AD and other dementing disorders has been studied (
Table 2). Postmortem studies have found decreased 5-HT levels in AD patients with a history of psychotic behaviors, compared with nonpsychotic AD patients, in some areas of the brain.
53 However, another study looking at the temporal cortex and both psychotic and nonpsychotic symptoms in patients with AD was unable to confirm these findings.
54 Chen et al.
26 found that concentrations of serotonin in the frontal cortex and 5-HIAA in the temporal cortex were significantly lower in patients on chronic neuroleptic treatment compared with patients not receiving neuroleptics. Because previous studies did not control for use of neuroleptics, it is unclear whether decreased serotonin is related to psychotic behaviors or to treatment with neuroleptics in AD. In Lewy body dementia, there is some evidence that preservation of 5-HT
2 receptors in the temporal cortex may be associated with hallucinations.
48Serotonin has also been related to nonpsychotic BPSD in both postmortem and clinical studies. A clinical study using CSF levels of the 5-HT metabolite 5-HIAA found that levels of 5-HIAA in the CSF were positively correlated with anxiety and fear/panic.
55 Decreased cortical levels of 5-HT were found post mortem in patients with a history of aggression compared with nonagitated patients,
19 whereas normal numbers of 5-HT
2 receptors were found in nonaggressive patients.
56 Clinical studies looking at putative peripheral markers of serotonergic activity have also been performed. Schneider et al.
57 evaluated the relationship between binding to the platelet serotonin transporter system (a marker of serotonergic activity) and BPSD in patients with AD. The BPSD were predominantly agitation and delusions. The agitated/delusional groups showed significantly lower peripheral serotonergic activity than AD subjects without BPSD.
57 However, a previous report by Suranyi-Cadotte et al.
58 found no differences in the density of platelet serotonin binding sites between 9 AD patients and 11 age-appropriate controls when subjects with depressive symptoms had been excluded. Although that study
58 demonstrated a link with depression, a recent postmortem study was unable to confirm this link.
27 Because depressed patients were not explicitly excluded in the Schneider et al. study
57 and the nonagitated group had apathy, it is unclear whether the association with agitation and delusions can be partially or wholly attributed to comorbid depression in some of these patients. Furthermore, platelet 5-HT
2 function may not be adequate as a peripheral marker for the central serotonergic system. For example, a negative correlation has been found between CSF 5-HIAA and receptor indexes in other neuropsychiatric patients.
59Recently, two neuroendocrine studies found that agitated aggressive patients had an increased response to the 5-HT releasing agent fenfluramine compared with nonagitated aggressive patients.
60,61 Both neuroendocrine BPSD studies excluded AD patients with significant depression. One study also found a significant gender effect.
62 Gender has not been controlled for in other studies looking at the serotonergic system and may be an important contributor to variations in the serotonergic system throughout the life span.
In summary, in vitro and in vivo studies provide inconsistent evidence to link serotonin dysfunction with psychotic symptoms and with anxiety and depression in AD. Failure to control for concomitant medication use and comorbidity are largely responsible for these inconsistencies. Current clinical evidence does support earlier work by Palmer et al.
19 and Procter et al.
56 linking aggressive agitation with the serotonergic system.
Evidence From Pharmacotherapy Studies
The neurochemical and neurophysiological evidence that the serotonergic system may contribute to BPSD has been supported by a number of studies involving pharmacologic agents specific for the serotonergic system. Clinical manipulation of the serotonergic system through pharmacotherapy provides only indirect evidence of the involvement of that system with the target behavior. However, because the goal of understanding BPSD is to improve treatment, this information is worth briefly reviewing. (More extensive reviews are available.
1,63) Considering there are 15 different serotonergic receptor subtypes, the number of serotonergic selective medications is limited. Agents reviewed here include the selective serotonin reuptake inhibitors (SSRIs), atypical antipsychotics, buspirone, ondansetron, and the serotonin and noradrenaline reuptake inhibitors (SNRIs).
The SSRIs alaproclate,
64 zimeldine,
65 fluoxetine,
66,67 citalopram,
68 fluvoxamine,
69 paroxetine,
70 and sertraline
71 have been given to patients with BPSD in randomized controlled trials. Many patients have also been treated with open-label SSRIs, with positive results. For example, in an open-label study by Pollock et al.,
72 9 of 16 patients receiving citalopram showed significant improvement in disinhibition, agitation, hostility, and suspicion. However, when SSRIs have been administered under double-blind, placebo-controlled conditions, results have been equivocal with respect to efficacy for BPSD. Randomized controlled trials in AD patients with alaproclate
64 and fluoxetine
66 have shown little efficacy for the SSRIs in treating BPSD, except for depression.
67 Placebo-controlled trials with mixed dementia populations have shown nonsignificant improvement in BPSD such as irritability, anxiety, fear/panic, mood, and restlessness with fluvoxamine,
69 and significant improvements in depression with paroxetine.
70 Fluoxetine was reported to slightly improve disinhibition, depressive symptoms, carbohydrate craving, and compulsions in about one-half of 11 patients with frontotemporal dementia.
73 The strongest efficacy was demonstrated by citalopram, which, among the SSRIs, has the greatest in vitro selectivity ratio for the serotonergic versus the noradrenergic system.
74 Nyth and Gottfries
68 administered citalopram for 4 weeks to patients with mild to moderate AD (
n=65) or multi-infarct dementia (
n=24) in a multicenter, placebo-controlled, parallel group study. In AD patients, significant improvements were noted on a geriatric rating scale in emotional bluntness and in all six BPSD (restlessness, confusion, irritability, anxiety, fear/panic, and depressed mood) for baseline versus citalopram scores. Irritability and depression improved significantly for citalopram versus placebo groups. There was no significant treatment effect in the multi-infarct group. The patients in this study had very mild BPSD at baseline, with mean scores of less than 2 out of 6 on each symptom.
The atypical antipsychotics (e.g., risperidone, clozapine, olanzapine) are being used more frequently in patients with BPSD because of their efficacy in treating psychotic symptoms and their favorable side effect profiles compared with their typical antipsychotic counterparts (i.e., reduced extrapyramidal effects). Most of the new antipsychotics have strong antagonistic affinity for the 5-HT
2 receptor. Because the hyperresponsivity of the serotonergic system noted in agitated and aggressive AD patients
60,61 suggests up-regulation of 5-HT
2 receptors, the atypicals may be of benefit in the management of BPSD. To date, risperidone, clozapine, olanzapine, and quetiapine have been assessed to treat behaviors related to dementia.
Risperidone, in a total of 534 patients from case reports,
75–81 case series,
82,83 retrospective chart reviews,
84–87 and open-label cohort studies,
88–90 was reported to produce significant reductions in behaviors such as psychosis and agitation in patients with various types of dementia. Further evidence of risperidone efficacy for BPSD was reported in two recent randomized controlled trials. Katz et al.
91 reported on 625 patients, diagnosed with AD and/or vascular dementia, who received either placebo or 0.5 mg, 1 mg, or 2 mg of risperidone for 12 weeks. The authors found a dose– response relationship with risperidone for reducing all behaviors and aggressiveness, and psychotic symptoms were significantly reduced with 1 mg, but not 2 mg, of risperidone compared with placebo. De Deyn et al.,
92 in a 12-week trial of patients with AD and/or vascular dementia, found risperidone to significantly reduce aggressiveness compared with haloperidol and placebo.
Clozapine has not been evaluated for the treatment of BPSD in any randomized controlled trials. In an open-label dechallenge and rechallenge case series by Oberholzer et al.,
93 clozapine was found to significantly improve “antisocial behavior,” “social competence,” “social interest,” and “irritability” in 7 patients with AD. In two retrospective chart reviews,
94,95 clozapine (doses of 100–300 mg per day) was reported to have minimal to moderate effects on behaviors (e.g., psychosis, agitation) in 5 patients with a variety of dementia diagnoses. Frankenburg et al.
96 reported a marked response (reduced hallucinations and verbal aggression) to low-dose clozapine (i.e., 18.5 mg/day) in a patient with parkinsonian dementia. However, 2 other patients with unspecified dementia were reported to have no changes or minimal change in behaviors with clozapine.
96 Finally, in a case report of 4 patients at varied stages of dementia, Pitner et al.
97 reported marked improvement in psychosis and other behaviors in 2 patients, and worsening of behavioral symptomatology in the remaining 2 patients.
In a large open-label trial of olanzapine, Kinon et al.
98 reported that patients diagnosed with AD and/or vascular dementia showed significant improvements in psychosis and/or agitation. A recent randomized controlled trial by Street et al.
99 reported olanzapine (5 mg and 10 mg doses) to significantly improve symptoms of psychosis and other behaviors compared with placebo in 206 AD patients over a 6-week period. The authors also reported that fewer patients without psychosis treated with olanzapine developed delusions and/or hallucinations by the end of the study compared with those receiving a placebo. Walker et al.
100 reported that 6 of 8 patients diagnosed with Lewy body dementia responded to open-label olanzapine with significant reductions in BPSD.
Quetiapine, in a single published open-label study by McManus et al.,
101 was found to significantly improve overall BPSD and psychotic symptoms in 106 patients with a variety of organic diseases over a 12-week period.
Although the atypical antipsychotics appear to improve BPSD, it is unclear whether this result is secondary to serotonergic or dopaminergic antagonism, since the typical antipsychotics (e.g., haloperidol, perphenazine, thioridazine) have also been demonstrated to have significant benefit.
102 Future research might help to determine if typical and atypical antipsychotics have different patterns of symptom improvement.
Buspirone (a 5-HT
1A partial agonist), in two randomized controlled double-blind trials, was compared with placebo
10 and haloperidol
103 to determine efficacy in treating BPSD. There was no significant overall reduction in behavioral symptoms with buspirone compared with placebo in the small number of patients tested (
N=10). However, in the other study, buspirone was as effective as haloperidol in treating 26 patients with probable AD and superior to haloperidol in decreasing anxiety and tension.
103 This latter study did not have a placebo control. Thus, the evidence for the efficacy of buspirone for BPSD is not strong.
Ondansetron (a 5-HT
3 selective antagonist), used primarily for the management of chemotherapy-induced nausea and vomiting, has been used in patients with Parkinson's disease accompanied by dementia.
104,105 Both open-label trials showed some benefit in alleviating visual hallucinations and delusions. Further evidence linking ondansetron to a reduction in behavioral problems awaits controlled clinical trials.
The SNRIs trazodone and clomipramine have also been tested for BPSD. Trazodone, in a number of open-label studies, was found to improve behavioral symptoms in 80% of demented elderly patients enrolled.
106–112 In a double-blind trial, trazodone showed a small but significant reduction in behavioral symptoms compared with buspirone and placebo.
10 In another double-blind study controlled with haloperidol, trends suggested that specific symptoms, including aggression, may respond preferentially to trazodone.
113 Clomipramine, another SNRI, produced significant overall improvement in depression in a small double-blind crossover study of AD patients.
114 Although evidence to date shows some benefit with SNRIs, these drugs have more affinity for the noradrenaline than the serotonergic receptors and cannot be considered serotonin-selective medications.
In summary, the evidence supporting the efficacy of SSRIs, atypical antipsychotics, 5-HT agonists and antagonists, and SNRIs in the management of BPSD is as yet inconclusive. However, it must be noted that, as with other trials for BPSD,
102 methodological issues with these studies included failure to blind or control,
71–73,75–90,93–98,100,101,106–111,115–130 insufficient power,
10,66,73,94–97,100,101,106–111,113 inclusion of a heterogeneous population (patients had varying etiologies for their dementias or were at varying stages of the illness or had different subtypes of BPSD),
64,68–70,77–79,81–83,85,86,88,89,91–98,113,117,129,130 and failure to use standardized and validated scales to identify and track target behaviors.
64,66,77,78,82,85,86,88,94–96,103,113–115,119,120,122–124,130 Furthermore, improper dosing and duration of treatment could contribute to some negative studies, since the effective dose and duration for treatment of different BPSD are unknown. Thus far, there is some evidence to support the use of serotonergic medications. The preliminary literature on SSRIs supports the use of the most serotonin-selective of these medications in patients with Alzheimer-type dementia, and this is corroborated by positive results with atypical antipsychotics. However, there are no predictors of response.