Trial of an Electronic Decision Support System to Facilitate Shared Decision Making in Community Mental Health

This study involved a cluster randomized controlled trial of a new EDSS designed to support greater client involvement in goal setting and to assist both clients and case managers in engaging in shared decision making. [A depiction of the study design is available as an online supplement at ps.psychiatryonline.org.] Case managers from three clinics were randomly assigned to the intervention group or treatment as usual. Clients were assigned to the same group to which their case manager was assigned. Because of the cluster randomization, all case managers participated multiple times, but all clients participated only one time. Because one clinic was larger than the others, one clinic recruited approximately one-half the participants and the other two recruited approximately one-quarter each. Two of the clinics provided traditional case management and community support. One clinic provided slightly more intensive services, because it previously provided assertive community treatment and continued to serve the same population. The participating agency the clinics belong to espouses a mission to help mental health clients maintain autonomy over their lives and achieve recovery-oriented goals (Wilkniss S, personal communication, 2009). Data were collected between September 2008 and January 2009.

For the intervention, a three-step EDSS process inverts the usual care planning procedures. First, clients indicate their top priorities and ideas for services at a touchscreen-enabled computer kiosk. Second, the information is electronically sent to the clients' case managers, who then complete a similar process. Finally, the two perspectives are merged electronically and presented graphically in a shared decision-making session with the dyad. [An appendix showing an example of EDSS screens is available as an online supplement at ps.psychiatryonline.org.]

The EDSS was intentionally designed so there would be minimal training burden on both clients and case managers. Clients in the study had not been exposed to the EDSS before. Case managers were given a brief manual and a one-hour didactic and practice session, in which they were able to try out the technology and ask questions. These sessions were conducted by the first author once for each site.

Clients who were assigned to receive usual care planning met with their case managers at the time that their six-month care plans were due. Case managers and clients were told to complete their care plans in the way that they normally would. Care plan discussions were audio recorded, but no research staff member was present. The usual care plan included use of a behavioral health electronic medical record designed to aid with billing and to theoretically help case managers create recovery-oriented care plans. There appeared to be significant heterogeneity in how care plans were completed, as is the case in real-world case management.

Clients who were assigned to the intervention group met with their case managers a minimum of three months before their regularly scheduled six-month care plans were due, so they could create care plans with the EDSS without worry that finishing the six-month care plan would bias the EDSS results. Clients in the intervention group eventually completed their six-month care plan in the usual way for billing purposes; however, they were able to use the intervention care plan as a guide.

Two questionnaires were developed for this study, one on the case manager satisfaction with the care planning process and the other on client satisfaction with the care planning process. The case manager version contains six statements related to the dimensions of case manager-client communication, time, flow, credibility as a clinical tool, and organization of information. The statements were rated on a 5-point Likert scale, ranging from 1, strongly disagree, to 5, strongly agree; higher summary scores indicate more satisfaction. The Cronbach's alpha for the entire scale was .74.

The case manager satisfaction measure asks case managers to rate their satisfaction with each care planning encounter, instead of their overall satisfaction with the care planning process. Pilot testing indicated that case managers strongly felt that the EDSS would work well for some clients but not others.

The client version contains seven statements rated by the client on a 5-point Likert scale, ranging from 1, strongly disagree, to 5, strongly agree; higher summary scores indicate more satisfaction. Based on the literature regarding what makes clients satisfied with a mental health encounter, the items were designed to assess case manager-client communication, involvement in decision making, being informed about decisions that were made, and having a clear management plan (31). The questionnaire also asks which types of information were communicated for clients who state that the EDSS helped them communicate important information about themselves to their counselor. The Cronbach's alpha for the entire scale was .62.

Clients were asked about their knowledge of the care plan two to four days after the care planning session. The researcher first asked the client to note which goal areas appeared in the care plan. The client was then provided with a list of goal areas not already indicated by the client and asked whether they were in the care plan. For each area correctly identified in either manner, the client was asked to relate what his or her individualized goals or objectives or shared decisions (as applicable) were within each quality-of-life area. All measures except the client knowledge of care plan were completed immediately after participation in the study.

Case managers were randomly assigned to intervention or control conditions within each of three clinics that were part of the same agency. Three clinics were necessary to achieve minimal sample size because clients could participate only if they were scheduled for a regularly occurring six-month care plan. Because case managers were randomly assigned to groups within sites instead of by site, no imbalances due to site characteristics should be present. Twenty case managers at the three clinics participated in the study with an average of four clients each. Clients were clustered by case manager, so all clients assigned to a particular case manager participated in the same condition. Clients participated only one time. Thus 40 dyads were exposed to the prototype EDSS at the time of their biannual case management planning review, and 40 dyads were in the control group and completed their usual care planning sessions. Researchers collected demographic, clinical, and workforce data before the planning sessions.

Both client and case manager questionnaires on satisfaction with the planning process were administered with a self-report computerized format. For the client questionnaire, a research assistant was present to provide assistance as needed. The knowledge of care plan questionnaire was administered by a research assistant over the telephone or in person at the mental health center if the client did not have access to a telephone. Each client was asked to recall which broad areas the care plan or shared-decision-making plan focused on. The proportion of the care plan goals that were recalled by clients was calculated by summing the number of care plan goals appearing in the care plan for the denominator, and summing the number of correctly recalled goals for the numerator. The maximum denominator was 11 goals.

The Dartmouth Committee for the Protection of Human Subjects and the agency's internal institutional review board approved the study. The study was deemed to be of minimal risk to participants. Full informed consent was obtained for research participants by the study research assistant.

Descriptive statistics were used to characterize the case manager and client samples. The differences between the intervention and the control groups' summary scores and individual items for the client and case manager questionnaires on satisfaction with the care planning process were examined via linear regression. Linear regression was also used to control for case manager age in the model that used the questionnaire on case manager satisfaction with the care planning process and to control for years working with the case manager in the model that used the questionnaire on client satisfaction with the care planning process.

Intraclass correlation coefficients (ICCs) were computed within case managers for all questionnaires and to adjust for clustering as appropriate (32,33). The ICCs indicated that case manager responses regarding individual sessions were highly correlated within individual counselors and required clustering of standard errors (ICC=.65) but that client responses did not (knowledge of care planning, ICC=.03; client satisfaction with the care planning process, ICC=.10) (34).

Because of a nonnormal distribution in the variable for client knowledge of the care plan, a nonparametric two-sample Wilcoxon rank-sum (Mann-Whitney U) test was used to assess the hypothesis that clients creating their care plans using the EDSS would have significantly greater knowledge of the goals in their care plans than clients in the usual care group.

This study has some limitations. First, case managers in the intervention group could have rated the EDSS more highly not because of its actual utility but because of its novelty (40). Second, there are many ways to define case manager and client satisfaction (35,36). The satisfaction measures were based on the literature. It is possible that the general literature on satisfaction with mental health services does not apply to satisfaction with the case management care planning process.

Research on how to maximize the potential benefits of information technology in the public sector mental health arena is in its infancy. Longitudinal studies that test systems that were designed to follow up on shared decisions over time and that test clinically meaningful outcomes are required in order to truly judge the usefulness of EDSSs in supporting shared decision making and improving care for people diagnosed as having severe mental illnesses.