A Best-Worst Scaling Experiment to Prioritize Caregiver Concerns About ADHD Medication for Children

Mixed methods were used to develop and test a best-worst scaling instrument to elicit caregivers’ priority concerns when deciding whether to use ADHD medication for their child. Best-worst scaling was preferred to a conjoint discrete-choice experiment, often used in health care research (10,11), for several reasons. Grounded in random utility theory, best-worst scaling evokes tradeoffs by asking individuals to select one best and one worst attribute among competing alternatives within a profile. By comparison, conjoint experiments force selections among two or more different profiles. With best-worst scaling, individuals select attributes that are of greatest value to them relative to other shown attributes; as a result, information about what matters most to individuals is gained (12–14). This provides more enriched information on heterogeneity of specific priority concerns than can be obtained from selecting one profile containing multiple priorities (12–14). In addition, best-worst scaling makes it possible to estimate and compare the average utility of a profile’s attributes, whereas in a conjoint discrete choice experiment, the reference group is the whole scenario (14).

Two separate convenience samples were recruited from two support organizations in metropolitan Baltimore for caregivers of children with mental health needs. First, a sample of 21 caregivers participated in focus groups as part of the development of the best-worst scaling instrument. A second sample of 25 caregivers of children ages four to 14 with an ADHD diagnosis participated in a pilot study of the best-worst scaling instrument. The demographic characteristics of the two samples were very similar. A majority (>75%) was African American, and most (>85%) were the children’s biological mothers.

Attribute statements for the best-worst scaling instrument were identified by using data from a previous qualitative study of caregivers’ experiences. The study examined the experiences of caregivers as they came to terms with the ADHD diagnosis and medication treatment (3,4). This prior work generated a model of caregivers’ priorities in initiating medication that was grounded in their views of treatment in term of appropriateness (for example, whether the child was too young), anticipated effects (whether the medication would harm the child), and symbolic representation (whether using medicine meant being a bad parent) (3). This model was cross-referenced with the published literature (5,6) to develop a list of attribute statements.

In October 2012, a family support group leader from one of the family organizations recruited caregivers for the first sample. Caregivers were asked to participate in focus groups to assess attribute statement relevance. Fifteen caregivers participating in the first of two focus groups were presented with 26 attribute statements reflecting the potential priorities of caregivers when considering whether to initiate ADHD medication for their child. They were asked to classify the statements into four categories (short-term concern, long-term impact, societal views, and supportive network) or, if needed, to suggest a new category. On the basis of this feedback, attribute statements were revised and presented to a second focus group of six members of the same support organization for verification and relevance. No further amendments were suggested.

Sixteen attribute statements for the best-worst scaling instrument were retained. The statements were divided evenly by category, with each category containing two positively and two negatively phrased statements. Two child psychiatrists reviewed the clinical and practical relevance of the attribute statements.

A balanced, incomplete block design was used to construct the choice task profiles so that each attribute statement was seen the same number of times and any two attribute statements appeared together the same number of times. This design ensured equal probability of selection for each attribute statement. The survey had 16 choice task profiles, each displaying six of the 16 attribute statements. [An example of a best-worse choice task profile is available online as a data supplement to this article.]

In each choice task profile, participants were asked to think back to when they first learned of their child’s ADHD diagnosis and some of the situations that influenced their decision to initiate ADHD medication. They were instructed to select one attribute statement from among the six choices that reflected the most important concern (best choice) and then select one attribute statement that reflected the least important concern (worst choice) that influenced their decision to initiate ADHD medication for their child.

The family support group leader from a different organization helped to recruit caregivers for the pilot from five support groups for families in the Baltimore metropolitan area. During the pilot, conducted from November 2012 to January 2013, 25 caregivers used paper and pencil to complete the best-worst scaling instrument. All of the participants had children between the ages of 4 and 14 who had been diagnosed as having ADHD, and all used medication for their child. Most of the children also were currently using psychotherapy or had an individualized education plan.

The principal investigator and a graduate research assistant attended the support group meetings, explained the purpose of the survey, and provided instructions for completing the choice tasks. The pilot survey was completed, on average, in 15 minutes. At the conclusion of the meeting, participants were asked to provide feedback regarding the clarity and relevance of the choice task profiles. No further modifications were recommended.

Survey responses for each choice task profile were coded into two binary variables. The statements chosen as best and worst each received a score of 1, and the statements that were not chosen as best or worst received a score of 0. Best-worst scores for each attribute statement were calculated as the sum of the best selections minus the sum of the worst selections across all respondents divided by 150 (the number of times each attribute statement was displayed [N=6] multiplied by 25 participants) (15). A t test assessed if scores differed significantly from 0 (α=.05), which would imply that selections were not made at random but reflected stated priorities.

A positive best-worst score indicated that the attribute statement was selected as most important more frequently than it was selected as least important, and negative best-worst scores indicated the opposite. The number of times each attribute statement was chosen as best and worse is shown in Table 1, along with each attribute’s mean best-worst score and 95% confidence intervals. All attribute statements except “ADHD medicine is not needed to control my child’s home behavior” were significant (p<.05).

Ranking of mean best-worst scores from largest to smallest was used to determine relative attribute importance. Overall, using medication to help their child become a successful adult (.41) was ranked highest, and concern that others would think badly of the child if he or she used ADHD medicine (–.53) was ranked lowest. [A list of attribute statements ranked by highest to lowest best-worse score is available online as a data supplementto this article.] Table 1 lists the attribute statements in each category by order of importance. Control of school behavior (.39) was the highest-ranked short-term concern (p<.001). The role of medicine in helping the child be a successful adult was a key long-term concern (p<.001). The only positive score in the supportive-network category was having a doctor who addressed the caregivers’ concerns about ADHD medicine (p<.001). Scores for the other attributes in the category indicated that family, friends, and school personnel were less important influences compared with doctors. With the exception of the child’s peer relations, all other attributes in the societal views category were negative.

This study demonstrates the feasibility of best-worst scaling for eliciting caregivers’ priorities in initiating medication for their child’s ADHD. Significant best-worst scores indicated that choices were not random selections. Caregivers completed the instrument with relative ease.

The caregiver-centered instrument holds great promise for advancing family-centered research and clinical practice. Children’s mental health services research has been limited by a lack of rigorous methods for eliciting caregiver priorities. Eliciting caregivers’ priorities early in the clinical encounter can guide family-centered treatment planning.

There were several limitations. The sample was limited in diversity, size, and geographic locale and may not generalize to all caregivers of children with ADHD. Although recruitment from different advocacy organizations can result in potentially different samples, convenience sampling was used to recruit caregivers from homogeneous sources. The perspectives reflected the priorities of one parent—the mother. Although we sought continuous caregiver feedback, this list of relevant attribute statements may not be exhaustive. However, attribute development was an iterative feedback process in which statements were confirmed separately by several individuals. Finally, stated priorities were not correlated with treatment adherence, but that was not the goal of this feasibility study.

The purpose of this feasibility study was to test a best-worst scaling instrument prior to use in a larger comprehensive survey. The instrument is currently being used in a study that is designed to capture clinical diagnoses and receipt of mental health care services in order to assess the association between priorities and treatment adherence.

Caregivers’ priorities are nuanced and have an impact on decisions about their child’s or adolescent’s mental health care. The methods described here may help to better define, recognize, and understand caregivers’ priorities so that clinicians may engage caregivers in shared decision making about treatment for their child. This could accelerate caregiver-centered outcomes research in children’s mental health.

This work was funded by a grant from the National Institute of Mental Health (R34 MH093502) to Dr. dosReis. The authors are grateful to Osler Andres, Pharm.D., and Katie Brant, Pharm.D., for their assistance in piloting the survey and to Bev Butler, Jane Walker, and Angela Vaughn-Lee for community outreach with the families who participated in the pilot.

The authors report no financial relationships with commercial interests.