Site maintenance Wednesday, November 13th, 2024. Please note that access to some content and account information will be unavailable on this date.
Skip to main content

Sources of Bias in Guideline Development

To the Editor: Treatment guidelines have become increasingly important for institutional policies, quality management, and jurisdiction in mental health care. We were engaged in the development of the treatment guideline on aggressive behavior for the German Association for Psychiatry and Psychotherapy from 2007 to 2008 ( 1 ). According to the standard most frequently used in guideline development, a clearly defined algorithm was utilized for the step from evidence to recommendations. This method was adopted from the U.S. Agency for Healthcare Research and Quality (AHRQ). It comprises different levels of evidence, from the highest (meta-analysis of at least three randomized controlled studies) to the lowest (expert opinions).
Using this method, for example, we found good evidence from several meta-analyses for use of antipsychotics and benzodiazepines as emergency medications. Consequently, we decided to recommend these drugs with the highest level of evidence. However, representatives of service users did not agree. Not surprisingly, they did not discuss issues such as p values and effect sizes but rather expressed their general concerns about the use of coercion and involuntary medication. Perhaps this was attributable to their unfamiliarity with methods of evidence-based medicine, but perhaps they were simply right in some way. This led us to reconsider some aspects of methodology in the development of guidelines. We identified at least five sources of bias in the AHRQ method and thus in many existing guidelines.
First, levels of evidence are related to the quality of studies, not to reported effect sizes. Thus a small amount of evidence of efficacy can lead to a strong recommendation. Second, external validity of randomized controlled trials is rather limited. This is particularly the case for issues such as violence and coercion: patients who give informed consent for randomized controlled studies often differ considerably from real-world patients. A third source of bias is that the absence of evidence for older treatment options leads to treatment recommendations for newer, well-examined, and frequently more expensive options without evidence of superiority.
Fourth, the ethical framework of many clinically relevant objectives cannot be represented sufficiently in randomized controlled trials. In particular, issues such as involuntary treatment and use of coercive measures have outcomes not only on the patient level but also on the level of staff, patients' relatives, and society as a whole, which should be taken into account. Finally, existing evidence is biased by a predominance of pharmacotherapy, which may decrease acceptance among service users.
The most recent guidelines, such as the update on schizophrenia by the National Institute for Health and Clinical Excellence (NICE) ( 2 ) and the German guideline on unipolar depression ( 3 ), weaken the strong link between evidence levels and recommendations. In the German guideline, four levels of evidence still correlate with four levels of recommendations. However, during guideline development the recommendations could be modified (upgraded or downgraded) after the developers took into account ethical obligations, clinical relevance of the effectiveness measures used, applicability of results to certain patient groups, preferences of patients, and the likelihood of implementation in routine clinical practice. The NICE update utilized a different approach. As a result of meta-analyses conducted by the guideline development group, a short "evidence summary" is given, which is the basis for making or not making a clinical recommendation.
Such modifications of links between evidence (more correctly, study quality) and recommendations can avoid much of the bias described above and highlight the role of consensus. The price, however, is a loss of transparency in regard to how recommendations are derived from evidence. Upgrading the role of consensus implies that the composition of the group and the type of applied consensus techniques have a high and rather unknown impact. This pertains also to use of the GRADE grid in developing guidelines, which has recently been suggested (5). In the GRADE grid, evidence is classified into high, moderate, low, and very low, and recommendations are classified as strong, weak, or none. Without a strict algorithm from evidence to recommendations, the latter can be upgraded by aspects such as high effect sizes and a strong dose-effect relationship and can be downgraded by study limitations, inconsistent results, and other bias. In addition to being affected by the level of evidence, the level of recommendation can be influenced by values and preferences, cost, and the relationship of desired and undesired effects.
In the development of future clinical guidelines, much attention should be paid to these methodological issues, with the aim of acquiring a maximum amount of transparency.

Acknowledgments and disclosures

The authors report no competing interests.

Footnote

Dr. Steinert is affiliated with the Centre of Psychiatry Weissenau and Dr. Bergk is with the Center for Psychiatry Suedwuerttemberg, Ulm University, Ravensburg, Germany. Dr. Richter is with the Department of Applied Sciences, Bern University, Bern, Switzerland.

References

1.
German Society for Psychiatry, Psychotherapy, and Neurology: Therapeutic Measures in Aggressive Behavior in Psychiatry and Psychotherapy [in German]. Darmstadt, Germany, Steinkopff, 2009
2.
Schizophrenia: The NICE Guideline on Core Interventions in the Treatment and Management of Schizophrenia in Adults in Primary and Secondary Care. London, National Institute for Health and Clinical Excellence, 2009. Available at www.nice.org.uk/nicemedia/pdf/cg82fullguideline.pdf
3.
Clinical Practice Guideline for Unipolar Depression [in German]. Berlin, German Society for Psychiatry, Psychotherapy, and Neurology, 2009. Available at www.depression.versorgungsleitlinien.de/
4.
Jaeschke R, Guyatt GH, Dellinger P, et al: Use of GRADE grid to reach decisions on clinical practice guidelines when consensus is elusive. British Medical Journal 337:a744, 2008

Information & Authors

Information

Published In

Go to Psychiatric Services
Go to Psychiatric Services
Psychiatric Services
Pages: 946 - 947
PubMed: 20810599

History

Published online: 1 September 2010
Published in print: September, 2010

Authors

Details

Tilman Steinert, Prof. Dr.med.
Dirk Richter, Prof. Dr.rer.soc.

Metrics & Citations

Metrics

Citations

Export Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

For more information or tips please see 'Downloading to a citation manager' in the Help menu.

Format
Citation style
Style
Copy to clipboard

View Options

View options

PDF/EPUB

View PDF/EPUB

Get Access

Login options

Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.

Personal login Institutional Login Open Athens login
Purchase Options

Purchase this article to access the full text.

PPV Articles - Psychiatric Services

PPV Articles - Psychiatric Services

Not a subscriber?

Subscribe Now / Learn More

PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.

Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).

Media

Figures

Other

Tables

Share

Share

Share article link

Share