Electronic Health Records in Mental Health Research: A Framework for Developing Valid Research Methods

Accuracy of information is critical to biomedical research, and all data sources have limitations. Paper medical records, which often serve as a standard to which other data sources are compared, do not necessarily capture what transpires in patient-physician encounters (12). An internal medicine study had physicians blindly evaluate actor-patients. Then both the physician and actor documented what transpired during the exam. The paper medical records agreed with the actors’ reports on 70% of the quality-of-care items measured; but for the actor-patients’ diagnoses, the extent of agreement was only 48% (12). EHRs have similar potential for inaccuracy. Like paper records and other sources of secondary data, EHRs are not designed primarily for research; instead they are designed to facilitate the delivery of clinical care and to support administrative (for example, billing) functions (13). These motivations strongly influence what is recorded and the thoroughness with which it is documented in EHRs.

Three key features of EHRs distinguish them from other sources of secondary data and point to the importance of using research methods that help increase data accuracy. First, EHRs often have complex designs that can affect data accuracy (13). EHRs commonly include structured elements, such as checklists or templates, for the mental status examination, physical examination, and other aspects of routine assessment. These elements are intended to aid clinical assessments and ensure documentation of elements required for billing. Clinicians enter data by selecting from a menu of predetermined options that may, in some cases, improve data completeness (14). In other cases, however, structured data may lead to inaccuracies. For example, an EHR may include templates with certain fields prepopulated as “normal” or may automatically insert information from prior notes or from other areas of the EHR to speed documentation. Structured templates require vigilance to edit incorrect information that would otherwise automatically be entered into the EHR. In analyzing structured data, it is important to consider that providers’ lapses in vigilance could lead to inaccurately entered or omitted data.

The second feature is unstructured or narrative textual elements, which may introduce opportunities for inaccuracies. An advantage of unstructured EHR data over the text in paper records is that large amounts of electronic text across clinical sites and patient populations can be searched and analyzed in an automated way, using natural language processing or similar methods (11). Narrative text, however, may also introduce opportunities for data coding errors. For example, a pediatric study of 465 EHRs for patients presenting for otitis media demonstrated 278 subtle variations on the documentation of a fever (for example, fever, T>101°F, and temperature>102°F) (13). In evaluating variable construction, it is important to consider all pertinent phrases that could refer to the variable of interest. Without an accurate data extraction rule—that is, a search strategy for collecting pertinent data in an EHR—inadvertently missed or misinterpreted pertinent data can lead to substantial inaccuracies.

The context in which a given EHR system is used may influence the description of clinical phenomena. For example, if providers use EHRs while interviewing and examining patients, then this distraction can lead to inaccurate data entry. In addition, EHR systems are routinely used by various personnel in a number of clinical settings, each with local customs influencing providers’ personal preferences in how they interface with the EHR (6). Inconsistent documentation practices across clinical settings and provider groups can lead to poor data comparability and inaccuracies (6). For example, depending on local preferences, diabetes associated with antipsychotic medications may be described with a host of terms (for example, diabetes, DM2, metabolic syndrome, and insulin resistance) and documented in numerous locations in an EHR—in the unstructured text of a progress note, in notes specifically dedicated to adverse medication effects, or in a section devoted to medical history.

This calls attention to a third unique feature of EHRs that affects secondary data use for clinical research. EHRs are optimized to present clinicians with a great deal of information about an individual patient. For data to be analyzed across many patients, however, data must typically be extracted from the EHR and transformed into a format that can be indexed, searched, and analyzed for research (13). Through this process, the fidelity of data can diminish, such as when specific, locally defined terms are cross-mapped to less specific categories. As a result of the transformation, pertinent data for variables of interest (such as a specific diagnosis or medication or a clinical event or outcome) may become lost or fragmented. A detailed working knowledge of the EHR’s content and the context of its use will help investigators identify and compile all of the pertinent information for their research.

In summary, EHRs have unique features that make them attractive sources for research data, combining the breadth of variables available in paper records with the automated search functions of administrative billing data or national registries. Alongside these advantages, however, EHRs have the potential to introduce inaccurate information. The full extent and degree of inaccuracies introduced into the research literature in this way are not currently quantified. This Open Forum addresses this issue by outlining a methodological framework that will allow researchers to reduce inaccuracies and enable peer reviewers to independently quantify the extent and degree of inaccuracies in future research that uses EHRs as a data source.

Some variables of interest are easily and accurately accessed from EHRs. Frequently, however, variables must be elicited by culling information from multiple areas in the EHR. Biomedical informatics researchers have described a variety of techniques for identifying patient cohorts with specific variables of interests (15). Data extraction rules locate variables of interest within the larger landscape of EHR information. This process is analogous to the case definition process that is a foundation of research using administrative data. Methods to validate these case definition algorithms are an important step in research using administrative data but are not yet standard in EHR research (16). An evaluation of 126 unique EHR-based studies in health outcomes research found that only 24% included validation methodologies (17).

The following recommendations are intended as a guide for mental health services researchers who are considering use of EHR data in their research. We describe the development of data extraction rules in detail and ways to validate and publish those methods, adapting lessons learned from research on administrative data sources and informatics in other fields (16,18). We use clozapine-associated agranulocytosis to illustrate the key steps.

EHRs offer key opportunities for mental health services research. By raising awareness of methodological issues and the importance of data accuracy in research using EHRs as a secondary data source, we highlight the need to conduct this work with the same rigor as that used in research on other data sources. We urge funders such as the Patient-Centered Outcomes Research Institute and the Agency for Healthcare Research and Quality to support work that promotes accuracy in EHR data and research methods. High-priority topics include development and evaluation of optimal research methods for identifying mental health EHR variables of interest that can be generalized across EHR platforms. This is especially, but not exclusively, important for variables such as depression and completed and attempted suicide, which were designated by the U.S. Food and Drug Administration’s Mini-Sentinel project as moderate and high research priorities, respectively, for future validation studies because of inconsistently or poorly performing algorithms to identify these variables in administrative claims data (18). Other high-priority topics are drafting expert consensus guidelines on types of EHR variables and the range of methods that can be used to validate them and drafting expert consensus standards on the reporting of validation methods in mental health EHR research.

Dr. Pincus has received travel funds from Value Options and funds for presentations from Johnson & Johnson. Dr. Stroup has participated in a CME program funded by Genentech and has received research funding from Auspex Pharmaceuticals.

The other authors report no financial relationships with commercial interests.