The Importance and Limitations of Null Findings

It is refreshing to read a brain imaging study, published in a prestigious psychiatry journal, that finds no evidence for key neurobiological hypotheses in the field. The paper by Kliemann and colleagues (1) on resting-state functional MRI (rs-fMRI) in autism spectrum disorder (ASD) demonstrates several features recommended as part of efforts to address the “replication crisis,” including protocol preregistration, larger samples, open data, and Bayesian statistics (2). Efforts by researchers to incorporate these features are valuable, but they also require support from journals willing to publish the null findings that may ensue.

Kliemann et al. argue that three key hypotheses about the neurobiology of ASD are underconnectivity between brain regions, atypical function of the amygdala, and higher interindividual variability. To test these hypotheses, they employed a publicly available rs-fMRI data set, the Autism Brain Imaging Data Exchange (ABIDE), and using a preregistered Bayesian approach that was focused on basolateral and cortico-centromedial amygdala, compared magnitude, pattern similarity, and variability of resting-state connectivity. To assess the robustness of their findings, they tested the hypotheses using different anatomical ranges, rs-fMRI pipelines, and amygdala segmentation approaches.

The authors found no evidence for the three neurobiological hypotheses. Instead, their results depended on choices regarding rs-fMRI analyses, including anatomical specificity and preprocessing pipelines. These findings underscore the importance of a focus on methodological rigor in science in general, and neuroimaging in particular (3–5). The paper goes against a tide of neurobiology of mental disorder publications claiming the substantive discovery of novel findings and promising the imminent development of treatments targeted accordingly. It reminds us that science typically progresses incrementally, and that we ought to be cautious about proposed “paradigm shifts” in psychiatry (6).

Still, just as the replicability crisis has appropriately encouraged skepticism about significant results, so we need to be cautious about overreliance on or hype about null findings. With this in mind, we emphasize a number of points about the important contribution of Kliemann et al.

First, the three neurobiological hypotheses align well with the three statistical comparisons made (of magnitude, pattern similarity, and variability of resting-state connectivity). However, there are many hypotheses about the psychobiology of autism, and these may not be entirely subsumed by the resting-state connectivity hypotheses examined in this study. Among these are postulates about executive dysfunction, mirror neuron impairment, cerebral overgrowth, altered cortical minicolumns, and excitation-inhibition imbalance (7). As the authors emphasize, autism clearly involves atypical brain dysfunction, and further research is required. Rigorous laboratory and clinical research has already made important contributions to the cognitive-affective neuroscience of the amygdala and to the psychobiology of ASD, not the least by the authors themselves (8), and the null finding here should not detract from the energy with which we go forward.

Second, as the authors acknowledge, their work has several methodological limitations, and this tempers interpretation of its conclusions. Importantly, a study with around 200 cases and 300 controls is only powered to detect a Cohen’s d (effect size) of around 0.2. By way of context, a considerably larger structural neuroimaging study of ASD, with around 1,600 cases and 1,600 controls, found that cases had higher frontal and lower temporal cortical thickness (d values, −0.21 to 0.20) as well as subcortical volume alterations (d values, 0.13 to −0.13) (9). Additionally, the ABIDE data set is characterized by significant site effects, thus increasing variability. Furthermore, the choices of rs-fMRI-based parcellation and targeted subsets of regions and networks (top 10%, default mode networks) require further empirical evaluation.

Third, as the term “autism spectrum disorder” helpfully informs us, individuals with ASD are characterized by varying combinations of symptoms and degrees of functional impairment. While “lumping” approaches are required in research to address the issue of statistical power, and may well lead to important findings, they run the risk of being unable to adequately address disorder heterogeneity. Rare structural gene variants may play a particularly important role in ASD, and different variants may be associated with entirely different sorts of psychobiological impairment (10). Research on other neuropsychiatric conditions has laid the foundations for integrating neuroimaging and genomic findings (11, 12), and there is an important opportunity for work on ASD to do the same.

A balanced appraisal of the progress and pitfalls of the emerging literature on large-sample neuroimaging data sets is needed to inform future research directions. Bearing in mind the importance of balanced appraisal, we wish to emphasize a number of points.

First, advances in neuroimaging methods open the way to a range of potentially productive work. There have been important innovations in our ability to address site differences, to provide more fine-grained analysis of brain structures, including the amygdala, to compare developmental trajectories, to triangulate findings, and to undertake multi-model analyses (13–17). With the increased sample sizes that large-scale neuroimaging collaborations have provided, the field has already contributed to our understanding of mechanisms underpinning phenotype heterogeneity (18). Furthermore, emerging artificial intelligence and deep learning methods may detect complex biosignatures that are hard to define a priori; a recent fMRI study of ABIDE and two independent cohorts developed a novel spatiotemporal deep neural network that yielded robust and interpretable dynamic brain markers that distinguished ASD from neurotypical controls and predicted clinical symptom severity (19).

Second, we agree with the authors that work on rs-fMRI requires close attention to analytic methods. Issues that need to be addressed include optimal parcellation, signal-to-noise ratio, and comparison of different processing pipelines (20). The possibility that significant group findings reflect differences in movement artifacts requires ongoing consideration. The Enhancing Neuroimaging Genetics Through Meta-Analysis (ENIGMA) collaboration has placed particular emphasis on comparing different analytic approaches and on developing rigorous pipelines to allow harmonized investigation, thus optimizing data integration and minimizing intersite variability (21). This work also provides the impetus to undertake prospective harmonized neuroimaging research, in which all aspects of work, including phenotype characterization and imaging data acquisition, are standardized (22).

Third, neurobiological investigations are increasingly recognizing the importance of including more diverse participant samples. The significance of this point has been well articulated in medical and psychiatric genetics (23, 24), and is also relevant to work on divergent environmental exposomes and social determinants (25). International collaborations in neuroimaging are growing in size and scope, and these should ultimately allow investigation of multiple diverse genetic and environmental factors, in order to explain relevant phenotypes (21).

In summary, we are pleased to see publication of a rigorous neuroimaging study with null findings in the Journal, and also caution against concluding that all rigorous analyses of large neuroimaging data sets will find insufficient support for hypotheses about the psychobiology of ASD and other mental disorders. We should commend the contributions of “big data” imaging and genomic studies of mental disorders to advancing our understanding of these conditions, and also further refine research methods so as to avoid the pitfalls of this work. Over the long haul, we hope to see further progress in delineating clinical heterogeneity, clarifying underlying psychobiological mechanisms, and developing innovative interventions.