Single subject prediction of brain disorders in neuroimaging: Promises and pitfalls
Introduction
Neuroimaging has opened up an exciting non-invasive window into the human brain over the past few decades. This interdisciplinary field has attracted scientists from areas such as medicine, engineering, mathematics, physics, statistics, computer science, and psychology (Epstein et al., 2001). Imaging modalities such as magnetic resonance imaging (MRI) and magnetoencephalography (MEG) along with more traditional methods such as electroencephalography (EEG) have made it possible to non-invasively study various aspects of the human brain with unprecedented accuracy. MRI-related techniques such as structural MRI (sMRI), functional MRI (fMRI) and diffusion MRI (dMRI) have the benefit of providing localized spatial information about the brain structure and function as well as detailed functional and structural connectivity maps. These techniques have provided new insight into the human brain and have brought hope to researchers trying to unravel the secrets of one of the most complex systems in the universe, the human brain.
Structural MRI has made it possible to visualize the brain at high spatial resolution (one cubic millimeter or less) (Liang and Lauterbur, 2000). sMRI high resolution images of the brain are ideal for studying various brain structures and also for detecting physical abnormalities, lesions and damages. dMRI is an imaging technique for visualization of anatomical connections between different brain regions (Le Bihan et al., 2001, Merboldt et al., 1985). Functional MRI measures brain activity by detecting changes in the blood oxygenation (DeYoe et al., 1994, Ogawa et al., 1990). fMRI makes it possible to study functional regions and networks of the brain as well as temporal associations among them.
Unfortunately, brain disorders are major health problems in the US and the rest of the world that not only impair the lives of millions of people but also impose huge financial burdens on societies (DiLuca and Olesen, 2014, Ernst and Hay, 1994, Rice, 1999). Moreover, there are no clinical tests to identify many brain disorders such as schizophrenia. One of the major hopes underlying the advanced neuroimaging tools mentioned above is to provide new understanding of brain disorders such as schizophrenia, bipolar disorder, autism spectrum disorder (ASD), Alzheimer's disease (AD), major depressive disorders, attention-deficit hyperactivity disorder (ADHD) and mild cognitive impairment (MCI). Brain disorder research aims at understanding the impact of each disease on the brain's function and structure from the cellular to system level, as well as the pathogenesis of these complex disorders. As a result, thousands of studies have been published on different aspects of brain disorders to show aberrations of some features (structural or functional) in a patient group usually in comparison with a healthy cohort (Jack et al., 1997, Jafri et al., 2008, Lorenzetti et al., 2009, McAlonan et al., 2005). While these studies are valuable in terms of finding relevant disease biomarkers, they are not sufficient for direct clinical diagnostic/prognostic adoption. The main reason is that many of these findings are statistically significant at the group level, but the individual discrimination ability of the proposed biomarkers is not typically evaluated. Since classification provides information for each individual subject, it is considered a much harder task than reporting group differences.
In recent years, there has been a growing trend in designing neuroimaging-based prognostic/diagnostic tools. As a result, there have been a lot of efforts using neuroimaging methods to automatically discriminate patients with brain disorders from healthy control or from each other (Klöppel et al., 2012). Many of these studies have reported promising prediction performances with the claim that complex diseases can be diagnosed robustly, accurately and rapidly in an automatic fashion. However, until now, these tools have not been integrated into the clinical realm. We believe that the main reason for this is that many of the studies of this nature, despite the promising results on a specific research dataset, are not designed to generalize to other datasets, specifically the clinical ones.
The purpose of this study was two-fold. First, we reviewed a large number of MRI-based brain disorder diagnostic/prognostic studies in schizophrenia, ASD, ADHD, depressive disorder, MCI and Alzheimer's disease. These studies are compared in a number of key aspects such as type of features, classifier and reported accuracies. Next, we formed our opinion on the issues associated with how machine learning is applied in neuroimaging and have suggested solutions that might address these pitfalls. Considering the immense potential of neuroimaging tools for clinical adoption, careful implementation and interpretation of machine learning in neuroimaging is crucial. Machine learning is a relatively new domain for many neuroimaging researchers coming from other fields and therefore pitfalls are unfortunately not rare. We attempt to identify and emphasize some common mistakes that resulted in these shortcomings and biases. At the end, we discuss emerging trends in neuroimaging such as data sharing, multimodal brain imaging and differential diagnosis.
As pointed out in the Introduction section, many brain disorder studies have shown abnormality in the average sense in one or more brain features in a patient cohort in comparison with a healthy group using statistical tests. The success of such methodology is usually measured by the means of p-values. On the other hand, the goal of single subject prediction is to automatically classify each subject into one of the groups in the study (e.g., healthy vs. patient). The success of classification studies is usually measured by accuracy.
These two problems are very different in essence as they try to address distinct research questions. In general, showing group differences is much easier compared to single subject prediction. To better illustrate the difference between these types of analysis, we show an example in Fig. 1. Suppose there are two groups each with 100 samples (subjects) and we have measurements of one brain feature for each subject. Fig. 1A shows a case where the mean values of the two groups are different as measured by a two-sample t-test. The difference is statistically significant (p-value = 0.001). However, if one tries to classify subjects based on a threshold on this brain feature (the dotted red line placed between the mean of two groups), a weak classification rate of 60.0% will be achieved. The reason for this is the range of values for that specific feature is highly overlapping for the two groups. So, a highly significant group difference does not necessarily translate into a strong classification result. But the opposite is also true, as high classification based on a feature doesn't necessarily mean that group-level mean differences exist. Fig. 1B shows a case where the two-sample t-test on the two groups is not significant (p-value = 0.86) but the classification based on two thresholds (red dotted lines placed between each mode of group 2 and mean of group 1) is very strong (94.5%). In this case, the abnormality is bidirectional, which does not cause significant mean differences but makes it possible to separate the groups with two thresholds (dotted lines). Interestingly, bidirectional abnormalities are observed in neuroimaging studies (Arbabshirani and Calhoun, 2011, Calhoun et al., 2006b). Fig. 1C shows a case where strong group differences and successful classification go hand in hand. The abnormality is one-directional and the mean difference is very significant (p-value < 2e −16). The mean of two groups is so far apart that the values of most of the samples of the two groups do not overlap. Therefore, a strong classification rate of 93.5% is achieved (based on one threshold).
The main purpose of example in Fig. 1 is to show that group level analysis and classification are two different methods for different problems. We will return to this example later for criticism of selecting features based on p-value.
Section snippets
Survey of MRI-based single-subject prediction of brain disorders
Based on a search on PubMed from 1990 to 2015,1
Common machine-learning pitfalls in neuroimaging
In this section, common pitfalls among the surveyed papers are discussed.
Machine learning in neuroimaging: shortcomings and emerging trends
Machine learning has more than two decades of history in neuroimaging and despite all of the promising results of numerous studies, it is still immature and not ready for integration into clinical healthcare. In this section, we review some of the challenges and emerging solutions.
Previous single-subject prediction surveys
In this study, we comprehensively reviewed past efforts in neuroimaging-based single subject prediction in several brain disorders such as MCI, AD, ASD, ADHD, schizophrenia and depressive disorders. Previous reviews include disease-specific surveys such as schizophrenia (Calhoun and Arbabshirani, 2012, Dazzan, 2014, Demirci et al., 2008b, Kambeitz et al., 2015, Veronese et al., 2013, Zarogianni et al., 2013), autism spectrum disorder (Retico et al., 2013, Retico et al., 2014), Alzheimer's
Acknowledgement
This work was supported by National Institutes of Health grants P20GM103472, R01EB005846, R01EB020407, 1R01DA040487 and NSF EPSCoR grant #1539067 (to V.D. Calhoun); “100 Talents Plan” of Chinese Academy of Sciences (to J. Sui); Chinese National Science Foundation No. 81471367 and the State High-Tech Development Plan (863) No. 2015AA020513. Also, we would like to thank Monica Jaramillo for the initial survey of neuroimaging studies. The authors report no financial relationships with commercial
References (371)
- et al.
Effects of hardware heterogeneity on the performance of SVM Alzheimer's disease classifier
NeuroImage
(2011) - et al.
How early can we predict Alzheimer's disease using computational anatomy?
Neurobiol. Aging
(2013) - et al.
Different multivariate techniques for automated classification of MRI data in Alzheimer's disease and mild cognitive impairment
Psychiatry Res.
(2013) - et al.
Outcome classification of preschool children with autism spectrum disorders using MRI brain measures
J. Am. Acad. Child Adolesc. Psychiatry
(2004) - et al.
The diagnosis of mild cognitive impairment due to Alzheimer's disease: Recommendations from the National Institute on Aging-Alzheimer's Association workgroups on diagnostic guidelines for Alzheimer's disease
Alzheimers Dement.
(2011) - et al.
Partial least squares for discrimination in fMRI data
Magn. Reson. Imaging
(2012) - et al.
Non-negative matrix factorization of multimodal MRI, fMRI and phenotypic data reveals differential changes in default mode subnetworks in ADHD
NeuroImage
(2014) - et al.
ApoE4 effects on automated diagnostic classifiers for mild cognitive impairment and Alzheimer's disease
Neuroimage Clin.
(2014) - et al.
Automated method for identification of patients with Alzheimer's disease based on three-dimensional MR images
Acad. Radiol.
(2008) - et al.
Altered resting state complexity in schizophrenia
NeuroImage
(2012)
Probability distribution function-based classification of structural MRI for the detection of Alzheimer's disease
Comput. Biol. Med.
Attention-deficit/hyperactivity disorder: a selective overview
Biol. Psychiatry
Machine learning fMRI classifier delineates subgroups of schizophrenia patients
Schizophr. Res.
Selection of relevant features and examples in machine learning
Artif. Intell.
Forecasting the global burden of Alzheimer's disease
Alzheimers Dement.
Shaving diffusion tensor images in discriminant analysis: a study into schizophrenia
Med. Image Anal.
Female children with autism spectrum disorder: an insight from mass-univariate and pattern classification analyses
NeuroImage
Application of principal component analysis to distinguish patients with schizophrenia from healthy controls based on fractional anisotropy measurements
NeuroImage
A multiple kernel learning approach to perform classification of groups from complex-valued fMRI data analysis: application to schizophrenia
NeuroImage
Characterization of groups using composite kernels and multi-source fMRI analysis data: application to schizophrenia
NeuroImage
Thalamus and posterior temporal lobe show greater inter-network connectivity at rest and across sensory paradigms in schizophrenia
NeuroImage
Gaussian process classification of Alzheimer's disease and mild cognitive impairment from resting-state fMRI
NeuroImage
Diagnostic classification of intrinsic functional connectivity highlights somatosensory, default mode, and visual regions in autism
Neuroimage Clin.
Detecting brain structural changes as biomarker from magnetic resonance images using a local feature based SVM approach
J. Neurosci. Methods
Nodal centrality of functional network in the differentiation of schizophrenia
Schizophr. Res.
Local MRI analysis approach in the diagnosis of early and prodromal Alzheimer's disease
NeuroImage
Does feature selection improve classification accuracy? impact of sample size and feature selection on classification using anatomical magnetic resonance images
NeuroImage
Computer aided diagnosis of schizophrenia on resting state fMRI data by ensembles of ELM
Neural Netw.
Automated hippocampal shape analysis predicts the onset of dementia in mild cognitive impairment
NeuroImage
Simultaneous segmentation and grading of anatomical structures for patient's classification: application to Alzheimer's disease
NeuroImage
Automated detection of amnestic mild cognitive impairment in community-dwelling elderly adults: a combined spatial atrophy and white matter alteration approach
NeuroImage
Automatic classification of patients with Alzheimer's disease from structural MRI: a comparison of ten methods using the ADNI database
NeuroImage
Discriminative analysis of early Alzheimer's disease using multi-modal imaging and multi-level characterization with multi-classifier (M3)
NeuroImage
Discriminant analysis of MRI measures as a method to determine the presence of dementia of the Alzheimer type
Psychiatry Res.
A projection pursuit algorithm to classify individuals using fMRI data: application to schizophrenia
NeuroImage
Attributed graph distance measure for automatic detection of attention deficit hyperactive disordered subjects
Front. Neural. Circuits
Functional magnetic resonance imaging (FMRI) of the human brain
J. Neurosci. Methods
The cost of brain diseases: a burden or a challenge?
Neuron
Meta-analysis based SVM classification enables accurate detection of Alzheimer's disease across different clinical centers using FDG-PET and MRI
Psychiatry Res.
The use of “overall accuracy” to evaluate the validity of screening or diagnostic tests
J. Gen. Intern. Med.
Decreased small-world functional network connectivity and clustering across resting state networks in schizophrenia: an fMRI classification tutorial
Front. Hum. Neurosci.
Functional connectivity magnetic resonance imaging classification of autism
Brain
Characterizing thalamo-cortical disturbances in schizophrenia and bipolar illness
Cereb. Cortex
Functional network connectivity during rest and task: comparison of healthy controls and schizophrenic patients
Classification of schizophrenia patients based on resting-state functional network connectivity
Front. Neurosci.
Diffusion tensor imaging reliably differentiates patients with schizophrenia from healthy volunteers
Hum. Brain Mapp.
Automatic Bayesian classification of healthy controls, bipolar disorder, and schizophrenia using intrinsic connectivity maps from fMRI data
IEEE Trans. Biomed. Eng.
Diagnostic and Statistical Manual of Mental Disorders: DSM-5
Principal component analysis for distributed data sets with updating
Prevalence of autism spectrum disorders: autism and developmental disabilities monitoring network, 14 sites, United States, 2008
Morb. Mortal. Wkly. Rep. Surveill. Summ.
Cited by (649)
In-house data adaptation to public data: Multisite MRI harmonization to predict Alzheimer's disease conversion
2024, Expert Systems with ApplicationsPhototherapy for age-related brain diseases: Challenges, successes and future
2024, Ageing Research ReviewsPerformance reserves in brain-imaging-based phenotype prediction
2024, Cell ReportsNeural networks from biological to artificial and vice versa
2024, BioSystems