Elsevier

NeuroImage

Volume 145, Part B, 15 January 2017, Pages 137-165
NeuroImage

Single subject prediction of brain disorders in neuroimaging: Promises and pitfalls

https://doi.org/10.1016/j.neuroimage.2016.02.079Get rights and content

Highlights

  • Past efforts on classification of brain disorders are comprehensively reviewed.

  • The common pitfalls from machine learning point of view are discussed.

  • Emerging trends related to single-subject prediction are reviewed and discussed.

Abstract

Neuroimaging-based single subject prediction of brain disorders has gained increasing attention in recent years. Using a variety of neuroimaging modalities such as structural, functional and diffusion MRI, along with machine learning techniques, hundreds of studies have been carried out for accurate classification of patients with heterogeneous mental and neurodegenerative disorders such as schizophrenia and Alzheimer's disease. More than 500 studies have been published during the past quarter century on single subject prediction focused on a multiple brain disorders. In the first part of this study, we provide a survey of more than 200 reports in this field with a focus on schizophrenia, mild cognitive impairment (MCI), Alzheimer's disease (AD), depressive disorders, autism spectrum disease (ASD) and attention-deficit hyperactivity disorder (ADHD). Detailed information about those studies such as sample size, type and number of extracted features and reported accuracy are summarized and discussed. To our knowledge, this is by far the most comprehensive review of neuroimaging-based single subject prediction of brain disorders. In the second part, we present our opinion on major pitfalls of those studies from a machine learning point of view. Common biases are discussed and suggestions are provided. Moreover, emerging trends such as decentralized data sharing, multimodal brain imaging, differential diagnosis, disease subtype classification and deep learning are also discussed. Based on this survey, there is extensive evidence showing the great potential of neuroimaging data for single subject prediction of various disorders. However, the main bottleneck of this exciting field is still the limited sample size, which could be potentially addressed by modern data sharing models such as the ones discussed in this paper. Emerging big data technologies and advanced data-intensive machine learning methodologies such as deep learning have coincided with an increasing need for accurate, robust and generalizable single subject prediction of brain disorders during an exciting time. In this report, we survey the past and offer some opinions regarding the road ahead.

Introduction

Neuroimaging has opened up an exciting non-invasive window into the human brain over the past few decades. This interdisciplinary field has attracted scientists from areas such as medicine, engineering, mathematics, physics, statistics, computer science, and psychology (Epstein et al., 2001). Imaging modalities such as magnetic resonance imaging (MRI) and magnetoencephalography (MEG) along with more traditional methods such as electroencephalography (EEG) have made it possible to non-invasively study various aspects of the human brain with unprecedented accuracy. MRI-related techniques such as structural MRI (sMRI), functional MRI (fMRI) and diffusion MRI (dMRI) have the benefit of providing localized spatial information about the brain structure and function as well as detailed functional and structural connectivity maps. These techniques have provided new insight into the human brain and have brought hope to researchers trying to unravel the secrets of one of the most complex systems in the universe, the human brain.

Structural MRI has made it possible to visualize the brain at high spatial resolution (one cubic millimeter or less) (Liang and Lauterbur, 2000). sMRI high resolution images of the brain are ideal for studying various brain structures and also for detecting physical abnormalities, lesions and damages. dMRI is an imaging technique for visualization of anatomical connections between different brain regions (Le Bihan et al., 2001, Merboldt et al., 1985). Functional MRI measures brain activity by detecting changes in the blood oxygenation (DeYoe et al., 1994, Ogawa et al., 1990). fMRI makes it possible to study functional regions and networks of the brain as well as temporal associations among them.

Unfortunately, brain disorders are major health problems in the US and the rest of the world that not only impair the lives of millions of people but also impose huge financial burdens on societies (DiLuca and Olesen, 2014, Ernst and Hay, 1994, Rice, 1999). Moreover, there are no clinical tests to identify many brain disorders such as schizophrenia. One of the major hopes underlying the advanced neuroimaging tools mentioned above is to provide new understanding of brain disorders such as schizophrenia, bipolar disorder, autism spectrum disorder (ASD), Alzheimer's disease (AD), major depressive disorders, attention-deficit hyperactivity disorder (ADHD) and mild cognitive impairment (MCI). Brain disorder research aims at understanding the impact of each disease on the brain's function and structure from the cellular to system level, as well as the pathogenesis of these complex disorders. As a result, thousands of studies have been published on different aspects of brain disorders to show aberrations of some features (structural or functional) in a patient group usually in comparison with a healthy cohort (Jack et al., 1997, Jafri et al., 2008, Lorenzetti et al., 2009, McAlonan et al., 2005). While these studies are valuable in terms of finding relevant disease biomarkers, they are not sufficient for direct clinical diagnostic/prognostic adoption. The main reason is that many of these findings are statistically significant at the group level, but the individual discrimination ability of the proposed biomarkers is not typically evaluated. Since classification provides information for each individual subject, it is considered a much harder task than reporting group differences.

In recent years, there has been a growing trend in designing neuroimaging-based prognostic/diagnostic tools. As a result, there have been a lot of efforts using neuroimaging methods to automatically discriminate patients with brain disorders from healthy control or from each other (Klöppel et al., 2012). Many of these studies have reported promising prediction performances with the claim that complex diseases can be diagnosed robustly, accurately and rapidly in an automatic fashion. However, until now, these tools have not been integrated into the clinical realm. We believe that the main reason for this is that many of the studies of this nature, despite the promising results on a specific research dataset, are not designed to generalize to other datasets, specifically the clinical ones.

The purpose of this study was two-fold. First, we reviewed a large number of MRI-based brain disorder diagnostic/prognostic studies in schizophrenia, ASD, ADHD, depressive disorder, MCI and Alzheimer's disease. These studies are compared in a number of key aspects such as type of features, classifier and reported accuracies. Next, we formed our opinion on the issues associated with how machine learning is applied in neuroimaging and have suggested solutions that might address these pitfalls. Considering the immense potential of neuroimaging tools for clinical adoption, careful implementation and interpretation of machine learning in neuroimaging is crucial. Machine learning is a relatively new domain for many neuroimaging researchers coming from other fields and therefore pitfalls are unfortunately not rare. We attempt to identify and emphasize some common mistakes that resulted in these shortcomings and biases. At the end, we discuss emerging trends in neuroimaging such as data sharing, multimodal brain imaging and differential diagnosis.

As pointed out in the Introduction section, many brain disorder studies have shown abnormality in the average sense in one or more brain features in a patient cohort in comparison with a healthy group using statistical tests. The success of such methodology is usually measured by the means of p-values. On the other hand, the goal of single subject prediction is to automatically classify each subject into one of the groups in the study (e.g., healthy vs. patient). The success of classification studies is usually measured by accuracy.

These two problems are very different in essence as they try to address distinct research questions. In general, showing group differences is much easier compared to single subject prediction. To better illustrate the difference between these types of analysis, we show an example in Fig. 1. Suppose there are two groups each with 100 samples (subjects) and we have measurements of one brain feature for each subject. Fig. 1A shows a case where the mean values of the two groups are different as measured by a two-sample t-test. The difference is statistically significant (p-value = 0.001). However, if one tries to classify subjects based on a threshold on this brain feature (the dotted red line placed between the mean of two groups), a weak classification rate of 60.0% will be achieved. The reason for this is the range of values for that specific feature is highly overlapping for the two groups. So, a highly significant group difference does not necessarily translate into a strong classification result. But the opposite is also true, as high classification based on a feature doesn't necessarily mean that group-level mean differences exist. Fig. 1B shows a case where the two-sample t-test on the two groups is not significant (p-value = 0.86) but the classification based on two thresholds (red dotted lines placed between each mode of group 2 and mean of group 1) is very strong (94.5%). In this case, the abnormality is bidirectional, which does not cause significant mean differences but makes it possible to separate the groups with two thresholds (dotted lines). Interestingly, bidirectional abnormalities are observed in neuroimaging studies (Arbabshirani and Calhoun, 2011, Calhoun et al., 2006b). Fig. 1C shows a case where strong group differences and successful classification go hand in hand. The abnormality is one-directional and the mean difference is very significant (p-value < 2e −16). The mean of two groups is so far apart that the values of most of the samples of the two groups do not overlap. Therefore, a strong classification rate of 93.5% is achieved (based on one threshold).

The main purpose of example in Fig. 1 is to show that group level analysis and classification are two different methods for different problems. We will return to this example later for criticism of selecting features based on p-value.

Section snippets

Survey of MRI-based single-subject prediction of brain disorders

Based on a search on PubMed from 1990 to 2015,1

Common machine-learning pitfalls in neuroimaging

In this section, common pitfalls among the surveyed papers are discussed.

Machine learning in neuroimaging: shortcomings and emerging trends

Machine learning has more than two decades of history in neuroimaging and despite all of the promising results of numerous studies, it is still immature and not ready for integration into clinical healthcare. In this section, we review some of the challenges and emerging solutions.

Previous single-subject prediction surveys

In this study, we comprehensively reviewed past efforts in neuroimaging-based single subject prediction in several brain disorders such as MCI, AD, ASD, ADHD, schizophrenia and depressive disorders. Previous reviews include disease-specific surveys such as schizophrenia (Calhoun and Arbabshirani, 2012, Dazzan, 2014, Demirci et al., 2008b, Kambeitz et al., 2015, Veronese et al., 2013, Zarogianni et al., 2013), autism spectrum disorder (Retico et al., 2013, Retico et al., 2014), Alzheimer's

Acknowledgement

This work was supported by National Institutes of Health grants P20GM103472, R01EB005846, R01EB020407, 1R01DA040487 and NSF EPSCoR grant #1539067 (to V.D. Calhoun); “100 Talents Plan” of Chinese Academy of Sciences (to J. Sui); Chinese National Science Foundation No. 81471367 and the State High-Tech Development Plan (863) No. 2015AA020513. Also, we would like to thank Monica Jaramillo for the initial survey of neuroimaging studies. The authors report no financial relationships with commercial

References (371)

  • I. Beheshti et al.

    Probability distribution function-based classification of structural MRI for the detection of Alzheimer's disease

    Comput. Biol. Med.

    (2015)
  • J. Biederman

    Attention-deficit/hyperactivity disorder: a selective overview

    Biol. Psychiatry

    (2005)
  • M. Bleich-Cohen et al.

    Machine learning fMRI classifier delineates subgroups of schizophrenia patients

    Schizophr. Res.

    (2014)
  • A.L. Blum et al.

    Selection of relevant features and examples in machine learning

    Artif. Intell.

    (1997)
  • R. Brookmeyer et al.

    Forecasting the global burden of Alzheimer's disease

    Alzheimers Dement.

    (2007)
  • M.W. a Caan et al.

    Shaving diffusion tensor images in discriminant analysis: a study into schizophrenia

    Med. Image Anal.

    (2006)
  • S. Calderoni et al.

    Female children with autism spectrum disorder: an insight from mass-univariate and pattern classification analyses

    NeuroImage

    (2012)
  • a. Caprihan et al.

    Application of principal component analysis to distinguish patients with schizophrenia from healthy controls based on fractional anisotropy measurements

    NeuroImage

    (2008)
  • E. Castro et al.

    A multiple kernel learning approach to perform classification of groups from complex-valued fMRI data analysis: application to schizophrenia

    NeuroImage

    (2014)
  • E. Castro et al.

    Characterization of groups using composite kernels and multi-source fMRI analysis data: application to schizophrenia

    NeuroImage

    (2011)
  • M.S. Cetin et al.

    Thalamus and posterior temporal lobe show greater inter-network connectivity at rest and across sensory paradigms in schizophrenia

    NeuroImage

    (2014)
  • E. Challis et al.

    Gaussian process classification of Alzheimer's disease and mild cognitive impairment from resting-state fMRI

    NeuroImage

    (2015)
  • C.P. Chen et al.

    Diagnostic classification of intrinsic functional connectivity highlights somatosensory, default mode, and visual regions in autism

    Neuroimage Clin.

    (2015)
  • Y. Chen et al.

    Detecting brain structural changes as biomarker from magnetic resonance images using a local feature based SVM approach

    J. Neurosci. Methods

    (2014)
  • H. Cheng et al.

    Nodal centrality of functional network in the differentiation of schizophrenia

    Schizophr. Res.

    (2015)
  • A. Chincarini et al.

    Local MRI analysis approach in the diagnosis of early and prodromal Alzheimer's disease

    NeuroImage

    (2011)
  • C. Chu et al.

    Does feature selection improve classification accuracy? impact of sample size and feature selection on classification using anatomical magnetic resonance images

    NeuroImage

    (2012)
  • D. Chyzhyk et al.

    Computer aided diagnosis of schizophrenia on resting state fMRI data by ensembles of ELM

    Neural Netw.

    (2015)
  • S.G. Costafreda et al.

    Automated hippocampal shape analysis predicts the onset of dementia in mild cognitive impairment

    NeuroImage

    (2011)
  • P. Coupé et al.

    Simultaneous segmentation and grading of anatomical structures for patient's classification: application to Alzheimer's disease

    NeuroImage

    (2012)
  • Y. Cui et al.

    Automated detection of amnestic mild cognitive impairment in community-dwelling elderly adults: a combined spatial atrophy and white matter alteration approach

    NeuroImage

    (2012)
  • R. Cuingnet et al.

    Automatic classification of patients with Alzheimer's disease from structural MRI: a comparison of ten methods using the ADNI database

    NeuroImage

    (2011)
  • Z. Dai et al.

    Discriminative analysis of early Alzheimer's disease using multi-modal imaging and multi-level characterization with multi-classifier (M3)

    NeuroImage

    (2012)
  • C. DeCarli et al.

    Discriminant analysis of MRI measures as a method to determine the presence of dementia of the Alzheimer type

    Psychiatry Res.

    (1995)
  • O. Demirci et al.

    A projection pursuit algorithm to classify individuals using fMRI data: application to schizophrenia

    NeuroImage

    (2008)
  • S. Dey et al.

    Attributed graph distance measure for automatic detection of attention deficit hyperactive disordered subjects

    Front. Neural. Circuits

    (2014)
  • E.A. DeYoe et al.

    Functional magnetic resonance imaging (FMRI) of the human brain

    J. Neurosci. Methods

    (1994)
  • M. DiLuca et al.

    The cost of brain diseases: a burden or a challenge?

    Neuron

    (2014)
  • J. Dukart et al.

    Meta-analysis based SVM classification enables accurate detection of Alzheimer's disease across different clinical centers using FDG-PET and MRI

    Psychiatry Res.

    (2013)
  • A.J. Alberg et al.

    The use of “overall accuracy” to evaluate the validity of screening or diagnostic tests

    J. Gen. Intern. Med.

    (2004)
  • A. Anderson et al.

    Decreased small-world functional network connectivity and clustering across resting state networks in schizophrenia: an fMRI classification tutorial

    Front. Hum. Neurosci.

    (2013)
  • J.S. Anderson et al.

    Functional connectivity magnetic resonance imaging classification of autism

    Brain

    (2011)
  • A. Anticevic et al.

    Characterizing thalamo-cortical disturbances in schizophrenia and bipolar illness

    Cereb. Cortex

    (2014)
  • M.R. Arbabshirani et al.

    Functional network connectivity during rest and task: comparison of healthy controls and schizophrenic patients

  • M.R. Arbabshirani et al.

    Classification of schizophrenia patients based on resting-state functional network connectivity

    Front. Neurosci.

    (2013)
  • B.a. Ardekani et al.

    Diffusion tensor imaging reliably differentiates patients with schizophrenia from healthy volunteers

    Hum. Brain Mapp.

    (2011)
  • J.I. Arribas et al.

    Automatic Bayesian classification of healthy controls, bipolar disorder, and schizophrenia using intrinsic connectivity maps from fMRI data

    IEEE Trans. Biomed. Eng.

    (2010)
  • A.P. Association

    Diagnostic and Statistical Manual of Mental Disorders: DSM-5

    (2003)
  • Z.-J. Bai et al.

    Principal component analysis for distributed data sets with updating

  • J. Baio

    Prevalence of autism spectrum disorders: autism and developmental disabilities monitoring network, 14 sites, United States, 2008

    Morb. Mortal. Wkly. Rep. Surveill. Summ.

    (2012)
  • Cited by (649)

    View all citing articles on Scopus
    View full text