Elsevier

NeuroImage

Volume 142, 15 November 2016, Pages 407-420
NeuroImage

Connectome sensitivity or specificity: which is more important?

https://doi.org/10.1016/j.neuroimage.2016.06.035Get rights and content

Highlights

  • Connectome specificity is paramount.

  • False positives are at least twice as detrimental as false negatives.

  • False positives are inter-modular and thereby critically alter network topology.

  • Binary networks from dense connectomes require stringent thresholding.

Abstract

Connectomes with high sensitivity and high specificity are unattainable with current axonal fiber reconstruction methods, particularly at the macro-scale afforded by magnetic resonance imaging. Tensor-guided deterministic tractography yields sparse connectomes that are incomplete and contain false negatives (FNs), whereas probabilistic methods steered by crossing-fiber models yield dense connectomes, often with low specificity due to false positives (FPs). Densely reconstructed probabilistic connectomes are typically thresholded to improve specificity at the cost of a reduction in sensitivity. What is the optimal tradeoff between connectome sensitivity and specificity? We show empirically and theoretically that specificity is paramount. Our evaluations of the impact of FPs and FNs on empirical connectomes indicate that specificity is at least twice as important as sensitivity when estimating key properties of brain networks, including topological measures of network clustering, network efficiency and network modularity. Our asymptotic analysis of small-world networks with idealized modular structure reveals that as the number of nodes grows, specificity becomes exactly twice as important as sensitivity to the estimation of the clustering coefficient. For the estimation of network efficiency, the relative importance of specificity grows linearly with the number of nodes. The greater importance of specificity is due to FPs occurring more prevalently between network modules rather than within them. These spurious inter-modular connections have a dramatic impact on network topology. We argue that efforts to maximize the sensitivity of connectome reconstruction should be realigned with the need to map brain networks with high specificity.

Introduction

Methods for mapping connectomes are imperfect. Structural connections can be erroneously inferred between pairs of nodes that are truly disconnected, giving rise to spurious connections known as false positives (FPs) and reducing the specificity of connectome reconstructions. Conversely, genuine connections can be overlooked, resulting in false negatives (FNs) and reducing connectome sensitivity. Despite current state of the art, it remains challenging to reconstruct micro-, meso and macro-scale connectomes that display both high sensitivity and high specificity (Azadbakht et al., 2015, Bastiani et al., 2012, Calabrese et al., 2015, Knosche et al., 2015, Reveley et al., 2015, Thomas et al., 2014).

This study primarily focuses on the sensitivity and specificity of macro-scale connectomes, which are most often mapped with automated fiber tracking methods (tractography) performed on diffusion-weighted magnetic resonance imaging data (Hagmann et al., 2008, Sporns et al., 2005). A considerable variety of tractography algorithms has been developed to reconstruct axonal fiber bundles and thereby infer where connections should be placed in network models of the brain. Typically, millions of streamlines that follow the trajectories of all major neural white matter pathways are initiated throughout the brain and the number of streamlines interconnecting pairs of brain regions comprising a predefined parcellation atlas are enumerated to yield a connectivity matrix of streamline counts (Li et al., 2012). Deterministic tractography algorithms (Conturo et al., 1999, Mori et al., 1999) guided by the diffusion tensor are criticized for their failure to resolve crossing-fiber geometries (Alexander et al., 2007). This failure predominantly results in FNs, but can also yield FPs as well, depending on the specific method, data quality and parcellation resolution (Zalesky et al., 2010a). Connectome sensitivity can be substantially improved with probabilistic tractography algorithms (Behrens et al., 2003, Koch et al., 2002) that are combined with sophisticated crossing-fiber models (Behrens et al., 2007, Tournier et al., 2008), but probabilistic methods can produce FPs.

These issues are most clearly borne out when considering the discrepancy in connection density between tractography methods. When reconstructed with tensor-guided deterministic tractography, the human connectome typically has a connection density ranging between 1% and 40% (e.g. Van den Heuvel et al., 2012, Zalesky et al., 2010a), whereas most probabilistic methods that model crossing fibers yield densities exceeding 50–60% and can even be as high as 99–100% (e.g. Roberts et al., 2016). In other words, probabilistic streamlines can be found between more than half of all pairs of brain regions. How can estimates of such a basic connectome property differ so drastically between these methods? Is it that tensor-based methods yield many FNs and are probabilistic crossing-fiber methods confounded by FPs (Thomas et al., 2014)?

One way to reconcile this discrepancy is to adopt a Bayesian view and assume probabilistic tractography provides an estimate of the likelihood of a connection. Whereas a single deterministic streamline might be considered adequate to indicate the presence of a connection, a single probabilistic streamline is unlikely to provide sufficient evidence to make such inference and might therefore be thresholded away when forming a binary network. However, the difficulty with Bayesian inference is that streamline counts and other tractography outputs do not differentiate between connection probabilities and connection strengths (Jones, 2010, Kaden et al., 2007). Does a high streamline count indicate a strong connection comprising many axonal projections, or a highly probable yet weak connection comprising few axons (Jones et al., 2013)? The difficulty in divorcing connection probability from connection strength challenges simple applications of Bayesian inference.

Despite these concerns, it is common to assume a monotonic relationship between connection probability and streamline count. This assumption enables the use of thresholding methods to eliminate likely FPs from dense connectomes reconstructed with probabilistic tractography. Thresholding involves progressively eliminating connections with the lowest streamline count until a desired connection density is attained (Fornito et al., 2013, Fornito et al., 2016, van Wijk et al., 2010). While eliminating connections with a low streamline count can improve connectome specificity, not all eliminated connections are necessarily FPs, and thus any gain in specificity is inevitably traded for a loss in sensitivity (Azadbakht et al., 2015, Knosche et al., 2015, Thomas et al., 2014). Therefore, while thresholding methods cannot yield connectomes displaying both high sensitivity and high specificity, they may allow a tradeoff to be achieved between these two measures, assuming a monotonic relation between streamline counts and connection probabilities. Lenient thresholds produce dense connectomes with high sensitivity, whereas stringent thresholds yield sparse connectomes with high specificity.

Thresholding is however in many senses an unsettling approach; all the finesse of a sophisticated crossing-fiber model and probabilistic tractography is largely overridden by a simple and arbitrary threshold, which ultimately determines the most fundamental property of a connectome—its connection density. In this way, the burden of connectome reconstruction is precariously balanced on a single threshold, with less faith placed in the accuracy of the reconstruction process itself.

An important choice must therefore be made between sensitivity and specificity. Should the dense and highly sensitive reconstructions yielded by cutting-edge crossing-fiber models and probabilistic tractography be favored over the sparse and specific reconstructions that are characteristic of tensor-guided deterministic methods? Moreover, should thresholding be used to strike a balance between these two extremes of the sensitivity-specificity continuum? And if so, where along this continuum is the optimal tradeoff between sensitivity and specificity? These questions can be addressed by quantifying the relative detriment of FPs versus FNs. Are FPs worse than FNs to connectome accuracy, and if so, by how much?

The answer to these questions depends on the application at hand. For example, sensitivity is vital in neurosurgical planning, in order to minimize the risk of injury to axonal connections that would result in postoperative deficits. Statistical connectomics is another important application where this question manifests. When statistically comparing connectomes between groups (Griffa et al., 2013), FPs lead to a linear increase in the number of multiple comparisons, whereas FNs can conceal genuine group difference.

The analysis of connectome topology with the use of graph theory is the focus of this study and represents an important application (Bullmore and Sporns, 2009) for which little is known about the impact of connectome sensitivity and specificity. Is it FPs or FNs that lead to poorer estimation of the topological properties of a complex network, such as its efficiency, modularity and small-world organization? Addressing this question is crucial to determine the most appropriate connectome reconstruction methodology for maximizing the accuracy of graph theoretical analyses of brain networks.

It is trivial to see that FPs and FNs are equally detrimental to the measurement of some network properties. An example is the average nodal degree, which for a binary, undirected network is given by ∑i‍di/N, where N is the total number of nodes and di is the number of connections incident to the ith node (Rubinov and Sporns, 2010). It can be seen that each FP increases the average nodal degree by 2/N, since the degree of exactly two distinct nodes is increased by unity with the addition of a new connection, whereas each FN decreases the average nodal degree by the same amount. FPs and FNs are therefore equally detrimental to the average nodal degree because they introduce identical amounts of absolute error. As we will demonstrate here, this parity between sensitivity and specificity does not hold for most measures of complex network organization. The purpose of this study is to determine whether sensitivity or specificity is more important in these cases.

Connectome sensitivity and specificity is also an important concern at the micro- and meso-scale. While tract tracing techniques (Zaborszky et al., 2006) are often considered a gold standard, they are not without problems. FNs can arise due to distance dependencies of some tracers (Ercsey-Ravasz et al., 2013, Markov et al., 2013). FPs can arise due to tracer uptake by axons traversing a tracer-injected site without making any synapses at that site, and due to the spread of the tracer substance around the injection site (Dyrby et al., 2007, Knosche et al., 2015). These effects can potentially explain some of the substantial variation in connection density that is evident between macro-scale connectomes mapped using tract tracing, ranging as high as 66% for the Markov-Kennedey macaque network (Markov et al., 2014) to densities well below 50% for mouse connectome(Oh et al., 2014). Also see Table 1.

The aim of this paper is to quantify the impact of FPs and FNs on the estimation of complex network properties of binary connectomes mapped at various scales. We systematically increase the number of FPs and FNs in micro-scale (worm), meso-scale (mouse) and macro-scale (human) connectomes to quantify the extent to which the estimation of three commonly used descriptors of connectome topology—network clustering, network efficiency and network modularity—are altered with the addition of each new FP and FN. We only consider binary networks here, in which case the goal of connectome reconstruction is to infer the absence or presence of connections. We first empirically quantify the relative impact of FPs to FNs on the ability to accurately estimate complex network properties in connectomes mapped at various scales. We then undertake an asymptotic analysis on small-world networks with idealized modular structure to verify our empirical findings analytically and to understand why FPs and FNs differ in the extent to which they affect graph theoretical analyses of connectomes.

Section snippets

Methods

We analyzed the impact of false positive and negative connections on macro-scale (human), meso-scale (mouse) and micro-sale (worm) anatomical brain networks. Erdös-Rényi (random) networks comprising 100 nodes with a connection density of 20% were also evaluated to provide a point of comparison. All networks were modeled as binary graphs. The mouse and worm networks were directed, whereas the human and random networks were undirected. This distinction is due to the inability to infer fiber

Results

Network clustering, network efficiency and network modularity were measured in empirical brain networks that were perturbed by incrementally adding FP or FN connections. We report on the extent to which these network properties were misestimated when the number of FPs or FNs was increased to model errors in connectome reconstruction. We begin with uniform placement of FPs and FNs, as dictated by the uniform model. The results of our evaluation are summarized in Table 2 and described in detail

Why are FPs more detrimental than FNs?

It remains to be determined why FPs are more detrimental than FNs to estimating topological properties of connectomes. To address this question, we consider a network model that is typical of brain networks. The model is a small-world network with idealized modular structure and a rich club (Fig. 6). The network comprises N modules. Each module comprises n nodes that are fully connected. The N modules are interconnected by a fully-connected rich club, with one rich-club node residing in each

Discussion

Developing axonal fiber tracking methods that provide high sensitivity to resolve intricate fiber architectures and navigate crossing-fiber geometries with ease is crucially important across all axonal scales, from individual axons to large axonal fiber bundles. However, when these methods are applied to reconstruct connectomes for the purpose of studying the topological properties of the brain (Bullmore and Sporns, 2009), upholding specificity becomes paramount and significantly more important

Acknowledgements

A.Z., A.F., L.C., L.L.H and M.B. are supported by the Australian National Health and Medical Research Council (grant identifiers: 1050504, 1066779, 1047648, 1037196, 1103252 and 1099082) and the Australian Research Council (ID: FT130100589, CE140100007). M.P.vdH. is supported by a VENI (451-12-001) grant from the Netherlands Organization for Scientific Research and a fellowship from MQ.

References (72)

  • A. Griffa et al.

    Structural connectomics in brain diseases

    NeuroImage

    (2013)
  • Y. Iturria-Medina et al.

    Studying the human brain anatomical network via diffusion-weighted MRI and Graph Theory

    NeuroImage

    (2008)
  • S. Jbabdi et al.

    A Bayesian framework for global tractography

    NeuroImage

    (2007)
  • D.K. Jones et al.

    White matter integrity, fiber count, and other fallacies: the do's and don'ts of diffusion MRI

    NeuroImage

    (2013)
  • E. Kaden et al.

    Parametric spherical deconvolution: inferring anatomical connectivity using diffusion MR imaging

    NeuroImage

    (2007)
  • M.A. Koch et al.

    An investigation of functional and anatomical connectivity using magnetic resonance imaging

    NeuroImage

    (2002)
  • D.E. Meskaldji et al.

    Improved statistical evaluation of group differences in connectomes by screening-filtering strategy with application to study maturation of brain connections between childhood and adolescence

    NeuroImage

    (2015)
  • B. Mišić et al.

    Cooperative and competitive spreading dynamics on the human connectome

    Neuron

    (2015)
  • J.A. Roberts et al.

    The contribution of geometry to the human connectome

    NeuroImage

    (2016)
  • M. Rubinov et al.

    Complex network measures of brain connectivity: uses and interpretations

    NeuroImage

    (2010)
  • C.T. Shih et al.

    Connectomics-based analysis of information flow in the Drosophila brain

    Curr. Biol.

    (2015)
  • R.E. Smith et al.

    SIFT: spherical-deconvolution informed filtering of tractograms

    NeuroImage

    (2013)
  • S.N. Sotiropoulos et al.

    Brain tractography using Q-ball imaging and graph theory: improved connectivities through fibre crossings via a model-based approach

    NeuroImage

    (2010)
  • O. Sporns et al.

    Organization, development and function of complex brain networks

    Trends Cogn. Sci.

    (2004)
  • J.D. Tournier et al.

    Resolving crossing fibres using constrained spherical deconvolution: validation using diffusion-weighted imaging phantom data

    NeuroImage

    (2008)
  • N. Tzourio-Mazoyer et al.

    Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain

    NeuroImage

    (2002)
  • M.P. van den Heuvel et al.

    Comparative connectomics

    Trends Cogn. Sci.

    (2016)
  • A. Zalesky et al.

    Whole-brain anatomical networks: does the choice of nodes matter?

    NeuroImage

    (2010)
  • B. Zingg et al.

    Neural networks of the mouse neocortex

    Cell

    (2014)
  • H. Azadbakht et al.

    Validation of high-resolution tractography against in vivo tracing in the macaque visual cortex

    Cereb. Cortex

    (2015)
  • D.S. Bassett et al.

    Efficient physical embedding of topologically complex information processing networks in brains and computer circuits

    PLoS Comput. Biol.

    (2010)
  • T.E. Behrens et al.

    Characterization and propagation of uncertainty in diffusion-weighted MR imaging

    Magn. Reson. Med.

    (2003)
  • R.F. Betzel et al.

    Multi-scale community organization of the human structural connectome and its relationship with restingstate functional connectivity

    Netw. Sci.

    (2013)
  • V.D. Blondel et al.

    Fast unfolding of communities in large networks

    J. Stat. Mech. Theory Exp.

    (2008)
  • M. Bota et al.

    Combining collation and annotation efforts toward completion of the rat and mouse connectomes in BAMS

    Front. Neuroinform.

    (2012)
  • E. Bullmore et al.

    Complex brain networks: graph theoretical analysis of structural and functional systems

    Nat. Rev. Neurosci.

    (2009)
  • Cited by (225)

    View all citing articles on Scopus
    View full text