Generative Sampling in Bundle Tractography using Autoencoders (GESTA)

https://doi.org/10.1016/j.media.2023.102761Get rights and content

Highlights

  • Autoencoder-based generative tractography method.

  • Capable of generating new, complete streamlines for a variable number of bundles.

  • Provides anatomically plausible streamlines for hard-to-track bundles.

  • Streamline plausibility evaluation framework for deep learning generative methods.

  • Consistent results across synthetic and human brain diffusion MRI tractography data.

Abstract

Current tractography methods use the local orientation information to propagate streamlines from seed locations. Many such seeds provide streamlines that stop prematurely or fail to map the true white matter pathways because some bundles are “harder-to-track” than others. This results in tractography reconstructions with poor white and gray matter spatial coverage. In this work, we propose a generative, autoencoder-based method, named GESTA (Generative Sampling in Bundle Tractography using Autoencoders), that produces streamlines achieving better spatial coverage. Compared to other deep learning methods, our autoencoder-based framework uses a single model to generate streamlines in a bundle-wise fashion, and does not require to propagate local orientations. GESTA produces new and complete streamlines for any given white matter bundle, including hard-to-track bundles. Applied on top of a given tractogram, GESTA is shown to be effective in improving the white matter volume coverage in poorly populated bundles, both on synthetic and human brain in vivo data. Our streamline evaluation framework ensures that the streamlines produced by GESTA are anatomically plausible and fit well to the local diffusion signal. The streamline evaluation criteria assess anatomy (white matter coverage), local orientation alignment (direction), and geometry features of streamlines, and optionally, gray matter connectivity. The GESTA framework offers considerable gains in bundle overlap using a reduced set of seeding streamlines with a 1.5x improvement for the “Fiber Cup”, and 6x for the ISMRM 2015 Tractography Challenge datasets. Similarly, it provides a 4x white matter volume increase on the BIL&GIN callosal homotopic dataset, and successfully populates bundles on the multi-subject, multi-site, whole-brain in vivo TractoInferno dataset. GESTA is thus a novel deep generative bundle tractography method that can be used to improve the tractography reconstruction of the white matter.

Introduction

White matter (WM) brain tractography has become an essential tool to study structural connectivity and track-specific properties in a wide range of applications. Tractography is the computational process of integrating local fiber orientation reconstructions of the white matter into long-range pathways reaching the gray matter (GM). Most commonly, such mapping is done through streamline propagation methods: given a map of local orientations, such as a voxel-wise fiber Orientation Distribution Function (fODF) estimated from diffusion Magnetic Resonance Imaging (dMRI) data, a map of initial locations (seeds), and possibly some anatomical constraints, continuous fiber trajectories are reconstructed using a numerical integration method (Jeurissen et al., 2019, O’Donnell et al., 2019). The result of this process is a “tractogram”, composed of a set of three-dimensional curves, called “streamlines”, estimating the white matter fiber pathways.

Conventional streamline propagation methods can either be deterministic or probabilistic, depending on whether they assume a unique fiber orientation in each voxel, or else, whether a distribution of possible trajectories are allowed at each location (Descoteaux et al., 2009). A number of other methods, including global optimization approaches (Reisert et al., 2011), particle filtering methods (Girard et al., 2014), or surface-enhanced (St-Onge et al., 2018, St-Onge et al., 2021), have emerged to try to adequately reconstruct the white matter pathways. A summary of these methods can be found in Jeurissen et al. (2019) and O’Donnell et al. (2019). More recently, deep learning-based tractography methods have been proposed as an alternative to conventional methods (Poulin et al., 2019).

However, modern tractography pipelines still miss to extract streamlines for some fiber pathways (Maffei et al., 2022, Maier-Hein et al., 2017). Essentially, models propagating local orientations produce tractography results that are inherently limited in their accuracy (Schilling et al., 2019a). Several works Rheault et al., 2020, Schilling et al., 2019b, Schilling et al., 2021b, Zhang et al., 2022 have studied the challenges faced by tractography methods, such as propagating streamlines through hard-to-track regions, which results in streamline groups (bundles) that show a poor volume occupancy and are relatively under-represented, among others (Jeurissen et al., 2019). Incorporating bundle-specific priors (anatomical or orientational, amongst others) to tractography was proposed in Chamberland et al., 2017, Rheault et al., 2019 as a way to increase the likelihood of reconstructing streamlines in hard-to-track regions.

Following our previous work using autoencoders in tractography (Legarreta et al., 2021), we show that an autoencoder can be used to generate new and complete streamlines to better fill-up the white matter, especially the hard-to-track regions and bundles. Our contributions are twofold: (i) a generative, autoencoder-based method for extracting new streamline candidates from the latent space in a bundle-wise fashion; and (ii) an evaluation framework to asses the anatomical plausibility and dMRI signal fit of the generated streamline data. We show that our approach can be successfully applied to both synthetic and human brain in vivo data with extensive experiments. Initiated on the results of an existing tractogram, our generative tractography method yields anatomically plausible streamlines that can be used to improve the spatial coverage of hard-to-track bundles or hard-to-extract clusters of streamlines. The proposed anatomical plausibility framework ensures that the provided streamlines comply with the required geometrical, brain tissue occupancy, diffusion signal fit, and connectivity features. To the best of our knowledge, GESTA (Generative Sampling in Bundle Tractography using Autoencoders) is the first deep generative tractography method.

A number of works have proposed ways to improve the results of tractography methods to provide a better white matter spatial coverage and/or cortical coverage, either using bundle-specific or whole-brain approaches.

Chamberland et al. (2014) proposed to iteratively improve the output of a tracking method by interactively changing the tractography parameters. By selectively seeding regions of interest, such as those with a poor spatial coverage, and adjusting the streamline propagation parameters, authors showed that their method produces tractograms with an improved white matter occupancy. The same authors later introduced a method (Chamberland et al., 2017) to improve tracking of the optic radiation by modifying the streamline propagation equation according to orientational priors within some given regions of interest. Bundle-specific tractography (Rheault et al., 2018, Rheault et al., 2019) requires a set of bundle templates to scale the fiber orientation distributions accordingly, and employs a multi-parametric approach to extract streamlines in a bundle-wise fashion. In Poulin et al. (2018) authors used a different deep recurrent neural network to reconstruct streamlines for each of the considered bundles.

Whole-brain tractography strategies have been introduced to offer generalized solutions to enhance the reconstruction of hard-to-track bundles. In Battocchio et al., 2020, Battocchio et al., 2021, the authors presented a method to improve the WM spatial coverage of a tractogram by re-parameterizing the existing set of streamlines using B-spline functions and computing new streamline trajectories using Markov chain Monte Carlo (MCMC) methods. St-Onge et al. (2021) proposed to seed the WM/GM interface in an adaptive manner based on GM local features. They also proposed to optionally dynamically (iteratively) add seeds in regions presenting a low streamline endpoint density to provide a new tractogram that features an improved spatial and cortical surface coverage. Although this strategy excels in improving the surface coverage, it relies on the surface data availability, and is still hindered by the limitations in the underlying local tracking procedure, which might still track preferably along given orientations in the deep white matter.

Numerous methods using neural networks, based on optimizing the propagation direction predictions according to a loss function, have been proposed for whole-brain tractography in recent years (Poulin et al., 2019). Similarly, deep reinforcement learning has been applied to whole-brain tractography (Théberge et al., 2021) as another choice to avoid detrimental local detours in streamline propagation. In this case, choosing the most appropriate streamline propagation step is learned according to the values of a reward function.

In the broader context of diffusion MRI, deep generative models have emerged as a method capable of successfully performing image synthesis and super-resolution. Adversarial methods have been proposed to synthesize diffusion data (or its derivatives) using structural data (e.g. Anctil-Robitaille et al., 2020), or to provide high-resolution diffusion MRI from low-resolution images (e.g. Luo et al., 2022). These methods employ an adversarial discriminator to iteratively allow the network to improve the generated data so as to increase its anatomical reliability. Instead of using a discriminator network, autoencoder-based generative methods with explicit anatomical constraints have also been used successfully in cardiac image segmentation tasks (e.g. Painchaud et al., 2020). However, to the best of our knowledge, deep generative models have not been employed to generate a tractography product yet.

In this work, we propose a generative, autoencoder-based tractography approach that, sourcing from a set of streamlines, is able to reconstruct new, anatomically plausible streamlines globally. Our method can be readily applied to both synthetic and human brain in vivo tractography data. Compared to other solutions, our method (i) uses a single model to yield new and complete streamlines; (ii) it does not involve iterative optimizations of the seeding strategy or streamline trajectories; and (iii) it does not require locally propagating an orientational field. Our procedure works by generating new streamline candidates for a bundle of choice in the representation space of an autoencoder, and evaluating the anatomical plausibility of the streamlines to accept or to discard them. Our anatomical plausibility evaluation framework includes structural and connectivity constraints, streamline geometry properties, and fixel-based features (Raffelt et al., 2015).

Section snippets

Material and methods

We leverage the FINTA autoencoder architecture presented in Legarreta et al. (2021) to introduce a generative tractography method, GESTA (Generative Sampling in Bundle Tractography using Autoencoders). GESTA uses the same convolutional deep neural autoencoder network model to allow extracting new, complete streamlines for tractography. To this end, the autoencoder does not need to be retrained to accomplish the generative task; the learned latent space is re-used as a “streamline yard” to

Experiments

Conventional streamline propagation methods were used on all datasets to obtain the tractography data for this work. Local probabilistic tracking was used for the “Fiber Cup” and ISMRM 2015 Tractography Challenge datasets. The BIL&GIN callosal homotopic tractograms were obtained using the probabilistic setting of the Particle Filtering Tracking (PFT) (Girard et al., 2014) method. Further details relevant to the tracking parameters can be found in our previous work Legarreta et al. (2021). The

Results

First, we show how GESTA can generate streamlines on the “Fiber Cup” and ISMRM 2015 Tractography Challenge datasets close to completing their ground truth coverage using only a small proportion of seed streamlines. GESTA’s performance using the full set of available seed streamlines (still relatively low for many segments) is analyzed thereafter for the BIL&GIN callosal homotopic dataset. The TractoInferno results are finally compared to the conventional tractography baselines provided by the

Discussion

We presented GESTA (Generative Sampling in Bundle Tractography using Autoencoders), a generative method to provide new, anatomically plausible streamlines that enhance the spatial coverage of an existing tractogram. We demonstrate that the method can be particularly useful to reliably fill bundles that are poorly populated in tractograms. The method allows to circumvent the difficulties of a given tracking method on regions where a local orientation signal may prevent a streamline to be

Future work

In this work, the same generative sampling parameters (namely, the Parzen estimator bandwidth factor) are used to generate streamlines across different bundles. This allows to consistently evaluate the effect of a given set of parameters on the used success measures. However, the framework allows to provide different parameters to different bundles, which might improve the success of the method at generating streamlines for a given bundle.

Similarly, we use the same feature thresholds for all

Conclusions

In this work, we introduced the first deep generative bundle tractography framework. The generative tractography framework uses (i) the latent space of a deep autoencoder; (ii) a set of seed streamlines; (iii) a data sampling method; and (iv) a streamline plausibility evaluation framework to globally reconstruct anatomically plausible streamlines bundle-wise from the trained latent space of the autoencoder. Our generative tractography method is able to reliably yield plausible streamlines for

CRediT authorship contribution statement

Jon Haitz Legarreta: Conceptualization, Methodology, Software, Writing – original draft, Writing – review & editing. Laurent Petit: Data curation, Writing – review & editing. Pierre-Marc Jodoin: Conceptualization, Methodology, Writing – review & editing. Maxime Descoteaux: Conceptualization, Methodology, Writing – review & editing.

Declaration of Competing Interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Jon Haitz Legarreta has patent #17/337,413 pending to Imeka Solutions inc.

Pierre-Marc Jodoin reports a relationship with Imeka Solutions inc. that includes: board membership and employment. Pierre-Marc Jodoin has patent #17/337,413 pending to Imeka Solutions inc.

Maxime Descoteaux reports a relationship with Imeka Solutions inc. that includes: board membership

Acknowledgments

This work has been partially supported by the Centre d’Imagerie Médicale de l’Université de Sherbrooke (CIMUS), Canada; the Axe d’Imagerie Médicale (AIM) of the Centre de Recherche du CHUS (CRCHUS), Canada; and the Réseau de Bio-Imagerie du Québec (RBIQ)/ Quebec Bio-imaging Network (QBIN) (FRSQ - Réseaux de recherche thématiques File: 35450). This research was enabled in part by support provided by Calcul Québec, Canada (www.calculquebec.ca) and the Digital Research Alliance of Canada Advanced

References (54)

  • LuoS. et al.

    Diffusion MRI super-resolution reconstruction via sub-pixel convolution generative adversarial network

    Magn. Reson. Imaging

    (2022)
  • MaffeiC. et al.

    Insights from the IronTract challenge: Optimal methods for mapping brain pathways from multi-shell diffusion MRI

    NeuroImage

    (2022)
  • MazoyerB. et al.

    BIL&GIN: A neuroimaging, cognitive, behavioral, and genetic database for the study of human brain lateralization

    NeuroImage

    (2016)
  • PoulinP. et al.

    Tractography and machine learning: Current state and open challenges

    Magn. Reson. Imaging

    (2019)
  • RaffeltD.A. et al.

    Connectivity-based fixel enhancement: Whole-brain statistical analysis of diffusion MRI measures in the presence of crossing fibres

    NeuroImage

    (2015)
  • ReisertM. et al.

    Global fiber reconstruction becomes practical

    NeuroImage

    (2011)
  • RheaultF. et al.

    Bundle-specific tractography with incorporated anatomical and orientational priors

    NeuroImage

    (2019)
  • SchillingK.G. et al.

    Challenges in diffusion MRI tractography - Lessons learned from international benchmark competitions

    Magn. Reson. Imaging

    (2019)
  • SchillingK.G. et al.

    Limits to anatomical accuracy of diffusion tractography using modern approaches

    NeuroImage

    (2019)
  • SchillingK.G. et al.

    Fiber tractography bundle segmentation depends on scanner effects, vendor effects, acquisition resolution, diffusion sampling scheme, diffusion sensitization, and bundle segmentation workflow

    NeuroImage

    (2021)
  • St-OngeE. et al.

    Surface-enhanced tractography (SET)

    NeuroImage

    (2018)
  • ThébergeA. et al.

    Track-to-learn: A general framework for tractography with deep reinforcement learning

    Med. Image Anal.

    (2021)
  • TournierJ.-D. et al.

    Robust determination of the fibre orientation distribution in diffusion MRI: Non-negativity constrained super-resolved spherical deconvolution

    NeuroImage

    (2007)
  • TournierJ.-D. et al.

    Direct estimation of the fiber orientation density function from diffusion-weighted MRI data using spherical deconvolution

    NeuroImage

    (2004)
  • ZhangF. et al.

    Quantitative mapping of the brain’s structural connectivity using diffusion MRI tractography: A review

    NeuroImage

    (2022)
  • ZhangY. et al.

    Atlas-guided tract reconstruction for automated and comprehensive examination of the white matter anatomy

    NeuroImage

    (2010)
  • Anctil-RobitailleB. et al.

    Manifold-aware cycleGAN for high-resolution structural-to-DTI synthesis

  • Cited by (4)

    • Variational Autoencoders for Generating Synthetic Tractography-Based Bundle Templates in a Low-Data Setting

      2023, Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS
    1

    Co-senior author. These authors contributed equally.

    View full text