Generative Sampling in Bundle Tractography using Autoencoders (GESTA)
Graphical abstract
Introduction
White matter (WM) brain tractography has become an essential tool to study structural connectivity and track-specific properties in a wide range of applications. Tractography is the computational process of integrating local fiber orientation reconstructions of the white matter into long-range pathways reaching the gray matter (GM). Most commonly, such mapping is done through streamline propagation methods: given a map of local orientations, such as a voxel-wise fiber Orientation Distribution Function (fODF) estimated from diffusion Magnetic Resonance Imaging (dMRI) data, a map of initial locations (seeds), and possibly some anatomical constraints, continuous fiber trajectories are reconstructed using a numerical integration method (Jeurissen et al., 2019, O’Donnell et al., 2019). The result of this process is a “tractogram”, composed of a set of three-dimensional curves, called “streamlines”, estimating the white matter fiber pathways.
Conventional streamline propagation methods can either be deterministic or probabilistic, depending on whether they assume a unique fiber orientation in each voxel, or else, whether a distribution of possible trajectories are allowed at each location (Descoteaux et al., 2009). A number of other methods, including global optimization approaches (Reisert et al., 2011), particle filtering methods (Girard et al., 2014), or surface-enhanced (St-Onge et al., 2018, St-Onge et al., 2021), have emerged to try to adequately reconstruct the white matter pathways. A summary of these methods can be found in Jeurissen et al. (2019) and O’Donnell et al. (2019). More recently, deep learning-based tractography methods have been proposed as an alternative to conventional methods (Poulin et al., 2019).
However, modern tractography pipelines still miss to extract streamlines for some fiber pathways (Maffei et al., 2022, Maier-Hein et al., 2017). Essentially, models propagating local orientations produce tractography results that are inherently limited in their accuracy (Schilling et al., 2019a). Several works Rheault et al., 2020, Schilling et al., 2019b, Schilling et al., 2021b, Zhang et al., 2022 have studied the challenges faced by tractography methods, such as propagating streamlines through hard-to-track regions, which results in streamline groups (bundles) that show a poor volume occupancy and are relatively under-represented, among others (Jeurissen et al., 2019). Incorporating bundle-specific priors (anatomical or orientational, amongst others) to tractography was proposed in Chamberland et al., 2017, Rheault et al., 2019 as a way to increase the likelihood of reconstructing streamlines in hard-to-track regions.
Following our previous work using autoencoders in tractography (Legarreta et al., 2021), we show that an autoencoder can be used to generate new and complete streamlines to better fill-up the white matter, especially the hard-to-track regions and bundles. Our contributions are twofold: (i) a generative, autoencoder-based method for extracting new streamline candidates from the latent space in a bundle-wise fashion; and (ii) an evaluation framework to asses the anatomical plausibility and dMRI signal fit of the generated streamline data. We show that our approach can be successfully applied to both synthetic and human brain in vivo data with extensive experiments. Initiated on the results of an existing tractogram, our generative tractography method yields anatomically plausible streamlines that can be used to improve the spatial coverage of hard-to-track bundles or hard-to-extract clusters of streamlines. The proposed anatomical plausibility framework ensures that the provided streamlines comply with the required geometrical, brain tissue occupancy, diffusion signal fit, and connectivity features. To the best of our knowledge, GESTA (Generative Sampling in Bundle Tractography using Autoencoders) is the first deep generative tractography method.
A number of works have proposed ways to improve the results of tractography methods to provide a better white matter spatial coverage and/or cortical coverage, either using bundle-specific or whole-brain approaches.
Chamberland et al. (2014) proposed to iteratively improve the output of a tracking method by interactively changing the tractography parameters. By selectively seeding regions of interest, such as those with a poor spatial coverage, and adjusting the streamline propagation parameters, authors showed that their method produces tractograms with an improved white matter occupancy. The same authors later introduced a method (Chamberland et al., 2017) to improve tracking of the optic radiation by modifying the streamline propagation equation according to orientational priors within some given regions of interest. Bundle-specific tractography (Rheault et al., 2018, Rheault et al., 2019) requires a set of bundle templates to scale the fiber orientation distributions accordingly, and employs a multi-parametric approach to extract streamlines in a bundle-wise fashion. In Poulin et al. (2018) authors used a different deep recurrent neural network to reconstruct streamlines for each of the considered bundles.
Whole-brain tractography strategies have been introduced to offer generalized solutions to enhance the reconstruction of hard-to-track bundles. In Battocchio et al., 2020, Battocchio et al., 2021, the authors presented a method to improve the WM spatial coverage of a tractogram by re-parameterizing the existing set of streamlines using B-spline functions and computing new streamline trajectories using Markov chain Monte Carlo (MCMC) methods. St-Onge et al. (2021) proposed to seed the WM/GM interface in an adaptive manner based on GM local features. They also proposed to optionally dynamically (iteratively) add seeds in regions presenting a low streamline endpoint density to provide a new tractogram that features an improved spatial and cortical surface coverage. Although this strategy excels in improving the surface coverage, it relies on the surface data availability, and is still hindered by the limitations in the underlying local tracking procedure, which might still track preferably along given orientations in the deep white matter.
Numerous methods using neural networks, based on optimizing the propagation direction predictions according to a loss function, have been proposed for whole-brain tractography in recent years (Poulin et al., 2019). Similarly, deep reinforcement learning has been applied to whole-brain tractography (Théberge et al., 2021) as another choice to avoid detrimental local detours in streamline propagation. In this case, choosing the most appropriate streamline propagation step is learned according to the values of a reward function.
In the broader context of diffusion MRI, deep generative models have emerged as a method capable of successfully performing image synthesis and super-resolution. Adversarial methods have been proposed to synthesize diffusion data (or its derivatives) using structural data (e.g. Anctil-Robitaille et al., 2020), or to provide high-resolution diffusion MRI from low-resolution images (e.g. Luo et al., 2022). These methods employ an adversarial discriminator to iteratively allow the network to improve the generated data so as to increase its anatomical reliability. Instead of using a discriminator network, autoencoder-based generative methods with explicit anatomical constraints have also been used successfully in cardiac image segmentation tasks (e.g. Painchaud et al., 2020). However, to the best of our knowledge, deep generative models have not been employed to generate a tractography product yet.
In this work, we propose a generative, autoencoder-based tractography approach that, sourcing from a set of streamlines, is able to reconstruct new, anatomically plausible streamlines globally. Our method can be readily applied to both synthetic and human brain in vivo tractography data. Compared to other solutions, our method (i) uses a single model to yield new and complete streamlines; (ii) it does not involve iterative optimizations of the seeding strategy or streamline trajectories; and (iii) it does not require locally propagating an orientational field. Our procedure works by generating new streamline candidates for a bundle of choice in the representation space of an autoencoder, and evaluating the anatomical plausibility of the streamlines to accept or to discard them. Our anatomical plausibility evaluation framework includes structural and connectivity constraints, streamline geometry properties, and fixel-based features (Raffelt et al., 2015).
Section snippets
Material and methods
We leverage the FINTA autoencoder architecture presented in Legarreta et al. (2021) to introduce a generative tractography method, GESTA (Generative Sampling in Bundle Tractography using Autoencoders). GESTA uses the same convolutional deep neural autoencoder network model to allow extracting new, complete streamlines for tractography. To this end, the autoencoder does not need to be retrained to accomplish the generative task; the learned latent space is re-used as a “streamline yard” to
Experiments
Conventional streamline propagation methods were used on all datasets to obtain the tractography data for this work. Local probabilistic tracking was used for the “Fiber Cup” and ISMRM 2015 Tractography Challenge datasets. The BIL&GIN callosal homotopic tractograms were obtained using the probabilistic setting of the Particle Filtering Tracking (PFT) (Girard et al., 2014) method. Further details relevant to the tracking parameters can be found in our previous work Legarreta et al. (2021). The
Results
First, we show how GESTA can generate streamlines on the “Fiber Cup” and ISMRM 2015 Tractography Challenge datasets close to completing their ground truth coverage using only a small proportion of seed streamlines. GESTA’s performance using the full set of available seed streamlines (still relatively low for many segments) is analyzed thereafter for the BIL&GIN callosal homotopic dataset. The TractoInferno results are finally compared to the conventional tractography baselines provided by the
Discussion
We presented GESTA (Generative Sampling in Bundle Tractography using Autoencoders), a generative method to provide new, anatomically plausible streamlines that enhance the spatial coverage of an existing tractogram. We demonstrate that the method can be particularly useful to reliably fill bundles that are poorly populated in tractograms. The method allows to circumvent the difficulties of a given tracking method on regions where a local orientation signal may prevent a streamline to be
Future work
In this work, the same generative sampling parameters (namely, the Parzen estimator bandwidth factor) are used to generate streamlines across different bundles. This allows to consistently evaluate the effect of a given set of parameters on the used success measures. However, the framework allows to provide different parameters to different bundles, which might improve the success of the method at generating streamlines for a given bundle.
Similarly, we use the same feature thresholds for all
Conclusions
In this work, we introduced the first deep generative bundle tractography framework. The generative tractography framework uses (i) the latent space of a deep autoencoder; (ii) a set of seed streamlines; (iii) a data sampling method; and (iv) a streamline plausibility evaluation framework to globally reconstruct anatomically plausible streamlines bundle-wise from the trained latent space of the autoencoder. Our generative tractography method is able to reliably yield plausible streamlines for
CRediT authorship contribution statement
Jon Haitz Legarreta: Conceptualization, Methodology, Software, Writing – original draft, Writing – review & editing. Laurent Petit: Data curation, Writing – review & editing. Pierre-Marc Jodoin: Conceptualization, Methodology, Writing – review & editing. Maxime Descoteaux: Conceptualization, Methodology, Writing – review & editing.
Declaration of Competing Interest
The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Jon Haitz Legarreta has patent #17/337,413 pending to Imeka Solutions inc.
Pierre-Marc Jodoin reports a relationship with Imeka Solutions inc. that includes: board membership and employment. Pierre-Marc Jodoin has patent #17/337,413 pending to Imeka Solutions inc.
Maxime Descoteaux reports a relationship with Imeka Solutions inc. that includes: board membership
Acknowledgments
This work has been partially supported by the Centre d’Imagerie Médicale de l’Université de Sherbrooke (CIMUS), Canada; the Axe d’Imagerie Médicale (AIM) of the Centre de Recherche du CHUS (CRCHUS), Canada; and the Réseau de Bio-Imagerie du Québec (RBIQ)/ Quebec Bio-imaging Network (QBIN) (FRSQ - Réseaux de recherche thématiques File: 35450). This research was enabled in part by support provided by Calcul Québec, Canada (www.calculquebec.ca) and the Digital Research Alliance of Canada Advanced
References (54)
- et al.
A reproducible evaluation of ANTs similarity metric performance in brain image registration
NeuroImage
(2011) - et al.
Sparse wars: A survey and comparative study of spherical deconvolution algorithms for diffusion MRI
NeuroImage
(2019) - et al.
Virtual in Vivo interactive dissection of white matter fasciculi in the human brain
NeuroImage
(2002) - et al.
A diffusion tensor imaging tractography Atlas for virtual in vivo dissections
Cortex
(2008) - et al.
Tractometer: Towards validation of tractography pipelines
Med. Image Anal.
(2013) - et al.
Quantitative evaluation of 10 tractography algorithms on a realistic diffusion MR phantom
NeuroImage
(2011) - et al.
Unbiased average age-appropriate Atlases for pediatric studies
NeuroImage
(2011) - et al.
Recognition of white matter bundles using local and global streamline-based registration and clustering
NeuroImage
(2018) - et al.
Towards quantitative connectivity analysis: Reducing tractography biases
NeuroImage
(2014) - et al.
Filtering in tractography using autoencoders (FINTA)
Med. Image Anal.
(2021)
Diffusion MRI super-resolution reconstruction via sub-pixel convolution generative adversarial network
Magn. Reson. Imaging
Insights from the IronTract challenge: Optimal methods for mapping brain pathways from multi-shell diffusion MRI
NeuroImage
BIL&GIN: A neuroimaging, cognitive, behavioral, and genetic database for the study of human brain lateralization
NeuroImage
Tractography and machine learning: Current state and open challenges
Magn. Reson. Imaging
Connectivity-based fixel enhancement: Whole-brain statistical analysis of diffusion MRI measures in the presence of crossing fibres
NeuroImage
Global fiber reconstruction becomes practical
NeuroImage
Bundle-specific tractography with incorporated anatomical and orientational priors
NeuroImage
Challenges in diffusion MRI tractography - Lessons learned from international benchmark competitions
Magn. Reson. Imaging
Limits to anatomical accuracy of diffusion tractography using modern approaches
NeuroImage
Fiber tractography bundle segmentation depends on scanner effects, vendor effects, acquisition resolution, diffusion sampling scheme, diffusion sensitization, and bundle segmentation workflow
NeuroImage
Surface-enhanced tractography (SET)
NeuroImage
Track-to-learn: A general framework for tractography with deep reinforcement learning
Med. Image Anal.
Robust determination of the fibre orientation distribution in diffusion MRI: Non-negativity constrained super-resolved spherical deconvolution
NeuroImage
Direct estimation of the fiber orientation density function from diffusion-weighted MRI data using spherical deconvolution
NeuroImage
Quantitative mapping of the brain’s structural connectivity using diffusion MRI tractography: A review
NeuroImage
Atlas-guided tract reconstruction for automated and comprehensive examination of the white matter anatomy
NeuroImage
Manifold-aware cycleGAN for high-resolution structural-to-DTI synthesis
Cited by (4)
Variational Autoencoders for Generating Synthetic Tractography-Based Bundle Templates in a Low-Data Setting
2023, Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS
- 1
Co-senior author. These authors contributed equally.