Multi-scale patch and multi-modality atlases for whole heart segmentation of MRI☆
Graphical abstract
Introduction
According to the World Health Organization, an estimated 17.3 million people died from cardiovascular diseases (CVDs) in 2008, accounting for 30% of deaths around the world (Organization, 2011). Early diagnosis and treatment play a vital role in reducing the mortality and morbidity of CVDs. Recently advances in medical imaging and computing technology and their application to the clinics have shown great potential towards achieving this goal. Cardiac magnetic resonance imaging (MRI) is a unique non-ionizing radiation technique which provides clear view of heart’s anatomy. To enable the development of novel clinical applications and thus improve cardiology, accurate and automatic extraction of the anatomical information becomes particularly important.
Whole heart segmentation (WHS) aims to extract the substructures of the heart, commonly including the four chamber blood cavities, the left ventricular myocardium, and sometimes the great vessels as well if they are of interest (Zheng, Barbu, Georgescu, Scheuering, Comaniciu, 2008, Peters, Ecabert, Meyer, Kneser, Weese, 2009, Zhuang, 2013). The results of WHS have a number of potential clinical applications. For instance, it can be directly used to calculate the functional indices of the heart such as the ejection fraction and myocardial mass; it can provide the initial geometric information of the heart for surgical guidance such as in radio-frequency ablation of the left atrium. It is also anticipated that the functional analysis of the whole heart has the potential of detecting subtle functional abnormalities or changes of the heart (Vasan et al., 1996). This is critical for the early diagnosis of patients who otherwise have normal systolic function of the ventricles but are suspected to have abnormal function in other regions. Two examples of WHS are provided in Fig. 1.
Obtaining fully automatic WHS is arduous due to the three major challenges: (1) the large shape variations of the cardiac anatomy, (2) the indistinct boundaries between substructures of the heart in cardiac MRI images, and (3) the low image quality (Zhuang, 2013). Zheng et al. (2008) developed a statistical shape model (SSM) and learning-based method to hierarchically detect boundary landmarks of the heart, based on the steerable features and marginal space learning. The SSM was used to regularize the resulting shape of the WHS defined by the detected landmarks. Peters et al. (2009) and Ecabert et al. (2011) developed a deformable model-based method for the WHS of both cardiac CT and MRI images. The large shape variations were tackled by the piecewise affine parametric adaptation and the boundary was detected by the simulated search of the optimal response of edges. Kirisli et al. (2010) constructed eight atlases and performed a multi-center, multi-vendor evaluation study on the WHS of CT data. The WHS performance was validated using a leave-one-out validation strategy on 8 datasets with manual segmentation. Zhuang et al. (2010) built a mean cardiac MRI atlas from 10 healthy subjects and developed a comprehensive registration algorithm for the atlas-based WHS of MRI. The authors employed the mean atlas to segment a test set involving nine different pathologies, relying on the locally affine registration method (LARM) to tackle the large shape variability of the cardiac anatomy. They also showed that the conventional multi-atlas segmentation (MAS) scheme did not performed better than the single mean atlas segmentation. Recently, Zhuang et al. (2015) developed a MAS scheme for the WHS of CT images, which adopted a new atlas ranking algorithm based on conditional entropy. The results showed that both the atlas ranking and label fusion could greatly affect the performance of a MAS scheme.
Previous studies show that the MAS reaches optimal performance by fusing a certain number of selected atlases using an effective atlas ranking method (Aljabar, Heckemann, Hammers, Hajnal, Rueckert, 2009, Zhuang, Bai, Song, Zhan, Qian, Shi, Lian, Rueckert, 2015). The global atlas selection idea has been extended to local atlas selection, based on local substructures or organs (Shi, Yap, Fan, Gilmore, Lin, Dinggang, 2010, van Rikxoort, Isgum, Arzhaeva, Staring, Klein, Viergever, Pluim, van Ginneken, 2010, Wolz, Chu, Misawa, Mori, Rueckert, 2013). Recently, the idea has been further extended to the pixel level, to provide atlas selection for each location (Wolz, Chu, Misawa, Mori, Rueckert, 2013, Tong, Wolz, Gao, Guerrero, Hajnal, Rueckert, 2014, Tong, Wolz, Wang, Gao, Misawa, Fujiwara, Mori, Hajnal, Rueckert, 2015). Currently, the atlas selection strategy generally requires the users to determine the number of atlases selected for label fusion, either on the global level or local level. In this manner, the selection is equivalent to applying a binary threshold on the contribution of atlases during label fusion.
Furthermore, to implement the multi-level atlas selection, one needs to define the masks of local regions according to the definition of anatomical structures (Wolz et al., 2013). This limits the flexibility of choosing the number of levels and consequently reduces the applicability of the method.
Another general limitation of the conventional MAS is that the atlases are constructed from one single modality. Since the number of atlases determines the potential optimal performance of MAS, including multi-modality atlases can be beneficial to the applications when different modality atlases are available. Iglesias et al. (2013) developed a MAS method where the atlas-to-target image registration and label fusion are solved simultaneously. Their proposed method was applied to cross-modality atlas-based segmentation, i.e. the proton density brain MRI atlases were used to segment the T1-weighted MRI and vice versa. Wang et al. (2015) proposed a learning-based multi-source atlas segmentation method. In their work, each atlas had intensity images from three modalities, including T1-weighted, T2-weighted and fractional anisotropy MRI, and the atlases were used to segment the target images from one modality such as T1 MRI. This is also different from the segmentation using multi-modality atlases, where one atlas is constructed solely from one imaging modality and different atlases can be built from different modalities.
Recently, a number of works adopt learning-based methods to predict the label of a target patch (Zhang, Zhan, Metaxas, 2012, Zhang, Zhan, Dewan, Huang, Metaxas, Zhou, 2011, Tong, Wolz, Gao, Guerrero, Hajnal, Rueckert, 2014, Tong, Wolz, Wang, Gao, Misawa, Fujiwara, Mori, Hajnal, Rueckert, 2015, Bai, Shi, Ledig, Rueckert, 2015). In these methods, the known patches, with gold standard labeling, from the atlases are used as training data and the target patch is considered as the test data. The voting weights of the known patches are implicitly implemented in the learning-and-prediction algorithm, such as the linear sparse encode using dictionary learning framework or the nonlinear kernel support vector machine method (Awate, Whitaker, 2014, Tong, Wolz, Gao, Guerrero, Hajnal, Rueckert, 2014, Tong, Wolz, Wang, Gao, Misawa, Fujiwara, Mori, Hajnal, Rueckert, 2015, Bai, Shi, Ledig, Rueckert, 2015). These approaches generally adopt the online learning scheme and a large number of atlases for training, and the performance is comparable to the state-of-the-art joint label fusion method.
To address the challenges of WHS and aforementioned limitations in current MAS research, we propose a multi-modality multi-atlas segmentation (M3AS) framework for WHS of cardiac MRI. M3AS adopts a new multi-scale patch (MSP) strategy, based on the multi-scale theory (Lindeberg, 1998), to obtain hierarchical local atlas ranking. The multi-scale space theory can handle different-level information within a limited window and has been applied to feature extraction/detection and image matching (Lindeberg, 1998, Lindeberg, 2011, Holden, Griffin, Saeed, Hill, 2004, Leutenegger, Chli, Siegwart, 2011, Wu, Kim, Sanroma, Wang, Munsell, Shen, 2015, Lowe, 2004, Wu, Liu, Huang, Guo, Jiang, Yang, Chen, Feng, 2014). We develop the MSP to compute the patches from different scale spaces to represent the different levels of structural information, with low scale capturing local fine structure and high scale suppressing fine structure but providing global structural information of the image. This is different from the conventional patch-based methods which only compute the local structural information within the patch. To avoid increasing the computational complexity, we adopt the multi-resolution implementation and couple it with the MSP where the high-scale patch can be efficiently computed using a low-resolution image space.
The main contributions of this work are summarized as follows: (1) the MSP for patch-based hierarchical local atlas ranking; (2) a framework for M3AS label fusion which evaluates both global image similarity and local pixel/patch similarity using information theoretic measures; (3) the non-binary global atlas ranking scheme which is formulated using a truncated Gaussian kernel regression model and does not require explicit atlas selection; (4) a validation study of WHS from cardiac MRI where comparisons between different patch parameterization schemes and label fusion methods are provided.
The remainder of the paper is organized as follows: Section 2 describes the methodologies of this work. Section 3 introduces the experiments and results, followed by the discussion and conclusion in Section 4.
Section snippets
Method
This section elaborates on the methodologies of this paper. First, Section 2.1 presents the framework of MAS and local weighted label fusion (LWF). Here, we propose a local similarity metric for multi-modality images on the pixel level and introduce three joint label fusion (JLF) methods which are used for comparisons in the experiments. Then, Section 2.2 describes the proposed MSP for multi-level hierarchical local atlas ranking. Finally, Section 2.3 provides the framework of the proposed M3
Materials
Twenty cardiac whole heart volumetric MRI data, provided by the Imaging Division at King’s College London, were used. The MRI sequence was the balanced steady state free precession (b-SSFP) for whole heart imaging. A 3D triggering b-SSFP turbo field echo (TFE) sequence, with arrhythmia rejection, was modified to enable the imaging at the end diastolic phase. The sequence was implemented on a 1.5T clinical scanner (Philips Healthcare, Best, The Netherlands) equipped with 32 independent receive
Discussion and conclusion
This work presents a new whole heart segmentation (WHS) method using multi-modality atlases, i.e., M3AS, for cardiac MRI. The label fusion algorithm is based on the proposed multi-scale patch (MSP) and a new global atlas ranking scheme. MSP uses the information of images from multi-scale space and thus is able to capture different levels of the structural information of images for hierarchical local atlas ranking. Also, MSP adopts the spatially varying weight (SVW) scheme to leverage the
Acknowledgment
This work was partially supported by the Chinese NSFC research fund (81301283, 11501123, 11571081), the NSFC-RS fund (81511130090), and the SRF for ROCS, SEM of China (47th).
References (48)
- et al.
Multi-atlas based segmentation of brain images: Atlas selection and its effect on accuracy
NeuroImage
(2009) - et al.
Multi-atlas segmentation with augmented features for cardiac MR images
Med. Image Anal.
(2015) - et al.
Segmentation of the heart and great vessels in CT images using a model-based adaptation framework
Med. Image Anal.
(2011) - et al.
BEaST: brain extraction based on nonlocal segmentation technique
Neuroimage
(2012) - et al.
Automatic anatomical brain MRI segmentation combining label propagation and decision fusion
NeuroImage
(2006) - et al.
Right ventricle segmentation from cardiac MRI: a collation study
Med. Image Anal.
(2015) - et al.
Adaptive local multi-atlas segmentation application to the heart and the caudate nucleus
Med. Image Anal.
(2010) - et al.
Evaluation of atlas selection strategies for atlas-based image segmentation with application to confocal microscopy images of bee brains
NeuroImage
(2004) - et al.
Multi-classifier framework for atlas-based image segmentation
Pattern Recognit. Lett.
(2005) - et al.
Construction of multi-region-multi-reference atlases for neonatal brain MRI segmentation
NeuroImage
(2010)
Multiple instance learning for classification of dementia in brain MRI
Med. Image Anal.
Discriminative dictionary learning for abdominal multi-organ segmentation
Med. Image Anal.
Entropy and Laplacian images: structural representations for multi-modal registration
Med. Image Anal.
Links: learning-based multi-source integration framework for segmentation of infant brain images
NeuroImage
Hierarchical multi-atlas label fusion with multi-scale feature representation and label-specific patch partition
NeuroImage
User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability
Neuroimage
Deformable segmentation via sparse representation and dictionary learning
Med. Image Anal.
Combination strategies in multi-atlas image segmentation: application to brain MR data
IEEE Trans. Med. Imaging
Multiatlas segmentation as nonparametric regression
IEEE Trans. Med. Imaging
A probabilistic patch-based label fusion model for multi-atlas segmentation with registration refinement: application to cardiac MR images
IEEE Trans. Med. Imaging
Nonlocal patch-based label fusion for hippocampus segmentation
Medical Image Computing and Computer-Assisted Intervention
Generalized overlap measures for evaluation and validation in medical image analysis
IEEE Trans. Med. Imaging
Multi-channel mutual information using scale space
Medical Image Computing and Computer-Assisted Intervention
Nonlocal regularization for active appearance model: application to medial temporal lobe segmentation
Hum. Brain Mapp.
Cited by (405)
Source-free unsupervised domain adaptation: A survey
2024, Neural NetworksEvidence-based uncertainty-aware semi-supervised medical image segmentation
2024, Computers in Biology and MedicineDeep learning for unsupervised domain adaptation in medical imaging: Recent advancements and future perspectives
2024, Computers in Biology and MedicineLearning with limited annotations: A survey on deep semi-supervised learning for medical image segmentation
2024, Computers in Biology and MedicineSemi-supervised medical image segmentation via hard positives oriented contrastive learning
2024, Pattern RecognitionAdvances in medical image analysis with vision Transformers: A comprehensive review
2024, Medical Image Analysis
- ☆
“This paper was recommended for publication by Dr. Nicholas Ayache”.