Elsevier

Medical Image Analysis

Volume 31, July 2016, Pages 77-87
Medical Image Analysis

Multi-scale patch and multi-modality atlases for whole heart segmentation of MRI

https://doi.org/10.1016/j.media.2016.02.006Get rights and content

Highlights

  • Multi-scale patch is proposed for hierarchical local atlas ranking.

  • A multi-modality multi-atlas scheme is developed for whole heart segmentation.

  • A validation and comparison study for whole heart MRI segmentation is performed.

  • Segmentation results demonstrate the performance of the proposed method.

  • The proposed method may be useful in clinical care of cardiovascular diseases.

Abstract

A whole heart segmentation (WHS) method is presented for cardiac MRI. This segmentation method employs multi-modality atlases from MRI and CT and adopts a new label fusion algorithm which is based on the proposed multi-scale patch (MSP) strategy and a new global atlas ranking scheme. MSP, developed from the scale-space theory, uses the information of multi-scale images and provides different levels of the structural information of images for multi-level local atlas ranking. Both the local and global atlas ranking steps use the information theoretic measures to compute the similarity between the target image and the atlases from multiple modalities. The proposed segmentation scheme was evaluated on a set of data involving 20 cardiac MRI and 20 CT images. Our proposed algorithm demonstrated a promising performance, yielding a mean WHS Dice score of 0.899 ± 0.0340, Jaccard index of 0.818 ± 0.0549, and surface distance error of 1.09 ± 1.11 mm for the 20 MRI data. The average runtime for the proposed label fusion was 12.58 min.

Introduction

According to the World Health Organization, an estimated 17.3 million people died from cardiovascular diseases (CVDs) in 2008, accounting for 30% of deaths around the world (Organization, 2011). Early diagnosis and treatment play a vital role in reducing the mortality and morbidity of CVDs. Recently advances in medical imaging and computing technology and their application to the clinics have shown great potential towards achieving this goal. Cardiac magnetic resonance imaging (MRI) is a unique non-ionizing radiation technique which provides clear view of heart’s anatomy. To enable the development of novel clinical applications and thus improve cardiology, accurate and automatic extraction of the anatomical information becomes particularly important.

Whole heart segmentation (WHS) aims to extract the substructures of the heart, commonly including the four chamber blood cavities, the left ventricular myocardium, and sometimes the great vessels as well if they are of interest (Zheng, Barbu, Georgescu, Scheuering, Comaniciu, 2008, Peters, Ecabert, Meyer, Kneser, Weese, 2009, Zhuang, 2013). The results of WHS have a number of potential clinical applications. For instance, it can be directly used to calculate the functional indices of the heart such as the ejection fraction and myocardial mass; it can provide the initial geometric information of the heart for surgical guidance such as in radio-frequency ablation of the left atrium. It is also anticipated that the functional analysis of the whole heart has the potential of detecting subtle functional abnormalities or changes of the heart (Vasan et al., 1996). This is critical for the early diagnosis of patients who otherwise have normal systolic function of the ventricles but are suspected to have abnormal function in other regions. Two examples of WHS are provided in Fig. 1.

Obtaining fully automatic WHS is arduous due to the three major challenges: (1) the large shape variations of the cardiac anatomy, (2) the indistinct boundaries between substructures of the heart in cardiac MRI images, and (3) the low image quality (Zhuang, 2013). Zheng et al. (2008) developed a statistical shape model (SSM) and learning-based method to hierarchically detect boundary landmarks of the heart, based on the steerable features and marginal space learning. The SSM was used to regularize the resulting shape of the WHS defined by the detected landmarks. Peters et al. (2009) and Ecabert et al. (2011) developed a deformable model-based method for the WHS of both cardiac CT and MRI images. The large shape variations were tackled by the piecewise affine parametric adaptation and the boundary was detected by the simulated search of the optimal response of edges. Kirisli et al. (2010) constructed eight atlases and performed a multi-center, multi-vendor evaluation study on the WHS of CT data. The WHS performance was validated using a leave-one-out validation strategy on 8 datasets with manual segmentation. Zhuang et al. (2010) built a mean cardiac MRI atlas from 10 healthy subjects and developed a comprehensive registration algorithm for the atlas-based WHS of MRI. The authors employed the mean atlas to segment a test set involving nine different pathologies, relying on the locally affine registration method (LARM) to tackle the large shape variability of the cardiac anatomy. They also showed that the conventional multi-atlas segmentation (MAS) scheme did not performed better than the single mean atlas segmentation. Recently, Zhuang et al. (2015) developed a MAS scheme for the WHS of CT images, which adopted a new atlas ranking algorithm based on conditional entropy. The results showed that both the atlas ranking and label fusion could greatly affect the performance of a MAS scheme.

Previous studies show that the MAS reaches optimal performance by fusing a certain number of selected atlases using an effective atlas ranking method (Aljabar, Heckemann, Hammers, Hajnal, Rueckert, 2009, Zhuang, Bai, Song, Zhan, Qian, Shi, Lian, Rueckert, 2015). The global atlas selection idea has been extended to local atlas selection, based on local substructures or organs (Shi, Yap, Fan, Gilmore, Lin, Dinggang, 2010, van Rikxoort, Isgum, Arzhaeva, Staring, Klein, Viergever, Pluim, van Ginneken, 2010, Wolz, Chu, Misawa, Mori, Rueckert, 2013). Recently, the idea has been further extended to the pixel level, to provide atlas selection for each location (Wolz, Chu, Misawa, Mori, Rueckert, 2013, Tong, Wolz, Gao, Guerrero, Hajnal, Rueckert, 2014, Tong, Wolz, Wang, Gao, Misawa, Fujiwara, Mori, Hajnal, Rueckert, 2015). Currently, the atlas selection strategy generally requires the users to determine the number of atlases selected for label fusion, either on the global level or local level. In this manner, the selection is equivalent to applying a binary threshold on the contribution of atlases during label fusion.

Furthermore, to implement the multi-level atlas selection, one needs to define the masks of local regions according to the definition of anatomical structures (Wolz et al., 2013). This limits the flexibility of choosing the number of levels and consequently reduces the applicability of the method.

Another general limitation of the conventional MAS is that the atlases are constructed from one single modality. Since the number of atlases determines the potential optimal performance of MAS, including multi-modality atlases can be beneficial to the applications when different modality atlases are available. Iglesias et al. (2013) developed a MAS method where the atlas-to-target image registration and label fusion are solved simultaneously. Their proposed method was applied to cross-modality atlas-based segmentation, i.e. the proton density brain MRI atlases were used to segment the T1-weighted MRI and vice versa. Wang et al. (2015) proposed a learning-based multi-source atlas segmentation method. In their work, each atlas had intensity images from three modalities, including T1-weighted, T2-weighted and fractional anisotropy MRI, and the atlases were used to segment the target images from one modality such as T1 MRI. This is also different from the segmentation using multi-modality atlases, where one atlas is constructed solely from one imaging modality and different atlases can be built from different modalities.

Recently, a number of works adopt learning-based methods to predict the label of a target patch (Zhang, Zhan, Metaxas, 2012, Zhang, Zhan, Dewan, Huang, Metaxas, Zhou, 2011, Tong, Wolz, Gao, Guerrero, Hajnal, Rueckert, 2014, Tong, Wolz, Wang, Gao, Misawa, Fujiwara, Mori, Hajnal, Rueckert, 2015, Bai, Shi, Ledig, Rueckert, 2015). In these methods, the known patches, with gold standard labeling, from the atlases are used as training data and the target patch is considered as the test data. The voting weights of the known patches are implicitly implemented in the learning-and-prediction algorithm, such as the linear sparse encode using dictionary learning framework or the nonlinear kernel support vector machine method (Awate, Whitaker, 2014, Tong, Wolz, Gao, Guerrero, Hajnal, Rueckert, 2014, Tong, Wolz, Wang, Gao, Misawa, Fujiwara, Mori, Hajnal, Rueckert, 2015, Bai, Shi, Ledig, Rueckert, 2015). These approaches generally adopt the online learning scheme and a large number of atlases for training, and the performance is comparable to the state-of-the-art joint label fusion method.

To address the challenges of WHS and aforementioned limitations in current MAS research, we propose a multi-modality multi-atlas segmentation (M3AS) framework for WHS of cardiac MRI. M3AS adopts a new multi-scale patch (MSP) strategy, based on the multi-scale theory (Lindeberg, 1998), to obtain hierarchical local atlas ranking. The multi-scale space theory can handle different-level information within a limited window and has been applied to feature extraction/detection and image matching (Lindeberg, 1998, Lindeberg, 2011, Holden, Griffin, Saeed, Hill, 2004, Leutenegger, Chli, Siegwart, 2011, Wu, Kim, Sanroma, Wang, Munsell, Shen, 2015, Lowe, 2004, Wu, Liu, Huang, Guo, Jiang, Yang, Chen, Feng, 2014). We develop the MSP to compute the patches from different scale spaces to represent the different levels of structural information, with low scale capturing local fine structure and high scale suppressing fine structure but providing global structural information of the image. This is different from the conventional patch-based methods which only compute the local structural information within the patch. To avoid increasing the computational complexity, we adopt the multi-resolution implementation and couple it with the MSP where the high-scale patch can be efficiently computed using a low-resolution image space.

The main contributions of this work are summarized as follows: (1) the MSP for patch-based hierarchical local atlas ranking; (2) a framework for M3AS label fusion which evaluates both global image similarity and local pixel/patch similarity using information theoretic measures; (3) the non-binary global atlas ranking scheme which is formulated using a truncated Gaussian kernel regression model and does not require explicit atlas selection; (4) a validation study of WHS from cardiac MRI where comparisons between different patch parameterization schemes and label fusion methods are provided.

The remainder of the paper is organized as follows: Section 2 describes the methodologies of this work. Section 3 introduces the experiments and results, followed by the discussion and conclusion in Section 4.

Section snippets

Method

This section elaborates on the methodologies of this paper. First, Section 2.1 presents the framework of MAS and local weighted label fusion (LWF). Here, we propose a local similarity metric for multi-modality images on the pixel level and introduce three joint label fusion (JLF) methods which are used for comparisons in the experiments. Then, Section 2.2 describes the proposed MSP for multi-level hierarchical local atlas ranking. Finally, Section 2.3 provides the framework of the proposed M3

Materials

Twenty cardiac whole heart volumetric MRI data, provided by the Imaging Division at King’s College London, were used. The MRI sequence was the balanced steady state free precession (b-SSFP) for whole heart imaging. A 3D triggering b-SSFP turbo field echo (TFE) sequence, with arrhythmia rejection, was modified to enable the imaging at the end diastolic phase. The sequence was implemented on a 1.5T clinical scanner (Philips Healthcare, Best, The Netherlands) equipped with 32 independent receive

Discussion and conclusion

This work presents a new whole heart segmentation (WHS) method using multi-modality atlases, i.e., M3AS, for cardiac MRI. The label fusion algorithm is based on the proposed multi-scale patch (MSP) and a new global atlas ranking scheme. MSP uses the information of images from multi-scale space and thus is able to capture different levels of the structural information of images for hierarchical local atlas ranking. Also, MSP adopts the spatially varying weight (SVW) scheme to leverage the

Acknowledgment

This work was partially supported by the Chinese NSFC research fund (81301283, 11501123, 11571081), the NSFC-RS fund (81511130090), and the SRF for ROCS, SEM of China (47th).

References (48)

  • TongT. et al.

    Multiple instance learning for classification of dementia in brain MRI

    Med. Image Anal.

    (2014)
  • TongT. et al.

    Discriminative dictionary learning for abdominal multi-organ segmentation

    Med. Image Anal.

    (2015)
  • WachingerC. et al.

    Entropy and Laplacian images: structural representations for multi-modal registration

    Med. Image Anal.

    (2012)
  • WangL. et al.

    Links: learning-based multi-source integration framework for segmentation of infant brain images

    NeuroImage

    (2015)
  • WuG. et al.

    Hierarchical multi-atlas label fusion with multi-scale feature representation and label-specific patch partition

    NeuroImage

    (2015)
  • YushkevichP.A. et al.

    User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability

    Neuroimage

    (2006)
  • ZhangS. et al.

    Deformable segmentation via sparse representation and dictionary learning

    Med. Image Anal.

    (2012)
  • ArtaechevarriaX. et al.

    Combination strategies in multi-atlas image segmentation: application to brain MR data

    IEEE Trans. Med. Imaging

    (2009)
  • AwateS. et al.

    Multiatlas segmentation as nonparametric regression

    IEEE Trans. Med. Imaging

    (2014)
  • BaiW. et al.

    A probabilistic patch-based label fusion model for multi-atlas segmentation with registration refinement: application to cardiac MR images

    IEEE Trans. Med. Imaging

    (2013)
  • CoupeP. et al.

    Nonlocal patch-based label fusion for hippocampus segmentation

    Medical Image Computing and Computer-Assisted Intervention

    (2010)
  • CrumW.R. et al.

    Generalized overlap measures for evaluation and validation in medical image analysis

    IEEE Trans. Med. Imaging

    (2006)
  • HoldenM. et al.

    Multi-channel mutual information using scale space

    Medical Image Computing and Computer-Assisted Intervention

    (2004)
  • HuS. et al.

    Nonlocal regularization for active appearance model: application to medial temporal lobe segmentation

    Hum. Brain Mapp.

    (2014)
  • Cited by (405)

    View all citing articles on Scopus

    “This paper was recommended for publication by Dr. Nicholas Ayache”.

    View full text