Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images

doi:10.1016/j.media.2019.101563

Medical Image Analysis

Volume 58, December 2019, 101563

https://doi.org/10.1016/j.media.2019.101563 Get rights and content

Highlights

•
A network targeted at simultaneous segmentation and classification of nuclei.
•
Introduce horizontal and vertical distance maps to separate clustered nuclei.
•
An interpretable evaluation framework that quantifies nuclear segmentation.
•
A new dataset of 24,319 exhaustively annotated nuclei with associated class labels.

Abstract

Nuclear segmentation and classification within Haematoxylin & Eosin stained histology images is a fundamental prerequisite in the digital pathology work-flow. The development of automated methods for nuclear segmentation and classification enables the quantitative analysis of tens of thousands of nuclei within a whole-slide pathology image, opening up possibilities of further analysis of large-scale nuclear morphometry. However, automated nuclear segmentation and classification is faced with a major challenge in that there are several different types of nuclei, some of them exhibiting large intra-class variability such as the nuclei of tumour cells. Additionally, some of the nuclei are often clustered together. To address these challenges, we present a novel convolutional neural network for simultaneous nuclear segmentation and classification that leverages the instance-rich information encoded within the vertical and horizontal distances of nuclear pixels to their centres of mass. These distances are then utilised to separate clustered nuclei, resulting in an accurate segmentation, particularly in areas with overlapping instances. Then, for each segmented instance the network predicts the type of nucleus via a devoted up-sampling branch. We demonstrate state-of-the-art performance compared to other methods on multiple independent multi-tissue histology image datasets. As part of this work, we introduce a new dataset of Haematoxylin & Eosin stained colorectal adenocarcinoma image tiles, containing 24,319 exhaustively annotated nuclei with associated class labels.

Graphical abstract

Introduction

Current manual assessment of Haematoxylin and Eosin (H&E) stained histology slides suffers from low throughput and is naturally prone to intra- and inter-observer variability (Elmore et al., 2015). To overcome the difficulty in visual assessment of tissue slides, there is a growing interest in digital pathology (DP), where digitised whole-slide images (WSIs) are acquired from glass histology slides using a scanning device. This permits efficient processing, analysis and management of the tissue specimens (Madabhushi and Lee, 2016). Each WSI contains tens of thousands of nuclei of various types, which can be further analysed in a systematic manner and used for predicting clinical outcome. Here, the type of nucleus refers to the cell type in which it is located. For example, nuclear features can be used to predict survival (Alsubaie et al., 2018) and also for diagnosing the grade and type of disease (Lu et al., 2018). Also, efficient and accurate detection and segmentation of nuclei can facilitate good quality tissue segmentation (Sirinukunwattana, Snead, Epstein, Aftab, Mujeeb, Tsang, Cree, Rajpoot, 2018, Javed, Fraz, Epstein, Snead, Rajpoot, 2018), which can in turn not only facilitate the quantification of WSIs but may also serve as an important step in understanding how each tissue component contributes to disease. In order to use nuclear features for downstream analysis within computational pathology, nuclear segmentation must be carried out as an initial step. However, this remains a challenge because nuclei display a high level of heterogeneity and there is significant inter- and intra-instance variability in the shape, size and chromatin pattern between and within different cell types, disease types or even from one region to another within a single tissue sample. Tumour nuclei, in particular, tend to be present in clusters, which gives rise to many overlapping instances, providing a further challenge for automated segmentation, due to the difficulty of separating neighbouring instances.

As well as extracting each individual nucleus, determining the type of each nucleus can increase the diagnostic potential of current DP pipelines. For example, accurately classifying each nucleus to be from tumour or lymphocyte enables downstream analysis of tumour infiltrating lymphocytes (TILs), which have been shown to be predictive of cancer recurrence (Corredor et al., 2019). Yet, similar to nuclear segmentation, classifying the type of each nucleus is difficult, due to the high variance of nuclear appearance within each WSI. Typically, nuclei are classified using two disjoint models: one for detecting each nucleus and then another for performing nuclear classification (Sharma, Zerbe, Heim, Wienert, Behrens, Hellwich, Hufnagl, 2015, Wang, Hu, Li, Liu, Zhu, 2016). However, it would be preferable to utilise a single unified model for nuclear instance segmentation and classification.

In this paper, we present a deep learning approach³ for simultaneous segmentation and classification of nuclear instances in histology images. The network is based on the prediction of horizontal and vertical distances (and hence the name HoVer-Net) of nuclear pixels to their centres of mass, which are subsequently leveraged to separate clustered nuclei. For each segmented instance, the nuclear type is subsequently determined via a dedicated up-sampling branch. To the best of our knowledge, this is the first approach that achieves instance segmentation and classification within the same network. We present comparative results on six independent multi-tissue histology image datasets and demonstrate state-of-the-art performance compared to other recently proposed methods. The main contributions of this work are listed as follows:

•
A novel network, targeted at simultaneous segmentation and classification of nuclei, where horizontal and vertical distance map predictions separate clustered nuclei.
•
We show that the proposed HoVer-Net achieves state-of-the-art performance on multiple H&E histology image datasets, as compared to over a dozen recently published methods.
•
An interpretable and reliable evaluation framework that effectively quantifies nuclear segmentation performance and overcomes the limitations of existing performance measures.
•
A new dataset⁴ of 24,319 exhaustively annotated nuclei within 41 colorectal adenocarcinoma image tiles.

Section snippets

Nuclear instance segmentation

Within the current literature, energy-based methods, in particular the watershed algorithm, have been widely utilised to segment nuclear instances. For example, Yang et al. (2006) used thresholding to obtain the markers and the energy landscape as input for watershed to extract the nuclear instances. Nonetheless, thresholding relies on a consistent difference in intensity between the nuclei and background, which does not hold for more complex images and hence often produces unreliable results.

Methods

Our overall framework for automatic nuclear instance segmentation and classification can be observed in Fig. 1 and the proposed network in Fig. 2. Here, nuclear pixels are first detected and then, a tailored post-processing pipeline is used to simultaneously segment nuclear instances and obtain the corresponding nuclear types. The framework is based upon the horizontal and vertical distance maps, which can be seen in Fig. 3. In the figure, each nuclear pixel denotes either the horizontal or

Nuclear instance segmentation evaluation

Assessment and comparison of different methods is usually given by an overall score that indicates which method is superior. However, to further investigate the method, it is preferable to break the problem into sub-tasks and measure the performance of the method on each sub-task. This enables an in depth analysis, thus facilitating a comprehensive understanding of the approach, which can help drive forward model development. For nuclear instance segmentation, the problem can be divided into

Datasets

As part of this work, we introduce a new dataset that we term as the colorectal nuclear segmentation and phenotypes (CoNSeP) dataset,⁶ consisting of 41 H&E stained image tiles, each of size 1000  ×  1000 pixels at 40 ×  objective magnification. Images were extracted from 16 colorectal adenocarcinoma (CRA) WSIs, each belonging to an individual patient, and scanned with an Omnyx VL120 scanner within the department

Discussion and conclusions

Analysis of nuclei in large-scale histopathology images is an important step towards automated downstream analysis for diagnosis and prognosis of cancer. Nuclear features have been often used to assess the degree of malignancy (Gurcan et al., 2009). However, visual analysis of nuclei is a very time consuming task because there are often tens of thousands of nuclei within a given whole-slide image (WSI). Performing simultaneous nuclear instance segmentation and classification enables subsequent

Declaration of Competing Interest

The authors confirm that there are no conflicts of interest.

Acknowledgements

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) (No. 2016R1C1B2012433) and by the Ministry of Science and ICT (MSIT) (No. 2018K1A3A1A74065728). We also acknowledge the financial support from EPSRC and MRC, provided as part of the Mathematics for Real-World Systems CDT. We thank Peter Naylor for his assistance in the implementation of the DIST network.

References (45)

A. LaTorre et al.
Segmentation of neuronal nuclei based on clump splitting and a two-step binarization of images
Expert Syst. Appl.
(2013)
G. Litjens et al.
A survey on deep learning in medical image analysis
Med. Image Anal.
(2017)
C. Lu et al.
Nuclear shape and orientation features from h&e images predict survival in early-stage estrogen receptor-positive breast cancers
Lab. Investig.
(2018)
P. Naylor et al.
Segmentation of nuclei in histopathology images by deep regression of the distance map
IEEE Trans. Med. Imaging
(2018)
K. Nguyen et al.
Prostate cancer detection: fusion of cytological and textural features
J. Pathol. Inform.
(2011)
Y. Yuan et al.
Quantitative image analysis of cellular heterogeneity in breast tumors complements genomic profiling
Sci. Transl. Med.
(2012)
M. Abadi et al.
Tensorflow: a system for large-scale machine learning.
OSDI
(2016)
S. Ali et al.
An integrated region-, boundary-, shape-based active contour for multiple object overlap resolution in histological imagery
IEEE Trans. Med. Imaging
(2012)
N. Alsubaie et al.
A bottom-up approach for tumour differentiation in whole slide images of lung adenocarcinoma
Medical Imaging 2018: Digital Pathology
(2018)
A. Arnab et al.
On the robustness of semantic segmentation models to adversarial attacks
CoRR
(2017)

V. Badrinarayanan et al.

Segnet: a deep convolutional encoder-decoder architecture for image segmentation

IEEE Trans. Pattern Anal. Mach. Intell.

(2017)

P. Bankhead et al.

Qupath: open source software for digital pathology image analysis

Sci. Rep.

(2017)

A.E. Carpenter et al.

Cellprofiler: image analysis software for identifying and quantifying cell phenotypes

Genome Biol.

(2006)

H. Chen et al.

Dcan: deep contour-aware networks for accurate gland segmentation

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

(2016)

J. Cheng et al.

Segmentation of clustered nuclei with shape markers and marking function

IEEE Trans. Biomed. Eng.

(2009)

G. Corredor et al.

Spatial architecture and arrangement of tumor-infiltrating lymphocytes for predicting likelihood of recurrence in early-stage non–small cell lung cancer

Clin. Cancer Res.

(2019)

Cui, Y., Zhang, G., Liu, Z., Xiong, Z., Hu, J., 2018. A deep learning algorithm for one-step contour aware nuclei...

J. Deng et al.

ImageNet: a large-scale hierarchical image database

CVPR09

(2009)

J.G. Elmore et al.

Diagnostic concordance among pathologists interpreting breast biopsy specimens

JAMA

(2015)

S. Graham et al.

Sams-net: stain-aware multi-scale network for instance-based nuclei segmentation in histology images

Biomedical Imaging (ISBI 2018), 2018 IEEE 15th International Symposium on

(2018)

M.N. Gurcan et al.

Histopathological image analysis: a review

IEEE reviews in biomedical engineering

(2009)

He, K., Gkioxari, G., Dollár, P., Girshick, R., 2017. Mask R-CNN....

Cited by (738)

Computational pathology: A survey review and the way forward
2024, Journal of Pathology Informatics
Computational Pathology (CPath) is an interdisciplinary science that augments developments of computational approaches to analyze and model medical histopathology images. The main objective for CPath is to develop infrastructure and workflows of digital diagnostics as an assistive CAD system for clinical pathology, facilitating transformational changes in the diagnosis and treatment of cancer that are mainly address by CPath tools. With evergrowing developments in deep learning and computer vision algorithms, and the ease of the data flow from digital pathology, currently CPath is witnessing a paradigm shift. Despite the sheer volume of engineering and scientific works being introduced for cancer image analysis, there is still a considerable gap of adopting and integrating these algorithms in clinical practice. This raises a significant question regarding the direction and trends that are undertaken in CPath. In this article we provide a comprehensive review of more than 800 papers to address the challenges faced in problem design all-the-way to the application and implementation viewpoints. We have catalogued each paper into a model-card by examining the key works and challenges faced to layout the current landscape in CPath. We hope this helps the community to locate relevant works and facilitate understanding of the field’s future directions. In a nutshell, we oversee the CPath developments in cycle of stages which are required to be cohesively linked together to address the challenges associated with such multidisciplinary science. We overview this cycle from different perspectives of data-centric, model-centric, and application-centric problems. We finally sketch remaining challenges and provide directions for future technical developments and clinical integration of CPath. For updated information on this survey review paper and accessing to the original model cards repository, please refer to GitHub. Updated version of this draft can also be found from arXiv.
Publicly available datasets of breast histopathology H&E whole-slide images: A scoping review
2024, Journal of Pathology Informatics
Advancements in digital pathology and computing resources have made a significant impact in the field of computational pathology for breast cancer diagnosis and treatment. However, access to high-quality labeled histopathological images of breast cancer is a big challenge that limits the development of accurate and robust deep learning models. In this scoping review, we identified the publicly available datasets of breast H&E-stained whole-slide images (WSIs) that can be used to develop deep learning algorithms. We systematically searched 9 scientific literature databases and 9 research data repositories and found 17 publicly available datasets containing 10 385 H&E WSIs of breast cancer. Moreover, we reported image metadata and characteristics for each dataset to assist researchers in selecting proper datasets for specific tasks in breast cancer computational pathology. In addition, we compiled 2 lists of breast H&E patches and private datasets as supplementary resources for researchers. Notably, only 28% of the included articles utilized multiple datasets, and only 14% used an external validation set, suggesting that the performance of other developed models may be susceptible to overestimation. The TCGA-BRCA was used in 52% of the selected studies. This dataset has a considerable selection bias that can impact the robustness and generalizability of the trained algorithms. There is also a lack of consistent metadata reporting of breast WSI datasets that can be an issue in developing accurate deep learning models, indicating the necessity of establishing explicit guidelines for documenting breast WSI dataset characteristics and metadata.
Improving generalization capability of deep learning-based nuclei instance segmentation by non-deterministic train time and deterministic test time stain normalization
2024, Computational and Structural Biotechnology Journal
With the advent of digital pathology and microscopic systems that can scan and save whole slide histological images automatically, there is a growing trend to use computerized methods to analyze acquired images. Among different histopathological image analysis tasks, nuclei instance segmentation plays a fundamental role in a wide range of clinical and research applications. While many semi- and fully-automatic computerized methods have been proposed for nuclei instance segmentation, deep learning (DL)-based approaches have been shown to deliver the best performances. However, the performance of such approaches usually degrades when tested on unseen datasets.
In this work, we propose a novel method to improve the generalization capability of a DL-based automatic segmentation approach. Besides utilizing one of the state-of-the-art DL-based models as a baseline, our method incorporates non-deterministic train time and deterministic test time stain normalization, and ensembling to boost the segmentation performance. We trained the model with one single training set and evaluated its segmentation performance on seven test datasets. Our results show that the proposed method provides up to 4.9%, 5.4%, and 5.9% better average performance in segmenting nuclei based on Dice score, aggregated Jaccard index, and panoptic quality score, respectively, compared to the baseline segmentation model.
BUA-Net: Boundary and uncertainty-aware attention network for lumbar multi-region magnetic resonance imaging segmentation
2024, Biomedical Signal Processing and Control
Multi-region segmentation by axial lumbar magnetic resonance imaging (MRI) is important for the diagnosis of lumbar spinal stenosis (LSS) and lumbar disc herniation (LDH) disease. Deep learning-based methods have been widely used in this field, but the segmentation task is still challenging due to intra-class grayscale unevenness, inter-class low contrast and unclear boundaries.
To solve the above problems, we propose a novel boundary and uncertainty-aware attention network (BUA-Net). It is structured around a single encoder and two decoders, each dedicated to either multi-region or boundary segmentation. We use an uncertainty-aware attention (UA) module to extract regions with large segmentation uncertainty and generate an attention mask. The learning and supervisory capabilities of the model are enhanced by a fine seg (FS) module and an attention supervision loss (ASL) in the boundaries and blurred regions. Moreover, a multi-scale spatial channel attention (MSCA) module is proposed that refines the high-level features without increasing any layers.
The results of the BUA-Net experiments were tested against seven networks and showed a 0.26% improvement in Dice scores, a 0.86% improvement in precision, and a 0.2% improvement in recall. In metrics related to boundary segmentation effectiveness, our method nearly matches the state-of-the-art in the ASD metric, surpasses it by 2.32 in the 95HD metric, and leads by 2.04, 2.68, and 1.22 in the IVD, TS, and PE categories, respectively.
The experimental results demonstrate that our proposed method outperforms other state-of-the-art methods and shows excellent segmentation ability in the boundaries and blurred regions.
Simultaneously segmenting and classifying cell nuclei by using multi-task learning in multiplex immunohistochemical tissue microarray sections
2024, Biomedical Signal Processing and Control
Quantitative analysis of tumor immune microenvironment (TIME) in immunohistochemical (IHC) tissue microarray (TMA) sections is crucial in diagnosis and treatment recommendations for cancer patients. Nuclei segmentation and classification are the prerequisites for the TIME quantification, but it still lacks of robust nuclear quantification models used for IHC histological slides. In this paper, we design an approach for simultaneously segmenting and classifying cell nuclei in multiplex IHC TMA sections. The large TMA tissue core is first divided into a set of small overlapping patches, where cell nuclei are then simultaneously segmented and classified by using our multi-task learning model. The model has one feature encoder with cascaded separable-ResUnit blocks, and three decoder branches that incorporate the Self-Attention modules and DenseUnit blocks to perform nuclear segmentation, classification and distance map regression, respectively. After processing all patches, the weighted loss map and vote mechanism are applied to seamlessly stitch patch-level predictions to form the tissue core level results. We finally exploit generalized Laplacian of Gaussian (gLoG) filters based algorithm to post-process segmentation results to further split overlapping cell nuclei. Quantitative evaluations have been performed on a IHC stained histological image dataset with 9725 manually identified cell nuclei and a public H&E stained dataset (CoNSep), which show that our model outperforms state-of-the-art nuclei segmentation and classification models. The qualitative evaluations on TMA sections show the potential of using our approach in clinical applications.
Multi-task global optimization-based method for vascular landmark detection
2024, Computerized Medical Imaging and Graphics
Vascular landmark detection plays an important role in medical analysis and clinical treatment. However, due to the complex topology and similar local appearance around landmarks, the popular heatmap regression based methods always suffer from the landmark confusion problem. Vascular landmarks are connected by vascular segments and have special spatial correlations, which can be utilized for performance improvement. In this paper, we propose a multi-task global optimization-based framework for accurate and automatic vascular landmark detection. A multi-task deep learning network is exploited to accomplish landmark heatmap regression, vascular semantic segmentation, and orientation field regression simultaneously. The two auxiliary objectives are highly correlated with the heatmap regression task and help the network incorporate the structural prior knowledge. During inference, instead of performing a max-voting strategy, we propose a global optimization-based post-processing method for final landmark decision. The spatial relationships between neighboring landmarks are utilized explicitly to tackle the landmark confusion problem. We evaluated our method on a cerebral MRA dataset with 564 volumes, a cerebral CTA dataset with 510 volumes, and an aorta CTA dataset with 50 volumes. The experiments demonstrate that the proposed method is effective for vascular landmark localization and achieves state-of-the-art performance.

View all citing articles on Scopus

¹: First authors contributed equally

²: Last authors contributed equally

View full text

Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images

Highlights

Abstract

Graphical abstract

Introduction

Section snippets

Nuclear instance segmentation

Methods

Nuclear instance segmentation evaluation

Datasets

Discussion and conclusions

Declaration of Competing Interest

Acknowledgements

Expert Syst. Appl.

Med. Image Anal.

Lab. Investig.

IEEE Trans. Med. Imaging

J. Pathol. Inform.

Sci. Transl. Med.

Tensorflow: a system for large-scale machine learning.

OSDI

An integrated region-, boundary-, shape-based active contour for multiple object overlap resolution in histological imagery

IEEE Trans. Med. Imaging

A bottom-up approach for tumour differentiation in whole slide images of lung adenocarcinoma

Medical Imaging 2018: Digital Pathology

On the robustness of semantic segmentation models to adversarial attacks

CoRR

Segnet: a deep convolutional encoder-decoder architecture for image segmentation

IEEE Trans. Pattern Anal. Mach. Intell.

Qupath: open source software for digital pathology image analysis

Sci. Rep.

Cellprofiler: image analysis software for identifying and quantifying cell phenotypes

Genome Biol.

Dcan: deep contour-aware networks for accurate gland segmentation

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

Segmentation of clustered nuclei with shape markers and marking function

IEEE Trans. Biomed. Eng.

Spatial architecture and arrangement of tumor-infiltrating lymphocytes for predicting likelihood of recurrence in early-stage non–small cell lung cancer

Clin. Cancer Res.

ImageNet: a large-scale hierarchical image database

CVPR09

Diagnostic concordance among pathologists interpreting breast biopsy specimens

JAMA

Sams-net: stain-aware multi-scale network for instance-based nuclei segmentation in histology images

Biomedical Imaging (ISBI 2018), 2018 IEEE 15th International Symposium on

Histopathological image analysis: a review

IEEE reviews in biomedical engineering