Generative adversarial network in medical imaging: A review

doi:10.1016/j.media.2019.101552

Medical Image Analysis

Volume 58, December 2019, 101552

https://doi.org/10.1016/j.media.2019.101552 Get rights and content

Highlights

•
The number of publications in medical imaging using adversarial training scheme are increasing rapidly.
•
By surveying 150 published articles (including preprints), we have observed the effectiveness of adversarial training in all canonical tasks in medical image analysis.
•
Generative adversarial networks are most popular in medical image synthesis and are used for data augmentation to alleviate the data scarcity and overfitting problem.
•
Well trained discriminator can be regarded as a learned prior for the normal images so that it can be used as a regularizer.
•
Other potential applications of GANs such as image captioning, cross modality translation, automatic Magnetic resonance protocol generation have also been discussed.

Abstract

Generative adversarial networks have gained a lot of attention in the computer vision community due to their capability of data generation without explicitly modelling the probability density function. The adversarial loss brought by the discriminator provides a clever way of incorporating unlabeled samples into training and imposing higher order consistency. This has proven to be useful in many cases, such as domain adaptation, data augmentation, and image-to-image translation. These properties have attracted researchers in the medical imaging community, and we have seen rapid adoption in many traditional and novel applications, such as image reconstruction, segmentation, detection, classification, and cross-modality synthesis. Based on our observations, this trend will continue and we therefore conducted a review of recent advances in medical imaging using the adversarial training scheme with the hope of benefiting researchers interested in this technique.

Graphical abstract

Introduction

With the resurgence of deep learning in computer vision starting from 2012 (Krizhevsky et al., 2012), the adoption of deep learning methods in medical imaging has increased dramatically. It is estimated that there were over 400 papers published in 2016 and 2017 in major medical imaging related conference venues and journals (Litjens et al., 2017). The wide adoption of deep learning in the medical imaging community is due to its demonstrated potential to complement image interpretation and augment image representation and classification. In this article, we focus on one of the most interesting recent breakthroughs in the field of deep learning - generative adversarial networks (GANs) - and their potential applications in the field of medical imaging.

GANs are a special type of neural network model where two networks are trained simultaneously, with one focused on image generation and the other centered on discrimination. The adversarial training scheme has gained attention in both academia and industry due to its usefulness in counteracting domain shift, and effectiveness in generating new image samples. This model has achieved state-of-the-art performance in many image generation tasks, including text-to-image synthesis (Xu et al., 2017), super-resolution (Ledig et al., 2017), and image-to-image translation (Zhu et al., 2017).

Unlike deep learning which has its roots traced back to the 1980s (Fukushima and Miyake, 1982), the concept of adversarial training is relatively new with significant recent progress (Goodfellow et al., 2014). This paper presents a general overview of GANs, describes their promising applications in medical imaging, and identifies some remaining challenges that need to be solved to enable their successful application in other medical imaging related tasks.

To present a comprehensive overview of all relevant works on GANs in medical imaging, we searched databases including PubMed, arXiv, proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), SPIE Medical Imaging, IEEE International Symposium on Biomedical Imaging (ISBI), and International conference on Medical Imaging with Deep Learning (MIDL). We also incorporated cross referenced works not identified in the above search process. Since there are research publications coming out every month, without losing generality, we set the cut off time of the search as January 1st, 2019. Works on arXiv that report only preliminary results are excluded from this review. Descriptive statistics of these papers based on task, imaging modality and year can be found in Fig. 1.

The remainder of the paper is structured as follows. We begin with a brief introduction of the principles of GANs and some of its structural variants in Section 2. It is followed by a comprehensive review of medical image analysis tasks using GANs in Section 3 including but not limited to the fields of radiology, histopathology and dermatology. We categorize all the works according to canonical tasks: reconstruction, image synthesis, segmentation, classification, detection, registration, and others. Section 4 summarizes the review and discusses prospective applications and identifies open challenges.

Section snippets

Vanilla GAN

The vanilla GAN (Goodfellow et al., 2014) is a generative model that was designed for directly drawing samples from the desired data distribution without the need to explicitly model the underlying probability density function. It consists of two neural networks: the generator G and the discriminator D. The input to G, z is pure random noise sampled from a prior distribution p(z), which is commonly chosen to be a Gaussian or a uniform distribution for simplicity. The output of G, x_g is expected

Applications in medical imaging

There are generally two ways GANs are used in medical imaging. The first is focused on the generative aspect, which can help in exploring and discovering the underlying structure of training data and learning to generate new images. This property makes GANs very promising in coping with data scarcity and patient privacy. The second focuses on the discriminative aspect, where the discriminator D can be regarded as a learned prior for normal images so that it can be used as regularizer or

Discussion

In the years 2017 and 2018, the number of studies applying GANs has risen significantly. The list of these papers reviewed for our study can be found on our¹ GitHub repository.

About 46% of these papers studied image synthesis, with cross modality image synthesis being the most important application of GANs. MR is ranked as the most common imaging modality explored in the GAN related literature. We believe one of the reasons for the

Declaration of Competing Interest

We wish to confirm that there are no known conflicts of interest associated with this publication and there has been no significant financial support for this work that could have influenced its outcome.

References (258)

C. Baur et al.
Generating highly realistic images of skin lesions with GANs
OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis
(2018)
A. Chartsias et al.
Adversarial image synthesis for unpaired multi-modal cardiac data
International Workshop on Simulation and Synthesis in Medical Imaging
(2017)
Y. Hu et al.
Freehand ultrasound image simulation with spatially-conditioned generative adversarial networks
Molecular Imaging, Reconstruction and Analysis of Moving Body Organs, and Stroke Imaging and Treatment
(2017)
Huang, H., Yu, P.S., Wang, C., 2018. An introduction to image synthesis with generative adversarial nets....
Y. Huo et al.
Synseg-net: synthetic segmentation without target modality ground truth
IEEE Trans. Med. Imaging
(2018)
D. Abramian et al.
Refacing: reconstructing anonymized facial features using GANs
H. Aerts et al.
Data from NSCLC-radiomics
Cancer Imaging Arch
(2015)
V. Alex et al.
Generative adversarial networks for brain lesion detection
SPIE Medical Imaging
(2017)
P. Appan et al.
Retinal image synthesis for cad development
International Conference Image Analysis and Recognition
(2018)
M. Arjovsky et al.
Wasserstein generative adversarial networks

Armanious, K., Küstner, T., Nikolaou, K., Gatidis, S., Yang, B., 2018a. Retrospective correction of rigid and non-rigid...

Armanious, K., Mecky, Y., Gatidis, S., Yang, B., 2018b. Adversarial inpainting of medical image modalities....

Armanious, K., Yang, C., Fischer, M., Küstner, T., Nikolaou, K., Gatidis, S., Yang, B., 2018c. Medgan: medical image...

S.G. Armato III et al.

Data from LIDC-IDRI

Cancer Imaging Arch

(2015)

L. Ballerini et al.

A color and texture based hierarchical k-NN approach to the classification of non-melanoma skin lesions

Color Medical Image Analysis

(2013)

S. Balocco et al.

Standardized evaluation methodology and reference database for evaluating IVUS image segmentation

Comput. Med. Imaging Graph.

(2014)

C.F. Baumgartner et al.

Visual feature attribution using Wasserstein GANs

Baur, C., Albarqouni, S., Navab, N., 2018b. Melanogans: high resolution skin lesion synthesis with GANs....

N. Bayramoglu et al.

Towards virtual H&E staining of hyperspectral lung histology images using conditional generative adversarial networks

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

(2017)

N. Bayramoglu et al.

Towards virtual H&E staining of hyperspectral lung histology images using conditional generative adversarial networks

International Conference on Computer Vision

(2017)

Beers, A., Brown, J., Chang, K., Campbell, J.P., Ostmo, S., Chiang, M.F., Kalpathy-Cramer, J., 2018. High-resolution...

Ben-Cohen, A., Klang, E., Raskin, S.P., Soffer, S., Ben-Haim, S., Konen, E., Amitai, M.M., Greenspan, H., 2018....

A. Bentaieb et al.

Adversarial stain transfer for histopathology image analysis

IEEE Trans. Med. Imaging

(2018)

C. Bermudez et al.

Learning implicit brain MRI manifolds with deep learning

Medical Imaging 2018: Image Processing

(2018)

Berthelot, D., Schumm, T., Metz, L., 2017. Began: boundary equilibrium generative adversarial networks....

L. Bi et al.

Synthesis of positron emission tomography (pet) images via multi-channel generative adversarial networks (GANs)

Molecular Imaging, Reconstruction and Analysis of Moving Body Organs, and Stroke Imaging and Treatment

(2017)

Bodnar, C., 2018. Text to image synthesis using generative adversarial networks....

Bowles, C., Chen, L., Guerrero, R., Bentley, P., Gunn, R., Hammers, A., Dickie, D.A., Hernández, M.V., Wardlaw, J.,...

C. Bowles et al.

Modelling the progression of alzheimer’s disease in MRI using generative adversarial networks

Medical Imaging 2018: Image Processing

(2018)

O. Breuleux et al.

Quickly generating representative samples from an RBM-derived process

Neural Comput.

(2011)

Brock, A., Donahue, J., Simonyan, K., 2018. Large scale GAN training for high fidelity natural image synthesis....

A. Budai et al.

Robust vessel segmentation in fundus images

Int. J. Biomed. Imaging

(2013)

E. Bullitt et al.

Vessel tortuosity and brain tumor malignancy: a blinded study1

Acad. Radiol.

(2005)

F. Calimeri et al.

Biomedical data augmentation using generative adversarial neural networks

International Conference on Artificial Neural Networks

(2017)

H. Chang et al.

Pairedcyclegan: Asymmetric style transfer for applying and removing makeup

2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

(2018)

C. Chen et al.

Semantic-aware generative adversarial nets for unsupervised domain adaptation in chest x-ray segmentation

Chen, X., Konukoglu, E., 2018. Unsupervised detection of lesions in brain MRI using constrained adversarial...

Y. Chen et al.

Efficient and accurate MRI super-resolution using a generative adversarial network and 3d multi-level densely connected network

International Conference on Medical Image Computing and Computer-Assisted Intervention

(2018)

H. Choi et al.

Generation of structural mr images from amyloid pet: application to mr-less quantification

J. Nucl. Med

(2017)

M.J. Chuquicusma et al.

How to fool radiologists with generative adversarial networks? A visual turing test for lung cancer diagnosis

CIHI, 2017. Wait times for priority procedures in Canada....

D.S.o.t. Clinical Practice Committee

Informed consent for medical photographs

Genet. Med.

(2000)

N.C. Codella et al.

Skin lesion analysis toward melanoma detection: achallenge at the 2017 international symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC)

Biomedical Imaging (ISBI 2018), 2018 IEEE 15th International Symposium on

(2018)

J.P. Cohen et al.

Distribution matching losses can hallucinate features in medical image translation

Costa, P., Galdran, A., Meyer, M.I., Abràmoff, M.D., Niemeijer, M., Mendonça, A.M., Campilho, A., 2017a. Towards...

P. Costa et al.

End-to-end adversarial retinal image synthesis

IEEE Trans. Med. Imaging

(2017)

A. Creswell et al.

Generative adversarial networks: an overview

IEEE Signal Process. Mag.

(2018)

A. Crimi et al.

Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: First International Workshop, Brainles 2015, Held in Conjunction with MICCAI 2015, Munich, Germany, October 5, 2015, Revised Selected Papers

(2016)

Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V., Le, Q.V., 2018. Autoaugment: learning augmentation policies from data....

Bo Dai et al.

Towards diverse and natural image descriptions via a conditional GAN

Cited by (1189)

IMPORTANT-Net: Integrated MRI multi-parametric increment fusion generator with attention network for synthesizing absent data
2024, Information Fusion
Magnetic resonance imaging (MRI) is highly sensitive for lesion detection. Sequences obtained with different settings can capture specific characteristics of lesions. Such multi-parametric MRI information has been shown to aid radiologist performance in lesion classification, as well as improving the performance of artificial intelligence models in various tasks. However, obtaining multi-parametric MRI makes the examination costly from both financial and time perspectives, and there may also be safety concerns for special populations, thus making acquisition of the full spectrum of MRI sequences less durable. In this study, a sophisticated Integrated MRI Multi-Parametric increment fusiOn generatoR wiTh AtteNTion Network (IMPORTANT-Net) is developed to generate absent sequences/parameters. First, the parameter reconstruction module is used to encode and restore the existing MRI parameters to obtain the corresponding latent representation information at any scale level. Then the multi-parametric fusion with attention module enables the interaction of the encoded information from different parameters through a set of algorithmic strategies, and applies different weights to the information through the attention mechanism after information fusion to obtain refined representation information. Finally, a increment fusion scheme embedded in a $V$ -shape generation module is used to combine the hierarchical representations to generate specified absent MRI parameter. Results showed that our IMPORTANT-Net is capable of synthesizing absent MRI, outperforms comparable state-of-the-art networks and more importantly benefit downstream tasks. The codes are available at https://github.com/Netherlands-Cancer-Institute/MRI_IMPORTANT_NET
Deep semi-supervised learning for medical image segmentation: A review
2024, Expert Systems with Applications
Deep learning has recently demonstrated considerable promise for a variety of computer vision tasks. However, in many practical applications, large-scale labeled datasets are not available, which limits the deployment of deep learning. To address this problem, semi-supervised learning has attracted a lot of attention in the computer vision community, especially in the field of medical image analysis. This paper analyzes existing deep semi-supervised medical image segmentation studies and categories them into five main categories (i.e., pseudo-labeling, consistency regularization, GAN-based methods, contrastive learning-based methods, and hybrid methods). Afterward, we empirically analyze several representative methods by conducting experiments on two common datasets. Besides, we also point out several promising directions for future research. In summary, this paper provides a comprehensive introduction to deep semi-supervised medical image segmentation, aiming to provide a reference and comparison of methods for researchers in this field.
Military target detection method based on EfficientDet and Generative Adversarial Network
2024, Engineering Applications of Artificial Intelligence
Military target identification is one of the first tasks of modern counter-terrorism operations, and military target detection methods based on unmanned system platforms can effectively reduce personnel casualties and improve combat effectiveness. Due to the complexity and variability of the actual combat environment and the demand for real-time target recognition, this paper proposes an image recognition method based on the combination of EfficientDet and Generative Adversarial Network (GAN), in which the gauged image features extracted from the EfficientDet model are used as the input of the GAN for the game learning of image categories and features. The learning results are also used as the feature reuse input of the EfficientDet for feature learning, so that this recognition model can obtain higher recognition speed and recognition accuracy. Through model testing with embedded experiments on unmanned platforms, the results show that the network has a higher mean-Average-Precision compared to the traditional one-stage approach, while improving the recognition accuracy of targets in different complex environments while maintaining no significant reduction in Frames Per Second.
Linear semantic transformation for semi-supervised medical image segmentation
2024, Computers in Biology and Medicine
Medical image segmentation is a focus research and foundation in developing intelligent medical systems. Recently, deep learning for medical image segmentation has become a standard process and succeeded significantly, promoting the development of reconstruction, and surgical planning of disease diagnosis. However, semantic learning is often inefficient owing to the lack of supervision of feature maps, resulting in that high-quality segmentation models always rely on numerous and accurate data annotations. Learning robust semantic representation in latent spaces remains a challenge. In this paper, we propose a novel semi-supervised learning framework to learn vital attributes in medical images, which constructs generalized representation from diverse semantics to realize medical image segmentation. We first build a self-supervised learning part that achieves context recovery by reconstructing space and intensity of medical images, which conduct semantic representation for feature maps. Subsequently, we combine semantic-rich feature maps and utilize simple linear semantic transformation to convert them into image segmentation. The proposed framework was tested using five medical segmentation datasets. Quantitative assessments indicate the highest scores of our method on IXI (73.78%), ScaF (47.50%), COVID-19-Seg (50.72%), PC-Seg (65.06%), and Brain-MR (72.63%) datasets. Finally, we compared our method with the latest semi-supervised learning methods and obtained 77.15% and 75.22% DSC values, respectively, ranking first on two representative datasets. The experimental results not only proved that the proposed linear semantic transformation was effectively applied to medical image segmentation, but also presented its simplicity and ease-of-use to pursue robust segmentation in semi-supervised learning. Our code is now open at: https://github.com/QingYunA/Linear-Semantic-Transformation-for-Semi-Supervised-Medical-Image-Segmentation.
GAN-based generation of realistic 3D volumetric data: A systematic review and taxonomy
2024, Medical Image Analysis
With the massive proliferation of data-driven algorithms, such as deep learning-based approaches, the availability of high-quality data is of great interest. Volumetric data is very important in medicine, as it ranges from disease diagnoses to therapy monitoring. When the dataset is sufficient, models can be trained to help doctors with these tasks. Unfortunately, there are scenarios where large amounts of data is unavailable. For example, rare diseases and privacy issues can lead to restricted data availability. In non-medical fields, the high cost of obtaining enough high-quality data can also be a concern. A solution to these problems can be the generation of realistic synthetic data using Generative Adversarial Networks (GANs). The existence of these mechanisms is a good asset, especially in healthcare, as the data must be of good quality, realistic, and without privacy issues. Therefore, most of the publications on volumetric GANs are within the medical domain. In this review, we provide a summary of works that generate realistic volumetric synthetic data using GANs. We therefore outline GAN-based methods in these areas with common architectures, loss functions and evaluation metrics, including their advantages and disadvantages. We present a novel taxonomy, evaluations, challenges, and research opportunities to provide a holistic overview of the current state of volumetric GANs.
Feature-enhanced multi-sequence MRI-based fusion mechanism for breast tumor segmentation
2024, Biomedical Signal Processing and Control
Multi-sequence MRI plays a crucial role in the effective segmentation of breast tumors, contributing to accurate clinical diagnosis and treatment. However, the problem of missing certain sequence images may occur in clinical practice, leading to a potential impact on network performance. In medical imaging tasks, researchers are prone to ignore the correlation between different sequences, resulting in lack of expression of the extracted features. To address these problems, we propose the Feature-enhanced Multi-sequence Feature Fusion Network (FMFF-Net), which fuses information from dynamic contrast-enhanced (DCE) and diffusion-weighted imaging (DWI) sequences to improve the accuracy of breast tumor segmentation. Utilizing an attention mechanism and adversarial learning, FMFF-Net identifies potential mappings and relationships between sequences, generating features for missing sequences and enhancing feature representation. The network incorporates a High-Frequency Edge Attention Block (HFAB) to accentuate tumor edge details, leading to more precise segmentation. Tested on a dataset of 98 high-risk breast cancer MRI images, FMFF-Net demonstrated superior performance, with a Dice Similarity Coefficient (DSC) of 83.6%, Intersection over Union (IoU) of 73.4%, F1 score of 88.1%, and Sensitivity of 74.1%. Comparative analysis against mainstream segmentation methods revealed that our proposed FMFF-Net exhibits a competitive edge in the field of breast tumor segmentation.

View all citing articles on Scopus

View full text

Generative adversarial network in medical imaging: A review

Highlights

Abstract

Graphical abstract

Introduction

Section snippets

Vanilla GAN

Applications in medical imaging

Discussion

Declaration of Competing Interest

IEEE Trans. Med. Imaging

Refacing: reconstructing anonymized facial features using GANs

Data from NSCLC-radiomics

Cancer Imaging Arch

Generative adversarial networks for brain lesion detection

SPIE Medical Imaging

Retinal image synthesis for cad development

International Conference Image Analysis and Recognition

Wasserstein generative adversarial networks

Data from LIDC-IDRI

Cancer Imaging Arch

A color and texture based hierarchical k-NN approach to the classification of non-melanoma skin lesions

Color Medical Image Analysis

Standardized evaluation methodology and reference database for evaluating IVUS image segmentation

Comput. Med. Imaging Graph.

Visual feature attribution using Wasserstein GANs

Towards virtual H&E staining of hyperspectral lung histology images using conditional generative adversarial networks

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

Towards virtual H&E staining of hyperspectral lung histology images using conditional generative adversarial networks

International Conference on Computer Vision

Adversarial stain transfer for histopathology image analysis

IEEE Trans. Med. Imaging

Learning implicit brain MRI manifolds with deep learning

Medical Imaging 2018: Image Processing

Synthesis of positron emission tomography (pet) images via multi-channel generative adversarial networks (GANs)

Molecular Imaging, Reconstruction and Analysis of Moving Body Organs, and Stroke Imaging and Treatment

Modelling the progression of alzheimer’s disease in MRI using generative adversarial networks

Medical Imaging 2018: Image Processing

Quickly generating representative samples from an RBM-derived process

Neural Comput.

Robust vessel segmentation in fundus images

Int. J. Biomed. Imaging

Vessel tortuosity and brain tumor malignancy: a blinded study1

Acad. Radiol.

Biomedical data augmentation using generative adversarial neural networks

International Conference on Artificial Neural Networks

Pairedcyclegan: Asymmetric style transfer for applying and removing makeup

2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

Semantic-aware generative adversarial nets for unsupervised domain adaptation in chest x-ray segmentation

Efficient and accurate MRI super-resolution using a generative adversarial network and 3d multi-level densely connected network

International Conference on Medical Image Computing and Computer-Assisted Intervention

Generation of structural mr images from amyloid pet: application to mr-less quantification

J. Nucl. Med

How to fool radiologists with generative adversarial networks? A visual turing test for lung cancer diagnosis

Informed consent for medical photographs

Genet. Med.

Skin lesion analysis toward melanoma detection: achallenge at the 2017 international symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC)

Biomedical Imaging (ISBI 2018), 2018 IEEE 15th International Symposium on

Distribution matching losses can hallucinate features in medical image translation

End-to-end adversarial retinal image synthesis

IEEE Trans. Med. Imaging

Generative adversarial networks: an overview

IEEE Signal Process. Mag.

Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: First International Workshop, Brainles 2015, Held in Conjunction with MICCAI 2015, Munich, Germany, October 5, 2015, Revised Selected Papers

Towards diverse and natural image descriptions via a conditional GAN