Generative adversarial network in medical imaging: A review
Graphical abstract
Introduction
With the resurgence of deep learning in computer vision starting from 2012 (Krizhevsky et al., 2012), the adoption of deep learning methods in medical imaging has increased dramatically. It is estimated that there were over 400 papers published in 2016 and 2017 in major medical imaging related conference venues and journals (Litjens et al., 2017). The wide adoption of deep learning in the medical imaging community is due to its demonstrated potential to complement image interpretation and augment image representation and classification. In this article, we focus on one of the most interesting recent breakthroughs in the field of deep learning - generative adversarial networks (GANs) - and their potential applications in the field of medical imaging.
GANs are a special type of neural network model where two networks are trained simultaneously, with one focused on image generation and the other centered on discrimination. The adversarial training scheme has gained attention in both academia and industry due to its usefulness in counteracting domain shift, and effectiveness in generating new image samples. This model has achieved state-of-the-art performance in many image generation tasks, including text-to-image synthesis (Xu et al., 2017), super-resolution (Ledig et al., 2017), and image-to-image translation (Zhu et al., 2017).
Unlike deep learning which has its roots traced back to the 1980s (Fukushima and Miyake, 1982), the concept of adversarial training is relatively new with significant recent progress (Goodfellow et al., 2014). This paper presents a general overview of GANs, describes their promising applications in medical imaging, and identifies some remaining challenges that need to be solved to enable their successful application in other medical imaging related tasks.
To present a comprehensive overview of all relevant works on GANs in medical imaging, we searched databases including PubMed, arXiv, proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), SPIE Medical Imaging, IEEE International Symposium on Biomedical Imaging (ISBI), and International conference on Medical Imaging with Deep Learning (MIDL). We also incorporated cross referenced works not identified in the above search process. Since there are research publications coming out every month, without losing generality, we set the cut off time of the search as January 1st, 2019. Works on arXiv that report only preliminary results are excluded from this review. Descriptive statistics of these papers based on task, imaging modality and year can be found in Fig. 1.
The remainder of the paper is structured as follows. We begin with a brief introduction of the principles of GANs and some of its structural variants in Section 2. It is followed by a comprehensive review of medical image analysis tasks using GANs in Section 3 including but not limited to the fields of radiology, histopathology and dermatology. We categorize all the works according to canonical tasks: reconstruction, image synthesis, segmentation, classification, detection, registration, and others. Section 4 summarizes the review and discusses prospective applications and identifies open challenges.
Section snippets
Vanilla GAN
The vanilla GAN (Goodfellow et al., 2014) is a generative model that was designed for directly drawing samples from the desired data distribution without the need to explicitly model the underlying probability density function. It consists of two neural networks: the generator G and the discriminator D. The input to G, z is pure random noise sampled from a prior distribution p(z), which is commonly chosen to be a Gaussian or a uniform distribution for simplicity. The output of G, xg is expected
Applications in medical imaging
There are generally two ways GANs are used in medical imaging. The first is focused on the generative aspect, which can help in exploring and discovering the underlying structure of training data and learning to generate new images. This property makes GANs very promising in coping with data scarcity and patient privacy. The second focuses on the discriminative aspect, where the discriminator D can be regarded as a learned prior for normal images so that it can be used as regularizer or
Discussion
In the years 2017 and 2018, the number of studies applying GANs has risen significantly. The list of these papers reviewed for our study can be found on our1 GitHub repository.
About 46% of these papers studied image synthesis, with cross modality image synthesis being the most important application of GANs. MR is ranked as the most common imaging modality explored in the GAN related literature. We believe one of the reasons for the
Declaration of Competing Interest
We wish to confirm that there are no known conflicts of interest associated with this publication and there has been no significant financial support for this work that could have influenced its outcome.
References (258)
- et al.
Generating highly realistic images of skin lesions with GANs
OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis
(2018) - et al.
Adversarial image synthesis for unpaired multi-modal cardiac data
International Workshop on Simulation and Synthesis in Medical Imaging
(2017) - et al.
Freehand ultrasound image simulation with spatially-conditioned generative adversarial networks
Molecular Imaging, Reconstruction and Analysis of Moving Body Organs, and Stroke Imaging and Treatment
(2017) - Huang, H., Yu, P.S., Wang, C., 2018. An introduction to image synthesis with generative adversarial nets....
- et al.
Synseg-net: synthetic segmentation without target modality ground truth
IEEE Trans. Med. Imaging
(2018) - et al.
Refacing: reconstructing anonymized facial features using GANs
- et al.
Data from NSCLC-radiomics
Cancer Imaging Arch
(2015) - et al.
Generative adversarial networks for brain lesion detection
SPIE Medical Imaging
(2017) - et al.
Retinal image synthesis for cad development
International Conference Image Analysis and Recognition
(2018) - et al.
Wasserstein generative adversarial networks
Data from LIDC-IDRI
Cancer Imaging Arch
A color and texture based hierarchical k-NN approach to the classification of non-melanoma skin lesions
Color Medical Image Analysis
Standardized evaluation methodology and reference database for evaluating IVUS image segmentation
Comput. Med. Imaging Graph.
Visual feature attribution using Wasserstein GANs
Towards virtual H&E staining of hyperspectral lung histology images using conditional generative adversarial networks
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Towards virtual H&E staining of hyperspectral lung histology images using conditional generative adversarial networks
International Conference on Computer Vision
Adversarial stain transfer for histopathology image analysis
IEEE Trans. Med. Imaging
Learning implicit brain MRI manifolds with deep learning
Medical Imaging 2018: Image Processing
Synthesis of positron emission tomography (pet) images via multi-channel generative adversarial networks (GANs)
Molecular Imaging, Reconstruction and Analysis of Moving Body Organs, and Stroke Imaging and Treatment
Modelling the progression of alzheimer’s disease in MRI using generative adversarial networks
Medical Imaging 2018: Image Processing
Quickly generating representative samples from an RBM-derived process
Neural Comput.
Robust vessel segmentation in fundus images
Int. J. Biomed. Imaging
Vessel tortuosity and brain tumor malignancy: a blinded study1
Acad. Radiol.
Biomedical data augmentation using generative adversarial neural networks
International Conference on Artificial Neural Networks
Pairedcyclegan: Asymmetric style transfer for applying and removing makeup
2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Semantic-aware generative adversarial nets for unsupervised domain adaptation in chest x-ray segmentation
Efficient and accurate MRI super-resolution using a generative adversarial network and 3d multi-level densely connected network
International Conference on Medical Image Computing and Computer-Assisted Intervention
Generation of structural mr images from amyloid pet: application to mr-less quantification
J. Nucl. Med
How to fool radiologists with generative adversarial networks? A visual turing test for lung cancer diagnosis
Informed consent for medical photographs
Genet. Med.
Skin lesion analysis toward melanoma detection: achallenge at the 2017 international symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC)
Biomedical Imaging (ISBI 2018), 2018 IEEE 15th International Symposium on
Distribution matching losses can hallucinate features in medical image translation
End-to-end adversarial retinal image synthesis
IEEE Trans. Med. Imaging
Generative adversarial networks: an overview
IEEE Signal Process. Mag.
Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: First International Workshop, Brainles 2015, Held in Conjunction with MICCAI 2015, Munich, Germany, October 5, 2015, Revised Selected Papers
Towards diverse and natural image descriptions via a conditional GAN
Cited by (1189)
Deep semi-supervised learning for medical image segmentation: A review
2024, Expert Systems with ApplicationsMilitary target detection method based on EfficientDet and Generative Adversarial Network
2024, Engineering Applications of Artificial IntelligenceLinear semantic transformation for semi-supervised medical image segmentation
2024, Computers in Biology and MedicineGAN-based generation of realistic 3D volumetric data: A systematic review and taxonomy
2024, Medical Image AnalysisFeature-enhanced multi-sequence MRI-based fusion mechanism for breast tumor segmentation
2024, Biomedical Signal Processing and Control