Skip to main content
SpringerLink
Log in
Book cover

Deep Learning in Medical Image Analysis pp 23–44Cite as

Medical Image Synthesis via Deep Learning

  • Chapter
  • First Online:

Part of the book series: Advances in Experimental Medicine and Biology ((AEMB,volume 1213))

Abstract

Medical images have been widely used in clinics, providing visual representations of under-skin tissues in human body. By applying different imaging protocols, diverse modalities of medical images with unique characteristics of visualization can be produced. Considering the cost of scanning high-quality single modality images or homogeneous multiple modalities of images, medical image synthesis methods have been extensively explored for clinical applications. Among them, deep learning approaches, especially convolutional neural networks (CNNs) and generative adversarial networks (GANs), have rapidly become dominating for medical image synthesis in recent years. In this chapter, based on a general review of the medical image synthesis methods, we will focus on introducing typical CNNs and GANs models for medical image synthesis. Especially, we will elaborate our recent work about low-dose to high-dose PET image synthesis, and cross-modality MR image synthesis, using these models.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    http://www.nap.edu/catalog/11340/health-risks-from-exposure-tolowlevels-of-ionizing-radiation.

References

  1. An L, Zhang P, Adeli E, Wang Y, Ma G, Shi F, Lalush DS, Lin W, Shen D (2016) Multi-level canonical correlation analysis for standard-dose PET image estimation. IEEE Trans Image Process 25(7):3303–3315

    Article  PubMed  PubMed Central  Google Scholar 

  2. Armanious K, Jiang C, Fischer M, Küstner T, Nikolaou K, Gatidis S, Yang B (2018) MedGAN: medical image translation using GANs. arXiv preprint. arXiv:1806.06397

    Google Scholar 

  3. Ben-Cohen A, Klang E, Raskin SP, Amitai MM, Greenspan H (2017) Virtual PET images from CT data using deep convolutional networks: initial results. In: International workshop on simulation and synthesis in medical imaging. Springer, Cham, pp 49–57

    Chapter  Google Scholar 

  4. Bi L, Kim J, Kumar A, Feng D, Fulham M (2017) Synthesis of positron emission tomography (PET) images via multi-channel generative adversarial networks (GANs). In: Molecular imaging, reconstruction and analysis of moving body organs, and stroke imaging and treatment. Springer, Cham, pp 43–51

    Chapter  Google Scholar 

  5. Bullitt E, Zeng D, Gerig G, Aylward S, Joshi S, Smith JK, Lin W, Ewend MG (2005) Vessel tortuosity and brain tumor malignancy: a blinded study1. Acad Radiol 12(10):1232–1240

    Article  PubMed  PubMed Central  Google Scholar 

  6. Bustin A, Voilliot D, Menini A, Felblinger J, de Chillou C, Burschka D, Bonnemains L, Odille F (2018) Isotropic reconstruction of MR images using 3D patch-based self-similarity learning. IEEE Trans Med Imaging 37:1932–1942

    Article  PubMed  Google Scholar 

  7. Chartsias A, Joyce T, Dharmakumar R, Tsaftaris SA (2017) Adversarial image synthesis for unpaired multi-modal cardiac data. In: International workshop on simulation and synthesis in medical imaging. Springer, Cham, pp 3–13

    Chapter  Google Scholar 

  8. Chartsias A, Joyce T, Giuffrida MV, Tsaftaris SA (2018) Multimodal MR synthesis via modality-invariant latent representation. IEEE Trans Med imaging 37(3):803–814

    Article  PubMed  Google Scholar 

  9. Chaudhari AS, Fang Z, Kogan F, Wood J, Stevens KJ, Gibbons EK, Lee JH, Gold GE, Hargreaves BA (2018) Super-resolution musculoskeletal MRI using deep learning. Magn Reson Med 80(5):2139–2154

    Article  PubMed  PubMed Central  Google Scholar 

  10. Chen H, Zhang Y, Kalra MK, Lin F, Chen Y, Liao P, Zhou J, Wang G (2017) Low-dose CT with a residual encoder-decoder convolutional neural network. IEEE Trans Med Imaging 36(12):2524–2535

    Article  PubMed  PubMed Central  Google Scholar 

  11. Chen H, Zhang Y, Zhang W, Liao P, Li K, Zhou J, Wang G (2017) Low-dose CT via convolutional neural network. Biomed Opt Express 8(2): 679–694

    Article  PubMed  PubMed Central  Google Scholar 

  12. Chen W (2007) Clinical applications of PET in brain tumors. J Nucl Med 48(9):1468–1481

    Article  PubMed  Google Scholar 

  13. Choi H, Lee DS (2018) Generation of structural MR images from amyloid PET: application to MR-less quantification. J Nucl Med 59(7):1111–1117

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Costa P, Galdran A, Meyer MI, Niemeijer M, Abràmoff M, Mendonça AM, Campilho A (2018) End-to-end adversarial retinal image synthesis. IEEE Trans Med Imaging 37(3):781–791

    Article  PubMed  Google Scholar 

  15. Dadar M, Pascoal TA, Manitsirikul S, Misquitta K, Fonov VS, Tartaglia MC, Breitner J, Rosa-Neto P, Carmichael OT, Decarli C et al (2017) Validation of a regression technique for segmentation of white matter hyperintensities in Alzheimer’s disease. IEEE Trans Med Imaging 36:1758–1768

    Article  PubMed  Google Scholar 

  16. Dar SUH, Yurt M, Karacan L, Erdem A, Erdem E, Çukur T (2018) Image synthesis in multi-contrast MRI with conditional generative adversarial networks. arXiv preprint. arXiv:1802.01221

    Google Scholar 

  17. Emami H, Dong M, Nejad-Davarani SP, Glide-Hurst CK (2018) Generating synthetic CTS from magnetic resonance images using generative adversarial networks. Med Phys 45(8):3627–3636

    Article  Google Scholar 

  18. Frangi AF, Tsaftaris SA, Prince JL (2018) Simulation and synthesis in medical imaging. IEEE Trans Med Imaging 37(3):673

    Article  PubMed  PubMed Central  Google Scholar 

  19. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680

    Google Scholar 

  20. Han X (2017) MR-based synthetic CT generation using a deep convolutional neural network method. Med Phys 44(4):1408–1419

    Article  CAS  PubMed  Google Scholar 

  21. Hiasa Y, Otake Y, Takao M, Matsuoka T, Takashima K, Carass A, Prince JL, Sugano N, Sato Y (2018) Cross-modality image synthesis from unpaired data using cycleGAN. In: International workshop on simulation and synthesis in medical imaging. Springer, Cham, pp 31–41

    Chapter  Google Scholar 

  22. Hsieh J et al (2009) Computed tomography: principles, design, artifacts, and recent advances. SPIE, Bellingham, WA

    Google Scholar 

  23. Huang Y, Shao L, Frangi AF (2017) Cross-modality image synthesis via weakly coupled and geometry co-regularized joint dictionary learning. IEEE Trans Med Imaging 37(3):815–827

    Article  Google Scholar 

  24. Huang Y, Shao L, Frangi AF (2017) Simultaneous super-resolution and cross-modality synthesis of 3D medical images using weakly-supervised joint convolutional sparse coding. arXiv preprint. arXiv:1705.02596

    Google Scholar 

  25. Isola P, Zhu JY, Zhou T, Efros AA (2016) Image-to-image translation with conditional adversarial networks. arXiv preprint. arXiv:1611.07004

    Google Scholar 

  26. Jog A, Carass A, Roy S, Pham DL, Prince JL (2017) Random forest regression for magnetic resonance image synthesis. Med Image Anal 35:475–488

    Article  PubMed  Google Scholar 

  27. Kamnitsas K, Ledig C, Newcombe VF, Simpson JP, Kane AD, Menon DK, Rueckert D, Glocker B (2017) Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Med Image Anal 36:61–78

    Article  PubMed  Google Scholar 

  28. Kang E, Min J, Ye JC (2017) A deep convolutional neural network using directional wavelets for low-dose x-ray CT reconstruction. Med Phys 44(10):e360–e375

    Article  CAS  PubMed  Google Scholar 

  29. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

    Google Scholar 

  30. Lê M, Delingette H, Kalpathy-Cramer J, Gerstner ER, Batchelor T, Unkelbach J, Ayache N (2017) Personalized radiotherapy planning based on a computational tumor growth model. IEEE Trans Med Imaging 36(3):815–825

    Article  PubMed  Google Scholar 

  31. Leynes AP, Yang J, Wiesinger F, Kaushik SS, Shanbhag DD, Seo Y, Hope TA, Larson PE (2017) Direct pseudoCT generation for pelvis PET/MRI attenuation correction using deep convolutional neural networks with multi-parametric MRI: zero echo-time and dixon deep pseudoCT (ZeDD-CT). J Nucl Med 59:852–858 (2017)

    Article  PubMed  Google Scholar 

  32. Liu F, Jang H, Kijowski R, Bradshaw T, McMillan AB (2017) Deep learning MR imaging–based attenuation correction for PET/MR imaging. Radiology 286(2):676–684

    Article  PubMed  Google Scholar 

  33. Menze BH, Jakab A, Bauer S, Kalpathy-Cramer J, Farahani K, Kirby J, Burren Y, Porz N, Slotboom J, Wiest R et al (2015) The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans Med Imaging 34(10):1993–2024

    Article  PubMed  Google Scholar 

  34. Mirza M, Osindero S (2014) Conditional generative adversarial nets. arXiv preprint. arXiv:1411.1784

    Google Scholar 

  35. Mok TC, Chung AC (2018) Learning data augmentation for brain tumor segmentation with coarse-to-fine generative adversarial networks. In: International MICCAI brainlesion workshop. Springer, Cham, pp 70–80

    Google Scholar 

  36. Nie D, Cao X, Gao Y, Wang L, Shen D (2016) Estimating CT image from MRI data using 3D fully convolutional networks. In: Deep learning and data labeling for medical applications. Springer, Cham, pp 170–178

    Chapter  Google Scholar 

  37. Nie D, Trullo R, Lian J, Wang L, Petitjean C, Ruan S, Wang Q, Shen D (2018) Medical image synthesis with deep convolutional adversarial networks. IEEE Trans Biomed Eng 65:2720–2730

    Article  PubMed  PubMed Central  Google Scholar 

  38. Olut S, Sahin YH, Demir U, Unal G (2018) Generative adversarial training for MRA image synthesis using multi-contrast MRI. In: International workshop on predictive intelligence in medicine. Springer, Cham, pp 147–154

    Chapter  Google Scholar 

  39. Peterfy C, Schneider E, Nevitt M (2008) The osteoarthritis initiative: report on the design rationale for the magnetic resonance imaging protocol for the knee. Osteoarthr Cartil 16(12):1433–1441

    Article  CAS  Google Scholar 

  40. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 234–241

    Google Scholar 

  41. Sahiner B, Pezeshk A, Hadjiiski LM, Wang X, Drukker K, Cha KH, Summers RM, Giger ML (2019) Deep learning in medical imaging and radiation therapy. Med Phys 46(1):e1–e36

    Article  PubMed  Google Scholar 

  42. Van Nguyen H, Zhou K, Vemulapalli R (2015) Cross-domain synthesis of medical images using efficient location-sensitive deep network. In: International conference on medical image computing and computer-assisted intervention. Springer, Cham. pp 677–684

    Google Scholar 

  43. van Tulder G, de Bruijne M (2015) Why does synthesized data improve multi-sequence classification? In: International conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 531–538

    Google Scholar 

  44. Wang Y, Zhang P, An L, Ma G, Kang J, Shi F, Wu X, Zhou J, Lalush DS, Lin W et al (2016) Predicting standard-dose PET image from low-dose PET and multimodal MR images using mapping-based sparse representation. Phys Med Biol 61(2):791

    Article  CAS  PubMed  Google Scholar 

  45. Wang Y, Ma G, An L, Shi F, Zhang P, Lalush DS, Wu X, Pu Y, Zhou J, Shen D (2016) Semisupervised tripled dictionary learning for standard-dose PET image prediction using low-dose PET and multimodal MRI. IEEE Trans Biomed Eng 64(3): 569–579

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  46. Wang Y, Yu B, Wang L, Zu C, Lalush DS, Lin W, Wu X, Zhou J, Shen D, Zhou L (2018) 3D conditional generative adversarial networks for high-quality PET image estimation at low dose. NeuroImage 174:550–562

    Article  PubMed  Google Scholar 

  47. Wang Y, Zhou L, Yu B, Wang L, Zu C, Lalush DS, Lin W, Wu X, Zhou J, Shen D (2019) 3D auto-context-based locality adaptive multi-modality GANs for PET synthesis. IEEE Trans Med Imaging 38(6):1328

    Article  PubMed  Google Scholar 

  48. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  PubMed  Google Scholar 

  49. Wei W, Poirion E, Bodini B, Durrleman S, Ayache N, Stankoff B, Colliot O (2018) Learning myelin content in multiple sclerosis from multimodal MRI through adversarial training. In: International conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 514–522

    Google Scholar 

  50. Welander P, Karlsson S, Eklund A (2018) Generative adversarial networks for image-to-image translation on multi-contrast MR images-a comparison of cycleGAN and unit. arXiv preprint. arXiv:1806.07777

    Google Scholar 

  51. Xiang L, Qiao Y, Nie D, An L, Lin W, Wang Q, Shen D (2017) Deep auto-context convolutional neural networks for standard-dose PET image estimation from low-dose PET/MRI. Neurocomputing 267:406–416

    Article  PubMed  PubMed Central  Google Scholar 

  52. Yang Q, Li N, Zhao Z, Fan X, Chang EI, Xu Y et al (2018) MRI cross-modality neuroimage-to-neuroimage translation. arXiv preprint. arXiv:1801.06940

    Google Scholar 

  53. Ye DH, Zikic D, Glocker B, Criminisi A, Konukoglu E (2013) Modality propagation: coherent synthesis of subject-specific scans with data-driven regularization. In: International conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 606–613

    Google Scholar 

  54. Yi X, Babyn P (2017) Sharpness-aware low dose CT denoising using conditional generative adversarial network. arXiv preprint. arXiv:1708.06453

    Google Scholar 

  55. Yi X, Walia E, Babyn P (2018) Generative adversarial network in medical imaging: a review. arXiv preprint. arXiv:1809.07294

    Google Scholar 

  56. Yu B, Zhou L, Wang L, Fripp J, Bourgeat P (2018) 3D cGAN based cross-modality MR image synthesis for brain tumor segmentation. In: 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018). IEEE, Washington, DC, pp 626–630

    Chapter  Google Scholar 

  57. Yu B, Zhou L, Wang L, Shi Y, Fripp J, Bourgeat P (2019) Ea-GANs: edge-aware generative adversarial networks for cross-modality MR image synthesis. IEEE Trans Med Imaging 38:1750–1762

    Article  PubMed  Google Scholar 

  58. Zeng K, Zheng H, Cai C, Yang Y, Zhang K, Chen Z (2018) Simultaneous single-and multi-contrast super-resolution for brain MRI images based on a convolutional neural network. Comput Biol Med 99:133–141

    Article  PubMed  Google Scholar 

  59. Zhang Z, Yang L, Zheng Y (2018) Translating and segmenting multimodal medical volumes with cycle-and shape-consistency generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 9242–9251

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computing and Information Technology, University of Wollongong, Wollongong, NSW, Australia

    Biting Yu & Lei Wang

  2. School of Computer Science, Sichuan University, Chengdu, China

    Yan Wang

  3. IDEA Lab, Department of Radiology and BRIC, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Dinggang Shen

  4. School of Electrical and Information Engineering, University of Sydney, Camperdown, NSW, Australia

    Luping Zhou

Authors
  1. Biting Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dinggang Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Luping Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luping Zhou .

Editor information

Editors and Affiliations

  1. College of Sciences & Engineering, Flinders University, Adelaide, SA, Australia

    Gobert Lee

  2. Faculty of Engineering, Gifu University, Gifu, Japan

    Hiroshi Fujita

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Yu, B., Wang, Y., Wang, L., Shen, D., Zhou, L. (2020). Medical Image Synthesis via Deep Learning. In: Lee, G., Fujita, H. (eds) Deep Learning in Medical Image Analysis . Advances in Experimental Medicine and Biology, vol 1213. Springer, Cham. https://doi.org/10.1007/978-3-030-33128-3_2

Download citation

Publish with us

Policies and ethics

Search

Navigation