Segmentation and classification in MRI and US fetal imaging: Recent trends and future prospects☆
Graphical abstract
Introduction
Prenatal imaging technology for fetal diagnosis has rapidly evolved (Yan (2017)). Two-dimensional ultrasound (2D US) is the primary screening modality for pregnancy evaluation because of its relative low cost, real-time imaging, lack of harmful effects to both fetus and mother, and high resolution. Three- and four -dimensional ultrasound (3D / 4D US) with additional sonography modalities such as color / power Doppler are increasingly available and have been successfully employed to detect fetal structural abnormalities (Roy-Lacroix et al. (2017a)).
Although US is still the standard tool for fetal imaging, it provides a small field-of-view, limited soft-tissue acoustic contrast, beam attenuation by adipose tissue, and poor image quality. Furthermore, important limitations arise in the context of an intrauterine environment, such as operator variability, fetal position, effects due to gestational age (poor visualization, skull ossification), and fetal tissue definition (Roy-Lacroix et al. (2017a); Reddy et al. (2008)). In cases where abnormalities cannot be adequately assessed with US, fetal magnetic resonance imaging (MRI) can be a good alternative. Early studies using MRI in the evaluation of fetal morphology were hindered due to image degradation by fetal motion on MRI standard sequences and the relatively long acquisition time. Nevertheless, with ultrafast MRI techniques such as single-shot fast-spin echo and half-Fourier acquisition turbo spin echo (HASTE), fetal motion artifacts are minimized. Dramatic reductions in image acquisition time and improved acquisition protocols for multi-planar imaging have enabled MRI to be used in fetal evaluation (Story and Rutherford (2015)). High-quality fetal images are now routinely achieved without maternal or fetal sedation. However, limitations of fetal MRI include the need of equipment availability and radiology expertise, higher cost, and longer time to perform a complete examination.
The increasing advancement and use of fetal imaging for computer-assisted prenatal organ assessment provides essential clinical feedback to design improved ad hoc surgical frameworks. A better understanding of fetal pathophysiology and early diagnosis has paved the way for the development of in utero therapies rather than postnatally (Maselli and Badillo (2016)). In this way, (3D) US and MRI provide crucial anatomic information that can be helpful in planning antenatal care and surgical procedures, as well as predicting fetal development and outcome. Fetal surgery has emerged as a multidisciplinary field capable of improving fetus outcomes for a wide range of interventions, such as laser treatment for twin-to-twin transfusion syndrome (TTTS) and open fetal surgery for spina bifida (Pratt et al. (2015); Slaghekke et al. (2014); Adzick et al. (2011)). Fetal interventions are expected to increase in breadth and prevalence as novel surgical or computational procedures demonstrate to be clinically valid (Pratt et al. (2015); Mathis et al. (2015)). As in any surgery, best results are achieved when doctors are prepared pre-operatively with a detailed understanding of the baby’s anatomy thanks to fetus-specific 3D visualizations obtained from medical images.
The pre-operative phase in fetal surgery involves the following steps (see Fig. 1) (Pratt et al. (2015)). First, images of the target structures are taken. Segmentation and classification methods are then applied to distinguish between the regions of interest and the background organs and tissues that can be discarded. The fusion of several image modalities allows completing the target structures from the different partial image views. Next, volume rendering algorithms are applied in the reconstruction stage to yield a 3D model in which the structural relationships are disclosed. In the simulation phase, surgeons benefit from this model to visualize the anatomical structures from different perspectives, and to understand their intricate relationships.
Moreover, segmentation and classification techniques can also provide imaging biomarkers for the prediction of fetal development and outcome. The shape, volume, morphometry and texture of fetal organs can be characterized during pregnancy to assess fetal health and predict complicated pregnancies (Dahdouh et al. (2018)). Early biomarkers of fetal organs disease that may impair fetal growth and well-being open up brand-new opportunities to intervene and protect vulnerable fetuses.
This review attempts to fill existing gaps in the literature. We sought to uncover both the literature in fetal imaging and to illustrate the usefulness of novel approaches to provide additional and improved diagnostic information. To the best of our knowledge, the state-of-the-art of segmentation and classification methods for fetal organs such as placenta, lungs, liver and heart, or even the whole fetus, in (3D) US and MRI have not been reviewed so far. Considering the relevance and the steady progress made in the field, we deemed it necessary to survey them in detail. For completeness, fetal brain is also covered. While several reviews of fetal brain MRI exist (see Section 3.2.1), considerably less literature can be found on fetal brain US. To our knowledge, this is the first review that addresses it.
The contents are structured as follows. Section 2 describes the search criteria for this work. Section 3 reviews the main works on fetal MRI and 2D/3D US imaging, categorized by fetal organs: the whole fetus, brain, placenta, lungs, liver, thorax, and heart. Section 4 provides examples of fetal surgery applications for which the methods reviewed in Section 3 can be employed at the pre-operative phase. Section 5 briefly overviews the validation performed on the state-of-the-art reviewed. Lastly, Section 6 discusses the most useful contributions to the field by highlighting some cutting-edge ideas.
Section snippets
Method
A systematic review on the use of segmentation and classification methods in fetal imaging has been performed. To scrutinize the literature, medical subject headings (MeSH)1 and keywords related to the scope of this review (segmentation, classification, computer vision, machine learning, fetal MRI and (3D) US imaging) were combined with medical terms associated to prenatal diagnosis. PUBMED2, EMBASE3
Categorization by fetal organ
The main challenge to automatically localize and classify the fetal organs such as brain, placenta, heart, lungs and liver, or even the whole fetus is the unpredictable fetus position and arbitrary orientation. This section gathers 123 articles that show quantitative / qualitative and robust validation of their methods, these have been assessed clinically by doctors and have undergone a peer-review process.
Examples of fetal surgery applications
There have been attempts to implement segmentation and classification algorithms for the aforementioned organs in real surgeries at pre-operative stage. In the following, examples of fetal diseases for which these scientific approaches have been applied (or could be employed in the near future) at the pre-operative level are highlighted.
General performance assessment
Table 2, Table 3, Table 4, Table 5 summarize the validation done on the 123 articles discussed in Section 3. They involve all fetal image segmentation and classification methods in terms of the fetal organs here presented, namely brain, placenta, lungs and liver (thorax), heart, and the whole fetus.
From Table 2, Table 3, Table 4, Table 5, some general comments on the state-of-art reviewed can be made.
- •
In the field of medical imaging, some simulation, phantom, and clinical studies are
Discussion and future prospects
Fetal MRI has gained wide acceptance in clinical practice as a valuable tool to complement prior prenatal (3D) US findings. Automatic segmentation and classification algorithms can nowadays facilitate the diagnosis of in utero abnormalities of fetal organs. In addition, thanks to the major breakthroughs in medical equipments and the recent advancements in the image acquisition protocols, more scalable and high-quality population reference studies can be performed. Crucial information is
Conclusion
This work provides the most recent landmarks on segmentation and classification methodologies in the context of fetal imaging. A total of 123 relevant studies in which state-of-the-art methods are used to analyze fetal structural anatomies to identify possible abnormalities or complications have been surveyed. Although fetal MRI and (3D) US are still improving, with higher resolution and enhanced signal-to-noise ratio, new processing approaches are emerging. Given the constraints that
Acknowledgements
This work was supported by CELLEX foundation and the Google Women Techmakers scholarship awarded to Jordina Torrents-Barrena. Also this work was funded by the Spanish Ministry of Economy and Competitiveness under the María de Maeztu Units of Excellence Programme [MDM-2015-0502].
References (164)
- et al.
Automated annotation and quantitative description of ultrasound videos of the fetal heart
Med. Image Anal.
(2017) - et al.
Segmentation of embryonic and fetal 3d ultrasound images based on pixel intensity distributions and shape priors
Med. Image Anal.
(2015) - et al.
Active cardiac model and its application on structure detection from early fetal ultrasound sequences
Comput. Medical Imaging Graphics
(2012) - et al.
Congenital anomalies: case definition and guidelines for data collection, analysis, and presentation of immunization safety data
Vaccine
(2016) - et al.
The ultrasonic changes in the maturing placenta and their relation to fetal pulmonic maturity
Am. J. Obst. Gynecol.
(1979) - et al.
Efficient multi-scale 3d CNN with fully connected CRF for accurate brain lesion segmentation
Med. Image Anal.
(2017) - et al.
Automated fetal brain segmentation from 2d MRI slices for motion correction
Neuroimage
(2014) - et al.
Automatic differentiation of placental perfusion compartments by time-to-peak analysis in mice
Placenta
(2015) - et al.
Comparison of septal defects in 2d and 3d echocardiography using active contour models
Comput. Med. ImagingGraph.
(2000) - et al.
Preoperative planning with magnetic resonance imaging and computerized volume rendering in twin-to-twin transfusion syndrome
Am. J. Obstetrics Gynecol.
(2001)
A review on automatic fetal and neonatal brain MRI segmentation
Neuroimage
A framework for analysis of linear ultrasound videos to detect fetal presentation and heartbeat
Med. Image Anal.
In utero perfusing fraction maps in normal and growth restricted pregnancy measured using IVIM echo-planar MRI
Placenta
Standard plane localization in ultrasound by radial component model and selective search
Ultrasound Med. Biol.
Markov random field segmentation based sonographic identification of prenatal ventricular septal defect
Procedia Computer Science - 7th International Conference on Communication, Computing and Virtualization
A randomized trial of prenatal versus postnatal repair of myelomeningocele
Eng. J. Med.
Fast fully automatic segmentation of the human placenta from motion corrupted MRI
Medical Image Computing and Computer-Assisted Intervention 2016 (MICCAI 2016)
Automatic segmentation of head structures on fetal MRI
IEEE International Symposium on Biomedical Imaging: From Nano to Macro
Automatic segmentation of antenatal 3d ultrasound images
IEEE Trans. Biomed. Eng.
Segmentation of the fetal envelope on ante-natal MRI
IEEE International Symposium on Biomedical Imaging: From Nano to Macro
Fetal responses to placental insufficiency: an update
BJOG:
Sononet: real-time detection and localisation of fetal standard scan planes in freehand ultrasound
IEEE Trans. Med. Imaging
Real-time standard scan plane detection and localisation in fetal ultrasound using fully convolutional neural networks
International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)
Automatic segmentation of the cerebellum of fetuses on 3D ultrasound images, using a 3D point distribution model
International Conference of the IEEE Engineering in Medicine and Biology Society
Three-dimensional sonographic calculation of the volume of intracranial structures in growth-restricted and appropriate-for-gestational age fetuses
Ultrasound Obstetrics Gynecol.
Toward the automatic quantification of in utero brain development in 3d structural MRI: a review
Hum. Brain Mapp.
Fetal lung maturity analysis using ultrasound image features
IEEE Trans. Inf. Technol. Biomed.
Systems biology approaches to identify developmental bases for lung diseases
Pediatr. Res.
Hybrid 3d pregnant woman and fetus modeling from medical imaging for dosimetry studies
Int. J. Comput. Assist. Radiol. Surg.
Quantitative ultrasound texture analysis of fetal lungs to predict neonatal respiratory morbidity
Ultrasound Obstetrics Gynecol.
Semantic-based indexing of fetal anatomies from 3-D ultrasound data using global / semi-local context and sequential sampling
IEEE Conference on Computer Vision and Pattern Recognition
Detection and measurement of fetal anatomies from ultrasound images using a constrained probabilistic boosting tree
IEEE Trans. Med. Imaging
Surgical planning system for twin-to-twin transfusion syndrome fetal surgery
International Conference of Computed Assisted Radiology and Surgery (CARS)
A novel surgical planning system for TTTS
Congreso Anual de la Sociedad Espaola de Bioingeniería
Fetal skull segmentation in 3D ultrasound via structured geodesic random forest
International Workshop on Fetal, Infant and Ophthalmic Medical Image Analysis (MICCAI)
Evaluation of placental maturity by the sonographic textures
Arch. Gynecol. Obstet.
Automatic fetal ultrasound standard plane detection using knowledge transferred recurrent neural networks
International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)
Standard plane localization in fetal ultrasound via domain transferred deep neural networks
IEEE J. Biomed. Health Inform.
Fetal abdominal standard plane localization through representation learning with knowledge transfer
International Workshop on Machine Learning in Medical Imaging
Segmentation and visualization of brain and lung volumes in fetal MRI using active contours and morphological operators
Int. Society for Magnetic Resonance in Medicine (ISMRM)
Variational level set algorithm in image segmentation for foetus ultrasound imaging system
Int. J. Simulation
Feasibility and reproducibility of fetal lung texture analysis by automatic quantitative ultrasound analysis and correlation with gestational age
Fetal. Diagn. Ther.
Where is my baby? a fast fetal head auto-alignment in 3D-ultrasound
IEEE International Symposium on Biomedical Imaging: From Nano to Macro
In vivo placental MRI shape and textural features predict fetal growth restriction and postnatal outcome
J. Magn. Reson. Imaging
Segmentation of fetal envelope from 3D ultrasound images based on pixel intensity statistical distribution and shape priors
IEEE 10th International Symposium on Biomedical Imaging
Automated detection of fetal cardiac structure from first-trimester ultrasound sequences
3rd International Conference on Biomedical Engineering and Informatics
Review on advanced techniques in 2-d fetal echocardiography: an image processing perspective
Adv. Med. Diagnostic Technol.
Automatic segmentation of low resolution fetal cardiac data using snake with shape priors
5th International Symposium on image and Signal Processing and Analysis
Level set snake algorithms on the fetal heart
4th IEEE International Symposium on Biomedical Imaging: From Nano to Macro
2D / 3d fetal cardiac dataset segmentation using a deformable model
Med. Phys.
Cited by (81)
Advances in the Application of Artificial Intelligence in Fetal Echocardiography
2024, Journal of the American Society of EchocardiographyAutomated classification of common maternal fetal ultrasound planes using multi-layer perceptron with deep feature integration
2023, Biomedical Signal Processing and ControlResidual attention based uncertainty-guided mean teacher model for semi-supervised breast masses segmentation in 2D ultrasonography
2023, Computerized Medical Imaging and GraphicsA review on deep-learning algorithms for fetal ultrasound-image analysis
2023, Medical Image AnalysisCitation Excerpt :Survey papers in the field have been published in the last years, even if a number of them focuses on the broader area of medical-image analysis (Liu et al., 2019) or surveys also DL algorithms for US image analysis outside the fetal field (Van Sloun et al., 2019; Akkus et al., 2019; Ouahabi and Taleb-Ahmed, 2021; Shen et al., 2021; Zaffino et al., 2020). Survey papers specifically dealing with US fetal images include: Torrents-Barrena et al. (2019), Song et al. (2021) and Sree and Vasanthanayaki (2019), where segmentation and classification algorithms are covered; Garcia-Canadilla et al. (2020) and Morris and Lopez (2021), that survey methods for fetal cardiology images; Rawat et al. (2018) and Bushra and Shobana (2021), that briefly summarize DL methods for fetal abnormality detection; and Chen et al. (2021c) and Diniz et al. (2020), that analyze research papers from a clinical perspective. An updated review that surveys the most recent work in the field of fetal US image analysis with DL could be a valuable and compact source of information for young researchers, and a reference overview document for those already working in the field.
- ☆
Conflict of interest: There are no conflicts of interest.