Combination of Non Invasive Medical Imaging Technologies and Virtual Reality Systems to Generate Immersive Fetal 3D Visualizations

Abstract

Advances in imaging technology have led to vast improvements in fetal evaluation. Ultrasound examination is the primary method of fetal assessment because it is patient friendly, effective, and cost efficient and is considered to be safe. Magnetic resonance imaging is generally used when ultrasound cannot provide sufficiently high-quality images. It offers high-resolution fetal and placental imaging with excellent contrast. The objective here is to describe the combination of non-invasive medical imaging technologies and virtual reality systems in fetal medicine.

1 Introduction

The purpose of this work is to present a study related to the combination of non-invasive medical imaging technologies and Virtual Reality (VR) immersive technologies as a complementary tool to assist fetal medicine studies (Fig. 1).

Advances in image-scanning technology have led to vast improvements in medicine, especially in the diagnosis of fetal anomalies. In general, two main technologies can be used to visualize and, as a result, obtain inner images of the maternal body during pregnancy: Ultrasonography (USG) and Magnetic Resonance Imaging (MRI) (Fig. 2).

Fig. 1.

Patient ready to start the magnetic resonance imaging scan and a radiologist analysing the MRI files during the scanning process.

Fig. 2.

3D model of the fetus generated from MRI DICOM (Digital Communication in Medicine protocol) files.

Apart from the technical differences, an interesting obvious difference between the technologies is related to the portability of USG equipment compared to MRI, which is carried out inside a tubular format, making it necessary for the patient to be positioned inside the equipment. The portability of some USG apparatus makes it possible for the equipment to travel to the patient, while for MRI scans the patient has to go to the equipment (Fig. 3).

Fig. 3.

Screen of the software “Unreal Engine” from Epic Games, the 3D model generated from the slices obtained on the non-invasive image technology to then apply the texture and lights and create the paths for the fly-through VR navigation.

Fig. 4.

Sequence of the virtual visualization inside a fetal heart at 38 weeks, after autopsy, from Micro-CT scanner (Zeiss Xradia Versa 510), and 3D printed full scale physical models (Projet 3510 HD Plus)

Computed tomography (CT) is used only in specific cases of suspected fetal malformation, particularly those related to the skeleton, because of potential risks associated with exposure of the fetus to radiation. Its use during pregnancy must be adequately justified and its application is limited to specific pathologies such as bone dysplasia, which can, in some cases, be difficult to diagnose by USG, especially in the absence of a family history of the disease. CT can also be used (after the birth, due to the X Ray radiation) and its physical principles are based on the amount of radiation absorbed by each body part, which means that tissues with different composition absorb X-rays in different ways. Also, Micro Computed Tomography (MicroCT) which has a higher image resolution can be used in cases of autopsy analysis (Fig. 4).

USG is the primary fetal monitoring method during pregnancy, and also the most commonly-used method, given its long record of safety, usefulness, and cost-effectiveness. The development of USG scanning during the 1960s opened a new window into the study of the fetus. Its applications are based on the detection and representation of acoustic waves reflected by interfaces within the body, providing the information needed to generate greyscale images of the uterine content (Fig. 5).

Fig. 5.

Pictures taken in September 2015 during the “Hospital Innovation Show” in São Paulo, Brasil where the project was presented for many physicians.

Fig. 6.

Two fathers having the opportunity to participate on the experiment visualizing their unborn babies during the pregnancy and the fetal medicine specialist Dr. Heron Werner during the equipment tests.

MRI is a non-invasive method that has been used in obstetrics since the 1980s. It offers high-resolution fetal images with excellent contrast that allow visualization of internal tissues. When USG yields unexpected results, MRI is generally used, because it provides additional information about fetal abnormalities and conditions for which USG cannot provide high-quality images. MRI files can generate detailed characteristics of the soft tissues of the fetus body as the face, hands or feet as well internal body structures as aerial paths [2].

2 Methodology

The construction process of the 3D accurate virtual model starts with the 3D modeling volume built from the obtained slices sequentially grouped, followed by the segmentation process where the Physician selects the important body parts to be visualized that will be then accurately reconstructed in 3D (Fig. 6).

Fig. 7.

3D view obtained from MRI of fetal cervical tumor at 37 gestational weeks of gestation.

Fig. 8.

Virtual fetoscopy highlights area of placenta.

Fig. 9.

Virtual fetoscopy after MRI highlights airway path in a fetus (28 weeks) with Diaphragmatic Hernia. In this case, MRI was performed after fetal surgery. The arrow shows the red structure corresponding to Balloon inflated inside the trachea. Note the presence of the stomach inside the thorax (*).

Having the accurate 3D model (womb, umbilical, cord, placenta and fetus) the final stage is the programming of the virtual reality (VR) for different devices as Oculus Rift DK2 and Samsung Gear VR. Through the use of the software “Unreal Engine” from Epic Games, the 3D model generated from the slices obtained on the non-invasive image technology as MRI, to then apply the texture and lights and create the paths for the fly-through VR navigation. According to Riva and Davide, VR can be used to explore the organs by “flying” around, behind, or even inside them. In this sense virtual environments can be used both as didactic and experiential educational tools, allowing a deeper understanding of the interrelationship of anatomical structures that cannot be achieved by any other means [3] (Fig. 7).

The third step is the definition of the Device. For Oculus Rift DK2, Unreal Engine have a plug-in that prepare the project for Oculus and we compile in one executable file “*.exe”. For Samsung Gear VR, the third step is render the project in a panoramic video that will be convert in 360º video on the device (this conversion is necessary because mobiles phones have limited graphic hardware). In this engine, we can include also the heartbeat sounds of the fetus to improve the immersive sensation (Fig. 8).

Important to observe that the navigation through internal paths should be pre-defined by the physician responsible for the patient in order to highlight the main subjects to be studied by the fetal medicine team as well for parents understanding. So far seven complete different studies have been developed to attend physicians in Fetal Medicine and Cardiology and the project was presented on September 2015 during the “Hospital Innovation Show” in São Paulo, Brasil (Fig. 9).

3 Conclusion

Virtual reality fetal 3D models based on non-invasive medical imaging technologies were successfully generated. They were remarkably similar to the postnatal appearance of the newborn baby, especially in cases with pathology, increasing the possibilities of digital tools to help fetal medicine researches. The 3D fetal models applied on virtual reality immersive technologies may improve our understanding of fetal anatomical characteristics, and can be used for educational purposes and as a method for parents to visualize their unborn baby.