Abstract
Recent technological innovations have created new opportunities for the increased adoption of virtual reality (VR) and augmented reality (AR) applications in medicine. While medical applications of VR have historically seen greater adoption from patient-as-user applications, the new era of VR/AR technology has created the conditions for wider adoption of clinician-as-user applications. Historically, adoption to clinical use has been limited in part by the ability of the technology to achieve a sufficient quality of experience. This article reviews the definitions of virtual and augmented reality and briefly covers the history of their development. Currently available options for consumer-level virtual and augmented reality systems are presented, along with a discussion of technical considerations for their adoption in the clinical environment. Finally, a brief review of the literature of medical VR/AR applications is presented prior to introducing a comprehensive conceptual framework for the viewing and manipulation of medical images in virtual and augmented reality. Using this framework, we outline considerations for placing these methods directly into a radiology-based workflow and show how it can be applied to a variety of clinical scenarios.
Similar content being viewed by others
References
Sutherland J, La Russa D: Virtual reality. In: Rybicki FJ, Grant GT Eds. 3D Printing in Medicine: A Practical Guide for Medical Professionals. Cham: Springer International Publishing, 2017, pp. 125–133
Satava RM: Virtual reality surgical simulator. Surg Endosc 7:203–205, 1993
Phillips JR: Virtual reality: A new vista for nurse researchers? Nurs Sci Q 6:5–7, 1993
Chinnock C: Virtual reality in surgery and medicine. Hosp Technol Ser 13:1–48, 1994
Rosen JM, Soltanian H, Redett RJ, Laub DR: Evolution of virtual reality. IEEE Eng Med Biol Mag 15:16–22, 1996
Pensieri C, Pennacchini M, Sivan Y, Gefen D, Boulos MK, Claudio P, Maddalena P: Overview: Virtual reality in medicine. J Virtual Worlds Res 7:1–34, 2014
Worldwide Spending on Augmented and Virtual Reality. Available at https://www.idc.com/getdoc.jsp?containerId=prUS42959717. Accessed January 2018.
Healthcare Augmented & Virtual Reality Market Worth $5.1 Billion By 2025. Available at https://www.grandviewresearch.com/press-release/global-augmented-reality-ar-virtual-reality-vr-in-healthcare-market. Accessed January 2018.
Rybicki FJ, Grant GT: 3D Printing in Medicine. Cham: Springer International Publishing, 2017
Burdea GC, Coiffet P: Virtual Reality Technology. New York: Wiley, 2003
Heilig ML: Stereoscopic-television apparatus for individual use. US Patent 2:955–156, 1960
Linte CA, Davenport KP, Cleary K, Peters C, Vosburgh KG, Navab N, Eddie P, Jannin P, Peters TM, Holmes DR, Robb RA: On mixed reality environments for minimally invasive therapy guidance: Systems architecture, successes and challenges in their implementation from laboratory to clinic. Comput Med Imaging Graph 37:83–97, 2013
Yaniv Z, Linte CA: Applications of augmented reality in the operating room. Fundam Wearable Comput Augment Real 485–510, 2016
Moro C, Štromberga Z, Raikos A, Stirling A: The effectiveness of virtual and augmented reality in health sciences and medical anatomy. Anat Sci Educ 1–11, 2017
Plasencia DM: One step beyond virtual reality: Connecting past and future developments. Crossroads ACM Mag Stud 22:18–23, 2015
Abrash M: Why You Won’t See Hard AR Anytime Soon. Available at http://blogs.valvesoftware.com/abrash/why-you-wont-see-hard-ar-anytime-soon/. Accessed May 2017
Milgram P, Takemura H, Utsumi A, Kishino F: Augmented reality: A class of display on the reality-virtuality continuum. Proc SPIE 2351:282–292, 1995
Milgram P, Kishino F: A taxonomy of mixed reality visual displays. IEICE Trans Inf Syst 77:1–15, 1994
Foxlin E, Harrington M, Pfeifer G: Constellation: A Wide-Range Wireless Motion-tracking System for Augmented Reality and Virtual Set Applications. Proc. 25th Annu. Conf. Comput. Graph. Interact. Tech. - SIGGRAPH ‘98 98, pp. 371–378, 1998
Valve: SteamVR Tracking. Available at https://partner.steamgames.com/vrtracking. Accessed May 2017
Valve Reveals Timeline of Vive Prototypes, We Chart it For You. Available at http://www.roadtovr.com/valve-reveals-timeline-of-vive-prototypes-we-chart-it-for-you/. Accessed May 2017
Tazartes D: An historical perspective on inertial navigation systems. In: Proc. Int. Symp. Inertial Sensors Syst, pp. 1–5, 2014
Riva G, Wiederhold BK: Gaggi: Being different: The transformative potential of virtual reality. Annu Rev Cybern Therapy Telemed 14:3–6, 2016
Tepper OM, Rudy HL, Lefkowitz A, Weimer KA, Marks SM, Stern CS, Garfein ES: Mixed reality with HoloLens: Where virtual reality meets augmented reality in the operating room. Plast Reconstr Surg 140:1066–1070, 2017
Wang S, Parsons M, Stone-McLean J, Rogers P, Boyd S, Hoover K, Meruvia-pastor O, Gong M, Smith A: Augmented reality as a telemedicine platform for remote procedural training. Sensors 17:1–21, 2017
Lee N: LG’s SteamVR headset is a bulky yet promising HTC Vive alternative. Available at https://www.engadget.com/2017/03/02/lg-steamvr-headset/. Accessed May 2017
Bharathan R, Vali S, Setchell T, Miskry T, Darzi A, Aggarwal R: Psychomotor skills and cognitive load training on a virtual reality laparoscopic simulator for tubal surgery is effective. Eur J Obstet Gynecol Reprod Biol 169:347–352, 2013
3D Systems Extends Leadership in Precision Healthcare with Technology Advancements to Quickly Create 3D Models. Available at https://www.3dsystems.com/press-releases/3d-systems-extends-leadership-precision-healthcare-technology-advancements-quickly. Accessed April 2018
3D Systems Leverages Virtual Reality to Advance Surgical Training. Available at https://www.3dsystems.com/press-releases/3d-systems-leverages-virtual-reality-advance-surgical-training. Accessed April 2018
Interactive Virtual Reality. Available at http://www.echopixeltech.com/interactive-virtual-reality/. Accessed April 2018
TeraRecon Launches Augmented Reality 3-D Imaging at HIMSS17. Available at https://www.itnonline.com/content/terarecon-launches-augmented-reality-3-d-imaging-himss17. Accessed April 2018
Rebenitsch L, Owen C: Review on cybersickness in applications and visual displays. Virtual Real 20:101–125, 2016
Rebenitsch L: Managing cybersickness in virtual reality. Crossroads ACM Mag Stud 22:46–51, 2015
Satava RM, Member A, Jones SB: Current and future applications of virtual reality for medicine. Proc IEEE 86:484–489, 1998
Cates CU, Lönn L, Gallagher AG: Prospective, randomised and blinded comparison of proficiency-based progression full-physics virtual reality simulator training versus invasive vascular experience for learning carotid artery angiography by very experienced operators. BMJ Simul Technol Enhanc Learn 2:1–5, 2016
Ali S, Qandeel M, Ramakrishna R, Yang CW: Virtual simulation in enhancing procedural training for fluoroscopy-guided lumbar puncture: A Pilot Study. Acad Radiol 25:235–239, 2018
Ramlogan R, Niazi AU, Jin R, Johnson J, Chan VW, Perlas A: A virtual reality simulation model of spinal ultrasound: Role in teaching spinal sonoanatomy. Reg Anesth Pain Med 42:217–222, 2017
Azagury DE, Ryou M, Shaikh SN, San José Estépar R, Lengyel BI, Jagadeesan J, Vosburgh KG, Thompson CC: Real-time computed tomography-based augmented reality for natural orifice transluminal endoscopic surgery navigation. Br J Surg 99:1246–1253, 2012
Johnston APR, Rae J, Ariotti N, Bailey B, Lija A, Webb R, Ferguson C, Maher S, Davis TP, Webb RI, Mcghee J, Parton RG: Journey to the centre of the cell: Virtual reality immersion into scientific data. Traffic 105–110, 2017.
Shanahan M: Use of a virtual radiography simulation enhances student learning. J Med Radiat Sci 63:27, 2016
Syed AZ, Zakaria A, Lozanoff S: Dark room to augmented reality: Application of HoloLens technology for oral radiological diagnosis. Oral Surg Oral Med Oral Pathol Oral Radiol 124:e33, 2017
Mitsouras D, Liacouras P, Imanzadeh A, Giannopoulos AA, Cai T, Kumamaru KK, George E, Wake N, Caterson EJ, Pomahac B, Ho VB, Grant GT, Rybicki FJ: Medical 3D printing for the radiologist. RadioGraphics 35:1965–1988, 2015
Giannopoulos AA, Mitsouras D, Yoo S-J, Liu PP, Chatzizisis YS, Rybicki FJ: Applications of 3D printing in cardiovascular diseases. Nat Rev Cardiol 13:701–718, 2016
Christensen A, Rybicki FJ: Maintaining safety and efficacy for 3D printing in medicine. 3D Print Med 3:1, 2017
Di Prima M, Coburn J, Hwang D, Kelly J, Khairuzzaman A, Ricles L: Additively manufactured medical products—The FDA perspective. 3D Print Med 2(1), 2015
Rybicki FJ: Medical 3D printing and the physician-artist. Lancet 391:651–652, 2018
Rybicki FJ: Message from Frank J. Rybicki, MD, incoming chair of ACR appropriateness criteria. J Am Coll Radiol 14:723–724, 2017
Douglas DB, Wilke CA, Gibson D, Petricoin EF, Liotta L: Virtual reality and augmented reality: Advances in surgery. Biol Eng Med 2:1–8, 2017
Gallagher K, Jain S, Okhravi N: Making and viewing stereoscopic surgical videos with smartphones and virtual reality headset. Eye 30:503–504, 2016
Huber T, Wunderling T, Paschold M, Lang H, Kneist W, Hansen C: Highly immersive virtual reality laparoscopy simulation: Development and future aspects. Int J Comput Assist Radiol Surg 13:281–290, 2018
Kersten-Oertel M, Gerard I, Drouin S, Mok K, Sirhan D, Sinclair DS, Collins DL: Augmented reality in neurovascular surgery: Feasibility and first uses in the operating room. Int J Comput Assist Radiol Surg 10:1823–1836, 2015
Moglia A, Ferrari V, Morelli L, Ferrari M, Mosca F, Cuschieri A: A systematic review of virtual reality simulators for robot-assisted surgery. Eur Urol 69:1065–1080, 2016
Londei R, Esposito M, Diotte B, Weidert S, Euler E, Thaller P, Navab N, Fallavollita P: Intra-operative augmented reality in distal locking. Int J Comput Assist Radiol Surg 10:1395–1403, 2015
King F, Jayender J, Bhagavatula SK, Shyn PB, Pieper S, Kapur T, Lasso A, Fichtinger G: An immersive virtual reality environment for diagnostic imaging. J Med Robot Res 01:1640003, 2016
Douglas DB, Petricoin EF, Liotta L, Wilson E: D3D augmented reality imaging system: Proof of concept in mammography. Med Devices Evid Res 9:277–283, 2016
Hanna MG, Ahmed I, Nine J, Prajapati S, Pantanowitz L: Augmented reality technology using Microsoft HoloLens in anatomic pathology. Arch Pathol Lab Med, 2018. https://doi.org/10.5858/arpa.2017-0189-OA
Calì C, Baghabra J, Boges DJ, Holst GR, Kreshuk A, Hamprecht FA, Srinivasan M, Lehväslaiho H, Magistretti PJ: Three-dimensional immersive virtual reality for studying cellular compartments in 3D models from EM preparations of neural tissues. J Comp Neurol 524:23–38, 2016
Zheng LL, He L, Yu CQ: Mobile virtual reality for ophthalmic image display and diagnosis. J Mob Technol Med 4:35–38, 2015
Dascal J, Reid M, Ishak WW, Spiegel B, Recacho J, Rosen B, Danovitch I: Virtual reality and medical inpatients: A systematic review of randomized, controlled trials. Innov Clin Neurosci 14:14–21, 2017
Schmitt YS, Hoffman HG, Blough DK, David R, Jensen MP, Soltani M, Carrougher GJ, Mn RN, Nakamura D, Otr L, Sharar SR: A randomized, controlled trial of immersive virtual reality analgesia during physical therapy for pediatric burn injuries. Burns 37:61–68, 2011
Carrougher GJ, Hoffman HG, Nakamura D, Lezotte D, Soltani M, Leahy L, Engrav LH, Patterson DR: The effect of virtual reality on pain and range of motion in adults with burn injuries. J Burn Care Res 30:785–791, 2009
Kipping B, Rodger S, Miller K, Kimble RM: Virtual reality for acute pain reduction in adolescents undergoing burn wound care: A prospective randomized controlled trial. Burns 38:650–657, 2012
Morris LD, Louw QA, Crous LC: Feasibility and potential effect of a low-cost virtual reality system on reducing pain and anxiety in adult burn injury patients during physiotherapy in a developing country. Burns 36:659–664, 2010
Hoffman HG, Patterson DR, Seibel E, Soltani M, Jewett-Leahy L, Sharar SR: Virtual reality pain control during burn wound debridement in the hydrotank. Clin J Pain 24:299–304, 2008
Patterson DR, Jensen MP, Wiechman SA, Sharar SR: Virtual reality hypnosis for pain associated with recovery from physical trauma. Int J Clin Exp Hypn 58:288–300, 2010
Li WH, Chung JO, Ho EK: The effectiveness of therapeutic play, using virtual reality computer games, in promoting the psychological well-being of children hospitalised with cancer. J Clin Nurs 20:2135–2143, 2011
Cesa GL, Manzoni GM, Bacchetta M, Castelnuovo G, Conti S, Gaggioli A, Mantovani F, Molinari E, Cárdenas-López G, Riva G: Virtual reality for enhancing the cognitive behavioral treatment of obesity with binge eating disorder: Randomized controlled study with one-year follow-up. J Med Internet Res 15:1–13, 2013
Manzoni GM, Pagnini F, Gorini A, Preziosa A, Castelnuovo G, Molinari E, Riva G: Can relaxation training reduce emotional eating in women with obesity? An exploratory study with 3 months of follow-up. J Am Diet Assoc 109:1427–1432, 2009
Larson EB, Ramaiya M, Zollman FS, Pacini S, Hsu N, Patton JL, Dvorkin AY: Tolerance of a virtual reality intervention for attention remediation in persons with severe TBI. Brain Inj 25:274–281, 2011
Saposnik G, Cohen LG, Mamdani M, Pooyania S, Ploughman M, Cheung D, Shaw J, Hall J, Nord P, Dukelow S, Nilanont Y, De los Rios F, Olmos L, Levin M, Teasell R, Cohen A, Thorpe K, Laupacis A, Bayley M: Efficacy and safety of non-immersive virtual reality exercising in stroke rehabilitation (EVREST): A randomised, multicentre, single-blind, controlled trial. Lancet Neurol 15:1019–1027, 2016
Silver B: Virtual reality versus reality in post-stroke rehabilitation. Lancet Neurol 15:996–997, 2016
Ambron E, Miller A, Kuchenbecker KJ, Laurel B, Coslett HB: Immersive low-cost virtual reality treatment for phantom limb pain: Evidence from two cases. Front Neurol 9:1–7, 2018
Freeman D, Reeve S, Robinson A, Ehlers A, Clark D, Spanlang B, Slater M: Virtual reality in the assessment, understanding, and treatment of mental health disorders. Psychol Med 47:2393–2400, 2017
Maples-Keller JL, Price M, Rauch S, Gerardi M, Rothbaum BO: Investigating relationships between PTSD symptom clusters within virtual reality exposure therapy for OEF/OIF veterans. Behav Ther 48:147–155, 2017
Miloff A, Lindner P, Hamilton W, Reuterskiöld L, Andersson G, Carlbring P: Single-session gamified virtual reality exposure therapy for spider phobia vs. traditional exposure therapy: Study protocol for a randomized controlled non-inferiority trial. Trials 17:1–8, 2016
Nararro-Haro MV, Hoffman HG, Garcia-Palacios A, Sampaio M, Alhalabi W, Hall K, Linehan M: The use of virtual reality to facilitate mindfulness skills training in dialectical behavioral therapy for borderline personality disorder: A case study. Front Psychol 7:1–9, 2016
Wiederhold BK, Wiederhold MD: Virtual Reality Therapy for Anxiety Disorders: Advances in Evaluation and Treatment. Washington, DC: American Psychological Association, 2005
Chepelev L, Giannopoulos A, Tang A, Mitsouras D, Rybicki FJ: Medical 3D printing: Methods to standardize terminology and report trends. 3D Print Med 3:1–9, 2017
Rybicki FJ: 3D printing in medicine: An introductory message from the editor-in-chief. 3D Print Med 1(1):1, 2015
Mitsouras D, Liacouras PC: 3D printing in medicine: 3D printing technologies. In: Rybicki FJ, Grant GT Eds. 3D Printing in Medicine: A Practical Guide for Medical Professionals. Cham: Springer International Publishing, 2017, pp. 5–22
American Association of Physicists in Medicine: Assessment of display performance for medical imaging systems. Med Phys:1–156, 2005
Norweck JT, Seibert JA, Andriole KP, Clunie DA, Curran BH, Flynn MJ, Krupinski E, Lieto RP, Peck DJ, Mian TA, Wyatt M: ACR-AAPM-SIIM technical standard for electronic practice of medical imaging. J Digit Imaging 26:38–52, 2013
Drebin R A, Carpenter L, Hanrahan P: Volume rendering. SIGGRAPH ’88. In: Proc. 15th Annu. Conf. Comput. Graph. Interact. Tech. 22, pp. 65–74, 1988
Levoy M: Volume rendering: Display of surface from volume data. IEEE Comput Graph Appl 8:29–37, 1988
Stegmaiert S, Kieint T, Stegmaier S, Strengert M, Klein T, Ertl T: A simple and flexible volume rendering framework for graphics-hardware-based raycasting. Fourth Int. Work. Vol. Graph, pp. 187–241, 2005
Ljung P, Krüger J, Groller E, Hadwiger M, Hansen CD, Ynnerman A, Gröller E, Hadwiger M, Hansen CD, Ynnerman A: State of the art in transfer functions for direct volume rendering. Comput Graph Forum 35:669–691, 2016
Arens S, Domik G: A survey of transfer functions suitable for volume rendering. IEEE/EG Int. Symp. Vol. Graph., pp. 77–83, 2010
Krüger J, Westermann R: Acceleration techniques for GPU-based volume rendering. IEEE Vis, pp. 287–292, 2003.
Vizua3D: https://vizua3d.com. Available at https://vizua3d.com. Accessed February 2018
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Sutherland, J., Belec, J., Sheikh, A. et al. Applying Modern Virtual and Augmented Reality Technologies to Medical Images and Models. J Digit Imaging 32, 38–53 (2019). https://doi.org/10.1007/s10278-018-0122-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10278-018-0122-7