5 Fascinating AR & VR Applications in Medicine

Move over, video games. AR, VR, and MR (mixed reality) are now being used not just for entertainment, but also to aid surgeons in the operating room, to train student doctors, and to educate patients. Here’s how.

1. Bringing AR and VR images to surgery prep and the operating room.

In 2016, the Neurosurgical Simulation and Virtual Reality Center opened at Stanford University’s School of Medicine. In the simulation center, VR technology “allows trainees to explore three-dimensional, digital renditions of brain structures.” With a headset and handheld controllers, students can explore a wide variety of neurosurgical cases.

The most exciting part is that these cases are built from 2D images (MRIs, CT scans, and angiograms) from real patients. This means that surgeons can use VR in the simulation center to study the individual details of their patients’ conditions. As if that wasn’t cool enough on its own, the system used for VR can also be wheeled into operating rooms and clinics, to either plan out a surgery or to teach a patient about his or her condition.

But wait! There’s more! The simulation center is located right next to an anatomy lab where surgery residents can practice procedures on cadaver heads. 3D technology plays an important role in the anatomy lab, too: a special microscope allows for the projection of 3D images onto the “patients” during practice surgeries.

Neurosurgeons aren’t alone in using VR to explore patient imaging data. In a study led by Dr. Egor Parkhomenko, urologists at the University of California, Irvine used 3D slicer software to compose VR models that surgeons and patients could view and interact with before partial nephrectomies.

Image source: Parkhomenko et al. 

The surgeons reported that using the VR images improved their “understanding of the size and shape of the renal mass, the anatomy, and the surgical approach.” After surgeons viewed the models, 56% of the cases saw changes to the surgical approach used. Patients also reported an increased understanding of their condition and the procedure that would be used.

Like VR, AR also has an important place in pre-operative analysis. St. Mary’s Hospital in London has teamed up with Microsoft to turn 2D diagnostic images into 3D projections that can be overlaid onto the patient’s actual body using the HoloLens (a mixed reality headset). The St. Mary’s researchers hope that, when streamlined, this technology will help save time during surgery.

Image source: Imperial College.

In the U.S., the OpenSight Augmented Reality System for the HoloLens has already been approved by the FDA for pre-surgical planning.

2. Replacing cadaver-based labs.

Many medical schools use technology (3D models, AR, VR) to augment cadaver dissection programs, but some are now beginning to build new labs that are cadaver-free.

This year, when Case Western Reserve University and the Cleveland Clinic open their new health campus, there will be an AR/VR anatomy lab instead of a traditional cadaver lab.

The most important reason for this switch is that VR and AR equipment is much cheaper and easier to sustain than a cadaver lab. In addition, students who were taught using only the HoloLens (and no cadavers) scored as well on tests as students taught with the traditional method. CWRU professor Mark Griswold noted that even with the new lab, first-year students will engage in cadaver dissections at a two-week boot camp program.

The Royal College of Surgeons in the U.K. plans to follow suit, replacing cadaver-based courses with mixed reality as the college prepares for a 2020 rebuild. Instructors there are teaming up with Pearson to create mixed reality medical education content for the Microsoft HoloLens.

Shafi Ahmed, one of the Royal College of Surgeons’ council members and founder of Medical Realities, is confident that VR and AR are the future of medical education. He told The Guardian that “[In the next five years] I think most people will be taught with this AR, VR, mixed reality. Learning will change immeasurably.”

Dr. Ahmed using a VR headset.  Image source: Medical Realities.

3. Helping pharmacy students understand drug effects.

It’s one thing to use words to describe how a drug affects a patient—it’s another thing entirely to provide a virtual visual experience of the physiology involved.

Dr. Kevin Sneed, Dean of the College of Pharmacy at the University of South Florida, intends to use VR as a new way to visualize the effects of different medications on the body. With VR learning, pharmacy students will gain a more thorough understanding of what the drugs they administer will do. Students will also be able to participate in multi-student operating room simulations, allowing them to better understand and practice their roles. Those who are visual learners will likely prefer the immersive nature and real-time movement of the models to PowerPoint slides and paragraphs of description. 

Patients can benefit from pharmacological VR as well. “If a patient is starting a new medication, for example, we can show them using this virtual reality technology what the medication is going to do when it enters the body and how it is going to affect their organs,” Sneed explained in an interview with the USF Oracle. His eventual goal is to have “an entire library of medication within virtual reality to show the patient’s activity inside of the body.”

4. Teaching dental students to administer local anesthetics.

In addition to medical and pharmacy schools, dental schools are also joining the world of VR. Denise Higgins from the University of Newcastle in Australia uses VR to help students practice their technical skills and gain confidence as they conduct simulated anesthesia injections. Within the simulation, students can perform virtual injections, keeping track of the volume of anesthetic they’re using and how quickly it is being administered. Students also practice giving injections to manikins designed by Higgins for that purpose. Though the manikins’ tissue is synthetic, the tactile experience of giving an injection provides valuable practice.

5. Training surgeons on new procedures and devices.

VR is becoming more and more popular as a method of training surgeons, since residents have limited training time and must stay up to date with constantly changing and improving medical device technology. Virtual Reality provides a cost-effective way to help surgeons learn and practice these various procedures with their associated devices.

A number of institutions—David Geffen School of Medicine at UCLA, Harvard Medical School, Vanderbilt University School of Medicine, Hospital for Special Surgery, Columbia, and Long Island Jewish Medical Center—are piloting Osso VR, a surgical training platform developed by orthopedic surgeon Dr. Justin Barad and Matt Newport, a professional game developer.

Osso VR contains an impressive library of tools and the associated hand/controller motions intended to simulate their use. You can see a sample of Osso VR's surgery training content here.

Though the initial content from Osso VR has focused on orthopedic procedures, Barad is optimistic about the future application of the platform to other surgical specializations.

Jeffery Hills, MD, an orthopedic resident at Vanderbilt, explained to Forbes.com that “...perfecting certain skills or reviewing the surgical approach through a quick session in the virtual operating room, will make you even more efficient in the cadaver lab, and ultimately more efficient in the operating room."

So what does all this mean? It’s clear that AR and VR are going to play an increasingly large and central role in medicine and medical education in the years to come. “Futuristic” devices like the HoloLens and Oculus Rift are hitting their stride as they become more affordable and as developers provide a wider range of content for them.

This technology isn’t just for doctors, either—VR headsets are being used for STEM(+) education in libraries as well as K–12 curricula. There’s a whole virtual world of educational possibilities out there, and we’ve only just begun to explore them!

Be sure to subscribe to the Visible Body Blog for more anatomy awesomeness! 

Are you a professor (or know someone who is)? We have awesome visuals and resources for your anatomy and physiology course! Learn more here. 

Additional Sources: 

• Parkhomenko et al. (2018) "Virtual Reality Immersion for Preoperative Planning and Education for Partial Nephrectomy". Presented at 36th World Congress of Endourology.
• Pratt et al. (2018) "Through the HoloLens looking glass: augmented reality for extremity reconstruction surgery using 3D vascular models with perforating vessels." European Radiology Experimental 2:2.