Mixed Reality Gives Surgeons Superpowers
@Simon Drouin
Overcoming the Limitations of Generic Images
Anatomy textbooks have long illustrated the details of most body parts very accurately. They provide surgeons with a clear understanding of three-dimensional anatomical structure and the spatial relationships between different parts. However, these illustrations are lacking when it comes to operating on a patient, because they remain generic. Medical imaging (e.g., magnetic resonance) is needed to show the patient’s anatomical peculiarities. These images are far from intuitive. They are composed of layers that scroll across a screen and form the volume of a body part.
What if, like Superman, surgeons were given the power to “see” through a patient’s skin? That’s the challenge we are tackling: making 3D images as eloquent as textbook pictures. To achieve this, we draw our inspiration not only from medical publications, but also from the principles of human visual perception. We aim to make these images as comprehensible as possible in terms of depth of detail. We also apply principles of computer graphics to produce real-time renderings of these images.
Rendering can be brought closer to reality by integrating mixed reality devices: pairing virtual reality and augmented reality. These devices are used to integrate processes that make more efficient use of the visual system, such as stereo vision (two views) and the parallax effect (changing viewpoint depending on positions of the head).
Facilitating the Work of Surgeons
A number of tools currently exist to help surgeons, but they are not very intuitive. The location of surgical instruments in relation to the anatomy of a person undergoing an operation can be tracked in real time and displayed on a computer screen. Surgeons must look alternately at the person and the screen, point by point, in an attempt to form a 3D mental image of the structures on which they are operating.
Layering images of the 3D rendering onto those of the surgical site creates a virtual reality that gives surgeons superpowers. Some operations last several hours without interruption, so any tool that can reduce the duration and difficulty of these operations can only be beneficial. Mixed reality can also be used to prepare for complex surgical procedures.
We measure cognitive load to assess the actual help provided by our tools. We aim to present the brain with images that it can interpret naturally, thus maintaining the surgeon’s energy throughout the operation.
