12 March 2025
Imagine a surgeon equipped with an augmented reality headset, capable of visualizing, through a patient’s skin, a 3D representation of the area to be operated on, with pinpoint accuracy. This process is being developed by École de technologie supérieure de Montréal (ÉTS) researcher Simon Drouin. He is collaborating with neurosurgery teams for whom this kind of tool is a real game changer.
Simon Drouin is not a doctor. He comes from the video game and animated film industry, and his career path transcends the boundaries between computer science and graphic design. His research focuses on handling virtual 3D objects using computer graphic principles, and is based on a keen understanding of human-machine interactive dynamics. Unexpected in his career path was his knowledge of virtual reality (VR) and augmented reality (AR) making him, after a few detours, take an interest in hospital issues, and specifically in the evolution of surgical guidance tools.
Filling the Gaps
Even today, surgery relies on techniques often limited by two-dimensional visualization, with relative precision. This young associate professor in the Department of Software Engineering and Information Technology aims to bring mixed reality into the operating room and to facilitate interaction between clinicians and 3D medical images. In other words, he’s working to “ensure that surgical teams have a direct, real-time, three-dimensional view during operations.”
This innovation is essential in neurosurgery,” he explains. “Precise manipulation in the human brain is crucial to avoid neurological damage. Every millimetre touched can be potentially dangerous.”
He adds: “I like to say that by using a skillful combination of VR and AR, members of the operating room will literally ‘see through skin.’ Not only will they be able to make finer, less invasive incisions—thus reducing trauma for patients—but they will also be supported by far more accurate surgical guidance.”
How Does it Work?
Virtual Reality (VR) in the Preoperative Phase
In the process Simon Drouin wants to implement, virtual reality is used to help surgeons plan their operations. Traditionally, these specialists examine scans of the patient on a computer screen displaying two-dimensional slices, which limits the understanding of three-dimensional structures. VR, on the other hand, will offer three-dimensional image visualization, significantly improving understanding of the geometric relationships between different anatomical structures. More specifically, surgeons would use virtual reality devices that offer “stereoscopic” vision, finding themselves in an immersive 3D simulation of patient anatomy.
Augmented Reality (AR) Applied to Guidance
Throughout the surgical procedure, augmented reality is superimposed on the specialist’s eye, providing a precise view of the area to operate. Displayed through glasses that superimpose virtual images on the real structures, AR images are revolutionizing surgical guidance functions. In other words, AR integrates digital information, such as views of the patient’s internal structures with the actual vision of the surgeon—the image they see through the surgical microscope—ensuring precise guidance throughout the operation. “It’s very real,” says Simon Drouin. “The surgeon can see exactly where the tumour is, where the blood vessels to avoid are, and so on. It’s practically like following a camera going inside [the body].”
The technology also relies on advanced human-machine interaction strategies, such as hand tracking for medical images in virtual reality.
This is a major innovation, as it relieves specialists of the demanding mental gymnastics required by the old guidance system. It is no longer necessary to alternate between the patient and the screen, and then to concentrate on superimposing the two views in your mind.
Should It Be Integrated to Medical Training?
ÉTS Professor Simon Drouin
Evidently, the potential for immersion in virtual surgical simulations is a promising avenue for training future surgeons. Imagine if all universities were to acquire the appropriate technologies: students would be able to perform surgical procedures in a virtual context that accurately reproduces real-life situations.
According to Simon Drouin, the process requires young specialists to wear a virtual reality headset and use tools connected to robots that simulate the pressure exerted during a real operation.
“A master’s student at ÉTS is currently working on a needle in the virtual environment simulating the insertion of a real needle. The simulated forces when inserting the needle into the virtual patient are designed to give young surgeons a realistic perception of the procedure, enhancing their hands-on experience without the risks associated with operating on real people.”
Robotics plays a crucial role in this approach, combined with mixed reality in a completely interdisciplinary perspective! This opens up new and promising horizons, concludes Simon Drouin. “Robotics isn’t my field of expertise, so I can’t say too much about it, but I do think that the future of mixed reality will be intimately linked to it. And that goes for medicine as well as many other fields!”
