From Special Effects to Augmented Reality for Neurosurgery

By the 1990s, and in a few short years, Montréal became a hub in the emerging special effects software industry. Simon Drouin quickly became fascinated by the technical prowess of Montréal companies, such as Softimage, whose software was used to produce revolutionary special effects in the film “Jurassic Park”. From then on, there was no doubt in his mind that he would become a computer engineer in order to participate in this movement. Since then, his various experiences have led him to work on almost every aspect of digital imaging.

His career path can certainly be described as eclectic. Professor Drouin conducted numerous research activities during his master’s, doctoral and postdoctoral studies—at Brigham and Women’s Hospital and at Harvard Medical School in Boston—just before joining ÉTS; he also has more than 15 years of practical experience as a programmer in various companies and organizations, notably at Ubisoft and the NFB. He has now joined the Department of Software Engineering and Information Technology team at ÉTS as a research professor.

In Drouin’s words: “For most of my career as a computer scientist and researcher, I studied the relationship between the users of a computer system and graphically represented virtual objects. I studied various applications, each time trying to identify ways for users to interact as naturally as possible with computer modelled entities.”

Indeed, he believes that handling virtual 3D objects should be as natural and intuitive as it is with physical objects. His aim is to combine principles of human visual perception, computer vision, human-computer interaction and computer graphics to achieve this goal.

Starting in Digital Arts

After earning a bachelor’s degree in computer engineering from Montreal’s École Polytechnique, Simon Drouin worked for a few years at Ubisoft, where he was part of a team whose mandate it was to implement a new model for the creation and rendering of animated game characters.

Simon Drouin, professor at ÉTS

He then pursued a master’s degree in computer science at the Mobile Robotics Lab at McGill University, focusing on assisted rotoscopy. Using Markov random fields, he modelled the relationship between the artists’ drawings, the reference video, and the images to be produced. Today, he believes that it would be relevant to revisit this project considering the advances achieved in deep learning.

After Ubisoft, Professor Drouin joined the National Film Board of Canada, where he worked on the development of animation software—the SANDDE system—designed to create three-dimensional lines using a localized 3D tool handled by the artist—an ancestor of Google TiltBrush.

Changing Course to Image-Guided Surgery

With an interest in medical imaging, Drouin earned a PhD in Biomedical Engineering from McGill University, working primarily at the Montreal Neurological Institute, focusing his research on the use of medical images to guide neurosurgical operations. Of particular interest to him was the improvement of graphics rendering techniques for augmented reality and the modes of surgeon-machine interaction likely to improve the understanding of the produced images.

According to him: “Image-guided neurosurgery (IGN) systems allow a surgeon to visualize a patient’s pre-operative scans within the operating room to better target an operation. They also make it possible to establish a correspondence between these scans and anatomy thanks to real-time monitoring of surgical instruments. In existing IGN systems, surgeon-computer interaction remains cognitively demanding and very limited in practice.”

The IBIS Platform – AR for Neurosurgery

For his PhD, Drouin developed the IBIS software platform, offering IGN technology to improve the visualization and surgeon-computer interaction paradigm of this type of system by enabling Augmented Reality (AR) visualization and direct interaction between the surgeon and the computer. The IBIS platform, released as open-source software, is now used by several other researchers at the Montreal Neurological Institute (MNI), at Concordia University, and at Université de Sherbrooke.

Beyond Medical Images

Newly appointed research professor at ÉTS, Drouin intends to focus much of his research on interactive medical image visualization applications and to pursue his collaboration with the Montreal Neurological Institute. He also wants to use his experience and contacts in the video game and digital arts community to offer students exciting practical projects in collaboration with organizations and companies, such as the NFB, the Society for Arts and Technology (SAT) and Ubisoft.