When 3D Makes the Invisible Visible

X-rays, videos, and photos show only a part of reality. Behind every two-dimensional image lies a three-dimensional world rich with information. This is what ÉTS Professor Carlos Vázquez is aiming to reveal. His work is based on a simple yet far-reaching idea: reconstructing and understanding 3D structures from 2D images to facilitate decision-making.

Whether it’s to improve musculoskeletal diagnoses or to offer immersive experiences, his research is grounded in the same scientific foundations: computer vision, image analysis, and three-dimensional modelling.

A better understanding of the human body through 3D

In the medical field, images are ubiquitous. However, the imaging tools used daily have their limitations.

Take scoliosis, for example—a curvature of the spine. The spine does not merely curve sideways; it also undergoes rotations and changes in all three spatial planes. Conventional X-rays primarily provide two-dimensional views, making it difficult to assess the condition accurately.

Carlos Vázquez is working on reconstructing 3D models from images obtained with the EOS system, a very-low-dose X-ray technology that simultaneously captures two orthogonal images. Using these two images, algorithms reconstruct the three-dimensional geometry of the bones and joints.

This approach has several advantages. For one thing, it significantly reduces patient radiation exposure compared to CT scanners. For another, it allows observing the body in a standing position under load, which reflects real-world conditions more accurately than exams performed in a supine position.

The same principles of 3D reconstruction can be applied to other musculoskeletal conditions, such as certain ankle disorders, where the distances between bones must be precisely measured to establish a diagnosis. This information is difficult, if not impossible, to obtain from 2D images alone. 3D reconstruction provides a perspective that is much more faithful to anatomical reality.

Observing movement without disturbing the patient

One major challenge of musculoskeletal assessment is understanding not only the body’s shape but also how it functions during movement.

Traditionally, biomechanical analysis relies on specialized systems that require markers to be attached to patients’ bodies. These devices offer high precision, but are expensive, complex to use, and can alter the natural gait of the subjects under observation.

In collaboration with Emovi, Carlos Vázquez is developing motion analysis techniques that rely solely on video. Using computer vision algorithms, it is now possible to estimate joint positions and track movements without markers or harnesses.

The goal is to make these assessments simpler, more accessible, and more representative of patients’ natural behaviour.

This expertise has also found applications in elite sports. In collaboration with the Institut national du sport du Québec, Professor Vázquez and his team used similar techniques to analyze divers’ movements, including joint angles and motion speed during their performances.

Facial reconstruction through augmented reality

Some applications of 3D modelling go beyond the framework of diagnosis.

Carlos Vázquez is involved in a project specifically designed for people who have lost part of their face as a result of head or neck cancer. These patients typically receive an epithesis—a silicone facial prosthesis designed to replace the missing part.

Traditionally, manufacturing these prostheses is a long and complex process. The team is now digitizing the entire process, from model design to 3D-printed fabrication.

Carlos Vázquez’s contribution involves integrating these models into an augmented reality environment. This allows patients to visualize their future prosthesis directly on their face using a virtual “magic mirror.”

The same tools used for immersive entertainment

At first glance, the world of multimedia seems a far cry from that of medicine. Yet the two fields are based on surprisingly similar principles.

Before turning his attention to biomedical applications, Carlos Vázquez was already working on stereoscopic imaging and three-dimensional reconstruction for digital media. The mathematical and algorithmic tools used to reconstruct a spine or an immersive scene share several common foundations.

At the ÉTS Multimedia Laboratory, he focuses on the entire video processing chain: acquisition, compression, transmission, and display.

One of his key initiatives is being conducted in collaboration with Summit Tech and Stéphane Coulombe, also a professor at ÉTS. This project aims to improve live immersive video experiences.

Imagine attending a concert filmed using 360-degree stereoscopic cameras. By wearing an immersive headset, viewers can move virtually within the scene, observe the surroundings from different angles, and enjoy an experience that closely resembles being there in person.

To make this immersion possible, several technical challenges must be overcome: capturing complex three-dimensional content, compressing it efficiently, transmitting it with low latency over 5G networks, and generating real-time viewpoints that mimic the user’s movements.

Predicting where we will be looking

In immersive environments, one challenge is transmitting vast amounts of visual information. Therefore, it is best to predict where users will be focusing their attention.

By analyzing head movements in the immersive headset, for example, the algorithms can anticipate the direction of gaze and focus computational resources on the areas that are actually being viewed. This strategy improves perceived quality while reducing the amount of data that needs to be transmitted.

Researchers are also working on depth estimation—compressing content from multiple 360-degree cameras—and accelerating computations using graphics processing units (GPUs), essential for achieving the required real-time performance.

Two worlds, one vision

Whether it’s helping a doctor better understand a spinal deformity or enabling a viewer to experience an immersive concert from a distance, Carlos Vázquez is pursuing the same goal: to extract more information from images.

His research shows how advances in computer vision can bridge the gaps between disciplines. Behind medical X-rays, motion videos, and immersive experiences lies the same fundamental question: how to reconstruct and understand the world in three dimensions from images that in themselves show only two.

Carlos Vázquez is a member of itechsanté, the ÉTS research institute for innovation in health technologies. To learn more about the institute, its mission, themes, flagship projects, and more, visit itechsanté