Toward a More Immersive Video Experience

Today, video lets us explore a space as if we were actually there. On location, a 360-degree camera captures a scene in all directions—whether in a concert hall, museum, or other spaces. From a distance, users wearing a virtual reality headset can then explore this view by turning their heads, as if they were standing exactly where the camera was placed. But could they also move around in that space?

This is one question that Stéphane Coulombe, professor at ÉTS and member of the International Laboratory on Learning Systems, is currently exploring. His work on video compression, transmission, and quality assessment is evolving concurrently with immersive technologies. After focusing on traditional video and then 360-degree video, his team is now taking the next step: offering users a view with six degrees of freedom, allowing users to move freely within a reconstructed real-world scene.

From three to six degrees of freedom

As part of the Summit Tech Research Chair, Stéphane Coulombe and his team—in collaboration with Carlos Vázquez, among others—are working to add three additional degrees of freedom. This would allow users to look around and also to move through the reconstructed environment as if they were actually there.

This advancement may seem minor, but it represents a significant technological challenge. To create the impression of movement, the system must generate a new view every time the user changes position. But this view may not necessarily have been filmed directly.

The solution involves using multiple cameras positioned throughout the space. The system captures the images and uses interpolation to calculate what the user would see from a spot located between the cameras. This requires synchronizing the cameras, knowing their exact positions, and reconstructing the environment geometry at the correct physical scale.

Compounding this is a key constraint: movement must appear natural. If the user moves a few meters within the virtual scene, the system must produce a seamless transition at a speed that matches human movement.

A challenge in computation and data transmission

Moving from three to six degrees of freedom naturally requires much more data. Multiple cameras must capture the scene simultaneously, sometimes at 30 to 60 frames per second. The system must then process this information and generate intermediate views almost instantly. The challenge is to find the optimal balance between image quality, data transmission rate, and computing power.

This trade-off is evident in many aspects of Stéphane Coulombe’s work. However, for immersive applications, transmitting all available information is not always necessary—for example, when part of the image is in an area the user is not looking at.

Consequently, the research team is also exploring the possibility of predicting areas where the users will focus their attention. By analyzing head movements inside the virtual reality headset, the algorithms can anticipate the gaze direction and allocate more resources to areas likely to be viewed. This reduces the quantity of data to be transmitted while maintaining high quality where it really matters.

Two eyes, twice the data?

Immersive viewing presents another challenge: to create a sense of depth, separate images must be generated for each eye. A 360-degree stereoscopic video therefore contains twice as much information to transmit.

Stéphane Coulombe and his team have developed a method to combine both views into a single representation, while preserving as much useful information as possible.

The principle is based on the geometry of a 360-degree image, typically represented in equirectangular format: like a world map, a spherical view unwrapped into a rectangle. Around the equator, the image is relatively accurate, but the regions near the poles are significantly stretched. As a result, these regions contain redundancies: more pixels are needed to represent the same amount of information.

By reducing this redundancy, the quantity of data to transmit can be decreased with no significant loss of quality. This optimization is especially valuable when every byte counts.

Reconstructing the world from images

Another area of research involves reconstructing three-dimensional environments from photographs. To achieve this, the team focuses specifically on Gaussian splats.

The concept can be visualized as a scene filled with small, oriented ellipsoids, each with its specific size, colour, and degree of transparency. Together, these elements can represent the geometry and appearance of a three-dimensional environment.

The advantage of this approach is its simplicity in rendering: once the scene is reconstructed, it can be displayed very efficiently. The main challenge is in the preliminary stages. Optimization methods, often based on artificial intelligence, serve to adapt the Gaussian splats to each scene.

When transmission fails, AI takes over

The quality of an immersive experience also depends on the transmission’s reliability. When a video is streamed in real time, it’s not feasible to wait several seconds to recover a lost data packet. The image would already be outdated.

It may therefore be better to discard a lost or corrupted frame and move on to the next one. But if losses or errors become frequent, the video may freeze or display artifacts.

So Stéphane Coulombe is also interested in correcting transmission errors. When a packet arrives damaged, certain parts of the information attached to the packet—namely the CRC check code—can help determine what has been altered. Artificial intelligence methods can then generate possible versions of the image and determine which are the most accurate.

This use of AI is playing an increasingly important role in Stéphane Coulombe’s research. It enables video quality assessment, predictable user movement, and content-based decision-making regarding compression or reconstruction.

Technology for new experiences

The potential applications of these technologies extend far beyond entertainment. The six degrees of freedom approach could be used in many situations, such as virtual tourism, concerts, sporting events, education, industrial training, inspections, digital twins, medical visualization, and even immersive videoconferencing.

For Stéphane Coulombe, the challenge lies precisely in pushing the boundaries without losing sight of the limitations of networks and devices. His goal is the same, regardless of the technology: to deliver the best possible video quality with the available resources.

The future of immersive video will not depend solely on higher-performance cameras or more sophisticated headsets. It will rely on a combination of innovations: camera synchronization, compression, 3D reconstruction, artificial intelligence, gaze and position prediction, transmission, and processing at the network edge. Bringing these technologies together will gradually transform video into a virtual space where users will not only look around but also move about.