5GXR

Volumetric video is an immersive media content produced by  capturing an object (e.g., people) using multiple, synchronized cameras in a three-dimensional space. This object can be later viewed from any angle at any point in time. Volumetric video enables the representation of six degrees-of-freedom (6DoF) content, through which a viewer can “move” within a video and view objects from different angles and distances.

The resulting data can be represented as point clouds [a set of points with the coordinates (x,y,z)] or meshes (vertices and texture).

Video-based Point Cloud Compression (V-PCC) standard compresses the object’s geometry, texture, and occupancy map separately and feeds the encoded bitstreams into a multiplexer along with metadata.

To process the mesh data, both the texture and vertices are encoded separately and fed into a multiplexer.

Volumetric video requires considerably more data (i.e., geometric information and texture) than its two-dimensional counterpart. Processing these data introduces challenges including: a lack of hardware decoders, difficulties associated with compression of point clouds, high bit rates (e.g., a raw point cloud object consisting of 2.8 million points requires a bandwidth of about 110 billion bits per second at 30 frames per second), and high processing power requirements for combining multiple volumetric objects.

The hardware found in current mobile devices is inadequate for decoding volumetric video content. A workaround is to offload the processing to the cloud. Here, a view from the three-dimensional object can be rendered into a two-dimensional video. This two-dimensional video can be compressed and transmitted to a user’s device. The rendered view is dynamically updated according to user interaction. Benefits include: leveraging the existing two-dimensional video processing pipeline on mobile phones, enabling the display of complex scenes on legacy devices, and achieving practicable bitrates for transmission of complex volumetric scenes. However, cloud-based rendering results in additional network latency (i.e., “motion-to-photon” latency).

The tradeoff for using cloud rendering is added network latency, which has been known to cause degraded user experience and motion sickness. To reduce network latency (beyond the benefits of using the 5G Radio Access Network), we use efficient hardware decoders, real-time streaming protocols, and predict the user’s head movement in 6DoF space.

This approach can reduce motion-to-photon latency and improve the user’s viewing experience. To accomplish this, algorithms predict the position of a user on the cloud server side for a given look-ahead time, and the corresponding rendered image is sent to the client.

In order to increase the immersive experience, we even go beyond the application of free-view­point volumetric video and present a new framework for the crea­tion of interactive volumetric video content of humans. Re-animation and alteration of an actor’s performance captured in a volumetric studio becomes possible through semantic enrichment of the captured volumetric video data and new hybrid geometry- and video-based animation methods that allow a direct animation of the high-quality data itself instead of creating an animatable model that resembles the captured data.

Details of our work on interactive volumetric video can be found here.