Due to the rapid development and the advances in eXtended Reality (XR) technologies and the related devices (headsets, interaction tools, ...), the realistic dynamic 3D representation of humans has achieved significant attention. Especially in use cases, where a convincing visualization of humans is required. Relevant use cases are interactive teaching and training, immersive experiences with contemporary witnesses, meet-ups with celebrities, and many more.
Fraunhofer Heinrich-Hertz-Institute HHI has combined computer vision and computer graphics to create highly realistic dynamic 3D representations of persons - the so-called Volumetric Video. Our concept for 3D human body reconstruction was awarded in 2019 with the Joseph-von-Fraunhofer Prize entitled by "Realistic people in virtual worlds - A movie as a true experience".
Technical Background
We developed a capture system and related workflow to create a dynamic three-dimensional model of a real person, which can then be observed from arbitrary viewpoints in the virtual scene. In this way, the user is able to virtually walk around this person and observe it from different perspectives. Facial expressions as well as moving clothes are reconstructed at high geometrical detail and texture quality. A maximum of naturalness of the visual impression is achieved.
Technically, the system consists of several camera-based, highly precise modules for depth estimation, which are arranged in a flexible way around the person. They capture its textured 3D shape simultaneously from various directions and angles. The data are fused and transformed to a consistent, natural and dynamic 3D representation of the person. Post-processing modules adapt the data structures and formats and allow for direct integration in standardized post-production workflows and real-time render engines.
Benefits
- Integration of real persons in computer generated scenes
- Realistic and lifelike impression and high-quality and naturally moving dynamic 3D models
- Flexible and modular multi-camera system for customized setups
- Fully equipped lighting and capture system for 360° full body reconstruction
- Real-time preview on set
- Efficient cloud processing of high-quality texture and geometry
- Hardware solution for scalable and local high-performance cloud server systems
- Integrated processing of data for virtual and augmented reality (VR/AR) glasses and systems
- Fast integration in professional post production workflows
- Recording up to 50 frames per second
- Possible extension to complete real-time workflow