March 4, 2022
The joint project "Virtual LiVe" (virtualization of live events through audio-visual immersion) has achieved a further validation step with the successful implementation of a hybrid live concert. The Fraunhofer Heinrich Hertz Institute (HHI), Fraunhofer FOKUS and Fraunhofer IIS are involved in the project. Last December, viewers experienced a concert in Berlin's Kesselhaus venue simultaneously in different locations, thanks to immersive media technologies in collaboration with the creative industries. Virtual LiVe, hosted by the Fraunhofer HHI transfer platform "3IT - Innovation Center for Immersive Imaging Technologies ", is funded by the 'KMU-akut' program "Research for SMEs" of the Fraunhofer-Gesellschaft. The aim of the project is to digitally complement or even replace classic event formats by means of immersive media technologies (e.g. 3D audio, 360° video, light fields, volumetric video) when physical presence is not possible.
How can we use state-of-the-art technology to transform live events into hybrid formats? The event industry has been asking this question at least since the start of the Covid-19 pandemic. In the Virtual LiVe project, the three Fraunhofer Institutes HHI, FOKUS and IIS are working together with companies from the creative industry to address this challenge. In order to blur the boundaries between physical and digital perception, they are combining high-end technologies in order to not only optimize the digitization of classic event formats, but also to open up new venues for interconnected artistic performances.
The project's first hybrid live event took place in December 2021: Artist Billy Andrews, lead singer of the band The Dark Tenor, performed the kick-off concert of his Classic Roxx Tour at four venues simultaneously. About 150 guests attended the show in the Kesselhaus of the Kulturbrauerei in Berlin. The singer was accompanied by pianist duo Queenz of Piano, who were integrated into the concert from their studio via an ultra-low latency audio connection through the Digital Stage platform. In this way, Queenz of Piano was projected onto a screen in the Kesselhaus.
The event was recorded using Fraunhofer HHI's OmniCam panoramic camera. This scalable, mirror-based multi-camera system, together with its Real-Time Stitching Engine (RTSE), is capable of recording a high-resolution 360° video stream and transmitting it in real time. This allowed the concert to be streamed in a resolution of up to 4K to the dome projectors in the planetarium in Bochum, to the Fraunhofer IIS cinema, and to VR glasses and mobile devices of viewers at home. The video streams were virtually transmitted with low latency and adaptive quality, including acoustic or visual feedback, to the end devices of more than 600 viewers from 16 countries with a low delay of a few seconds.
To complete the immersive experience, the concert sound was produced using MPEG-H audio; a 3D audio mix was created in real time. Using a Spatial Audio Designer, the researchers created a MPEG-H scene, then encoded and transferred it to the cloud. The audio and video stream were decoded at the planetarium and then distributed to more than 60 speakers in the planetarium dome. The team also synchronized the various video and audio streams frame-by-frame to ensure a perfect audiovisual experience. The lab concert was thus able to demonstrate that a technologically innovative production workflow is capable of delivering immersive, real-time content to audiences both on-site and at remote venues.
The goal of Virtual LiVe is to develop a platform that provides scalable high-end solutions in modular form for the event streaming needs of various players in different industries. From this modular system, SMEs can put together a program for their respective event needs, depending on their needs and requirements. This provides them with a technology-based solution for the realization of their formats. Further validation projects are already being planned. You can find all the latest information on the project here .