SOMA
Sensorfusion for Object based Multimodal Analysis
July 2021 - June 2024
Objectives
The objectives of the research project involve the development of a proprietary sensor system with microphones, cameras, LiDAR, and GPS for multimodal data acquisition. The emphasis from the outset is on a spherical sensor system. Additionally, there is a focus on simplifying data fusion through the synchronization of all data types before processing, utilizing Precision Time Protocol (PTP).
Approach
Firstly, the goal is to improve the quality of multimodal sensor data. To achieve this, a high-density microphone system is being developed, consisting of at least 64 microphones to ensure an adequate signal-to-noise ratio without aliasing up to 20 kHz. A camera system with 6 cameras, providing a wide-angle view and Full HD resolution, as well as a proprietary LiDAR scanning system and GPS for time and position tracking, are also integrated.
The spherical array geometry is a central aspect of the design. The microphone system is located at the center, the frame structure around the microphone system houses the camera modules, the LiDAR scanner is positioned beneath the frame structure, and the GPS module is placed beneath the LiDAR scanner.
Crucial to data synchronization is the use of AES67 for transmitting the microphone array data (inherent PTP synchronization). The camera data can be synchronized at the operating system level through PTP (Linux PTP). The LiDAR scanner is inherently synchronized using PTP (e.g., Ouster OS0). Additionally, the GPS module receives timestamps from the same source as the PTP Grandmaster, maintaining a constant offset for synchronization purposes.
Prospects
The applications of the project results lie in the simultaneous acquisition of noise and geoinformation with concurrent scene visualization, as well as the modeling of extensive digital twins.
The strategic potentials include the simultaneous use of multiple sensor systems for exploring data evaluation with six degrees of freedom and establishing a comprehensive analysis system for future data-related research and development projects.