AI-based, real-time image analysis for human-robot collaboration at the Fraunhofer Edge Cloud over TSN and 5G
We implemented a unique human-robot collaboration application with an intensive use of AI-based machine vision in the Edge Cloud. The work situation in an experimental production cell with robots and conveyors was captured with flexible 2D industrial cameras and analysed and controlled from a distance of two kilometres.
To this end, the ICS group at HHI has developed, optimized, and evaluated a low-latency video processing chain for wired data transmission via Time-Sensitive Networking (TSN) and for wireless transmission via 5G. Furthermore, Fraunhofer HHI has developed machine vision AI models for the detection of hand positions and hand gestures, used to control the demo process and security features. Additional models were developed and optimized for the workpiece identification and its position tracking, as well as for spontaneous learning of new workpiece modifications by means of few-shot training. These models processed the video signals from the remote production cell in real time in the Edge Cloud.
Our AI-based machine vision modules and the actuator control were implemented as Docker instances on the FEC. By the means of our Multimodal Interaction Middleware all components were integrated in a decentralized manner to form a cyber-physical system for human-robot collaboration. As one of the first users of FEC, TSN and private 5G technologies, Fraunhofer HHI´s ICS Group implemented an ambitious video analysis demonstrator and gained extensive experience for future implementations. Our AI-based object and hand gesture recognition technologies are optimally designed for edge cloud infrastructure and, in combination with IPK actuator control, allow flexibility and cost reduction of future human-robot interaction scenarios.
We are looking for partners who would like to test innovative, real-time edge cloud applications with us in this unique test environment, and in particular address challenges in human-technology interaction e.g. between humans and robots.
Project Consortium
This project was partly funded by our industrial partner German Edge Cloud (GEC). The physical programmable logic controller (PLC) was replaced by Fraunhofer IPK with a virtualized programmable logic controller (vPLC) on the FEC. Fraunhofer IPK also implemented the production cell and programmed its actuators. Our colleagues from Fraunhofer FOKUS installed the TSN connection. Fraunhofer HHI experts from the Wireless Network (WN) department installed the 5G connection and the Photonic Netwok departments Digital Signal Processing Group, together with our industrial Partner GEC implemented the FEC at HHI premises.
About the Fraunhofer Edge Cloud
The question of trustworthy and secure data infrastructures has become increasingly important in recent years and months. Collecting, processing and sharing data enables new business models, faster product development and new forms of resilience in many areas.
In order to be able to offer industry and research such a data infrastructure, Fraunhofer and its research partner German Edge Cloud have developed the Fraunhofer Edge Cloud, an open test environment with which concepts of a distributed data infrastructure can be tested in practice and soon be implemented on an industrial scale.
The Fraunhofer Edge Cloud (FEC) consists of individual computing nodes that have been distributed and networked at individual institutes of the Fraunhofer-Gesellschaft. Thanks to the computing power of the individual nodes, it offers the advantages of cloud technologies together with the data security of operation in the company's own network. All data is analyzed within the FEC, the use of international hyperscalers is therefore not necessary.
