April 2, 2020
© Fraunhofer HHI
Robots are being deployed in more and more situations, many of which involve collaboration between humans and robots – for example relieving humans of onerous tasks in the workplace. The challenges are how to integrate the robot into the working environment and how to operate it. In a joint project with Volkswagen AG the Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute, HHI, will demonstrate the advantages that the use of humanrobot collaboration (HRC) can bring to the inspection of welding seams in the automotive industry.
Flawless welding seams are a critical quality criterion in automotive engineering. In the future, welds will be inspected by a human-robot team, with each contributing its specific skills and expertise. Controlled by gesture and voice commands, the robot will hold and maneuver the specific components into position, while the employee marks and records any defects in weld Quality.
The joint research project EASY COHMO (Ergonomics Assistance Systems for Contactless Human-Machine-Operation) draws on the many years of experience that Fraunhofer HHI has accumulated in the fields of 3D capture, 3D information processing and 3D visualization. This system for the visual inspection of welding seams on key components in automobile production offers a great example of how HRC can function in the industry. In the coming years, this technology will be providing concrete assistance with inspection procedures at Volkswagen.
From manual inspection to Industry 4.0 Standards
The inspection process has remained unchanged for decades. First of all, each component has to be manually fixed in a rotary positioner so that it can be inspected from all angles. Inevitably, this involves the employee having to assume unergonomic positions, which may lead to repetitive strain injuries. Furthermore, only a short time is set aside for this complex procedure, which can have a negative impact on the quality of inspection.
At present, employees are free to conduct the inspection in their own manner. This inevitably results in a non-standardized procedure that makes it more difficult to recognize systematic defects on the basis of different inspectors’ observations. In addition, any such defects often go unrecorded or have to be laboriously entered into a separate system via a keyboard and mouse.
In the future, this inspection process will be interactive, with robots used to hold heavy components and maneuver them into a position in which employees can inspect them in an ergonomic manner. Such robots will have at least six axes and will therefore be able to move, turn and tilt a component in every possible direction. The robot will automatically remove a component from the line and present it for inspection. The employee will interact with the robot by means of explicit and implicit gestures, thereby steering the component into the desired position. Employees can concentrate their attention on identifying defects and, as a result, will overlook fewer defects.
Multimodal control
The middleware developed by Fraunhofer HHI coordinates the various sensors that serve to capture the overall work situation. Based on the employee’s position and gestures, the software calculates the required movement of the robotic arm. This also ensures user safety. For example, whenever the employee ceases to look directly at the component, the software halts the robotic arm as a precaution.
“The robot can also be set to respond to personalized instructions,” explains Paul Chojecki, project manager at Fraunhofer HHI. “Our new perceptual interface is able to process a user’s individual gestures and voice commands. This means the system can be quickly customized to a workstation’s specific requirements.”
Gestures are used to mark, categorize and confirm defects on the component. By the means of precise object tracking and projector based augmentation employees are provided with an interface directly on the component, in their working space area. This offers an efficient and intuitive way of generating a digital 3D record of any defects in the welding seams. By contrast, the current inspection procedure lacks any systematic record of defects, because there is very little formalized exchange of information between different employees. With the new procedure, any defects can be immediately entered into the system and then statistically evaluated. This means that systematic defects are more rapidly identified and can thereby be eliminated at the welding stage.
Extendible to all human-robot workstations
The system features a large number of sensors combined with multimodal controls based on an enhanced middleware along with customized operating instructions and machine learning. It is an approach that could well bring the breakthrough for further HRC applications – and also expand the scope for collaboration and interaction between humans and robots. Further areas of use include milling machines, for instance, or interactive robotic assistants in nonindustrial settings – e.g., in the healthcare or services sectors.
The joint project EASY COHMO is part of the 3Dsensation research alliance and is financed by the German Federal Ministry of Education and Research (BMBF) as part of the government funding program “Zwanzig20 – Partnerschaft für Innovationen.” Partners alongside Fraunhofer HHI (IMC group) include Carl Zeiss AG, Carl Zeiss Microscopy GmbH, Charité – Universitätsmedizin Berlin, Humboldt-Universität zu Berlin (Department of Computer Science), Pilz GmbH, Otto von Guericke University Magdeburg, Volkswagen AG and Zentrum für Bild- und Signalverarbeitung e.V.