K3VR

Objectives

The aim of the three-year research project is to develop an innovative, multi-sensory and artificial intelligence (AI)-supported virtual reality (VR) training environment in which police forces can explicitly train de-escalating communication. Another innovative feature is the recording of unconscious body reactions, which are made usable via AI-supported sensor evaluation and represent a fundamentally new approach to human-machine interaction.

Approach

A virtual training environment based on artificial intelligence is being created as part of the K3VR project. At the beginning, numerous interviews are conducted with police officers and the general public to identify communication patterns that favor or counteract the escalation of a situation. At the same time, the development of the VR-environment will begin, which will be used to investigate the effectiveness of the identified communication patterns under laboratory conditions. In the final phase of the project, the VR environment and the AI software will be further developed by the continuously learning AI. Emergency services will then be able to train the use of de-escalating communication virtually and realistically.

Innovations and Perspectives

With the help of this new training option, police officers can approach the public even better prepared when situations are charged with strong emotions such as fear, anger, or aggression. A possible escalation of these situations also depends on the communicative approach - verbally, but above all non-verbally through posture, facial expressions, or behavior. The use of de-escalating communication can increase the safety of both citizens and emergency services and ensure rapid assistance. The research project will create the scientific basis for making existing training courses more effective and efficient.

The data obtained from innovative technological and social science research makes it possible to train interactive artificial intelligence. This can be used to develop new training approaches for social interactions and communication training. The multi-sensory recording of partially unconscious processes and their effects, such as non-verbal communication through facial expressions, gestures, body language or voice tone, is innovative. For the first time, these can be made usable as an interaction tool using AI-supported sensor evaluation.

Contribution by Fraunhofer HHI to K3VR

In this project, the Interactive and Cognitive Systems (ICS) and Capture & Display Systems (CDS) groups are jointly developing an AI-based system to capture affective states and behaviors that have the potential to escalate or de-escalate a social conflict situation. Specifically, we are developing technologies that 1) neurophysiologically assess the subjectively perceived internal affective states, 2) recognize the contribution to the escalation dynamics of overt behavior, including gestures and facial expressions, and 3) integrate them into an interactive simulation environment for training security personnel.

To achieve this goal, key technical and scientific questions from the fields of neurotechnology, computer vision, machine learning, ergonomics and software engineering are addressed.

Assessment of inner affective states. These normally hidden internal arousal states can be captured via psychophysiological approaches such as electroencephalography, pupillometry, electrodermal activity and electrocardiography. As part of this project, the ICS group is developing real-time approaches that ensure a sufficient signal-to-noise ratio for mobile users and enable high synchronicity between sensors and displayed content. Such real-time analyses usually require machine learning methods to ensure a high signal-to-noise ratio in real time.

Recognition of overt and potentially escalatory behavior. Non-verbal communication can be analyzed on the basis of the video signal. The ICS group is developing AI-supported detection methods from the video stream, which, together with automated interpretation, enable the extraction of users' postures, gestures and facial expressions in relation to their social impact and their contribution to the escalation dynamics of human interaction.

Recognition of affective content in verbal expressions. Based on the extraction of relevant prosodic markers from the audio signal, the CDS group develops AI-based methods for the computer-aided automatic analysis of the escalation potential and the prediction of the social impact of verbal communication.

Data fusion and integration into interactive storybook. In the context of a conflict situation, the extracted objective information about the internal affective state, the potentially escalating behavior and the affective content in verbal utterances can represent markers for a potential escalation or de-escalation of this situation. If these markers are recognized in real time, they enable interactive navigation through the narrative tree within the storybook, resulting in an individual experience that is optimized to the user's learning level. The interactive simulation environment is experienced by the user through VR goggles, allowing for maximum immersion.

Assessment of the necessary degree of immersion. Another aim is to investigate the level of immersion required for the simulated training environment. Specifically, the full VR solution via a head-mounted display (VR goggles) will be compared with a less intrusive, panorama-based solution in our Fraunhofer HHI internal TiME Lab.

Project Consortium

• Akkon-University of Applied Human Sciences
• Fraunhofer HHI
• AspectEins GmbH

Associated Partners

• Berlin police
• Bavarian police
• Bavarian Center for Special Operations 

Publications

Nierula, B., Lafci, M. T., Prescher, A., Bosse, S. (2024, April 10-12). Responses in oscillatory brain activity and electrodermal activity to interpersonal space intrusion. Neural Traces 2024, Berlin, Germany.