Concept Relevance Propagation

The research of the eXplainable AI group fundamentally focuses on the algorithmic development of methods to understand and visualize the predictions of state-of-the-art AI models.

In 2022, our group introduced Concept Relevance Propagation (CRP) [1] to explain the predictions of deep neural networks (DNNs) on the concept level in a human understandable way. As an extension of our Layer-wise Relevance Propagation (LRP) [2] method, CRP allows a deeper model understanding compared to traditional heatmap-based approaches. Whereas traditional heatmaps show the overall relevant input features for a prediction outcome, CRP disentangles the explanation into the contributions of individual concepts (learned by the AI) during individual predictions.

Concretely, CRP communicates which hidden concepts a model has learned, to which degree each concept contributes to a prediction, and precisely locates the concepts in the input data.

The application of CRP consists of a pre-processing step (that is performed once) and an ad-hoc step specific to the model prediction of choice.

In the pre-processing step, the example visualizations for concepts in the model's hidden space are collected from arbitrary data sources. Besides finding visualizations leading to strong stimuli (as is the common approach in the literature), we propose Relevance Maximization, which refines the visualizations showing more precisely how the model uses a concept.

In the ad-hoc step, firstly, a traditional XAI explanation is computed for a model prediction of choice by propagating relevance scores from the output to the input, layer by layer. During this backward pass, the relevances of latent concepts are determined.

To also precisely localize concepts in the input data, CRP introduces the idea of conditional relevance computation. Specifically, for a concept of interest, the backward pass is restricted to the concept's relevant paths, resulting in a concept-specific heatmap.

Together, CRP visualizes concepts and communicates for a prediction outcome of choice, which hidden concepts are relevant and precisely localizes the concepts in the input, rendering complex AI prediction processes readily interpretable to non-experts/AI users.

The concept-based explanations of CRP allow for deeper insights into the decision process of DNNs compared to traditional heatmaps, therefore improving the detection of potential model biases. In [3], we apply CRP to segmentation and object detection models to reveal concepts that focus on the background instead of the localized object, highlighting potential context bias. Targeting all kinds of biases, we present in [4] an Explainable AI life cycle with CRP to reveal and revise spurious model behavior.