Prototypical Concept-based Explanations

Concretely, PCX allows to

• quickly understand model (sub-)strategies in detail on the concept level via prototypes
• spot spurious model behavior or data quality issues
• catch outlier predictions or assign valid predictions to prototypes
• identify missing and overused features in a prediction w.r.t. the prototypical behavior.

PCX prototypes are computed by first generating concept-based explanations using, e.g., our CRP method, on the whole training set. For each sample and the corresponding model prediction outcome, we here receive relevance scores for the concepts used by the model. Having generated and collected all explanations, we continue with fitting a Gaussian Mixture Model (GMM) on the respective distributions for each output class. A single Gaussian distribution of a GMM then corresponds to one prototype that summarizes multiple similar explanations. To visualize prototypes, we show the closest sample to a Gaussian's mean in the training set as a prototypical example.

By inspecting the PCX prototypes together with the explanation distribution as in Fig. 1 (left), we get a quick idea of the prediction strategies the model has learned. In such a “strategy map’’, similar explanations are closely located. As our prototypes are based on concept-based explanations, we can now dive deeper and study the characteristic concepts of each prototype.

Further, during deployment, PCX allows for more objective explanations that are also practical for model (prediction) validation. Concretely, with PCX, we can understand for single predictions their (a-)typicality. On the one hand, PCX allows studying the difference to the expected model behavior qualitatively in terms of concepts. Notably, contrary to many other explainability methods, we can understand not only which features are present, but also missing w.r.t. prototypes, as illustrated in Fig. 1 (right). On the other hand, as prototypes are based on Gaussian distributions, the probability density function of a distribution allows for a quantitative likelihood measure. The less usual the concepts used for an output class, the lower the measured likelihood.

Concept-based prototypes of PCX allow for a detailed yet quick understanding of the decision-making processes of DNNs, requiring significantly less human effort compared to studying single prediction explanations. In [2], we showcase PCX and find sub-strategies and data quality issues/biases on public benchmark datasets. Notably, the identification of spurious behavior is a key step for ensuring trust and safety in AI. Further, PCX also opens up new possibilities for knowledge discovery in science applications.