Problems with Shapley-value-based explanations as feature importance measures (2002.11097v2)

Abstract: Game-theoretic formulations of feature importance have become popular as a way to "explain" machine learning models. These methods define a cooperative game between the features of a model and distribute influence among these input elements using some form of the game's unique Shapley values. Justification for these methods rests on two pillars: their desirable mathematical properties, and their applicability to specific motivations for explanations. We show that mathematical problems arise when Shapley values are used for feature importance and that the solutions to mitigate these necessarily induce further complexity, such as the need for causal reasoning. We also draw on additional literature to argue that Shapley values do not provide explanations which suit human-centric goals of explainability.

• The paper highlights that the axiomatic foundations of Shapley values can misrepresent feature contributions, especially in non-additive models.
• It identifies practical challenges with choosing between interventional and conditional distributions, leading to potential misinterpretations of model behavior.
• The paper recommends developing tailored, human-centric explanation methods to deliver actionable, contrastive insights beyond aggregated numerical attributions.

Analysis of Shapley-value-based Explanations as Feature Importance Measures

The paper "Problems with Shapley-value-based explanations as feature importance measures" provides a critical examination of the use of Shapley values for feature importance in machine learning models. It addresses both theoretical and practical concerns, highlighting that, although Shapley values are grounded in robust game-theoretic principles, their application in feature importance produces several significant challenges.

The fundamental critique stems from the mathematical constraints and assumptions inherent in using Shapley values in this context. The traditional application of Shapley values as a solution to cooperative game theory assumes additivity, symmetry, and a dummy feature property, which are leveraged as justifications for their use as feature importance measures. However, the paper argues that these axiomatic properties may not align with meaningful interpretations of model explanations.

Key Mathematical Concerns

• Interventional vs. Conditional Distributions: The paper identifies the dilemma in choosing between interventional and conditional value functions. Interventional methods require evaluating models on out-of-distribution data, which can be misleading because models are not validated on such data regions. Conditional approaches, while more aligned with the data distribution, necessitate significant modeling of feature interdependencies, potentially introducing complexity and inaccuracies in feature attribution.
• Axiomatic Limitations: The adherence to the Shapley axioms, particularly additivity, imposes limitations on explanations. For non-additive models, where interactions between features are critical, Shapley values can produce unintuitive results. For instance, multiplicative models might attribute equal importance to all features regardless of their specific contributions, which is conceptually misleading.

Human-centric Explanability Issues

The practical applicability of Shapley values in aiding human understanding of model decisions is questioned. The approach offers limited value in answering the typical "why-question" that practitioners seek: why a particular decision was made over another. Furthermore, the averaged marginal contributions do not naturally align with actionable or intuitive explanations that decision-makers or affected individuals may require.

• Contrastive Explanation: Humans typically prefer contrastive explanations, which compare why one event happened over another potential event. Shapley values struggle to offer such directed, contrastive insights because they diffuse the importance over all permutations of feature subsets instead of specific alternative scenarios.
• Operationalization for Decision-making: The paper notes that while Shapley values provide a numeric measure of influence, they fail at offering concrete guidance for action. This is particularly evident in scenarios where stakeholders need directions on how to alter inputs to achieve a desired outcome.

Implications for Practice and Future Work

While Shapley-value-based methods have become popular for explaining machine learning models, the paper advocates for caution in their uncritical adoption. It underscores the necessity of clear interpretative guidelines and the development of model-specific and task-specific methods that focus on end-user requirements. Importantly, this research suggests that future work should prioritize establishing a more direct alignment between mathematical formulations and the practical needs of model interpretation, such as actionable insights and alignment with human intuition.

Practical Recommendations

• Model-specific Explanation Development: Tailor interpretative methods to align with the specific characteristics of the model architecture and the task at hand. Avoid blindly applying game-theoretic solutions without adjusting for model-specific dynamics.
• Focus on Human-Centric Interpretability: Develop explanation tools that prioritize user-defined contrastive questions and actionable insights over mathematically complex feature attributions that may not lead to improved understanding or outcomes.

In conclusion, the analysis presented in the paper illuminates the critical gaps in the current use of Shapley values for model explanation, mapping clear pathways for more nuanced, context-aware interpretative strategies that satisfactory address both theoretical foundations and practical human-centric needs.