Decompose-and-Compose: A Compositional Approach to Mitigating Spurious Correlation

Decompose-and-Compose: Enhancing Robustness to Spurious Correlation in Image Classification

Introduction

Empirical Risk Minimization (ERM) has been the cornerstone of training strategies for image classification tasks, particularly when the goal is to optimize performance on in-distribution (ID) samples. However, the effectiveness of ERM wanes in the face of out-of-distribution (OOD) samples, primarily due to its vulnerability to spurious correlations. This limitation is exacerbated in realistic scenarios where images comprise both causal and non-causal components relative to the target label. The paper undertakes the challenge of spurious correlation through a novel approach dubbed Decompose-and-Compose (DaC), which focuses on balancing groups by intervening on non-causal components of images to generate new, counterfactual examples. This method proposes a solution that does not require explicit knowledge of group labels or spurious features, making it a significant step forward in developing robust models capable of generalizing across varied distributions.

Methodology

The foundation of DaC lies in two critical observations:

• Models trained with ERM tend to focus either on causal parts or on parts that exhibit a high spurious correlation with the target label, especially on samples where the model is confident about its predictions.
• A granular, compositional analysis, facilitated by class activation maps, indicates that the emphasis (causal or spurious) depends on the relative ease of predicting the label based on these components.

This leads to the inception of DaC, which operates by first identifying and decomposing images into causal and non-causal parts. This identification process leverages the class activation maps of models pre-trained with ERM. The pivotal aspect of DaC is the generation of novel data points through the combination of deconstructed image parts from different samples, followed by retraining the model on this augmented dataset, which incorporates these synthetically generated counterfactual instances.

Evaluation

DaC is evaluated against baseline methods such as DFR, MaskTune, and Group DRO on benchmarks including Waterbirds, CelebA, MetaShift, and Dominoes datasets. These benchmarks are selected to cover a range of distribution and spurious correlation challenges.

The contributions are multi-fold, emphasizing:

• A detailed analysis of ERM-trained models' behavior, particularly noting their tendency to focus on either causal or spuriously correlated components based on confidence and loss metrics.
• An innovative method to identify causal parts within images, capitalizing on the insights obtained from the models' attribution maps.
• A group-balancing strategy reliant on the strategic combination of image parts to construct new, balanced training data points without the necessity for group label information.
• Superior performance in worst-group accuracy metrics across a majority of the considered benchmarks, showcasing DaC's efficacy in mitigating the adverse effects of spurious correlation.

Implications and Future Directions

The insights and methodology introduced in this paper have profound implications for both theoretical and practical advancements in AI. Theoretically, DaC provides a nuanced understanding of how models attend to various components of an image and how this attention can be manipulated to foster more robust learning. Practically, the ability to enhance model robustness without explicit reliance on group labels or detailed knowledge of spurious features has far-reaching applications across different domains where robustness and generalizability are paramount.

As for future research avenues, the exploration could extend into:

• Refining the decomposition and composition mechanisms to increase the precision of causal and non-causal part identification.
• Applying DaC's principles to other forms of data beyond images, such as text or audio, where spurious correlation is also a significant challenge.
• Investigating the intersection of DaC with other robustness-enhancing techniques like adversarial training, to further bolster model resilience.

Conclusion

This work stands as a testament to the evolving understanding of how models perceive and process information, moving beyond the monolithic treatment of images towards a more dissected and recombinant strategy. Decompose-and-Compose not only sheds light on the nuanced dynamics of spurious correlation but also provides a practical toolkit to address it, signaling a shift towards developing AI systems that are not only performant but robust and fair by design.