Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control

Evaluating Sparse Feature Dictionaries in Language Models

Introduction

When it comes to understanding what goes on inside LLMs like GPT-2, interpretability is a big deal. The core idea here is to disentangle the complex representations these models use into meaningful features. Recently, sparse autoencoders (SAEs) have been suggested as a promising way to achieve this. The paper we're discussing dives deep into validating these methods by comparing them against dictionaries built using supervised features—essentially, features we "know" to be meaningful beforehand.

What’s the Problem?

We don't have a clear "ground truth" for what these model features should be, making it a tough job to evaluate new methods. This paper's approach is to create a framework for evaluating these feature dictionaries on specific tasks, comparing them against supervised dictionaries for context. They focus on the indirect object identification (IOI) task using GPT-2 Small to find out how well these dictionaries perform.

The Core Approach

To break it down, here’s the framework the researchers proposed:

1. Supervised Feature Dictionaries: First, they demonstrate that supervised dictionaries can do an excellent job of approximating, controlling, and interpreting model computations.
2. Contextualizing Unsupervised Dictionaries: They then use these supervised dictionaries to benchmark unsupervised ones (like those learned via SAEs).

Key Results

They trained SAEs on datasets specific to the task (IOI) and a larger, general dataset (OpenWebText). Here’s what they found:

• Interpretability: Both task-specific and full-dataset SAEs capture interpretable features, but task-specific ones were better at this.
• Sparse Controllability: Supervised dictionaries allowed for more precise editing of features to change model behavior compared to SAEs. Task-specific SAEs fared better than general ones but were still not as good as supervised features.
• Occlusion and Over-splitting: They noticed two phenomena in SAE training:
• Feature Occlusion: Important features can be overshadowed by even slightly stronger features.
• Feature Over-splitting: Binary features can get split into several smaller features, making them less interpretable.

Practical Implications

These findings imply that while SAEs can indeed produce some meaningful and interpretable features, there's still a way to go before they can match the precision and control offered by supervised methods. This matters because the better we can understand what our models are doing, the more effectively we can use and refine them.

Future Directions

Future work could involve tuning SAE training procedures to improve their performance or exploring ways to lessen the issues of occlusion and over-splitting. Also, it might be valuable to apply this evaluation framework to a wider range of tasks and models, making it more robust and generalizable.

Conclusion

In a nutshell, the paper provides a robust framework for evaluating the effectiveness of sparse feature dictionaries in LLMs. It sets a benchmark using supervised features and uses it to measure how well unsupervised methods like SAEs stack up. While SAEs show promise, there's still a gap between them and the more reliable supervised methods. The ongoing challenge is to close this gap, making LLMs more interpretable and controllable along the way.