On Early Detection of Hallucinations in Factual Question Answering (2312.14183v3)

Abstract: While LLMs have taken great strides towards helping humans with a plethora of tasks, hallucinations remain a major impediment towards gaining user trust. The fluency and coherence of model generations even when hallucinating makes detection a difficult task. In this work, we explore if the artifacts associated with the model generations can provide hints that the generation will contain hallucinations. Specifically, we probe LLMs at 1) the inputs via Integrated Gradients based token attribution, 2) the outputs via the Softmax probabilities, and 3) the internal state via self-attention and fully-connected layer activations for signs of hallucinations on open-ended question answering tasks. Our results show that the distributions of these artifacts tend to differ between hallucinated and non-hallucinated generations. Building on this insight, we train binary classifiers that use these artifacts as input features to classify model generations into hallucinations and non-hallucinations. These hallucination classifiers achieve up to $0.80$ AUROC. We also show that tokens preceding a hallucination can already predict the subsequent hallucination even before it occurs.

• The paper introduces binary classifiers using softmax, integrated gradients, self-attention, and fully-connected activations to detect hallucinations in LLM outputs with up to 0.82 AUROC.
• It leverages internal model artifacts to differentiate between accurate and hallucinatory responses by analyzing confidence, token attribution, and activation clustering.
• Experimental results across datasets like TriviaQA reveal that self-attention and fully-connected activations are most effective for early detection of hallucinated answers.

On Early Detection of Hallucinations in Factual Question Answering

Introduction

The paper "On Early Detection of Hallucinations in Factual Question Answering" (2312.14183) explores the challenge of detecting hallucinations in responses generated by LLMs. The fluency of these models can produce coherent text even when they provide incorrect factual information, termed as hallucinations. The work investigates the feasibility of early detection of hallucinations by analyzing artifacts derived from various stages of the LLM's question-answering pipeline.

Artifacts for Hallucination Detection

The paper systematically analyzes several types of artifacts associated with LLM generations: Softmax probabilities, Integrated Gradients (IG) feature attributions, self-attention scores, and fully-connected layer activations. These artifacts explore different aspects of the model's internal state and output behavior:

Softmax Probabilities

The Softmax probabilities at the output level are a direct measure of model confidence. The paper posits that hallucinations tend to correlate with higher entropy values in the Softmax probability distributions, suggesting a lower certainty in the model's predictions.

Figure 1: Softmax probability distributions for hallucinated versus non-hallucinated responses, highlighting differences in confidence.

Integrated Gradients Attributions

IG attributions provide insights into which input tokens are deemed important for the generation of specific output tokens. The paper hypothesizes that hallucinated outputs exhibit dispersed attribution scores over the input tokens rather than focusing on key tokens, indicating potential uncertainty or irrelevance.

Figure 2: IG Attributions showing differences in token importance between hallucinated and accurate generations.

Self-Attention Scores and Fully-Connected Activations

Self-attention scores and fully-connected activations offer a window into the model's internal processing states. Variations in these activations between hallucinated and non-hallucinated outputs can be detected, especially at the deeper layers of the Transformer model. The paper demonstrates that these activations vary significantly, which can be leveraged to identify hallucinations.

Figure 3: TSNE clustering of self-attention scores differentiating hallucinated from non-hallucinated outputs.

Methodology

The authors develop binary classifiers trained on the aforementioned artifacts to categorize model outputs into hallucinations and non-hallucinations. These classifiers, built using different combinations of the artifacts, achieve AUROC scores as high as 0.82 for certain datasets and models, demonstrating significant potential for early hallucination detection. The classifiers highlight the effectiveness of self-attention and fully-connected activation scores in discerning hallucinations across various dataset types.

Experimental Setup

The evaluation encompasses a variety of datasets, including T-REx with subject-specific questions, and the broader TriviaQA dataset. The models tested include variants of OpenLLaMA, OPT, and Falcon, catering to different parameter sizes and architectures. Each experiment rigorously assesses both the base model tasks and the hallucination detection capabilities of the artifacts.

Results

Qualitative findings emphasize the discernible differences in entropy and clustering between artifacts generated from hallucinated versus correct responses. Quantitatively, self-attention scores and fully-connected activations consistently outperform IG attributions and Softmax probabilities in hallucination detection, achieving over 0.70 AUROC across most conditions.

Figure 4: AUROC scores for different hallucinatory detectors using self-attention and fully-connected activations.

Conclusion

The demonstrated efficacy of early hallucination detection through model artifacts paves the way for enhanced reliability in LLM applications. Real-world implementations could leverage these classifiers to flag potential inaccuracies in model outputs before reaching the user, contributing to increased trust and applicability in critical domains such as web search and information retrieval. Future work may explore combining artifact-based detection with advanced retrieval mechanisms and fine-tuning strategies for broader applicability and robustness.