Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability
(2401.08574)Abstract
While language models (LMs) can sometimes generate factually correct text and estimate truth values of individual claims, these generally do not reflect a globally coherent, manipulable model of the world. As a consequence, current LMs also generate incorrect or nonsensical content, and are difficult to edit and bring up to date. We present a method called Deductive Closure Training (DCT) that uses LMs themselves to identify implications of (and contradictions within) the text that they generate, yielding an efficient self-supervised procedure for improving LM factuality. Given a collection of seed documents, DCT prompts LMs to generate additional text implied by these documents, reason globally about the correctness of this generated text, and finally fine-tune on text inferred to be correct. Given seed documents from a trusted source, DCT provides a tool for supervised model updating; if seed documents are sampled from the LM itself, DCT enables fully unsupervised fine-tuning for improved coherence and accuracy. Across the CREAK, MQUaKE, and Reversal Curse datasets, supervised DCT improves LM fact verification and text generation accuracy by 3-26%; on CREAK fully unsupervised DCT improves verification accuracy by 12%. These results show that LMs' reasoning capabilities during inference can be leveraged during training to improve their reliability.
Overview
-
The paper introduces Deductive Closure Training (DCT), a fine-tuning procedure aimed at enhancing language models (LMs) for coherence, accuracy, and updatability by leveraging their inference capabilities.
-
DCT involves an iterative process where the LM generates documents from seed inputs, evaluates their logical coherence and truthfulness, and is fine-tuned on these inferred-true documents to improve factual accuracy.
-
Comprehensive evaluations demonstrate DCT's effectiveness in improving fact verification, model updating, question answering, and addressing logical consistency issues compared to traditional fine-tuning and state-of-the-art methods.
Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability
The paper "Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability" introduces a new fine-tuning procedure dubbed Deductive Closure Training (DCT). The primary objective of DCT is to enhance language models (LMs) in terms of factual coherence and updatability by ensuring that they maintain a logically consistent and complete model of the world. This method leverages the models' existing inference capabilities to refine and improve their own generated predictions without the need for extensive external supervision.
Methodology
DCT operates through an iterative cycle of interaction between seeded input documents and model-generated text. The process can be broken down into several distinct stages:
- Document Generation: Starting with a collection of seed documents, DCT uses the LM to generate related documents. These documents can be either directly implied by the seed documents or contradictory to them, effectively broadening the deductive neighborhood of the initial facts.
- Consistency Evaluation: The LM evaluates the generated documents for logical coherence and truthfulness. This involves assessing each document's probability of being correct in conjunction with the other generated documents. The objective is to identify a subset of documents that form a logically consistent set while maximizing the probability of correctness.
- Fine-Tuning: The model is then fine-tuned on the inferred-true documents, adjusting its parameters to better reflect the coherent and complete set of factual information.
- Seed Data Sources: Depending on the application, the seed documents can come from various origins: trusted external sources for supervised adaptation, new factual updates for model editing, or even the LM's own outputs for unsupervised fine-tuning.
Experimental Results and Analysis
The paper presents comprehensive evaluations of DCT across several benchmarks:
- Fact Verification: Using the CREAK dataset, DCT significantly improved the model's ability to classify factual claims correctly. In unsupervised settings, DCT increased fact verification accuracy by up to 12%, while in supervised settings, the improvement reached up to 26%.
- Model Updating and QA: On the counterfactual subset from the MQaUE dataset, DCT exhibited superior performance in propagating updates to the model, substantially outperforming both fine-tuning and state-of-the-art retrieval methods like MeLLo. The generated correlative implications were particularly effective, illustrating DCT's ability to incorporate and propagate new knowledge through logical reasoning.
- Reversal Curse Benchmark: DCT also showed promise in addressing issues of logical consistency and implication recognition, performing better than traditional fine-tuning methods in recognizing and correctly answering reversed and cloze-style prompts involving simple logical implications.
Theoretical Implications and Future Directions
The theoretical underpinning of DCT is supported by a formal analysis demonstrating that under certain conditions, DCT is guaranteed to improve model accuracy. This highlights a fundamental characteristic: some factual inaccuracies in LMs arise not purely from data limitations but from the inadequacies in training algorithms. Integrating reasoning at the training stage bridges this gap, optimizing for a model that is both logically coherent and factually correct.
Future research could explore extensions of DCT:
- Probabilistic and Contrastive Approaches: Incorporating probabilistic methods to handle uncertainty in document evaluation or employing contrastive learning objectives to differentiate correct from incorrect statements more effectively.
- Enhanced Logical Inference: Extending the inference capabilities to multi-hop or more complex logical reasoning tasks could further enhance the model's robustness and applicability in diverse domains.
- Application in Misinformation Detection: Given its strengths in factual consistency, DCT could be adapted to detect and mitigate the spread of misinformation by ensuring the generation of logically consistent true facts.
Practical Implications
Practically, DCT represents a significant step toward more reliable and updatable LMs. It paves the way for better fact-checking tools and more dependable AI systems capable of integrating and acting upon new information dynamically. Whether applied for model refinement, continuous learning, or real-time updates in mission-critical applications, the DCT method offers a potent mechanism to align LMs closely with trustworthy data sources and evolving knowledge bases.
In conclusion, Deductive Closure Training presents a robust and innovative approach to addressing fundamental challenges in language model training. By leveraging the model's reasoning abilities, DCT ensures that LMs can be both factually accurate and logically coherent, thus achieving significant strides in model reliability and factual correctness.
Create an account to read this summary for free: