Chain of Hindsight Aligns Language Models with Feedback (2302.02676v8)

Abstract: Learning from human preferences is important for LLMs to match human needs and to align with human and social values. Prior works have achieved remarkable successes by learning from human feedback to understand and follow instructions. Nonetheless, these methods are either founded on hand-picked model generations that are favored by human annotators, rendering them inefficient in terms of data utilization and challenging to apply in general, or they depend on reinforcement learning, which often suffers from imperfect reward functions and relies on extremely challenging optimizations. In this work, we propose a novel technique, Chain of Hindsight, that is easy to optimize and can learn from any form of feedback, regardless of its polarity. Our idea is inspired by how humans learn from extensive feedback presented in the form of languages. We convert all types of feedback into sequences of sentences, which are then used to fine-tune the model, allowing us to take advantage of the language comprehension capabilities of LLMs. We condition the model on a sequence of model generations paired with feedback. By doing so, the model is trained to generate outputs based on feedback, while learning to identify and correct negative attributes or errors. Applying our method to LLMs, we observed that Chain of Hindsight significantly surpasses previous methods in aligning LLMs with human preferences. We report significant improvements on summarization and dialogue benchmarks, with our approach markedly preferred in human evaluations.

• The paper introduces Chain of Hindsight, a novel approach that integrates natural language feedback into LLM finetuning.
• It blends the strengths of SFT and RLHF by using feedback sequences for error correction and improved output quality.
• Evaluation demonstrates that CoH significantly outperforms traditional methods in aligning models with human preferences in summarization and dialogue tasks.

Chain of Hindsight: Aligning LLMs with Feedback

The paper "Chain of Hindsight aligns LLMs with Feedback" introduces a novel approach for enhancing the alignment of LLMs with human preferences. This work is situated within the ongoing effort to make LLMs more attuned to human values by learning from human feedback, a crucial aspect for the models’ broader acceptance in society.

The primary innovation presented in the paper is the Chain of Hindsight (CoH), which proposes an efficient technique for finetuning LLMs by leveraging human feedback in the form of natural language comparisons. Unlike conventional methods reliant on selective positive feedback data or reinforcement learning (RLHF), which can be cumbersome and imperfect, CoH utilizes a straightforward optimization process to condition models on sequences of outputs paired with feedback. These sequences enable the models to learn to correct errors based on comparisons and improve upon negative attributes of the generated outputs.

The paper outlines that existing methods like Supervised Finetuning (SFT) and RLHF have limitations. SFT hinges on labeled datasets that emphasize positively-rated outputs, subsequently limiting the model's exposure to error correction paradigms. RLHF, while more generalized in terms of data utilization, demands meticulous reward function design and challenging optimization processes. CoH innovatively synergizes the strengths of both methods without inheriting their respective drawbacks.

In practical terms, CoH transforms all forms of feedback—positive or negative—into sequences, allowing LLMs to use their inherent language comprehension abilities to optimize and fine-tune their outputs. This is achieved by presenting the models with feedback alongside previous model generations, which guides them through comparative analysis towards more aligned outputs.

The researchers demonstrate that CoH significantly outperforms SFT, Conditional SFT, SFT with unlikelihood loss, and state-of-the-art RLHF baselines on tasks involving summarization and dialogue, as evidenced by both human evaluation and automated metrics. Notably, CoH integrates feedback through natural language descriptions, enhancing the models' flexibility and scalability. For example, when conditioned on feedback indicators like 'Good' or 'Bad', models using CoH produce more accurate, coherent, and comprehensive summaries compared to baselines.

An important contribution of the paper is the scalability of the CoH method. Unlike RLHF, CoH retains the training objective consistency with the pretraining phase, suggesting a much broader applicability and ease of integration into existing LLM training pipelines. Furthermore, CoH's ability to incorporate both fine-grained language feedback and binary feedback without requiring reinforcement signals indicates its potential to significantly reduce the alignment tax typically associated with preference-trained models.

The implications of this work are substantial both theoretically and practically. Theoretically, CoH provides a robust framework for feedback integration, offering a methodological departure from reliance on hand-engineered reward functions. Practically, it points towards more resource-effective and scalable ways to align LLMs with human norms, potentially making these models more reliable for real-world applications.

Future developments could explore the integration of external feedback sources beyond human comparative feedback, such as technical evaluations and user-generated performance metrics, which could further bolster the efficacy and alignment potential of LLMs. In conclusion, the Chain of Hindsight presents a promising paradigm for aligning LLMs with human preferences, offering significant improvements and efficiencies over existing approaches. Its continued development and application could mark a substantial step forward in the responsible deployment of AI systems.