Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
104 tokens/sec
GPT-4o
12 tokens/sec
Gemini 2.5 Pro Pro
40 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Robust Preference Optimization through Reward Model Distillation (2405.19316v2)

Published 29 May 2024 in cs.LG and cs.CL

Abstract: LLM (LM) post-training (or alignment) involves maximizing a reward function that is derived from preference annotations. Direct Preference Optimization (DPO) is a popular offline alignment method that trains a policy directly on preference data without the need to train a reward model or apply reinforcement learning. However, the empirical evidence suggests that DPO typically assigns implicit rewards that overfit, and trend towards infinite magnitude. This frequently leads to degenerate policies, sometimes causing even the probabilities of the preferred generations to go to zero. In this work, we analyze this phenomenon and use distillation to get a better proxy for the true preference distribution over generation pairs: we train the LM such that its induced implicit reward, i.e., the scaled log-likelihood ratio of the model to the reference model, matches an explicit reward model trained on the preference data. Moreover, to account for uncertainty in the reward model we are distilling from, we optimize against a family of reward models that, as a whole, is likely to include at least one reasonable proxy for the preference distribution. Our results show that distilling from such a family of reward models leads to improved robustness to distribution shift in preference annotations, while preserving the simple supervised nature of DPO.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (41)
  1. Direct preference optimization with an offset. arXiv preprint arXiv:2402.10571, 2024.
  2. Palm 2 technical report, 2023.
  3. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR, 2024.
  4. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. ISSN 00063444. URL http://www.jstor.org/stable/2334029.
  5. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning, pages 3852–3878. PMLR, 2022.
  6. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
  7. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.02743, 2023.
  8. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863, 2024.
  9. Helping or herding? Reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09244, 2023.
  10. Born again neural networks. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1607–1616. PMLR, 10–15 Jul 2018. URL https://proceedings.mlr.press/v80/furlanello18a.html.
  11. Impact of preference noise on the alignment performance of generative language models. arXiv preprint arXiv:2404.09824, 2024.
  12. Learn your reference model for real good alignment. arXiv preprint arXiv:2404.09656, 2024.
  13. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792, 2024.
  14. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. URL http://arxiv.org/abs/1503.02531.
  15. RL with KL penalties is better viewed as bayesian inference. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1083–1091, 2022.
  16. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179–1191, 2020.
  17. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023.
  18. Provably good batch off-policy reinforcement learning without great exploration. Advances in neural information processing systems, 33:1264–1274, 2020.
  19. Confronting reward model overoptimization with constrained rlhf. arXiv preprint arXiv:2310.04373, 2023.
  20. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228, 2024.
  21. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.19159, 2024.
  22. From r𝑟ritalic_r to q∗superscript𝑞q^{*}italic_q start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT: Your language model is secretly a q-function. arXiv preprint arXiv:2404.12358, 2024a.
  23. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024b.
  24. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36, 2024.
  25. WARM: On the benefits of weight averaged reward models. arXiv preprint arXiv:2401.12187, 2024.
  26. Fitnets: Hints for thin deep nets. In In Proceedings of ICLR, 2015.
  27. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
  28. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023.
  29. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020.
  30. Preference fine-tuning of llms should leverage suboptimal, on-policy data. arXiv preprint arXiv:2404.14367, 2024.
  31. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749, 2024.
  32. TL;DR: Mining Reddit to learn automatic summarization. In Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu, editors, Proceedings of the Workshop on New Frontiers in Summarization, pages 59–63, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/W17-4508. URL https://aclanthology.org/W17-4508.
  33. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34:6683–6694, 2021.
  34. Is DPO superior to PPO for LLM alignment? a comprehensive study. arXiv preprint arXiv:2404.10719, 2024.
  35. Training deep neural networks in generations: a more tolerant teacher educates better students. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’19/IAAI’19/EAAI’19. AAAI Press, 2019. ISBN 978-1-57735-809-1. doi: 10.1609/aaai.v33i01.33015628. URL https://doi.org/10.1609/aaai.v33i01.33015628.
  36. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954–28967, 2021.
  37. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems, 34:13626–13640, 2021.
  38. Token-level direct preference optimization. arXiv preprint arXiv:2404.11999, 2024.
  39. Uncertainty-penalized reinforcement learning from human feedback with diverse reward lora ensembles. arXiv preprint arXiv:2401.00243, 2023.
  40. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023.
  41. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf. arXiv preprint arXiv:2401.16335, 2024.
Citations (11)
View on Semantic Scholar

Summary

  • The paper introduces reward model distillation to mitigate DPO's overconfidence and misalignment issues.
  • It uses classical knowledge distillation with a pessimistic extension to align LM outputs with a reward-based distribution.
  • Empirical results, including TL;DR summarization, show improved robustness over traditional offline preference optimization methods.

Robust Preference Optimization Through Reward Model Distillation

The paper "Robust Preference Optimization through Reward Model Distillation," authored by Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant, explores the landscape of LLM alignment methods. It specifically addresses the limitations of Direct Preference Optimization (DPO), proposing an alternative methodology that couples explicit reward modeling with a distillation approach, aiming for improved robustness in LLM policies.

Background and Motivation

LLM (LM) post-training or alignment often utilizes reward learning from human feedback via preference annotations. Traditional methods rely on reinforcement learning from human feedback (RLHF) to optimize these models. DPO, although popular for being an efficient offline method, tends to suffer from overconfidence issues and misalignment due to idiosyncrasies in preference data. The paper advocates for the importance of explicit reward modeling even in offline settings and proposes a method that merges the advantages of both direct preference optimization and reward model distillation. The objective is to mitigate issues arising from distribution shifts in preference annotations while maintaining simple and supervised training frameworks.

Methodology

The proposed method leverages classical knowledge distillation techniques, reformulating the problem to match the LLM's output probability distribution to a distribution derived from a reward model, which has itself been trained on preference data. The approach entails training the LM to produce probabilities that align with an underlying reward model distribution. This method provides a regularization effect, addressing the overconfidence issue in DPO.

Reward Model Distillation

The core of the approach involves distillation from a reward model under uncertainty. The reward model serves as a proxy for the true preference distribution. The paper presents the theoretical underpinnings that relate this distillation approach to optimization in classical reinforcement learning settings, demonstrating that adequately diverse samples from preference data can enable a straightforward transition from optimizing an RLHF objective to a distillation loss.

Pessimistic Distillation

To further refine robustness, the paper introduces a "pessimistic" extension of the reward model distillation. This extension optimizes policy alignment by considering the worst-case performance across a family of plausible reward models drawn from preference data. The technique is inspired by conservative offline RL methods and adds KL-divergence regularization to ensure that the policy remains stable even when faced with biased or noisy preference annotations.

Theoretical and Empirical Analysis

The paper extensively analyzes the degenerative tendencies of DPO using theoretical constructs. It finds that DPO can lead to policies that heavily overfit to training preference data, sometimes resulting in degenerate policies that ignore useful responses present in training data. This overfitting is mitigated by the proposed distillation and pessimistic distillation methods, leading to more reliable and robust policy outputs.

Empirical results further support these theoretical insights. Comparisons are drawn using the TL;DR summarization task, where preference data introduces a spurious bias between the length of summaries and their preference scores. The distilled and pessimistic methods show significant improvements in alignment performance over DPO and Identity Preference Optimization (IPO), particularly in scenarios where the preference datasets show biases.

Implications and Future Directions

The research presented in this paper has significant practical and theoretical implications. Practically, it suggests that the overarching strategy for LM alignment should not solely rely on direct optimization from preference data but should incorporate robust distillation techniques that account for uncertainty and potential biases in the data. Theoretically, it opens paths for future research into combining offline and online methods for more effective and efficient LM post-training.

The approach has shown promise in yielding robust policies that outperform traditional DPO and IPO methods, particularly in complex scenarios involving biased preference data. Future research could explore broader application domains, evaluate varying formats of distributional shifts, and refine the balance between reward model fidelity and computational efficiency.

In conclusion, the exploration of reward model distillation—both regular and pessimistic—marks a significant stride in the robustness and efficacy of LLM alignment. By combining the benefits of explicit reward models with the simplicity and efficiency of direct preference optimization, this approach promises a more stable path toward robust AI systems.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown