Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
166 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Training Language Models to Generate Text with Citations via Fine-grained Rewards (2402.04315v3)

Published 6 Feb 2024 in cs.CL

Abstract: While recent LLMs have proven useful in answering user queries, they are prone to hallucination, and their responses often lack credibility due to missing references to reliable sources. An intuitive solution to these issues would be to include in-text citations referring to external documents as evidence. While previous works have directly prompted LLMs to generate in-text citations, their performances are far from satisfactory, especially when it comes to smaller LLMs. In this work, we propose an effective training framework using fine-grained rewards to teach LLMs to generate highly supportive and relevant citations, while ensuring the correctness of their responses. We also conduct a systematic analysis of applying these fine-grained rewards to common LLM training strategies, demonstrating its advantage over conventional practices. We conduct extensive experiments on Question Answering (QA) datasets taken from the ALCE benchmark and validate the model's generalizability using EXPERTQA. On LLaMA-2-7B, the incorporation of fine-grained rewards achieves the best performance among the baselines, even surpassing that of GPT-3.5-turbo.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511, 2023.
  2. Language models are few-shot learners. In Advances in Neural Information Processing Systems.
  3. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314.
  4. Eli5: Long form question answering. In Association for Computational Linguistics (ACL), pages 3558––3567.
  5. Rarr: Researching and revising what language models say, using language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page 16477–16508.
  6. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6465––6488.
  7. Realm: Retrieval augmented language model pre-training. In Proceedings of International Conference on Machine Learning, 2020, pages 33–40.
  8. Rethinking with retrieval: Faithful large language model inference. arXiv preprint arXiv:2301.00303, 2022.
  9. True: Re-evaluating factual consistency evaluation. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 3905–3920.
  10. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  11. Atlas: Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299, 2022.
  12. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1––38.
  13. Active retrieval augmented generation. arXiv preprint arXiv:2305.06983, 2023.
  14. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations.
  15. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
  16. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems.
  17. Towards verifiable generation: A benchmark for knowledge-aware language model attribution. arXiv preprint arXiv:2310.05634.
  18. Ra-dit: Retrievalaugmented dual instruction tuning. arXiv preprint arXiv:2310.01352, 2022.
  19. Sail: Search-augmented instruction learning. arXiv preprint arXiv:2305.15225, 2023.
  20. Expertqa: Expert-curated questions and attributed answers. arXiv preprint arXiv:2309.07852, 2023.
  21. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147, 2022.
  22. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251v1.
  23. Webgpt: Browser-assisted question answering with human feedback. arXiv preprint arXiv:2112.09332, 2021.
  24. Large dual encoders are generalizable retrievers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9844––9855.
  25. Talm: Tool augmented language models. arXiv preprint arXiv:2205.12255, 2022.
  26. The web is your oyster - knowledge-intensive nlp against a very large web corpus. arXiv preprint arXiv:2112.09924, 2021.
  27. Mauve: Measuring the gap between neural text and human text using divergence frontiers. In Advances in Neural Information Processing Systems.
  28. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research (JMLR), 21(140).
  29. In-context retrieval-augmented language models. In Transactions of the Association for Computational Linguistics, 2023.
  30. Qampari: An open-domain question answering benchmark for questions with many answers from multiple paragraphs. arXiv preprint arXiv:2205.12665, 2022.
  31. Efficient rlhf: Reducing the memory usage of ppo. arXiv preprint arXiv:2309.00754, 2023.
  32. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023.
  33. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
  34. Asqa: Factoid questions meet long-form answers. arXiv preprint arXiv:2204.06092, 2022.
  35. Conditionalqa: A complex reading comprehension dataset with conditional answers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, page 3627–3637.
  36. Towards verifiable text generation with evolving memory and self-reflection. arXiv preprint arXiv:2312.09075.
  37. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  38. Fine-grained human feedback gives better rewards for language model training. In Advances in Neural Information Processing Systems.
  39. Effective large language model adaptation for improved grounding. arXiv preprint arXiv:2311.09533.
  40. Making retrieval-augmented language models robust to irrelevant context. arXiv preprint arXiv:2310.01558, 2023.
  41. Automatic evaluation of attribution by large language models. arXiv preprint arXiv:2305.06311.
  42. Training language models with memory augmentation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5657–5673.
  43. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406, 2023.
Citations (17)
View on Semantic Scholar

Summary

  • The paper introduces a training framework using rejection sampling and reinforcement learning with fine-grained rewards to enhance citation accuracy and response correctness.
  • The framework achieves significant improvements on QA benchmarks by aligning sentence-level and citation-level evaluation metrics.
  • These results pave the way for more credible and verifiable LLM outputs, setting new standards in attributable text generation.

Enhancing Attributable Text Generation in LLMs with Fine-Grained Rewards

Introduction to Attributable Text Generation

LLMs have shown remarkable capabilities in generating human-like text responses. However, their tendency to produce unverifiable or incorrect information, commonly referred to as "hallucinations," has raised questions regarding their reliability and trustworthiness. As a solution to increase their credibility and utility, recent efforts have been directed towards enabling LLMs to generate text with in-text citations, thereby linking generated claims to credible sources. This process, known as attributable text generation, aims to make the text output more transparent and verifiable by the end-users.

Challenges and Proposed Approach

The generation of accurate and relevant citations represents a multifaceted challenge, involving the assessment of both citation quality and information correctness. Prior attempts to tackle this problem through direct prompts or retrieval-augmented generation (RAG) have shown limitations, particularly for smaller LLMs. In response, this research introduces a new training framework leveraging fine-grained rewards, designed to enhance the LLM’s capabilities in producing both accurate responses and high-quality citations. This approach employs two training algorithms, rejection sampling (RS) and reinforcement learning (RL), utilizing fine-grained automatic evaluation functions for reward signals at both sentence-level and citation-level.

Methodology

The developed training framework initializes the LLM using distillation from a powerful model like ChatGPT, providing a starting point before applying RS or RL training. The methodology focuses on fine-grained rewards across three primary dimensions: correctness of the generated response, citation recall, and citation precision. These rewards aim to ensure that each sentence in the response is not only accurate but also appropriately supported by relevant citations. Experimental analysis is conducted on the Question Answering (QA) datasets from the ALCE benchmark, with further validation on the EXPERTQA for generalizability.

Results and Implications

The implementation of fine-grained rewards for attributable generation significantly outperforms traditional holistic reward approaches, demonstrating substantial improvements across all evaluated metrics. Specifically, the combination of RS and RL, when trained with fine-grained rewards, yields the best performance, surpassing established models like ChatGPT in certain metrics. These findings underscore the potential of fine-grained reward systems in refining the accuracy and credibility of LLM outputs, particularly in the context of generating text with citations.

Future Directions

The research suggests several avenues for further exploration. Improving LLMs’ reading comprehension and their capability to synthesize information from multiple documents could enhance correctness recall in QA tasks. Additionally, the iterative application of imitation learning and reinforcement learning might push the limits of small LLMs in attributable text generation. Addressing the initial reliance on distillation from larger models like ChatGPT could pave the way for a more accessible and iterative training process, independent of proprietary LLMs.

Conclusion

The introduction of fine-grained rewards in the training of LLMs for attributable text generation marks a significant advancement in enhancing the models' ability to generate trustworthy and verifiable content. By focusing on citation quality and information correctness, this approach addresses a critical gap in existing LLM applications, promising a future where LLM-generated text can be both informative and credible.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube

Reddit Logo Streamline Icon: https://streamlinehq.com

Reddit

  1. Training LLMs to generate text with citations via fine-grained rewards (1 point, 1 comment)
  2. Training Language Models to Generate Text with Citations via Fine-grained Rewards (0 points, 0 comments)