Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding (2401.04398v2)

Published 9 Jan 2024 in cs.CL

Abstract: Table-based reasoning with LLMs is a promising direction to tackle many table understanding tasks, such as table-based question answering and fact verification. Compared with generic reasoning, table-based reasoning requires the extraction of underlying semantics from both free-form questions and semi-structured tabular data. Chain-of-Thought and its similar approaches incorporate the reasoning chain in the form of textual context, but it is still an open question how to effectively leverage tabular data in the reasoning chain. We propose the Chain-of-Table framework, where tabular data is explicitly used in the reasoning chain as a proxy for intermediate thoughts. Specifically, we guide LLMs using in-context learning to iteratively generate operations and update the table to represent a tabular reasoning chain. LLMs can therefore dynamically plan the next operation based on the results of the previous ones. This continuous evolution of the table forms a chain, showing the reasoning process for a given tabular problem. The chain carries structured information of the intermediate results, enabling more accurate and reliable predictions. Chain-of-Table achieves new state-of-the-art performance on WikiTQ, FeTaQA, and TabFact benchmarks across multiple LLM choices.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Table-to-text generation and pre-training with TabT5. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6758–6766, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  2. Palm 2 technical report. arXiv preprint arXiv:2305.10403.
  3. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  4. Webtables: Exploring the power of tables on the web. Proc. VLDB Endow., 1(1):538–549.
  5. Wenhu Chen. 2023. Large language models are few(1)-shot table reasoners. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1120–1130, Dubrovnik, Croatia. Association for Computational Linguistics.
  6. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588.
  7. Tabfact: A large-scale dataset for table-based fact verification. In International Conference on Learning Representations.
  8. Binding language models in symbolic languages. In International Conference on Learning Representations.
  9. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186.
  10. Handling divergent reference texts when evaluating table-to-text generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4884–4895.
  11. Understanding tables with intermediate pre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 281–296, Online. Association for Computational Linguistics.
  12. PAL: Program-aided language models. In International Conference on Machine Learning, pages 10764–10799. PMLR.
  13. PASTA: Table-operations aware fact verification via sentence-table cloze pre-training. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4971–4983, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  14. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992.
  15. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320–4333, Online. Association for Computational Linguistics.
  16. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguistics.
  17. MathPrompter: Mathematical reasoning using large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 37–42, Toronto, Canada. Association for Computational Linguistics.
  18. OmniTab: Pretraining with natural and synthetic data for few-shot table-based question answering. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 932–942, Seattle, United States. Association for Computational Linguistics.
  19. A survey on table question answering: recent advances. In China Conference on Knowledge Graph and Semantic Computing, pages 174–186. Springer.
  20. Ziqi Jin and Wei Lu. 2023. Tab-cot: Zero-shot tabular chain of thought. arXiv preprint arXiv:2305.17812.
  21. George Katsogiannis-Meimarakis and Georgia Koutrika. 2023. A survey on deep learning approaches for text-to-sql. The VLDB Journal, pages 1–32.
  22. Decomposed prompting: A modular approach for solving complex tasks. In International Conference on Learning Representations.
  23. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems.
  24. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  25. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172.
  26. TAPEX: Table pre-training via learning a neural sql executor. In International Conference on Learning Representations.
  27. From zero to hero: Examining the power of symbolic tasks in instruction tuning. arXiv preprint arXiv:2304.07995.
  28. Benchmarking large language model capabilities for conditional generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9194–9213.
  29. FeTaQA: Free-form table question answering. Transactions of the Association for Computational Linguistics, 10:35–49.
  30. Lever: Learning to verify language-to-code generation with execution. In International Conference on Machine Learning, pages 26106–26128. PMLR.
  31. OpenAI. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
  32. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  33. Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470–1480, Beijing, China. Association for Computational Linguistics.
  34. Richard Pönighaus. 1995. ’favourite’sql-statements—an empirical analysis of sql-usage in commercial applications. In International Conference on Information Systems and Management of Data, pages 75–91. Springer.
  35. Evaluating the text-to-sql capabilities of large language models. arXiv preprint arXiv:2204.00498.
  36. On the potential of lexico-logical alignments for semantic parsing to sql queries. Findings of the Association for Computational Linguistics: EMNLP 2020.
  37. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  38. Large language models still can’t plan (a benchmark for llms on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop.
  39. TUTA: Tree-based transformers for generally structured table pre-training. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1780–1790.
  40. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
  41. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601.
  42. Large language models are versatile decomposers: Decompose evidence and questions for table-based reasoning. arXiv preprint arXiv:2301.13808.
  43. Least-to-most prompting enables complex reasoning in large language models. In International Conference on Learning Representations.
Citations (58)
View on Semantic Scholar

Summary

  • The paper introduces a dynamic framework that evolves tables through intermediate reasoning steps to enhance LLM interpretation of tabular data.
  • It uses iterative operations such as adding columns and selecting rows to incrementally build a coherent reasoning chain.
  • Experimental validation on WikiTQ, FeTaQA, and TabFact benchmarks demonstrates its significant edge over existing table understanding models.

Overview of Chain-of-Table Framework

The field of natural language processing has taken interest in developing methodologies for interpreting tabular data using sophisticated LLMs. A notable development is the introduction of the Chain-of-Table framework, which seeks to augment the reasoning capabilities of LLMs for tasks involving tables. It specifically addresses the challenge of synthesizing the various steps involved in understanding and answering questions about complex tabular information.

Understanding Tabular Data with LLMs

Tables, by their structured nature, store information in an organized manner that differs fundamentally from plain text. This creates a distinct challenge when LLMs are expected to comprehend and reason over the data. Traditional approaches to equip LLMs with the capability to process tabular data involve pre-training models to recognize table structure through specialized embedding layers or attention mechanisms or pre-train models as neural SQL executors using synthetic query-response pairs. The novelty of the Chain-of-Table methodology lies in its ability to dynamically update the table to reflect the reasoning process, step by step, as the model arrives at the answer to a given question.

The Mechanics of Chain-of-Table

This method starts with a table subjected to a sequence of operations—such as adding columns, selecting rows, or grouping—which serve as “intermediate thoughts” recorded in the evolving table. LLMs, through iterative prompting, choose and execute a sequence of these operations, with each operation building upon the previous ones to gradually spin a narrative of the reasoning process. Key parts of this approach include dynamic planning for the selection of the next operation, arguments generation for those operations, and final querying for the answer after the table has been sufficiently transformed.

Experimental Validation

Chain-of-Table has not only been a conceptual success but has also shown remarkable performance in practical evaluations. When subjected to public benchmarks for table understanding, it has outperformed existing state-of-the-art models across various metrics. The strategic iterations of table transformation allow LLMs such as PaLM 2, GPT 3.5, and LLaMA 2 to navigate through the intricacies of tabular reasoning, ultimately enhancing accuracy and producing reliable outcomes. This has been demonstrated on WikiTQ, FeTaQA, and TabFact datasets, where Chain-of-Table leads in performance indicators.

Implications and Future Directions

The Chain-of-Table framework is a significant milestone in the effort to fully exploit the potential of LLMs in understanding and using tabular data, which is prevalent in diverse segments such as finance, health, and numerous scientific domains. The methodology stands as a testament to the innovative ways machine learning can be guided to interpret complex information structures and make accurate predictions. As the capabilities of LLMs grow, frameworks like Chain-of-Table are pivotal in advancing our interaction with knowledge and information in tabular formats. This line of research opens up new avenues for more nuanced and contextually aware artificial intelligence systems.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube