· Trending Papers
Emergent Mind

Complementary Explanations for Effective In-Context Learning

(2211.13892)
Published Nov 25, 2022 in cs.CL

Abstract

LLMs have exhibited remarkable capabilities in learning from explanations in prompts, but there has been limited understanding of exactly how these explanations function or why they are effective. This work aims to better understand the mechanisms by which explanations are used for in-context learning. We first study the impact of two different factors on the performance of prompts with explanations: the computation trace (the way the solution is decomposed) and the natural language used to express the prompt. By perturbing explanations on three controlled tasks, we show that both factors contribute to the effectiveness of explanations. We further study how to form maximally effective sets of explanations for solving a given test query. We find that LLMs can benefit from the complementarity of the explanation set: diverse reasoning skills shown by different exemplars can lead to better performance. Therefore, we propose a maximal marginal relevance-based exemplar selection approach for constructing exemplar sets that are both relevant as well as complementary, which successfully improves the in-context learning performance across three real-world tasks on multiple LLMs.

We're not able to analyze this paper right now due to high demand.

Please check back later (sorry!).

Generate a detailed summary of this paper with a premium account.

We ran into a problem analyzing this paper.

Subscribe by Email

Get summaries of trending comp sci papers delivered straight to your inbox:

Unsubscribe anytime.

References
  1. Explanations for CommonsenseQA: New Dataset and Models. In Proceedings of the Annual Conference of the Association for Computational Linguistics (ACL).
  2. PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
  3. Language models are few-shot learners. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS).
  4. e-snli: Natural language inference with natural language explanations. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS).
  5. Jaime Carbonell and Jade Goldstein. 1998. The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). Association for Computing Machinery.
  6. Evaluating Large Language Models Trained on Code
  7. Meta-learning via language model in-context tuning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
  8. PaLM: Scaling Language Modeling with Pathways
  9. Training Verifiers to Solve Math Word Problems
  10. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
  11. Prototypical Calibration for Few-shot Learning of Language Models
  12. Surface form competition: Why the highest probability answer isn’t always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
  13. Maieutic prompting: Logically consistent reasoning with recursive explanations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
  14. Large Language Models are Zero-Shot Reasoners
  15. What Makes Good In-Context Examples for GPT-$3$?
  16. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
  17. Language models of code are few-shot commonsense learners. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
  18. Noisy channel language model prompting for few-shot text classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
  19. MetaICL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  20. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)
  21. Show Your Work: Scratchpads for Intermediate Computation with Language Models
  22. Training language models to follow instructions with human feedback
  23. Measuring and Narrowing the Compositionality Gap in Language Models
  24. Evaluating the impact of model scale for compositional generalization in semantic parsing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
  25. Learning to retrieve prompts for in-context learning. In Proceedings of the Annual Conference of the Association for Computational Linguistics (ACL).
  26. Constrained language models yield few-shot semantic parsers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
  27. Selective Annotation Makes Language Models Better Few-Shot Learners
  28. Rationale-Augmented Ensembles in Language Models
  29. Self-Consistency Improves Chain of Thought Reasoning in Language Models
  30. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  31. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations.
  32. The unreliability of explanations in few-shot prompting for textual reasoning. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS).
  33. OPT: Open Pre-trained Transformer Language Models
  34. BERTScore: Evaluating Text Generation with BERT. In Proceedings of the International Conference on Learning Representations (ICLR).
  35. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the International Conference on Learning Representations (ICLR).
  36. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
  37. Teaching Algorithmic Reasoning via In-context Learning

Show All 37