· Trending Papers
Emergent Mind

Compositional Exemplars for In-context Learning

(2302.05698)
Published Feb 11, 2023 in cs.CL , cs.AI , and cs.LG

Abstract

Large pretrained language models (LMs) have shown impressive In-Context Learning (ICL) ability, where the model learns to do an unseen task via a prompt consisting of input-output examples as the demonstration, without any parameter updates. The performance of ICL is highly dominated by the quality of the selected in-context examples. However, previous selection methods are mostly based on simple heuristics, leading to sub-optimal performance. In this work, we formulate in-context example selection as a subset selection problem. We propose CEIL (Compositional Exemplars for In-context Learning), which is instantiated by Determinantal Point Processes (DPPs) to model the interaction between the given input and in-context examples, and optimized through a carefully-designed contrastive learning objective to obtain preference from LMs. We validate CEIL on 12 classification and generation datasets from 7 distinct NLP tasks, including sentiment analysis, paraphrase detection, natural language inference, commonsense reasoning, open-domain question answering, code generation, and semantic parsing. Extensive experiments demonstrate not only the state-of-the-art performance but also the transferability and compositionality of CEIL, shedding new light on effective and efficient in-context learning. Our code is released at https://github.com/HKUNLP/icl-ceil.

We're not able to analyze this paper right now due to high demand.

Please check back later (sorry!).

Generate a detailed summary of this paper with a premium account.

We ran into a problem analyzing this paper.

Subscribe by Email

Get summaries of trending comp sci papers delivered straight to your inbox:

Unsubscribe anytime.

References
  1. In-context Examples Selection for Machine Translation
  2. Cont: Contrastive neural text generation. NeurIPS
  3. Task-oriented dialogue as dataflow synthesis. Transactions of the Association for Computational Linguistics, 8:556–571
  4. Learning detection with diverse proposals. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.  7149–7157

  5. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp.  1533–1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. https://www.aclweb.org/anthology/D13-1160.

  6. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. https://doi.org/10.5281/zenodo.5297715.

  7. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html.

  8. Fast greedy map inference for determinantal point process to improve recommendation diversity. Advances in Neural Information Processing Systems, 31
  9. Evaluating Large Language Models Trained on Code
  10. Evaluating Large Language Models Trained on Code

  11. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp.  4171–4186, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. https://aclanthology.org/N19-1423.

  12. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pp.  350–356
  13. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science (New York, NY), 378(6624):1067–1074

  14. Improving text-to-SQL evaluation methodology. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  351–360, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1033. https://aclanthology.org/P18-1033.

  15. The Pile: An 800GB Dataset of Diverse Text for Language Modeling
  16. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.  6894–6910, 2021b.
  17. Near-optimal map inference for determinantal point processes. Advances in Neural Information Processing Systems, 25
  18. Diverse sequential subset selection for supervised video summarization. Advances in neural information processing systems, 27
  19. Faster greedy map inference for determinantal point processes. In International Conference on Machine Learning, pp. 1384–1393. PMLR
  20. Question decomposition with dependency graphs. In 3rd Conference on Automated Knowledge Base Construction
  21. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  9729–9738
  22. Activitynet: A large-scale video benchmark for human activity understanding. In 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp.  961–970. IEEE
  23. The curious case of neural text degeneration. In International Conference on Learning Representations
  24. Unsupervised Dense Information Retrieval with Contrastive Learning
  25. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.  6769–6781
  26. Adam: A method for stochastic optimization. In Proceedings of ICLR
  27. An exact algorithm for maximum entropy sampling. Operations Research, 43(4):684–691
  28. k-dpps: Fixed-size determinantal point processes. In ICML
  29. Determinantal point processes for machine learning. Foundations and Trends® in Machine Learning, 5(2–3):123–286
  30. Kulis, B. et al. Metric learning: A survey. Foundations and Trends® in Machine Learning, 5(4):287–364
  31. Diverse Demonstrations Improve In-context Compositional Generalization
  32. Mtop: A comprehensive multilingual task-oriented semantic parsing benchmark. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2950–2962
  33. Making Large Language Models Better Reasoners with Step-Aware Verifier
  34. Nl2bash: A corpus and semantic parser for natural language interface to the linux operating system. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation LREC 2018, Miyazaki (Japan), 7-12 May, 2018.
  35. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pp.  100–114
  36. Liu, T.-Y. et al. Learning to rank for information retrieval. Foundations and Trends® in Information Retrieval, 3(3):225–331
  37. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  8086–8098
  38. WebGPT: Browser-assisted question-answering with human feedback
  39. Representation Learning with Contrastive Predictive Coding
  40. OpenAI, T. Chatgpt: Optimizing language models for dialogue. OpenAI

  41. Improving compositional generalization with latent structure and data augmentation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.  4341–4362, Seattle, United States, July 2022a. Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.323. https://aclanthology.org/2022.naacl-main.323.

  42. Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing
  43. Language models are unsupervised multitask learners. 2019.
  44. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3:333–389, 01 2009. doi: 10.1561/1500000019.
  45. Movie description. International Journal of Computer Vision, 123:94–120

  46. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.  2655–2671, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.191. https://aclanthology.org/2022.naacl-main.191.

  47. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press

  48. Compositional generalization and natural language variation: Can a semantic parsing approach handle both? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp.  922–938, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.75. https://aclanthology.org/2021.acl-long.75.

  49. Natural Language to Code Translation with Execution
  50. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp.  1631–1642
  51. Selective Annotation Makes Language Models Better Few-Shot Learners

  52. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp.  4149–4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1421. https://aclanthology.org/N19-1421.

  53. Attention is all you need. Advances in neural information processing systems, 30
  54. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp.  353–355
  55. Self-Consistency Improves Chain of Thought Reasoning in Language Models
  56. Emergent abilities of large language models. Transactions on Machine Learning Research
  57. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp.  1112–1122

  58. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp.  38–45, Online, October 2020. Association for Computational Linguistics. https://www.aclweb.org/anthology/2020.emnlp-demos.6.

  59. Break it down: A question understanding benchmark. Transactions of the Association for Computational Linguistics, 8:183–198
  60. Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering
  61. Deep determinantal point process for large-scale multi-label classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.

  62. ProGen: Progressive zero-shot dataset generation via in-context feedback. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp.  3671–3683, Abu Dhabi, United Arab Emirates, December 2022a. Association for Computational Linguistics. https://aclanthology.org/2022.findings-emnlp.269.

  63. Generating data for symbolic language with large language models. 2023.
  64. Complementary Explanations for Effective In-Context Learning
  65. Compositional generalization for neural semantic parsing via span-level supervised attention. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.  2810–2823

  66. Learning to parse database queries using inductive logic programming. In AAAI/IAAI, pp.  1050–1055, Portland, OR, August 1996. AAAI Press/MIT Press. http://www.cs.utexas.edu/users/ai-lab?zelle:aaai96.

  67. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

  68. Extractive summarization as text matching. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.  6197–6208, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.552. https://aclanthology.org/2020.acl-main.552.

Show All 68