"In-Context Learning" or: How I learned to stop worrying and love "Applied Information Retrieval" (2405.01116v1)

Abstract: With the increasing ability of LLMs, in-context learning (ICL) has evolved as a new paradigm for NLP, where instead of fine-tuning the parameters of an LLM specific to a downstream task with labeled examples, a small number of such examples is appended to a prompt instruction for controlling the decoder's generation process. ICL, thus, is conceptually similar to a non-parametric approach, such as $k$-NN, where the prediction for each instance essentially depends on the local topology, i.e., on a localised set of similar instances and their labels (called few-shot examples). This suggests that a test instance in ICL is analogous to a query in IR, and similar examples in ICL retrieved from a training set relate to a set of documents retrieved from a collection in IR. While standard unsupervised ranking models can be used to retrieve these few-shot examples from a training set, the effectiveness of the examples can potentially be improved by re-defining the notion of relevance specific to its utility for the downstream task, i.e., considering an example to be relevant if including it in the prompt instruction leads to a correct prediction. With this task-specific notion of relevance, it is possible to train a supervised ranking model (e.g., a bi-encoder or cross-encoder), which potentially learns to optimally select the few-shot examples. We believe that the recent advances in neural rankers can potentially find a use case for this task of optimally choosing examples for more effective downstream ICL predictions.

• The paper demonstrates that integrating IR principles, such as dynamic example scoring, significantly boosts predictive accuracy in In-Context Learning.
• The study introduces an adaptive ICL method that mirrors k-nearest neighbors by adjusting the few-shot prompt based on local data similarities.
• The research highlights the use of supervised neural rankers and Query Performance Prediction to optimize example selection and reduce bias.

In-Context Learning and Information Retrieval in AI

This essay provides an analysis of the paper titled "In-Context Learning" or: How I learned to stop worrying and love "Applied Information Retrieval" (2405.01116). This work examines the convergence between In-Context Learning (ICL) — a burgeoning paradigm in NLP facilitated by LLMs — and established techniques in the field of Information Retrieval (IR). The authors propose that the principles and advancements in IR can significantly enhance the effectiveness of ICL.

Introduction

The paper presents a novel perspective by arguing that the foundation of In-Context Learning (ICL) aligns closely with principles found in Information Retrieval (IR). It juxtaposes the role ICL plays in the field of NLP with non-parametric approaches such as $k$-Nearest Neighbors ($k$-NN), emphasizing the reliance on local similarities within a dataset. In the context of ICL, a few labeled examples from a training set are appended to a prompt to guide an LLM's generation, which modifies the prediction process from relying on pre-trained model fine-tuning to emphasizing prompt-based guidance (Figure 1).

Figure 1: A workflow diagram illustrating how three verticals of IR research fit into the workflow of in-context learning (ICL).

While traditional IR focuses on relevance ranking, which can be leveraged to retrieve few-shot examples for ICL, this research introduces the concept of "usefulness-specific relevance," where an example is considered valuable if it improves predictive accuracy. The work highlights how advanced neural rankers might be used to select optimal demonstrations for ICL, stressing that the field of Information Retrieval offers robust methodologies that can enhance ICL workflows effectively.

In-Context Learning: Conceptual Framework

In-Context Learning differs fundamentally from traditional supervised learning, as it bypasses iterative parameter fine-tuning in favor of leveraging few-shot labeled examples via prompt engineering to guide the decoder's generative process. The prediction in ICL depends heavily on the context provided by a few-shot set of examples appended to the prompt, making it akin to $k$-NN, although with frozen LLM parameters (Figure 2). The framework of ICL consists of embedding test instances as queries to retrieve relevant, high-utility examples from a training set, which align with the predictive task's objectives such as text classification or generative tasks like QA or summarization.

Figure 2: Example workflow of In-Context Learning for sentiment classification.

Query Performance Prediction (QPP) and Adaptive ICL

The paper explores the parallel between ICL's demand for useful training examples and Information Retrieval's (IR) ranking tasks, such as QPP, which could enhance ICL by efficiently selecting relevant training examples. The authors suggest that both unsupervised and supervised retrieval models can be calibrated to select optimal few-shot examples, employing ranking objectives and predictive models learned from training instances to enhance the efficacy of example retrieval.

Adaptive Example Selection

A proposed approach to optimize the number of examples in a prompt involves adaptive ICL, where the number of examples is dynamically determined based on the test instance, much like adjusting the neighborhood size in $k$-NN for variable density distributions (Figure 3).

Figure 3: Motivation behind using a variable sized neighborhood for $k$-NN classification.

This approach necessitates scoring each example based on its potential utility for a correct prediction, which is a non-trivial task that the paper describes through a supervised learning approach. The prediction accuracy relies significantly on these localized examples, emphasizing their critical role in informing the model's decision-making process.

Practical and Theoretical Implications

The investigation outlines how the integration of IR strategies, particularly QPP, faceted, and diversified search paradigms, into ICL can create synergies that enhance ICL. Such integration would allow AI systems to perform better in tasks with varying demands on training data, such as adapting the number of examples dynamically — an area where they could borrow significantly from the vast IR literature.

Furthermore, the paper underscores the significance of orchestrating relevance and diversity in training examples to reduce the bias towards certain topical aspects, essential for a balanced exploration of the input space. Highlighting the challenges faced when utilizing score distributions and pointing out the necessity of adopting supervised approaches using neural rankers for more accurate ICL example selection.

Conclusion

This paper positions itself at the intersection of In-Context Learning and Information Retrieval, postulating that enhanced retrieval and diversity mechanisms can significantly contribute towards more effective ICL applications. By recommending IR methodologies such as Query Performance Prediction, supervised ranking, and faceted IR, the research opens new avenues for the refinement of ICL. Continued exploration along these lines may lead to paradigms that further heighten the adaptability and efficiency of AI models, providing tangible benefits for complex downstream AI tasks. Importantly, this research provides a framework to guide future work in merging these distinct yet synergistic fields.