A Comparison of Methods for Evaluating Generative IR (2404.04044v2)

Abstract: Information retrieval systems increasingly incorporate generative components. For example, in a retrieval augmented generation (RAG) system, a retrieval component might provide a source of ground truth, while a generative component summarizes and augments its responses. In other systems, a LLM might directly generate responses without consulting a retrieval component. While there are multiple definitions of generative information retrieval (Gen-IR) systems, in this paper we focus on those systems where the system's response is not drawn from a fixed collection of documents or passages. The response to a query may be entirely new text. Since traditional IR evaluation methods break down under this model, we explore various methods that extend traditional offline evaluation approaches to the Gen-IR context. Offline IR evaluation traditionally employs paid human assessors, but increasingly LLMs are replacing human assessment, demonstrating capabilities similar or superior to crowdsourced labels. Given that Gen-IR systems do not generate responses from a fixed set, we assume that methods for Gen-IR evaluation must largely depend on LLM-generated labels. Along with methods based on binary and graded relevance, we explore methods based on explicit subtopics, pairwise preferences, and embeddings. We first validate these methods against human assessments on several TREC Deep Learning Track tasks; we then apply these methods to evaluate the output of several purely generative systems. For each method we consider both its ability to act autonomously, without the need for human labels or other input, and its ability to support human auditing. To trust these methods, we must be assured that their results align with human assessments. In order to do so, evaluation criteria must be transparent, so that outcomes can be audited by human assessors.

• The paper introduces five evaluative methods for generative IR that combine LLM automation with human auditability.
• It demonstrates that pairwise preferences and subtopic relevance methods offer nuanced differentiation in response quality.
• The findings underscore the need for advanced hybrid evaluation frameworks to effectively assess generative IR outputs.

Evaluative Methods for Generative Information Retrieval Systems

Introduction

The increasing integration of generative components in information retrieval (IR) systems necessitates a reevaluation of traditional offline evaluation methods. Gen-IR systems, characterized by their ability to produce responses not confined to a pre-existing corpus, present unique challenges for evaluation. This paper investigates various methods extending traditional offline IR evaluation to suit the Gen-IR context, emphasizing the operationalization of LLMs in evaluation processes.

Methods Explored

The exploration covers five distinct methods, each with its potential for autonomous operation and capacity for human auditability:

• Binary Relevance: Engages LLMs to assess query/response pairs for relevancy, supporting straightforward auditing by human assessors.
• Graded Relevance: Amplifies binary relevance by introducing multiple grades of relevance, although it suffers slightly from the need for calibrating human and LLM assessors to these grades.
• Subtopic Relevance: Utilizes LLM-generated subtopics to refine relevance evaluation, promising greater detail in relevancy assessments and offering an optimal balance between autonomy and auditability.
• Pairwise Preferences: Prioritizes direct comparison between two responses, showing higher performance in distinguishing the nuances between responses but requires exemplars for comparison.
• Embeddings: Leverages cosine similarity between the embeddings of an exemplar and generated responses, providing a method that, while not directly auditable, aligns well with human assessments in comparative contexts.

Validation and Results

The validation employed TREC Deep Learning Track datasets, applying the aforementioned methods to assess the alignment with human judgments and their efficacy in distinguishing between generative models' outputs. Key insights include:

• Methods like subtopic relevance and pairwise preferences showed promise in nuanced differentiation between responses.
• Pairwise preferences, while computationally demanding, provided a clear advantage in performance recognition but hinged on the availability of exemplars.
• Subtopic relevance emerged as a method offering substantial detail, allowing for a nuanced understanding of response relevance without extensive human input aside from auditing.

Implications and Future Directions

This work underscores the evolving need for Gen-IR evaluation methodologies that can effectively measure the novel outputs of generative systems. It highlights the potential of LLMs not only as tools in generating responses but also as critical components in the evaluation infrastructure of Gen-IR systems. The future of IR evaluation, as indicated by these findings, will likely rely more heavily on advanced models and autonomous methods, with human oversight ensuring alignment with user expectations and real-world relevance.

The exploration points to several directions for future research, including extending these evaluative methods to broader datasets and contexts, refining the balance between autonomous evaluations and human auditability, and adapting methodologies to the evolving capabilities of Gen-IR systems.

Conclusion

The transition towards generative models in information retrieval poses significant challenges and opportunities for the field of IR evaluation. This paper provides a foundational step towards understanding and developing evaluation methodologies suitable for Gen-IR. By leveraging the capabilities of LLMs within a structured evaluative framework, it opens avenues for more sophisticated, nuanced, and accurate assessments of generative information retrieval systems.