Leveraging Large Language Models for NLG Evaluation: A Survey

Introduction to LLM-based NLG Evaluation

Natural Language Generation (NLG) is a critical aspect of modern AI-driven communication, with its applications sprawling across various fields such as machine translation and content creation. With the advancement of LLMs, our ability to generate text has seen a leap in quality. This, in turn, necessitates robust evaluation methods that can accurately assess the quality of the generated content. Traditional NLG evaluation metrics often fail to fully capture semantic coherence or fail to align with human judgments. In contrast, the emergent capacities of LLMs offer promising new methods for NLG evaluation through improved interpretability and alignment with human preferences.

A Structured Framework for Evaluation

This paper presents a detailed overview of utilizing LLMs for NLG evaluation and establishes a formalized taxonomy to categorize various LLM-based evaluation metrics. By identifying the core dimensions of evaluation tasks, references, and functions, a structured perspective emerges that enhances our understanding of different approaches. Moreover, the paper investigates the role of LLMs in NLG evaluation, acknowledging their potential in evaluating tasks. It explores the novel applications of LLMs in generating evaluation metrics directly, considering continuous scoring, likelihood estimations, and comparative pairwise analyses. The taxonomy presented herein provides clarity on the landscape of LLM-based evaluators, delineating between generative-based methods and matching-based approaches.

Advancement and Meta-Evaluation

Emphasizing the ability to measure alignment with human judgment, the survey reviews meta-evaluation benchmarks across diverse NLG tasks, including machine translation, text summarization, and more. These benchmarks offer important platforms for testing evaluator efficacy by incorporating human annotations and by assessing agreement with human preferences. The paper recognizes the evolution of LLMs in general generation tasks and outlines the development of multi-scenario benchmarks that contribute to a richer understanding of evaluator performances.

The Road Ahead for NLG Evaluation

Despite the progress, several challenges linger in the domain of LLM-based NLG evaluation, such as biases inherent in LLM evaluators, their robustness against adversarial inputs, the need for domain-specific evaluation, and the quest for unified evaluation across a variety of complex tasks. Addressing these challenges is crucial for advancing the field and developing more reliable and effective evaluators. The paper concludes by advocating for future research to tackle these open problems and propel the NLG evaluation landscape forward.