A Survey on Evaluation of Large Language Models
(2307.03109)Abstract
LLMs are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate, where to evaluate, and how to evaluate. Firstly, we provide an overview from the perspective of evaluation tasks, encompassing general natural language processing tasks, reasoning, medical usage, ethics, educations, natural and social sciences, agent applications, and other areas. Secondly, we answer the where' andhow' questions by diving into the evaluation methods and benchmarks, which serve as crucial components in assessing performance of LLMs. Then, we summarize the success and failure cases of LLMs in different tasks. Finally, we shed light on several future challenges that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to researchers in the realm of LLMs evaluation, thereby aiding the development of more proficient LLMs. Our key point is that evaluation should be treated as an essential discipline to better assist the development of LLMs. We consistently maintain the related open-source materials at: https://github.com/MLGroupJLU/LLM-eval-survey.
Overview
-
This paper systematically reviews the methods for evaluating LLMs, stressing the importance of multifaceted assessment strategies.
-
Evaluation is framed within three dimensions: tasks (what), benchmarks and datasets (where), and methodologies (how).
-
LLMs demonstrate proficiency in areas like fluency and arithmetic reasoning, but struggle with robustness and tasks requiring nuanced, current knowledge.
-
Methodologies for evaluation are categorized as either automatic or human-involved, each with its own benefits and shortcomings.
-
The authors advocate for evaluation as a discipline and suggest future directions to enhance LLM measurement and development.
Overview of LLMs Evaluation Metrics and Benchmarks
LLMs have become pivotal in various applications and their evaluation has grown increasingly complex. It is essential to assess these models using multifaceted techniques that reflect their performance and ability to interact with humans effectively. The paper entitled "A Survey on Evaluation of LLMs" by Chang et al. presents a systematic review of the evaluation methods employed for LLMs.
Evaluation Dimensions
The paper delineates an evaluation framework consisting of three primary dimensions: the categories of tasks evaluated (what to evaluate), the datasets and benchmarks applied (where to evaluate), and the methodologies implemented (how to evaluate).
Concerning What to Evaluate, the paper categorizes evaluation tasks into areas such as natural language processing, robustness, and medical inquiries. It reveals that while LLMs excel in fluency and certain reasoning tasks, they fall short in aspects such as robustness to adversarial prompts and tasks requiring current, real-time knowledge.
As for Where to Evaluate, the paper highlights the need for comprehensive benchmarks that can accommodate the rapid development of LLM capabilities. It references an array of benchmarks assessing general language tasks, specific downstream tasks, and multi-modal tasks, emphasizing that no single benchmark is universally best suited for all types of LLMs.
Insights from Numerical Results and Strong Claims
Significant numerical results are discussed, indicating the strengths and weaknesses of LLMs across various tasks. For instance, LLMs show impressive performance in arithmetic reasoning and factual input handling, yet have limitations in areas like abstract reasoning and our ever-evolving understanding of human-like qualities like humor.
The paper also underscores strong claims, such as the efficacy of LLMs in education-related functions but notes their susceptibility to generating biased or inaccurate content, presenting both ethical concerns and challenges for reliable application.
Methodologies for Evaluation
When explaining How to Evaluate, the paper differentiates between automatic and human-involved evaluations. Automatic evaluations, although efficient, might not capture the complete spectrum of LLM capabilities, especially in cases where nuanced judgment is required. Conversely, human evaluations, despite being more labor-intensive, offer richer insights into the practical usability and interaction quality of LLMs.
Future Directions
The authors stress that evaluation should be considered a discipline in its own right, guiding the progression of LLMs. They posit key future challenges, including the development of benchmarks capable of measuring AGI (Artificial General Intelligence), comprehensive behavioral evaluation, robustness against diverse inputs, dynamic and adaptive evaluation protocols, and beyond.
Moreover, the paper looks to contribute beyond raw evaluation, suggesting that a credible evaluation system should foster LLM enhancements through insightful analysis and actionable guidance.
Conclusion
In summary, "A Survey on Evaluation of LLMs" is an extensive study that not only provides a current overview of LLM evaluation strategies but also paves the way for future research and development. As we advance in our understanding and integration of LLMs, such a survey is invaluable for improving the models' reliability, fairness, and applicability across various domains.