BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues (2310.13650v1)

Abstract: Interacting with human via high-quality multi-turn dialogues is a key feature of LLMs. However, human-based evaluation of such capability involves intensive manual labor. This report provides a preliminary evaluation of existing LLMs for human-style multi-turn chatting, through an LLM-based approach. We start from real-world human dialogues and keep the very first utterances as the ChatSEED. Then we prompt LLMs to generate a full multi-turn dialogue (tens of utterances) based on the ChatSEED, utterance by utterance. Finally, we adopt state-of-the-art LLMs (GPT-4, \etc) as the judge to evaluate the generated dialogues. With different evaluation protocols, we come to substantially identical conclusions. We find that GPT-4 can generate human-style multi-turn dialogues with impressive quality, significantly outperforms its counterparts. It's difficult for a discriminator to distinguish between GPT-4 generated dialogues and human dialogues. In contrast, other LLMs struggle to generate multi-turn dialogues of satisfactory quality due to poor instruction-following capability, tendency to generate lengthy utterances, or limited general capability. All data and codes will be provided in https://github.com/open-compass/BotChat/ and we hope they can serve as a valuable resource for evaluating multi-turn chatting capabilities of LLMs.

• The paper introduces BotChat, an automated framework leveraging LLMs to generate and evaluate multi-turn dialogues, offering a scalable alternative to human assessment.
• Key findings show GPT-4 consistently excels at generating human-like dialogue quality, maintaining performance significantly better than other models, especially across longer conversations.
• Evaluation using BotChat reveals that while some open-source LLMs perform well in short dialogues, they struggle with coherence and consistency in extended multi-turn interactions.

Evaluating LLMs' Capabilities in Multi-Turn Dialogues: Insights from the BotChat Framework

Overview

The paper entitled "BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues" introduces an automated evaluation method for assessing the multi-turn conversational abilities of contemporary LLMs. Recognizing the impracticality and resource intensity of human-based evaluations, the authors propose a framework called BotChat. This framework leverages LLMs themselves to both generate multi-turn dialogues and evaluate their quality, offering a unique, labor-efficient means of testing conversational models. The principal focus is on comparing the performance of various LLMs, including the state-of-the-art model GPT-4, across different multi-turn dialogue scenarios.

Methodology and Implementation

BotChat operates through two primary stages: dialogue generation and quality assessment. The procedure begins with the extraction of initial dialogues, referred to as ChatSEEDs, from real-world conversation datasets. These seeds serve as the foundation from which LLMs generate full-fledged multi-turn dialogues. The generation process employs a structured, turn-by-turn format, with LLMs prompted to emulate human-like dialogue behavior.

Evaluation occurs in three distinct forms: Unitary Evaluation (UniEval), BotChat Arena, and Ground-Truth Evaluation (GTEval). In UniEval, a judge LLM independently assesses each generated dialogue to ascertain its human-likeness. BotChat Arena introduces a competitive framework where two dialogues generated by different LLMs are compared to determine which better mimics human conversation. Finally, GTEval compares LLM-generated dialogues with ground truth human dialogues from datasets, providing a benchmark for assessing the degree of fidelity to human interaction.

Key Findings

The paper presents several notable findings from their expansive evaluation across 14 LLMs. GPT-4 consistently excels, producing dialogues that are difficult to distinguish from human conversations. This model stands out not only for its superior generation of human-like dialogue but also for its ability to maintain quality over longer interactions. Other LLMs, such as Vicuna-13B and InternLM-20B, also show commendable performance, albeit falling short of GPT-4's standard in extended dialogues.

Several open-source LLMs deliver satisfactory results in short dialogue scenarios; however, their performance declines significantly with increased dialogue length. This degradation highlights issues in maintaining contextual relevance and natural flow over multiple turns. Factors contributing to poorer performance include excessive verbosity, tendency for AI assistants to reveal their non-human nature, and contextually inconsistent or repetitive responses.

Implications and Future Directions

The implications of this research are far-reaching. From a practical standpoint, BotChat offers a scalable, efficient alternative to human labor-intensive evaluations, enabling rapid assessment of LLM dialogue capabilities across diverse models. This methodology could evolve into a standard tool for benchmarking future LLMs and variant updates.

From a theoretical perspective, understanding the nuances in dialogue generation quality across models guides research into refining LLM architectures and training paradigms. Addressing identified flaws, such as contextual inconsistency and verbosity, could enhance the development of LLMs adept in generating seamless, human-like discourse.

Future research may focus on refining evaluation strategies, potentially incorporating more nuanced aspects of human dialogue such as emotional intelligence and situational awareness. Additionally, expanding BotChat's evaluation protocols to include multilingual dialogues could advance capabilities in global communication contexts.

Conclusion

This paper provides a robust framework for evaluating LLM dialogue capabilities and highlights GPT-4's exceptional proficiency in maintaining human-style conversations over multiple turns. The authors underscore challenges faced by lesser-performing models, offering a roadmap for ongoing enhancements in LLMs’ conversational skills. By employing LLMs for both dialogue generation and assessment, BotChat represents a significant shift towards more efficient and automated evaluation methodologies in AI research.