"Is ChatGPT a Better Explainer than My Professor?": Evaluating the Explanation Capabilities of LLMs in Conversation Compared to a Human Baseline

Evaluating the Explanation Capabilities of LLMs in Conversation Compared to a Human Baseline

The paper, titled "Is ChatGPT a Better Explainer than My Professor?: Evaluating the Explanation Capabilities of LLMs in Conversation Compared to a Human Baseline" by Grace Li, Milad Alshomary, and Smaranda Muresan, contributes to the domain of science communication by focusing on how LLMs, particularly GPT-4, perform in generating explanatory dialogues compared to human experts. This work is pivotal as it provides empirical evidence and methodologies for assessing the quality of LLM-generated explanations in conversational settings.

Background and Framework

The research targets conversational explanations due to their adaptive and interactive nature, leveraging the 5-Levels dataset from WIRED's YouTube series. This dataset, annotated with "explanatory acts", provides a rich ground for examining the diverse strategies used by explainers and explainees in dialogues. The annotations clarify how these dialogues are structured, providing a nuanced understanding of various explanatory tactics.

Methodology

The study evaluates three distinct strategies for generating explainer responses:

1. S1: Baseline - Human expert responses extracted directly from the 5-Levels dataset.
2. S2: GPT4 Standard - GPT-4 generated responses based solely on the previous conversational context.
3. S3: GPT4 w/ Explanatory Acts (EAs) - GPT-4 responses constructed using the previous context and a sequence of predefined explanatory acts.

Human annotators were recruited to assess these responses on eight dimensions (coherence, conciseness, conversational nature, acknowledgment, appropriateness, depth, active guidance, and engagement), and rank the responses to facilitate a comprehensive evaluation.

Results

The inter-annotator agreement was measured using Krippendorff’s alpha for the rating dimensions and Kendall's Tau for the ranking agreement. Notably, the study found:

• S2: GPT Standard was preferred in 49% of the instances, outperforming both the human baseline and the explanatory acts-aided approaches in terms of overall perceived quality.
• S3: GPT w/ EA demonstrated value in deepening conversations and maintaining engagement, although it was less preferred due to the verbosity and complexity of some responses.
• S1: Baseline received the lowest preference, primarily due to a perceived lack of engagement and adaptability compared to the LLM-generated responses.

Discussion

The paper's results indicate that LLMs, particularly GPT-4, are capable of not only matching but sometimes exceeding human performance in generating conversational explanations. The effectiveness of LLMs in this context has significant implications:

1. Practical Implications: LLMs could serve as valuable tools in augmenting education and science communication. They can provide scalable, adaptive, and engaging explanations, potentially bridging the gap between experts and laypersons.
2. Theoretical Implications: The ability of LLMs to follow structured explanatory acts suggests that LLMs possess an advanced understanding of conversational dynamics. This offers a pathway for future research to explore and refine strategies that optimize LLM performance in educational settings.

Future Directions

The standing results from this study pave the way for several future research ventures. These include:

• Enhanced Personalization: Developing frameworks that automatically tailor explanations based on the explainee’s preferences and background.
• Integrative Systems: Exploring hybrid models where human experts and LLMs collaboratively generate explanations, leveraging the strengths of both.
• Deeper Evaluation Criteria: Expanding evaluation metrics to include longer-term impacts on learning and understanding, as well as exploring the role of multimodal inputs in explanatory dialogues.

Through rigorous experimentation and detailed analysis, this paper makes a substantial contribution to the understanding of how LLMs can enhance science communication. By addressing both the strengths and shortcomings of current LLM capabilities, it lays a robust groundwork for further advancements in the field of AI-driven education and communication systems.