EIPE-text: Evaluation-Guided Iterative Plan Extraction for Long-Form Narrative Text Generation (2310.08185v1)

Abstract: Plan-and-Write is a common hierarchical approach in long-form narrative text generation, which first creates a plan to guide the narrative writing. Following this approach, several studies rely on simply prompting LLMs for planning, which often yields suboptimal results. In this paper, we propose a new framework called Evaluation-guided Iterative Plan Extraction for long-form narrative text generation (EIPE-text), which extracts plans from the corpus of narratives and utilizes the extracted plans to construct a better planner. EIPE-text has three stages: plan extraction, learning, and inference. In the plan extraction stage, it iteratively extracts and improves plans from the narrative corpus and constructs a plan corpus. We propose a question answer (QA) based evaluation mechanism to automatically evaluate the plans and generate detailed plan refinement instructions to guide the iterative improvement. In the learning stage, we build a better planner by fine-tuning with the plan corpus or in-context learning with examples in the plan corpus. Finally, we leverage a hierarchical approach to generate long-form narratives. We evaluate the effectiveness of EIPE-text in the domains of novels and storytelling. Both GPT-4-based evaluations and human evaluations demonstrate that our method can generate more coherent and relevant long-form narratives. Our code will be released in the future.

• The paper introduces a novel evaluation-guided iterative plan extraction framework that refines narrative plans through QA-based feedback.
• The methodology combines tree-structured plan sketching with reinforcement-like iterative refinement, yielding coherent and domain-specific narratives.
• Experimental results show improved narrative coherence by 5.0% and relevance by 13.3%, outperforming state-of-the-art methods.

Evaluation-Guided Iterative Plan Extraction for Long-Form Narrative Text Generation

Introduction

The paper "EIPE-text: Evaluation-Guided Iterative Plan Extraction for Long-Form Narrative Text Generation" (2310.08185) presents a novel framework for generating coherent and relevant long-form narratives. EIPE-text tackles the challenges posed by existing hierarchical generation approaches that rely predominantly on LLMs for planning, and often yield suboptimal results without thorough domain-specific adaptations. The proposed system enhances the planning capabilities by automatically extracting high-quality plans from narrative corpora, thus enabling better planner learning.

Figure 1: A Comprehensive Visual Overview of the EIPE-text Framework. The Plan Extraction stage initiates with Plan Sketching, followed by QA-based Evaluation, refining, and constructing a plan corpus for learning.

Methodology

Plan Extraction

EIPE-text's methodology is divided into three distinct stages: plan extraction, learning, and inference. The plan extraction phase initiates with the creation of tree-structured plans based on the narrative corpus, leveraging LLMs in a sketching step. Subsequently, QA-pairs are generated to evaluate the plan's comprehensiveness. The self-evaluation process employs a QA-based mechanism to assess quality and alignment with the source narratives, offering detailed refinement instructions for iterative improvement.

Figure 2: An Example of the Plan Refinement Process.

This refinement is crucial for addressing errors identified during the evaluation, ensuring high-quality plans that preserve the narrative's thematic integrity. The iterative process continues until the plan meets predefined evaluation standards, a self-improving loop resembling reinforcement learning, effectively enhancing plan quality.

Learning

In the learning stage, EIPE-text employs two enhancement strategies: fine-tuning with the plan corpus and in-context learning with demonstration examples. Fine-tuning adjusts planner capabilities across multiple domains, while in-context learning permits rapid adaptation to specific styles using selected demonstrations, contributing to generating domain-specific high-quality plans.

Inference

The inference stage consists of two principal activities: plan generation and narrative generation. Here, the planner first generates a plan based on an input topic, followed by the creation of narratives using the structured plan. The hierarchical approach ensures logical organization and detailed narratives, improving coherence and relevance.

Figure 3: Average accuracy curve of iterative refinement process.

Experimental Results

The authors tested EIPE-text in domains such as novels and storytelling, demonstrating notable improvements over state-of-the-art baselines like recurrentGPT. Evaluations conducted using both GPT-4 and human judges revealed superior coherence and relevance in narratives generated by EIPE-text. Specifically, human evaluations reflected substantial coherence and relevance enhancements of 5.0% and 13.3%, respectively.

Implications

The implications of EIPE-text extend beyond practical applications in narrative generation to theoretical contributions in improving LLM planning capabilities. By integrating QA-based evaluations with iterative refinement, the framework offers a robust method for enhancing narrative coherence in diverse domains, including scriptwriting and journalism.

The paper suggests that EIPE-text could serve as a cornerstone for future AI systems that require narrative generation with high domain specificity, potentially leading to more creative and expressive writing tasks.

Limitations and Future Research

The effectiveness of EIPE-text largely depends on the reasoning capabilities of the underlying LLMs, thus limiting its application to models like GPT-4 and Claude that possess strong reasoning faculties. Additionally, the framework's reliance on domain-specific data implies challenges in out-of-domain generalization.

Future research may explore expansions of EIPE-text across diverse narrative domains and investigate methods for boosting its out-of-domain performance and scalability.

Conclusion

EIPE-text successfully addresses the inherent challenges in long-form narrative text generation, offering a sophisticated framework for plan extraction, learning, and inference. Its rigorous evaluation mechanism and iterative refinement process significantly enhance planning and narrative outcomes, paving the way for further exploration and advancements in AI-driven storytelling and text generation.