Formal-LLM: Integrating Formal Language and Natural Language for Controllable LLM-based Agents (2402.00798v4)

Abstract: Recent advancements on LLMs enable AI Agents to automatically generate and execute multi-step plans to solve complex tasks. However, since LLM's content generation process is hardly controllable, current LLM-based agents frequently generate invalid or non-executable plans, which jeopardizes the performance of the generated plans and corrupts users' trust in LLM-based agents. In response, this paper proposes a novel "Formal-LLM" framework for LLM-based agents by integrating the expressiveness of natural language and the precision of formal language. Specifically, the framework allows agent developers to express their requirements or constraints for the planning process as an automaton. A stack-based LLM plan generation process is then conducted under the supervision of the automaton to ensure that the generated plan satisfies the constraints, making the planning process controllable. We conduct experiments on both benchmark tasks and practical real-life tasks, and our framework achieves over 50% overall performance increase, which validates the feasibility and effectiveness of employing Formal-LLM to guide the plan generation of agents, preventing the agents from generating invalid and unsuccessful plans. Further, more controllable LLM-based agents can facilitate the broader utilization of LLM in application scenarios where high validity of planning is essential. The source code of this work is available at https://github.com/agiresearch/Formal-LLM.

• The paper introduces the Formal-LLM framework that integrates context-free grammars and pushdown automata to rigorously enforce constraints on LLM-generated plans.
• It employs a combination of backtracking and reinforcement learning to enhance plan execution validity by 50% over traditional LLM methods.
• The framework demonstrates practical applicability in complex, multi-step tasks such as risk management, highlighting its potential for real-world deployment.

"Formal-LLM: Integrating Formal Language and Natural Language for Controllable LLM-based Agents" (2402.00798)

Introduction

The intersection of formal languages and LLMs presents an opportunity to enhance the controllability of LLM-based agents. LLMs, while proficient in natural language generation, often lack the necessary precision for complex task planning. The paper proposes the "Formal-LLM" framework, which leverages the strength of formal languages to impose constraints and improve the execution of multi-step plans by agents.

Framework Overview

The Formal-LLM framework integrates context-free grammars (CFGs) and pushdown automata (PDA) to encode and enforce constraints on LLM-generated plans.

• Formal Language Integration: Users define constraints using CFGs that are subsequently converted into PDAs (Figure 1). This ensures rigorous validation of plans against user constraints.
• Pushdown Automaton: PDAs enable the representation of hierarchical and recursive structures in task plans, providing a formal mechanism to assess the validity of decisions made by the LLM.

  Figure 1: The Formal-LLM workflow with a toy example, illustrating how CFGs and PDAs are utilized to enforce constraints on LLM plan generation.

Implementation Details

Workflow Construction: Human users specify task constraints via CFGs, which are translated into PDAs (Figure 2).

• Example PDA: An automaton is detailed, elucidating transitions based on stack operations that uphold plan validity (Figure 3).

  Figure 2: A PDA example illustrating the state transitions and stack operations for a given CFG.

  

  Figure 3: An equivalent PDA for a CFG, showcasing how specific transitions map to plan constraints.

Planning Adherence: During plan generation, the LLM is guided through state transitions defined by the PDA, ensuring adherence to the constraints. When a dead-end is reached, backtracking mechanisms are employed.

• Backtracking and Reinforcement Learning: Enhancements like backtracking allow the re-evaluation of prior decisions, while reinforcement learning improves plan quality by iteratively refining the policy based on successful executions.

Experiments

Benchmark Evaluation: Formal-LLM is applied to both benchmark and real-world tasks, achieving a notable 50% improvement in plan execution validity compared to traditional LLM approaches.

Task Complexity: Addressed complex, multi-input-output tasks requiring recursive decision processes, notably exceeding baseline performance in terms of executable plan percentage and accuracy.

Real-World Applications: Implementations in domains such as risk management (Figure 4) demonstrate the framework's potential for use in areas demanding high execution validity and precision.

Figure 4: A complex PDA for planning daily activities demonstrating the transition management via Formal-LLM.

Conclusion

The Formal-LLM framework bridges the gap between natural and formal language domains, enhancing agent-based task execution by enforcing constraint adherence and plan validity. Future work may explore automating constraint formulation and extending the framework to probabilistic automata for adaptability in dynamic environments.