Lean Copilot: Large Language Models as Copilots for Theorem Proving in Lean (2404.12534v3)

Abstract: Neural theorem proving combines LLMs with proof assistants such as Lean, where the correctness of formal proofs can be rigorously verified, leaving no room for hallucination. With existing neural theorem provers pretrained on a fixed collection of data and offering valuable suggestions at times, it is challenging for them to continually prove novel theorems in a fully autonomous mode, where human insights may be critical. In this paper, we explore LLMs as copilots that assist humans in proving theorems. We introduce Lean Copilot, a general framework for running LLM inference natively in Lean. It enables programmers to build various LLM-based proof automation tools that integrate seamlessly into the workflow of Lean users. Lean users can use our pretrained models or bring their own ones that run either locally (with or without GPUs) or on the cloud. Using Lean Copilot, we build LLM-based tools that suggest proof steps, complete proof goals, and select relevant premises. Experimental results on the Mathematics in Lean textbook demonstrate the effectiveness of our method compared to existing rule-based proof automation in Lean (aesop). When assisting humans, Lean Copilot requires only 2.08 manually-entered proof steps on average (3.86 required by aesop); when automating the theorem proving process, Lean Copilot automates 74.2% proof steps on average, 85% better than aesop (40.1%). We open source all code and artifacts under a permissive MIT license to facilitate further research.

• The paper introduces Lean Copilot, a framework that integrates LLMs into Lean to suggest tactics, automate proof search, and enhance premise selection.
• The paper details a direct C++ FFI integration of LLM inference, ensuring fast, seamless performance without relying on external API calls.
• The paper demonstrates improved theorem proving efficiency and reduced manual intervention, making Lean more accessible to users.

Lean Copilot: Integrating LLMs into Lean for Enhanced Theorem Proving Assistance

Introduction

The advent of LLMs has heralded promising advancements in the domain of automated theorem proving, traditionally handled by interactive theorem proving systems (ITPs) and proof assistants. Research has predominantly focused on autonomous theorem proving using LLMs. However, existing models often fall short when tackling novel or complex problems outside their trained domains, underscoring the necessity of human insight. Addressing this, we explore a paradigm where LLMs operate as copilots, assisting rather than replacing human reasoners in theorem proving. We introduce "Lean Copilot," a framework specifically designed for embedding LLM capabilities directly into the Lean proof assistant environment.

Framework Overview

Lean Copilot is engineered to facilitate the integration of LLM inference within the Lean environment, enabling the development of tools that assist in various aspects of theorem proving without disrupting existing user workflows. The core functionalities provided include:

• Tactic Suggestion: Proposes possible next steps in a proof, aiding users in navigating through proofs by suggesting logical subsequent tactics based on the current state of the theorem.
• Proof Search: Enhances existing rule-based search methods by integrating LLM-generated tactics, dynamically adapting strategy based on the context of the proof.
• Premise Selection: Efficiently selects relevant premises that are likely to contribute toward goal resolution, streamlining the proof process.

These tools are designed to operate seamlessly within the Lean environment, requiring minimal setup and compatible with common hardware configurations, including those without GPUs.

Technical Implementation

Lean Copilot operates by integrating LLM inference directly into Lean through the Foreign Function Interface (FFI) with C++, leveraging efficiency and avoiding the overhead of external API calls or process communications. This direct integration ensures that the tool maintains the interaction speed users expect from their native Lean environment.

The framework supports a variety of inference models but by default uses a pretrained ByT5 model from the LeanDojo project, optimized for the tactical generation of Lean proofs. Users have the flexibility to employ custom models, which the system can execute either locally or via a server, thus accommodating a wide range of computational and operational requirements.

Evaluation and Impact

Lean Copilot was evaluated against standard proof environments and showed considerable improvements in facilitating theorem proving. It proved capable of automating significant portions of proofs that traditionally required human intervention. This not only speeds up the theorem proving process but also makes Lean more accessible to users who may not be expert theorem provers.

In practical terms, Lean Copilot has been shown to reduce the number of manual interventions needed in proving theorems and to enhance the capability of the existing aesop tool in Lean through its sophisticated LLM-based proof search strategy. With these advancements, Lean Copilot is set to make a substantial impact on how theorems are proved, making formal mathematics both more efficient and accessible.

Future Directions

Looking forward, the successful integration of LLMs within the Lean environment opens the door to further innovations in automated reasoning and theorem proving. As more data becomes available from enhanced usage of tools like Lean Copilot, LLMs themselves can be improved, potentially leading to a virtuous cycle of enhancements in both tool capabilities and base model accuracies. Moreover, the open-source nature of Lean Copilot encourages ongoing collaboration and innovation, potentially leading to developments that could expand its applicability beyond mathematics into other domains requiring formal verification and logical reasoning.

Conclusion

The introduction of Lean Copilot marks a significant step forward in the integration of machine learning models with interactive theorem proving environments. By bridging the gap between LLM capabilities and the needs of theorem provers, Lean Copilot not only enhances the efficiency of proving theorems but also enriches the theorem proving process through effective human-AI collaboration.