Exploration of Neural Machine Translation in Autoformalization of Mathematics in Mizar (1912.02636v2)

Abstract: In this paper we share several experiments trying to automatically translate informal mathematics into formal mathematics. In our context informal mathematics refers to human-written mathematical sentences in the LaTeX format; and formal mathematics refers to statements in the Mizar language. We conducted our experiments against three established neural network-based machine translation models that are known to deliver competitive results on translating between natural languages. To train these models we also prepared four informal-to-formal datasets. We compare and analyze our results according to whether the model is supervised or unsupervised. In order to augment the data available for auto-formalization and improve the results, we develop a custom type-elaboration mechanism and integrate it in the supervised translation.

• The paper presents an NMT framework that transforms informal LaTeX expressions into formal Mizar code, evaluated using BLEU scores, perplexity, and edit distances.
• It employs both supervised and unsupervised models, including cross-lingual pretraining and back-translation, to handle varied dataset alignments.
• The study demonstrates that iterative data augmentation through type elaboration can enhance translation accuracy and formalization quality.

Exploration of Neural Machine Translation in Autoformalization of Mathematics in Mizar

Introduction

The paper explores the use of neural machine translation (NMT) techniques to automate the formalization of mathematics, transforming informal mathematical expressions written in LaTeX into formal code in the Mizar language. Autoformalization is a critical area aiming to convert broad swathes of informal mathematical literature into formal language comprehensible by interactive theorem proving (ITP) systems. This task leverages NMT models, traditionally successful in NLP, to bridge the gap between informal and formal mathematical communication.

Informal-To-Formal Datasets

The research utilizes several datasets that reflect different facets of mathematical literature. The primary dataset is a synthetic LaTeX-Mizar corpus, generated using a tool that transcribes theorems and proof statements from Mizar articles into LaTeX format, producing aligned data suitable for training supervised NMT models. In addition, ProofWiki provides a source of aligned data, linking human-written LaTeX sentences with Mizar formal representations. The choice of Mizar is justified based on its comprehensive mathematical library and use of Tarski-Grothendieck set theory, aligning well with the project's goals.

Neural Machine Translation Models

Three NMT models underpin the experimental framework:

1. Supervised NMT (Luong et al.): This model uses an encoder-decoder architecture, augmented with attention mechanisms, trained on aligned LaTeX-Mizar data. The flexible architecture supports adjustments in hyperparameters to optimize translation performance between closely aligned datasets.
2. Unsupervised NMT (UNMT): This model operates without direct alignment data, utilizing a shared encoder and separate decoders for each language. The innovative use of back-translation transforms the translation problem into a series of supervised tasks, increasing versatility across more varied datasets.
3. Cross-lingual Pretraining (XLM): Building on UNMT, XLM integrates BERT-style pretraining, drastically enhancing unsupervised learning capabilities by refining language representations across both source and target languages prior to translation tasks.

Experiments and Results

The models were evaluated using BLEU scores, perplexity, and edit distances, showcasing traditional and advanced metrics in NLP to assess translation quality. The supervised NMT exhibits superior performance on the synthetic dataset due to its reliance on aligned data. However, innovations in unsupervised translation methods, particularly XLM, demonstrate promise in unaligned contexts despite lower initial performance.

Data Augmentation Using Type Elaboration

A feedback loop integrating type elaboration with supervised NMT is proposed to improve translation quality. By generating multiple potential translations and validating them using type elaboration, additional training data is iteratively generated. This process enhances the model's ability to produce syntactically correct translations, shown by improved exact match rates over successive iterations with modest data increases.

Conclusion

The exploration of NMT for the autoformalization of mathematics suggests that while substantial progress can be made with existing models, further advancements require composite strategies incorporating augmented datasets and novel training methodologies. The method outlined using type elaboration illustrates a potential pathway for significantly expanding formalization efforts, providing a framework for future research that could extend beyond current limitations in dataset size and translation accuracy. Collaboration between mathematical and AI communities will be paramount in realizing the full potential of automated formalization.

Continued exploration of dataset enrichment techniques, multi-modal input formats, and cross-platform library integration remains crucial, reflecting a persistent focus on aligning and leveraging the cognitive capabilities of machine learning with the rigorous demands of mathematical proof systems.