Language Modelling for Source Code with Transformer-XL (2007.15813v1)

Abstract: It has been found that software, like natural language texts, exhibits "naturalness", which can be captured by statistical LLMs. In recent years, neural LLMs have been proposed to represent the naturalness of software through deep learning. In this paper, we conduct an experimental evaluation of state-of-the-art neural LLMs for source code, including RNN-based models and Transformer-XL based models. Through experiments on a large-scale Python code corpus, we find that the Transformer-XL model outperforms RNN-based models (including LSTM and GRU models) in capturing the naturalness of software, with far less computational cost.