HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization (2402.16694v2)
Abstract: LLMs have made significant progress in generating codes from textual prompts. However, existing benchmarks have mainly concentrated on translating English prompts to multilingual codes or have been constrained to very limited natural languages (NLs). These benchmarks have overlooked the vast landscape of massively multilingual NL to multilingual code, leaving a critical gap in the evaluation of multilingual LLMs. In response, we introduce HumanEval-XL, a massively multilingual code generation benchmark specifically crafted to address this deficiency. HumanEval-XL establishes connections between 23 NLs and 12 programming languages (PLs), and comprises of a collection of 22,080 prompts with an average of 8.33 test cases. By ensuring parallel data across multiple NLs and PLs, HumanEval-XL offers a comprehensive evaluation platform for multilingual LLMs, allowing the assessment of the understanding of different NLs. Our work serves as a pioneering step towards filling the void in evaluating NL generalization in the area of multilingual code generation. We make our evaluation code and data publicly available at \url{https://github.com/FloatAI/humaneval-xl}.
- Multi-lingual evaluation of code generation models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
- Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
- Ernie-code: Beyond english-centric cross-lingual pretraining for programming languages. In Annual Meeting of the Association for Computational Linguistics.
- Training and evaluating a jupyter notebook data science assistant.
- Evaluating large language models trained on code.
- Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
- Measuring coding challenge competence with apps. NeurIPS.
- Training compute-optimal large language models. CoRR, abs/2203.15556.
- The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Computational Linguistics.
- Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pages 18319–18345. PMLR.
- Starcoder: may the source be with you! CoRR, abs/2305.06161.
- Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019–9052.
- Codegen2: Lessons for training llms on programming and natural languages. arXiv preprint arXiv:2305.02309.
- Codegen: An open large language model for code with multi-turn program synthesis. ICLR.
- OpenAI. 2022. Introducing chatgpt.
- OpenAI. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
- Bloom: A 176b-parameter open-access multilingual language model. ArXiv, abs/2211.05100.
- Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922.
- Execution-based evaluation for open-domain code generation. ArXiv, abs/2212.10481.
- Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
- Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’23, page 5673–5684, New York, NY, USA. Association for Computing Machinery.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.