Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement (2402.14658v2)

Published 22 Feb 2024 in cs.SE, cs.AI, and cs.CL

Abstract: The introduction of LLMs has significantly advanced code generation. However, open-source models often lack the execution capabilities and iterative refinement of advanced systems like the GPT-4 Code Interpreter. To address this, we introduce OpenCodeInterpreter, a family of open-source code systems designed for generating, executing, and iteratively refining code. Supported by Code-Feedback, a dataset featuring 68K multi-turn interactions, OpenCodeInterpreter integrates execution and human feedback for dynamic code refinement. Our comprehensive evaluation of OpenCodeInterpreter across key benchmarks such as HumanEval, MBPP, and their enhanced versions from EvalPlus reveals its exceptional performance. Notably, OpenCodeInterpreter-33B achieves an accuracy of 83.2 (76.4) on the average (and plus versions) of HumanEval and MBPP, closely rivaling GPT-4's 84.2 (76.2) and further elevates to 91.6 (84.6) with synthesized human feedback from GPT-4. OpenCodeInterpreter brings the gap between open-source code generation models and proprietary systems like GPT-4 Code Interpreter.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Program synthesis with large language models. ArXiv preprint, abs/2108.07732.
  2. Qwen technical report. ArXiv preprint, abs/2309.16609.
  3. GPT-NeoX-20B: An open-source autoregressive language model. In Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models, pages 95–136, virtual+Dublin. Association for Computational Linguistics.
  4. Sahil Chaudhary. 2023. Code Alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/codealpaca. Accessed: 2024-02-13.
  5. Evaluating large language models trained on code. ArXiv preprint, abs/2107.03374.
  6. Teaching large language models to self-debug. ArXiv preprint, abs/2304.05128.
  7. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113.
  8. Pangu-coder: Program synthesis with function-level language modeling. ArXiv preprint, abs/2207.11280.
  9. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  10. Codefuse-13b: A pretrained multi-lingual code large language model. ArXiv preprint, abs/2310.06266.
  11. Stepcoder: Improve code generation with reinforcement learning from compiler feedback. ArXiv preprint, abs/2402.01391.
  12. Incoder: A generative model for code infilling and synthesis. ArXiv preprint, abs/2204.05999.
  13. GitHub. 2023. Github copilot. https://github.com/features/copilot. Accessed: 2024-02-14.
  14. Gemini: a family of highly capable multimodal models. ArXiv preprint, abs/2312.11805.
  15. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. ArXiv preprint, abs/2401.14196.
  16. Mistral 7b. ArXiv preprint, abs/2310.06825.
  17. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314–21328.
  18. Starcoder: may the source be with you! ArXiv preprint, abs/2305.06161.
  19. Taco: Topics in algorithmic code generation dataset. ArXiv preprint, abs/2312.14852.
  20. Competition-level code generation with alphacode. Science, 378(6624):1092–1097.
  21. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems.
  22. Wizardcoder: Empowering code large language models with evol-instruct. ArXiv preprint, abs/2306.08568.
  23. Memory-assisted prompt editing to improve GPT-3 after deployment. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2833–2861, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  24. Self-refine: Iterative refinement with self-feedback. ArXiv preprint, abs/2303.17651.
  25. Codegen: An open large language model for code with multi-turn program synthesis. ArXiv preprint, abs/2203.13474.
  26. OpenAI. 2022. ChatGPT: Optimizing Language Models for Dialogue. https://openai.com/blog/chatgpt/. Accessed on 14 Feb. 2024.
  27. OpenAI. 2023. Gpt-4 technical report.
  28. Phind. 2023. Phind/phind-codellama-34b-v2.
  29. Scaling language models: Methods, analysis & insights from training gopher. ArXiv preprint, abs/2112.11446.
  30. Code llama: Open foundation models for code. ArXiv preprint, abs/2308.12950.
  31. Self-critiquing models for assisting human evaluators. ArXiv preprint, abs/2206.05802.
  32. Training language models with natural language feedback. ArXiv preprint, abs/2204.14146.
  33. In-context pretraining: Language modeling beyond document boundaries. ArXiv preprint, abs/2310.10638.
  34. Execution-based code generation using deep reinforcement learning. ArXiv preprint, abs/2301.13816.
  35. speechless. 2023. speechless-codellama-34b-v2.0.
  36. Interscript: A dataset for interactive learning of scripts through error feedback. ArXiv preprint, abs/2112.07867.
  37. Galactica: A large language model for science. ArXiv preprint, abs/2211.09085.
  38. Lamda: Language models for dialog applications. ArXiv preprint, abs/2201.08239.
  39. Debugbench: Evaluating debugging capability of large language models. ArXiv preprint, abs/2401.04621.
  40. Llama: Open and efficient foundation language models. ArXiv preprint, abs/2302.13971.
  41. Openchat: Advancing open-source language models with mixed-quality data. ArXiv preprint, abs/2309.11235.
  42. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 13484–13508.
  43. Codet5+: Open code large language models for code understanding and generation. ArXiv preprint, abs/2305.07922.
  44. Magicoder: Source code is all you need. ArXiv preprint, abs/2312.02120.
  45. Xwin-LM. 2023. Xwin-lm.
  46. Wavecoder: Widespread and versatile enhanced instruction tuning with refined data generation. ArXiv preprint, abs/2312.14187.
  47. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. ArXiv preprint, abs/2303.17568.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Tianyu Zheng (28 papers)
  2. Ge Zhang (170 papers)
  3. Tianhao Shen (15 papers)
  4. Xueling Liu (5 papers)
  5. Bill Yuchen Lin (72 papers)
  6. Jie Fu (229 papers)
  7. Wenhu Chen (134 papers)
  8. Xiang Yue (72 papers)
Citations (67)
View on Semantic Scholar

Summary

  • The paper presents an open-source system that integrates code generation with live execution and iterative refinement through multi-turn feedback.
  • It leverages a diverse Code-Feedback dataset of 68,000 interactions to improve performance across benchmarks like HumanEval and MBPP.
  • Experimental results show the model achieving performance on par with GPT-4, highlighting its potential for handling complex coding tasks.

OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement

Introduction

The paper introduces OpenCodeInterpreter, an open-source code generation system designed to integrate code generation with execution and iterative refinement. A significant challenge within code generation has been the disparity between proprietary systems, such as the GPT-4 Code Interpreter, and open-source models, which generally lack the same level of execution capabilities and dynamic refinement through feedback. OpenCodeInterpreter targets this gap by building on a richly constructed dataset called Code-Feedback, consisting of 68,000 multi-turn interactions which include elements of both execution feedback and human feedback.

Methodology

The design of OpenCodeInterpreter hinges on three core components:

  1. Code Generation: Utilizing pre-trained LLMs on extensive code-centric datasets.
  2. Execution: Directly running generated code to gather diagnostic feedback.
  3. Iterative Refinement: Incorporating multi-turn interactions to refine code iteratively based on feedback.

Dataset Construction: Code-Feedback

The Code-Feedback dataset is critical to OpenCodeInterpreter's success. It comprises a mix of queries sourced from open-source datasets and coding challenges from LeetCode, processed to ensure a diverse and challenging collection of tasks. Noteworthy elements include:

  • Multi-Turn Dialogues: Structured interactions that simulate real-world coding scenarios, focusing on iterative feedback.
  • Diverse Data Collection Methods: Encompassing single-turn packing, interaction simulation, code correction, LeetCode similar problems, and LeetCode follow-up.

These methods ensure that Code-Feedback not only covers a wide array of coding challenges but also fosters robust engagement with execution and human feedback.

Experimental Evaluation

The paper presents a thorough evaluation of OpenCodeInterpreter against established benchmarks such as HumanEval, MBPP, and their enhanced versions from EvalPlus, demonstrating its performance across model scales (7B, 13B, 34B, and 70B parameters). Remarkably, OpenCodeInterpreter-33B achieves an average accuracy of 83.2 (76.4 for the plus versions) on HumanEval and MBPP, closely matching GPT-4's 84.2 (76.2). With synthesized human feedback from GPT-4, this performance further elevates to 91.6 (84.6), showcasing the efficacy of iterative refinement and feedback integration.

Results

Single-Turn Code Generation: OpenCodeInterpreter significantly outperforms other open-source models, achieving results on par with or surpassing proprietary models. This is evident across different scales and configurations, highlighting its robustness.

Multi-Turn Code Generation: When evaluated on multi-turn tasks involving execution feedback and synthetic human feedback, OpenCodeInterpreter demonstrates superior refinement capabilities. It consistently performs better than leading models in incorporating iterative feedback to correct and enhance code functionality.

Discussion

The research denotes several key implications:

  • Integrated Execution and Feedback: The dynamic incorporation of execution results and human feedback into the code generation process bridges a notable gap between open-source and proprietary models.
  • Model Adaptability: Iterative refinement allows for better handling of complex and ambiguous user intents, improving overall code accuracy and reliability.
  • Scalability of Training Data: The construction and utilization of Code-Feedback emphasize the importance of diverse, multi-turn interactions to train more effective code generation models.

Future Developments

Future progress in AI-driven code generation could see:

  • Enhanced Feedback Mechanisms: Incorporating more sophisticated diagnostic tools and nuanced human feedback to further boost model performance.
  • Cross-Domain Applications: Expanding code generation capabilities into other domains such as data science or low-level programming, leveraging the adaptability of models like OpenCodeInterpreter.
  • Community Contributions: Engaging the open-source community to expand and refine datasets, continually improving the robustness and applicability of models.

Conclusion

OpenCodeInterpreter marks a significant advancement in integrating code generation with execution and refinement. By effectively leveraging multi-turn interactions through execution and human feedback, OpenCodeInterpreter narrows the gap between open-source and proprietary systems. This approach not only elevates performance standards but also sets a new precedent for future research and development in the field of automated code generation.

Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube