QACP: An Annotated Question Answering Dataset for Assisting Chinese Python Programming Learners (2402.07913v2)
Abstract: In online learning platforms, particularly in rapidly growing computer programming courses, addressing the thousands of students' learning queries requires considerable human cost. The creation of intelligent assistant LLMs tailored for programming education necessitates distinct data support. However, in real application scenarios, the data resources for training such LLMs are relatively scarce. Therefore, to address the data scarcity in intelligent educational systems for programming, this paper proposes a new Chinese question-and-answer dataset for Python learners. To ensure the authenticity and reliability of the sources of the questions, we collected questions from actual student questions and categorized them according to various dimensions such as the type of questions and the type of learners. This annotation principle is designed to enhance the effectiveness and quality of online programming education, providing a solid data foundation for developing the programming teaching assists (TA). Furthermore, we conducted comprehensive evaluations of various LLMs proficient in processing and generating Chinese content, highlighting the potential limitations of general LLMs as intelligent teaching assistants in computer programming courses.
- Can chatgpt play the role of a teaching assistant in an introductory programming course?, 2023.
- Unmasking the giant: A comprehensive evaluation of chatgpt’s proficiency in coding algorithms and data structures, 2023.
- Qwen technical report, 2023.
- A comparative study of ai-generated (gpt-4) and human-crafted mcqs in programming education. In Proceedings of the 26th Australasian Computing Education Conference, ACE 2024. ACM, Jan. 2024. doi: 10.1145/3636243.3636256. URL http://dx.doi.org/10.1145/3636243.3636256.
- Ai-ta: Towards an intelligent question-answer teaching assistant using open-source llms, 2023.
- Y. Katz and S. Romi. Affective education: The nature and characteristics of teachers and students attitudes toward school. Educational Practice and Theory, 25:35–47, 01 2003. doi: 10.7459/ept/25.1.04.
- Studying the effect of ai code generators on supporting novice learners in introductory programming. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023. URL https://api.semanticscholar.org/CorpusID:256868626.
- N. Kiesler and D. Schiffner. Large language models in introductory programming education: Chatgpt’s performance and implications for assessments, 2023.
- Exploring the potential of large language models to generate formative programming feedback, 2023.
- Learning from teaching assistants to program with subgoals: Exploring the potential for ai teaching assistants, 2023.
- J. Leppink. Cognitive load theory: Practical implications and an important challenge. Journal of Taibah University Medical Sciences, 12(5):385–391, 2017. ISSN 1658-3612. doi: https://doi.org/10.1016/j.jtumed.2017.05.003. URL https://www.sciencedirect.com/science/article/pii/S1658361217300835.
- Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021.
- C. Li. External knowledge augmented polyphone disambiguation using large language model, 2023.
- C.-Y. Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004.
- Retrieval augmented generation and representative vector summarization for large unstructured textual data in medical education, 2023.
- Augmented language models: a survey, 2023.
- M. Mizbani and A. Chalak. Analyzing reading and writing activities of iranian efl textbook prospect 3 based on bloom’s revised taxonomy. 4:13–27, 01 2017.
- OpenAI. Chatgpt: Optimizing language models for dialogue, 2023. URL https://openai.com/chatgpt. Software available from OpenAI.
- Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
- Direct preference optimization: Your language model is secretly a reward model, 2023.
- Thrilled by your progress! large language models (gpt-4) no longer struggle to pass assessments in higher education programming courses. In Proceedings of the 2023 ACM Conference on International Computing Education Research V.1, ICER 2023. ACM, Aug. 2023a. doi: 10.1145/3568813.3600142. URL http://dx.doi.org/10.1145/3568813.3600142.
- Large language models (gpt) struggle to answer multiple-choice questions about code, 2023b.
- Sotana: The open-source software development assistant, 2023.
- Llama 2: Open foundation and fine-tuned chat models, 2023.
- Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015.
- Clinicalgpt: Large language models finetuned with diverse medical data and comprehensive evaluation, 2023a.
- Data management for large language models: A survey, 2023b.
- Baichuan 2: Open large-scale language models, 2023.
- Glm-130b: An open bilingual pre-trained model, 2023.
- L. Zhong and Z. Wang. Can chatgpt replace stackoverflow? a study on robustness and reliability of large language model code generation, 2023.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.