From Human Days to Machine Seconds: Automatically Answering and Generating Machine Learning Final Exams
(2206.05442)Abstract
A final exam in machine learning at a top institution such as MIT, Harvard, or Cornell typically takes faculty days to write, and students hours to solve. We demonstrate that LLMs pass machine learning finals at a human level, on finals available online after the models were trained, and automatically generate new human-quality final exam questions in seconds. Previous work has developed program synthesis and few-shot learning methods to solve university-level problem set questions in mathematics and STEM courses. In this work, we develop and compare methods that solve final exams, which differ from problem sets in several ways: the questions are longer, have multiple parts, are more complicated, and span a broader set of topics. We curate a dataset and benchmark of questions from machine learning final exams available online and code for answering these questions and generating new questions. We show how to generate new questions from other questions and course notes. For reproducibility and future research on this final exam benchmark, we use automatic checkers for multiple-choice, numeric, and questions with expression answers. We perform ablation studies comparing zero-shot learning with few-shot learning and chain-of-thought prompting using GPT-3, OPT, Codex, and ChatGPT across machine learning topics and find that few-shot learning methods perform best. We highlight the transformative potential of language models to streamline the writing and solution of large-scale assessments, significantly reducing the workload from human days to mere machine seconds. Our results suggest that rather than banning LLMs such as ChatGPT in class, instructors should teach students to harness them by asking students meta-questions about correctness, completeness, and originality of the responses generated, encouraging critical thinking in academic studies.
Overview
-
This paper explores the capabilities of LLMs in answering and generating university-level machine learning final exam questions, presenting a novel dataset from prestigious institutions.
-
It benchmarks various LLMs, including GPT-3, Codex, OPT, and ChatGPT, revealing that few-shot learning methods excel in both solving existing exams and generating new questions.
-
The use of LLMs in educational settings could streamline exam preparation, enhance learning and assessment, and signal a transformative potential of AI in education.
-
The study highlights a limitation in addressing only text and mathematical notation-based questions, suggesting future research on multimodal LLMs for image-based content.
Automatically Answering and Generating Machine Learning Final Exams: An Exploration with LLMs
Introduction
As the field of ML continues to evolve, so too do the methods for educating the next generation of ML professionals. This paper introduces a robust examination of the capabilities of LLMs in the context of university-level machine learning final exams. It presents a novel dataset comprised of ML final exam questions from prestigious institutions and benchmarks various LLMs' abilities to both solve and generate new exam content. The findings reveal that these models can perform at human levels on existing finals and generate new exam questions that are indistinguishable from those created by humans.
Dataset and Benchmarks
At the core of this exploration is a meticulously curated dataset of final exam questions from MIT, Harvard, and Cornell's introductory ML courses, creating a diverse spectrum of topics within the field. This dataset includes a total of 646 question parts across 149 questions, offering a broad examination of the subject matter taught in these courses.
The benchmarking effort compares a variety of LLMs, including GPT-3, Codex, OPT, and ChatGPT, with different prompting schemes such as zero-shot and few-shot learning, as well as chain-of-thought prompting. The results from these benchmarks are telling: few-shot learning methods consistently outperform their counterparts across different semesters and topics. For example, when evaluating non-image-based questions from MIT's Spring 2022 exams, few-shot learning with GPT-3 achieved a notable performance, aligning closely with human levels.
Implications and Future Directions
The successful application of LLMs in solving and generating ML exam content has several immediate implications:
- Educational Efficiency: The ability of LLMs to generate new, quality exam questions in seconds can significantly reduce the workload on faculty and teaching assistants, thereby streamlining the examination preparation process.
- Learning and Assessment: Instead of outright banning LLMs in academic settings due to concerns over cheating, this paper advocates for a constructive approach. By integrating LLMs into the learning process, educators can develop meta-questions that help students refine their critical thinking and problem-solving skills.
- Future of AI in Education: The findings hint at the transformative potential of AI in educational settings. This could redefine not just how content is created but also how students interact with and learn from this content.
A notable limitation of this study is its focus on text and mathematical notation-based questions, excluding those that require visual aids for solving. Future research could look into overcoming this limitation by incorporating multimodal LLMs capable of understanding and generating image-based content.
Conclusion
This research marks a significant stride in understanding the capacity of LLMs to support and enhance learning in complex fields like machine learning. The ability of these models to automate the time-consuming task of generating exam content—and to do so at a level comparable to human experts—opens up new avenues for educational tools and methodologies. Moving forward, it will be crucial to explore how these technologies can be integrated responsibly into educational frameworks to augment the learning experience without diminishing the value of human instruction and insight.