Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions (2401.09395v6)
Abstract: Recent advancements in LLMs have showcased striking results on existing logical reasoning benchmarks, with some models even surpassing human performance. However, the true depth of their competencies and robustness in reasoning tasks remains an open question. To this end, in this paper, we focus on two popular reasoning tasks: arithmetic reasoning and code generation. Particularly, we introduce (i) a general ontology of perturbations for math and coding questions, (ii) a semi-automatic method to apply these perturbations, and (iii) two datasets, GSMORE and HUMANEVAL-CORE, respectively, of perturbed math and coding problems to probe LLM capabilities in numeric reasoning and coding tasks. Through comprehensive evaluations of both closed-source and open-source LLMs, we show a significant performance drop across all the models against the perturbed questions, suggesting that the current LLMs lack robust problem solving skills and structured reasoning abilities in many areas, as defined by our ontology. We open-source the datasets and source codes at: https://github.com/declare-lab/LLM-ReasoningTest.
- Training verifiers to solve math word problems. ArXiv, abs/2110.14168, 2021a. URL https://api.semanticscholar.org/CorpusID:239998651.
- Measuring massive multitask language understanding. ArXiv, abs/2009.03300, 2020. URL https://api.semanticscholar.org/CorpusID:221516475.
- Measuring mathematical problem solving with the math dataset, 2021.
- Beyond accuracy: Behavioral testing of nlp models with checklist. arXiv preprint arXiv:2005.04118, 2020.
- Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks, 2023.
- Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023.
- Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct, 2023a.
- Wizardcoder: Empowering code large language models with evol-instruct, 2023b.
- Time travel in llms: Tracing data contamination in large language models, 2023.
- Measuring reliability of large language models through semantic consistency, 2023.
- Metamath: Bootstrap your own mathematical questions for large language models, 2023.
- Training verifiers to solve math word problems. ArXiv, abs/2110.14168, 2021b. URL https://api.semanticscholar.org/CorpusID:239998651.
- OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
- Mistral 7b. ArXiv, abs/2310.06825, 2023. URL https://api.semanticscholar.org/CorpusID:263830494.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.