The remarkable progress of Multi-modal LLMs (MLLMs) has garnered unparalleled attention, due to their superior performance in visual contexts. However, their capabilities in visual math problem-solving remain insufficiently evaluated and understood. We investigate current benchmarks to incorporate excessive visual content within textual questions, which potentially assist MLLMs in deducing answers without truly interpreting the input diagrams. To this end, we introduce MathVerse, an all-around visual math benchmark designed for an equitable and in-depth evaluation of MLLMs. We meticulously collect 2,612 high-quality, multi-subject math problems with diagrams from publicly available sources. Each problem is then transformed by human annotators into six distinct versions, each offering varying degrees of information content in multi-modality, contributing to 15K test samples in total. This approach allows MathVerse to comprehensively assess whether and how much MLLMs can truly understand the visual diagrams for mathematical reasoning. In addition, we propose a Chain-of-Thought (CoT) evaluation strategy for a fine-grained assessment of the output answers. Rather than naively judging True or False, we employ GPT-4(V) to adaptively extract crucial reasoning steps, and then score each step with detailed error analysis, which can reveal the intermediate CoT reasoning quality by MLLMs. We hope the MathVerse benchmark may provide unique insights to guide the future development of MLLMs. Project page: https://mathverse-cuhk.github.io
MathVerse is a benchmark created to evaluate Multi-modal LLMs (MLLMs) on visual math problems, containing 2,612 problems transformed into 15,672 test samples.
The benchmark aims to assess MLLMs' abilities to interpret diagrams, by providing problems with varying degrees of information content across modalities, pushing models beyond relying on textual data.
Features of MathVerse include a rich dataset covering various math areas, a Chain-of-Thought evaluation method for detailed reasoning analysis, and a comprehensive assessment through diverse problem sets.
Experiments with leading MLLMs revealed a reliance on textual cues over visual interpretation, highlighting a performance gap between models and human solvers in visual reasoning capabilities.
MathVerse is an innovative benchmark designed to rigorously assess the capabilities of Multi-modal LLMs (MLLMs) in solving visual math problems. This benchmark distinguishes itself by focusing on the actual interpretation of diagrams by MLLMs, rather than relying predominantly on accompanying textual descriptions. MathVerse comprises 2,612 visual math problems, each meticulously transformed into six versions with varying degrees of information content across modalities, resulting in a comprehensive dataset of 15,672 test samples.
Current benchmarks often do not accurately evaluate an MLLM's ability to interpret visual information within math problems. They typically contain redundant textual information which MLLMs could exploit, bypassing the need for genuine diagram understanding. MathVerse addresses this by offering problems with progressively reduced textual content and enhanced diagram details, compelling models to rely more on visual interpretation for problem solving.
Through extensive experiments involving leading MLLMs, MathVerse reveals significant insights:
MathVerse stands as a pivotal step towards truly understanding and enhancing the visual mathematical reasoning abilities of MLLMs. The insights gained from this benchmark pave the way for future development in this area. Possible directions include improving visual encoders within MLLMs, developing richer training datasets encompassing a broader range of mathematical concepts and increasing the diversity of problem types to include multilingual and higher-difficulty problems.