Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B

Accessing GPT-4 Level Mathematical Olympiad Solutions via Monte Carlo Tree Self-Refine with LLaMa-3 8B

The paper "Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B" by Di Zhang et al., proposes the MCT Self-Refine (MCTSr) algorithm, which integrates LLMs with Monte Carlo Tree Search (MCTS) to improve performance in complex mathematical reasoning tasks. The primary objective is to address the accuracy and reliability issues faced by LLMs in strategic and logical reasoning contexts, such as mathematical Olympiads.

Methodology

The core methodology involves the systematic application of MCTS combined with LLMs' self-refine capabilities to construct a Monte Carlo search tree. The authors have tailored the traditional MCTS approach to fit the stochastic nature of LLM outputs:

1. Initialization: A root node is generated based on naive model-generated answers or dummy responses to minimize overfitting.
2. Selection: Nodes are selected based on their $Q$ value, which is computed using the model's self-reward mechanism.
3. Self-Refine: Nodes undergo iterative refinement, where the model generates feedback to improve the initial solution.
4. Self-Evaluation: Refined answers are scored, with constraints to ensure strict and fair evaluation.
5. Backpropagation: Values are propagated back to parent nodes to update the search tree.
6. UCT and Selection Updates: The Upper Confidence Bound (UCB) is updated to balance exploration and exploitation, guiding the selection for further refinement.
7. Termination: The process stops based on pre-defined criteria such as maximum depth or diminishing returns from additional rollouts.

The integration of these steps aims to refine answers iteratively and systematically, resulting in more accurate and reliable solutions to mathematical problems.

Experimental Evaluation

The performance of the MCTSr algorithm was evaluated using LLaMa3-8B on several datasets, including GSM8K, GSM-Hard, MATH, AIME, Math Odyssey, and OlympiadBench. The evaluations compared the results of MCTSr (with varying rollouts) against state-of-the-art models like GPT-4, Claude 3, and Gemini 1.5-Pro.

GSM Benchmarks

• GSM8K: MCTSr showed improvement from 74.07% (Zero-Shot CoT) to 96.66% (8-rollouts), indicating a significant enhancement in solving typical mathematical problems.
• GSM-Hard: The performance improved from 25.47% (Zero-Shot CoT) to 45.49% (8-rollouts), although the improvement plateaued, suggesting a limitation in solving more challenging problems.

MATH Benchmark

The MCTSr algorithm was also tested on the MATH dataset across five difficulty levels. Notable results include:

• Level 1: Success rate improved from 57.21% (Zero-Shot CoT) to 90.16% (8-rollouts).
• Level 5: The success rate increased from 7.10% (Zero-Shot CoT) to 34.06% (8-rollouts).

Overall, the cumulative success rate across all levels was enhanced from 24.36% to 58.24% with 8-rollouts MCTSr.

Olympiad-Level Benchmarks

The algorithm's efficacy was further validated on the AIME, Math Odyssey, and OlympiadBench datasets:

• AIME: Improved from 2.36% (Zero-Shot CoT) to 11.79% (8-rollouts).
• Math Odyssey: Showed substantial improvement from 17.22% (Zero-Shot CoT) to 49.36% (8-rollouts).
• OlympiadBench: Enhanced from 1.25% (Zero-Shot CoT) to 7.76% (8-rollouts).

Discussion and Implications

The results demonstrate that integrating MCTS with LLMs via MCTSr can significantly enhance the mathematical problem-solving capabilities of LLMs, reaching performance levels comparable to current state-of-the-art models. This algorithm shows promise in various applications, including educational technologies and automated reasoning systems.

Limitations and Future Work

While the MCTSr algorithm displays considerable potential, further research is necessary to explore its application in other decision-making frameworks such as black-box optimization and self-driven model alignment. Additionally, further refinement and comparison of component algorithms are essential to improve the algorithm's practical applicability and effectiveness.

Conclusion

The MCTSr algorithm successfully integrates MCTS with LLMs to enhance mathematical problem-solving capabilities, addressing critical challenges in accuracy and reliability. The significant improvements across various datasets underscore the potential for future innovations in AI-driven decision-making and reasoning tasks. The research sets a foundation for further exploration and optimization of AI technologies in complex problem-solving environments.