Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Discovering Mathematical Formulas from Data via GPT-guided Monte Carlo Tree Search (2401.14424v3)

Published 24 Jan 2024 in cs.LG and cs.AI

Abstract: Finding a concise and interpretable mathematical formula that accurately describes the relationship between each variable and the predicted value in the data is a crucial task in scientific research, as well as a significant challenge in artificial intelligence. This problem is referred to as symbolic regression, which is an NP-hard problem. In the previous year, a novel symbolic regression methodology utilizing Monte Carlo Tree Search (MCTS) was advanced, achieving state-of-the-art results on a diverse range of datasets. although this algorithm has shown considerable improvement in recovering target expressions compared to previous methods, the lack of guidance during the MCTS process severely hampers its search efficiency. Recently, some algorithms have added a pre-trained policy network to guide the search of MCTS, but the pre-trained policy network generalizes poorly. To optimize the trade-off between efficiency and versatility, we introduce SR-GPT, a novel algorithm for symbolic regression that integrates Monte Carlo Tree Search (MCTS) with a Generative Pre-Trained Transformer (GPT). By using GPT to guide the MCTS, the search efficiency of MCTS is significantly improved. Next, we utilize the MCTS results to further refine the GPT, enhancing its capabilities and providing more accurate guidance for the MCTS. MCTS and GPT are coupled together and optimize each other until the target expression is successfully determined. We conducted extensive evaluations of SR-GPT using 222 expressions sourced from over 10 different symbolic regression datasets. The experimental results demonstrate that SR-GPT outperforms existing state-of-the-art algorithms in accurately recovering symbolic expressions both with and without added noise.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Neural symbolic regression that scales. In International Conference on Machine Learning, pp.  936–945. PMLR, 2021.
  2. Root mean square error (rmse) or mean absolute error (mae)?–arguments against avoiding rmse in the literature. Geoscientific model development, 7(3):1247–1250, 2014.
  3. Recent advances and clinical applications of deep learning in medical image analysis. Medical Image Analysis, pp.  102444, 2022.
  4. Recent advances and applications of deep learning methods in materials science. npj Computational Materials, 8(1):59, 2022.
  5. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
  6. Combining online and offline knowledge in uct. In Proceedings of the 24th international conference on Machine learning, pp.  273–280, 2007.
  7. Learning to forget: Continual prediction with lstm. Neural computation, 12(10):2451–2471, 2000.
  8. Graves, A. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013.
  9. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp.  1861–1870. PMLR, 2018.
  10. Shape-constrained multi-objective genetic programming for symbolic regression. Applied Soft Computing, 132:109855, 2023.
  11. Semantic linear genetic programming for symbolic regression. IEEE Transactions on Cybernetics, 2022.
  12. Discovery of implicit relationships from data using linear programming and mixed integer linear programming. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp.  558–561, 2022.
  13. Deep generative symbolic regression with monte-carlo-tree-search. arXiv preprint arXiv:2302.11223, 2023.
  14. Machine learning at the service of meta-heuristics for solving combinatorial optimization problems: A state-of-the-art. European Journal of Operational Research, 296(2):393–422, 2022.
  15. Integration of neural network-based symbolic regression in deep learning for scientific discovery. IEEE transactions on neural networks and learning systems, 32(9):4166–4177, 2020.
  16. Koza, J. R. et al. Evolution of subsumption using genetic programming. In Proceedings of the first European conference on artificial life, pp.  110–119. MIT Press Cambridge, MA, USA, 1992.
  17. Transformer-based model for symbolic regression via joint supervised learning. In The Eleventh International Conference on Learning Representations.
  18. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1-3):503–528, 1989.
  19. Extrapolation and learning equations. arXiv preprint arXiv:1610.02995, 2016.
  20. Sympy: symbolic computing in python. PeerJ Computer Science, 3:e103, 2017.
  21. Symbolic regression via neural-guided genetic programming population seeding. arXiv preprint arXiv:2111.00053, 2021.
  22. Opening the black box: the promise and limitations of explainable machine learning in cardiology. Canadian Journal of Cardiology, 38(2):204–213, 2022.
  23. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. arXiv preprint arXiv:1912.04871, 2019.
  24. Distilling free-form natural laws from experimental data. science, 324(5923):81–85, 2009.
  25. Learning to converse with noisy data: Generation with calibration. In IJCAI, volume 7, 2018.
  26. Transformer-based planning for symbolic regression. arXiv preprint arXiv:2303.06833, 2023.
  27. Mastering the game of go without human knowledge. nature, 550(7676):354–359, 2017.
  28. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140–1144, 2018.
  29. Symbolic physics learner: Discovering governing equations via monte carlo tree search. arXiv preprint arXiv:2205.13134, 2022.
  30. Ai feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16):eaay2631, 2020.
  31. Ai feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity. Advances in Neural Information Processing Systems, 33:4860–4871, 2020.
  32. Van Laarhoven, T. L2 regularization versus batch and weight normalization. arXiv preprint arXiv:1706.05350, 2017.
  33. Maximum entropy deep inverse reinforcement learning. arXiv preprint arXiv:1507.04888, 2015.
  34. Yeh, I.-C. Modeling of strength of high-performance concrete using artificial neural networks. Cement and Concrete research, 28(12):1797–1808, 1998.
  35. Application of deep learning algorithms in geotechnical engineering: a short critical review. Artificial Intelligence Review, pp.  1–41, 2021.
  36. Self-learning gene expression programming. IEEE Transactions on Evolutionary Computation, 20(1):65–80, 2015.
  37. Algorithm 778: L-bfgs-b: Fortran subroutines for large-scale bound-constrained optimization. ACM Transactions on mathematical software (TOMS), 23(4):550–560, 1997.
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets