Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
149 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Hybrid Parameter Search and Dynamic Model Selection for Mixed-Variable Bayesian Optimization (2206.01409v4)

Published 3 Jun 2022 in cs.LG, math.ST, stat.ML, and stat.TH

Abstract: This paper presents a new type of hybrid model for Bayesian optimization (BO) adept at managing mixed variables, encompassing both quantitative (continuous and integer) and qualitative (categorical) types. Our proposed new hybrid models (named hybridM) merge the Monte Carlo Tree Search structure (MCTS) for categorical variables with Gaussian Processes (GP) for continuous ones. hybridM leverages the upper confidence bound tree search (UCTS) for MCTS strategy, showcasing the tree architecture's integration into Bayesian optimization. Our innovations, including dynamic online kernel selection in the surrogate modeling phase and a unique UCTS search strategy, position our hybrid models as an advancement in mixed-variable surrogate models. Numerical experiments underscore the superiority of hybrid models, highlighting their potential in Bayesian optimization.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Automatic kernel selection for gaussian processes regression with approximate bayesian computation and sequential monte carlo. Frontiers in Built Environment 3, 52.
  2. Auer, P. (2002). Using Confidence Bounds for Exploitation-Exploration Trade-Offs. Journal of Machine Learning Research 3, 397–422.
  3. Autotuning in high-performance computing applications. Proceedings of the IEEE 106(11), 2068–2083.
  4. Algorithms for Hyper-Parameter Optimization. Advances in neural information processing systems 24.
  5. Optimization by Gradient Boosting. In Advances in Contemporary Statistics and Econometrics, pp. 23–44. Springer.
  6. Structural kernel search via bayesian optimization and symbolical optimal transport. Advances in Neural Information Processing Systems 35, 39047–39058.
  7. Black-box Combinatorial Optimization using Models with Integer-valued Minima. Annals of Mathematics and Artificial Intelligence 89(7), 639–653.
  8. Similarity Encoding for Learning with Dirty Categorical Variables. Machine Learning 107(8), 1477–1494.
  9. BART: Bayesian Additive Regression Trees. The Annals of Applied Statistics 4(1), 266–298.
  10. Kernel methods for deep learning. Advances in neural information processing systems 22.
  11. Additive Gaussian Process for Computer Models With Qualitative and Quantitative Factors. Technometrics 59(3), 283–292.
  12. Bayesian Optimization over Hybrid Spaces. arXiv preprint arXiv:2106.04682.
  13. Neural architecture search: A survey. The Journal of Machine Learning Research 20(1), 1997–2017.
  14. Scaling multi-armed bandit algorithms. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp.  1449–1459.
  15. Friedman, J. H. (1991). Multivariate Adaptive Regression Splines. The Annals of Statistics, 1–67.
  16. Dealing with Categorical and Integer-valued Variables in Bayesian Optimization with Gaussian Processes . Neurocomputing 380, 20–35.
  17. STRUMPACK.
  18. An Efficient Multicore Implementation of a Novel HSS-structured Multifrontal Solver using Randomized Sampling. SIAM Journal on Scientific Computing 38(5), S358–S384.
  19. Algorithmic Assurance: An Active Approach to Algorithmic Testing Using Bayesian Optimisation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp.  5470–5478.
  20. Gramacy, R. B. (2020). Surrogates: Gaussian Process Modeling, Design, and Optimization for the Applied Sciences.
  21. Hannan, E. J. and B. G. Quinn (1979). The Determination of the Order of an Autoregression. Journal of the Royal Statistical Society: Series B (Methodological) 41(2), 190–195.
  22. Harrison Jr, D. and D. L. Rubinfeld (1978). Hedonic Housing Prices and the Demand for Clean Air. Journal of environmental economics and management 5(1), 81–102.
  23. Scikit-Optimize/Scikit-Optimize. Zenodo.
  24. Recurrent kernel machines: Computing with infinite echo state networks. Neural Computation 24(1), 104–133.
  25. Sequential Model-Based Optimization for General Algorithm Configuration. In International Conference on Learning and Intelligent Optimization, pp.  507–523. Springer.
  26. Efficient global optimization of expensive black-box functions. Journal of Global optimization 13, 455–492.
  27. Continuous Surrogate-based Optimization Algorithms are Well-suited for Expensive Discrete Problems. In Benelux Conference on Artificial Intelligence, pp.  48–63. Springer.
  28. Bandit Based Monte-Carlo Planning. In D. Hutchison, T. Kanade, J. Kittler, J. M. Kleinberg, F. Mattern, J. C. Mitchell, M. Naor, O. Nierstrasz, C. Pandu Rangan, B. Steffen, M. Sudan, D. Terzopoulos, D. Tygar, M. Y. Vardi, G. Weikum, J. Fürnkranz, T. Scheffer, and M. Spiliopoulou (Eds.), Machine Learning: ECML 2006, Volume 4212, pp.  282–293. Berlin, Heidelberg: Springer Berlin Heidelberg.
  29. Efficient and robust bayesian selection of hyperparameters in dimension reduction for visualization. arXiv preprint arXiv:2306.00357.
  30. Non-smooth Bayesian Optimization in Tuning Problems. arXiv preprint arXiv:2109.07563.
  31. Luo, H. and M. T. Pratola (2022). Sharded Bayesian Additive Regression Trees. arXiv:2306.00361, 1–46.
  32. Optimism and Model Complexity Measure for Linear Models. In preparation.
  33. Bandits with adversarial scaling. In International Conference on Machine Learning, pp. 6511–6521. PMLR.
  34. Bayesian Optimization for Automated Model Selection. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Eds.), Advances in neural information processing systems, Volume 29. Curran Associates, Inc.
  35. Bayesian Optimization for Categorical and Category-Specific Continuous Inputs. arXiv:1911.12473 [cs, stat].
  36. Mixed Variable Bayesian Optimization with Frequency Modulated Kernels. arXiv preprint arXiv:2102.12792.
  37. Olson, R. S. and J. H. Moore (2019). TPOT: A Tree-Based Pipeline Optimization Tool for Automating Machine Learning. In F. Hutter, L. Kotthoff, and J. Vanschoren (Eds.), Automated Machine Learning, pp.  151–160. Cham: Springer International Publishing.
  38. Automated Machine Learning with Monte-Carlo Tree Search. arXiv:1906.00170 [cs, stat].
  39. Rasmussen, C. E. and C. K. I. Williams (2006). Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning. Cambridge, Mass: MIT Press.
  40. Bayesian Optimisation over Multiple Continuous and Categorical Inputs. arXiv:1906.08878 [cs, stat].
  41. Sethuraman, J. and R. C. Tiwari (1982). Convergence of Dirichlet Measures and the Interpretation of Their Parameter. In Statistical Decision Theory and Related Topics III, pp. 305–315. Elsevier.
  42. Taking the Human Out of the Loop: A Review of Bayesian Optimization. Proceedings of the IEEE 104(1), 148–175.
  43. Multitask and Transfer Learning for Autotuning Exascale Applications. arXiv:1908.05792 [cs, stat].
  44. GPTune User Guide.
  45. 3d-hst wfc3-selected photometric catalogs in the five candels/3d-hst fields: photometry, photometric redshifts, and stellar masses. The Astrophysical Journal Supplement Series 214(2), 24.
  46. Practical Bayesian Optimization of Machine Learning Algorithms. arXiv:1206.2944 [cs, stat].
  47. Virtual library of simulation experiments: Test functions and datasets. Retrieved June 2, 2022, from http://www.sfu.ca/~ssurjano.
  48. Raiders of the lost architecture: Kernels for bayesian optimization in conditional parameter spaces. arXiv preprint arXiv:1409.4011.
  49. Bayesian Inference in Monte-Carlo Tree Search. arXiv preprint arXiv:1203.3519.
  50. Bayesian Optimization for Auto-Tuning GPU kernels. arXiv:2111.14991 [cs, math].
  51. Ezgp: Easy-to-interpret gaussian process models for computer experiments with both quantitative and qualitative factors. SIAM/ASA Journal on Uncertainty Quantification 9(2), 333–353.
  52. Ye, J. (1998). On measuring and correcting the effects of data mining and model selection. Journal of the American Statistical Association 93(441), 120–131.
  53. Progressive sampling-based bayesian optimization for efficient and automatic machine learning model selection. Health information science and systems 5, 1–21.
  54. Bayesian optimization for materials design with mixed quantitative and qualitative variables. Scientific reports 10(1), 4924.
  55. A latent variable approach to Gaussian process modeling with qualitative and quantitative factors. Technometrics 62(3), 291–302.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now