Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Obeying the Order: Introducing Ordered Transfer Hyperparameter Optimisation (2306.16916v1)

Published 29 Jun 2023 in cs.LG

Abstract: We introduce ordered transfer hyperparameter optimisation (OTHPO), a version of transfer learning for hyperparameter optimisation (HPO) where the tasks follow a sequential order. Unlike for state-of-the-art transfer HPO, the assumption is that each task is most correlated to those immediately before it. This matches many deployed settings, where hyperparameters are retuned as more data is collected; for instance tuning a sequence of movie recommendation systems as more movies and ratings are added. We propose a formal definition, outline the differences to related problems and propose a basic OTHPO method that outperforms state-of-the-art transfer HPO. We empirically show the importance of taking order into account using ten benchmarks. The benchmarks are in the setting of gradually accumulating data, and span XGBoost, random forest, approximate k-nearest neighbor, elastic net, support vector machines and a separate real-world motivated optimisation problem. We open source the benchmarks to foster future research on ordered transfer HPO.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Transfer learning for Bayesian optimization: A survey. arXiv preprint arXiv:2302.05927.
  2. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems 33.
  3. Collecting empirical data about hyperparameters for data driven AutoML. Democratizing Machine Learning Contributions in AutoML and Fairness, page 93.
  4. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486.
  5. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 785–794, New York, NY, USA. ACM.
  6. HEBO: pushing the limits of sample-efficient hyperparameter optimisation. Journal of Artificial Intelligence Research, 74:1269–1349.
  7. UCI machine learning repository.
  8. Simopt: A testbed for simulation-optimization experiments. INFORMS Journal on Computing.
  9. Hyperparameter optimization. Automated machine learning: Methods, systems, challenges, pages 3–33.
  10. Practical transfer learning for Bayesian optimization. arXiv preprint arXiv:1802.02219.
  11. Initializing Bayesian hyperparameter optimization via meta-learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29.
  12. Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1):1–22.
  13. Google Vizier: A service for black-box optimization. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1487–1495.
  14. Hyperparameter transfer learning with adaptive complexity. In International Conference on Artificial Intelligence and Statistics, pages 1378–1386. PMLR.
  15. Non-stochastic best arm identification and hyperparameter optimization. In Artificial intelligence and statistics, pages 240–248. PMLR.
  16. Transfer learning for Bayesian HPO with end-to-end landmark meta-features. In Fifth Workshop on Meta-Learning at the Conference on Neural Information Processing Systems.
  17. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455.
  18. Fast Bayesian optimization of machine learning hyperparameters on large datasets. In Artificial intelligence and statistics, pages 528–536. PMLR.
  19. Model-based asynchronous hyperparameter and neural architecture search. arXiv preprint arXiv:2003.10865.
  20. Multidimensional curve classification using passing-through regions. Pattern Recognition Letters, 20(11-13):1103–1111.
  21. Kushmerick, N. (1999). Learning to remove internet advertisements. In Proceedings of the third annual conference on Autonomous Agents, pages 175–181.
  22. Hyperband: A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research, 18(1):6765–6816.
  23. Gesture unit segmentation using support vector machines: segmenting gestures from rest positions. In Proceedings of the 28th Annual ACM Symposium on Applied Computing, pages 46–52.
  24. Automatic termination for hyperparameter optimization. In Guyon, I., Lindauer, M., van der Schaar, M., Hutter, F., and Garnett, R., editors, Proceedings of the First International Conference on Automated Machine Learning, volume 188 of Proceedings of Machine Learning Research, pages 7/1–21. PMLR.
  25. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence, 42(4):824–836.
  26. Organizers of KDD Cup 2012 and Tencent Inc (2012). 2012 KDD Cup data set.
  27. Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning. Advances in Neural Information Processing Systems, 32.
  28. YAHPO Gym – an efficient multi-objective multi-fidelity benchmark for hyperparameter optimization. In International Conference on Automated Machine Learning, pages 3–1. PMLR.
  29. Quinlan, J. R. (1987). Simplifying decision trees. International journal of man-machine studies, 27(3):221–234.
  30. Syne Tune: A library for large scale hyperparameter tuning and reproducible research. In First Conference on Automated Machine Learning (Main Track).
  31. A quantile-based approach for hyperparameter transfer learning. In International conference on machine learning, pages 8438–8448. PMLR.
  32. Simonoff, J. S. (2003). Analyzing categorical data, volume 496. Springer.
  33. Practical Bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25.
  34. Bayesian optimization with robust Bayesian neural networks. In Proceedings of the 29th International Conference on Advances in Neural Information Processing Systems (NIPS’16).
  35. Hyperparameter transfer across developer adjustments. arXiv preprint arXiv:2010.13117.
  36. BORE: Bayesian optimization by density-ratio estimation. In International Conference on Machine Learning, pages 10289–10300. PMLR.
  37. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734.
  38. OpenML: networked science in machine learning. SIGKDD Explorations, 15(2):49–60.
  39. Few-shot Bayesian optimization with deep kernel surrogates. arXiv preprint arXiv:2101.07667.
  40. Learning data set similarities for hyperparameter optimization initializations. In Metasel@ pkdd/ecml, pages 15–26.
  41. Sequential model-free hyperparameter tuning. In 2015 IEEE international conference on data mining, pages 1033–1038. IEEE.
  42. ranger: A fast implementation of random forests for high dimensional data in C++ and R. Journal of Statistical Software, 77:1–17.
  43. A resource-efficient method for repeated hpo and nas problems. arXiv:2103.16111 [cs.LG].
  44. Lifelong Bayesian optimization. arXiv preprint arXiv:1905.12280.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now