An Oblivious Stochastic Composite Optimization Algorithm for Eigenvalue Optimization Problems (2306.17470v1)
Abstract: In this work, we revisit the problem of solving large-scale semidefinite programs using randomized first-order methods and stochastic smoothing. We introduce two oblivious stochastic mirror descent algorithms based on a complementary composite setting. One algorithm is designed for non-smooth objectives, while an accelerated version is tailored for smooth objectives. Remarkably, both algorithms work without prior knowledge of the Lipschitz constant or smoothness of the objective function. For the non-smooth case with $\mathcal{M}-$bounded oracles, we prove a convergence rate of $ O( {\mathcal{M}}/{\sqrt{T}} ) $. For the $L$-smooth case with a feasible set bounded by $D$, we derive a convergence rate of $ O( {L2 D2}/{(T{2}\sqrt{T})} + {(D_02+\sigma2)}/{\sqrt{T}} )$, where $D_0$ is the starting distance to an optimal solution, and $ \sigma2$ is the stochastic oracle variance. These rates had only been obtained so far by either assuming prior knowledge of the Lipschitz constant or the starting distance to an optimal solution. We further show how to extend our framework to relative scale and demonstrate the efficiency and robustness of our methods on large scale semidefinite programs.
- A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM journal on imaging sciences, 2(1):183–202, 2009.
- Ashok Cutkosky. Anytime online-to-batch conversions, optimism, and acceleration. arXiv 1903.00974, 2019.
- Optimal algorithms for stochastic complementary composite minimization. arXiv 2211.01758, 2022.
- A stochastic smoothing algorithm for semidefinite programming. arXiv 1204.0665, 2014.
- Learning-rate-free learning by D-adaptation. arXiv preprint arXiv:2301.07733, 2023.
- Complementary composite minimization, small gradients in general norms, and applications. arXiv 2101.11041, 2023.
- Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res., 12(null):2121–2159, jul 2011.
- Adaptive gradient methods for constrained convex optimization and variational inequalities. arXiv 2007.08840, 2020.
- Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization i: A generic algorithmic framework. SIAM Journal on Optimization, 22(4):1469–1492, 2012.
- Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization, ii: Shrinking procedures and optimal algorithms. SIAM Journal on Optimization, 23(4):2061–2089, 2013.
- Parameter-free mirror descent. arXiv 2203.00444, 2022.
- A simpler approach to accelerated optimization: iterative averaging meets optimism. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4984–4993. PMLR, 13–18 Jul 2020.
- Deterministic and stochastic primal-dual subgradient algorithms for uniformly convex minimization. Stochastic Systems, 4(1):44 – 80, 2014.
- Guanghui Lan. An optimal method for stochastic composite optimization. Mathematical Programming, 133:1–33, 06 2012.
- Online adaptive methods, universality and acceleration. arXiv 1809.02864, 2018.
- Arkadi Nemirovski. Prox-method with rate of convergence o(1/t) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM Journal on Optimization, 15(1):229–251, 2004.
- Robust stochastic approximation approach to stochastic programming. Society for Industrial and Applied Mathematics, 19:1574–1609, 01 2009.
- Problem complexity and method efficiency in optimization. Wiley-Interscience series in discrete mathematics. Wiley, 1983.
- Yu Nesterov. Gradient methods for minimizing composite functions. Mathematical Programming, 2013.
- Yurii Nesterov. A method for solving the convex programming problem with convergence rate o(1/k2)𝑜1superscript𝑘2o(1/k^{2})italic_o ( 1 / italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ). Proceedings of the USSR Academy of Sciences, 269:543–547, 1983.
- Yurii Nesterov. How to make the gradients small. Optima. Mathematical Optimization Society Newsletter, 2012.
- Yurii Nesterov. Gradient methods for minimizing composite functions. Mathematical Programming, 140(1):125–161, 2013.
- Randomized minimization of eigenvalue functions. arXiv 2301.08352, 2023.
- Boris Polyak. Introduction to optimization, 07 2020.
- Goldfarb Donald Scheinberg Katya and Xi Bai. Fast first-order methods for composite convex optimization with backtracking. Foundations of Computational Mathematics, 2014.
- Scalable semidefinite programming. SIAM Journal on Mathematics of Data Science, 3(1):171–200, jan 2021.