Optimal rates for first-order stochastic convex optimization under Tsybakov noise condition

Abstract: We focus on the problem of minimizing a convex function $f$ over a convex set $S$ given $T$ queries to a stochastic first order oracle. We argue that the complexity of convex minimization is only determined by the rate of growth of the function around its minimizer $x<sup>*_{f,S}$, as quantified by a Tsybakov-like noise condition. Specifically, we prove that if $f$ grows at least as fast as $|x-x<sup>*_{f,S}|<sup>\kappa$ around its minimum, for some $\kappa &gt; 1$, then the optimal rate of learning $f(x<sup>*_{f,S})$ is $\Theta(T<sup>{-\frac{\kappa}{2\kappa-2}})$. The classic rate $\Theta(1/\sqrt T)$ for convex functions and $\Theta(1/T)$ for strongly convex functions are special cases of our result for $\kappa \rightarrow \infty$ and $\kappa=2$, and even faster rates are attained for $\kappa &lt;2$. We also derive tight bounds for the complexity of learning $x_{f,S}<sup>*$, where the optimal rate is $\Theta(T<sup>{-\frac{1}{2\kappa-2}})$. Interestingly, these precise rates for convex optimization also characterize the complexity of active learning and our results further strengthen the connections between the two fields, both of which rely on feedback-driven queries.