An Optimal Algorithm for Strongly Convex Min-min Optimization

In this paper we study the smooth strongly convex minimization problem $\min{x}\miny f(x,y)$. The existing optimal first-order methods require $\mathcal{O}(\sqrt{\max{\kappax,\kappay}} \log 1/\epsilon)$ of computations of both $\nablax f(x,y)$ and $\nablay f(x,y)$, where $\kappax$ and $\kappay$ are condition numbers with respect to variable blocks $x$ and $y$. We propose a new algorithm that only requires $\mathcal{O}(\sqrt{\kappax} \log 1/\epsilon)$ of computations of $\nablax f(x,y)$ and $\mathcal{O}(\sqrt{\kappay} \log 1/\epsilon)$ computations of $\nablay f(x,y)$. In some applications $\kappax \gg \kappay$, and computation of $\nablay f(x,y)$ is significantly cheaper than computation of $\nablax f(x,y)$. In this case, our algorithm substantially outperforms the existing state-of-the-art methods.