Quasi-Newton Methods for Saddle Point Problems and Beyond

Abstract: This paper studies quasi-Newton methods for solving strongly-convex-strongly-concave saddle point problems (SPP). We propose greedy and random Broyden family updates for SPP, which have explicit local superlinear convergence rate of $${\mathcal O}\big(\big(1-\frac{1}{n\kappa<sup>2}\big)<sup>{k(k-1)/2}\big)$$, where $n$ is dimensions of the problem, $\kappa$ is the condition number and $k$ is the number of iterations. The design and analysis of proposed algorithm are based on estimating the square of indefinite Hessian matrix, which is different from classical quasi-Newton methods in convex optimization. We also present two specific Broyden family algorithms with BFGS-type and SR1-type updates, which enjoy the faster local convergence rate of $$\mathcal O\big(\big(1-\frac{1}{n}\big)<sup>{k(k-1)/2}\big)$$. Additionally, we extend our algorithms to solve general nonlinear equations and prove it enjoys the similar convergence rate.