On the Optimal Weighted $\ell_2$ Regularization in Overparameterized Linear Regression

Abstract: We consider the linear model $$\mathbf{y} = \mathbf{X} \mathbf{\beta}<em>\star + \mathbf{\epsilon}$$ with $\mathbf{X}\in \mathbb{R}^{n\times p}$ in the overparameterized regime $p&gt;n$. We estimate $\mathbf{\beta}</em>\star$ via generalized (weighted) ridge regression: $$\hat{\mathbf{\beta}}<em>\lambda = \left(\mathbf{X}^T\mathbf{X} + \lambda \mathbf{\Sigma}_w\right)^\dagger \mathbf{X}<sup>T\mathbf{y}$$, where $\mathbf{\Sigma}_w$ is the weighting matrix. Under a random design setting with general data covariance $\mathbf{\Sigma}_x$ and anisotropic prior on the true coefficients $$\mathbb{E}\mathbf{\beta}</em>\star\mathbf{\beta}<em>\star^T = \mathbf{\Sigma}</em>\beta$$, we provide an exact characterization of the prediction risk $$\mathbb{E}(y-\mathbf{x}<sup>T\hat{\mathbf{\beta}}_\lambda)<sup>2$$ in the proportional asymptotic limit $p/n\rightarrow \gamma \in (1,\infty)$. Our general setup leads to a number of interesting findings. We outline precise conditions that decide the sign of the optimal setting $\lambda_{\rm opt}$ for the ridge parameter $\lambda$ and confirm the implicit $\ell_2$ regularization effect of overparameterization, which theoretically justifies the surprising empirical observation that $\lambda_{\rm opt}$ can be negative in the overparameterized regime. We also characterize the double descent phenomenon for principal component regression (PCR) when both $\mathbf{X}$ and $\mathbf{\beta}<em>\star$ are anisotropic. Finally, we determine the optimal weighting matrix $\mathbf{\Sigma}_w$ for both the ridgeless ($\lambda\to 0$) and optimally regularized ($\lambda = \lambda</em>{\rm opt}$) case, and demonstrate the advantage of the weighted objective over standard ridge regression and PCR.