Spectral Statistics of the Sample Covariance Matrix for High Dimensional Linear Gaussians

Performance of ordinary least squares(OLS) method for the \emph{estimation of high dimensional stable state transition matrix} $A$(i.e., spectral radius $\rho(A)<1$) from a single noisy observed trajectory of the linear time invariant(LTI)\footnote{Linear Gaussian (LG) in Markov chain literature} system $X{-}:(x0,x1, \ldots,x{N-1})$ satisfying \begin{equation} x{t+1}=Ax{t}+w{t}, \hspace{10pt} \text{ where } w{t} \thicksim N(0,I{n}), \end{equation} heavily rely on negative moments of the sample covariance matrix: $(X{-}X{-}^{*})=\sum{i=0}^{{N-1}x{i}x{i}{*}$} and singular values of $EX{-}^{*}$, where $E$ is a rectangular Gaussian ensemble $E=[w0, \ldots, w{N-1}]$. Negative moments requires sharp estimates on all the eigenvalues $\lambda{1}\big(X{-}X{-}^{*}\big) \geq \ldots \geq \lambda{n}\big(X{-}X{-}^{*}\big) \geq 0$. Leveraging upon recent results on spectral theorem for non-Hermitian operators in \cite{naeem2023spectral}, along with concentration of measure phenomenon and perturbation theory(Gershgorins' and Cauchys' interlacing theorem) we show that only when $A=A^{*}$, typical order of $\lambda{j}\big(X{-}X{-}^{*}\big) \in \big[N-n\sqrt{N}, N+n\sqrt{N}\big]$ for all $j \in [n]$. However, in \emph{high dimensions} when $A$ has only one distinct eigenvalue $\lambda$ with geometric multiplicity of one, then as soon as eigenvalue leaves \emph{complex half unit disc}, largest eigenvalue suffers from curse of dimensionality: $\lambda{1}\big(X{-}X{-}^{{*}\big)=\Omega\big(} \lfloor\frac{N}{n}\rfloor e^{\alpha{\lambda}n} \big)$, while smallest eigenvalue $\lambda{n}\big(X{-}X_{-}^{*}\big) \in (0, N+\sqrt{N}]$. Consequently, OLS estimator incurs a \emph{phase transition} and becomes \emph{transient: increasing iteration only worsens estimation error}, all of this happening when the dynamics are generated from stable systems.