Enhanced RMT estimator for signal number estimation in the presence of colored noise (2211.12942v2)

Abstract: The subspace-based techniques are widely utilized to estimate the parameters of sums of complex sinusoids corrupted by noise, and they need accurate estimation of the signal subspace dimension. The classic RMT estimator for model order estimation based on random matrix theory (RMT) assumes that the noise is white Gaussian, and performs poorly in the presence of colored noise with unknown covariance matrix. In order to deal with this problem, this paper proposes a novel algorithm to estimate the number of signals for the case of colored noise with unknown covariance matrix based on the analysis of the behavior of information theoretic criteria utilized in model order selection. Firstly, a first criterion is defined as the ratio of the current eigenvalue and the mean of the next ones, and its properties is analyzed with respect to the over-modeling and under-modeling. Secondly, a second criterion is designed as the ratio of the current value and the next value of the first criterion, and its properties is analyzed with respect to the over-modeling and under-modeling. Then, a novel enhanced RMT estimator is proposed for signal number estimation by analyzing the detection properties among the signal number estimates obtained by these two criteria and the RMT estimator to determine which eigenvalue is arising from a signal. Finally, simulation results are presented to illustrate that the proposed enhanced RMT estimator has better estimation performance and works better in the presence of colored noise than the existing methods.