Double criterion-based estimator for signal number estimation for the colored noise with unknown covariance matrix

The subspace-based techniques are widely utilized to estimate the parameters of sums of complex sinusoids corrupted by noise, and the zoom ESPRIT algorithm utilizes the zoom technique to apply the ESPRIT to a narrow frequency band to improve the accuracy of frequency estimation. However, the Gaussian noise becomes non-Gaussian in the zoomed baseband after being filtered by a low-pass filter, and thus has an unknown covariance matrix. However, most exiting algorithms for model order estimation performs poorly for the case of colored noise with unknown covariance matrix. In order to accurately estimate the dimension of the signal subspace for the zoom ESPRIT algorithm, this paper proposes a novel strategy to estimate the number of signals for the case of colored noise with unknown covariance matrix. The proposed strategy is based on the analysis of the behavior of information theoretic criteria utilized in model order selection. Firstly, a first criterion is defined as the ratio of the current eigenvalue and the mean of the next ones, and its properties is analyzed with respect to the over-modeling and under-modeling. Secondly, a novel second criterion is designed as the ratio of the current value and the next value of the first criterion, and its properties is also analyzed with respect to the over-modeling and under-modeling. Then, a novel signal number estimation method is proposed by combining the second criterion with the first criterion to check whether the eigenvalue being tested is arising from a signal or from noise. The resulted signal number estimation method is called as the double criterion-based estimator as it utilizes two criteria to separate the signal eigenvalues from the noise eigenvalues. Finally, simulation results are presented to illustrate the performance of the proposed double criterion-based estimator and compare it with the existing methods.