Analytical Performance of Rank-One Signal Detection in Subspace Interference Plus Gaussian Noise

Abstract

The problem of detecting rank-one signals is considered in the presence of subspace interference and Gaussian noise with unknown covariance matrix. A generalized likelihood ratio test (GLRT) detector has already been designed for this detection problem. In this correspondence, we provide analytical performance analysis for the GLRT detector in the mismatched case where the nominal target steering vector is not aligned with the true one. All theoretical results are confirmed using Monte Carlo techniques. It is revealed that the detector bears a constant false alarm rate property against the noise covariance matrix. The expressions derived for the probabilities of false alarm and detection can greatly facilitate the threshold setting and performance evaluations. Numerical examples indicate that the GLRT is sensitive to steering vector mismatches.