Optimum linear detector at small and large noise power for a general binary composite hypothesis testing problem

Abstract

The problem of finding the optimum linear detector for a general binary composite hypothesis testing problem in additive white Gaussian noise is addressed in the paper. The signal set consists of a limited number of known signals with known a priori probabilities on each binary hypothesis. The a priori probability for each hypothesis is also assumed known. The linear detector to this binary decision problem consists of a linear filter and a comparison with a threshold. In the paper we show how to find the optimum filter and threshold for this linear detector, for the limiting cases of infinitely large and vanishingly small noise power, respectively. An analytical solution is given for the optimum solution in the case of infinitely large noise power and a recursive algorithm, giving the optimum solution in the case of vanishingly small noise power, is presented. These solutions are valid without any restrictions on signals and a priori probabilities.