Invariance and the Bayesian approach to generalized coherence tests

Abstract

This paper considers the problem of testing for mutual independence of multiple sets of complex Gaussian vectors. This problem has classical roots in statistics and has been of recent interest in the signal processing literature in connection with multi-channel signal detection. The probability distribution of the maximal invariants under the action of a subgroup of the full invariance group of the problem is derived for both hypotheses. It is shown that for parameter space, the maximal invariants under the action of this subgroup form a compact space on which proper non-informative prior distributions can be constructed. Bayesian likelihood ratios for the maximal invariants are derived for various proper prior distributions. Previously, Bayesian likelihood ratios associated with non-informative prior distributions for this problem could only be constructed through considerably less satisfactory limiting techniques.