Magnitude Square Coherence (MSC) Estimation via an ARMA Model Based on Analytic DCT and Group Delay

Abstract

A new method of magnitude square coherence (MSC) estimation by an auto-regressive moving average (ARMA) model based on analytic discrete cosine transform (ADCT) and group delay (GD) property is proposed. The estimation is achieved by modeling the Welch-MSC derived from ADCT and the ARMA model realized by GD. The ADCT provides twice frequency resolution and reduced variance compared to those of conventional DFT. The proposed MSC estimate is superior to that based on MSC estimation using DFT, in terms of normalized sum of the sample mean square error (NSSMSE), maximum sample root-mean-square error and bias. The proposed method for the two examples involving two stationary stochastic processes reduces the NSSMSE by 60 and 30% over those of ARMA-MSC based on DFT. Further, the Welch-MSC based on ADCT itself reduces the performance indices over that based on DFT (significant for example-2). The proposed method is also applied to signals with sinusoids whose frequencies are located at off the DFT grid bin and closely spaced. The minimum variance distortionless response method of MSC estimation known for its frequency resolution fails to detect and resolve closely spaced off bin frequencies, whereas the proposed ADCT-based Welch-MSC and its ARMA detect and resolve those frequencies. The DFT-based Welch-MSC and its ARMA version though detect off the bin sinusoids, but due to their poor frequency resolution cannot resolve the closely spaced sinusoids. The average variance per frequency bin of ADCT-ARMA and DFT-ARMA is 0.0061 and 0.0108 (i.e., 43.52% reduction by ADCT), respectively.