Roundoff error in multidimensional generalized discrete transforms

Abstract

The analysis of rounding error in the one-dimensional fast Fourier transform (FFT) is extended to a class of generalized orthogonal transforms [1] with a common fast algorithm similar to the FFT algorithm. This class includes the BInary FOurier REpresentation (BIFORE) transform (BT) [2], the complex BT (CBT) [3], and the discrete Fourier transform (DFT). Expressions for the mean square error (MSE) in the two-dimensional BT, CBT, and FFT are derived. In the case of white input data, the mean square error-to-signal ratio is derived for the multidimensional generalized transforms. The error-to-signal ratio for the one-dimensional FFT derived by Kaneko and Liu is modified with improvement. Some comparisons among BIFORE, DFT, and Haar transforms are also included. The theoretical results for the two-dimensional FFT and BIFORE have been verified experimentally. The experimental results are in good agreement with the theoretical results for lower order sequences, but deviate as the order increases due to the actual manner of rounding in the digital computer.