Generalized Independent Component Analysis Over Finite Alphabets

Abstract

Independent component analysis (ICA) is a statistical method for transforming an observable multi-dimensional random vector into components that are as statistically independent as possible from each other. Usually, the ICA framework assumes a model according to which the observations are generated (such as a linear transformation with additive noise). ICA over finite fields is a special case of ICA in which both the observations and the independent components are over a finite alphabet. In this paper, we consider a generalization of this framework in which an observation vector is decomposed to its independent components (as much as possible) with no prior assumption on the way it was generated. This generalization is also known as Barlow's minimal redundancy representation problem and is considered an open problem. We propose several theorems and show that this hard problem can be accurately solved with a branch and bound search tree algorithm, or tightly approximated with a series of linear problems. Our contribution provides the first efficient set of solutions to Barlow's problem. The minimal redundancy representation (also known as factorial code) has many applications, mainly in the fields of neural networks and deep learning. The binary ICA is also shown to have applications in several domains, including medical diagnosis, multi-cluster assignment, network tomography, and internet resource management. In this paper, we show that this formulation further applies to multiple disciplines in source coding, such as predictive coding, distributed source coding, and coding of large alphabet sources.