Classification and Reconstruction of High-Dimensional Signals From Low-Dimensional Features in the Presence of Side Information

Francesco Renna,Lawrence Carin,Galen Reeves,Xin Yuan,Robert Calderbank,Jianbo Yang,Liming Wang,Miguel R D Rodrigues

doi:10.1109/tit.2016.2606646

Abstract

This paper offers a characterization of fundamental limits on the classification and reconstruction of high-dimensional signals from low-dimensional features, in the presence of side information. We consider a scenario where a decoder has access both to linear features of the signal of interest and to linear features of the side information signal; while the side information may be in a compressed form, the objective is recovery or classification of the primary signal, not the side information. The signal of interest and the side information are each assumed to have (distinct) latent discrete labels; conditioned on these two labels, the signal of interest and side information are drawn from a multivariate Gaussian distribution. With joint probabilities on the latent labels, the overall signal-(side information) representation is defined by a Gaussian mixture model. We then provide sharp sufficient and/or necessary conditions for these quantities to approach zero when the covariance matrices of the Gaussians are nearly low-rank. These conditions, which are reminiscent of the well-known Slepian-Wolf and Wyner-Ziv conditions, are a function of the number of linear features extracted from the signal of interest, the number of linear features extracted from the side information signal, and the geometry of these signals and their interplay. Moreover, on assuming that the signal of interest and the side information obey such an approximately low-rank model, we derive expansions of the reconstruction error as a function of the deviation from an exactly low-rank model; such expansions also allow identification of operational regimes where the impact of side information on signal reconstruction is most relevant. Our framework, which offers a principled mechanism to integrate side information in high-dimensional data problems, is also tested in the context of imaging applications.

Highlights

A SIGNIFICANT focus of recent research concerns approaches to represent and extract the salient information of a high-dimensional signal from low-dimensional signal features
We report a series of numerical results, both with synthetic and real data, that cast further light on the role of side information to aid signal classification or reconstruction
We have considered a model where the joint distribution of the signal of interest and the side information, conditioned on some underlying class labels is a multivariate Gaussian, which embodies the correlation between these signals

Summary

Introduction

A SIGNIFICANT focus of recent research concerns approaches to represent and extract the salient information of a high-dimensional signal from low-dimensional signal features. Methods such as feature extraction, supervised dimensionality reduction and unsupervised dimensionality reduction have been studied in various disciplines [1]–[4]. Linear dimensionality reduction methods based on the second-order statistics of the source have been developed, such as linear discriminant analysis (LDA) [1] or principal component analysis (PCA) [1]. Linear dimensionality reduction methods based on higher-order statistics of the data have been developed [5]–[17].

Objectives

Results

Conclusion