Unsupervised Learning Algorithms and Latent Variable Models: PCA/SVD, CCA/PLS, ICA, NMF, etc.

Abstract

Constrained matrix and tensor factorizations, also called penalized matrix/tensor decompositions play a key role in Latent Variable Models (LVM), Multilinear Blind Source Separation (MBSS), and (multiway) Generalized Component Analysis (GCA) and they are important unifying topics in signal processing and linear and multilinear algebra. This chapter introduces basic linear and multilinear models for matrix and tensor factorizations and decompositions. The “workhorse” of this chapter consists of constrained matrix decompositions and their extensions, including multilinear models which perform multiway matrix or tensor factorizations, with various constraints such as orthogonality, statistical independence, nonnegativity and/or sparsity. The constrained matrix and tensor decompositions are very attractive because they take into account spatial, temporal and/or spectral information and provide links among the various extracted factors or latent variables while providing often physical or physiological meanings and interpretations. In fact matrix/tensor decompositions are important techniques for blind source separation, dimensionality reduction, pattern recognition, object detection, classification, multiway clustering, sparse representation and coding and data fusion.