Low-dimensional Linear Subspace Research Articles

Pursuit of a discriminative representation for multiple subspaces via sequential games

We consider the problem of learning discriminative representations for data in a high-dimensional space with distribution supported on or around multiple low-dimensional linear subspaces. That is, we wish to compute a linear injective map of the data such that the features lie on multiple orthogonal subspaces. Instead of solving this learning problem using multiple instances of principal component analysis (PCA), we cast it as a sequential game using the closed-loop transcription (CTRL) framework recently proposed for learning discriminative and generative representations for general low-dimensional submanifolds. We prove that the equilibrium solutions to the game indeed give the correct representations. Our approach unifies classical methods of learning subspaces with modern machine learning practice, by showing that subspace learning problems may be provably solved using the modern toolkit of representation learning. In addition, our work provides the first theoretical justification for the CTRL framework in the important special case of linear subspaces. We support our theoretical findings with compelling empirical evidence. We also generalize the sequential game formulation to more general representation learning problems. Our code is publicly available on GitHub.

A unified performance analysis of likelihood-informed subspace methods

The likelihood-informed subspace (LIS) method offers a viable route to reducing the dimensionality of high-dimensional probability distributions arising in Bayesian inference. LIS identifies an intrinsic low-dimensional linear subspace where the target distribution differs the most from some tractable reference distribution. Such a subspace can be identified using the leading eigenvectors of a Gram matrix of the gradient of the log-likelihood function. Then, the original high-dimensional target distribution is approximated through various forms of marginalization of the likelihood function, in which the approximated likelihood only has support on the intrinsic low-dimensional subspace. This approximation enables the design of inference algorithms that can scale sub-linearly with the apparent dimensionality of the problem. Intuitively, the accuracy of the approximation, and hence the performance of the inference algorithms, are influenced by three factors—the dimension truncation error in identifying the subspace, Monte Carlo error in estimating the Gram matrices, and Monte Carlo error in constructing marginalizations. This work establishes a unified framework to analyze each of these three factors and their interplay. Under mild technical assumptions, we establish error bounds for a range of existing dimension reduction techniques based on the principle of LIS. Our error bounds also provide useful insights into the accuracy of these methods. In addition, we analyze the integration of LIS with sampling methods such as Markov Chain Monte Carlo (MCMC) and sequential Monte Carlo (SMC). We also demonstrate the applicability of our analysis on a linear inverse problem with Gaussian prior, which shows that all the estimates can be dimension-independent if the prior covariance is a trace-class operator. Finally, we demonstrate various aspects of our theoretical claims on two nonlinear inverse problems.

Improved Balanced Steady-State Free Precession Based MR Fingerprinting with Deep Autoencoders.

Magnetic Resonance (MR) Fingerprinting is an emerging transient-state imaging paradigm, which enables the quantization of multiple MR tissue parameters in a single experiment. Balanced steady-state free precession (bSSFP)-based MR Fingerprinting has excellent signal-to-noise characteristics and also allows for acquiring both tissue parameter maps and field inhomogeneity maps. However, field inhomogeneity often results in complex magnetization evolutions in bSSFP-based MR Fingerprinting, which creates significant challenges in image reconstruction. In this paper, we introduce a new method to address the image reconstruction problem. The proposed method incorporates a low-dimensional nonlinear manifold learned from an ensemble of magnetization evolutions using a deep autoencoder. It provides much better representation power than a low-dimensional linear subspace in capturing complex magnetization evolutions. We formulate the image reconstruction problem with this nonlinear model and solve the resulting optimization problem using an algorithm based on variable splitting and the alternating direction method of multipliers. We evaluate the performance of the proposed method using numerical experiments and demonstrate that it significantly outperforms the state-of-art method using a linear subspace model.

Probabilistic methods for approximate archetypal analysis

Abstract Archetypal analysis (AA) is an unsupervised learning method for exploratory data analysis. One major challenge that limits the applicability of AA in practice is the inherent computational complexity of the existing algorithms. In this paper, we provide a novel approximation approach to partially address this issue. Utilizing probabilistic ideas from high-dimensional geometry, we introduce two preprocessing techniques to reduce the dimension and representation cardinality of the data, respectively. We prove that provided data are approximately embedded in a low-dimensional linear subspace and the convex hull of the corresponding representations is well approximated by a polytope with a few vertices, our method can effectively reduce the scaling of AA. Moreover, the solution of the reduced problem is near-optimal in terms of prediction errors. Our approach can be combined with other acceleration techniques to further mitigate the intrinsic complexity of AA. We demonstrate the usefulness of our results by applying our method to summarize several moderately large-scale datasets.

A fast and accurate physics-informed neural network reduced order model with shallow masked autoencoder

Traditional linear subspace reduced order models (LS-ROMs) are able to accelerate physical simulations in which the intrinsic solution space falls into a subspace with a small dimension, i.e., the solution space has a small Kolmogorov n-width. However, for physical phenomena not of this type, e.g., any advection-dominated flow phenomena such as in traffic flow, atmospheric flows, and air flow over vehicles, a low-dimensional linear subspace poorly approximates the solution. To address cases such as these, we have developed a fast and accurate physics-informed neural network ROM, namely nonlinear manifold ROM (NM-ROM), which can better approximate high-fidelity model solutions with a smaller latent space dimension than the LS-ROMs. Our method takes advantage of the existing numerical methods that are used to solve the corresponding full order models. The efficiency is achieved by developing a hyper-reduction technique in the context of the NM-ROM. Numerical results show that neural networks can learn a more efficient latent space representation on advection-dominated data from 1D and 2D Burgers' equations. A speedup of up to 2.6 for 1D Burgers' and a speedup of 11.7 for 2D Burgers' equations are achieved with an appropriate treatment of the nonlinear terms through a hyper-reduction technique. Finally, a posteriori error bounds for the NM-ROMs are derived that take account of the hyper-reduced operators.

A relative multifractal analysis: box-dimensions, densities, and projections

We are interested in the behavior of relative multifractal box-dimensions, density results and multifractal dimensions under projections onto a lower-dimensional linear subspace. The results in this paper establish the connections with various multifractal and fractal dimensions and their projections, and generalize many known results about Hausdorff and packing measures and dimensions of a subset of ℝ n .

Deep self-representative subspace clustering network

Deep learning based subspace clustering networks have been a significant technique for motion segmentation, unsupervised image segmentation, image representation and compression, and face clustering by separating the high-dimensional data points into their representative low-dimensional linear subspaces. Effective feature selection is critical to remove redundant samples and select the representative feature subset from high-dimensional data space; hence deriving the number of subspaces, their dimensions, data segmentation, and a basis for each subspace. The effective self-representative feature selection and emphasis by scaling the feature map in the learned embedded space is required for deep learning based subspace clustering to reduce the number of parameters and dimension of the self-representative layer. In this paper, we propose a self-representative feature extraction deep neural network for unsupervised subspace clustering to improve representativeness and clustering ability. The extensive relevant results on various data demonstrate that deep subspace clustering employing self-representative features from high-dimensional data can effectively reduce the dimension of the self-representative layer while improving performance.

Local low-rank approach to nonlinear matrix completion

This paper deals with a problem of matrix completion in which each column vector of the matrix belongs to a low-dimensional differentiable manifold (LDDM), with the target matrix being high or full rank. To solve this problem, algorithms based on polynomial mapping and matrix-rank minimization (MRM) have been proposed; such methods assume that each column vector of the target matrix is generated as a vector in a low-dimensional linear subspace (LDLS) and mapped to a pth order polynomial and that the rank of a matrix whose column vectors are dth monomial features of target column vectors is deficient. However, a large number of columns and observed values are needed to strictly solve the MRM problem using this method when p is large; therefore, this paper proposes a new method for obtaining the solution by minimizing the rank of the submatrix without transforming the target matrix, so as to obtain high estimation accuracy even when the number of columns is small. This method is based on the assumption that an LDDM can be approximated locally as an LDLS to achieve high completion accuracy without transforming the target matrix. Numerical examples show that the proposed method has a higher accuracy than other low-rank approaches.

Matched Manifold Detection for Group-Invariant Registration and Classification of Images

Consider the set of possible observations turned out by geometric and radiometric transformations of an object. This set is generally a manifold in the ambient space of observations. It has been shown [1] that in those cases where the geometric deformations are affine and the radiometric deformations are monotonic, the radiometry invariant universal manifold embedding (RIUME) provides a mapping from the orbit of deformed observations to a single low dimensional linear subspace of Euclidean space. This linear subspace is invariant to the geometric and radiometric transformations and hence is a representative of the orbit. It thus naturally serves as an invariant statistic for solving problems of joint transformation estimation and detection or classification. In the unsupervised detection problem, subspaces evaluated from two observations are tested for the similarity of the observed object and their relative transformation is estimated from the RIUME matrix representation. In the classification set-up the RIUME subspace extracted from an experimental observation is tested against a set of subspaces representing the different object manifolds, in search for the nearest class. We show how to extract a set of mutually orthogonal subspaces, where each subspace represents a different object manifold. In the presence of observation noise, the observations do not lie strictly on the manifold and the resulting RIUME subspaces are noisy. We derive a method for estimating the mean subspace representation of a manifold of deformed observations. To optimize the performance of the matched manifold detector in the presence of observation noise, an analytic solution for choosing the RIUME nonlinear operators is derived, achieving the effect of simultaneous denoising of the object manifolds. The invariant representation of the object is the basis of a matched manifold detection and tracking framework for objects that undergo complex geometric and radiometric deformations. The experimental results on natural scenes demonstrate the generality and applicability of the RIUME framework for classification, detection, and dense registration.

Subspace clustering using ensembles of K-subspaces

Abstract Subspace clustering is the unsupervised grouping of points lying near a union of low-dimensional linear subspaces. Algorithms based directly on geometric properties of such data tend to either provide poor empirical performance, lack theoretical guarantees or depend heavily on their initialization. We present a novel geometric approach to the subspace clustering problem that leverages ensembles of the $K$-subspace (KSS) algorithm via the evidence accumulation clustering framework. Our algorithm, referred to as ensemble $K$-subspaces (EKSSs), forms a co-association matrix whose $(i,j)$th entry is the number of times points $i$ and $j$ are clustered together by several runs of KSS with random initializations. We prove general recovery guarantees for any algorithm that forms an affinity matrix with entries close to a monotonic transformation of pairwise absolute inner products. We then show that a specific instance of EKSS results in an affinity matrix with entries of this form, and hence our proposed algorithm can provably recover subspaces under similar conditions to state-of-the-art algorithms. The finding is, to the best of our knowledge, the first recovery guarantee for evidence accumulation clustering and for KSS variants. We show on synthetic data that our method performs well in the traditionally challenging settings of subspaces with large intersection, subspaces with small principal angles and noisy data. Finally, we evaluate our algorithm on six common benchmark datasets and show that unlike existing methods, EKSS achieves excellent empirical performance when there are both a small and large number of points per subspace.

Cross-domain transfer person re-identification via topology properties preserved local fisher discriminant analysis

Person re-identification (Re-ID) systems aim to identify a person appeared in non-overlapping cameras. However, a sufficient amount of pairwise cross-camera-view person images are often not available in a new scenario. Transfer learning can assist the new Re-ID system through leveraging knowledge from other related scenarios. Since the images from existed scenarios may not be the exact representative samples, how to learn a robust transfer Re-ID model with limited labeled person images is still a challenge so far. To solve this problem, a novel cross-domain transfer person Re-ID via topology properties preserved local Fisher discriminant analysis (TPPLFDA) method is proposed in this paper. Making an assumption that all person images in the new and related scenarios share common manifold, TPPLFDA projects all cross-domain images into a low dimensional linear subspace, while preserves the topology properties according to the geodesic distances on manifold. Then, multiple cross-domain datasets as source domains are considered and kernel TPPLFDA for multi-source domain transfer is proposed, so that TPPLFDA can handle more complex cross-domain transfer Re-ID tasks. Extensive experiments on several transfer Re-ID datasets show that TPPLFDA is effective.

On the effect of projections on the Billingsley dimensions

We are interested in the behavior of Billingsley dimensions under projections onto a lower dimensional linear subspace. The results in this paper establish the connections with various dimensions of subsets [Formula: see text] of [Formula: see text] and their projections, and generalize many known results about Hausdorff and packing dimensions of projections of [Formula: see text]. In particular, we improve, through these results, one of the main theorems of Selmi et al. in [Multifractal variation for projections of measures, Chaos Solitons Fractals 91 (2016) 414–420] and treat an unsolved case in their work.

Hyper-Reduction Over Nonlinear Manifolds for Large Nonlinear Mechanical Systems

Common trends in model reduction of large nonlinear finite element (FE)-discretized systems involve Galerkin projection of the governing equations onto a low-dimensional linear subspace. Though this reduces the number of unknowns in the system, the computational cost for obtaining the reduced solution could still be high due to the prohibitive computational costs involved in the evaluation of nonlinear terms. Hyper-reduction methods are then used for fast approximation of these nonlinear terms. In the finite element context, the energy conserving sampling and weighing (ECSW) method has emerged as an effective tool for hyper-reduction of Galerkin-projection-based reduced-order models (ROMs). More recent trends in model reduction involve the use of nonlinear manifolds, which involves projection onto the tangent space of the manifold. While there are many methods to identify such nonlinear manifolds, hyper-reduction techniques to accelerate computation in such ROMs are rare. In this work, we propose an extension to ECSW to allow for hyper-reduction using nonlinear mappings, while retaining its desirable stability and structure-preserving properties. As a proof of concept, the proposed hyper-reduction technique is demonstrated over models of a flat plate and a realistic wing structure, whose dynamics have been shown to evolve over a nonlinear (quadratic) manifold. An online speed-up of over one thousand times relative to the full system has been obtained for the wing structure using the proposed method, which is higher than its linear counterpart using the ECSW.

A Theoretical Analysis of Noisy Sparse Subspace Clustering on Dimensionality-Reduced Data

Subspace clustering is the problem of partitioning unlabeled data points into a number of clusters so that data points within one cluster lie approximately on a low-dimensional linear subspace. In many practical scenarios, the dimensionality of data points to be clustered is compressed due to the constraints of measurement, computation, or privacy. In this paper, we study the theoretical properties of a popular subspace clustering algorithm named sparse subspace clustering (SSC) and establish formal success conditions of SSC on dimensionality-reduced data. Our analysis applies to the most general fully deterministic model, where both underlying subspaces and data points within each subspace are deterministically positioned, and also a wide range of dimensionality reduction techniques (e.g., Gaussian random projection, uniform subsampling, and sketching) that fall into a subspace embedding framework. Finally, we apply our analysis to a differentially private SSC algorithm and established both privacy and utility guarantees of the proposed method.

A Novel Outlier Immune Multipath Fingerprinting Model for Indoor Single-Site Localization

Multipath fingerprinting is a promising indoor location technique, which contains abundant position features of the received array signals. Effectively representing the position information is one critical issue in fingerprinting localization. Meanwhile, positional accuracy is prone to reduction caused by abnormal measurement readings, which are referred to as outliers, and it has received a little attention in the existing literature. A multipath fingerprinting model for indoor single-site localization is proposed. In this model, the location fingerprint is composed of the spatial-temporal covariance matrix of the multipath signals received by the base station antenna array. The low-dimensional linear subspace of the location fingerprinting is introduced as feature descriptors. Based on the fact that the Grassmann manifold maintains the orthogonality of the linear subspace, the Binet-Cauchy kernel is employed to map the multipath fingerprinting to a higher dimensional reproducing kernel Hilbert space. The Euclidean distance of the nearest point between multipath fingerprinting affine hulls is adopted to represent the similarity of the position. Moreover, an augmented Lagrangian and alternating direction solution is given to remove the influence of outliers. We extensively evaluated the proposed method with the indoor multi-scenario benchmark data set. All the results demonstrate that the location accuracy of the proposed positioning model outperforms the existing method in an indoor environment. As the proportion of outliers increases, the positional accuracy loss of the proposed model is negligible.

Generalizing CoSaMP to signals from a union of low dimensional linear subspaces

The idea that signals reside in a union of low dimensional subspaces subsumes many low dimensional models that have been used extensively in the recent decade in many fields and applications. Until recently, the vast majority of works have studied each one of these models on its own. However, a recent approach suggests providing general theory for low dimensional models using their Gaussian mean width, which serves as a measure for the intrinsic low dimensionality of the data. In this work we use this novel approach to study a generalized version of the popular compressive sampling matching pursuit (CoSaMP) algorithm, and to provide general recovery guarantees for signals from a union of low dimensional linear subspaces, under the assumption that the measurement matrix is Gaussian. We discuss the implications of our results for specific models, and use the generalized algorithm as an inspiration for a new greedy method for signal reconstruction in a combined sparse-synthesis and cosparse-analysis model. We perform experiments that demonstrate the usefulness of the proposed strategy.

Learning binary hash codes for finger vein image retrieval

High dimensional real-valued features have been shown to be effective for finger vein identification, but result in high computational cost especially in a large-scale finger vein database. Therefore compact binary codes are generally used to obtain fast query speed as well as reduce storage requirements. However, only a few strategies are available for finger vein retrieval in Hamming space, and existing binary representation of finger veins can become unstable and undiscriminating due to finger vein variations induced by translation, rotation and illumination. In this paper, we propose a binary hash codes learning algorithm to map finger vein images in the original feature space to Hamming space. First, to obtain the discriminative finger vein image features, a novel finger vein image representation method called Nonlinearly Subspace Coding (NSC) is proposed. The codebook is a union of low-dimensional linear subspaces instead of visual words. For a given local finger vein texton, its top-k nearest subspaces are found and the texton is nonlinearly mapped into these subspaces. Then, a finger vein Binary Hash Codes (BHC) learning method is proposed by jointly considering the discriminability and the stability of the binary code. Experimental results on a two-session public finger vein database and a large fused finger vein database demonstrate the effectiveness and efficiency of our binary hash coding learning algorithm for large-scale finger vein retrieval.

A General Framework for Understanding Compressed Subspace Clustering Algorithms

Subspace clustering (SC) refers to the problem of clustering unlabeled high-dimensional data into a union of low-dimensional linear subspaces. In many practical scenarios, one may have access to only the compressed data due to constraints of measurement or computation. In this paper, based on the recently proposed restricted isometric property of Gaussian random projection for low-dimensional subspaces, we propose a general framework for analyzing the performance of various subspace clustering algorithms when applied to the compressed data. Our framework captures the connection between the problems of compressed subspace clustering (CSC) and noisy subspace clustering. With the existing study on noisy SC, our framework makes it possible to easily extend the results in noisy SC to CSC. In this paper, we select the most commonly used subspace clustering algorithms, i.e. , sparse SC (SSC), SSC-orthogonal matching pursuit (SSC-OMP), and thresholding based SC (TSC), as representatives and analyze their performance using the proposed framework. Our results are consistent with related work obtained by previous researchers, where our methodology is much more direct and has more universal implications. Finally, the practicability and efficiency of CSC are verified by numerical experiments.

Fast online low-rank tensor subspace tracking by CP decomposition using recursive least squares from incomplete observations

This paper considers the problem of online tensor subspace tracking of a partially observed high-dimensional data stream corrupted by noise, where we assume that the data lie in a low-dimensional linear subspace. This problem is cast as an online low-rank tensor completion problem. We propose a novel online tensor subspace tracking algorithm based on the CANDECOMP/PARAFAC (CP) decomposition, dubbed OnLine Low-rank Subspace tracking by TEnsor CP Decomposition (OLSTEC). The proposed algorithm specifically addresses the case in which data of interest are fed into the algorithm over time infinitely, and their subspace are dynamically time-varying. To this end, we build up our proposed algorithm exploiting the recursive least squares (RLS), which is a second-order gradient algorithm. Numerical evaluations on synthetic datasets and real-world datasets such as communication network traffic, environmental data, and surveillance videos, show that the proposed OLSTEC algorithm outperforms state-of-the-art online algorithms in terms of the convergence rate per iteration.

Directional Correlation Filter Bank for Robust Head Pose Estimation and Face Recognition

During the past few decades, face recognition has been an active research area in pattern recognition and computer vision due to its wide range of applications. However, one of the most challenging problems encountered by face recognition is the difficulty of handling large head pose variations. Therefore, the efficient and effective head pose estimation is a critical step of face recognition. In this paper, a novel feature extraction framework, called Directional Correlation Filter Bank (DCFB), is presented for head pose estimation. Specifically, in the proposed framework, the 1-Dimensional Optimal Tradeoff Filters (1D-OTF) corresponding to different head poses are simultaneously and jointly designed in the low-dimensional linear subspace. Different from the traditional methods that heavily rely on the precise localization of the key facial feature points, our proposed framework exploits the frequency domain of the face images, which effectively captures the high-order statistics of faces. As a result, the obtained features are compact and discriminative. Experimental results on public face databases with large head pose variations show the superior performance obtained by the proposed framework on the tasks of both head pose estimation and face recognition.