Union Of Low-dimensional Subspaces Research Articles

A Multiple Subspaces-Based Model: Interpreting Urban Functional Regions with Big Geospatial Data

Analyzing the urban spatial structure of a city is a core topic within urban geographical information science that has the ability to assist urban planning, site selection, location recommendation, etc. Among previous studies, comprehending the functionality of places is a central topic and corresponds to understanding how people use places. With the help of big geospatial data which contain affluent information about human mobility and activity, we propose a novel multiple subspaces-based model to interpret the urban functional regions. This model is based on the assumption that the temporal activity patterns of places lie in a high-dimensional space and can be represented by a union of low-dimensional subspaces. These subspaces are obtained through finding sparse representations using the data science method known as sparse subspace clustering (SSC). The paper details how to use this method in the context of detecting functional regions. With these subspaces, we can detect the functionality of urban regions in a designated study area and further explore the characteristics of functional regions. We conducted experiments using real data in Shanghai. The experimental results and outperformance of our proposed model against the single subspace-based method prove the efficacy and feasibility of our model.

Dynamic Sparse Subspace Clustering for Evolving High-Dimensional Data Streams.

In an era of ubiquitous large-scale evolving data streams, data stream clustering (DSC) has received lots of attention because the scale of the data streams far exceeds the ability of expert human analysts. It has been observed that high-dimensional data are usually distributed in a union of low-dimensional subspaces. In this article, we propose a novel sparse representation-based DSC algorithm, called evolutionary dynamic sparse subspace clustering (EDSSC). It can cope with the time-varying nature of subspaces underlying the evolving data streams, such as subspace emergence, disappearance, and recurrence. The proposed EDSSC consists of two phases: 1) static learning and 2) online clustering. During the first phase, a data structure for storing the statistic summary of data streams, called EDSSC summary, is proposed which can better address the dilemma between the two conflicting goals: 1) saving more points for accuracy of subspace clustering (SC) and 2) discarding more points for the efficiency of DSC. By further proposing an algorithm to estimate the subspace number, the proposed EDSSC does not need to know the number of subspaces. In the second phase, a more suitable index, called the average sparsity concentration index (ASCI), is proposed, which dramatically promotes the clustering accuracy compared to the conventionally utilized SCI index. In addition, the subspace evolution detection model based on the Page-Hinkley test is proposed where the appearing, disappearing, and recurring subspaces can be detected and adapted. Extinct experiments on real-world data streams show that the EDSSC outperforms the state-of-the-art online SC approaches.

Matrix completion with side information using manifold optimisation

The authors solve the matrix completion (MC) problem based on manifold optimisation by incorporating the side information under which the columns of the intended matrix are drawn from a union of low-dimensional subspaces. It is proved that this side information leads us to construct new manifolds, as an embedded sub-manifold of the manifold of constant rank matrices, using which the MC problem is solved more accurately. The required geometrical properties of the aforementioned manifold are then presented for MC. Simulation results over both synthetic and real-world data show that the proposed method outperforms some recent techniques either based on side information or not.

Efficient Solvers for Sparse Subspace Clustering

Sparse subspace clustering (SSC) clusters n points that lie near a union of low-dimensional subspaces. The SSC model expresses each point as a linear or affine combination of the other points, using either ℓ1 or ℓ0 regularization. Using ℓ1 regularization results in a convex problem but requires O(n2) storage, and is typically solved by the alternating direction method of multipliers which takes O(n3) flops. The ℓ0 model is non-convex but only needs memory linear in n, and is solved via orthogonal matching pursuit and cannot handle the case of affine subspaces. This paper shows that a proximal gradient framework can solve SSC, covering both ℓ1 and ℓ0 models, and both linear and affine constraints. For both ℓ1 and ℓ0, algorithms to compute the proximity operator in the presence of affine constraints have not been presented in the SSC literature, so we derive an exact and efficient algorithm that solves the ℓ1 case with just O(n2) flops. In the ℓ0 case, our algorithm retains the low-memory overhead, and is the first algorithm to solve the SSC-ℓ0 model with affine constraints. Experiments show our algorithms do not rely on sensitive regularization parameters, and they are less sensitive to sparsity misspecification and high noise.

Provable Subspace Clustering: When LRR Meets SSC

An important problem in analyzing big data is subspace clustering, i.e., to represent a collection of points in a high-dimensional space via the union of low-dimensional subspaces. Sparse subspace clustering (SSC) and Low-rank representation (LRR) are the state-of-the-art methods for this task. These two methods are fundamentally similar in that both are based on convex optimization exploiting the intuition of “Self-Expressiveness”. The main difference is that the SSC minimizes the vector ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> norm of the representation matrix to induce sparsity while LRR minimizes the nuclear norm (aka trace norm) to promote a low-rank structure. Because the representation matrix is often simultaneously sparse and low-rank, we propose a new algorithm, termed Low-rank sparse subspace clustering (LRSSC), by the combining SSC and LRR, and develop theoretical guarantees of the success of the algorithm. The results reveal interesting insights into the strengths and the weaknesses of SSC and LRR, and demonstrate how the LRSSC can take advantage of both methods in preserving the “Self-Expressiveness Property” and “Graph Connectivity” at the same time. A byproduct of our analysis is that it also expands the theoretical guarantee of SSC to handle cases when the subspaces have arbitrarily small canonical angles but are “nearly independent”.

Subspace clustering via structure-enforced dictionary learning

Many real world data, which we deal with today, often have very high dimensions. These high-dimensional data can be seen as collections of data points from a union of low-dimensional subspaces. Subspace clustering, one solution to the high-dimensional data problem, refers to a method by which a set of data points is divided into multiple clusters by finding multiple subspaces that fit each cluster. Most existing subspace clustering approaches construct an affinity matrix using the self-representation model, which can propagate disturbing noisy information because even noisy data points are used to represent the data; then, they perform spectral clustering on the obtained affinity matrix, which contains the irrelevant information about noise or outliers. This paper proposes a novel subspace clustering method based on the structured sparse PCA-based dictionary learning. Our proposed method learns the reduced dimensional dictionary and coefficient matrices using the structural information as well as sparsity of data. Then, the affinity matrix is constructed from inner product of the learned dictionary coefficient vectors, which shows the correlation among data points. The experimental results on three benchmark datasets verify that the proposed method outperforms state-of-the-art subspace clustering methods.

Motion Segmentation Based on Model Selection in Permutation Space for RGB Sensors.

Motion segmentation is aimed at segmenting the feature point trajectories belonging to independently moving objects. Using the affine camera model, the motion segmentation problem can be viewed as a subspace clustering problem—clustering the data points drawn from a union of low-dimensional subspaces. In this paper, we propose a solution for motion segmentation that uses a multi-model fitting technique. We propose a data grouping method and a model selection strategy for obtaining more distinguishable data point permutation preferences, which significantly improves the clustering. We perform extensive testing on the Hopkins 155 dataset, and two real-world datasets. The experimental results illustrate that the proposed method can deal with incomplete trajectories and the perspective effect, comparing favorably with the current state of the art.

Simultaneous Subspace Clustering and Cluster Number Estimating Based on Triplet Relationship.

In this paper, we propose a unified framework to discover the number of clusters and group the data points into different clusters using subspace clustering simultaneously. Real data distributed in a high-dimensional space can be disentangled into a union of low-dimensional subspaces, which can benefit various applications. To explore such intrinsic structure, state-of-the-art subspace clustering approaches often optimize a self-representation problem among all samples, to construct a pairwise affinity graph for spectral clustering. However, a graph with pairwise similarities lacks robustness for segmentation, especially for samples which lie on the intersection of two subspaces. To address this problem, we design a hyper-correlation-based data structure termed as the triplet relationship, which reveals high relevance and local compactness among three samples. The triplet relationship can be derived from the self-representation matrix, and be utilized to iteratively assign the data points to clusters. Based on the triplet relationship, we propose a unified optimizing scheme to automatically calculate clustering assignments. Specifically, we optimize a model selection reward and a fusion reward by simultaneously maximizing the similarity of triplets from different clusters while minimizing the correlation of triplets from the same cluster. The proposed algorithm also automatically reveals the number of clusters and fuses groups to avoid over-segmentation. Extensive experimental results on both synthetic and real-world datasets validate the effectiveness and robustness of the proposed method.

Generalizing CoSaMP to signals from a union of low dimensional linear subspaces

The idea that signals reside in a union of low dimensional subspaces subsumes many low dimensional models that have been used extensively in the recent decade in many fields and applications. Until recently, the vast majority of works have studied each one of these models on its own. However, a recent approach suggests providing general theory for low dimensional models using their Gaussian mean width, which serves as a measure for the intrinsic low dimensionality of the data. In this work we use this novel approach to study a generalized version of the popular compressive sampling matching pursuit (CoSaMP) algorithm, and to provide general recovery guarantees for signals from a union of low dimensional linear subspaces, under the assumption that the measurement matrix is Gaussian. We discuss the implications of our results for specific models, and use the generalized algorithm as an inspiration for a new greedy method for signal reconstruction in a combined sparse-synthesis and cosparse-analysis model. We perform experiments that demonstrate the usefulness of the proposed strategy.

Bi-sparsity pursuit: A paradigm for robust subspace recovery

Abstract The success of sparse models in computer vision and machine learning is due to the fact that, high dimensional data is distributed in a union of low dimensional subspaces in many real-world applications. The underlying structure may, however, be adversely affected by sparse errors. In this paper, we propose a bi-sparse model as a framework to analyze this problem, and provide a novel algorithm to recover the union of subspaces in the presence of sparse corruptions. We further show the effectiveness of our method by experiments on real-world vision data.

Accelerated orthogonal least-squares for large-scale sparse reconstruction

We study the problem of inferring a sparse vector from random linear combinations of its components. We propose the Accelerated Orthogonal Least-Squares (AOLS) algorithm that improves performance of the well-known Orthogonal Least-Squares (OLS) algorithm while requiring significantly lower computational costs. While OLS greedily selects columns of the coefficient matrix that correspond to non-zero components of the sparse vector, AOLS employs a novel computationally efficient procedure that speeds up the search by anticipating future selections via choosing L columns in each step, where L is an adjustable hyper-parameter. We analyze the performance of AOLS and establish lower bounds on the probability of exact recovery for both noiseless and noisy random linear measurements. In the noiseless scenario, it is shown that when the coefficients are samples from a Gaussian distribution, AOLS with high probability recovers a k-sparse m-dimensional sparse vector using O(klog⁡mk+L−1) measurements. Similar result is established for the bounded-noise scenario where an additional condition on the smallest nonzero element of the unknown vector is required. The asymptotic sampling complexity of AOLS is lower than the asymptotic sampling complexity of the existing sparse reconstruction algorithms. In simulations, AOLS is compared to state-of-the-art sparse recovery techniques and shown to provide better performance in terms of accuracy, running time, or both. Finally, we consider an application of AOLS to clustering high-dimensional data lying on the union of low-dimensional subspaces and demonstrate its superiority over existing methods.

Sparse Subspace Modeling for Query by Example Spoken Term Detection

This paper focuses on the problem of query by example spoken term detection (QbE-STD) in zero-resource scenario. Current state-of-the-art approaches to tackle this problem rely on dynamic programming based template matching techniques using phone posterior features extracted at the output of a deep neural network. Previously, it has been shown that the space of phone posteriors is highly structured, as a union of low-dimensional subspaces. To exploit the temporal and sparse structure of the speech data, we investigate here three different QbE-STD systems based on sparse model recovery. More specifically, we use query examples to model the query subspace using dictionary for sparse coding. Reconstruction errors calculated using sparse representation of feature vectors are then used to characterize the underlying subspaces. The first approach uses these reconstruction errors in a dynamic programming framework to detect the spoken query, resulting in a much faster search compared to standard template matching. The other two methods aim at merging template matching and sparsity-based approaches to further improve the performance. The first one proposes to regularize the template matching local distances using sparse reconstruction errors. The second approach aims at using the sparse reconstruction errors to rescore (improve) the template matching likelihood. Experiments on two different databases (AMI and MediaEval) show that the proposed hybrid systems perform better than a highly competitive QbE-STD baseline system.

High Rank Matrix Completion With Side Information

We address the problem of high-rank matrix completion with side information. In contrast to existing work dealing with side information, which assume that the data matrix is low-rank, we consider the more general scenario where the columns of the data matrix are drawn from a union of low-dimensional subspaces, which can lead to a high rank matrix. Our goal is to complete the matrix while taking advantage of the side information. To do so, we use the self-expressive property of the data, searching for a sparse representation of each column of matrix as a combination of a few other columns. More specifically, we propose a factorization of the data matrix as the product of side information matrices with an unknown interaction matrix, under which each column of the data matrix can be reconstructed using a sparse combination of other columns. As our proposed optimization, searching for missing entries and sparse coefficients, is non-convex and NP-hard, we propose a lifting framework, where we couple sparse coefficients and missing values and define an equivalent optimization that is amenable to convex relaxation. We also propose a fast implementation of our convex framework using a Linearized Alternating Direction Method. By extensive experiments on both synthetic and real data, and, in particular, by studying the problem of multi-label learning, we demonstrate that our method outperforms existing techniques in both low-rank and high-rank data regimes.

Active Orthogonal Matching Pursuit for Sparse Subspace Clustering

Sparse Subspace Clustering (SSC) is a state-of-the-art method for clustering high-dimensional data points lying in a union of low-dimensional subspaces. However, while $\ell_1$ optimization-based SSC algorithms suffer from high computational complexity, other variants of SSC, such as Orthogonal Matching Pursuit-based SSC (OMP-SSC), lose clustering accuracy in pursuit of improving time efficiency. In this letter, we propose a novel Active OMP-SSC, which improves clustering accuracy of OMP-SSC by adaptively updating data points and randomly dropping data points in the OMP process, while still enjoying the low computational complexity of greedy pursuit algorithms. We provide heuristic analysis of our approach, and explain how these two active steps achieve a better tradeoff between connectivity and separation. Numerical results on both synthetic data and real-world data validate our analyses and show the advantages of the proposed active algorithm.

A Structured Sparse Plus Structured Low-Rank Framework for Subspace Clustering and Completion

Recent advances in a low-rank matrix completion have enabled the exact recovery of incomplete data drawn from a low-dimensional subspace of a high-dimensional observation space. However, in many applications, the data are drawn from multiple low-dimensional subspaces without knowing which point belongs to which subspace. In such cases, using a single low-dimensional subspace to complete the data may lead to erroneous results, because the complete data matrix need not be low rank. In this paper, we propose a structured sparse plus structured low-rank ( $\text{S}^3$ LR) optimization framework for clustering and completing data drawn from a union of low-dimensional subspaces. The proposed $\text{S}^3$ LR framework exploits the fact that each point in a union of subspaces can be expressed as a sparse linear combination of all other points and that the matrix of the points within each subspace is low rank . This framework leads to a nonconvex optimization problem, which we solve efficiently by using a combination of a linearized alternating direction method of multipliers and spectral clustering. In addition, we discuss the conditions that guarantee the exact matrix completion in a union of subspaces. Experiments on synthetic data, motion segmentation data, and cancer gene data validate the effectiveness of the proposed approach.

Discriminative Subspace Alignment for Unsupervised Visual Domain Adaptation

We address the problem of unsupervised visual domain adaptation for transferring category models from one visual domain or image data set to another. We present a new unsupervised domain adaptation algorithm based on subspace alignment. The core idea of our approach is to reduce the discrepancy between the source domain and the target domain in a latent discriminative subspace. Specifically, we first generate pseudo-labels for the target data by applying spectral clustering to a cross-domain similarity matrix, which is built from sparse coefficients found in a low-dimensional latent space. This coarse alignment between the two domains exploits the assumption that the collection of data of different classes from both domains can be viewed as samples from a union of low-dimensional subspaces. Then, we create discriminative subspaces for both domains using partial least squares correlation. Finally, a mapping which aligns the discriminative source subspace into the target one is learned by minimizing a Bregman matrix divergence function. Experimental results on benchmark cross-domain visual object recognition data sets and cross-view scene classification data sets demonstrate that the proposed method outperforms the baselines and several state-of-the-art competing methods.

Robust Subspace Clustering via Thresholding

The problem of clustering noisy and incompletely observed high-dimensional data points into a union of low-dimensional subspaces and a set of outliers is considered. The number of subspaces, their dimensions, and their orientations are assumed unknown. We propose a simple low-complexity subspace clustering algorithm, which applies spectral clustering to an adjacency matrix obtained by thresholding the correlations between data points. In other words, the adjacency matrix is constructed from the nearest neighbors of each data point in spherical distance. A statistical performance analysis shows that the algorithm exhibits robustness to additive noise and succeeds even when the subspaces intersect. Specifically, our results reveal an explicit tradeoff between the affinity of the subspaces and the tolerable noise level. We furthermore prove that the algorithm succeeds even when the data points are incompletely observed with the number of missing entries allowed to be (up to a log-factor) linear in the ambient dimension. We also propose a simple scheme that provably detects outliers, and we present numerical results on real and synthetic data.

Study on Different Representation Methods for Subspace Segmentation

With many engineering and science application problems, we must deal with a lot of high-dimensional data, such as videos, images, web documents, text, etc. In the areas of computer vision, image processing and machine learning, high-dimensional data are widespread. However, it is very hard for obtaining meaningful learning and inference from these high-dimensional data directly, the computational complexity of highdimensional data is often exponential. However, under many conditions, highdimensional data lie in low-dimensional data corresponding to some classes of the data. Thus, finding the low-dimensional structure from the high-dimensional data is very important. The aim of subspace segmentation is to cluster data that lie in a union of lowdimensional subspaces. In recent years, based on the research of representation methods, many subspace segmentation algorithms appeared. Although these methods are all effective for handling subspace segmentation problems, they all have advantages and disadvantages. This paper focuses on the performance comparison of different subspace segmentation algorithms currently used in handling subspace segmentation problems and views other conventional methods that can be applied in this field.

Linear-time subspace clustering via bipartite graph modeling.

We present a linear-time subspace clustering approach that combines sparse representations and bipartite graph modeling. The signals are modeled as drawn from a union of low-dimensional subspaces, and each signal is represented by a sparse combination of basis elements, termed atoms, which form the columns of a dictionary matrix. The sparse representation coefficients are arranged in a sparse affinity matrix, which defines a bipartite graph of two disjoint sets: 1) atoms and 2) signals. Subspace clustering is obtained by applying low-complexity spectral bipartite graph clustering that exploits the small number of atoms for complexity reduction. The complexity of the proposed approach is linear in the number of signals, thus it can rapidly cluster very large data collections. Performance evaluation of face clustering and temporal video segmentation demonstrates comparable clustering accuracies to state-of-the-art at a significantly lower computational load.

Band selection using sparse nonnegative matrix factorization with the thresholded Earth’s mover distance for hyperspectral imagery classification

A sparse nonnegative matrix factorization method with the thresholded ground distance (SNMF-TEMD) is proposed to solve the band selection problem in hyperspectral imagery (HSI) classification. The SNMF-TEMD assumes that band vectors are sampled from a union of low-dimensional subspaces and approximates a HSI data matrix with the product of a basis matrix constructed from subspaces and a sparse coefficient matrix. The SNMF-TEMD utilizes the TEMD metric to better measures approximation errors during the optimization of HSI data factorization. The TEMD metric makes up the theoretical drawbacks in the Euclidean distance (ED) and Kullback–Leibler divergence (KLD) metrics when measuring the approximation errors in HSI datasets. The SNMF-TEMD is solved by the combination of min-cost-flow algorithm and multiplicative update rules. The band cluster assignments are found according to positions of largest entries in columns of the coefficient matrix and the desired band subset constitutes with the bands closest to their cluster centers. Three groups of experiments on two HSI datasets are performed to explore the performance of SNMF-TEMD. Four popular band selection methods are used to make comparisons: affinity propagation (AP), maximum-variance principal component analysis (MVPCA), SNMF with ED metric (SNMF-ED) and SNMF with KLD metric (SNMF-KLD). Experimental results show that SNMF-TEMD outperforms all four methods in classification accuracy and its computational speed is slower than SNMF-ED and SNMF-KLD. SNMF-TEMD is a better choice for band selection among all five methods because of its overwhelming advantage in classification and the popular speed remedy scheme from parallel computing and high-performance computers.