Sparse Subspace Clustering Research Articles

An ordered sparse subspace clustering algorithm based on p‐Norm

AbstractImages in video may include both Gaussian noise and geometric rotation. Thus, it is challenging to represent an image sequence in its intrinsically low‐dimensional space in a noise‐robust and rotation‐robust manner. In this paper, we propose a novel‐ordered sparse subspace clustering algorithm based on a p‐norm to achieve an effective clustering of sequential data under heavy noise conditions. We also use the wavelet‐histogram of oriented gradient (HOG) transform in the kernel view to extract both the global features (with the wavelet process) and the local features (with the HOG process) from the image. In addition, we assign different weights to different features to obtain a sparse coefficient matrix that helps to emphasize the global and local correlations in each sample. Similarly, the clustering algorithm based on the p‐norm for sequential images emphasizes the within‐class correlations amongst samples. Therefore, in this paper, we select additional denoising main components under a Laplacian constraint to achieve a better block‐diagonal structure and highlight the independence of different clusters. Extensive experiments performed on various public datasets (including the ordered face dataset, handwritten recognition dataset, video scene segmentation dataset, and object recognition dataset) demonstrate that the proposed method is more resilient to noise and rotation than other representative sparse subspace clustering methods.

Robust Stereo Visual Odometry Based on Probabilistic Decoupling Ego-Motion Estimation and 3D SSC

The outliers caused by noise and mismatching severely restrict the precision of visual odometry. Moreover, the dynamic environment is also a crucial element that decreases the robustness of the systems. This paper presents a robust stereo visual odometry by decoupled ego-motion estimation based on probabilistic matches and rejecting the outliers of dynamic objects through motion segmentation. Fast ZNCC method, based on local sum table and partition upper bound schemes, is presented for selecting probabilistic matches while keeping run-time efficiency. The selection of multi-correspondences can avoid mismatching of corresponding points. In consideration of noise interference, the essential matrix is computed in a probabilistic framework to estimate the initial value of the rotation matrix without estimated depth errors involved. Then, in order to estimate pose robustly in dynamic environment, a modified sparse subspace clustering (SSC) method is discussed, which aims to cluster the tracked 3D points cloud to avoid errors caused by affine transformation. The non-negative constraint makes the method suitable for fast moving camera. The proposed 3D-SSC method removes the outliers belonging to dynamic objects effectively. Finally, the detected inliers and depths are employed to estimate the translation matrix and refine rotation matrix. The proposed method is evaluated on the KITTI benchmark and compared with the state-of-the-art methods. The results show that our method is more robust as it can detect outliers more accurately in dynamic environments and achieve higher precision in motion estimation.

Forest-Fire Recognition by Sparse and Collaborative Subspace Clustering

Traditional forest-fire recognition based on the characteristics of smoke, temperature and light fails to accurately detect and respond to early fires. By analyzing the characteristics of flame, the methods based on aerial image recognition have been widely used, such as RGB-based and HIS-based methods. However, these methods are susceptible to background factors causing interference and false detection. To alleviate these problems, we investigate two subspace clustering methods based on sparse and collaborative representation, respectively, to detect and locate forest fires. Firstly, subspace clustering segments flame from the whole image. Afterwards, sparse or collaborative representation is employed to represent most of the flame information in a dictionary with l1-regularization or l2-regularization term, which results in fewer reconstruction errors. Experimental results show that the proposed SSC and CSC substantially outperform the state-of-the art methods.

L0 -Motivated Low-Rank Sparse Subspace Clustering.

In many applications, high-dimensional data points can be well represented by low-dimensional subspaces. To identify the subspaces, it is important to capture a global and local structure of the data which is achieved by imposing low-rank and sparseness constraints on the data representation matrix. In low-rank sparse subspace clustering (LRSSC), nuclear and l1 -norms are used to measure rank and sparsity. However, the use of nuclear and l1 -norms leads to an overpenalized problem and only approximates the original problem. In this paper, we propose two l0 quasi-norm-based regularizations. First, this paper presents regularization based on multivariate generalization of minimax-concave penalty (GMC-LRSSC), which contains the global minimizers of a l0 quasi-norm regularized objective. Afterward, we introduce the Schatten-0 ( S0 ) and l0 -regularized objective and approximate the proximal map of the joint solution using a proximal average method ( S0/l0 -LRSSC). The resulting nonconvex optimization problems are solved using an alternating direction method of multipliers with established convergence conditions of both algorithms. Results obtained on synthetic and four real-world datasets show the effectiveness of GMC-LRSSC and S0/l0 -LRSSC when compared to state-of-the-art methods.

Robust Affine Subspace Clustering via Smoothed $$\ell _{0}$$ ℓ 0 -Norm

In the past few years, sparse representation based method has been used in many fields with breathtaking speed due to its superior sparse recovery performance. Sparse subspace clustering (SSC), as one of its application hot-spots, has attracted considerable attention. Traditional sparse subspace clustering methods employ the $$\ell _{1}$$ -norm to induce sparse representation of data points. Typically, the use of $$\ell _{1}$$ -regularization instead of the $$\ell _{0}$$ one can make the objective function convex while it also causes large errors on large coefficients in some cases. In this work, we propose using the non-convex smoothed $$\ell _{0}$$ -norm to replace the $$\ell _{0}$$ one for affine subspace clustering. This leads to a non-convex minimization problem. We then propose an effective method to solve the problem which minimizes the objective function by using the gradient method and proximal projection. In addition, the proposed algorithm is robust to noise and can provide a fast solution. Extensive experiments on real datasets demonstrate the effectiveness of our proposed method.

Coded Aperture Design for Compressive Spectral Subspace Clustering

Compressive spectral imaging (CSI) acquires compressed observations of a spectral scene by applying different coding patterns at each spatial location and then performing a spectral-wise integration. Relying on compressive sensing, spectral image reconstruction is achieved by using nonlinear and relatively expensive optimization-based algorithms. In the CSI literature, several works have focused on improving reconstructions quality by properly designing the set of coding patterns. However, signal recovery is not actually necessary in many signal processing applications. For instance, assuming that compressed measurements with similar characteristics lie on the same subspace, unsupervised methods such as subspace clustering can be used to separate them into the same cluster. Since the structure of compressed measurements is defined by the applied codification, it is possible to improve clustering performance. This paper proposes to design a set of coding patterns such that inter-class and intra-class data structure is preserved after the CSI acquisition in order to improve clustering results directly on the compressed domain. To validate the coding pattern design, an algorithm based on sparse subspace clustering (SSC) is proposed to perform clustering on the compressed measurements. The proposed algorithm adds a three-dimensional (3-D) spatial regularizer to the SSC problem exploiting the spatial correlation of spectral images. In general, an overall accuracy up to 83.81% is obtained, when noisy measurements are assumed. In addition, a difference of at most 4% in terms of overall accuracy was observed when comparing the clustering results obtained by the full 3-D data with those achieved using CSI measurements acquired with the proposed coding pattern design.

A General Framework for Understanding Compressed Subspace Clustering Algorithms

Subspace clustering (SC) refers to the problem of clustering unlabeled high-dimensional data into a union of low-dimensional linear subspaces. In many practical scenarios, one may have access to only the compressed data due to constraints of measurement or computation. In this paper, based on the recently proposed restricted isometric property of Gaussian random projection for low-dimensional subspaces, we propose a general framework for analyzing the performance of various subspace clustering algorithms when applied to the compressed data. Our framework captures the connection between the problems of compressed subspace clustering (CSC) and noisy subspace clustering. With the existing study on noisy SC, our framework makes it possible to easily extend the results in noisy SC to CSC. In this paper, we select the most commonly used subspace clustering algorithms, i.e. , sparse SC (SSC), SSC-orthogonal matching pursuit (SSC-OMP), and thresholding based SC (TSC), as representatives and analyze their performance using the proposed framework. Our results are consistent with related work obtained by previous researchers, where our methodology is much more direct and has more universal implications. Finally, the practicability and efficiency of CSC are verified by numerical experiments.

On Geometric Analysis of Affine Sparse Subspace Clustering

Sparse subspace clustering (SSC) is a state-of-the-art method for segmenting a set of data points drawn from a union of subspaces into their respective subspaces. It is now well understood that SSC produces subspace-preserving data affinity under broad geometric conditions but suffers from a connectivity issue. In this paper, we develop a novel geometric analysis for a variant of SSC, named affine SSC (ASSC), for the problem of clustering data from a union of affine subspaces. Our contributions include a new concept called affine independence for capturing the arrangement of a collection of affine subspaces. Under the affine independence assumption, we show that ASSC is guaranteed to produce subspace-preserving affinity. Moreover, inspired by the phenomenon that the $\ell_1$ regularization no longer induces sparsity when the solution is nonnegative, we further show that subspace-preserving recovery can be achieved under much weaker conditions for all data points other than the extreme points of samples from each subspace. In addition, we confirm a curious observation that the affinity produced by ASSC may be subspace-dense---which could guarantee the subspace-preserving affinity of ASSC to produce correct clustering under rather weak conditions. We validate the theoretical findings on carefully designed synthetic data and evaluate the performance of ASSC on several real data sets.

Sparse and Low-Rank Subspace Data Clustering with Manifold Regularization Learned by Local Linear Embedding

Data clustering is an important research topic in data mining and signal processing communications. In all the data clustering methods, the subspace spectral clustering methods based on self expression model, e.g., the Sparse Subspace Clustering (SSC) and the Low Rank Representation (LRR) methods, have attracted a lot of attention and shown good performance. The key step of SSC and LRR is to construct a proper affinity or similarity matrix of data for spectral clustering. Recently, Laplacian graph constraint was introduced into the basic SSC and LRR and obtained considerable improvement. However, the current graph construction methods do not well exploit and reveal the non-linear properties of the clustering data, which is common for high dimensional data. In this paper, we introduce the classic manifold learning method, the Local Linear Embedding (LLE), to learn the non-linear structure underlying the data and use the learned local geometry of manifold as a regularization for SSC and LRR, which results the proposed LLE-SSC and LLE-LRR clustering methods. Additionally, to solve the complex optimization problem involved in the proposed models, an efficient algorithm is also proposed. We test the proposed data clustering methods on several types of public databases. The experimental results show that our methods outperform typical subspace clustering methods with Laplacian graph constraint.

Time series clustering based on sparse subspace clustering algorithm and its application to daily box-office data analysis

Movie box-office research is an important work for the rapid development of the film industry, and it is also a challenging task. Our study focuses on finding the regular box-office revenue patterns. Clustering algorithm is unsupervised machine learning algorithm which classifies the data in the absence of early knowledge of the classes. Unlike static data, the time series data vary with time. The work focused on time series clustering analysis is relatively less than those focused on static data. In this paper, the sparse subspace clustering (SSC) algorithm is introduced to analyze the time series data. The SSC algorithm has a better performance both on the artificial data set and the daily box-office data than recently developed well-known clustering algorithm such as K-means and spectral clustering algorithm. On the artificial data set, SSC is more suitable for time series, whether from the angle of clustering error or visualization. On the actual data, movies are divided into five clusters by SSC algorithm, and each cluster represents a distinct type of distribution pattern. And these patterns can be used in movie recommendation, film evaluation and can guide theater exhibitors and distributors. In addition, this is the first time to apply SSC to deal with time series clustering problem and get a pleasant effect.

Low-Rank Sparse Subspace for Spectral Clustering

Traditional graph clustering methods consist of two sequential steps, i.e., constructing an affinity matrix from the original data and then performing spectral clustering on the resulting affinity matrix. This two-step strategy achieves optimal solution for each step separately, but cannot guarantee that it will obtain the globally optimal clustering results. Moreover, the affinity matrix directly learned from the original data will seriously affect the clustering performance, since high-dimensional data are usually noisy and may contain redundancy. To address the above issues, this paper proposes a Low-rank Sparse Subspace (LSS) clustering method via dynamically learning the affinity matrix from low-dimensional space of the original data. Specifically, we learn a transformation matrix to project the original data to their low-dimensional space, by conducting feature selection and subspace learning in the sample self-representation framework. Then, we utilize the rank constraint and the affinity matrix directly obtained from the original data to construct a dynamic and intrinsic affinity matrix. Moreover, each of these three matrices is updated iteratively while fixing the other two. In this way, the affinity matrix learned from the low-dimensional space is the final clustering results. Extensive experiments are conducted on both synthetic and real datasets to show that our proposed LSS method outperforms the state-of-the-art clustering methods.

Fast subspace segmentation via Random Sample Probing

Abstract Subspace segmentation is to group a given set of n data points into multiple clusters, with each cluster corresponding to a subspace. Prevalent methods such as Sparse Subspace Clustering (SSC) and Low-Rank Representation (LRR) are effective in terms of segmentation accuracy, but computationally inefficient while applying to gigantic datasets where n is very large as they possess a complexity of O(n3). In this paper, we propose an iterative method called Random Sample Probing (RANSP). In each iteration, RANSP finds the members of one subspace by randomly choosing a data point (called “seed”) at first, and then using Ridge Regression (RR) to retrieve the other points that belong to the same subspace as the seed. Such a procedure is repeated until all points have been classified. RANSP has a computational complexity of O(n) and can therefore handle large-scale datasets. Experiments on synthetic and real datasets confirm the effectiveness and efficiency of RANSP.

Approximate sparse spectral clustering based on local information maintenance for hyperspectral image classification.

Sparse spectral clustering (SSC) has become one of the most popular clustering approaches in recent years. However, its high computational complexity prevents its application to large-scale datasets such as hyperspectral images (HSIs). In this paper, we propose two efficient approximate sparse spectral clustering methods for HSIs clustering in which clustering performance is improved by utilizing local information among the data. Firstly, we construct a smaller representative dataset on which sparse spectral clustering is performed. Then the labels of ground object are extending to whole dataset based on the local information according to two extending strategies. The first one is that the local interpolation is utilized to improve the extension of the clustering result. The other one is that the label extension is turned to a problem of subspace embedding, and is fulfilled by locally linear embedding (LLE). Several experiments on HSIs demonstrated that the proposed algorithms are effective for HSIs clustering.

Subspace Learning by $$\ell ^{0}$$ ℓ 0 -Induced Sparsity

Subspace clustering methods partition the data that lie in or close to a union of subspaces in accordance with the subspace structure. Such methods with sparsity prior, such as sparse subspace clustering (SSC) (Elhamifar and Vidal in IEEE Trans Pattern Anal Mach Intell 35(11):2765–2781, 2013) with the sparsity induced by the $$\ell ^{1}$$ -norm, are demonstrated to be effective in subspace clustering. Most of those methods require certain assumptions, e.g. independence or disjointness, on the subspaces. However, these assumptions are not guaranteed to hold in practice and they limit the application of existing sparse subspace clustering methods. In this paper, we propose $$\ell ^{0}$$ -induced sparse subspace clustering ( $$\ell ^{0}$$ -SSC). In contrast to the required assumptions, such as independence or disjointness, on subspaces for most existing sparse subspace clustering methods, we prove that $$\ell ^{0}$$ -SSC guarantees the subspace-sparse representation, a key element in subspace clustering, for arbitrary distinct underlying subspaces almost surely under the mild i.i.d. assumption on the data generation. We also present the “no free lunch” theorem which shows that obtaining the subspace representation under our general assumptions can not be much computationally cheaper than solving the corresponding $$\ell ^{0}$$ sparse representation problem of $$\ell ^{0}$$ -SSC. A novel approximate algorithm named Approximate $$\ell ^{0}$$ -SSC (A $$\ell ^{0}$$ -SSC) is developed which employs proximal gradient descent to obtain a sub-optimal solution to the optimization problem of $$\ell ^{0}$$ -SSC with theoretical guarantee. The sub-optimal solution is used to build a sparse similarity matrix upon which spectral clustering is performed for the final clustering results. Extensive experimental results on various data sets demonstrate the superiority of A $$\ell ^{0}$$ -SSC compared to other competing clustering methods. Furthermore, we extend $$\ell ^{0}$$ -SSC to semi-supervised learning by performing label propagation on the sparse similarity matrix learnt by A $$\ell ^{0}$$ -SSC and demonstrate the effectiveness of the resultant semi-supervised learning method termed $$\ell ^{0}$$ -sparse subspace label propagation ( $$\ell ^{0}$$ -SSLP).

Exploration of Neural Activity under Cognitive Reappraisal Using Simultaneous EEG-fMRI Data and Kernel Canonical Correlation Analysis.

Background Neural activity under cognitive reappraisal can be more accurately investigated using simultaneous EEG- (electroencephalography) fMRI (functional magnetic resonance imaging) than using EEG or fMRI only. Complementary spatiotemporal information can be found from simultaneous EEG-fMRI data to study brain function. Method An effective EEG-fMRI fusion framework is proposed in this work. EEG-fMRI data is simultaneously sampled on fifteen visually stimulated healthy adult participants. Net-station toolbox and empirical mode decomposition are employed for EEG denoising. Sparse spectral clustering is used to construct fMRI masks that are used to constrain fMRI activated regions. A kernel-based canonical correlation analysis is utilized to fuse nonlinear EEG-fMRI data. Results The experimental results show a distinct late positive potential (LPP, latency 200-700ms) from the correlated EEG components that are reconstructed from nonlinear EEG-fMRI data. Peak value of LPP under reappraisal state is smaller than that under negative state, however, larger than that under neutral state. For correlated fMRI components, obvious activation can be observed in cerebral regions, e.g., the amygdala, temporal lobe, cingulate gyrus, hippocampus, and frontal lobe. Meanwhile, in these regions, activated intensity under reappraisal state is obviously smaller than that under negative state and larger than that under neutral state. Conclusions The proposed EEG-fMRI fusion approach provides an effective way to study the neural activities of cognitive reappraisal with high spatiotemporal resolution. It is also suitable for other neuroimaging technologies using simultaneous EEG-fMRI data.

Successive Convex Approximation Algorithms for Sparse Signal Estimation With Nonconvex Regularizations

In this paper, we propose a successive convex approximation framework for sparse optimization where the nonsmooth regularization function in the objective function is nonconvex and it can be written as the difference of two convex functions. The proposed framework is based on a nontrivial combination of the majorization-minimization framework and the successive convex approximation framework proposed in literature for a convex regularization function. The proposed framework has several attractive features, namely, i) flexibility, as different choices of the approximate function lead to different type of algorithms; ii) fast convergence, as the problem structure can be better exploited by a proper choice of the approximate function and the stepsize is calculated by the line search; iii) low complexity, as the approximate function is convex and the line search scheme is carried out over a differentiable function; iv) guaranteed convergence to a stationary point. We demonstrate these features by two example applications in subspace learning, namely, the network anomaly detection problem and the sparse subspace clustering problem. Customizing the proposed framework by adopting the best-response type approximation, we obtain soft-thresholding with exact line search algorithms for which all elements of the unknown parameter are updated in parallel according to closed-form expressions. The attractive features of the proposed algorithms are illustrated numerically.

Noisy Subspace Clustering via Matching Pursuits

Sparsity-based subspace clustering algorithms have attracted significant attention thanks to their excellent performance in practical applications. A prominent example is the sparse subspace clustering (SSC) algorithm by Elhamifar and Vidal, which performs spectral clustering based on an adjacency matrix obtained by sparsely representing each data point in terms of all the other data points via the Lasso. When the number of data points is large or the dimension of the ambient space is high, the computational complexity of SSC quickly becomes prohibitive. Dyer et al. observed that SSC-OMP obtained by replacing the Lasso by the greedy orthogonal matching pursuit (OMP) algorithm results in significantly lower computational complexity, while often yielding comparable performance. The central goal of this paper is an analytical performance characterization of SSC-OMP for noisy data. Moreover, we introduce and analyze the SSC-MP algorithm, which employs matching pursuit (MP) in lieu of OMP. Both SSC-OMP and SSC-MP are proven to succeed even when the subspaces intersect and when the data points are contaminated by severe noise. The clustering conditions we obtain for SSC-OMP and SSC-MP are similar to those for SSC and for the thresholding-based subspace clustering (TSC) algorithm due to Heckel and B\"olcskei. Analytical results in combination with numerical results indicate that both SSC-OMP and SSC-MP with a data-dependent stopping criterion automatically detect the dimensions of the subspaces underlying the data. Moreover, experiments on synthetic and on real data show that SSC-MP compares very favorably to SSC, SSC-OMP, TSC, and the nearest subspace neighbor algorithm, both in terms of clustering performance and running time. In addition, we find that, in contrast to SSC-OMP, the performance of SSC-MP is very robust with respect to the choice of parameters in the stopping criteria.

Structural Reweight Sparse Subspace Clustering

Subspace clustering aims to segment a group of data points into a union of subspaces. Reweight sparse subspace clustering is one of the state-of-the-art algorithms which proposed an iterative weighted subspace clustering. The reweight matrix helps to improve the performance of the affinity matrix construction process but it easily falls into a local minimization. In this paper, we propose a structural reweight sparse subspace clustering algorithm which introduces the structural information into reweight subspace clustering. The structural information achieved in spectral clustering process is useful for the subsequent iterative optimization process which helps to obtain a better local minimization. The experimental results on the Extended Yale B, Hopkins 155, and COIL 20 datasets demonstrate that our algorithm achieves a better performance on subspace clustering problem.

Structured Sparse Subspace Clustering with Within-Cluster Grouping

Many high-dimensional data in computer vision essentially lie in multiple low-dimensional subspaces. Recently developed subspace clustering methods have shown good effectiveness in recovering the underlying low-dimensional subspace structure of high-dimensional data. The state-of-the-art methods show that sparseness and grouping effect of the affinity matrix are important for subspace clustering. The Structured Sparse Subspace Clustering (SSSC) model is a unified optimization framework for learning both the self-representation of the data and their subspace segmentation. But the SSSC only considers structured sparseness property of the affinity matrix. In this work, we define a concept of grouping-effect-within-cluster (GEWC) to group data from the same subspace together. Based on GEWC, we design a new regularization term coupling the self-representation matrix and the segmentation matrix. The new regularization term interactively enforces both to have the expected properties: the segmentation matrix enforces the self-representation coefficient vectors to have large cosine similarity, or GEWC, whenever the data points are drawn from the same subspace and they have the same cluster labels. On the other hand, the self-representation matrix enforces data to have the same cluster labels whenever their self-representation coefficient vectors have large cosine similarity. Incorporating the new penalty into the SSSC model, we present a new unified minimization framework for affinity learning and subspace clustering. The new model considers not only structured sparseness but also GEWC. Experimental results on several commonly used datasets demonstrate that our method outperforms other state-of-the-art methods in revealing the subspace structure of high-dimensional data.

Kernel sparse representation on Grassmann manifolds for visual clustering

Image sets and videos can be modeled as subspaces, which are actually points on Grassmann manifolds. Clustering of such visual data lying on Grassmann manifolds is a hard issue based on the fact that the state-of-the-art methods are only applied to vector space instead of non-Euclidean geometry. Although there exist some clustering methods for manifolds, the desirable method for clustering on Grassmann manifolds is lacking. We propose an algorithm termed as kernel sparse subspace clustering on the Grassmann manifold, which embeds the Grassmann manifold into a reproducing kernel Hilbert space by an appropriate Gaussian projection kernel. This kernel is applied to obtain kernel sparse representations of data on Grassmann manifolds utilizing the self-expressive property and exploiting the intrinsic Riemannian geometry within data. Although the Grassmann manifold is compact, the geodesic distances between Grassmann points are well measured by kernel sparse representations based on linear reconstruction. With the kernel sparse representations, clustering results of experiments on three prevalent public datasets outperform a number of existing algorithms and the robustness of our algorithm is demonstrated as well.