Norm Sparsity Research Articles

L0 -Regularized Image Downscaling.

In this paper, we propose a novel -regularized optimization framework for image downscaling. The optimization is driven by two -regularized priors. The first prior, gradient-ratio prior, is based on the observation that the number of edges in the downscaled image is approximately inverse square proportional to the downscaling factor. By introducing norm sparsity to the gradient ratio, the downscaled image is able to preserve the most salient edges as well as the visual perception of the original image. The second prior, downsampling prior, is to constrain the downsampling matrix so that pixels of the downscaled image are estimated according to those optimal neighboring pixels. Extensive experiments on the Urban100 and BSDS500 data sets show that the proposed algorithm achieves superior performance over the state-of-the-arts, in terms of both quality and robustness.

Orthogonal Nonnegative Matrix Factorization by Sparsity and Nuclear Norm Optimization

In this paper, we study orthogonal nonnegative matrix factorization. We demonstrate the coefficient matrix can be sparse and low-rank in the orthogonal nonnegative matrix factorization. By using these properties, we propose to use a sparsity and nuclear norm minimization for the factorization and develop a convex optimization model for finding the coefficient matrix in the factorization. Numerical examples including synthetic and real-world data sets are presented to illustrate the effectiveness of the proposed algorithm and demonstrate that its performance is better than other testing methods.

Joint Discriminative Dictionary and Classifier Learning for ALS Point Cloud Classification

To efficiently recognize on-ground objects in airborne laser scanning (ALS) point clouds, we design a method that jointly learns a discriminative dictionary and a classifier. In the method, the point cloud is segmented into hierarchical point clusters, which are organized by a tree structure. Then, the feature of each point cluster is extracted. The feature of a leaf node is obtained by aggregating the features of all its parent nodes. The feature of the leaf node is called the hierarchical aggregation feature. The hierarchical aggregation features are encoded by sparse coding. We introduce a new label consistency constraint called “discriminative sparse-code error,” and combine it with the reconstruction error, the classification error, and $L_{1}$ -norm sparsity constraint to form a unified objective function. The objective function is efficiently solved by using the proposed label consistency feature sign method. We obtain an overcomplete discriminative dictionary and an optimal linear classifier. Experiments performed on different ALS point cloud scenes have shown that the hierarchical aggregation features combined with the learned classifier can significantly enhance the classification results, and also demonstrated the superior performance of our method over other techniques in point cloud classification.

Sparsity constraints using a Laplacian kernel to get geological structures from potential field data

ABSTRACTWe propose a new method for including sparsity constraints into potential field data inversion using a Laplacian kernel. The method obtains three‐dimensional density distributions, which are at best a proxy for geological structures. Compressive sensing has been used in many domains to recover the original data from the acquisition data. Compressive sensing is an inversion philosophy where sparsity constraints are applied, and thus, we use this principle into geophysical inversion. Here, we extend this algorithm to potential field data inversion. We introduced the Laplacian smoothed L0 norm into the stabiliser function as a sparse constraint, and the density constraint function has been introduced into our inversion method to guarantee the inverted density to be in the geological and physical meaning range. Compared to the traditional smooth inverse algorithm, our proposed method can obtain accurate geological structures with sharp boundary and sub‐surface “block” structures. This method permits reconstruction of (non‐smooth) density functions that represent a blocky geological structure. Our results using synthetic gravity data show that the Laplacian smoothed L0 norm inversion method with sparsity constraints predicts more focused and accurate depth and density anomalies than smooth inversion method. Application of this Laplacian smoothed L0 norm sparsity constraints method to the inversion of gravity data collected over the Humble salt dome, Harris County, near Houston, United States, leads to improved interpretation of geological structures. These results confirm the validity of the proposed method and its potential application for other potential field data inversions to explore geological structures.

Local and Nonlocal Context-Aware Elastic Net Representation-Based Classification for Hyperspectral Images

By representing a query sample as a linear combination of all labeled samples and then classifying it by evaluating which class leads to the minimal representation error, the representation-based classification methods have been successfully used for the classification of hyperspectral images (HSI). According to the usage of different norms, the sparse representation-based classification (SRC) and collaborative representation-based classification (CRC) methods have been presented in two different paradigms. The SRC promotes the use of few labeled samples, while the CRC encourages the use of all labeled samples to collaboratively represent the query one from all classes. However, when the limited labeled samples of different classes are unbalance, the learnt representation is hardly to reflect the particular characteristics of each class. To overcome this problem, this paper presents a novel graph based context-aware elastic net (ELN) model for the HSI classification. Under a generalized ELN framework, the proposed model is able to take full advantages of SRC and CRC. Specifically, by evaluating the spectral and spatial self-similarity of local and nonlocal neighbors, an ELN-coding neighborhood graph is constructed with image patch distance. Owing to the exploitation of the spectral-spatial context, a centralized sparsity norm is integrated into the optimization model and it can promote the local and global consistence preserving. Finally, an efficient solver for the proposed model is developed by using the well-known alternating direction method of multiplier. Experiments on several real datasets validated that the proposed method can outperform state-of-the-art algorithms in terms of classification accuracy. Furthermore, even with the limited unbalanced labeled samples the proposed method is robust.

Robust auto-weighted multi-view subspace clustering with common subspace representation matrix.

In many computer vision and machine learning applications, the data sets distribute on certain low-dimensional subspaces. Subspace clustering is a powerful technology to find the underlying subspaces and cluster data points correctly. However, traditional subspace clustering methods can only be applied on data from one source, and how to extend these methods and enable the extensions to combine information from various data sources has become a hot area of research. Previous multi-view subspace methods aim to learn multiple subspace representation matrices simultaneously and these learning task for different views are treated equally. After obtaining representation matrices, they stack up the learned representation matrices as the common underlying subspace structure. However, for many problems, the importance of sources and the importance of features in one source both can be varied, which makes the previous approaches ineffective. In this paper, we propose a novel method called Robust Auto-weighted Multi-view Subspace Clustering (RAMSC). In our method, the weight for both the sources and features can be learned automatically via utilizing a novel trick and introducing a sparse norm. More importantly, the objective of our method is a common representation matrix which directly reflects the common underlying subspace structure. A new efficient algorithm is derived to solve the formulated objective with rigorous theoretical proof on its convergency. Extensive experimental results on five benchmark multi-view datasets well demonstrate that the proposed method consistently outperforms the state-of-the-art methods.

SIRSE: A secure identity recognition scheme based on electroencephalogram data with multi-factor feature

As a common bioelectric phenomenon, brainwave is produced by electric field change in human brain. The use of electroencephalogram (EEG) in identity recognition greatly enhances recognition security and accuracy. In this work, we propose a secure identity recognition method by using EEG with multi-factor feature. Machine learning is utilized to extract, classify and select features of EEG data. Structural sparse norm in EEG data has consistency. The feature is helpful in optimization. The experiments show that SIRSE has superiority on security, recognition success rate and reliability in comparison.

Recovering Hidden Diagonal Structures via Non-Negative Matrix Factorization with Multiple Constraints.

Revealing data with intrinsically diagonal block structures is particularly useful for analyzing groups of highly correlated variables. Earlier researches based on non-negative matrix factorization (NMF) have been shown to be effective in representing such data by decomposing the observed data into two factors, where one factor is considered to be the feature and the other the expansion loading from a linear algebra perspective. If the data are sampled from multiple independent subspaces, the loading factor would possess a diagonal structure under an ideal matrix decomposition. However, the standard NMF method and its variants have not been reported to exploit this type of data via direct estimation. To address this issue, a non-negative matrix factorization with multiple constraints model is proposed in this paper. The constraints include an sparsity norm on the feature matrix and a total variational norm on each column of the loading matrix. The proposed model is shown to be capable of efficiently recovering diagonal block structures hidden in observed samples. An efficient numerical algorithm using the alternating direction method of multipliers model is proposed for optimizing the new model. Compared with several benchmark models, the proposed method performs robustly and effectively for simulated and real biological data.

Dictionary learning with the $${{\ell }_{1/2}}$$-regularizer and the coherence penalty and its convergence analysis

The \({{\ell }_{1/2}}\)-regularizer has been studied widely in compressed sensing, but there have been few studies about dictionary learning problems. The dictionary learning method with the \({{\ell }_{1/2}}\)-regularizer aims to learn a dictionary, which requires solving a very challenging nonconvex and nonsmooth optimization problem. In addition, the low mutual coherence of a dictionary is an important property that ensures the optimality of the sparse representation in the dictionary. In this paper, we address a dictionary learning problem involving the \({{\ell }_{1/2}}\)-regularizer and the coherence penalty, which is difficult to solve quickly and efficiently. We employ a decomposition scheme and an alternating optimization, which transforms the overall problem into a set of minimizations of single-vector-variable subproblems. Although the subproblems are nonsmooth and even nonconvex, we propose the use of proximal operator technology to conquer them, which leads to a rapid and efficient dictionary learning algorithm. In a theoretical analysis, we establish the algorithm’s global convergence. Experiments were performed for dictionary learning using both synthetic data and real-world data. For the synthetic data, we demonstrated that our algorithm performed better than state-of-the-art algorithms. Using real-world data, the learned dictionaries were shown to be more efficient than algorithms using \({{\ell }_{1}}\)-norm for sparsity.

Sparse-promoting full-waveform inversion based on online orthonormal dictionary learning

Full-waveform inversion (FWI) delivers high-resolution images of the subsurface by minimizing iteratively the misfit between recorded and calculated seismic data. We have attacked this misfit successfully with the Gauss-Newton method and sparsity-promoting regularization based on fixed multiscale transforms that permit significant subsampling of the seismic data when the model perturbation at each FWI data-fitting iteration can be represented with sparse coefficients. Rather than using analytical transforms with predefined dictionaries to achieve sparse representation, we developed an adaptive transform called the sparse orthonormal transform (SOT), whose dictionary is learned from many small training patches taken from the model perturbations in previous iterations. The patch-based dictionary is constrained to be orthonormal and trained with an online approach to provide the best sparse representation of the complex features and variations in the entire model perturbation. The complexity of the training method is proportional to the cube of the number of samples in one small patch. By incorporating compressive subsampling and the adaptive SOT-based representation into the Gauss-Newton least-squares problem for each FWI iteration, the model perturbation can be recovered after an [Formula: see text]-norm sparsity constraint is applied on the SOT coefficients. Numerical experiments on synthetic models determined that the SOT-based sparsity-promoting regularization can provide robust FWI results with reduced computation.

Adaptive low-rank subspace learning with online optimization for robust visual tracking

In recent years, sparse and low-rank models have been widely used to formulate appearance subspace for visual tracking. However, most existing methods only consider the sparsity or low-rankness of the coefficients, which is not sufficient enough for appearance subspace learning on complex video sequences. Moreover, as both the low-rank and the column sparse measures are tightly related to all the samples in the sequences, it is challenging to incrementally solve optimization problems with both nuclear norm and column sparse norm on sequentially obtained video data. To address above limitations, this paper develops a novel low-rank subspace learning with adaptive penalization (LSAP) framework for subspace based robust visual tracking. Different from previous work, which often simply decomposes observations as low-rank features and sparse errors, LSAP simultaneously learns the subspace basis, low-rank coefficients and column sparse errors to formulate appearance subspace. Within LSAP framework, we introduce a Hadamard production based regularization to incorporate rich generative/discriminative structure constraints to adaptively penalize the coefficients for subspace learning. It is shown that such adaptive penalization can significantly improve the robustness of LSAP on severely corrupted dataset. To utilize LSAP for online visual tracking, we also develop an efficient incremental optimization scheme for nuclear norm and column sparse norm minimizations. Experiments on 50 challenging video sequences demonstrate that our tracker outperforms other state-of-the-art methods.

Tree-Structured Nuclear Norm Approximation with Applications to Robust Face Recognition.

Structured sparsity, as an extension of standard sparsity, has shown the outstanding performance when dealing with some highly correlated variables in computer vision and pattern recognition. However, the traditional mixed (L1, L2) or (L1, L∞) group norm becomes weak in characterizing the internal structure of each group since they cannot alleviate the correla-tions between variables. Recently, nuclear norm has been vali-dated to be useful for depicting a spatially structured matrix variable. It considers the global structure of the matrix variable but overlooks the local structure. To combine the advantages of structured sparsity and nuclear norm, this paper presents a tree-structured nuclear norm approximation (TSNA) model as-suming that the representation residual with tree-structured prior is a random matrix variable and follows a dependent matrix dis-tribution. The Extended Alternating Direction Method of Multi-pliers (EADMM) is utilized to solve the proposed model. An effi-cient bound condition based on the extended restricted isometry constants is provided to show the exact recovery of the proposed model under the given noisy case. In addition, TSNA is connected with some newest methods such as sparse representation based classifier (SRC), nuclear-L1 norm joint regression (NL1R) and nuclear norm based matrix regression (NMR), which can be re-garded as the special cases of TSNA. Experiments with face re-construction and recognition demonstrate the benefits of TSNA over other approaches.

Exploring canonical correlation analysis with subspace and structured sparsity for web image annotation

Canonical correlation analysis (CCA) has been extensively exploited for modelling Internet multimedia. However, two major challenges are raised for the classical CCA. First, CCA frequently fails to remove noisy and irrelevant features. Second, CCA cannot effectively capture the correlation between semantic labels, which is especially beneficial for annotating web images. In this paper, we propose a new framework that integrates structural sparsity and low-rank shared subspace into the least-squares formulation of CCA. Under this framework, multiple label interactions can be uncovered by the shared common structure of the input space. Meanwhile, a few highly discriminative features can be decided via the structural sparse norm. Owing to the presence of non-smooth structured sparsity, a new efficient iterative algorithm is derived with guaranteed convergence. The empirical studies over several popular web image data collections consistently deliver the effectiveness of our new formulation in comparison with competing algorithms.

Learning Dictionary of Discriminative Part Detectors for Image Categorization and Cosegmentation

This paper proposes a novel approach to learning mid-level image models for image categorization and cosegmentation. We represent each image class by a dictionary of part detectors that best discriminate that class from the background. We learn category-specific part detectors in a weakly supervised setting in which the training images are only annotated with category labels without part/object location information. We use a latent SVM model regularized using the $$\ell _{2,1}$$l2,1 group sparsity norm to learn the part detectors. Starting from a large set of initial parts, the group sparsity regularizer forces the model to jointly select and optimize a set of discriminative part detectors in a max-margin framework. We propose a stochastic version of a proximal algorithm to solve the corresponding optimization problem. We apply the learned part detectors to image classification and cosegmentation, and present extensive comparative experiments with standard benchmarks.

3D face reconstruction via landmark depth estimation and shape deformation

Since a human face could be represented by a few landmarks with less redundant information, and calculated by a linear combination of a small number of prototypical faces, we propose a two-step 3D face reconstruction approach including landmark depth estimation and shape deformation. The proposed approach allows us to reconstruct a realistic 3D face from a 2D frontal face image. First, we apply a coupled dictionary learning method based on sparse representation to explore the underlying mappings between pair of 2D and 3D training landmarks. In the method, a weighted l1 norm sparsity function is introduced to better pursuit the l0 norm sparsity. Then, the depth of the landmarks could be estimated. Second, we propose a novel shape deformation method to reconstruct the 3D face by combining a small number of most relevant deformed faces which are obtained by the estimated landmarks. The sparsity regulation is also introduced to find the relevant faces in the second step. The proposed approach could explore the distributions of 2D and 3D faces and the underlying mappings between them well, because human faces are represented by low-dimensional landmarks, and their distributions are described by sparse representations. Moreover, it is much more flexible since we can make any change in any step. Extensive experiments are conducted on BJUT_3D database, and the results validate the effectiveness of the proposed approach.

Zero Attracting Recursive Least Squares Algorithms

The $l_{1}$ -norm sparsity constraint is a widely used technique for constructing sparse models. In this paper, two zero-attracting recursive least squares algorithms, which are referred to as ZA-RLS-I and ZA-RLS-II, are derived by employing the $l_{1}$ -norm of the parameter vector constraint to facilitate model sparsity. To achieve a closed-form solution, the $l_{1}$ -norm of the parameter vector is approximated by an adaptively weighted $l_{2}$ -norm in which the weighting factors are set as the inversion of the associated $l_{1}$ -norm of parameter estimates that are readily available in the adaptive learning environment. ZA-RLS-II is computationally more efficient than ZA-RLS-I by exploiting the known results from linear algebra and the sparsity of the system. The proposed algorithms are proven to converge, and adaptive sparse channel estimation is used to demonstrate the effectiveness of the proposed approach.

Structured occlusion coding for robust face recognition

Occlusion in face recognition is a common yet challenging problem. While sparse representation based classification (SRC) has been shown promising performance in laboratory conditions (i.e. noiseless or random pixel corrupted), it performs much worse in practical scenarios. In this paper, we consider the practical face recognition problem, where the occlusions are predictable and available for sampling. We propose the structured occlusion coding (SOC) to address occlusion problems. The structured coding here lies in two folds. On one hand, we employ a structured dictionary for recognition. On the other hand, we propose to use the structured sparsity in this formulation. Specifically, SOC simultaneously separates the occlusion and classifies the image. In this way, the problem of recognizing an occluded image is turned into seeking a structured sparse solution on occlusion-appended dictionary. In order to construct a well-performing occlusion dictionary, we propose an occlusion mask estimating technique via locality constrained dictionary (LCD), showing striking improvement in occlusion sample. On a category-specific occlusion dictionary, we replace l1 norm sparsity with the structured sparsity which is shown more robust, further enhancing the robustness of our approach. Moreover, SOC achieves significant improvement in handling large occlusion in real world. Extensive experiments are conducted on public data sets to validate the superiority of the proposed algorithm.

On Efficient Feature Ranking Methods for High-Throughput Data Analysis.

Efficient mining of high-throughput data has become one of the popular themes in the big data era. Existing biology-related feature ranking methods mainly focus on statistical and annotation information. In this study, two efficient feature ranking methods are presented. Multi-target regression and graph embedding are incorporated in an optimization framework, and feature ranking is achieved by introducing structured sparsity norm. Unlike existing methods, the presented methods have two advantages: (1) the feature subset simultaneously account for global margin information as well as locality manifold information. Consequently, both global and locality information are considered. (2) Features are selected by batch rather than individually in the algorithm framework. Thus, the interactions between features are considered and the optimal feature subset can be guaranteed. In addition, this study presents a theoretical justification. Empirical experiments demonstrate the effectiveness and efficiency of the two algorithms in comparison with some state-of-the-art feature ranking methods through a set of real-world gene expression data sets.

Simultaneously Structured Models With Application to Sparse and Low-Rank Matrices

Recovering structured models (e.g., sparse or group-sparse vectors, low-rank matrices) given a few linear observations have been well-studied recently. In various applications in signal processing and machine learning, the model of interest is structured in several ways, for example, a matrix that is simultaneously sparse and low rank. Often norms that promote the individual structures are known, and allow for recovery using an orderwise optimal number of measurements (e.g., $\ell _{1}$ norm for sparsity, nuclear norm for matrix rank). Hence, it is reasonable to minimize a combination of such norms. We show that, surprisingly, using multiobjective optimization with these norms can do no better, orderwise, than exploiting only one of the structures, thus revealing a fundamental limitation in sample complexity. This result suggests that to fully exploit the multiple structures, we need an entirely new convex relaxation. Further, specializing our results to the case of sparse and low-rank matrices, we show that a nonconvex formulation recovers the model from very few measurements (on the order of the degrees of freedom), whereas the convex problem combining the $\ell _{1}$ and nuclear norms requires many more measurements, illustrating a gap between the performance of the convex and nonconvex recovery problems. Our framework applies to arbitrary structure-inducing norms as well as to a wide range of measurement ensembles. This allows us to give sample complexity bounds for problems such as sparse phase retrieval and low-rank tensor completion.

Sparse and Low-Rank Covariance Matrix Estimation

This paper aims at achieving a simultaneously sparse and low-rank estimator from the semidefinite population covariance matrices. We first benefit from a convex optimization which develops $$\ell _1$$ -norm penalty to encourage the sparsity and nuclear norm to favor the low-rank property. For the proposed estimator, we then prove that with high probability, the Frobenius norm of the estimation rate can be of order $${\fancyscript{O}}\left({\sqrt{(s\log p)/n}}\right) $$ under a mild case, where s and p denote the number of nonzero entries and the dimension of the population covariance, respectively and n notes the sample capacity. Finally, an efficient alternating direction method of multipliers with global convergence is proposed to tackle this problem, and merits of the approach are also illustrated by practicing numerical simulations.