Non-negative Sparse Coding Research Articles

Nonnegative sparse coding induced hashing for image copy detection

Among the existing hashing methods, the Self-taught hashing (STH) is regarded as the state-of-the-art work. However, it still suffers the problem of semantic loss, which mainly comes from the fact that the original optimization objective of in-sample data is NP-hard and therefore is compromised into the combination of Laplacian Eigenmaps (LE) and binarization. Obviously, the shape associated with the embedding of LE is quite dissimilar to that of binary code. As a result, binarization of the LE embedding readily leads to significant semantic loss. To overcome this drawback, we combine the constrained nonnegative sparse coding and the Support Vector Machine (SVM) to propose a new hashing method, called nonnegative sparse coding induced hashing (NSCIH). Here, nonnegative sparse coding is exploited for seeking a better intermediate representation, which can make sure that the binarization can be smoothly conducted. In addition, we build an image copy detection scheme based on the proposed hashing methods. The extensive experiments show that the NSCIH is superior to the state-of-the-art hashing methods. At the same time, this copy detection scheme can be used for performing copy detection over very large image database.

Image classification using Harr-like transformation of local features with coding residuals

Recently, the bag-of-visual-words (BoW) model has been proven very effective for image classification. However, most researchers used local features directly while neglecting their spatial information and correlations. Besides, the encoding of local features causes some information loss which also hinders the final image classification performance. To tackle these problems, in this paper, we proposed a novel image classification method using Harr-like transformation of local features with additional consideration of coding residuals. We apply Harr-like transformation on local features to combine the spatial information as well as the correlations of local features. These Harr-like transformed local features are then encoded using non-negative sparse coding. We jointly consider the coding parameters and the coding residuals as the local representation in order to reduce the information loss during the local feature encoding process. Experiments on several public datasets demonstrate the effectiveness of the proposed method.

Online learning and generalization of parts-based image representations by non-negative sparse autoencoders

We present an efficient online learning scheme for non-negative sparse coding in autoencoder neural networks. It comprises a novel synaptic decay rule that ensures non-negative weights in combination with an intrinsic self-adaptation rule that optimizes sparseness of the non-negative encoding. We show that non-negativity constrains the space of solutions such that overfitting is prevented and very similar encodings are found irrespective of the network initialization and size. We benchmark the novel method on real-world datasets of handwritten digits and faces. The autoencoder yields higher sparseness and lower reconstruction errors than related offline algorithms based on matrix factorization. It generalizes to new inputs both accurately and without costly computations, which is fundamentally different from the classical matrix factorization approaches.

Palmprint recognition method based on localized non-negative sparse coding

To more effectively extract localized features of images,on the basis of the traditional Non-negative Sparse Coding(Hoyer NNSC) algorithm,a novel localized NNSC(LNNSC) algorithm with sparse constraint was proposed.This algorithm considered the sparse measure constraint of feature basis vectors and the maximized representativeness of features,and could obtain the strengthened localized image features.At the same time,this algorithm utilized the Laplace density model as the feature coefficients sparse punitive function to ensure an image's sparse structure.Furthermore,on the basis of feature extraction,by utilizing the Radial Basis Probabilistic Neural Networks(RBPNN),the palmprint recognition task could be implemented automatically.Compared with the palmprint recognition methods of Non-negative Matrix Factorization(NMF),Local NMF(LNMF) and Hoyer-NNSC,simulation results show that our method proposed here displays feasibility and practicality in palmprint recognition.

Denoising via truncated sparse decomposition

This paper proposes a denoising algorithm called truncated sparse decomposition (TSD) algorithm, which combines the advantage of the sparse decomposition with that of the minimum energy model truncation operation. Experimental results on two real chaotic signals show that the TSD algorithm outperforms the recently reported denoising algorithms—non-negative sparse coding and singular value decomposition based method.

A hierarchical framework for spectro-temporal feature extraction

In this paper we present a hierarchical framework for the extraction of spectro-temporal acoustic features. The design of the features targets higher robustness in dynamic environments. Motivated by the large gap between human and machine performance in such conditions we take inspirations from the organization of the mammalian auditory cortex in the design of our features. This includes the joint processing of spectral and temporal information, the organization in hierarchical layers, competition between coequal features, the use of high-dimensional sparse feature spaces, and the learning of the underlying receptive fields in a data-driven manner. Due to these properties we termed the features as hierarchical spectro-temporal (HIST) features. For the learning of the features at the first layer we use Independent Component Analysis (ICA). At the second layer of our feature hierarchy we apply Non-Negative Sparse Coding (NNSC) to obtain features spanning a larger frequency and time region. We investigate the contribution of the different subparts of this feature extraction process to the overall performance. This includes an analysis of the benefits of the hierarchical processing, the comparison of different feature extraction methods on the first layer, the evaluation of the feature competition, and the investigation of the influence of different receptive field sizes on the second layer. Additionally, we compare our features to MFCC and RASTA-PLP features in a continuous digit recognition task in noise. On a wideband dataset we constructed ourselves based on the Aurora-2 task, as well as on the actual Aurora-2 database. We show that a combination of the proposed HIST features and RASTA-PLP features yields significant improvements and that the proposed features carry complementary information to RASTA-PLP and MFCC features.

Iterative Reweighted $\ell_1$ and $\ell_2$ Methods for Finding Sparse Solutions

A variety of practical methods have recently been introduced for finding maximally sparse representations from overcomplete dictionaries, a central computational task in compressive sensing applications as well as numerous others. Many of the underlying algorithms rely on iterative reweighting schemes that produce more focal estimates as optimization progresses. Two such variants are iterative reweighted l <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> and l <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> minimization; however, some properties related to convergence and sparse estimation, as well as possible generalizations, are still not clearly understood or fully exploited. In this paper, we make the distinction between <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">separable</i> and <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">non-separable</i> iterative reweighting algorithms. The vast majority of existing methods are separable, meaning the weighting of a given coefficient at each iteration is only a function of that individual coefficient from the previous iteration (as opposed to dependency on all coefficients). We examine two such separable reweighting schemes: an l <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> method from Chartrand and Yin (2008) and an l <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> approach from Cande's (2008), elaborating on convergence results and explicit connections between them. We then explore an interesting non-separable alternative that can be implemented via either l <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> or l <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> reweighting and maintains several desirable properties relevant to sparse recovery despite a highly non-convex underlying cost function. For example, in the context of canonical sparse estimation problems, we prove uniform superiority of this method over the minimum l <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> solution in that, 1) it can never do worse when implemented with reweighted l <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> , and 2) for any dictionary and sparsity profile, there will always exist cases where it does better. These results challenge the prevailing reliance on strictly convex (and separable) penalty functions for finding sparse solutions. We then derive a new non-separable variant with similar properties that exhibits further performance improvements in empirical tests. Finally, we address natural extensions to group sparsity problems and non-negative sparse coding.

SENSC: a Stable and Efficient Algorithm for Nonnegative Sparse Coding

摘要: 非负稀疏编码(Nonnegative sparse coding, NSC)已成功应用在很多领域的研究中. 目前使用的NSC算法通过梯度投影法和基于辅助函数的乘性更新法相结合来实现, 其性能受迭代步长的影响很大, 且效率较低. 为增强NSC的可应用性, 本文通过对一组凸超抛物面函数做交替最小化来实现NSC, 并依据凸超抛物面特性、点到非负数集合的投影规则以及点到原点处单位超球的投影规则构造了一个无用户定义优化参数的稳定高效的NSC算法---SENSC. 从数学角度, 文中推断了SENSC比现有算法高效且它的解优于当前算法的解, 证明了它的稳定性和收敛性. 实验验证了上述理论推断的正确, 说明了SENSC调节编码稀疏性的能力比已有算法更强. 关键词: 非负稀疏编码 / 非负矩阵分解 / 稀疏编码 / 超抛物面

Non-negative sparse coding shrinkage for image denoising using normal inverse Gaussian density model

This paper proposes a novel image denoising technique based on the normal inverse Gaussian (NIG) density model and the extended non-negative sparse coding (NNSC). The NIG density function, which is fully specified by four real-valued parameters, represents a class of flexible closed form distribution and is quite suitable for modeling sparse data. By choosing appropriate parameters, one can describe a variety of data distributions. In this paper, we demonstrate that the NIG density function provides good fit to non-negative sparse data. With the aid of NIG-based maximum a posteriori estimator (MAP), significant denoising can be achieved for non-negatively and sparsely coded images corrupted with additive Gaussian noise. It is also shown that the proposed NNSC shrinkage technique is adaptive to various distribution properties of natural image data. Experimental results confirm the effectiveness of the proposed NIG based NNSC shrinkage method for image denoising. The comparison with other denoising methods are also made and it is shown that the proposed method produces the best denoising effect.

Learning Sparse Overcomplete Codes for Images

Images can be coded accurately using a sparse set of vectors from a learned overcomplete dictionary, with potential applications in image compression and feature selection for pattern recognition. We present a survey of algorithms that perform dictionary learning and sparse coding and make three contributions. First, we compare our overcomplete dictionary learning algorithm (FOCUSS-CNDL) with overcomplete independent component analysis (ICA). Second, noting that once a dictionary has been learned in a given domain the problem becomes one of choosing the vectors to form an accurate, sparse representation, we compare a recently developed algorithm (sparse Bayesian learning with adjustable variance Gaussians, SBL-AVG) to well known methods of subset selection: matching pursuit and FOCUSS. Third, noting that in some cases it may be necessary to find a non-negative sparse coding, we present a modified version of the FOCUSS algorithm that can find such non-negative codings. Efficient parallel implementations in VLSI could make these algorithms more practical for many applications.

Face recognition using localized features based on non-negative sparse coding

Neural networks in the visual system may be performing sparse coding of learnt local features that are qualitatively very similar to the receptive fields of simple cells in the primary visual cortex, V1. In conventional sparse coding, the data are described as a combination of elementary features involving both additive and subtractive components. However, the fact that features can ‘cancel each other out’ using subtraction is contrary to the intuitive notion of combining parts to form a whole. Thus, it has recently been argued forcefully for completely non-negative representations. This paper presents Non-Negative Sparse Coding (NNSC) applied to the learning of facial features for face recognition and a comparison is made with the other part-based techniques, Non-negative Matrix Factorization (NMF) and Local-Non-negative Matrix Factorization (LNMF). The NNSC approach has been tested on the Aleix–Robert (AR), the Face Recognition Technology (FERET), the Yale B, and the Cambridge ORL databases, respectively. In doing so, we have compared and evaluated the proposed NNSC face recognition technique under varying expressions, varying illumination, occlusion with sunglasses, occlusion with scarf, and varying pose. Tests were performed with different distance metrics such as the L1-metric, L2-metric, and Normalized Cross-Correlation (NCC). All these experiments involved a large range of basis dimensions. In general, NNSC was found to be the best approach of the three part-based methods, although it must be observed that the best distance measure was not consistent for the different experiments.

Learning Sparse Overcomplete Codes for Images

Images can be coded accurately using a sparse set of vectors from a learned overcomplete dictionary, with potential applications in image compression and feature selection for pattern recognition. We present a survey of algorithms that perform dictionary learning and sparse coding and make three contributions. First, we compare our overcomplete dictionary learning algorithm (FOCUSS-CNDL) with overcomplete independent component analysis (ICA). Second, noting that once a dictionary has been learned in a given domain the problem becomes one of choosing the vectors to form an accurate, sparse representation, we compare a recently developed algorithm (sparse Bayesian learning with adjustable variance Gaussians, SBL-AVG) to well known methods of subset selection: matching pursuit and FOCUSS. Third, noting that in some cases it may be necessary to find a non-negative sparse coding, we present a modified version of the FOCUSS algorithm that can find such non-negative codings. Efficient parallel implementations in VLSI could make these algorithms more practical for many applications.

A novel shrinkage technique based on the normal inverse Gaussian density model

This paper proposes a novel image denoising technique based on the normal inverse Gaussian (NIG) density model using an extended non-negative sparse coding (NNSC) algorithm proposed by us. This algorithm can converge to feature basis vectors, which behave in the locality and orientation in spatial and frequency domain. Here, we demonstrate that the NIG density provides a very good fitness to the non-negative sparse data. In the denoising process, by exploiting a NIG-based maximum a posteriori estimator (MAP) of an image corrupted by additive Gaussian noise, the noise can be reduced successfully. This shrinkage technique, also referred to as the NNSC shrinkage technique, is self-adaptive to the statistical properties of image data. This denoising method is evaluated by values of the normalized signal to noise rate (SNR). Experimental results show that the NNSC shrinkage approach is indeed efficient and effective in denoising. Otherwise, we also compare the effectiveness of the NNSC shrinkage method with methods of standard sparse coding shrinkage, wavelet-based shrinkage and the Wiener filter. The simulation results show that our method outperforms the three kinds of denoising approaches mentioned above.

Noise removal using a novel non-negative sparse coding shrinkage technique

A novel and successful method for denoising natural images using an extended non-negative sparse coding (NNSC) shrinkage technique is proposed. The main idea is to utilize the selected shrinkage function to the non-negative sparse components to remove noises hidden in an image. This method is evaluated by values of the normalized mean squared error (MSE) and signal to noise ratio (SNR). Compared with other denoising methods, the simulation results show that the NNSC shrinkage technique is indeed effective and efficient.

Statistical model of natural stimuli predicts edge-like pooling of spatial frequency channels in V2

BackgroundIt has been shown that the classical receptive fields of simple and complex cells in the primary visual cortex emerge from the statistical properties of natural images by forcing the cell responses to be maximally sparse or independent. We investigate how to learn features beyond the primary visual cortex from the statistical properties of modelled complex-cell outputs. In previous work, we showed that a new model, non-negative sparse coding, led to the emergence of features which code for contours of a given spatial frequency band.ResultsWe applied ordinary independent component analysis to modelled outputs of complex cells that span different frequency bands. The analysis led to the emergence of features which pool spatially coherent across-frequency activity in the modelled primary visual cortex. Thus, the statistically optimal way of processing complex-cell outputs abandons separate frequency channels, while preserving and even enhancing orientation tuning and spatial localization. As a technical aside, we found that the non-negativity constraint is not necessary: ordinary independent component analysis produces essentially the same results as our previous work.ConclusionWe propose that the pooling that emerges allows the features to code for realistic low-level image features related to step edges. Further, the results prove the viability of statistical modelling of natural images as a framework that produces quantitative predictions of visual processing.

Nonnegative Matrix Factorization for Rapid Recovery of Constituent Spectra in Magnetic Resonance Chemical Shift Imaging of the Brain

We present an algorithm for blindly recovering constituent source spectra from magnetic resonance (MR) chemical shift imaging (CSI) of the human brain. The algorithm, which we call constrained nonnegative matrix factorization (cNMF), does not enforce independence or sparsity, instead only requiring the source and mixing matrices to be nonnegative. It is based on the nonnegative matrix factorization (NMF) algorithm, extending it to include a constraint on the positivity of the amplitudes of the recovered spectra. This constraint enables recovery of physically meaningful spectra even in the presence of noise that causes a significant number of the observation amplitudes to be negative. We demonstrate and characterize the algorithm's performance using 31P volumetric brain data, comparing the results with two different blind source separation methods: Bayesian spectral decomposition (BSD) and nonnegative sparse coding (NNSC). We then incorporate the cNMF algorithm into a hierarchical decomposition framework, showing that it can be used to recover tissue-specific spectra given a processing hierarchy that proceeds coarse-to-fine. We demonstrate the hierarchical procedure on 1H brain data and conclude that the computational efficiency of the algorithm makes it well-suited for use in diagnostic work-up.

Modeling receptive fields with non-negative sparse coding

An important approach in visual neuroscience considers how the processing of the early visual system is dependent on the statistics of the natural environment. A particularly influential model in this respect has been sparse coding. In this paper we argue for a non-negative variant of the model. This is based partly on neurophysiological grounds and partly on the intuitive understanding of parts-based representations. We discuss the logic behind our reasoning and show experiments on natural images demonstrating the usefulness of the new model.