Linear Dictionary Research Articles

Randomized algorithms for large-scale dictionary learning

Dictionary learning is an important sparse representation algorithm which has been widely used in machine learning and artificial intelligence. However, for massive data in the big data era, classical dictionary learning algorithms are computationally expensive and even can be infeasible. To overcome this difficulty, we propose new dictionary learning methods based on randomized algorithms. The contributions of this work are as follows. First, we find that dictionary matrix is often numerically low-rank. Based on this property, we apply randomized singular value decomposition (RSVD) to the dictionary matrix, and propose a randomized algorithm for linear dictionary learning. Compared with the classical K-SVD algorithm, an advantage is that one can update all the elements of the dictionary matrix simultaneously. Second, to the best of our knowledge, there are few theoretical results on why one can solve the involved matrix computation problems inexactly in dictionary learning. To fill-in this gap, we show the rationality of this randomized algorithm with inexact solving, from a matrix perturbation analysis point of view. Third, based on the numerically low-rank property and Nyström approximation of the kernel matrix, we propose a randomized kernel dictionary learning algorithm, and establish the distance between the exact solution and the computed solution, to show the effectiveness of the proposed randomized kernel dictionary learning algorithm. Fourth, we propose an efficient scheme for the testing stage in kernel dictionary learning. By using this strategy, there is no need to form nor store kernel matrices explicitly both in the training and the testing stages. Comprehensive numerical experiments are performed on some real-world data sets. Numerical results demonstrate the rationality of our strategies, and show that the proposed algorithms are much efficient than some state-of-the-art dictionary learning algorithms. The MATLAB codes of the proposed algorithms are publicly available from https://github.com/Jiali-yang/RALDL_RAKDL.

Sparse linear dictionary reconstruction for removing microphonic noise from nuclear spectrometry measurements

The deployment of precise and sensitive nuclear spectroscopy instrumentation, using high purity germanium (HPGe), requires sustained cryogenic temperatures that can be achieved with cryocooler heat pumps. However, these pumps introduce microphonic noise that broaden the peaks of the spectra. This work demonstrates the efficacy of L1-norm regularization for the purpose of removing microphonic noise generated by the cryocooler, while still recovering the correct detector pulse amplitudes. Data are first collected from an He3 neutron detector with an attached cryocooler which enables the controlled addition of microphonic noise. A dictionary is then created by combining two relatively incoherent matrices, the first is the Fourier basis and the second is a matrix created from normalized and shifted pulses. By applying this dictionary to background noise, the regularization parameter λ is obtained. Using an L1-regularization least squares solver, microphonic noise can be effectively separated, therefore reducing the variance of the detector pulses while preserving the correct amplitude.

Convex Nonparanormal Regression

Quantifying uncertainty in predictions or, more generally, estimating the posterior conditional distribution, is a core challenge in machine learning and statistics. We introduce Convex Nonparanormal Regression (CNR), a conditional nonparanormal approach for coping with this task. CNR involves a convex optimization of a posterior defined via a rich dictionary of pre-defined non linear transformations on Gaussians. It can fit an arbitrary conditional distribution, including multimodal and non-symmetric posteriors. For the special but powerful case of a piecewise linear dictionary, we provide a closed form of the posterior mean which can be used for point-wise predictions. Finally, we demonstrate the advantages of CNR over classical competitors using synthetic and real world data.

A Hierarchical Discriminative Sparse Representation Classifier for EEG Signal Detection.

Classification of electroencephalogram (EEG) signal data plays a vital role in epilepsy detection. Recently sparse representation-based classification (SRC) methods have achieved the good performance in EEG signal automatic detection, by which the EEG signals are sparsely represented using a few active coefficients in the dictionary and classified according to the reconstruction criteria. However, most of SRC learn a linear dictionary for encoding, and cannot extract enough information and nonlinear relationship of data for classification. To solve this problem, a hierarchical discriminative sparse representation classification model (called HD-SRC) for EEG signal detection is proposed. Based on the framework of neural network, HD-SRC learns the hierarchical nonlinear transformation and maps the signal data into the nonlinear transformed space. Through incorporating this idea into label consistent K singular value decomposition (LC-KSVD) at the top layer of neural network, HD-SRC seeks discriminative representation together with dictionary, while minimizing errors of classification, reconstruction and discriminative sparse-code for pattern classification. By learning the hierarchical feature mapping and discriminative dictionary simultaneously, more discriminative information of data can be exploited. In the experiment the proposed model is evaluated on the Bonn EEG database, and the results show it obtains satisfactory classification performance in multiple EEG signal detection tasks.

Block-wise weighted sparse representation-based classification

We present a block-wise weighted sparse representation-based classification (BW-SRC) method, an extension of sparse representation-based classification (SRC), useful when the input features can be treated in a block-wise manner. We apply the method, which extracts features (coefficients) embedded in a supervised scheme on three known datasets. We show that, depending on the blocks of features used, our method outperforms linear dictionary learning and sparse coding methods without learning a dictionary. As an additional advantage of our method, the input feature blocks are automatically given an interpretation based on their importance for their ease of representation.

Nonlinear dictionary learning with application to image classification

In this paper, we propose a new nonlinear dictionary learning (NDL) method and apply it to image classification. While a variety of dictionary learning algorithms have been proposed in recent years, most of them learn only a linear dictionary for feature learning and encoding, which cannot exploit the nonlinear relationship of image samples for feature extraction. Even though kernel-based dictionary learning methods can address this limitation, they still suffer from the scalability problem. Unlike existing dictionary learning methods, our NDL employs a feed-forward neural network to seek hierarchical feature projection matrices and dictionary simultaneously, so that the nonlinear structure of samples can be well exploited for feature learning and encoding. To better exploit the discriminative information, we extend the NDL into supervised NDL (SNDL) by learning a class-specific dictionary with the labels of training samples. Experimental results on four image datasets show the effectiveness of the proposed methods.

The influence of parameter initialization on the training time and accuracy of a nonlinear regression model

In order to construct a nonlinear regression model we have to accurately (in some sense) initialize parameters of the model. In this work we performed comparison of several widely used methods and several newly developed approached for initialization of parameters of a regression model, represented as a decomposition in a linear dictionary of some parametric functions (sigmoids). We proposed a general deterministic approach for initialization, providing repeatability of results, reduction of a learning time and in some cases increase of a regression model accuracy; we developed two new algorithms (based on a piecewise-linear approximation and based on local properties of approximable dependency) in the framework of the proposed approach; we developed randomized initialization algorithm (spherical initialization) for effective approximation of high-dimensional dependencies; we improved the classical initialization method SCAWI (by locating centers of sigmoids in sample points), providing a regression model accuracy improvement on specific classes of dependencies (smooth functions and discontinuous functions with a number of local peculiarities in an input domain) when using RProp algorithm for learning; we performed comparison of classical and newly proposed initialization methods and highlighted the most efficient ones.

ACCELERATING DYNAMIC MAGNETIC RESONANCE IMAGING BY NONLINEAR SPARSE CODING.

Although being high-dimensional, dynamic magnetic resonance images usually lie on low-dimensional manifolds. Nonlinear models have been shown to capture well that latent low-dimensional nature of data, and can thus lead to improvements in the quality of constrained recovery algorithms. This paper advocates a novel reconstruction algorithm for dynamic magnetic resonance imaging (dMRI) based on nonlinear dictionary learned from low-spatial but high-temporal resolution images. The nonlinear dictionary is initially learned using kernel dictionary learning, and the proposed algorithm subsequently alternates between sparsity enforcement in the feature space and the data-consistency constraint in the original input space. Extensive numerical tests demonstrate that the proposed scheme is superior to popular methods that use linear dictionaries learned from the same set of training data.

Linearized Kernel Dictionary Learning

In this paper we present a new approach of incorporating kernels into dictionary learning. The kernel K-SVD algorithm (KKSVD), which has been introduced recently, shows an improvement in classification performance, with relation to its linear counterpart K-SVD. However, this algorithm requires the storage and handling of a very large kernel matrix, which leads to high computational cost, while also limiting its use to setups with small number of training examples. We address these problems by combining two ideas: first we approximate the kernel matrix using a cleverly sampled subset of its columns using the Nystr\"{o}m method; secondly, as we wish to avoid using this matrix altogether, we decompose it by SVD to form new "virtual samples," on which any linear dictionary learning can be employed. Our method, termed "Linearized Kernel Dictionary Learning" (LKDL) can be seamlessly applied as a pre-processing stage on top of any efficient off-the-shelf dictionary learning scheme, effectively "kernelizing" it. We demonstrate the effectiveness of our method on several tasks of both supervised and unsupervised classification and show the efficiency of the proposed scheme, its easy integration and performance boosting properties.

Abundance Estimation for Bilinear Mixture Models via Joint Sparse and Low-Rank Representation

Sparsity-based unmixing algorithms, exploiting the sparseness property of the abundances, have recently been proposed with promising performances. However, these algorithms are developed for the linear mixture model (LMM), which cannot effectively handle the nonlinear effects. In this paper, we extend the current sparse regression methods for the LMM to bilinear mixture models (BMMs), where the BMMs introduce additional bilinear terms in the LMM in order to model second-order photon scattering effects. To solve the abundance estimation problem for the BMMs, we propose to perform a sparsity-based abundance estimation by using two dictionaries: a linear dictionary containing all the pure endmembers and a bilinear dictionary consisting of all the possible second-order endmember interaction components. Then, the abundance values can be estimated from the sparse codes associated with the linear dictionary. Moreover, to exploit the spatial data structure where the adjacent pixels are usually homogeneous and are often mixtures of the same materials, we first employ the joint-sparsity (row-sparsity) model to enforce structured sparsity on the abundance coefficients. However, the joint-sparsity model is often a strict assumption, which might cause some aliasing artifacts for the pixels that lie on the boundaries of different materials. To deal with this problem, the low-rank-representation model, which seeks the lowest rank representation of the data, is further introduced to better capture the spatial data structure. Our simulation results demonstrate that the proposed algorithms provide much enhanced performance compared with state-of-the-art algorithms.

Face illumination compensation dictionary

Face images captured under distinct lighting conditions have totally different overall appearances, which greatly degrade the recognition accuracy. In this paper, an illumination compensation strategy is worked out to assist linear representation based face recognition. In the past few years, linear representation based face recognition approaches such as SRC and CRC_RLS attract great attention, but their effectiveness greatly depends on a large number of training samples, which seriously restricts their application values. We will illustrate that face illumination distinction could be compensated just through a general linear dictionary, and after enrolling our illumination compensation strategy, even there is only single gallery image for each subject, linear representation recognition approaches can still be relatively robust to probe illumination variance. The proposed strategy is experimented on the Extended Yale B and CMU PIE database.