Sparse Weight Vector Research Articles

Penalized Mallow’s model averaging

This article proposes penalized Mallow’s model averaging (pMMA) in the linear regression framework given non nested candidate models. Compared to the MMA, additional constraints are imposed on model weights. We introduce a general framework and allow for non convex constraints such as SCAD, MCP, and TLP. We establish the asymptotic optimality of our proposed penalized MMA (pMMA) estimator and show that the pMMA can achieve a higher sparsity level than the classic MMA. A coordinate-wise descent algorithm has been developed to compute the pMMA estimator efficiently. We conduct simulation and empirical studies to show that our pMMA estimator produces a more sparse weight vector than the MMA, but with better out-of-sample performance.

Complex sparse Bayesian learning for guided wave dispersion curve estimation in plate-like structures

This paper proposes a new dispersion curve estimation method that employs the complex sparse Bayesian learning (CSBL). It is well accepted that guided wave packets are distorted because of the differences in propagation velocities at different frequencies; thus, the preceding velocity–frequency curve estimation is beneficial for wave packet recovery, feature recognition and defect localization. Conventional dispersion curve estimation methods, such as two-dimensional Fourier transform and multiple signal classification, are suitable for array signal and are restricted by the transducer aperture, leading to a small application scope. According to the frequency–response model of the guided wave, for each frequency, the responses obtained by the transducers can be sparsely represented based on an overcomplete dictionary matrix constructed using multiple discretized wavenumbers and known distances. A CSBL algorithm was developed to infer the posterior probability density function of the weight vector in the sparse representation. The non-zero elements in the sparse weight vector mean that the corresponding dictionary atoms indicated wavenumbers are contained in the frequency response, and the velocity–frequency curve can be finally derived from the wavenumber–frequency curve. The proposed CSBL method has a satisfactory capability to solve this sparse representation of the complex frequency response because a hierarchical form of the Laplace prior is employed to achieve a high degree of sparsity of the weight vector. This method effectively incorporates the real and imaginary parts of the complex frequency response by employing the same hyperparameter to integrate the known information. This method requires only a few randomly placed transducers; thus, it has a wide range of applications. The effectiveness of the proposed approach was validated using multiple guided-wave signals obtained through numerical simulations and an experimental study on a plate structure.

Sparse oblique decision trees: a tool to understand and manipulate neural net features

The widespread deployment of deep nets in practical applications has lead to a growing desire to understand how and why such black-box methods perform prediction. Much work has focused on understanding what part of the input pattern (an image, say) is responsible for a particular class being predicted, and how the input may be manipulated to predict a different class. We focus instead on understanding which of the internal features computed by the neural net are responsible for a particular class. We achieve this by mimicking part of the neural net with an oblique decision tree having sparse weight vectors at the decision nodes. Using the recently proposed Tree Alternating Optimization (TAO) algorithm, we are able to learn trees that are both highly accurate and interpretable. Such trees can faithfully mimic the part of the neural net they replaced, and hence they can provide insights into the deep net black box. Further, we show we can easily manipulate the neural net features in order to make the net predict, or not predict, a given class, thus showing that it is possible to carry out adversarial attacks at the level of the features. These insights and manipulations apply globally to the entire training and test set, not just at a local (single-instance) level. We demonstrate this robustly in the MNIST and ImageNet datasets with LeNet5 and VGG networks.

Adaptive weighted robust iterative closest point

The Iterative Closest Point (ICP) algorithm is one of the most important methods for rigid registration between point sets. However, its performance begins to degenerate with the point data are overly contaminated by noise, outliers, and missing data. In this paper, we propose an adaptive weighted robust ICP method (AW-RICP). A sparse weight vector can be automatically learned by adaptive neighbors assigning process. The sparseness of the weight vector can be achieved by fitting the number of selected samples. The adaptive weight vector leads to the robustness of AW-RICP by eliminating the negative effect caused by point cloud pairs with largest registration errors. Further, the retained point pairs are assigned suitable weights to suppress the noises and outliers. The new error metric can effectively improve the robustness of ICP method. Additionally, we propose an efficient iterative solution to optimize our problem. Experiments on both synthetic and real data sets show that the proposed method can achieve superior performance with other state-of-the-art methods.

ℓ0-based sparse canonical correlation analysis with application to cross-language document retrieval

Most of existing sparse CCA algorithms compute sparse weight vectors by minimizing the ℓ1 norm, which imposes essential difficulty for the analysis of the solution. Different from existing ones, this paper develops a novel sparse CCA algorithm by ℓ0 penalty. The resulting ℓ0 minimization problem is solved by means of residual, which has one merit that no regularization parameter or shrinkage parameter needs to be tuned. We also provide consistency analysis of the proposed method using the concept of Restricted Isometry Property (RIP) condition, while no theoretical guarantee was given for most of existing sparse CCA methods. Sparsity bound of the CCA solutions is also studied. Experimental results on both simulated dataset and real-world datasets in cross-language document retrieval task demonstrate the effectiveness and competitiveness of the proposed algorithm, when compared with several state-of-the-art sparse CCA methods.

Robust Matching Pursuit Extreme Learning Machines

Extreme learning machine (ELM) is a popular learning algorithm for single hidden layer feedforward networks (SLFNs). It was originally proposed with the inspiration from biological learning and has attracted massive attentions due to its adaptability to various tasks with a fast learning ability and efficient computation cost. As an effective sparse representation method, orthogonal matching pursuit (OMP) method can be embedded into ELM to overcome the singularity problem and improve the stability. Usually OMP recovers a sparse vector by minimizing a least squares (LS) loss, which is efficient for Gaussian distributed data, but may suffer performance deterioration in presence of non-Gaussian data. To address this problem, a robust matching pursuit method based on a novel kernel risk-sensitive loss (in short KRSLMP) is first proposed in this paper. The KRSLMP is then applied to ELM to solve the sparse output weight vector, and the new method named the KRSLMP-ELM is developed for SLFN learning. Experimental results on synthetic and real-world data sets confirm the effectiveness and superiority of the proposed method.

Rule Extraction from Decision Trees Ensembles: New Algorithms Based on Heuristic Search and Sparse Group Lasso Methods

Decision trees are examples of easily interpretable models whose predictive accuracy is normally low. In comparison, decision tree ensembles (DTEs) such as random forest (RF) exhibit high predictive accuracy while being regarded as black-box models. We propose three new rule extraction algorithms from DTEs. The RF[Formula: see text]DHC method, a hill climbing method with downhill moves (DHC), is used to search for a rule set that decreases the number of rules dramatically. In the RF[Formula: see text]SGL and RF[Formula: see text]MSGL methods, the sparse group lasso (SGL) method, and the multiclass SGL (MSGL) method are employed respectively to find a sparse weight vector corresponding to the rules generated by RF. Experimental results with 24 data sets show that the proposed methods outperform similar state-of-the-art methods, in terms of human comprehensibility, by greatly reducing the number of rules and limiting the number of antecedents in the retained rules, while preserving the same level of accuracy.

Sparse Spatio-Spectral LapSVM With Semisupervised Kernel Propagation for Hyperspectral Image Classification

The information contained in the hyperspectral data allows the characterization, identification, and classification of the land covers with improved accuracy and robustness. Many methods have been explored in the hyperspectral image classification (HIC). Among these methods, spatio-spectral Laplacian support vector machine (SS-LapSVM) combines the spatial and spectral information on both the labeled and unlabeled samples together through the weight sum of a spectral regularization term and a spatial regularization term. Thus, it can achieve accurate classification with very few labeled samples and has proved to be effective in HIC. In this paper, a sparse SS-LapSVM with semisupervised Kernel Propagation (S3LapSVM-KP) is constructed to achieve higher accuracy and efficiency in HIC. First, data-driven semisupervised KP is proposed to carefully learn a kernel matrix from a small number of labeled pixels. Furthermore, a one-step sparse pruning algorithm is advanced by solving sparse weight vectors associated with network nodes in SS-LapSVM. By combining semisupervised KP with sparse coding, S3LapSVM-KP can not only automatically determine kernels from data, but also avoid overfitting and reduce computation cost resulted from the nonsparse topology of SS-LapSVM. The performance of S3LapSVM-KP is evaluated on several real hyperspectral datasets, and the results show that S3LapSVM-KP can achieve accurate and rapid classification with very few labeled data.

Joint Sparsity-Based Range-Angle-Dependent Beampattern Synthesis for Frequency Diverse Array

For frequency diverse array (FDA) range-angle-dependent beampattern synthesis, the objective is to obtain the desired beampattern performance using fewer antenna elements or smaller aperture. This paper proposes a FDA single-beam pattern synthesis by joint $\ell _{1}$ -norm minimization and convex optimization in the beginning. A virtual uniform FDA array with small element spacing and corresponding frequency increment is created first. Then, we formulate the array pattern synthesis (APS) problem as finding a joint sparse weight vector, which can be obtained by solving a convex optimization problem with the joint sparse constraint between angle dimension weight vector and range dimension weight vector, where some small entries of the vector can be regarded as zeros without significantly changing the array pattern performance. The antenna elements corresponding to the mapping positions of nonzero values of the joint sparse weight vector are placed to form a non-uniform FDA. Finally, convex optimization is further conducted to obtain the optimal weight vector of the non-uniform FDA. Besides, we expand the idea to FDA multi-beam pattern synthesis based on the convex optimization problem with joint sparse constraint between angle dimension weight matrix and range dimension weight matrix. Numerical examples are provided to verify the efficiency of achieving the desired radiation pattern with the fewer antenna elements, and better APS performance for given array elements.

Sparse multiple maximum scatter difference for dimensionality reduction

Sparse subspace learning has drawn more and more attentions recently. However, most of the present sparse subspace learning methods neglect the sparse reconstructive relationship between classes that the given samples belong to, so the extracted features are not so discriminative for classification. To address this shorting, a new sparse subspace learning algorithm called sparse multiple maximum scatter difference (SMMSD) is proposed for dimensionality reduction in this paper. SMMSD seeks a low-dimensional subspace in which the within-class sparse reconstructive residual is minimized, and meanwhile the between-class reconstructive residual is maximized based on multiple maximum scatter difference (MMSD) criterion. Hence, SMMSD can capture more potential discriminative information for classification, and naturally avoid the singularity problem. Additionally, SMMSD can be performed very simply by solving the l2-norm related optimization problem to compute the sparse reconstructive weight vector. Extensive experimental results on three popular face databases (FERET, Extended Yale B and AR) and PolyU finger-knuckle-print database demonstrate the effectiveness and efficiency of the proposed SMMSD method.

Variational sparse Bayesian learning for optimal design of interleaved linear arrays with region constraint

An innovative approach based on the variational sparse Bayesian learning method is presented for the optimal design of interleaved linear arrays with a fewer number of subarray elements. In this study, with a consideration of practical array assembling, the acceptable minimum element spacing can be freely adjusted and used to construct the constrained regions for subarray interleaving. The design of interleaved sparse arrays is then recast as a sparse constrained optimisation problem under region constraint, and a fast relevance vector machine is utilised to estimate the sparse weight vectors, with which the number of the selected antenna elements as well as their positions and excitations can be obtained. A set of representative numerical simulations validate the high accuracy and computational efficiency of the proposed method.

Diffusion Sign Subband Adaptive Filtering Algorithm with Enlarged Cooperation and Its Variant

The recently proposed diffusion sign subband adaptive filtering (DSSAF) algorithm is more robust than most of mean-square error minimization criterion-based diffusion distributed estimation algorithms in an impulsive interference environment. To enhance its convergence rate and steady-state misalignment, this paper proposes a DSSAF algorithm with enlarged cooperation (DSSAF-EC). The DSSAF-EC algorithm exchanges not only the weight information but also measurements within individual neighborhoods. Moreover, a variant of the DSSAF-EC algorithm, called the proportionate DSSAF-EC (PDSSAF-EC) algorithm, is presented. It incorporates an adaptive gain matrix into the DSSAF-EC algorithm to proportionately adapt the weight vectors of agents. Simulation results verify that both the DSSAF-EC and PDSSAF-EC algorithms are robust against impulsive interference and that the PDSSAF-EC algorithm can obtain faster convergence rate than the DSSAF-EC algorithm in estimating a sparse unknown weight vector.

A multiple hold-out framework for Sparse Partial Least Squares

BackgroundSupervised classification machine learning algorithms may have limitations when studying brain diseases with heterogeneous populations, as the labels might be unreliable. More exploratory approaches, such as Sparse Partial Least Squares (SPLS), may provide insights into the brain's mechanisms by finding relationships between neuroimaging and clinical/demographic data. The identification of these relationships has the potential to improve the current understanding of disease mechanisms, refine clinical assessment tools, and stratify patients. SPLS finds multivariate associative effects in the data by computing pairs of sparse weight vectors, where each pair is used to remove its corresponding associative effect from the data by matrix deflation, before computing additional pairs. New methodWe propose a novel SPLS framework which selects the adequate number of voxels and clinical variables to describe each associative effect, and tests their reliability by fitting the model to different splits of the data. As a proof of concept, the approach was applied to find associations between grey matter probability maps and individual items of the Mini-Mental State Examination (MMSE) in a clinical sample with various degrees of dementia. ResultsThe framework found two statistically significant associative effects between subsets of brain voxels and subsets of the questions/tasks. Comparison with existing methodsSPLS was compared with its non-sparse version (PLS). The use of projection deflation versus a classical PLS deflation was also tested in both PLS and SPLS. ConclusionsSPLS outperformed PLS, finding statistically significant effects and providing higher correlation values in hold-out data. Moreover, projection deflation provided better results.

Sparse canonical correlation analysis applied to ‐omics studies for integrative analysis and biomarker discovery

AbstractWith the rapid development of new ‐omics measurement methods, there is an increasing interest in studying the correlation structure between two or more data sets. Multivariate methods such as canonical correlation analysis (CCA) have been proposed to analyze the intrinsic correlation relationship by integrating two data sets. However, because of the high dimensionality of data and the relative scarcity of samples, the ordinary CCA is usually faced with variable selection problems and thereby fails to obtain a satisfactory relationship. Here, we explored the potential of sparse CCA (SCCA) to find the correlative components in two sparse views. SCCA aims at finding sparse projection directions to well extract the correlation between two data sets. We applied this method to one simulation data and one real ‐omics data to illustrate the performance of SCCA. The results from two studies show that SCCA could effectively find the correlated patterns between two data sets, which are of high importance for understanding the relationship between two underlying chemical or biological processes. The corresponding variable subsets selected by sparse weight vectors can assist in a better interpretation of the chemical or biological process. The integrative analysis from two views by SCCA helps in improving the discriminative ability of classification models for various ‐omics studies. Copyright © 2015 John Wiley & Sons, Ltd.

Time series prediction using sparse regression ensemble based on $$\ell _2$$ ℓ 2 – $$\ell _1$$ ℓ 1 problem

Sparse regression ensemble (SRE) is to sparsely combine the outputs of multiple learners using a sparse weight vector. This paper deals with SRE based on the $$\ell _2$$ l 2 --- $$\ell _1$$ l 1 problem and applies it to time series prediction problems. The $$\ell _2$$ l 2 --- $$\ell _1$$ l 1 problem consists of $$\ell _2$$ l 2 -norm and $$\ell _1$$ l 1 -norm regularization terms, where the former denotes the total ensemble empirical risk, and the latter represents the ensemble complexity. Thus, the goal is both to minimize the total ensemble training error and control the ensemble complexity. Experiments on real-world data for regression and time series prediction are given.

Finger Vein Recognition with Personalized Feature Selection

Finger veins are a promising biometric pattern for personalized identification in terms of their advantages over existing biometrics. Based on the spatial pyramid representation and the combination of more effective information such as gray, texture and shape, this paper proposes a simple but powerful feature, called Pyramid Histograms of Gray, Texture and Orientation Gradients (PHGTOG). For a finger vein image, PHGTOG can reflect the global spatial layout and local details of gray, texture and shape. To further improve the recognition performance and reduce the computational complexity, we select a personalized subset of features from PHGTOG for each subject by using the sparse weight vector, which is trained by using LASSO and called PFS-PHGTOG. We conduct extensive experiments to demonstrate the promise of the PHGTOG and PFS-PHGTOG, experimental results on our databases show that PHGTOG outperforms the other existing features. Moreover, PFS-PHGTOG can further boost the performance in comparison with PHGTOG.

Non-uniform array pattern synthesis using reweighted ℓ1-norm minimization

This paper presents a new array pattern synthesis algorithm by reweighted ℓ1-norm minimization and convex optimization. A virtual uniform array with much smaller element spacing and a weight matrix are created first; then, we obtain a weight vector by solving a weighted ℓ1-norm minimization convex problem, and update the weight matrix by the weight vector. Some entries of the weight vector are so small and thus negligible without significantly changing the array pattern performance. Based on this notion, the pruned weight vector becomes a sparse weight vector, and a non-uniform array is formed according to the nonzero valued positions of the sparse weight vector. Determining the weight vector, updating the weight matrix and creating a non-uniform array are repeated until the final array synthesis performance is satisfactory. After optimizing the array geometric construction, we obtain the optimal weight vector of the non-uniform array by using convex optimization. One numerical example is presented to show the high efficiency of achieving the desired radiation pattern with the minimum number of antenna elements.

Biomarker discovery using 1-norm regularization for multiclass earthworm microarray gene expression data

Novel biomarkers can be discovered through mining high dimensional microarray datasets using machine learning techniques. Here we propose a novel recursive gene selection method which can handle the multiclass setting effectively and efficiently. The selection is performed iteratively. In each iteration, a linear multiclass classifier is trained using 1-norm regularization, which leads to sparse weight vectors, i.e., many feature weights are exactly zero. Those zero-weight features are eliminated in the next iteration. The empirical results demonstrate that the selected features (genes) have very competitive discriminative power. In addition, the selection process has fast rate of convergence.

Margin distribution based bagging pruning

Bagging is a simple and effective technique for generating an ensemble of classifiers. It is found there are a lot of redundant base classifiers in the original Bagging. We design a pruning approach to bagging for improving its generalization power. The proposed technique introduces the margin distribution based classification loss as the optimization objective and minimizes the loss on training samples, which leads to an optimal margin distribution. Meanwhile, in order to derive a sparse ensemble, l1 regularization is introduced to control the size of ensembles. By this way, we can obtain a sparse weight vector of base classifiers. Then we rank the base classifiers with respect to their weights and combine the base classifiers with large weights. We call this technique MArgin Distribution base Bagging pruning (MAD-Bagging). Simple voting and weighted voting are tried to combine the outputs of selected base classifiers. The performance of this pruned ensemble is evaluated with several UCI benchmark tasks, where base classifiers are trained with SVM, CART, and the nearest neighbor (1NN) rule, respectively. The results show that margin distribution based CART pruned Bagging can significantly improve classification accuracies. However, SVM and 1NN pruned Bagging improve little compared with single classifiers.

Sparse ensembles using weighted combination methods based on linear programming

An ensemble of multiple classifiers is widely considered to be an effective technique for improving accuracy and stability of a single classifier. This paper proposes a framework of sparse ensembles and deals with new linear weighted combination methods for sparse ensembles. Sparse ensemble is to sparsely combine the outputs of multiple classifiers by using a sparse weight vector. When the continuous outputs of multiple classifiers are provided in our methods, the problem of solving sparse weight vector can be formulated as linear programming problems in which the hinge loss or/and the 1-norm regularization are exploited. Both the hinge loss and the 1-norm regularization are techniques inducing sparsity used in machine learning. We only ensemble classifiers with nonzero weight coefficients. In these LP-based methods, the ensemble training error is minimized while the weight vector of ensemble learning is controlled, which can be thought as implementing the structure risk minimization rule and naturally explains good performance of these methods. The promising experimental results over UCI data sets and the radar high-resolution range profile data are presented.