L1-norm Penalty Research Articles

The Support Vector Regression with Adaptive Norms

This study proposes a new method for regression – lp-norm support vector regression (lp SVR). Some classical SVRs minimize the hinge loss function subject to the l2-norm or l1-norm penalty. These methods are non-adaptive since their penalty forms are fixed and pre-determined for any types of data. Our new model is an adaptive learning procedure with lp-norm (0 <p < 1), where the best p is automatically chosen by data. By adjusting the parameter p, lp SVR can not only select relevant features but also improve the regression accuracy. An iterative algorithm is suggested to solve the lp SVR efficiently. Simulations and real data applications support the effectiveness of the proposed procedure.

Improved dynamic MRI reconstruction by exploiting sparsity and rank-deficiency

In this paper we address the problem of dynamic MRI reconstruction from partially sampled K-space data. Our work is motivated by previous studies in this area that proposed exploiting the spatiotemporal correlation of the dynamic MRI sequence by posing the reconstruction problem as a least squares minimization regularized by sparsity and low-rank penalties. Ideally the sparsity and low-rank penalties should be represented by the l0-norm and the rank of a matrix; however both are NP hard penalties. The previous studies used the convex l1-norm as a surrogate for the l0-norm and the non-convex Schatten-q norm (0<q≤1) as a surrogate for the rank of matrix. Following past research in sparse recovery, we know that non-convex lp-norm (0<p≤1) is a better substitute for the NP hard l0-norm than the convex l1-norm. Motivated by these studies, we propose improvements over the previous studies by replacing the l1-norm sparsity penalty by the lp-norm. Thus, we reconstruct the dynamic MRI sequence by solving a least squares minimization problem regularized by lp-norm as the sparsity penalty and Schatten-q norm as the low-rank penalty. There are no efficient algorithms to solve the said problems. In this paper, we derive efficient algorithms to solve them. The experiments have been carried out on Dynamic Contrast Enhanced (DCE) MRI datasets. Both quantitative and qualitative analysis indicates the superiority of our proposed improvement over the existing methods.

Sparse kernel logistic regression based on L 1/2 regularization

The sparsity driven classification technologies have attracted much attention in recent years, due to their capability of providing more compressive representations and clear interpretation. Two most popular classification approaches are support vector machines (SVMs) and kernel logistic regression (KLR), each having its own advantages. The sparsification of SVM has been well studied, and many sparse versions of 2-norm SVM, such as 1-norm SVM (1-SVM), have been developed. But, the sparsification of KLR has been less studied. The existing sparsification of KLR is mainly based on L1 norm and L2 norm penalties, which leads to the sparse versions that yield solutions not so sparse as it should be. A very recent study on L1/2 regularization theory in compressive sensing shows that L1/2 sparse modeling can yield solutions more sparse than those of 1 norm and 2 norm, and, furthermore, the model can be efficiently solved by a simple iterative thresholding procedure. The objective function dealt with in L1/2 regularization theory is, however, of square form, the gradient of which is linear in its variables (such an objective function is the so-called linear gradient function). In this paper, through extending the linear gradient function of L1/2 regularization framework to the logistic function, we propose a novel sparse version of KLR, the 1/2 quasi-norm kernel logistic regression (1/2-KLR). The version integrates advantages of KLR and L1/2 regularization, and defines an efficient implementation scheme of sparse KLR. We suggest a fast iterative thresholding algorithm for 1/2-KLR and prove its convergence. We provide a series of simulations to demonstrate that 1/2-KLR can often obtain more sparse solutions than the existing sparsity driven versions of KLR, at the same or better accuracy level. The conclusion is also true even in comparison with sparse SVMs (1-SVM and 2-SVM). We show an exclusive advantage of 1/2-KLR that the regularization parameter in the algorithm can be adaptively set whenever the sparsity (correspondingly, the number of support vectors) is given, which suggests a methodology of comparing sparsity promotion capability of different sparsity driven classifiers. As an illustration of benefits of 1/2-KLR, we give two applications of 1/2-KLR in semi-supervised learning, showing that 1/2-KLR can be successfully applied to the classification tasks in which only a few data are labeled.

Multikernel Adaptive Filtering

This paper exemplifies that the use of multiple kernels leads to efficient adaptive filtering for nonlinear systems. Two types of multikernel adaptive filtering algorithms are proposed. One is a simple generalization of the kernel normalized least mean square (KNLMS) algorithm [2], adopting a coherence criterion for dictionary designing. The other is derived by applying the adaptive proximal forward-backward splitting method to a certain squared distance function plus a weighted block l1 norm penalty, encouraging the sparsity of an adaptive filter at the block level for efficiency. The proposed multikernel approach enjoys a higher degree of freedom than those approaches which design a kernel as a convex combination of multiple kernels. Numerical examples show that the proposed approach achieves significant gains particularly for nonstationary data as well as insensitivity to the choice of some design-parameters.

A Geometric Approach to Support Vector Regression and Its Application to Fermentation Process Fast Modeling

Support vector machine (SVM) has shown great potential in pattern recognition and regressive estimation. Due to the industrial development demands, such as the fermentation process modeling, improving the training performance on increasingly large sample sets is an important problem. However, solving a large optimization problem is computationally intensive and memory intensive. In this paper, a geometric interpretation of SVM regression (SVR) is derived, and μ-SVM is extended for both L1-norm and L2-norm penalty SVR. Further, Gilbert algorithm, a well-known geometric algorithm, is modified to solve SVR problems. Theoretical analysis indicates that the presented SVR training geometric algorithms have the same convergence and almost identical cost of computation as their corresponding algorithms for SVM classification. Experimental results show that the geometric methods are more efficient than conventional methods using quadratic programming and require much less memory.

EEG-based vigilance estimation using extreme learning machines

For many human machine interaction systems, techniques for continuously estimating the vigilance of operators are highly desirable to ensure work safety. Up to now, various signals are studied for vigilance analysis. Among them, electroencephalogram (EEG) is the most commonly used signal. In this paper, extreme learning machine (ELM) and its modifications with L1 norm and L2 norm penalties are adopted for EEG-based vigilance estimation. A comparative study on system performance is conducted among ordinary ELM, its modifications, and support vector machines (SVMs). Experimental results show that, compared with SVMs, the ordinary ELM and its modifications can all dramatically speed up the training process while still achieving similar or better vigilance estimation accuracy. In addition, the following three observations have been made from the experiment results: (a) the ordinary ELM and the ELM with L1 norm penalty (LARS-ELM) are sensitive on the number of hidden nodes; (b) the ELM with L2 norm penalty (regularized-ELM) and the ELMs with both L1 norm and L2 norm penalties (LARS-EN-ELM, TROP-ELM) are stable and insensitive on the number of hidden nodes; and (c) regularized-ELM has a much faster training speed, while LARS-EN-ELM can achieve better vigilance estimation accuracy.

Sparse covariance fitting for direction of arrival estimation

This article proposes a new algorithm for finding the angles of arrival of multiple uncorrelated sources impinging on a uniform linear array of sensors. The method is based on sparse signal representation and does not require either the knowledge of the number of the sources or a previous initialization. The proposed technique considers a covariance matrix model based on overcomplete basis representation and tries to fit the unknown signal powers to the sample covariance matrix. Sparsity is enforced by means of a l1-norm penalty. The final problem is reduced to an objective function with a non-negative constraint that can be solved efficiently using the LARS/homotopy algorithm. The method described herein is able to provide high resolution with a low computational burden. It proceeds in an iterative fashion solving at each iteration a small linear system of equations until a stopping condition is fulfilled. The proposed stopping criterion is based on the residual spectrum and arises in a natural way when the LARS/homotopy is applied to the considered objective function.

Joint L1 and total variation regularization for fluorescence molecular tomography

Fluorescence molecular tomography (FMT) is an imaging modality that exploits the specificity of fluorescent biomarkers to enable 3D visualization of molecular targets and pathways in vivo in small animals. Owing to the high degree of absorption and scattering of light through tissue, the FMT inverse problem is inherently ill-conditioned making image reconstruction highly susceptible to the effects of noise and numerical errors. Appropriate priors or penalties are needed to facilitate reconstruction and to restrict the search space to a specific solution set. Typically, fluorescent probes are locally concentrated within specific areas of interest (e.g., inside tumors). The commonly used L2 norm penalty generates the minimum energy solution, which tends to be spread out in space. Instead, we present here an approach involving a combination of the L1 and total variation norm penalties, the former to suppress spurious background signals and enforce sparsity and the latter to preserve local smoothness and piecewise constancy in the reconstructed images. We have developed a surrogate-based optimization method for minimizing the joint penalties. The method was validated using both simulated and experimental data obtained from a mouse-shaped phantom mimicking tissue optical properties and containing two embedded fluorescent sources. Fluorescence data were collected using a 3D FMT setup that uses an EMCCD camera for image acquisition and a conical mirror for full-surface viewing. A range of performance metrics was utilized to evaluate our simulation results and to compare our method with the L1, L2 and total variation norm penalty-based approaches. The experimental results were assessed using the Dice similarity coefficients computed after co-registration with a CT image of the phantom.

DOA Estimation Based on Sparse Signal Recovery Utilizing Weighted $l_{1}$-Norm Penalty

In this letter, a new DOA estimation method based on sparse signal recovery is proposed. We utilize the Capon spectrum to design a weighted <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> -norm penalty in order to further enforce the sparsity and approximate the original <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> -norm. A theoretical guidance for choosing a proper regularization parameter is also presented according to the dual form of the original problem. Simulation results demonstrate the effectiveness and efficiency of the proposed method.

Contingency Constrained VAr Planning Using Penalty Successive Conic Programming

This paper presents a new method for VAr planning under multiple operating scenarios, optimizing sizes and locations of new reactive compensation equipment to ensure that both the system voltage profile and voltage stability requirements are met. The approach is based on mbi L1-norm regularization for finding a solution with minimum VAr installation sites and on the mbi L2-norm penalty function for satisfying the multiple state constraints. The mbi L2-norm penalty function is exact in the sense that a finite penalty parameter is required to establish equivalence with the VAr planning problem, thus avoiding numerical ill-conditioning. The solution is obtained from a successive conic programming algorithm which makes use of adaptive trust-region control. The results of the new method are compared with those of an optimal power flow-based program for VAr planning.

Pedestrian detection in images via cascaded L1-norm minimization learning method

A new cascaded L1-norm minimization learning (CLML) method for pedestrian detection in images is proposed in this paper. The proposed CLML method, which is designed from the perspective of Vapnic's theory in the statistical learning, integrates feature selection with classifier construction via solving meaningful optimization models. The method incorporates three stages: weak classifier learning, strong classifier learning and the cascaded classifier construction. In the weak classifier learning, the L1-norm minimization learning (LML) and min-max penalty function model are presented. In the strong classifier learning, an integer programming optimization model is built, equaling the reformulation of LML in the integer space. Finally, a cascade of LML classifiers is constructed to promote detection speed. During the classifier learning and pedestrian detection, Histograms of Oriented Gradients of variable-sized blocks (v-HOG) are used as feature descriptors. Experimental results on the INRIA and SDL human datasets show that the proposed method achieves a higher performance and speed than the state-of-the-art methods.

Network, Anatomical, and Non-Imaging Measures for the Prediction of ADHD Diagnosis in Individual Subjects

Brain imaging methods have long held promise as diagnostic aids for neuropsychiatric conditions with complex behavioral phenotypes such as Attention-Deficit/Hyperactivity Disorder. This promise has largely been unrealized, at least partly due to the heterogeneity of clinical populations and the small sample size of many studies. A large, multi-center dataset provided by the ADHD-200 Consortium affords new opportunities to test methods for individual diagnosis based on MRI-observable structural brain attributes and functional interactions observable from resting-state fMRI. In this study, we systematically calculated a large set of standard and new quantitative markers from individual subject datasets. These features (>12,000 per subject) consisted of local anatomical attributes such as cortical thickness and structure volumes, and both local and global resting-state network measures. Three methods were used to compute graphs representing interdependencies between activations in different brain areas, and a full set of network features was derived from each. Of these, features derived from the inverse of the time series covariance matrix, under an L1-norm regularization penalty, proved most powerful. Anatomical and network feature sets were used individually, and combined with non-imaging phenotypic features from each subject. Machine learning algorithms were used to rank attributes, and performance was assessed under cross-validation and on a separate test set of 168 subjects for a variety of feature set combinations. While non-imaging features gave highest performance in cross-validation, the addition of imaging features in sufficient numbers led to improved generalization to new data. Stratification by gender also proved to be a fruitful strategy to improve classifier performance. We describe the overall approach used, compare the predictive power of different classes of features, and describe the most impactful features in relation to the current literature.

SPACE-TIME ADAPTIVE PROCESSING BASED ON WEIGHTED REGULARIZED SPARSE RECOVERY

In this paper, novel space-time adaptive processing algorithms based on sparse recovery (SR-STAP) that utilize weighted l1-norm penalty are proposed to further enforce the sparsity and approximate the original l0-norm. Because the amplitudes of the clutter components from difierent snapshots are random variables, we design the corresponding weights according to two difierent ways, i.e., the Capon's spectrum using limited snapshots and the Fourier spectrum using the current snapshot. Moreover, we apply the weighted idea to both the direct data domain (D3) SR-STAP and SR- STAP using multiple snapshots from adjacent target-free range bins. Simulation results illustrate that our proposed algorithms outperform the existing SR-STAP and D3SR-STAP algorithms.

Application of L1-norm regularization to epicardial potential reconstruction based on gradient projection

The epicardial potential (EP)-targeted inverse problem of electrocardiography (ECG) has been widely investigated as it is demonstrated that EPs reflect underlying myocardial activity. It is a well-known ill-posed problem as small noises in input data may yield a highly unstable solution. Traditionally, L2-norm regularization methods have been proposed to solve this ill-posed problem. But the L2-norm penalty function inherently leads to considerable smoothing of the solution, which reduces the accuracy of distinguishing abnormalities and locating diseased regions. Directly using the L1-norm penalty function, however, may greatly increase computational complexity due to its non-differentiability. We propose an L1-norm regularization method in order to reduce the computational complexity and make rapid convergence possible. Variable splitting is employed to make the L1-norm penalty function differentiable based on the observation that both positive and negative potentials exist on the epicardial surface. Then, the inverse problem of ECG is further formulated as a bound-constrained quadratic problem, which can be efficiently solved by gradient projection in an iterative manner. Extensive experiments conducted on both synthetic data and real data demonstrate that the proposed method can handle both measurement noise and geometry noise and obtain more accurate results than previous L2- and L1-norm regularization methods, especially when the noises are large.

Brain Effective Connectivity Modeling for Alzheimer's Disease by Sparse Gaussian Bayesian Network.

Recent studies have shown that Alzheimer's disease (AD) is related to alteration in brain connectivity networks. One type of connectivity, called effective connectivity, defined as the directional relationship between brain regions, is essential to brain function. However, there have been few studies on modeling the effective connectivity of AD and characterizing its difference from normal controls (NC). In this paper, we investigate the sparse Bayesian Network (BN) for effective connectivity modeling. Specifically, we propose a novel formulation for the structure learning of BNs, which involves one L1-norm penalty term to impose sparsity and another penalty to ensure the learned BN to be a directed acyclic graph - a required property of BNs. We show, through both theoretical analysis and extensive experiments on eleven moderate and large benchmark networks with various sample sizes, that the proposed method has much improved learning accuracy and scalability compared with ten competing algorithms. We apply the proposed method to FDG-PET images of 42 AD and 67 NC subjects, and identify the effective connectivity models for AD and NC, respectively. Our study reveals that the effective connectivity of AD is different from that of NC in many ways, including the global-scale effective connectivity, intra-lobe, interlobe, and inter-hemispheric effective connectivity distributions, as well as the effective connectivity associated with specific brain regions. These findings are consistent with known pathology and clinical progression of AD, and will contribute to AD knowledge discovery.

Sparsity regularised recursive least squares adaptive filtering

The authors propose a new approach for the adaptive identification of sparse systems. This approach improves on the recursive least squares (RLS) algorithm by adding a sparsity inducing weighted l 1 norm penalty to the RLS cost function. Subgradient analysis is utilised to develop the recursive update equations for the calculation of the optimum system estimate, which minimises the regularised cost function. Two new algorithms are introduced by considering two different weighting scenarios for the l 1 norm penalty. These new l 1 relaxation-based RLS algorithms emphasise sparsity during the adaptive filtering process, and they allow for faster convergence than standard RLS when the system under consideration is sparse. The authors test the performance of the novel algorithms and compare it with standard RLS and other adaptive algorithms for sparse system identification.

Sparse Weighted Norm Minimum Variance Portfolio

We propose to impose a weighted L1 and squared L2 norm penalty on the portfolio weights to improve out-of-sample (OOS) performances of portfolio optimization when the number of assets becomes large. We show that under certain conditions, the realized risk of the optimal minimum variance portfolio (MVP) obtained from the strategy can asymptotically be lower than those of benchmark portfolios with a high probability. Our theoretical results imply that penalty parameters for the weighted norm penalty can be specified as a simple function of the number of assets and sample size. With the theoretical results, we also develop an automatic calibration procedure for choosing the penalty parameters. We demonstrate superior OOS performances of the weighted-norm MVP with two real data sets. Finally, we propose several alternative norm penalties and show that their OOS performances are comparable to the weighted-norm strategy.

Multi-Label Transfer Learning With Sparse Representation

Due to the visually polysemous barrier, videos and images may be annotated by multiple tags. Discovering the correlations among different tags can significantly help predicting precise labels for videos and images. Many of recent studies toward multi-label learning construct a linear subspace embedding with encoded multi-label information, such that data points sharing many common labels tend to be close to each other in the embedded subspace. Motivated by the advances of compressive sensing research, a sparse representation that selects a compact subset to describe the input data can be more discriminative. In this paper, we propose a sparse multi-label learning method to circumvent the visually polysemous barrier of multiple tags. Our approach learns a multi-label encoded sparse linear embedding space from a related dataset, and maps the target data into the learned new representation space to achieve better annotation performance. Instead of using l1-norm penalty (lasso) to induce sparse representation, we propose to formulate the multi-label learning as a penalized least squares optimization problem with elastic-net penalty. By casting the video concept detection and image annotation tasks into a sparse multi-label transfer learning framework in this paper, ridge regression, lasso, elastic net, and the multi-label extended sparse discriminant analysis methods are, respectively, well explored and compared.

Convergence analysis of sparse LMS algorithms with l1-norm penalty based on white input signal

The zero-attracting LMS (ZA-LMS) algorithm is one of the recently published sparse LMS algorithms. It usesan l 1-norm penalty in the standard LMS cost function. In this paper, we perform convergence analysis of the ZA-LMS algorithm based on white input signals. The stability condition is examined and the steady-state mean square deviation (MSD) is derived in terms of the system sparsity, system response length, and filter parameters (step size and zero-attractor controller). In addition, we propose a criterion for parameter selection such that the ZA-LMS algorithm outperforms the standard LMS algorithm. The results are demonstrated through computer simulations.

Application of L1-Norm Regularization to Epicardial Potential Solution of the Inverse Electrocardiography Problem

The electrocardiographic inverse problem of computing epicardial potentials from multi-electrode body-surface ECG measurements, is an ill-posed problem. Tikhonov regularization is commonly employed, which imposes penalty on the L2-norm of the potentials (zero-order) or their derivatives. Previous work has indicated superior results using L2-norm of the normal derivative of the solution (a first order regularization). However, L2-norm penalty function can cause considerable smoothing of the solution. Here, we use the L1-norm of the normal derivative of the potential as a penalty function. L1-norm solutions were compared to zero-order and first-order L2-norm Tikhonov solutions and to measured 'gold standards' in previous experiments with isolated canine hearts. Solutions with L1-norm penalty function (average relative error [RE] = 0.36) were more accurate than L2-norm (average RE = 0.62). In addition, the L1-norm method localized epicardial pacing sites with better accuracy (3.8 +/- 1.5 mm) compared to L2-norm (9.2 +/- 2.6 mm) during pacing in five pediatric patients with congenital heart disease. In a pediatric patient with Wolff-Parkinson-White syndrome, the L1-norm method also detected and localized two distinct areas of early activation around the mitral valve annulus, indicating the presence of two left-sided pathways which were not distinguished using L2 regularization.