L1 Penalty Term Research Articles

Hausdorff metric based training of kernels to learn attractors with application to 133 chaotic dynamical systems

Kernel methods have been successfully used to learn surrogate models for dynamical systems by regressing the vector field of such systems. However, choosing an appropriate kernel for a specific problem is an essential challenge in practice. Kernel Flows (KFs) have emerged as an effective tool for learning data-adaptive kernels used to interpolate dynamical systems. In this paper, we introduce Hausdorff Metric based Kernel Flows (HMKFs) to discover a’good’ kernel from time series when a system under consideration has an attractor. First, under the premise that a kernel is good if there is no significant loss in accuracy when half of the data is used to reconstruct the attractor, we design an objective function based on the distance between true and forecasted attractors. Then, we optimize the proposal by adding an ℓ1 penalty term when the base kernel contains many terms, thereby generating sparse HMKFs. Furthermore, we apply HMKFs and its sparse version to a library of 133 chaotic systems from various fields such as biochemistry, fluid mechanics, and astrophysics. The results showcase the potential of HMKFs in accurately modeling complex dynamical systems using data-driven kernels.

Combining regularization and logistic regression model to validate the Q-matrix for cognitive diagnosis model.

Q-matrix is an important component of most cognitive diagnosis models (CDMs); however, it mainly relies on subject matter experts' judgements in empirical studies, which introduces the possibility of misspecified q-entries. To address this, statistical Q-matrix validation methods have been proposed to aid experts' judgement. A few of these methods, including the multiple logistic regression-based (MLR-B) method and the Hull method, can be applied to general CDMs, but they are either time-consuming or lack accuracy under certain conditions. In this study, we combine the L1 regularization and MLR model to validate the Q-matrix. Specifically, an L1 penalty term is imposed on the log-likelihood of the MLR model to select the necessary attributes for each item. A simulation study with various factors was conducted to examine the performance of the new method against the two existing methods. The results show that the regularized MLR-B method (a) produces the highest Q-matrix recovery rate (QRR) and true positive rate (TPR) for most conditions, especially with a small sample size; (b) yields a slightly higher true negative rate (TNR) than either the MLR-B or the Hull method for most conditions; and (c) requires less computation time than the MLR-B method and similar computation time as the Hull method. A real data set is analysed for illustration purposes.

EEG signal classification based on improved variational mode decomposition and deep forest

The study of EEG signals is of great significance for the diagnosis and prevention of brain disease. Most of the previous studies are based on the binary classification of nonictal and ictal EEG signals, and there are few studies on the detailed division of EEG signals. In this paper, in addition to the binary classification of EEG signals, the multiclassification of EEG signals is also studied. An EEG signals recognition framework based on improved variational mode decomposition (VMD) and deep forest is proposed. Firstly, the L1 penalty term is introduced into the variational problem of VMD to improve the Tikhonov regularization term. The improved VMD algorithm is used to decompose the original signal. Second, a weighted minimum redundancy maximum relevance criterion is constructed for feature selection. Finally, a deep forest model is built to classify EEG signals. The feasibility of the proposed method is verified by EEG data from Bonn University and the Centre for Neurology and Sleep, Hauz Khas, New Delhi. The experimental results are compared with the traditional machine learning methods and the existing methods. Experimental results show that this method can recognize epileptic EEG signals effectively.

A Novel Method to Enhance the Inversion Speed and Precision of the NMR T2 Spectrum by the TSVD Based Linearized Bregman Iteration

The low-field nuclear magnetic resonance (NMR) technique has been used to probe the pore size distribution and the fluid composition in geophysical prospecting and related fields. However, the speed and accuracy of the existing numerical inversion methods are still challenging due to the ill-posed nature of the first kind Fredholm integral equation and the contamination of the noises. This paper proposes a novel inversion algorithm to accelerate the convergence and enhance the precision using empirical truncated singular value decompositions (TSVD) and the linearized Bregman iteration. The L1 penalty term is applied to construct the objective function, and then the linearized Bregman iteration is utilized to obtain fast convergence. To reduce the complexity of the computation, empirical TSVD is proposed to compress the kernel matrix and determine the appropriate truncated position. This novel inversion method is validated using numerical simulations. The results indicate that the proposed novel method is significantly efficient and can achieve quick and effective data solutions with low signal-to-noise ratios.

Portfolio Construction with LASSO

It is well known that standard Markowitz portfolio optimization suffers from empirical instability. Brodie, Daubechies, De Mol, Giannone, and Loris (2009) show that inclusion of an L1 penalty term in the optimization leads to sparse and stable solutions. The L1 penalty term can be used to model transaction costs and gross leverage and offers a way to control the trade-off between Sharpe ratio and implementation constraints such as turn over, transaction costs and gross leverage. The methodology can be useful to institutional investors in construction of target-return portfolios and replication of illiquid assets. I compare Standard Markowitz optimization, LASSO optimization and equal volatility weighting approaches to portfolio construction using a practical case study and show that LASSO optimization is superior to the other two approaches.

Fast distributed control design for DC linked microgrids

A centralized controller for a DC linked hybrid microgrid requires costly communication infrastructure. To minimize the costs, in this paper, a sparse distributed controller is formulized by introducing an l1 penalty term to the cost function to enforce at most not more than neighbor-to neighbor links. This makes the design nonlinear which exhausts large amount of computation times. Therefore, by some transformations, the nonlinear cost function is converted to an approximate quadratic one; which has better optimization characteristics, i.e. fast convergence rate and unimodal. The proposed fast distributed controller Design (FDCD) algorithm is employed for a 3-bus and a 69-bus (consist of 7 hybrid microgrid) distribution systems controller design. The results indicate that the obtained sparse distributed controller performs well as if a centralized controller is in operation. It can successfully suppress various types of faults injected to the AC, DC, and DC link sides of the network. The algorithm convergence rate is substantially lower than some other contestant algorithms. A problem that is solved by FDCD in 5 min, it takes 10 min for the others to be completed.

Sparse model-based estimation of functional dependence in high-dimensional field and spike multiscale networks

Objective. Behavior is encoded across multiple scales of brain activity, from binary neuronal spikes to continuous fields including local field potentials (LFP). Multiscale models need to describe both the encoding of behavior and the conditional dependencies in simultaneously recorded spike and field signals, which form a high-dimensional multiscale network. However, learning spike-field dependencies in high-dimensional recordings is challenging due to the prohibitively large number of spike-field signal pairs, which makes standard learning techniques subject to overfitting. Approach. We present a sparse model-based estimation algorithm to learn these multiscale network dependencies. We develop a multiscale encoding model consisting of a point process model of binary spikes for each neuron whose firing rate is a function of the LFP network features and behavioral states. Doing so, spike-field dependencies constitute the model parameters to be learned. We resolve the parameter learning challenge by forming a constrained optimization problem to maximize the likelihood with an L1 penalty term that eases the detection of significant spike-LFP dependencies. We then apply the Akaike information criterion (AIC) to force a sparse number of nonzero dependency parameters in the model. Main results. We validate the algorithm using simulations and spike-field data from two non-human primates (NHP) in a 3D motor task with motor cortical recordings and a pro-saccade visual task with prefrontal recordings. We find that by identifying a model with a sparse set of dependency parameters, the algorithm improves spike prediction compared with models without dependencies. Further, the algorithm identifies significantly fewer dependency parameters compared with standard methods while improving their spike prediction likely due to detecting fewer spurious dependencies. Also, spike prediction on any electrode improves by including LFP features from all electrodes compared with using only those on the same electrode. Finally, unlike standard methods, the algorithm uncovers patterns of spike-field network dependencies as a function of distance, brain region, and frequency band. Significance. This algorithm can help study functional dependencies in high-dimensional spike-field networks and leads to more accurate multiscale encoding models.

Hyperspectral Band Selection Using Weighted Kernel Regularization

A band selection method named weighted kernel regularization (WKR) is proposed for hyperspectral imagery (HSI) classification. The WKR aims to select dissimilar and class-separable bands to better model the relationship between labeled samples. First, the WKR considers nonlinear structure of hyperspectral data and models nonlinear relations between HSI samples and their class labels using a weighted kernel ridge regression (WKRR) program with respect to sample coefficients. Second, it combines the L1 penalty term of weights on all bands with the above WKRR program into the unified framework of WKR. The L1 penalty term considers divergent contributions from different bands in describing nonlinear relations and guarantees the sparsity of band weights. Third, the WKR algorithm implements the KerNel Iterative-based Feature Extraction (KNIFE) algorithm to estimate the proper band weights. The KNIFE linearizes the nonlinear kernels to avoid high computational cost, and iteratively minimizes two convex subproblems with respect to the sample coefficients and band weights. Finally, the first k bands with larger weights and larger dissimilarity with other bands are automatically chosen to form the band subset. Experimental results show that the WKR outperforms the state-of-the-art methods in classification accuracies with a lower computational cost.

A Projection Neural Network for the Generalized Lasso.

The generalized lasso (GLasso) is an extension of the lasso regression in which there is an l1 penalty term (or regularization) of the linearly transformed coefficient vector. Finding the optimal solution of GLasso is not straightforward since the penalty term is not differentiable. This brief presents a novel one-layer neural network to solve the generalized lasso for a wide range of penalty transformation matrices. The proposed neural network is proven to be stable in the sense of Lyapunov and converges globally to the optimal solution of the GLasso. It is also shown that the proposed neural solution can solve many optimization problems, including sparse and weighted sparse representations, (weighted) total variation denoising, fused lasso signal approximator, and trend filtering. Disparate experiments on the above problems illustrate and confirm the excellent performance of the proposed neural network in comparison to other competing techniques.

A reconstruction algorithm for full waveform inversion based on sparsity regularization

Full waveform inversion (FWI) is an important ill-posed problem in mathematical physics inverse problem. In order to address considerable over-smoothing of the Tikhonov regularization solution, sparsity regularization method is applied to FWI. The sparsity of the ‘inhomogeneity’ with respect to a certain basis is priori assumed. The proposed approach is motivated by a Tikhonov functional incorporating a sparsity-promoting l1–penalty term, and it allows us to obtain quantitative results when the assumption is valid. We have proposed an applicable sparse reconstruction algorithm of iterative soft shrinkage type. This algorithm is compared with a conventional reconstruction approach on the basis of smoothness regularization. The numerical examples showed that very good reconstruction results can be obtained by the proposed algorithms.

On the edge detection of an image by numerical differentiations for gray function

The detection of image edges is of great importance in image processing. One of the efficient implementations for this image recovery problem is based on the identification of sharp jumps of the gray function of the image. Mathematically, this problem can be modeled by the numerical differentiation of the gray function with 2 variables. For this ill‐posed problem with nonsmooth solution, we investigate the regularization schemes with total variation and L1 penalty term, respectively. We prove that the regularizing parameter under the Tikhonov regularization framework can be uniquely chosen in terms of the Morozov's discrepancy principle and then establish the convergence rate of the regularizing solutions in terms of the Bregman distance. The discrete schemes are performed by the lagged diffusivity fixed point iteration, with numerical implementations showing the validity of the proposed scheme.

A Band-Weighted Support Vector Machine Method for Hyperspectral Imagery Classification

A band-weighted support vector machine (BWSVM) method is proposed to classify hyperspectral imagery (HSI). The BWSVM presents an L1 penalty term of band weight vector to regularize the regular SVM model. The L1 norm regularization term guarantees the sparsity of band weights and describes potentially divergent contributions from different bands in modeling the binary SVM model. The BWSVM adopts the KerNel iterative feature extraction algorithm to minimize the nonconvex program. It linearizes nonlinear kernels and iteratively optimizes two convex subproblems with respect to both sample coefficients and band weights. The class label is determined by picking the largest sample coefficients from all its binary models of BWSVM. Two popular HSI data sets are utilized to testify the classification performance of BWSVM. Experimental results show that the BWSVM outperforms three state-of-the-art classifiers including SVM, random forest, and k-nearest neighbor.

Convergence analysis of BP neural networks via sparse response regularization

Backpropagation (BP) algorithm is the typical strategy to train the feedforward neural networks (FNNs). Gradient descent approach is the popular numerical optimal method which is employed to implement the BP algorithm. However, this technique frequently leads to poor generalization and slow convergence. Inspired by the sparse response character of human neuron system, several sparse-response BP algorithms were developed which effectively improve the generalization performance. The essential idea is to impose the responses of hidden layer as a specific L1 penalty term on the standard error function of FNNs. In this paper, we mainly focus on the two remaining challenging tasks: one is to solve the non-differential problem of the L1 penalty term by introducing smooth approximation functions. The other aspect is to provide a rigorous convergence analysis for this novel sparse response BP algorithm. In addition, an illustrative numerical simulation has been done to support the theoretical statement.

Orbital minimization method with ℓ1 regularization

We consider a modification of the orbital minimization method (OMM) energy functional which contains an ℓ1 penalty term in order to find a sparse representation of the low-lying eigenspace of self-adjoint operators. We analyze the local minima of the modified functional as well as the convergence of the modified functional to the original functional. Algorithms combining soft thresholding with gradient descent are proposed for minimizing this new functional. Numerical tests validate our approach. In addition, we also prove the unanticipated and remarkable property that every local minimum of the OMM functional without the ℓ1 term is also a global minimum.

The inversion of 2D NMR relaxometry data using L1 regularization

NMR relaxometry has been used as a powerful tool to study molecular dynamics. Many algorithms have been developed for the inversion of 2D NMR relaxometry data. Unlike traditional algorithms implementing L2 regularization, high order Tikhonov regularization or iterative regularization, L1 penalty term is involved to constrain the sparsity of resultant spectra in this paper. Then fast iterative shrinkage-thresholding algorithm (FISTA) is proposed to solve the L1 regularization problem. The effectiveness, noise vulnerability and practical utility of the proposed algorithm are analyzed by simulations and experiments. The results demonstrate that the proposed algorithm has a more excellent capability to reveal narrow peaks than traditional inversion algorithms. The L1 regularization implemented by our algorithm can be a useful complementary to the existing algorithms.

Soft Shrinkage Thresholding Algorithm for Nonlinear Microwave Imaging

In this paper, we analyze a sparse nonlinear inverse scattering problem arising in microwave imaging and numerically solved it for retrieving dielectric contrast from measured fields. In sparsity reconstruction, contrast profiles are a priori assumed to be sparse with respect to a certain base. We proposed an approach which is motivated by a Tikhonov functional incorporating a sparsity promoting l1-penalty term. The proposed iterative algorithm of soft shrinkage type enforces the sparsity constraint at each nonlinear iteration. The scheme produces sharp and good reconstruction of dielectric profiles in sparse domains by adapting Barzilai and Borwein (BB) step size selection criteria and positivity by maintaining its convergence during the reconstruction.

An MM-Based Algorithm for -Regularized Least-Squares Estimation With an Application to Ground Penetrating Radar Image Reconstruction

An estimation method known as least absolute shrinkage and selection operator (LASSO) or ℓ1-regularized LS estimation has been found to perform well in a number of applications. In this paper, we use the majorize-minimize method to develop an algorithm for minimizing the LASSO objective function, which is the sum of a linear LS objective function plus an ℓ1 penalty term. The proposed algorithm, which we call the LASSO estimation via majorization-minimization (LMM) algorithm, is straightforward to implement, parallelizable, and guaranteed to produce LASSO objective function values that monotonically decrease. In addition, we formulate an extension of the LMM algorithm for reconstructing ground penetrating radar (GPR) images, that is much faster than the standard LMM algorithm and utilizes significantly less memory. Thus, the GPR specific LMM (GPR-LMM) algorithm is able to accommodate the big data associated with GPR imaging. We compare our proposed algorithms to the state-of-the-art ℓ1-regularized LS algorithms using a time and space complexity analysis. The GPR-LMM greatly outperforms the competing algorithms in terms of the performance metrics we considered. In addition, the reconstruction results of the standard LMM and GPR-LMM algorithms are evaluated using both simulated and real GPR data.

Stability of the elastic net estimator

The elastic net is a regularized least squares regression method that has been widely used in learning and variable selection. The elastic net regularization linearly combines an l1 penalty term (like the lasso) and an l2 penalty term (like ridge regression). The l1 penalty term enforces sparsity of the elastic net estimator, whereas the l2 penalty term ensures democracy among groups of correlated variables. Compressed sensing is currently an extensively studied technique for efficiently reconstructing a sparse vector from much fewer samples/observations. In this paper we study the elastic net in the setting of sparse vector recovery. For recovering sparse vectors from few observations by employing the elastic net regression, we prove in this paper that the elastic net estimator is stable provided that the underlying measurement/design matrix satisfies the commonly required restricted isometry property or the sparse approximation property. It is well known that many independent random measurement matrices satisfy the restricted isometry property while random measurement matrices generated by highly correlated Gaussian random variables satisfy the sparse approximation property. As a byproduct, we establish a uniform bound for the grouping effect of the elastic net. Some numerical experiments are provided to illustrate our theoretical results on stability and grouping effect of the elastic net estimator.

A novel sparse boosting method for crater detection in the high resolution planetary image

Impact craters distributed on planetary surface become one of the main barriers during the soft landing of planetary probes. In order to accelerate the crater detection, in this paper, we present a new sparse boosting (SparseBoost) method for automatic detection of sub-kilometer craters. The SparseBoost method integrates an improved sparse kernel density estimator (RSDE-WL1) into the Boost algorithm and the RSDE-WL1 estimator is achieved by introducing weighted l1 penalty term into the reduced set density estimator. An iterative algorithm is proposed to implement the RSDE-WL1. The SparseBoost algorithm has the advantage of fewer selected features and simpler representation of the weak classifiers compared with the Boost algorithm. Our SparseBoost based crater detection method is evaluated on a large and high resolution image of Martian surface. Experimental results demonstrate that the proposed method can achieve less computational complexity in comparison with other crater detection methods in terms of selected features.

A sparser reduced set density estimator by introducing weighted l1 penalty term

Reduced set density estimator (RSDE), employing a small percentage of available data samples, is an effective and important nonparametric technique for probability density function estimation. Despite that RSDE has been demonstrated to perform better in the computational complexity compared with several existing approaches, it still faces the critical challenge in practical applications when training the estimator on large data sets. Dealing with its high complexity both in time and space, a sparser reduced set density estimator with weighted l1 penalty term (RSDE-WL1) is proposed in this paper. By introducing the weighted l1 norm of the estimated coefficients as additional penalty term, the sparse weights of density estimation are estimated, in which small weight coefficients are more likely to be driven to zero. The proposed iterative algorithm is used to solve the corresponding convex optimization problem efficiently. Some discussions about the choice of parameters and the trade-off between sparsity and accuracy are also given. The simulations of several examples demonstrate that the proposed RSDE-WL1 is superior to the related methods including the RSDE in sparsity and complexity. While requiring less number of kernels in density estimation, our approach is comparable to the full-sample optimized Parzen window estimator in terms of test accuracy.