Sparse Regularization Research Articles

Fast orthogonal locality-preserving projections for unsupervised feature selection

Graph-based sparsity learning is one of the most successful unsupervised feature selection methods that has been widely adopted in many real-world applications. However, traditional graph-based unsupervised feature selection methods have several drawbacks: (1) being time-consuming and unable to deal with large-scale problems; (2) having difficulty tuning the regularization parameter with the sparsity regularization term; and (3) being unable to find explicit solutions owing to the limitation of sparsity, that is, feature selection with the ℓ2,1-norm constrained problem. Thus, this paper proposes OLPPFS, a method to preserve the local geometric structure within the feature subspace by imposing the ℓ2,0-norm constraint. First, the linear mapping capability of the proposed model is enhanced using locality-preserving projections (LPPs), whichpreserve the local and global geometric manifold structure of the data while enhancing the ability to reconstruct data. Second, the graph-embedding learning method can accelerate the construction of a sparsity affinity graph and describe the intrinsic structure of the dataset well. More importantly, we propose a method for solving a projection matrix with the ℓ2,0-norm constrained, which can accurately select a explicit group of discriminative feature subsets. This method can yield a more accurate sparse projection matrix than the ℓ2,1-norm. We also adopt FOLPPFS, an effective anchor-based strategy to further accelerate our model with two flexible options. Extensive experiments on eight datasets demonstrate that the proposed method is superior to the other methods and can preserve a better local geometric structure of the dataset with less time consumption.

Sparse non-convex regularization based explainable DBN in the analysis of brain abnormalities in schizophrenia

Deep belief networks have been widely used in medical image analysis. However, the high-dimensional but small-sample-size characteristic of medical image data makes the model prone to dimensional disaster and overfitting. Meanwhile, the traditional DBN is driven by performance and ignores the explainability which is important for medical image analysis. In this paper, a sparse non-convex based explainable deep belief network is proposed by combining DBN with non-convex sparsity learning. For sparsity, the non-convex regularization and Kullback–Leibler divergence penalty are embedded into DBN to obtain the sparse connection and sparse response representation of the network. It effectively reduces the complexity of the model and improves the generalization ability of the model. Considering explainability, the crucial features for decision-making are selected through the feature back-selection based on the row norm of each layer’s weight after network training. We apply the model to schizophrenia data and demonstrate it achieves the best performance among several typical feature selection models. It reveals 28 functional connections highly correlated with schizophrenia, which provides an effective foundation for the treatment and prevention of schizophrenia and methodological assurance for similar brain disorders.

Non-convex sparse regularization via convex optimization for impact force identification

Instead of traditional Tikhonov regularization, sparse regularization methods like ℓ1 regularization have been a popular choice for impact force identification because it can encode sparse prior information into the inverse problem solving. However, ℓ1 regularization tends to have a limited ability to promote sparsity and underestimate high-amplitude components in the solution. Non-convex regularization holds the potential to promote sparsity more efficiently, and then further improves the accuracy of solutions. In this paper, we consider a family of non-convex regularizers with convexity-preserving characteristics to promote sparsity. By virtue of the non-convex regularizers, we propose a non-convex sparse regularization method to simultaneously localize and reconstruct impact events with a highly under-determined sensor setting. To avoid the inherent difficulties concerning non-convex optimization, we turn to the Alternating Direction Method of Multipliers(ADMM) algorithm. ADMM helps separate the primal non-convex problem into a sequence of convex sub-problems, each of which is easy-to-solve. The non-convex exponential penalty is chosen as the representative of this class of regularizers through theoretical analysis. Several simulations and experiments are conducted on a stiffened composite laminated panel to validate the proposed method. The proposed method outperforms the standard ℓ1 regularization in both localization accuracy and time-history reconstruction accuracy.

Elimination of accelerometer mass loading effects in sparse identification of impact forces

Identifying impact force from measured structural responses has been extensively studied. Contact sensors such as accelerometers are often applied for recording the structural responses. Masses of the contact sensors would change the structural dynamic properties, especially for lightweight structures. To eliminate the effects of additional masses in force identification, this paper proposes a novel method for reconstructing the impact force. The proposed method is developed from existing sparse regularization-based method. In the process of forward analysis, a key step is to introduce some pseudo-forces for simulation of the additional mass loading effects. The measured responses are regarded as the sum of two components. One is induced by the impact force, and the other one is caused via pseudo-forces. In the process of inverse analysis, impact force identification involving notable sensor mass is treated as a problem of multi-force identification, in which both impact force and pseudo-forces are simultaneously identified. The impact force is identified via sparse regularization which determines more accurate results insensitive to noise. Masses of installed sensors are estimated from the identified pseudo-forces because the pseudo-forces are a translated expression of mass loading effects. Therefore, the effects of additional masses can be eliminated in force identification. Numerical simulations and experimental studies on a cantilever beam are carried out. Cases studies show that the proposed method can be applied for estimating the impact force and additional masses with high accuracy and strong robustness. Some related issues are discussed as well.

An Adaptive Sparse Regularization Method for Response Covariance-Based Structural Damage Detection

Structural damage detection is usually an ill-posed inverse problem due to the contamination of measurement noise and model error in structural health monitoring. To deal with the ill-posed damage detection problem, l2-regularization is widely used. However, l2-regularization tends to provide nonsparse solutions and distribute identified damage to many undamaged elements, potentially leading to false alarms. Therefore, an adaptive sparse regularization method is proposed, which considers spatially sparse damage as a prior constraint since structural damage often occurs in some locations with stiffness reduction at the sparse elements out of the large total number of elements in an entire structure. First, a response covariance-based convex cost function is established by incorporating an l1-regularized term and an adaptive regularization factor to formulate the sparse regularization-based damage detection problem. Then, optimal sensor placement is conducted to determine the optimal measurement locations where the acceleration responses are adopted for computing the response covariance-based damage index and cost function. Further, the predictor-corrector primal-dual path-following approach, an efficient and robust convex optimization algorithm, is applied to search for solutions to the damage detection problem. Finally, a comparison study with the Tikhonov regularization-based damage detection method is conducted to examine the performance of the proposed adaptive sparse regularization-based method by using an overhanging beam model subjected to different damage scenarios and noise levels. The numerical study demonstrates that the proposed method can effectively and accurately identify damage under multiple damage scenarios with various noise levels, and it outperforms the Tikhonov regularization-based method in terms of high accuracy and few false alarms. The analyses on time consumption, adaptiveness of the sparse regularization factor, model-error resistance, and sensor number influence are conducted for further discussions of the proposed method.

L0-norm based Short-term Sparse Portfolio Optimization Algorithm Based on Alternating Direction Method of Multipliers

This paper proposes a novel short-term sparse portfolio optimization (SSPO) model based on ℓ0-norm. Compared with existing approaches, this model selects the portfolio based on the short-term increasing potential of assets, and an ℓ0-norm constraint is introduced to directly control the maximum number of non-zero assets in selected portfolios. Unlike the ℓ1-norm based methods, the no-short-selling constraints can be directly used in our proposed model. Besides, a sparse regularization term is introduced to eliminate trivial trades in the SSPO system. Moreover, to solve the contained non-convex optimization system, an algorithm based on the concept of the alternating direction method of multipliers (ADMM) is developed. The convergence of the proposed algorithm is also investigated. Finally, the effectiveness of the proposed approach is demonstrated by some numerical experiments on four real-world datasets.

Safe RuleFit: Learning Optimal Sparse Rule Model by Meta Safe Screening.

We consider the problem of learning a sparse rule model, a prediction model in the form of a sparse linear combination of rules, where a rule is an indicator function defined over a hyper-rectangle in the input space. Since the number of all possible such rules is extremely large, it has been computationally intractable to select the optimal set of active rules. In this paper, to solve this difficulty for learning the optimal sparse rule model, we propose Safe RuleFit (SRF). Our basic idea is to develop meta safe screening (mSS), which is a non-trivial extension of well-known safe screening (SS) techniques. While SS is used for screening out one feature, mSS can be used for screening out multiple features by exploiting the inclusion-relations of hyper-rectangles in the input space. SRF provides a general framework for fitting sparse rule models for regression and classification, and it can be extended to handle more general sparse regularizations such as group regularization. We demonstrate the advantages of SRF through intensive numerical experiments.

An Application of Elastic-Net Regularized Linear Inverse Problem in Seismic Data Inversion

In exploration geophysics, seismic impedance is a physical characteristic parameter of underground formations. It can mark rock characteristics and help stratigraphic analysis. Hence, seismic data inversion for impedance is a key technology in oil and gas reservoir prediction. To invert impedance from seismic data, one can perform reflectivity series inversion first. Then, under a simple exponential integration transformation, the inverted reflectivity series can give the final inverted impedance. The quality of the inverted reflectivity series directly affects the quality of impedance. Sparse-spike inversion is the most common method to obtain reflectivity series with high resolution. It adopts a sparse regularization to impose sparsity on the inverted reflectivity series. However, the high resolution of sparse-spike-like reflectivity series is obtained at the cost of sacrificing small reflectivity. This is the inherent problem of sparse regularization. In fact, the reflectivity series from the actual impedance well log is not strictly sparse. It contains not only the sparse major large reflectivity, but also small reflectivity between major reflectivity. That is to say, the large reflectivity is sparse, but the small reflectivity is dense. To combat this issue, we adopt elastic-net regularization to replace sparse regularization in seismic impedance inversion. The elastic net is a hybrid regularization that combines sparse regularization and dense regularization. The proposed inversion method was performed on a synthetic seismic trace, which is created from an actual well log. Then, a real seismic data profile was used to test the practice application. The inversion results showed that it provides an effective new alternative method to invert impedance.

PAEDID: Patch Autoencoder-based Deep Image Decomposition for pixel-level defective region segmentation

Unsupervised pixel-level defective region segmentation is an important task in image-based anomaly detection for various industrial applications. The state-of-the-art methods have their own advantages and limitations: matrix-decomposition-based methods are robust to noise, but lack complex background image modeling capability; representation-based methods are good at defective region localization, but lack accuracy in defective region shape contour extraction; reconstruction-based methods detected defective region match well with the ground truth defective region shape contour, but are noisy. To combine the best of both worlds, we present an unsupervised Patch AutoEncoder-based Deep Image Decomposition (PAEDID) method for defective region segmentation. In the training stage, we learn the common background as a deep image prior by a patch autoencoder network. In the inference stage, we formulate anomaly detection as an image decomposition problem with the deep image prior and sparsity regularizations. By adopting the proposed approach, the defective regions in the image can be accurately extracted in an unsupervised fashion. We demonstrate the effectiveness of the PAEDID method in simulation studies and an industrial dataset in the case study.

Pruning-aware Sparse Regularization for Network Pruning

Structural neural network pruning aims to remove the redundant channels in the deep convolutional neural networks (CNNs) by pruning the filters of less importance to the final output accuracy. To reduce the degradation of performance after pruning, many methods utilize the loss with sparse regularization to produce structured sparsity. In this paper, we analyze these sparsity-training-based methods and find that the regularization of unpruned channels is unnecessary. Moreover, it restricts the network’s capacity, which leads to under-fitting. To solve this problem, we propose a novel pruning method, named MaskSparsity, with pruning-aware sparse regularization. MaskSparsity imposes the fine-grained sparse regularization on the specific filters selected by a pruning mask, rather than all the filters of the model. Before the fine-grained sparse regularization of MaskSparity, we can use many methods to get the pruning mask, such as running the global sparse regularization. MaskSparsity achieves a 63.03% float point operations (FLOPs) reduction on ResNet-110 by removing 60.34% of the parameters, with no top-1 accuracy loss on CIFAR-10. On ILSVRC-2012, MaskSparsity reduces more than 51.07% FLOPs on ResNet-50, with only a loss of 0.76% in the top-1 accuracy. The code of this paper is released at https://github.com/CASIA-IVA-Lab/MaskSparsity . We have also integrated the code into a self-developed PyTorch pruning toolkit, named EasyPruner, at https://gitee.com/casia_iva_engineer/easypruner .

Unsupervised domain adaptation via re-weighted transfer subspace learning with inter-class sparsity

Unsupervised domain adaptation is intended to respond to a challenging scenario where the training and testing samples are related yet distributed differently. A common technique to realize the alignment of distributions is feature transformation, which is gaining increasing attention in the transfer learning community. Many of existing studies concentrate on learning domain-invariant feature representations in order to transfer classifiers from the source to the target domain. However, the class discriminability of features is also crucial for the cross-domain learning, which could lead to decreased performance if ignored. To extract both the transferable and discriminative feature representation, a novel unsupervised domain adaptation method termed re-weighted transfer subspace learning with inter-class sparsity (ICS-RTSL) is proposed. Central to the proposed approach is the introduction of a class-wise sparsity regularization that aims to reduce the intra-class distances by maintaining the structural consistency of the source samples, thus making it possible to learn a more discriminative representation. In addition, a residual term through the least absolute criterion is constructed to mitigate the negative impact possibly brought by outliers. This is followed by the proposal of an efficient algorithm with iteratively re-weighted least squares to optimize the learning model. Supported by a set of comparative experiments, the effectiveness of the proposed ICS-RTSL is demonstrated through cross-domain image recognition tasks.

Sparse deep neural networks for modeling aluminum electrolysis dynamics

Deep neural networks have become very popular in modeling complex nonlinear processes due to their extraordinary ability to fit arbitrary nonlinear functions from data with minimal expert intervention. However, they are almost always overparameterized and challenging to interpret due to their internal complexity. Furthermore, the optimization process to find the learned model parameters can be unstable due to the process getting stuck in local minima. In this work, we demonstrate the value of sparse regularization techniques to significantly reduce the model complexity. We demonstrate this for the case of an aluminum extraction process, which is highly nonlinear system with many interrelated subprocesses. We trained a densely connected deep neural network to model the process and then compared the effects of sparsity promoting ℓ1 regularization on generalizability, interpretability, and training stability. We found that the regularization significantly reduces model complexity compared to a corresponding dense neural network. We argue that this makes the model more interpretable, and show that training an ensemble of sparse neural networks with different parameter initializations often converges to similar model structures with similar learned input features. Furthermore, the empirical study shows that the resulting sparse models generalize better from small training sets than their dense counterparts.

A DCA‐based sparse coding for video summarization with MCP

AbstractVideo summarization offers a summary version that conveys the primary information of a longer video. The main challenges of video summarization are related to keyframe extraction and saliency mapping. Thus, this work proposes a sparse coding model for keyframe extraction and saliency mapping applications. Specifically, the minimax concave penalty (MCP) is utilized as a sparse regularization scheme and the regularized non‐convex MCP problem is solved by decomposing MCP into two convex functions and the convex function's algorithm difference is relied on to solve the resulting sub‐problems. The experimental results demonstrate higher compressed keyframes and saliency maps than current state‐of‐the‐art algorithms. In particular, the model attains a lower summary length of 34% and 19% compared to sparse modeling representation selection (SMRS) and sparse modeling using the determinant sparsity measure (SC‐det), respectively. In addition, the developed scheme has a shorter computation time, requiring 82% and 33% less time than the ITTI and the dense and sparse reconstruction (DSR) methods.

Parameter choices for sparse regularization with the norm *

We consider a regularization problem whose objective function consists of a convex fidelity term and a regularization term determined by the norm composed with a linear transform. Empirical results show that the regularization with the norm can promote sparsity of a regularized solution. The goal of this paper is to understand theoretically the effect of the regularization parameter on the sparsity of the regularized solutions. We establish a characterization of the sparsity under the transform matrix of the solution. When the objective function is block-separable or an error bound of the regularized solution to a known function is available, the resulting characterization can be taken as a regularization parameter choice strategy with which the regularization problem has a solution having a sparsity of a certain level. When the objective function is not block-separable, we propose an iterative algorithm which simultaneously determines the regularization parameter and its corresponding solution with a prescribed sparsity level. Moreover, we study choices of the regularization parameter so that the regularization term can alleviate the ill-posedness and promote sparsity of the resulting regularized solution. Numerical experiments demonstrate that the proposed algorithm is effective and efficient, and the choices of the regularization parameters can balance the sparsity of the regularized solution and its approximation to the minimizer of the fidelity function.

Hyperspectral Image Denoising: Reconciling Sparse and Low-Tensor-Ring-Rank Priors in the Transformed Domain

Recently, the transform-based tensor nuclear norm (TNN) framework has yielded promising results for hyperspectral image (HSI) denoising as compared with previous original-domain tensor-based models. However, the TNN framework only exploits the low-rankness of each band of HSIs (tensors) under a single spectral transform. The correlation between all bands under the transform (i.e., the global low-rankness of the transformed tensor) and the sparsity of the transformed HSI, which are beneficial for HSI denoising, is usually neglected in the TNN framework. In this article, we propose to reconcile sparse and low-tensor-ring (TR)-rank priors in the learned transformed domain (called T-RSTR model) for HSI denoising. In T-RSTR, the transform-based low-TR-rank and sparse regularizers are designed to characterize the global low-rankness and sparsity of the transformed tensors, respectively, and then the transform-based low-TR-rank and sparse regularizers are organically integrated and benefit from each other for substantially boosting denoising performance. To tackle the T-RSTR model, we elaborately design a proximal alternating minimization-based algorithm with the theoretical convergence. Extensive numerical results demonstrate that T-RSTR is superior to the competing methods.

Cross-Track Illumination Correction for Hyperspectral Pushbroom Sensor Images Using Low-Rank and Sparse Representations

A hyperspectral pushbroom sensor scans objects line-by-line using a detector array, and a cross-track illumination error (CTIE) exists in the imagery acquired in this way. When the illumination of the individual cells of the detector is not aligned well, or if some of the cells are degraded or old, the acquired images will exhibit nonuniform illumination in the cross-track direction. As additive Gaussian noise is found widely in hyperspectral images (HSIs), we develop a unified mathematical model that describes the image formation process corrupted by the CTIE and additive Gaussian noise. The CTIE produced by line-by-line scanning is replicated and modeled as an offset term with the equivalent values in the direction of flight. The main contribution of this study is the development of a hyperspectral image cross-track illumination correction (HyCIC) method, which corrects the cross-track illumination using column (along-track) mean compensation with total variation and sparsity regularizations, and attenuates the Gaussian noise by using a form of low-rank constraint. The effectiveness of the proposed method is illustrated using semireal data and real HSIs. The performance of the proposed HyCIC is found to be better than other existing methods.

Visual Motion Segmentation in Crowd Videos Based on Spatial-Angular Stacked Sparse Autoencoders

Visual motion segmentation (VMS) is an important and key part of many intelligent crowd systems. It can be used to figure out the flow behavior through a crowd and to spot unusual life-threatening incidents like crowd stampedes and crashes, which pose a serious risk to public safety and have resulted in numerous fatalities over the past few decades. Trajectory clustering has become one of the most popular methods in VMS. However, complex data, such as a large number of samples and parameters, makes it difficult for trajectory clustering to work well with accurate motion segmentation results. This study introduces a spatial-angular stacked sparse autoencoder model (SA-SSAE) with <i>l2</i>-regularization and softmax, a powerful deep learning method for visual motion segmentation to cluster similar motion patterns that belong to the same cluster. The proposed model can extract meaningful high-level features using only spatial-angular features obtained from refined tracklets (a.k.a ‘trajectories’). We adopt <i>l2</i>-regularization and sparsity regularization, which can learn sparse representations of features, to guarantee the sparsity of the autoencoders. We employ the softmax layer to map the data points into accurate cluster representations. One of the best advantages of the SA-SSAE framework is it can manage VMS even when individuals move around randomly. This framework helps cluster the motion patterns effectively with higher accuracy. We put forward a new dataset with its manual ground truth, including 21 crowd videos. Experiments conducted on two crowd benchmarks demonstrate that the proposed model can more accurately group trajectories than the traditional clustering approaches used in previous studies. The proposed SA-SSAE framework achieved a 0.11 improvement in accuracy and a 0.13 improvement in the F-measure compared with the best current method using the CUHK dataset.

Discriminative restricted Boltzmann machine with trainable sparsity

Discriminative restricted Boltzmann machine (DRBM) is a probabilistic three-layered neural network, consisting of the input, hidden, and output layers, that helps to solve classification problems. This study attempts to improve the generalization property of the DRBM. Regularization methods such as L1 or L2 regularizations can be used to control the representation power of a learning model and suppress over-fitting to a dataset. To control the representation power of the DRBM, an alternative regularization approach is proposed, in which sparse regularization is imposed on the values of the hidden variables of the DRBM. In the resultant model, the sparse regularization controls the effective size of the hidden layer of the DRBM. Unlike standard regularization methods, in the proposed model, parameters that control the sparsity strength are trainable. The method is validated through numerical experiments based on benchmark datasets.

Hyperspectral Unmixing Using Higher-Order Graph Regularized NMF With Adaptive Feature Selection

Recently, graph learning methods have attracted much research attention, which uses first-order nearest-neighbor relation between pixels to construct adjacency graphs for capturing smooth abundance. However, the first-order nearest-neighbor information neglects the shared domain structure of pixels making the higher-order nearest-neighbor relation missing resulting in limited spatial structure learning. Additionally, the feature values in hyperspectral images vary greatly in different bands, tiny value contributions are easily neglected due to the statistical properties of the model, which hinders the learning efficiency of the approach. To address these shortcomings, a higher-order graph regularizer nonnegative matrix factorization with adaptive feature selection method is proposed. Specifically, we introduce pixel second-order nearest-neighbor relation in graph learning to capture the nearest-neighbor domains of pixels with missing connection in first-order neighborhoods and enhance the aggregation of homogeneous regions. Then, the first-order and second-order nearest-neighbor relation are combined to construct higher-order graph regularizer to better preserve the global spatial structure of the image. Additionally, adaptive weight is introduced in the data reconstruction to balance the effects of different feature values by adaptively selecting spectral features to enhance the contribution of tiny features. Finally, the objective function is designed by fusing the higher-order graph regularization, adaptive feature selection and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1/2</sub> sparse regularizer into the nonnegative matrix factorization framework, and the optimization algorithm of the model is derived using a multiplicative update rule. We conducted extensive experiments on synthetic and real datasets, and the effectiveness and superiority of the proposed method were verified by comparing it with several state-of-the-art methods.

Sparse regularization based feature selection: A survey

Feature selection, as an essential preprocessing tool, aims to identify a subset of crucial features by eliminating redundant and noisy features according to a predefined criterion. In recent years, sparse learning has received considerable attention. As an important component of sparse learning, sparse regularization has been widely used in feature selection to enforce irrelevant feature coefficients to be small or exactly zero, which is especially suitable for selecting the discriminative and relevant features. In this paper, we attempt to provide a survey on various sparse regularization based methods. We group the existing sparse regularization based methods into two categories, i.e., convex sparse regularization based feature selection and non-convex sparse regularization based feature selection. This survey not only compares the common and different characters of sparse regularizations based feature selection methods, but also offers useful guidance for practitioners of feature selection.