Nonconvex Penalty Research Articles

Enhanced Sparse Regularization Based on Logarithm Penalty and Its Application to Gearbox Compound Fault Diagnosis

A vibration signal observed from a gearbox is normally composed of the vibrations of bearing and gear components as well as the strong background noise. Sparse regularization performs as an effective method to deal with vibration signal denoising and compound fault diagnosis for the gearbox. In this article, a multivariate and non-convex logarithm penalty based on the generalized infimal convolution smoothing (GICS) is deduced. To guarantee the global minimum of the overall cost function, we also derive a convexity condition to prescribe the non-convex penalty. In this way, the optimal sparse solution to the cost function can be calculated by a convex algorithm. Multiplied by respective transform matrices, all fault components in a compound signal can be simultaneously reconstructed, and then each fault characteristic frequency can be extracted. Numerical and experimental analyses demonstrate the effectiveness of the proposed method. Compared with the classical convex $L1$ norm and newly developed non-convex generalized minimax-concave (GMC) penalties, the proposed method is superior in terms of inducing sparsity effectively and enhancing the estimation accuracy of signal components.

Sparse Learning with Non-convex Penalty in Multi-classification

Multi-classification is commonly encountered in data science practice, and it has broad applications in many areas such as biology, medicine, and engineering. Variable selection in multiclass problems is much more challenging than in binary classification or regression problems. In addition to estimating multiple discriminant functions for separating different classes, we need to decide which variables are important for each individual discriminant function as well as for the whole set of functions. In this paper, we address the multi-classification variable selection problem by proposing a new form of penalty, supSCAD, which first groups all the coefficients of the same variable associated with all the discriminant functions altogether and then imposes the SCAD penalty on the supnorm of each group. We apply the new penalty to both soft and hard classification and develop two new procedures: the supSCAD multinomial logistic regression and the supSCAD multi-category support vector machine. Our theoretical results show that, with a proper choice of the tuning parameter, the supSCAD multinomial logistic regression can identify the underlying sparse model consistently and enjoys oracle properties even when the dimension of predictors goes to infinity. Based on the local linear and quadratic approximation to the non-concave SCAD and nonlinear multinomial log-likelihood function, we show that the new procedures can be implemented efficiently by solving a series of linear or quadratic programming problems. Performance of the new methods is illustrated by simulation studies and real data analysis of the Small Round Blue Cell Tumors and the Semeion Handwritten Digit data sets.

Clustering by Orthogonal NMF Model and Non-Convex Penalty Optimization

The non-negative matrix factorization (NMF) model with an additional orthogonality constraint on one of the factor matrices, called the orthogonal NMF (ONMF), has been found a promising clustering model and can outperform the classical K-means. However, solving the ONMF model is a challenging optimization problem because the coupling of the orthogonality and non-negativity constraints introduces a mixed combinatorial aspect into the problem due to the determination of the correct status of the variables (positive or zero). Most of the existing methods directly deal with the orthogonality constraint in its original form via various optimization techniques, but are not scalable for large-scale problems. In this paper, we propose a new ONMF based clustering formulation that equivalently transforms the orthogonality constraint into a set of norm-based non-convex equality constraints. We then apply a non-convex penalty (NCP) approach to add them to the objective as penalty terms, leading to a problem that is efficiently solvable. One smooth penalty formulation and one non-smooth penalty formulation are respectively studied. We build theoretical conditions for the penalized problems to provide feasible stationary solutions to the ONMF based clustering problem, as well as proposing efficient algorithms for solving the penalized problems of the two NCP methods. Experimental results based on both synthetic and real datasets are presented to show that the proposed NCP methods are computationally time efficient, and either match or outperform the existing K-means and ONMF based methods in terms of the clustering performance.

Sharpening Sparse Regularizers via Smoothing

Non-convex sparsity-inducing penalties outperform their convex counterparts, but generally sacrifice the cost function convexity. As a middle ground, we propose the sharpening sparse regularizers (SSR) framework to design non-separable non-convex penalties that induce sparsity more effectively than convex penalties such as $\ell _1$ and nuclear norms, but without sacrificing the cost function convexity. The overall problem convexity is preserved by exploiting the data fidelity relative strong convexity. The framework constructs penalties as the difference of convex functions , namely the difference between convex sparsity-inducing penalties and their smoothed versions. We propose a generalized infimal convolution smoothing technique to obtain the smoothed versions. Furthermore, SSR recovers and generalizes several non-convex penalties in the literature as special cases. The SSR framework is applicable to any sparsity regularized least squares ill-posed linear inverse problem. Beyond regularized least squares, the SSR framework can be extended to accommodate Bregman divergence, and other sparsity structures such as low-rankness. The SSR optimization problem can be formulated as a saddle point problem, and solved by a scalable forward-backward splitting algorithm. The effectiveness of the SSR framework is demonstrated by numerical experiments in different applications.

Improving Network Slimming With Nonconvex Regularization

Convolutional neural networks (CNNs) have developed to become powerful models for various computer vision tasks ranging from object detection to semantic segmentation. However, most of the state-of-the-art CNNs cannot be deployed directly on edge devices such as smartphones and drones, which need low latency under limited power and memory bandwidth. One popular, straightforward approach to compressing CNNs is network slimming, which imposes $\ell _{1}$ regularization on the channel-associated scaling factors via the batch normalization layers during training. Network slimming thereby identifies insignificant channels that can be pruned for inference. In this paper, we propose replacing the $\ell _{1}$ penalty with an alternative nonconvex, sparsity-inducing penalty in order to yield a more compressed and/or accurate CNN architecture. We investigate $\ell _{p} (0 , transformed $\ell _{1}$ ( $\text{T}\ell _{1}$ ), minimax concave penalty (MCP), and smoothly clipped absolute deviation (SCAD) due to their recent successes and popularity in solving sparse optimization problems, such as compressed sensing and variable selection. We demonstrate the effectiveness of network slimming with nonconvex penalties on three neural network architectures – VGG-19, DenseNet-40, and ResNet-164 – on standard image classification datasets. Based on the numerical experiments, $\text{T}\ell _{1}$ preserves model accuracy against channel pruning, $\ell _{1/2, 3/4}$ yield better compressed models with similar accuracies after retraining as $\ell _{1}$ , and MCP and SCAD provide more accurate models after retraining with similar compression as $\ell _{1}$ . Network slimming with $\text{T}\ell _{1}$ regularization also outperforms the latest Bayesian modification of network slimming in compressing a CNN architecture in terms of memory storage while preserving its model accuracy after channel pruning.

Hyperspectral images destriping approach with weighted block sparsity regularization and non-convex low-rank penalty

Hyperspectral images destriping approach with weighted block sparsity regularization and non-convex low-rank penalty

Rolling bearing fault diagnosis based on GMCP sparse enhanced signal decomposition and TFM

This paper presents a rolling bearing fault diagnosis method based on GMCP sparse enhancement signal decomposition and TFM. First, algorithm use the non-convex penalty function that not only increases the sparsity but also maintains the convexity of the cost function, that is, the characteristics of the generalized maximum minimum concave penalty (GMCP) to obtain a more accurate estimate. Then the time-frequency manifold (TFM) is used to process the signal. The advantage of TFM is its powerful noise reduction performance to reduce the noise of the signal. Finally, the fault frequency is extracted to realize the fault diagnosis of the rolling bearing. Through the analysis of experimental data, the excellent performance and application value of the proposed method in the diagnosis of rolling bearings are confirmed.

Non-Convex Penalized Estimation of Count Data Responses via Generalized Linear Model (GLM)

This study provided a non-convex penalized estimation procedure via Smoothed Clipped Absolute Deviation (SCAD) and Minimax Concave Penalty (MCP) for count data responses to checkmate the problem of covariates exceeding the sample size . The Generalized Linear Model (GLM) approach was adopted in obtaining the penalized functions needed by the MCP and SCAD non-convex penalizations of Binomial, Poisson and Negative-Binomial related count responses regression. A case study of the colorectal cancer with six (6) covariates against sample size of five (5) was subjected to the non-convex penalized estimation of the three distributions. It was revealed that the non-convex penalization of Binomial regression via MCP and SCAD best explained four un-penalized covariates needed in determining whether surgical or therapy ideal for treating the turmoil.

A fast DC-based dictionary learning algorithm with the SCAD penalty

In recent years, there has been growing concerns on the study of dictionary learning with the nonconvex sparsity-including penalty. However, how to efficiently address the dictionary learning with the nonconvex penalty is still an open problem. In this paper, we present an efficient DC-based algorithm for dictionary learning with the nonconvex smoothly clipped absolute deviation (SCAD) penalty for strong sparsity and accurate estimation. The optimization problem we considered can be generalized as a minimization of the representation error with the SCAD penalty. The approach we proposed is based on a decomposition scheme which decomposes the whole problem into a set of subproblems with regard to single-vector factors. For handling the nonconvexity of the representation error in the subproblems, we use an alternating optimization scheme to update one factor with the other factor fixed. For tackling the nonconvexity of the SCAD penalty in the subproblems, we apply the Difference of Convex functions (DC) technology to convert the nonconvex subproblem into the resulting convex problems and thus employ DC algorithm to solve the corresponding optimization; thus the simple and straightforward solutions in the closed form can be easily derived. As verified by the numerical experiments with synthetic and real-world data, the proposed algorithm performs better than the state-of-the-art algorithms with different sparsity-including constraints.

Nonconvex Total Generalized Variation Model for Image Inpainting

It is a challenging task to prevent the staircase effect and simultaneously preserve sharp edges in image inpainting. For this purpose, we present a novel nonconvex extension model that closely incorporates the advantages of total generalized variation and edge-enhancing nonconvex penalties. This improvement contributes to achieve the more natural restoration that exhibits smooth transitions without penalizing fine details. To efficiently seek the optimal solution of the resulting variational model, we develop a fast primal-dual method by combining the iteratively reweighted algorithm. Several experimental results, with respect to visual effects and restoration accuracy, show the excellent image inpainting performance of our proposed strategy over the existing powerful competitors.

Structured sparse support vector machine with ordered features

ABSTRACT In the application of high-dimensional data classification, several attempts have been made to achieve variable selection by replacing the -penalty with other penalties for the support vector machine (SVM). However, these high-dimensional SVM methods usually do not take into account the special structure among covariates (features). In this article, we consider a classification problem, where the covariates are ordered in some meaningful way, and the number of covariates p can be much larger than the sample size n. We propose a structured sparse SVM to tackle this type of problems, which combines the non-convex penalty and cubic spline estimation procedure (i.e. penalizing second-order derivatives of the coefficients) to the SVM. From a theoretical point of view, the proposed method satisfies the local oracle property. Simulations show that the method works effectively both in feature selection and classification accuracy. A real application is conducted to illustrate the benefits of the method.

An Accurate Sparse SAR Imaging Method for Enhancing Region-Based Features Via Nonconvex and TV Regularization

With the rapid development of compressed sensing theories and applications, sparse signal processing has been widely used in synthetic aperture radar (SAR) imaging during the recent years. As an efficient tool for sparse reconstruction, ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> optimization induces sparsity the most effectively, and the ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> -norm penalty is usually combined with the total variation norm (TV-norm) penalty to construct a compound regularizer in order to enhance the point-based features as well as the region-based features. However, as a convex optimizer, the analytic solution of ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> regularization-based sparse signal reconstruction is usually a biased estimation. Aiming at this issue, in this article, we quantitatively analyzed the variation of reconstruction bias with respect to the complex reflectivity of targets, the undersampling ratio and the noise power. In order to reduce the bias effect and improve the reconstruction accuracy, we adopted the nonconvex regularization-based sparse SAR imaging method with a nonconvex penalty family. Furthermore, we linearly combined the nonconvex penalty and the TV-norm penalty to form a compound regularizer in the imaging model, which can improve the reconstruction accuracy of distributed targets and maintain the continuity of the backscattering coefficient. Simulation results showed that compared with ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> regularization, nonconvex regularization can reduce the average relative bias from 10.88% to 0.25%; compared with the matched filtering method and ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> and TV regularization, nonconvex & TV regularization can reduce the variance of the uniformly distributed targets by 80% without losing of reconstruction accuracy. Experiments on Gaofen-3 SAR data are also exploited to verify the effectiveness of the proposed method.

Simultaneous Feature Selection and Classification for Data-Adaptive Kernel-Penalized SVM

Simultaneous feature selection and classification have been explored in the literature to extend the support vector machine (SVM) techniques by adding penalty terms to the loss function directly. However, it is the kernel function that controls the performance of the SVM, and an imbalance in the data will deteriorate the performance of an SVM. In this paper, we examine a new method of simultaneous feature selection and binary classification. Instead of incorporating the standard loss function of the SVM, a penalty is added to the data-adaptive kernel function directly to control the performance of the SVM, by firstly conformally transforming the kernel functions of the SVM, and then re-conducting an SVM classifier based on the sparse features selected. Both convex and non-convex penalties, such as least absolute shrinkage and selection (LASSO), moothly clipped absolute deviation (SCAD) and minimax concave penalty (MCP) are explored, and the oracle property of the estimator is established accordingly. An iterative optimization procedure is applied as there is no analytic form of the estimated coefficients available. Numerical comparisons show that the proposed method outperforms the competitors considered when data are imbalanced, and it performs similarly to the competitors when data are balanced. The method can be easily applied in medical images from different platforms.

Sparse Regression: Scalable Algorithms and Empirical Performance

In this paper, we review state-of-the-art methods for feature selection in statistics with an application-oriented eye. Indeed, sparsity is a valuable property and the profusion of research on the topic might have provided little guidance to practitioners. We demonstrate empirically how noise and correlation impact both the accuracy - the number of correct features selected - and the false detection - the number of incorrect features selected - for five methods: the cardinality-constrained formulation, its Boolean relaxation, $\ell_1$ regularization and two methods with non-convex penalties. A cogent feature selection method is expected to exhibit a two-fold convergence, namely the accuracy and false detection rate should converge to $1$ and $0$ respectively, as the sample size increases. As a result, proper method should recover all and nothing but true features. Empirically, the integer optimization formulation and its Boolean relaxation are the closest to exhibit this two properties consistently in various regimes of noise and correlation. In addition, apart from the discrete optimization approach which requires a substantial, yet often affordable, computational time, all methods terminate in times comparable with the \verb|glmnet| package for Lasso. We released code for methods that were not publicly implemented. Jointly considered, accuracy, false detection and computational time provide a comprehensive assessment of each feature selection method and shed light on alternatives to the Lasso-regularization which are not as popular in practice yet.

Fault detection of planetary subassemblies in a wind turbine gearbox using TQWT based sparse representation

Planetary subassemblies in wind turbine gearbox are subject to compound faults due to harsh environment and complex structure. Disturbed by the meshing vibration from higher-speed transmission stage and intensive noise, fault diagnosis of planetary subassemblies with low rotational speed is challenging. In this paper, a tunable Q-factor wavelet transform based sparse representation method is proposed, which integrates the property of tunable Q wavelet transform, non-convex penalty and noise optimization into sparse decomposition. This method makes it possible to accurately decompose vibration signal from faulty planetary subassemblies into two resonance components and noise, relying less on the setting of regularization parameters due to the noise restriction. It is easier to detect potential fault information in decomposed low or high resonance component than in the original signal. Further, normalized multi-stage enveloping spectrogram is presented to reveal the fault characteristic frequencies of planetary gears and bearings even though they are weak. The effectiveness of the proposed methods is verified by the analysis of a simulated faulty signal and an on-site case from one 850 kW wind turbine gearbox.

Nonconvex Haar-TV denoising

The anisotropic total variation (TV) denoising model suppresses noise for two-dimensional signals that are vertically and horizontally piecewise constant. However, two-dimensional signals may have sparse derivatives in other directions. We propose a modification of the classical anisotropic two-dimensional TV regularizer from a spectral point of view. In the frequency domain, the TV regularizer can be considered as penalizing the high-frequency component of original signals and promoting only low-frequency components. The classical anisotropic TV, which applies l1-norm on vertical and horizontal differences, suppresses high-frequency components of the signals. The proposed operator, named Haar total variation (Haar-TV), penalizes two-dimensional signals that have more varied high-frequency regions. Furthermore, we propose non-convex penalties based on the Haar-TV operator since non-convex penalties can preserve edges and thus enhance the quality of the estimation. We derive a condition that preserves the strong convexity of the total cost function so the global minimizer can be reached.

A Modified Higher-Order Singular Value Decomposition Framework With Adaptive Multilinear Tensor Rank Approximation for Three-Dimensional Magnetic Resonance Rician Noise Removal.

The magnetic resonance (MR) images are acknowledged to be inevitably corrupted by Rician distributed noise, which adversely affected the image quality for diagnosis purpose. However, the traditional denoising methods may recover the images from corruptions with severe loss of detailed structure and edge information, which would affect the lesion detections and diagnostic decision making. In this study, we challenged improving the Rician noise removal from three-dimensional (3D) MR volumetric data through a modified higher-order singular value decomposition (MHOSVD) method. The proposed framework of MHOSVD involved a parameterized logarithmic nonconvex penalty function for low-rank tensor approximation (LRTA) algorithm optimization to suppress the image noise in MR dataset. Reference cubes were extracted from the noisy image volume, and block matching was performed according to nonlocal similarity for a fourth-order tensor construction. Then the LRTA problem was implemented by tensor factorization approaches, and the ranks of unfolding matrices along different modes of the tensor were estimated utilizing an adaptive nonconvex low-rank method. The denoised MR images were finally restored through aggregating all recovered cubes. We investigated the proposed algorithm MHOSVD on both the synthetic and real clinic 3D MR images for Rician noise removal, and relative results demonstrated that the MHOSVD can recover images with fine structures and detailed edge preservation with heavy noise even as high as 15% of the maximum intensity. The experimental results were also compared along with several classical denoising methods; the MHOSVD exhibited a sufficient improvement in noise-removal performance at various noise conditions in terms of different measurement indices such as peak signal-to-noise ratio and structural similarity index metrics. Based upon the comparison, the proposed MHOSVD has proved a relative state-of-the-art performance with excellent detailed structure reservation.

Analytical and Simple Form of Shrinkage Functions for Non-Convex Penalty Functions in Fused Lasso Algorithm

In some circumstances, the performance of machine learning (ML) tasks are based on the quality of signal (data) that is processed in these tasks. Therefore, the pre-processing techniques, such as reconstruction and denoising methods, are important techniques in ML tasks. In reconstructed (estimated) method, the fused lasso algorithm with non-convex penalty function is an efficient method when the signal corrupted by additive white Gaussian noise (AWGN) is considered. Therefore, this paper proposes new shrinkage functions for non-convex penalty functions, modified arctangent and exponential models, in fused lasso formulation. A lot of works present the shrinkage function for arctangent penalty function. Unfortunately, there is no closed-form solution. The numerical solution is required for shrinkage function of this penalty function. However, the analytical solution is derived in this paper. Moreover, the shrinkage function of modified exponential penalty function is proposed. This shrinkage function obtains from simple iterative method, fixed-point algorithm. We demonstrate the proposed methods through simulations with standard one-dimensional signals contaminated by AWGN. The proposed techniques are compared with traditional estimation methods, such as total variation (TV) and wavelet denoising methods. In experimental results, our proposed methods outperform several exiting methods both visual quality and in terms of root mean square error (RMSE). In fact, the proposed methods can better preserve the feature of noise-free signal than the compared methods. The denoised signals produced by the proposed methods are less smooth than the denoised signals produced by the compared methods.

Sparse Principal Component Analysis via Fractional Function Regularity

In this paper, we describe a novel approach to sparse principal component analysis (SPCA) via a nonconvex sparsity-inducing fraction penalty function SPCA (FP-SPCA). Firstly, SPCA is reformulated as a fraction penalty regression problem model. Secondly, an algorithm corresponding to the model is proposed and the convergence of the algorithm is guaranteed. Finally, numerical experiments were carried out on a synthetic data set, and the experimental results show that the FP-SPCA method is more adaptable and has a better performance in the tradeoff between sparsity and explainable variance than SPCA.

Inertial proximal gradient methods with Bregman regularization for a class of nonconvex optimization problems

This paper proposes an inertial Bregman proximal gradient method for minimizing the sum of two possibly nonconvex functions. This method includes two different inertial steps and adopts the Bregman regularization in solving the subproblem. Under some general parameter constraints, we prove the subsequential convergence that each generated sequence converges to the stationary point of the considered problem. To overcome the parameter constraints, we further propose a nonmonotone line search strategy to make the parameter selections more flexible. The subsequential convergence of the proposed method with line search is established. When the line search is monotone, we prove the stronger global convergence and linear convergence rate under Kurdyka–Łojasiewicz framework. Moreover, numerical results on SCAD and MCP nonconvex penalty problems are reported to demonstrate the effectiveness and superiority of the proposed methods and line search strategy.