Nonconvex Regularization Research Articles

Proximal methods for reweighted l-regularization of sparse signal recovery

Abstract To recover a sparse signal from a noised linear measurement system A x = b + e , convex lp regularization methods (i.e., 1 ≤ p p = 1 ) are commonly used under certain conditions. Recently, however, more attentions have been paid to nonconvex lq regularization methods (i.e., 0 q = 1 / 2 ) for recovering a sparse signal. In this paper, we use proximal methods to study both convex and nonconvex reweighted lQ regularization for recovering a sparse signal. Convex lQ regularization is introduced by S. Voronin and I. Daubechies [19]. We extend it to the nonconvex case and our results therefore supplement those of Voronin and Daubechies [19] . We also study Nesterov’s acceleration method for the nonconvex case. Our numerical experiments show that nonconvex lQ regularization can more effectively recover sparse signals.

DigGCN: Learning Compact Graph Convolutional Networks via Diffusion Aggregation.

Recent interests in graph neural networks (GNNs) have received increasing concerns due to their superior ability in the network embedding field. The GNNs typically follow a message passing scheme and represent nodes by aggregating features from neighbors. However, the current aggregation methods assume that the network structure is static and define the local receptive fields under visible connections, which consequently fails to consider latent or high-order structures. Besides, the aggregation methods are known to have a depth dilemma due to the over-smoothness issues. To solve the above shortcomings, we present in this article a compact graph convolutional network framework which defines the graph receptive fields based on diffusion paths and explicitly compresses the neural networks with sparsity regularization. The proposed model seeks to learn from invisible connections and recover the latent proximity. First, we infer the high-order proximity and construct diffusion paths by diffusion samplings. Compared with random walk samplings, the diffusion samplings are based on regions instead of paths. The network inference then obtains accurate weights that can be leveraged to build small but informative receptive fields with salient neighbors. Second, to utilize the deep information while avoiding overfitting, we propose learning a lightweight model by introducing a nonconvex regularizer. Numerical comparisons with the existing network embedding methods under unsupervised feature learning and supervised classification show the effectiveness of our model.

On Solving SAR Imaging Inverse Problems Using Nonconvex Regularization With a Cauchy-Based Penalty

Synthetic aperture radar (SAR) imagery can provide useful information in a multitude of applications, including climate change, environmental monitoring, meteorology, high dimensional mapping, ship monitoring, or planetary exploration. In this article, we investigate solutions for several inverse problems encountered in SAR imaging. We propose a convex proximal splitting method for the optimization of a cost function that includes a nonconvex Cauchy-based penalty. The convergence of the overall cost function optimization is ensured through careful selection of model parameters within a forward-backward (FB) algorithm. The performance of the proposed penalty function is evaluated by solving three standard SAR imaging inverse problems, including super-resolution, image formation, and despeckling, as well as ship wake detection for maritime applications. The proposed method is compared to several methods employing classical penalty functions such as total variation (TV) and L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> norms, and to the generalized minimax-concave (GMC) penalty. We show that the proposed Cauchy-based penalty function leads to better image reconstruction results when compared to the reference penalty functions for all SAR imaging inverse problems in this article.

Compressed sensing with nonconvex sparse regularization and convex analysis for duct mode detection

Recently, research interests are increasing in mode detection methods based on compressed sensing, as it can reduce the number of sensors required by the classical Shannon-Nyquist sampling theory. This paper proposes a nonconvex sparse regularization method for azimuthal mode detection for aero engine fan noise. The nonconvex sparse regularization is based on the generalized minimax-concave (GMC) penalty, which can maintain the convexity of the sparse-regularized least squares cost function, and thus the global optimal solution can be solved by convex optimization algorithms. The main advantage of the GMC method over conventional compressed sensing method in mode detection is that the GMC method can better recover the mode amplitudes with a small number of sensors. Besides, the GMC method can suppress effectively the irrelated modes induced by the background noise or sensor installation errors. Therefore, the proposed method for duct mode detection can significantly improve the accuracy of the detected modes. Numerical simulations and experimental tests verify the effectiveness of the GMC method in mode detection for aero engine fan noise, and comparison studies show that the GMC method provides more accurate mode detection results than l1 minimization in the category of compressed sensing, as well as traditional mode detection methods.

A data-driven group-sparse feature extraction method for fault detection of wind turbine transmission system

Vibration monitoring using sensors mounted on machines is widely used for rotating machinery fault diagnosis. The periodic overlapping group sparsity (POGS) method has been developed in previous work of authors, and is an effective technique for detecting faults induced in rotating machines. However, the regularization parameter of the POGS problem is roughly specified via a look-up table provided in the original work. To address this problem, a data-driven diagnostic method, which is termed the adaptively regularized periodic overlapping group sparsity (ARPOGS), is proposed in this paper. The non-stationary fault feature ratio which is defined in the Hilbert domain is employed to guide the optimal regularization parameter. The criterion of setting the interval of candidate regularization parameters is also discussed. The ARPOGS is developed in terms of a convex optimization problem, while non-convex regularizations are used to further promote the sparsity. Since the non-convex penalty term is used and the whole objective function is constrained as a convex optimization problem, the sparsity of useful fault features is maximally induced. A simulated signal is formulated to verify the performance of the proposed method for periodic feature extraction. Finally, the effectiveness of the proposed ARPOGS method is validated by analyzing real data collected from a wind turbine transmission system. The results demonstrated that the proposed method can effectively and automatically extract periodic-group-sparse features from noisy vibration signals.

On constrained optimization with nonconvex regularization

In many engineering applications, it is necessary to minimize smooth functions plus penalty (or regularization) terms that violate smoothness and convexity. Specific algorithms for this type of problems are available in recent literature. Here, a smooth reformulation is analyzed and equivalence with the original problem is proved both from the points of view of global and local optimization. Moreover, for the cases in which the objective function is much more expensive than the constraints, model-intensive algorithms, accompanied by their convergence and complexity theories, are introduced. Finally, numerical experiments are presented.

A nonconvex [formula omitted] model for image restoration with impulse noise

In this paper, we propose a model for image restoration with blur and impulse noise, it is composed of l1 data fitting term and a nonconvex regularization term, which is the weighted difference of l1-norm and l2-norm based on wavelet frame. The combined model is difficult to solve by the corresponding Euler Lagrangian equations, here we solve it by alternating direction method of multipliers (ADMM). We describe the detailed process of the algorithm, and establish the convergence of the algorithm. The experimental outcomes on different blurs and different impulse noises demonstrate that the proposed approach is better than those existing methods in terms of standard signal-to-noise ratio (PSNR), relative error (ReErr), and visual quality.

Block diagonal representation learning for robust subspace clustering

Subspace clustering groups a set of data into their underlying subspaces according to the low-dimensional subspace structure of data. The performance of spectral clustering-based approaches heavily depends on the learned block diagonal structure of the affinity matrix. However, this structure is fragile in the presence of noise within data. As such, the clustering performance is degraded significantly. On the other hand, in practice, we often do not have a prior knowledge of error distribution at all, which results in that we cannot model the error with suitable norms. To this end, in this paper, we propose a robust block diagonal representation learning for subspace clustering. Specifically, a non-convex regularizer is directly utilized to constrain the affinity matrix for exploiting the block diagonal structure. Furthermore, we use a penalty matrix to adaptively weight the reconstruction error so that we can handle noise without prior knowledge. We also devise an effective method to compute the parameters related to this matrix, reducing the complexity of the parameter trains. Experimental results show that our method outperformed the state-of-the-art methods on both synthetic data and real-world datasets.

Meta-Analysis Based on Nonconvex Regularization

The widespread applications of high-throughput sequencing technology have produced a large number of publicly available gene expression datasets. However, due to the gene expression datasets have the characteristics of small sample size, high dimensionality and high noise, the application of biostatistics and machine learning methods to analyze gene expression data is a challenging task, such as the low reproducibility of important biomarkers in different studies. Meta-analysis is an effective approach to deal with these problems, but the current methods have some limitations. In this paper, we propose the meta-analysis based on three nonconvex regularization methods, which are L1/2 regularization (meta-Half), Minimax Concave Penalty regularization (meta-MCP) and Smoothly Clipped Absolute Deviation regularization (meta-SCAD). The three nonconvex regularization methods are effective approaches for variable selection developed in recent years. Through the hierarchical decomposition of coefficients, our methods not only maintain the flexibility of variable selection and improve the efficiency of selecting important biomarkers, but also summarize and synthesize scientific evidence from multiple studies to consider the relationship between different datasets. We give the efficient algorithms and the theoretical property for our methods. Furthermore, we apply our methods to the simulation data and three publicly available lung cancer gene expression datasets, and compare the performance with state-of-the-art methods. Our methods have good performance in simulation studies, and the analysis results on the three publicly available lung cancer gene expression datasets are clinically meaningful. Our methods can also be extended to other areas where datasets are heterogeneous.

False Lipschitz Penalty Sparse Low-Rank Matrix and Chaotic Bionic Optimization for Prognosis of Bearing Degradation

Accurately predicting the degradation trend of a pivotal component is essential for scheduling maintenance decisions, avoiding abrupt shutdowns of mechanical equipment, and improving overall systems reliability. This article proposes a hybrid data-driven approach for bearing degradation prognosis that utilizes sparse low-rank matrix (SLRM) and chaotic bionic optimization algorithms based upon bearings health indicator time series (HITS). First, by mean of SLRM method, the HITS of bearings can be adaptively decomposed into low frequency trend component (LFC) and high frequency oscillation component (HFC), respectively. A novel non-convex regularizer named false Lipschitz penalty is designed, and the strict convexity of the proposed cost function can be ensured. Meanwhile, an efficient iterative algorithm using the forward–backward splitting technique is presented. Second, a metaheuristic algorithm named chaotic cuckoo search (CCS) is utilized to optimize the key parameters of the support vector machine (SVM) model, and the decomposed LFC and HFC are, respectively, predicted via the CCS-SVM approach. The final predicted time series is obtained after a summation operation. The proposed hybrid approach is evaluated using accelerated degradation test data of rolling bearings. Experimental results demonstrate the effectiveness and superiority of the proposed approach in comparison with five benchmark methods.

Matrix completion with nonconvex regularization: spectral operators and scalable algorithms

In this paper, we study the popularly dubbed matrix completion problem, where the task is to "fill in" the unobserved entries of a matrix from a small subset of observed entries, under the assumption that the underlying matrix is of low-rank. Our contributions herein, enhance our prior work on nuclear norm regularized problems for matrix completion (Mazumder et al., 2010) by incorporating a continuum of nonconvex penalty functions between the convex nuclear norm and nonconvex rank functions. Inspired by SOFT-IMPUTE (Mazumder et al., 2010; Hastie et al., 2016), we propose NC-IMPUTE- an EM-flavored algorithmic framework for computing a family of nonconvex penalized matrix completion problems with warm-starts. We present a systematic study of the associated spectral thresholding operators, which play an important role in the overall algorithm. We study convergence properties of the algorithm. Using structured low-rank SVD computations, we demonstrate the computational scalability of our proposal for problems up to the Netflix size (approximately, a $500,000 \times 20, 000$ matrix with $10^8$ observed entries). We demonstrate that on a wide range of synthetic and real data instances, our proposed nonconvex regularization framework leads to low-rank solutions with better predictive performance when compared to those obtained from nuclear norm problems. Implementations of algorithms proposed herein, written in the R programming language, are made available on github.

Non-convex sparse regularization for impact force identification

Many convex regularization methods, such as the classical Tikhonov regularization based on l2-norm penalty and the standard sparse regularization method based on l1-norm penalty, have been widely investigated for impact force identification. However, in many practical applications, these regularization methods commonly underestimate the true solution. In this paper, we propose a non-convex sparse regularization method based on lp-norm (0 < p < 1) penalty, to seek the sufficiently sparse and highly accurate solution of impact force identification. Firstly, a non-convex optimization model based on lp-norm penalty instead of l2-norm penalty or l1-norm penalty is developed for regularizing inverse problems of impact force identification to overcome the mismatch between l0-norm and l1-norm regularizations. Secondly, an iteratively reweighed l1-norm algorithm is introduced to solve such a non-convex model through transforming it into a series of l1-norm regularizations. Finally, numerical simulation and experimental validation are carried out to evaluate the effectiveness and performance of lp-norm regularization when 0 = p ≤ 1. Our numerical results demonstrate that, compared to the state-of-the-art Tikhonov regularization and l1-norm regularization, the solution of lp-norm regularization achieves more stability, sparseness and accuracy when dealing with the heavily noisy response. Additionally, both numerical and experimental results demonstrate that the peak relative error of the identified impact force using lp-norm regularization has a decreasing tendency as p is approaching 0; while the identification of lp-norm regularization with 0 = p ≤ 1/2 bears no significant difference, which always outperforms that the identification with 1/2 = p ≤ 1.

Separating variables to accelerate non-convex regularized optimization

In this paper, a novel variable separation algorithm stemmed from the idea of orthogonalization EM is proposed to find the minimization of general function with non-convex regularizer. The main idea of our algorithm is to construct a new function by adding an item that allows minimization to be solved separately on each component. Several attractive theoretical properties concerning the new algorithm are established. The new algorithm converges to one of the critical points with the condition that the objective function is coercive or the generated sequence is in a compact set. The convergence rate of the algorithm is also obtained. The Barzilai–Borwein (BB) rule and Nesterov’s method are also used to accelerate our algorithm. The new algorithm can also be used to solve the minimization of general function with group structure regularizer. The simulation and real data results show that these methods can accelerate our method obviously.

Prognosis of Bearing Degeneration Using Adaptive Quaternion Least Mean Biquadrate Under Framework of Hypercomplex Data

Prognostics of bearing degeneration play a crucial role in implementation of systems maintenance strategies. Although numerous classical prognostic models have been developed, these models are carried out based on channel-wise processing and therefore fail to capture the inherent nonlinear and coupling nature of multi-dimensional or multi-channel data. To address this issue, a novel prognostics method based on adaptive quaternion-valued least-mean biquadrate (AQ-LMB) algorithm is proposed for a unified processing of hypercomplex data by the virtue of quaternion algebra. The cost function of the proposed AQ-LMB algorithm is designed via the biquadrate form of system output error to adapt to the more common nonlinear and non-Gaussian data, and the update weight vector is derived through Hamilton calculus instead of traditional complex gradient calculation. First, the time series of health indicators (e.g., root-mean-square, RMS) derived from historical data are decomposed by a nonconvex sparse regularization (SR) algorithm associated with a nonconvex penalty, that is, the low frequency trend component (LFC) and high frequency noise component (HFC) are obtained. Then both LFC and HFC are respectively predicted by the AQ-LMB algorithm. The final predicted health indicators can be obtained by integrating the correspondingly predicted LFC and HFC. The separate analysis of both sub-components makes it possible to distinguish their respective contributions to the entire degeneration process, thus avoiding false deviation and improving the prediction accuracy. Finally, the effectiveness of the proposed nonconvex SR and AQ-LMB approach in improving prognostic accuracy is illustrated via three-and four-dimensional run-to-failure datasets of the rolling bearings.

Sparsity-based fractional spline wavelet denoising via overlapping group shrinkage with non-convex regularization and convex optimization for bearing fault diagnosis

Vibration monitoring has become a relatively common method for the fault diagnosis of rolling bearings in recent years. Vibration-based fault diagnosis technologies essentially rely upon accurate fault signature extraction from noisy vibration signals. To this end, this paper presents a sparsity-assisted denoising method using the fractional spline wavelet transform (FrSWT) and overlapping group shrinkage (OGS) with non-convex regularization and convex optimization (OGSNCRCO) to diagnose bearing faults. The FrSWT is used to promote sparsity of the wavelet coefficients for bearing fault signals. The OGSNCRCO balances data consistency and sparsity by using non-convex regularization and convex optimization, and is applied to shrink the wavelet coefficients of noisy signals in the fractional spline wavelet domain to extract bearing fault signals. Simulations and experiments validate the efficiency of the proposed method for fault diagnosis of rolling bearings. The results of analysis suggest that the proposed method is superior to other state-of-art techniques also commonly used for extracting fault signatures of rolling bearings, including the L1-norm denoising method (FrSWT with L1-norm regularization) and spectral kurtosis method.

Sparse multiple instance learning with non-convex penalty

Multiple instance learning (MIL) usually plays an important role in weakly labeled learning. It is able to reduce cost noted by accurate annotations and remove noisy samples with useless labels in large scale databases. However, in many real applications, the training data always contains many redundant features. In order to remove those residual features and improve the computational efficiency, some convex and nonconvex regularization terms are applied. In this paper we apply the MCP function to propose an efficient sparse modle SyMIL, and then use the proximal stochastic subgradient method (PSSG) to solve it. The convergence analysis of the algorithm is given in details. Extensive experiments show that our model can reduce redundant features effectively while ensuring the accuracy. Quantitative comparisons against several regularization terms demonstrate the advantages of the MCP regularizer in this model. Moreover, PSSG can be observed to have slightly faster convergence rate than some other methods.

Self-calibrated brain network estimation and joint non-convex multi-task learning for identification of early Alzheimer's disease.

Detection of early stages of Alzheimer's disease (AD) (i.e., mild cognitive impairment (MCI)) is important to maximize the chances to delay or prevent progression to AD. Brain connectivity networks inferred from medical imaging data have been commonly used to distinguish MCI patients from normal controls (NC). However, existing methods still suffer from limited performance, and classification remains mainly based on single modality data. This paper proposes a new model to automatically diagnosing MCI (early MCI (EMCI) and late MCI (LMCI)) and its earlier stages (i.e., significant memory concern (SMC)) by combining low-rank self-calibrated functional brain networks and structural brain networks for joint multi-task learning. Specifically, we first develop a new functional brain network estimation method. We introduce data quality indicators for self-calibration, which can improve data quality while completing brain network estimation, and perform correlation analysis combined with low-rank structure. Second, functional and structural connected neuroimaging patterns are integrated into our multi-task learning model to select discriminative and informative features for fine MCI analysis. Different modalities are best suited to undertake distinct classification tasks, and similarities and differences among multiple tasks are best determined through joint learning to determine most discriminative features. The learning process is completed by non-convex regularizer, which effectively reduces the penalty bias of trace norm and approximates the original rank minimization problem. Finally, the most relevant disease features classified using a support vector machine (SVM) for MCI identification. Experimental results show that our method achieves promising performance with high classification accuracy and can effectively discriminate between different sub-stages of MCI.

Non-convex Total Variation Regularization for Convex Denoising of Signals

Total variation (TV) signal denoising is a popular nonlinear filtering method to estimate piecewise constant signals corrupted by additive white Gaussian noise. Following a ‘convex non-convex’ strategy, recent papers have introduced non-convex regularizers for signal denoising that preserve the convexity of the cost function to be minimized. In this paper, we propose a non-convex TV regularizer, defined using concepts from convex analysis, that unifies, generalizes, and improves upon these regularizers. In particular, we use the generalized Moreau envelope which, unlike the usual Moreau envelope, incorporates a matrix parameter. We describe a novel approach to set the matrix parameter which is essential for realizing the improvement we demonstrate. Additionally, we describe a new set of algorithms for non-convex TV denoising that elucidate the relationship among them and which build upon fast exact algorithms for classical TV denoising.

Moreau-Enhanced Total Variation and Subspace Factorization for Hyperspectral Denoising

Hyperspectral images (HSIs) denoising aims at recovering noise-free images from noisy counterparts to improve image visualization. Recently, various prior knowledge has attracted much attention in HSI denoising, e.g., total variation (TV), low-rank, sparse representation, and so on. However, the computational cost of most existing algorithms increases exponentially with increasing spectral bands. In this paper, we fully take advantage of the global spectral correlation of HSI and design a unified framework named subspace-based Moreau-enhanced total variation and sparse factorization (SMTVSF) for multispectral image denoising. Specifically, SMTVSF decomposes an HSI image into the product of a projection matrix and abundance maps, followed by a ‘Moreau-enhanced’ TV (MTV) denoising step, i.e., a nonconvex regularizer involving the Moreau envelope mechnisam, to reconstruct all the abundance maps. Furthermore, the schemes of subspace representation penalizing the low-rank characteristic and ℓ 2 , 1 -norm modelling the structured sparse noise are embedded into our denoising framework to refine the abundance maps and projection matrix. We use the augmented Lagrange multiplier (ALM) algorithm to solve the resulting optimization problem. Extensive results under various noise levels of simulated and real hypspectral images demonstrate our superiority against other competing HSI recovery approaches in terms of quality metrics and visual effects. In addition, our method has a huge advantage in computational efficiency over many competitors, benefiting from its removal of most spectral dimensions during iterations.

Sparse Reconstruction for Enhancement of the Empirical Mode Decomposition-Based Signal Denoising

Effective signal denoising methods are essential for science and engineering. In general, denoising algorithms may be either linear or non-linear. Most of the linear ones are unable to remove the noise from the real-world measurements. More suitable methods are usually based on non-linear approaches. One of the possible algorithms to signal denoising is based on empirical mode decomposition. The typical approach to the empirical mode decomposition-based signal denoising is the partial reconstruction. More recently, a new concept inspired by the wavelet thresholding principle was proposed. The method is named the interval thresholding. In this article, we further extend the concept by the application of the sparse reconstruction to the empirical mode decomposition-based signal denoising algorithm. To this end, we state and then solve the problem of signal denoising as a regularization problem. In the article, we consider three cases, that is, three types of penalty functions. The first algorithm is combining total variation denoising with empirical mode decomposition approach. In the second one, we applied the fused LASSO Signal Approximator to design the empirical mode decomposition-based signal denoising algorithm. The third approach solves the denoising problem by applying a non-convex sparse regularization. The proposed algorithms were validated on synthetic and real-world signals. We found that the proposed methods have the ability to improve the accuracy of the signal denoising in comparison to the reference methods. Significant improvements from both the synthetic and the real-world signals were obtained for the algorithm based on non-convex sparse regularization. The presented results show that the proposed approach to signal denoising based on empirical mode decomposition algorithm and sparse regularization gives a great improvement of accuracy, and it is the promising direction of future research.