Non-convex Method Research Articles

Learning Proximal Operator Methods for Nonconvex Sparse Recovery with Theoretical Guarantee

Sparse recovery has attracted considerable attention in signal processing community these years, because of its widespread usage in many applications. Though lots of convex and nonconvex methods have been proposed for this topic, most of them are computationally expensive. Recently, learned iterative shrinkage-thresholding algorithm (LISTA) and its variants are incorporated to sparse recovery, which are based on specific deep neural networks designed by unfolding ISTA. However, the problem they are able to solve is regularized by 11 norm, which is less effective at promoting sparseness than some nonconvex sparsity-inducing penalties (SIPs). Motivated by the fact that the problems regularized by these SIPs can be solved by proximal operator methods, we devise a network named learning proximal operator method (LePOM) for sparse recovery. For LePOM on a general class of SIPs, a necessary condition of its convergence is established. Based on this condition, a simplified network is proposed with less parameters. Theoretical analysis of this network inspires us to further reduce the number of parameters and arrive at proposing Analytical LePOM (ALePOM). ALePOM determines most parameters by solving an optimization problem and significantly reduces the number of parameters. Theoretical analysis shows if the signal is sufficiently sparse, ALePOM converges linearly. Simulations confirm our analyses and demonstrate that the proposed solutions outperform state-of-the-art sparse recovery algorithms and neural network-based methods.

Sampling and Inference of Networked Dynamics Using Log-Koopman Nonlinear Graph Fourier Transform

Networked nonlinear dynamics underpin the complex functionality of many engineering, social, biological, and ecological systems. Monitoring the networked dynamics via the minimum subset of nodes is essential for a variety of scientific and operational purposes. When there is a lack of a explicit model and networked signal space, traditional evolution analysis and non-convex methods are insufficient. An important data-driven state-of-the-art method use the Koopman operator to generate a linear evolution model for a vector-valued observable of original state-space. As a result, one can derive a sampling strategy via the linear evolution property of observable. However, current polynomial Koopman operators result in a large sampling space due to: (i) the large size of polynomial based observables ($O(N^2)$, $N$ number of nodes in network), and (ii) not factoring in the nonlinear dependency between observables. In this work, to achieve linear scaling ($O(N)$) and a small set of sampling nodes, we propose to combine a novel Log-Koopman operator and nonlinear Graph Fourier Transform (NL-GFT) scheme. First, the Log-Koopman operator is able to reduce the size of observables by transforming multiplicative poly-observable to logarithm summation. Second, a nonlinear GFT concept and sampling theory are provided to exploit the nonlinear dependence of observables for Koopman linearized evolution analysis. Combined, the sampling and reconstruction algorithms are designed and demonstrated on two established application areas. The results demonstrate that the proposed Log-Koopman NL-GFT scheme can (i) linearize unknown nonlinear dynamics using $O(N)$ observables, and (ii) achieve lower number of sampling nodes, compared with the state-of-the art polynomial Koopman linear evolution analysis.

Vector-Valued Graph Trend Filtering with Non-Convex Penalties.

This work studies the denoising of piecewise smooth graph signals that exhibit inhomogeneous levels of smoothness over a graph, where the value at each node can be vector-valued. We extend the graph trend filtering framework to denoising vector-valued graph signals with a family of non-convex regularizers, which exhibit superior recovery performance over existing convex regularizers. Using an oracle inequality, we establish the statistical error rates of first-order stationary points of the proposed non-convex method for generic graphs. Furthermore, we present an ADMM-based algorithm to solve the proposed method and establish its convergence. Numerical experiments are conducted on both synthetic and real-world data for denoising, support recovery, event detection, and semi-supervised classification.

Hybrid non-convex second-order total variation with applications to non-blind image deblurring

Total variation is a very popular method for image restoration, yet it produces undesirable staircase artifacts. In this paper, a combined second-order non-convex total variation with overlapping group sparse regularizer for staircase artifact removal is proposed. The non-convex higher-order TV regularizer is introduced to model the observation that the staircase effect must be appropriately smoothen out in the restored image while preserving its edges. However, using the non-convex higher-order TV alone tends to smoothen the image while amplifying speckle artifacts. To deal with this, the overlapping group sparse regularizer is added to balance the effects produced by the non-convex higher-order TV regularizer. An efficient re-weighted $$\ell _1$$ alternating direction method is formulated to solve the corresponding iterative scheme. Comparative analysis with three algorithms, namely overlapping group sparse total variation, total generalized variation and the fast non-smooth non-convex method, with two different blur kernels is carried out. The results from peak signal-to-noise ratio and structural similarity index measure show the effectiveness of our proposed method when compared to the mentioned algorithms.

A fractional 0–1 program for task assignment with respect to preferences

We consider a task assignment problem with respect to preferences. The objective is to maximize total weighted satisfaction while maintaining a fair distribution. The problem’s mathematical model turns out to be a multiple-ratio, constrained, fractional 0–1 program since satisfaction is defined as a ratio of two linear functions. We discuss the computational complexity of the problem, give three equivalent mixed-integer linear formulations, and propose a naive heuristic the usefulness of which is demonstrated by a numeric study. This paper not only provides a new application of fractional 0–1 programming, but it also suggests an alternative solution approach for a similar problem dealt in the literature before with nonlinear nonconvex methods.

Convergence rate of SVM for kernel-based robust regression

It is known that to alleviate the performance deterioration caused by the outliers, the robust support vector (SV) regression is proposed, which is essentially a convex optimization problem associated with a non-convex loss function. The theory analysis for its performance cannot be finished by the usual convex analysis approach. For a robust SV regression algorithm containing two homotopy parameters, a non-convex method is developed with the quasiconvex analysis theory and the error estimate is given. An explicit convergence rate is provided, and the effect degree of outliers on the performance is quantitatively shown.

A Non-Euclidean Gradient Descent Framework for Non-Convex Matrix Factorization

We study convex optimization problems that feature low-rank matrix solutions. In such scenarios, non-convex methods offer significant advantages over convex methods due to their lower space complexity, as well as practical faster convergence. Under mild assumptions, these methods feature global convergence guarantees. In this paper, we extend the results on this matter by following a different path. We derive a non-Euclidean optimization framework in the non-convex setting that takes nonlinear gradient steps on the factors. Our framework enables the possibility to further exploit the underlying problem structures, such as sparsity or low-rankness on the factorized domain, or better dimensional dependence of the smoothness parameters of the objectives. We prove that the non-Euclidean methods enjoy the same rigorous guarantees as their Euclidean counterparts under appropriate assumptions. Numerical evidence with Fourier ptychography and FastText applications, using real data, shows that our approach can enhance solution quality, as well as convergence speed over the standard non-convex approaches.

Robust low-rank kernel multi-view subspace clustering based on the Schatten p-norm and correntropy

Multi-view subspace clustering, which studies the similarities and differences among data in multiple views, is an efficient clustering problem. However, to deal with problems that have non-linear structures and non-Gaussian noise in multi-view data, existing clustering methods relax the original problem convexly. The solutions that are generated by these convex relaxations are not the optimal solutions to the original problem. To overcome this deficiency, this paper presents a robust low-rank kernel multi-view subspace clustering approach that combines the non-convex Schatten p-norm (0<p≤1) regularizer with the “kernel trick”, which can efficiently deal with problems that have non-linear structures in multi-view data via non-convex methods. In addition, the correntropy is introduced into our model, which is a robust measure of the corruptions that are caused by non-Gaussian noise. Moreover, our method can learn the joint subspace representation of all views. Because it learns a low-rank kernel mapping, the data in the feature space are both low-rank and self-expressed. The optimization problems are solved efficiently via an iterative algorithm (HQ-ADMM). This algorithm can ensure that each iteration has closed-form solutions, which simplifies the optimization problems substantially. Experimental comparisons on five real-world datasets demonstrate that the proposed algorithm outperforms state-of-the-art multi-view subspace clustering algorithms.

Identifiability of Multichannel Blind Deconvolution and Nonconvex Regularization Algorithm

In this paper, we consider the multichannel blind deconvolution problem, where we observe the output of channels $\mathbf {h}_{i}\in \mathbb {R}^n$ ( $i=1,...,N$ ) that all convolve with the same unknown input signal $\mathbf {x}\in \mathbb {R}^n$ . We wish to estimate the input signal and blur kernels simultaneously. Existing theoretical results showed that the original inputs are identifiable under subspace assumptions. However, the subspaces discussed before were randomly or generically chosen. Here, we propose deterministic subspace assumption, which is widely used in practice, and give some theoretical results. First of all, we derive tight sufficient condition for identifiability of signal and convolution kernels, which is only violated on a set of Lebesgue measure zero. Then, we present a nonconvex regularization algorithm by a lifting method and approximate the rank-one constraint via the difference of nuclear norm and Frobenius norm. The global minimizer of the proposed nonconvex algorithm is rank-one matrix under mild conditions on parameters and noise level. The stability result is also shown under the assumption that the inputs lie in a compact set. Besides, the computation of our regularization model is carried out and any limit point of iterations converges to a stationary point of our model. Finally, we provide numerical experiments to show that our nonconvex regularization model outperforms convex relaxation models, such as nuclear norm minimization and some nonconvex methods, such as alternating minimization method and spectral method.

Provable compressed sensing quantum state tomography via non-convex methods

With nowadays steadily growing quantum processors, it is required to develop new quantum tomography tools that are tailored for high-dimensional systems. In this work, we describe such a computational tool, based on recent ideas from non-convex optimization. The algorithm excels in the compressed sensing setting, where only a few data points are measured from a low-rank or highly-pure quantum state of a high-dimensional system. We show that the algorithm can practically be used in quantum tomography problems that are beyond the reach of convex solvers, and, moreover, is faster and more accurate than other state-of-the-art non-convex approaches. Crucially, we prove that, despite being a non-convex program, under mild conditions, the algorithm is guaranteed to converge to the global minimum of the quantum state tomography problem; thus, it constitutes a provable quantum state tomography protocol.

Sparse non-convex Lp regularization for cone-beam X-ray luminescence computed tomography

ABSTRACTCone-beam X-ray luminescence computed tomography (CB-XLCT) is an attractive hybrid imaging modality, and it has the potential of monitoring the metabolic processes of nanophosphors-based drugs in vivo. However, the XLCT imaging suffers from a severe ill-posed problem. In this work, a sparse nonconvex Lp (0 < p < 1) regularization was utilized for the efficient reconstruction for early detection of small tumour in CB-XLCT imaging. Specifically, we transformed the non-convex optimization problem into an iteratively reweighted scheme based on the L1 regularization. Further, an iteratively reweighted split augmented lagrangian shrinkage algorithm (IRW_SALSA-Lp) was proposed to efficiently solve the non-convex Lp (0 < p < 1) model. We studied eight different non-convex p-values (1/16, 1/8, 1/4, 3/8, 1/2, 5/8, 3/4, 7/8) in both 3D digital mouse experiments and in vivo experiments. The results demonstrate that the proposed non-convex methods outperform L2 and L1 regularization in accurately recovering sparse targets in CB-XLCT. And among all the non-convex p-values, our Lp(1/4 < p < 1/2) methods give the best performance.

Quadratic Majorization for Nonconvex Loss with Applications to the Boosting Algorithm

ABSTRACTClassical robust statistical methods dealing with noisy data are often based on modifications of convex loss functions. In recent years, nonconvex loss-based robust methods have been increasingly popular. A nonconvex loss can provide robust estimation for data contaminated with outliers. The significant challenge is that a nonconvex loss can be numerically difficult to optimize. This article proposes quadratic majorization algorithm for nonconvex (QManc) loss. The QManc can decompose a nonconvex loss into a sequence of simpler optimization problems. Subsequently, the QManc is applied to a powerful machine learning algorithm: quadratic majorization boosting algorithm (QMBA). We develop QMBA for robust classification (binary and multi-category) and regression. In high-dimensional cancer genetics data and simulations, the QMBA is comparable with convex loss-based boosting algorithms for clean data, and outperforms the latter for data contaminated with outliers. The QMBA is also superior to boosting when directly implemented to optimize nonconvex loss functions. Supplementary material for this article is available online.

A non-smooth and non-convex regularization method for limited-angle CT image reconstruction

Abstract Restricted by the practical applications and radiation exposure of computed tomography (CT), the obtained projection data is usually incomplete, which may lead to a limited-angle reconstruction problem. Whereas reconstructing an object from limited-angle projection views is a challenging and ill-posed inverse problem. Fortunately, the regularization methods offer an effective way to deal with that. Recently, several researchers are absorbed in ℓ 1 {\ell_{1}} regularization to address such problem, but it has some problems for suppressing the limited-angle slope artifacts around edges due to incomplete projection data. In this paper, in order to surmount the ill-posedness, a non-smooth and non-convex method that is based on ℓ 0 {\ell_{0}} and ℓ 1 {\ell_{1}} regularization is presented to better deal with the limited-angle problem. Firstly, the splitting technique is utilized to deal with the presented approach called LWPC-ST-IHT. Afterwards, some propositions and convergence analysis of the presented approach are established. Numerical implementations show that our approach is more capable of suppressing the slope artifacts compared with the classical and state of the art iterative reconstruction algorithms.

Bundle-based descent method for nonsmooth multiobjective DC optimization with inequality constraints

Multiobjective DC optimization problems arise naturally, for example, in data classification and cluster analysis playing a crucial role in data mining. In this paper, we propose a new multiobjective double bundle method designed for nonsmooth multiobjective optimization problems having objective and constraint functions which can be presented as a difference of two convex (DC) functions. The method is of the descent type and it generalizes the ideas of the double bundle method for multiobjective and constrained problems. We utilize the special cutting plane model angled for the DC improvement function such that the convex and the concave behaviour of the function is captured. The method is proved to be finitely convergent to a weakly Pareto stationary point under mild assumptions. Finally, we consider some numerical experiments and compare the solutions produced by our method with the method designed for general nonconvex multiobjective problems. This is done in order to validate the usage of the method aimed specially for DC objectives instead of a general nonconvex method.

Data based modelling of driveability for internal combustion engines

ABSTRACTNon-convex hull concepts are used to describe shapes or images in data processing. The most widely used non-convex hull methods were introduced to visualize two- or three-dimensional data obtained via 3D scans or GPS signals. Data hulls have also been used as feasibility models in automotive control systems. For these higher-dimensional problems the computational complexity bears an important factor for data processing software. Commonly used methods, such as the convex hull, involve triangulations, which are numerically intractable in practice for dimensions above 10. Simple, spatial models can be computed and evaluated fast, using and floating point operations, respectively, but they perform poorly as feasibility classifiers if the data is close to infeasible regions. This article introduces a class of conic hulls that can be calculated for any input dimension with sub-quadratic, and evaluated with sub-linear, CPU complexity, respectively. These hulls are designed to include screening information of the measurement procedures embedded in the data generating processes.

Finding Low-Rank Solutions via Nonconvex Matrix Factorization, Efficiently and Provably

A rank-$r$ matrix $X \in \mathbb{R}^{m \times n}$ can be written as a product $U V^\top$, where $U \in \mathbb{R}^{m \times r}$ and $V \in \mathbb{R}^{n \times r}$. One could exploit this observation in optimization: e.g., consider the minimization of a convex function $f(X)$ over rank-$r$ matrices, where the set of low-rank matrices is modeled via $UV^\top$. Though such parameterization reduces the number of variables and is more computationally efficient (of particular interest is the case $r \ll \min\{m, n\}$), it comes at a cost: $f(U V^\top)$ becomes a nonconvex function w.r.t. $U$ and $V$. We study such parameterization on generic convex objectives $f$ and focus on first-order, gradient descent algorithms. We propose the bifactored gradient descent (\textttBFGD) algorithm, an efficient first-order method that operates directly on the $U, V$ factors. We show that when $f$ is (restricted) smooth, \textttBFGD has local sublinear convergence; when $f$ is both (restricted) smooth and (restricted) strongly convex, it has local linear convergence. For several applications, we provide simple and efficient initialization schemes that provide initial conditions, good enough for the above convergence results to hold, globally. Extensive experimental results support our arguments that \textttBFGD is an efficient and accurate nonconvex method, compared to state-of-the-art approaches.

A New Approach for Sparse Signal Recovery in Compressed Sensing Based on Minimizing Composite Trigonometric Function

Accurate signal recovery from an underdetermined system of linear equation (USLE) is a topic of considerable interest; such as compressed sensing (CS), recovery of low-rank matrix, blind source separation, and related fields. In order to improve the accuracy of signal recovery from an USLE in CS, we develop a new algorithm called composite trigonometric function null-space re-weighted approximate ${\ell _{0}}$ -norm (CTNRAL0). The proposed algorithm deploys composite trigonometric function as a nonconvex penalty for sparsity which can better approximate ${\ell _{0}}$ -norm and can yield more accurate solution. To solve nonconvex minimization formulation efficiently, we adopt a gradual nonconvexity method. In addition, the null space measurement matrix is applied in the CTNRAL0 algorithm, which reduces the dimension of the matrix. The proposed algorithm has been compared with the smoothed ${\ell _{0}}$ -norm and null-space re-weighted approximate ${\ell _{0}}$ -norm algorithm via numerical simulations to show its improved performance in the noise environment, while computation cost required is comparable. Furthermore, compared with the interior-point LP solvers and iteratively re-weighted least squares algorithm, the proposed algorithm computation cost can reduce by 1 or 2 orders of magnitude.

Pattern Discovery in Brain Imaging Genetics via SCCA Modeling with a Generic Non-convex Penalty

Brain imaging genetics intends to uncover associations between genetic markers and neuroimaging quantitative traits. Sparse canonical correlation analysis (SCCA) can discover bi-multivariate associations and select relevant features, and is becoming popular in imaging genetic studies. The L1-norm function is not only convex, but also singular at the origin, which is a necessary condition for sparsity. Thus most SCCA methods impose {ell }_{{bf{1}}}-norm onto the individual feature or the structure level of features to pursuit corresponding sparsity. However, the {ell }_{{bf{1}}}-norm penalty over-penalizes large coefficients and may incurs estimation bias. A number of non-convex penalties are proposed to reduce the estimation bias in regression tasks. But using them in SCCA remains largely unexplored. In this paper, we design a unified non-convex SCCA model, based on seven non-convex functions, for unbiased estimation and stable feature selection simultaneously. We also propose an efficient optimization algorithm. The proposed method obtains both higher correlation coefficients and better canonical loading patterns. Specifically, these SCCA methods with non-convex penalties discover a strong association between the APOE e4 rs429358 SNP and the hippocampus region of the brain. They both are Alzheimer’s disease related biomarkers, indicating the potential and power of the non-convex methods in brain imaging genetics.

Augmented Lagrangian alternating direction method for low-rank minimization via non-convex approximation

This paper concerns the low-rank minimization problems which consist of finding a matrix of minimum rank subject to linear constraints. Many existing approaches, which used the nuclear norm as a convex surrogate of the rank function, usually result in a suboptimal solution. To seek a tighter rank approximation, we develop a non-convex surrogate to approximate the rank function based on the Laplace function. An iterative algorithm based on the augmented Lagrangian multipliers method is developed. Empirical studies for practical applications including robust principal component analysis and low-rank representation demonstrate that our proposed algorithm outperforms many other state-of-the-art convex and non-convex methods developed recently in the literature.

The Overcomplete Dictionary-Based Directional Estimation Model and Nonconvex Reconstruction Methods.

In this paper, it is proposed the directional estimation model on the overcomplete dictionary, which bridges the compressed measurements of the image blocks and the directional structures of the dictionary. In the model, it is established the analytical method to estimate the structure type of a block as either smooth, single-oriented, or multioriented. Furthermore, the structures of each type of blocks are described by the structured subdictionaries. Then based on the obtained estimations and the constrains on the sparse dictionaries, the original image will be estimated. To verify the model, the nonconvex methods are designed for compressed sensing. Specifically, the greedy pursuit-based methods are established to search the subdictionaries obtained by the model, which achieve better local structural estimation than the methods without the directional estimation. More importantly, it is proposed the nonconvex image reconstruction method with direction-guided dictionaries and evolutionary searching strategies (NR_DG), where the evolutionary searching strategies are delicately designed for each type of the blocks based on the directional estimation. By the experimental results, it is shown that the NR_DG method performs better than the available two-stage evolutionary reconstruction method.