non-Lipschitz Optimization Problem Research Articles

An Iterative Threshold Algorithm of Log-Sum Regularization for Sparse Problem

The log-sum function as a penalty has always been drawing widespread attention in the field of sparse problems. However, it brings a non-convex, non-smooth and non-Lipschitz optimization problem that is difficult to tackle. To overcome the problem, an iterative threshold algorithm for the sparse optimization problems with log-sum function is proposed in this paper. For brevity, the sparse optimization problems with log-sum function are named log-sum regularization. Firstly, by introducing an intermediate function to construct another new function, a property theorem about solution for log-sum regularization is established. Secondly, based on the above theorem, the optimal setting rules of the compromising parameters are elaborated, and an iterative log-sum threshold algorithm is proposed. Thirdly, under the situation that the compromising parameters of log-sum regularization are relatively small, it can be proven that the proposed algorithm converges to a local minimizer of log-sum regularization. Finally, a series of simulations are implemented to examine performance of the proposed algorithm, and the results exhibit that the proposed algorithm outperforms the state-of-the-art algorithms.

Robust amplitude method with <inline-formula><tex-math id="M1">$ L_{1/2} $</tex-math></inline-formula>-regularization for compressive phase retrieval

Compressive phase retrieval aims to recover a sparse signal from the amplitude of its linear measurements when the number of measurements required is far less than the dimension of the signal. In real applications, the measurements are often corrupted by outliers and asymmetrical distribution noise. In this paper, we introduce a novel method for compressive phase retrieval which consists of least absolute deviation loss function and an $ L_{1/2}- $regularization term. It is a nonconvex, nonsmooth, and non-Lipschitz optimization problem. We design an efficient alternating direction method of multipliers(ADMM) to solve the problem and establish its convergence. Extensive numerical experiments demonstrate that the proposed method has higher success probability when the number of measurements is relatively less and is robust to outliers, dense bounded noise as well as Laplace noise.

An Interior Stochastic Gradient Method for a Class of Non-Lipschitz Optimization Problems

An Interior Stochastic Gradient Method for a Class of Non-Lipschitz Optimization Problems

<i>L</i><sub>1/2 </sub>-Regularized Quantile Method for Sparse Phase Retrieval

The sparse phase retrieval aims to recover the sparse signal from quadratic measurements. However, the measurements are often affected by outliers and asymmetric distribution noise. This paper introduces a novel method that combines the quantile regression and the L1/2-regularizer. It is a non-convex, non-smooth, non-Lipschitz optimization problem. We propose an efficient algorithm based on the Alternating Direction Methods of Multiplier (ADMM) to solve the corresponding optimization problem. Numerous numerical experiments show that this method can recover sparse signals with fewer measurements and is robust to dense bounded noise and Laplace noise.

Isotropic non-Lipschitz regularization for sparse representations of random fields on the sphere

In this paper, we consider an infinite-dimensional isotropic non-Lipschitz optimization problem with ℓ 2 , p \ell _{2,p} ( 0 > p > 1 0>p>1 ) regularizer for random fields on the unit sphere with spherical harmonic representations. The regularizer not only gives a group sparse solution, but also preserves the isotropy of the regularized random field represented by the solution. We present first order and second order necessary optimality conditions for local minimizers of the optimization problem. We also derive two lower bounds for the nonzero groups of stationary points, which are used to prove that the infinite-dimensional optimization problem can be reduced to a finite-dimensional problem. Moreover, we propose an iteratively reweighted algorithm for the finite-dimensional problem and prove its convergence. Finally, numerical experiments on Cosmic Microwave Background data are presented to show the efficiency of the non-Lipschitz regularization.

Smoothing Neural Network for Non‐Lipschitz Optimization with Linear Inequality Constraints

This paper presents a smoothing neural network to solve a class of non-Lipschitz optimization problem with linear inequality constraints. The proposed neural network is modelled with a differential inclusion equation, which introduces the smoothing approximate techniques. Under certain conditions, we prove that the trajectory of neural network reaches the feasible region in finite time and stays there thereafter, and that any accumulation point of the solution is a stationary point of the original optimization problem. Furthermore, if all stationary points of the optimization problem are isolated, then the trajectory converges to a stationary point of the optimization problem. Two typical numerical examples are given to verify the effectiveness of the proposed neural network.

Projected Neural Network for a Class of Non-Lipschitz Optimization Problems With Linear Constraints.

In this article, we consider a class of nonsmooth, nonconvex, and non-Lipschitz optimization problems, which have wide applications in sparse optimization. We generalize the Clarke stationary point and define a kind of generalized stationary point of the problems with a stronger optimal capability. Based on the smoothing method, we propose a projected neural network for solving this kind of optimization problem. Under the condition that the level set of objective function in the feasible region is bounded, we prove that the solution of the proposed neural network is globally existent and bounded. The uniqueness of the solution of the proposed network is also analyzed. When the feasible region is bounded, any accumulation point of the proposed neural network is a generalized stationary point of the optimization model. Based on some suitable conditions, any solution of the proposed neural network is asymptotic convergent to one stationary point. In particular, we give some deep analysis on the proposed network for solving a special class of the non-Lipschitz optimization problem, which indicates a lower bound property and the unify identification for the nonzero elements of all accumulation points. Finally, some numerical results are presented to show the efficiency of the proposed neural network for solving some kinds of sparse optimization models.

Global optimality condition and fixed point continuation algorithm for non-Lipschitz ℓ p regularized matrix minimization

Regularized minimization problems with nonconvex, nonsmooth, even non-Lipschitz penalty functions have attracted much attention in recent years, owing to their wide applications in statistics, control, system identification and machine learning. In this paper, the non-Lipschitz $\ell_p~(0<p<1)$ regularized matrix minimization problem is studied. A global necessary optimality condition for this non-Lipschitz optimization problem is firstly obtained, specifically, the global optimal solutions for the problem are fixed points of the so-called $p$-thresholding operator which is matrix-valued and set-valued. Then a fixed point iterative scheme for the non-Lipschitz model is proposed, and the convergence analysis is also addressed in detail. Moreover, some acceleration techniques are adopted to improve the performance of this algorithm. The effectiveness of the proposed $p$-thresholding fixed point continuation ($p$-FPC) algorithm is demonstrated by numerical experiments on randomly generated and real matrix completion problems.

Sparse Support Vector Machine with L p Penalty for Feature Selection

We study the strategies in feature selection with sparse support vector machine (SVM). Recently, the socalled Lp-SVM (0 < p < 1) has attracted much attention because it can encourage better sparsity than the widely used L1-SVM. However, Lp-SVM is a non-convex and non-Lipschitz optimization problem. Solving this problem numerically is challenging. In this paper, we reformulate the Lp-SVM into an optimization model with linear objective function and smooth constraints (LOSC-SVM) so that it can be solved by numerical methods for smooth constrained optimization. Our numerical experiments on artificial datasets show that LOSC-SVM (0 < p < 1) can improve the classification performance in both feature selection and classification by choosing a suitable parameter p. We also apply it to some real-life datasets and experimental results show that it is superior to L1-SVM.

Alternating Direction Method of Multipliers for a Class of Nonconvex and Nonsmooth Problems with Applications to Background/Foreground Extraction

In this paper, we study a general optimization model, which covers a large class of existing models for many applications in imaging sciences. To solve the resulting possibly nonconvex, nonsmooth, and non-Lipschitz optimization problem, we adapt the alternating direction method of multipliers (ADMM) with a general dual step-size to solve a reformulation that contains three blocks of variables and analyze its convergence. We show that for any dual step-size less than the golden ratio, there exists a computable threshold such that if the penalty parameter is chosen above such a threshold and the sequence thus generated by our ADMM is bounded, then the cluster point of the sequence gives a stationary point of the nonconvex optimization problem. We achieve this via a potential function specifically constructed for our ADMM. Moreover, we establish the global convergence of the whole sequence if, in addition, this special potential function is a Kurdyka--Łojasiewicz function. Furthermore, we present a simple st...

Penalty Methods for a Class of Non-Lipschitz Optimization Problems

We consider a class of constrained optimization problems with a possibly nonconvex non-Lipschitz objective and a convex feasible set being the intersection of a polyhedron and a possibly degenerate ellipsoid. Such problems have a wide range of applications in data science, where the objective is used for inducing sparsity in the solutions while the constraint set models the noise tolerance and incorporates other prior information for data fitting. To solve this class of constrained optimization problems, a common approach is the penalty method. However, there is little theory on exact penalization for problems with nonconvex and non-Lipschitz objective functions. In this paper, we study the existence of exact penalty parameters regarding local minimizers, stationary points, and $\epsilon$-minimizers under suitable assumptions. Moreover, we discuss a penalty method whose subproblems are solved via a nonmonotone proximal gradient method with a suitable update scheme for the penalty parameters and prove the convergence of the algorithm to a KKT point of the constrained problem. Preliminary numerical results demonstrate the efficiency of the penalty method for finding sparse solutions of underdetermined linear systems.

Convergence of ℓ2/3 Regularization for Sparse Signal Recovery

In this paper, we consider the problem of finding the sparsest solution to underdetermined linear systems. Unlike the literatures which use the ℓ1 regularization to approximate the original problem, we consider the ℓ2/3 regularization which leads to a better approximation but a nonconvex, nonsmooth, and non-Lipschitz optimization problem. Through developing a fixed point representation theory associated with the two thirds thresholding operator for ℓ2/3 regularization solutions, we propose a fixed point iterative thresholding algorithm based on two thirds norm for solving the k-sparsity problems. Relying on the restricted isometry property, we provide subsequentional convergence guarantee for this fixed point iterative thresholding algorithm on recovering a sparse signal. By discussing the preferred regularization parameters and studying the phase diagram, we get an adequate and efficient algorithm for the high-dimensional sparse signal recovery. Finally, comparing with the existing algorithms, such as the standard ℓ1 minimization, the iterative reweighted ℓ2 minimization, the iterative reweighted ℓ1 minimization, and iterative Half thresholding algorithm, we display the results of the experiment which indicate that the two thirds norm fixed point iterative thresholding algorithm applied to sparse signal recovery and large scale imageries from noisy measurements can be accepted as an effective solver for ℓ2/3 regularization.

An Interior Point Method forL1/2-SVM and Application to Feature Selection in Classification

This paper studies feature selection for support vector machine (SVM). By the use of theL1/2regularization technique, we propose a new modelL1/2-SVM. To solve this nonconvex and non-Lipschitz optimization problem, we first transform it into an equivalent quadratic constrained optimization model with linear objective function and then develop an interior point algorithm. We establish the convergence of the proposed algorithm. Our experiments with artificial data and real data demonstrate that theL1/2-SVM model works well and the proposed algorithm is more effective than some popular methods in selecting relevant features and improving classification performance.

L1/2 regularization: a thresholding representation theory and a fast solver.

The special importance of L1/2 regularization has been recognized in recent studies on sparse modeling (particularly on compressed sensing). The L1/2 regularization, however, leads to a nonconvex, nonsmooth, and non-Lipschitz optimization problem that is difficult to solve fast and efficiently. In this paper, through developing a threshoding representation theory for L1/2 regularization, we propose an iterative half thresholding algorithm for fast solution of L1/2 regularization, corresponding to the well-known iterative soft thresholding algorithm for L1 regularization, and the iterative hard thresholding algorithm for L0 regularization. We prove the existence of the resolvent of gradient of ||x||1/2(1/2), calculate its analytic expression, and establish an alternative feature theorem on solutions of L1/2 regularization, based on which a thresholding representation of solutions of L1/2 regularization is derived and an optimal regularization parameter setting rule is formulated. The developed theory provides a successful practice of extension of the well- known Moreau's proximity forward-backward splitting theory to the L1/2 regularization case. We verify the convergence of the iterative half thresholding algorithm and provide a series of experiments to assess performance of the algorithm. The experiments show that the half algorithm is effective, efficient, and can be accepted as a fast solver for L1/2 regularization. With the new algorithm, we conduct a phase diagram study to further demonstrate the superiority of L1/2 regularization over L1 regularization.

Smoothing neural network for constrained non-Lipschitz optimization with applications.

In this paper, a smoothing neural network (SNN) is proposed for a class of constrained non-Lipschitz optimization problems, where the objective function is the sum of a nonsmooth, nonconvex function, and a non-Lipschitz function, and the feasible set is a closed convex subset of . Using the smoothing approximate techniques, the proposed neural network is modeled by a differential equation, which can be implemented easily. Under the level bounded condition on the objective function in the feasible set, we prove the global existence and uniform boundedness of the solutions of the SNN with any initial point in the feasible set. The uniqueness of the solution of the SNN is provided under the Lipschitz property of smoothing functions. We show that any accumulation point of the solutions of the SNN is a stationary point of the optimization problem. Numerical results including image restoration, blind source separation, variable selection, and minimizing condition number are presented to illustrate the theoretical results and show the efficiency of the SNN. Comparisons with some existing algorithms show the advantages of the SNN.