Nonsmooth Nonconvex Minimization Research Articles

An algorithm for the minimization of nonsmooth nonconvex functions using inexact evaluations and its worst-case complexity

An adaptive regularization algorithm using inexact function and derivatives evaluations is proposed for the solution of composite nonsmooth nonconvex optimization. It is shown that this algorithm needs at most $$O(|\log (\epsilon )|\,\epsilon ^{-2})$$ evaluations of the problem’s functions and their derivatives for finding an $$\epsilon $$ -approximate first-order stationary point. This complexity bound therefore generalizes that provided by Bellavia et al. (Theoretical study of an adaptive cubic regularization method with dynamic inexact Hessian information. arXiv:1808.06239 , 2018) for inexact methods for smooth nonconvex problems, and is within a factor $$|\log (\epsilon )|$$ of the optimal bound known for smooth and nonsmooth nonconvex minimization with exact evaluations. A practically more restrictive variant of the algorithm with worst-case complexity $$O(|\log (\epsilon )|+\epsilon ^{-2})$$ is also presented.

Approaching nonsmooth nonconvex minimization through second-order proximal-gradient dynamical systems

We investigate the asymptotic properties of the trajectories generated by a second-order dynamical system of proximal-gradient type stated in connection with the minimization of the sum of a nonsmooth convex and a (possibly nonconvex) smooth function. The convergence of the generated trajectory to a critical point of the objective is ensured provided a regularization of the objective function satisfies the Kurdyka–Łojasiewicz property. We also provide convergence rates for the trajectory formulated in terms of the Łojasiewicz exponent.

Combination of steepest descent and BFGS methods for nonconvex nonsmooth optimization

In this paper, a method is developed for solving nonsmooth nonconvex minimization problems. This method extends the classical BFGS framework. First, we generalize the Wolfe conditions for locally Lipschitz functions and prove that this generalization is well defined. Then, a line search algorithm is presented to find a step length satisfying the generalized Wolfe conditions. Next, the Goldstein e-subgradient is approximated by an iterative method and a descent direction is computed using a positive definite matrix. This matrix is updated using the BFGS method. Finally, a minimization algorithm based on the BFGS method is described. The algorithm is implemented in MATLAB and numerical results using it are reported.

A regularized matrix factorization approach to induce structured sparse-low-rank solutions in the EEG inverse problem

We consider the estimation of the Brain Electrical Sources (BES) matrix from noisy electroencephalographic (EEG) measurements, commonly named as the EEG inverse problem. We propose a new method to induce neurophysiological meaningful solutions, which takes into account the smoothness, structured sparsity, and low rank of the BES matrix. The method is based on the factorization of the BES matrix as a product of a sparse coding matrix and a dense latent source matrix. The structured sparse-low-rank structure is enforced by minimizing a regularized functional that includes the ℓ21-norm of the coding matrix and the squared Frobenius norm of the latent source matrix. We develop an alternating optimization algorithm to solve the resulting nonsmooth-nonconvex minimization problem. We analyze the convergence of the optimization procedure, and we compare, under different synthetic scenarios, the performance of our method with respect to the Group Lasso and Trace Norm regularizers when they are applied directly to the target matrix.

Approximated Function Based Spectral Gradient Algorithm for Sparse Signal Recovery

Numerical algorithms for the l0-norm regularized non-smooth non-convex minimization problems have recently became a topic of great interest within signal processing, compressive sensing, statistics, and machine learning. Nevertheless, the l0-norm makes the problem combinatorial and generally computationally intractable. In this paper, we construct a new surrogate function to approximate l0-norm regularization, and subsequently make the discrete optimization problem continuous and smooth. Then we use the well-known spectral gradient algorithm to solve the resulting smooth optimization problem. Experiments are provided which illustrate this method is very promising.

Approximated Function Based Spectral Gradient Algorithm for Sparse Signal Recovery

Numerical algorithms for the l0-norm regularized non-smooth non-convex minimization problems have recently became a topic of great interest within signal processing, compressive sensing, statistics, and machine learning. Nevertheless, the l0-norm makes the problem combinatorial and generally computationally intractable. In this paper, we construct a new surrogate function to approximate l0-norm regularization, and subsequently make the discrete optimization problem continuous and smooth. Then we use the well-known spectral gradient algorithm to solve the resulting smooth optimization problem. Experiments are provided which illustrate this method is very promising.

A Generalized Robust Minimization Framework for Low-Rank Matrix Recovery

This paper considers the problem of recovering low-rank matrices which are heavily corrupted by outliers or large errors. To improve the robustness of existing recovery methods, the problem is solved by formulating it as a generalized nonsmooth nonconvex minimization functional via exploiting the Schattenp-norm(0 < p ≤1)andLq(0 < q ≤1)seminorm. Two numerical algorithms are provided based on the augmented Lagrange multiplier (ALM) and accelerated proximal gradient (APG) methods as well as efficient root-finder strategies. Experimental results demonstrate that the proposed generalized approach is more inclusive and effective compared with state-of-the-art methods, either convex or nonconvex.

A splitting bundle approach for non-smooth non-convex minimization

We present a bundle-type method for minimizing non-convex non-smooth functions. Our approach is based on the partition of the bundle into two sets, taking into account the local convex or concave behaviour of the objective function. Termination at a point satisfying an approximate stationarity condition is proved and numerical results are provided.

An Efficient Algorithm for Multiphase Image Segmentation With Intensity Bias Correction

This paper presents a variational model for simultaneous multiphase segmentation and intensity bias estimation for images corrupted by strong noise and intensity inhomogeneity. Since the pixel intensities are not reliable samples for region statistics due to the presence of noise and intensity bias, we use local information based on the joint density within image patches to perform image partition. Hence, the pixel intensity has a multiplicative distribution structure. Then, the maximum-a-posteriori (MAP) principle with those pixel density functions generates the model. To tackle the computational problem of the resultant nonsmooth nonconvex minimization, we relax the constraint on the characteristic functions of partition regions, and apply primal-dual alternating gradient projections to construct a very efficient numerical algorithm. We show that all the variables have closed-form solutions in each iteration, and the computation complexity is very low. In particular, the algorithm involves only regular convolutions and pointwise projections onto the unit ball and canonical simplex. Numerical tests on a variety of images demonstrate that the proposed algorithm is robust, stable, and attains significant improvements on accuracy and efficiency over the state-of-the-arts.

Non-Lipschitz $\ell_{p}$-Regularization and Box Constrained Model for Image Restoration

Nonsmooth nonconvex regularization has remarkable advantages for the restoration of piecewise constant images. Constrained optimization can improve the image restoration using a priori information. In this paper, we study regularized nonsmooth nonconvex minimization with box constraints for image restoration. We present a computable positive constant θ for using nonconvex nonsmooth regularization, and show that the difference between each pixel and its four adjacent neighbors is either 0 or larger than θ in the recovered image. Moreover, we give an explicit form of θ for the box-constrained image restoration model with the non-Lipschitz nonconvex l(p)-norm regularization. Our theoretical results show that any local minimizer of this imaging restoration problem is composed of constant regions surrounded by closed contours and edges. Numerical examples are presented to validate the theoretical results, and show that the proposed model can recover image restoration results very well.

Efficient Reconstruction of Piecewise Constant Images Using Nonsmooth Nonconvex Minimization

We consider the restoration of piecewise constant images where the number of the regions and their values are not fixed in advance, with a good difference of piecewise constant values between neighboring regions, from noisy data obtained at the output of a linear operator (e.g., a blurring kernel or a Radon transform). Thus we also address the generic problem of unsupervised segmentation in the context of linear inverse problems. The segmentation and the restoration tasks are solved jointly by minimizing an objective function (an energy) composed of a quadratic data-fidelity term and a nonsmooth nonconvex regularization term. The pertinence of such an energy is ensured by the analytical properties of its minimizers. However, its practical interest used to be limited by the difficulty of the computational stage which requires a nonsmooth nonconvex minimization. Indeed, the existing methods are unsatisfactory since they (implicitly or explicitly) involve a smooth approximation of the regularization term and often get stuck in shallow local minima. The goal of this paper is to design a method that efficiently handles the nonsmooth nonconvex minimization. More precisely, we propose a continuation method where one tracks the minimizers along a sequence of approximate nonsmooth energies $\{J_\eps\}$, the first of which being strictly convex and the last one the original energy to minimize. Knowing the importance of the nonsmoothness of the regularization term for the segmentation task, each $J_\eps$ is nonsmooth and is expressed as the sum of an $\ell_1$ regularization term and a smooth nonconvex function. Furthermore, the local minimization of each $J_{\eps}$ is reformulated as the minimization of a smooth function subject to a set of linear constraints. The latter problem is solved by the modified primal-dual interior point method, which guarantees the descent direction at each step. Experimental results are presented and show the effectiveness and the efficiency of the proposed method. Comparison with simulated annealing methods further shows the advantage of our method.