Nonsmooth Minimization Problems Research Articles

Convergence rate of inertial forward–backward algorithms based on the local error bound condition

Abstract The ‘inertial forward–backward algorithm’ (IFB) is a powerful algorithm for solving a class of convex non-smooth minimization problems, IFB relies on an inertial parameter $\gamma _{k}$ whose tuning is crucial for achieving accelerated convergence speeds as compared to the classical forward–backward algorithm. Under the local error bound condition, it is known that IFB converges R-linearly as soon as the inertial parameter satisfies ${\sup _{k}}{\gamma _{k}} \leqslant \tilde{\gamma } <1.$ On the contrary, we are not aware of any convergence result for the case ${\sup _{k}}{\gamma _{k}} = 1.$ In this paper, we consider six different choices of inertial parameters satisfying this last condition, and show convergence of the corresponding IFB algorithms under the local error bound condition. Finally, we propose a class of inertial forward–backward algorithm with an adaptive modification (IFB_AdapM) and show that it enjoys the same convergence results.

Truncated nonsmooth Newton multigrid for phase-field brittle-fracture problems, with analysis

AbstractWe propose the truncated nonsmooth Newton multigrid method (TNNMG) as a solver for the spatial problems of the small-strain brittle-fracture phase-field equations. TNNMG is a nonsmooth multigrid method that can solve biconvex, block-separably nonsmooth minimization problems with linear time complexity. It exploits the variational structure inherent in the problem, and handles the pointwise irreversibility constraint on the damage variable directly, without regularization or the introduction of a local history field. In the paper we introduce the method and show how it can be applied to several established models of phase-field brittle fracture. We then prove convergence of the solver to a solution of the nonsmooth Euler–Lagrange equations of the spatial problem for any load and initial iterate. On the way, we show several crucial convexity and regularity properties of the models considered here. Numerical comparisons to an operator-splitting algorithm show a considerable speed increase, without loss of robustness.

A Bregman stochastic method for nonconvex nonsmooth problem beyond global Lipschitz gradient continuity

ABSTRACT In this paper, we consider solving a broad class of large-scale nonconvex and nonsmooth minimization problems by a Bregman proximal stochastic gradient (BPSG) algorithm. The objective function of the minimization problem is the composition of a differentiable and a nondifferentiable function, and the differentiable part does not admit a global Lipschitz continuous gradient. Under some suitable conditions, the subsequential convergence of the proposed algorithm is established. And under expectation conditions with the Kurdyka-Łojasiewicz (KL) property, we also prove that the proposed method converges globally. We also apply the BPSG algorithm to solve sparse nonnegative matrix factorization (NMF), symmetric NMF via non-symmetric relaxation, and matrix completion problems under different kernel generating distances, and numerically compare it with other algorithms. The results demonstrate the robustness and effectiveness of the proposed algorithm.

A generalized inertial proximal alternating linearized minimization method for nonconvex nonsmooth problems

We propose a general inertial version of the proximal alternating linearized minimization (PALM) (denoted by NiPALM) for a class of nonconvex and nonsmooth minimization problems, whose objective function is the sum of a smooth function of the entire variables and two nonsmooth functions of each variable. NiPALM is general in the sense that it contains the popular PALM, the inertial PALM (iPALM) and Gauss-Seidel type inertial PALM (GiPALM) as special cases. Under mild assumptions, namely, the underlying functions satisfy the Kurdyka-Łojasiewicz (KL) property and some suitable conditions on the parameters, we prove that each bounded sequence generated by NiPALM globally converges to a critical point. We also apply NiPALM to nonnegative matrix factorization, sparse principal component analysis, and weighted low-rank matrix restoration problems. Comparing results with those by PALM, iPALM, and GiPALM demonstrate the robustness and effectiveness of the proposed algorithm.

Proximal Linearized Iteratively Reweighted Algorithms for Nonconvex and Nonsmooth Optimization Problem

The nonconvex and nonsmooth optimization problem has been attracting increasing attention in recent years in image processing and machine learning research. The algorithm-based reweighted step has been widely used in many applications. In this paper, we propose a new, extended version of the iterative convex majorization–minimization method (ICMM) for solving a nonconvex and nonsmooth minimization problem, which involves famous iterative reweighted methods. To prove the convergence of the proposed algorithm, we adopt the general unified framework based on the Kurdyka–Łojasiewicz inequality. Numerical experiments validate the effectiveness of the proposed algorithm compared to the existing methods.

Assimilation of Images via Dictionary Learning-Based Sparsity Regularization Strategy: An Application for Retrieving Fluid Flows

In this work, we propose a structure sparsity regularization strategy in the framework of 4-D variational data assimilation (4-D Var). In meteorology and oceanography, the number of unknown model variables is far fewer than that of image observations, often leading to solve an underdetermined nonlinear inverse problem. In recent years, the l¹-norm-based sparsity regularization approach has attracted great attention in the field of 4-D Var because of its data structure preservation and noise suppression. To avoid little underlying physical priors considered, we introduce a widely used dictionary learning (DL) method to adaptively derive an efficient sparse approximation via learning a basis from a given dataset. For our target application of estimating sea surface flows, we consider a DL sparsity constraint on the variable of flow vorticity due to its rich spatial variation related to flows evolution. A novel anisotropic regularization method combined with fluid dynamics characteristics could overcome magnitude underestimation and staircase artifacts appearing in the gradient regularization-based 4-D Var method. The split Bregman iteration with fast convergence property is employed to solve the l¹+l² nonsmooth minimization problem. The promising fluid flows estimation performance in real test cases (assimilation of image sequences collected from CORIOLIS experimental turntable) demonstrates the efficiency of our approach.

Maximum Penalty Function in Linear Programming

A linear program can be equivalently reformulated as an unconstrained nonsmooth minimization problem, whose objective is the sum of the original objective and a penalty function with a sufficiently large penalty parameter. The article presents two methods for choosing this parameter. The first one applies to linear programs with usual linear inequality constraints. Then, we use a corresponding theorem by N.Z. Shor on the equivalence of a convex program to an unconstrained nonsmooth minimization problem. The second method is for linear programs of a special type. This means that all inequalities are of the form that a linear expression on the left-hand side is less or equal to a positive constant on the right-hand side. For this special type, we use a corresponding theorem of B.N. Pshenichny on establishing a penalty parameter for convex programs. For differently sized linear programs of the special type, we demonstrate that suitable penalty parameters can be computed by a procedure in GNU Octave based on GLPK software.

Bregman linearized reweighted alternating minimization for robust sparse recovery

In this paper, we consider a class of nonconvex and nonsmooth minimization problems with linear constraints in the area of compressed sensing (CS). The classical alternating minimization (AM) had difficulties in solving formulated nonconvex or nonsmooth subproblems. In view of this problem, we propose a Bregman linearized reweighted alternating minimization with continuation (BLRAMC). In this algorithm, all subproblems are convex and can be thus solved efficiently. Moreover, with the introduced Kurdyka-Łojasiewicz property, it is proved that the proposed algorithm converges to a stationary point of the objective function. Numerical experiments for the robust sparse recovery with different impulsive noise are presented to substantiate the effectiveness of the proposed algorithm.

Sample-based online learning for bi-regular hinge loss

Support vector machine (SVM), a state-of-the-art classifier for supervised classification task, is famous for its strong generalization guarantees derived from the max-margin property. In this paper, we focus on the maximum margin classification problem cast by SVM and study the bi-regular hinge loss model, which not only performs feature selection but tends to select highly correlated features together. To solve this model, we propose an online learning algorithm that aims at solving a non-smooth minimization problem by alternating iterative mechanism. Basically, the proposed algorithm alternates between intrusion samples detection and iterative optimization, and at each iteration it obtains a closed-form solution to the model. In theory, we prove that the proposed algorithm achieves $$O(1/\sqrt{T})$$ convergence rate under some mild conditions, where T is the number of training samples received in online learning. Experimental results on synthetic data and benchmark datasets demonstrate the effectiveness and performance of our approach in comparison with several popular algorithms, such as LIBSVM, SGD, PEGASOS, SVRG, etc.

Linear convergence of proximal incremental aggregated gradient method for nonconvex nonsmooth minimization problems

In this paper, we study the proximal incremental aggregated gradient (PIAG) method for minimizing the sum of smooth component functions and a nonsmooth function, both of which are allowed to be nonconvex. We show the linear convergence rate result under the metric subregularity and proper separation condition. Our method is developed based on a special auxiliary Lyapunov function sequence, and we also show that this auxiliary sequence is Q-linearly convergent. Using this result, we obtain an explicit computable stepsize threshold to guarantee that the objective value and iterative sequences are R-linearly convergent. Preliminary computational experience is also reported.

Image restoration using overlapping group sparsity on hyper-Laplacian prior of image gradient

Due to the ill-posed nature of image restoration, seeking a meaningful image prior is still a great challenge in the field of image processing. The total variation with overlapping group sparsity (OGS-TV) has been successfully applied for image denoising/deblurring. In this paper, we further study the overlapping group sparsity of the image gradient. The sparsity is measured by the ℓq quasi-norm (0<q<1). The proposed regularizer comes down to the well-known hyper-Laplacian prior if the overlapping group size is 1. Although it seems to be a simple extensive study compared with the previous works, its regularization capability and corresponding mathematical problems are still in demand for imaging science. To solve the non-convex and non-smooth minimization problem, we use the alternating direction method of multipliers as the main algorithm framework. The difficult inner subproblem is tackled by the majorization-minimization method with the sophisticatedly derived majorizer. We carry out some numerical experiments to demonstrate the effectiveness of the proposed regularizer in terms of PSNR and SSIM values.

Efficient iterative solution of finite element discretized nonsmooth minimization problems

For the iterative solution of finite element discretized, nonsmooth minimization problems the alternating direction method of multipliers (ADMM) is considered, which is a flexible method to solve a large class of convex minimization problems. Particular features are its unconditional convergence with respect to the involved step size and its direct applicability. In particular, this article deals with the ADMM with variable step sizes and devises an adjustment rule for the step size relying on the monotonicity of the residual and discusses proper stopping criteria. The proposed scheme is applied to finite element formulations of the obstacle problem and the Rudin–Osher–Fatemi image denoising problem, and the numerical experiments show significant improvements over established variants of the ADMM.

Least-Square-Based Three-Term Conjugate Gradient Projection Method for ℓ1-Norm Problems with Application to Compressed Sensing

In this paper, we propose, analyze, and test an alternative method for solving the ℓ 1 -norm regularization problem for recovering sparse signals and blurred images in compressive sensing. The method is motivated by the recent proposed nonlinear conjugate gradient method of Tang, Li and Cui [Journal of Inequalities and Applications, 2020(1), 27] designed based on the least-squares technique. The proposed method aims to minimize a non-smooth minimization problem consisting of a least-squares data fitting term and an ℓ 1 -norm regularization term. The search directions generated by the proposed method are descent directions. In addition, under the monotonicity and Lipschitz continuity assumption, we establish the global convergence of the method. Preliminary numerical results are reported to show the efficiency of the proposed method in practical computation.

A nonconvex and nonsmooth anisotropic total variation model for image noise and blur removal

In this paper, a nonconvex and nonsmooth anisotropic total variation model is proposed, which can provide a very sparser representation of the derivatives of the function in horizontal and vertical directions. The new model can preserve sharp edges and alleviate the staircase effect often arising in total variation (TV) based models. We use graduated nonconvexity (GNC) algorithm to solve the proposed nonconvex and nonsmooth minimization problem. Starting with a convex initialization, it uses a family of nonconvex functional to gradually approach the original nonconvex functional. For each subproblem, we use function splitting technique to separately address the nonconvex and nonsmooth properties, and use augmented Lagrangian method (ALM) to solve it. Experiments are conducted for both synthetic and real images to demonstrate the effectiveness of the proposed model. In addition, we compare it with several state-of-the-art models in denoising and deblurring applications. The numerical results show that our model has the best performance in terms of PSNR and MSSIM indexes.

A Gauss–Seidel type inertial proximal alternating linearized minimization for a class of nonconvex optimization problems

In this paper we study a broad class of nonconvex and nonsmooth minimization problems, whose objective function is the sum of a smooth function of the entire variables and two nonsmooth functions of each variable. We adopt the framework of the proximal alternating linearized minimization (PALM), together with the inertial strategy to accelerate the convergence. Since the inertial step is performed once the x-subproblem/y-subproblem is updated, the algorithm is a Gauss–Seidel type inertial proximal alternating linearized minimization (GiPALM) algorithm. Under the assumption that the underlying functions satisfy the Kurdyka–Łojasiewicz (KL) property and some suitable conditions on the parameters, we prove that each bounded sequence generated by GiPALM globally converges to a critical point. We apply the algorithm to signal recovery, image denoising and nonnegative matrix factorization models, and compare it with PALM and the inertial proximal alternating linearized minimization.

Gradient-enhanced thermomechanical 3D model for simulation of transformation patterns in pseudoelastic shape memory alloys

Stress-induced martensitic transformation in polycrystalline NiTi under tension often proceeds through formation and propagation of macroscopic phase transformation fronts, i.e., diffuse interfaces that separate the transformed and untransformed domains. A gradient-enhanced 3D finite-strain model of pseudoelasticity is developed in this work with the aim to describe the related phenomena. The underlying softening response is regularized by enhancing the Helmholtz free energy of a non-gradient model with a gradient term expressed in terms of the martensite volume fraction. To facilitate finite-element implementation, a micromorphic-type regularization is then introduced following the approach developed recently in the 1D small-strain context. The complete evolution problem is formulated within the incremental energy minimization framework, and the resulting non-smooth minimization problem is solved by employing the augmented Lagrangian technique. In order to account for the thermomechanical coupling effects, a general thermomechanical framework, which is consistent with the second law of thermodynamics and considers all related couplings, is also developed. Finite-element simulations of representative 3D problems show that the model is capable of representing the loading-rate effects in a NiTi dog-bone specimen and complex transformation patterns in a NiTi tube under tension. A parametric study is also carried out to investigate the effect of various parameters on the characteristics of the macroscopic transformation front.

On the convergence of the iterates of proximal gradient algorithm with extrapolation for convex nonsmooth minimization problems

In this paper, we consider the proximal gradient algorithm with extrapolation for solving a class of convex nonsmooth minimization problems. We show that for a large class of extrapolation parameters including the extrapolation parameters chosen in FISTA (Beck and Teboulle in SIAM J Imaging Sci 2:183–202, 2009), the successive changes of iterates go to 0. Moreover, based on the Łojasiewicz inequality, we establish the global convergence of iterates generated by the proximal gradient algorithm with extrapolation with an additional assumption on the extrapolation coefficients. The assumption is general enough to allow the threshold of the extrapolation coefficients to be 1. In particular, we prove the length of the iterates is finite. Finally, we perform numerical experiments on the least squares problems with $$\ell _1$$ regularization to illustrate our theoretical results.

Bregman Proximal Gradient Algorithm With Extrapolation for a Class of Nonconvex Nonsmooth Minimization Problems

In this paper, we consider an accelerated method for solving nonconvex and nonsmooth minimization problems. We propose a Bregman Proximal Gradient algorithm with extrapolation (BPGe). This algorithm extends and accelerates the Bregman Proximal Gradient algorithm (BPG), which circumvents the restrictive global Lipschitz gradient continuity assumption needed in Proximal Gradient algorithms (PG). The BPGe algorithm has a greater generality than the recently introduced Proximal Gradient algorithm with extrapolation (PGe) and, in addition, due to the extrapolation step, BPGe converges faster than the BPG algorithm. Analyzing the convergence, we prove that any limit point of the sequence generated by BPGe is a stationary point of the problem by choosing the parameters properly. Besides, assuming Kurdyka-Łojasiewicz property, we prove that all the sequences generated by BPGe converge to a stationary point. Finally, to illustrate the potential of the new method BPGe, we apply it to two important practical problems that arise in many fundamental applications (and that not satisfy global Lipschitz gradient continuity assumption): Poisson linear inverse problems and quadratic inverse problems. In the tests the accelerated BPGe algorithm shows faster convergence results, giving an interesting new algorithm.

Optimized projections for compressed sensing via direct mutual coherence minimization

Abstract Compressed Sensing (CS) is a new data acquisition theory based on the existence of a sparse representation of a signal and a projected dictionary PD, where P ∈ R m × d is the projection matrix and D ∈ R d × n is the dictionary. To recover the signal from a small number m of measurements, it is expected that the projected dictionary PD is of low mutual coherence. Several previous methods attempt to find the projection P such that the mutual coherence of PD is low. However, they do not minimize the mutual coherence directly and thus they may be far from optimal. Their used solvers lack convergence guarantee and thus the quality of their solutions is not guaranteed. This work aims to address these issues. We propose to find an optimal projection matrix by minimizing the mutual coherence of PD directly. This leads to a nonconvex nonsmooth minimization problem. We approximate it by smoothing, solve it by alternating minimization and prove the convergence of our algorithm. To the best of our knowledge, this is the first work which directly minimizes the mutual coherence of the projected dictionary and has convergence guarantee. Numerical experiments demonstrate that our method can recover sparse signals better than existing ones.

First Order Methods Beyond Convexity and Lipschitz Gradient Continuity with Applications to Quadratic Inverse Problems

We focus on nonconvex and nonsmooth minimization problems with a composite objective, where the differentiable part of the objective is freed from the usual and restrictive global Lipschitz gradien...