Non-convex Optimization Method Research Articles

Gradient Methods for Non-convex Optimization

Non-convex optimization forms bedrock of most modern machine learning (ML) techniques such as deep learning. While non-convex optimization problems have been studied for the past several decades, ML-based problems have significantly different characteristics and requirements due to large datasets and high-dimensional parameter spaces along with the statistical nature of the problem. Over the last few years, there has been a flurry of activity in non-convex optimization for such ML problems. This article surveys a few of the foundational approaches in this domain.

Newton-type methods for non-convex optimization under inexact Hessian information

We consider variants of trust-region and cubic regularization methods for non-convex optimization, in which the Hessian matrix is approximated. Under mild conditions on the inexact Hessian, and using approximate solution of the corresponding sub-problems, we provide iteration complexity to achieve $ \epsilon $-approximate second-order optimality which have shown to be tight. Our Hessian approximation conditions constitute a major relaxation over the existing ones in the literature. Consequently, we are able to show that such mild conditions allow for the construction of the approximate Hessian through various random sampling methods. In this light, we consider the canonical problem of finite-sum minimization, provide appropriate uniform and non-uniform sub-sampling strategies to construct such Hessian approximations, and obtain optimal iteration complexity for the corresponding sub-sampled trust-region and cubic regularization methods.

Robust support vector machine with generalized quantile loss for classification and regression

A new robust loss function (called Lq-loss) is proposed based on the concept of quantile and correntropy, which can be seen as an improved version of quantile loss function. The proposed Lq-loss has some important properties such as asymmetry, non-convexity and boundedness, which has received a lot of attention recently. The Lq-loss includes and extends the traditional loss functions such as pinball loss, rescaled hinge loss, L1-norm loss and zero-norm loss. Additionally, we demonstrate that the Lq-loss is a kernel-induced loss by reproducing piecewise kernel function. Further, two robust SVM frameworks are presented to handle robust classification and regression problems by applying Lq-loss to support vector machine, respectively. Last but not least, we demonstrate that the proposed classification framework satisfies Bayes’ optimal decision rule. However, the non-convexity of the proposed Lq-loss makes it difficult to optimize. A non-convex optimization method, concave–convex procedure (CCCP) technique, is used to solve the proposed models, and the convergence of the algorithms is proved theoretically. For classification and regression tasks, experiments are carried out on three databases including UCI benchmark datasets, artificial datasets and a practical application dataset. Compared to some classical and advanced methods, numerical simulations under different noise setting and different evaluation criteria show that the proposed methods have good robustness to feature noise and outliers in both classification and regression applications.

Run-and-Inspect Method for nonconvex optimization and global optimality bounds for R-local minimizers

Many optimization algorithms converge to stationary points. When the underlying problem is nonconvex, they may get trapped at local minimizers and occasionally stagnate near saddle points. We propose the Run-and-Inspect Method, which adds an "inspect" phase to existing algorithms that helps escape from non-global stationary points. The inspection samples a set of points in a radius $R$ around the current point. When a sample point yields a sufficient decrease in the objective, we move there and resume an existing algorithm. If no sufficient decrease is found, the current point is called an approximate $R$-local minimizer. We show that an $R$-local minimizer is globally optimal, up to a specific error depending on $R$, if the objective function can be implicitly decomposed into a smooth convex function plus a restricted function that is possibly nonconvex, nonsmooth. For high-dimensional problems, we introduce blockwise inspections to overcome the curse of dimensionality while still maintaining optimality bounds up to a factor equal to the number of blocks. Our method performs well on a set of artificial and realistic nonconvex problems by coupling with gradient descent, coordinate descent, EM, and prox-linear algorithms.

Three-Phase Unbalanced Optimal Power Flow Using Holomorphic Embedding Load Flow Method

Distribution networks are typically unbalanced due to loads being unevenly distributed over the three phases and untransposed lines. Additionally, unbalance is further increased with high penetration of single-phased distributed generators. Load and optimal power flows, when applied to distribution networks, use models developed for transmission grids with limited modification. The performance of optimal power flow depends on external factors such as ambient temperature and irradiation, since they have strong influence on loads and distributed energy resources such as photo voltaic systems. To help mitigate the issues mentioned above, the authors present a novel class of optimal power flow algorithm which is applied to low-voltage distribution networks. It involves the use of a novel three-phase unbalanced holomorphic embedding load flow method in conjunction with a non-convex optimization method to obtain the optimal set-points based on a suitable objective function. This novel three-phase load flow method is benchmarked against the well-known power factory Newton-Raphson algorithm for various test networks. Mann-Whitney U test is performed for the voltage magnitude data generated by both methods and null hypothesis is accepted. A use case involving a real network in Austria and a method to generate optimal schedules for various controllable buses is provided.

On integer and MPCC representability of affine sparsity

In addition to sparsity, practitioners of statistics and machine learning often wish to promote additional structures in their variable selection process to incorporate prior knowledge. Borrowing the modeling power of linear systems with binary variables, many of such structures can be faithfully modeled by affine sparsity constraints (ASC). In this note we study conditions under which an ASC system can be represented by sets in integer programs and mathematical programs with complementarity conditions (MPCC). Results of this note facilitate developing nonconvex optimization methods for variable selection with structured sparsity.

Misspecified nonconvex statistical optimization for sparse phase retrieval

Existing nonconvex statistical optimization theory and methods crucially rely on the correct specification of the underlying “true” statistical models. To address this issue, we take a first step towards taming model misspecification by studying the high-dimensional sparse phase retrieval problem with misspecified link functions. In particular, we propose a simple variant of the thresholded Wirtinger flow algorithm that, given a proper initialization, linearly converges to an estimator with optimal statistical accuracy for a broad family of unknown link functions. We further provide extensive numerical experiments to support our theoretical findings.

Stochastic Successive Convex Approximation for Non-Convex Constrained Stochastic Optimization

This paper proposes a constrained stochastic successive convex approximation (CSSCA) algorithm to find a stationary point for a general non-convex stochastic optimization problem, whose objective and constraint functions are non-convex and involve expectations over random states. Most existing methods for non-convex stochastic optimization, such as the stochastic (average) gradient and stochastic majorization-minimization, only consider minimizing a stochastic non-convex objective over a deterministic convex set. The proposed CSSCA algorithm can also handle stochastic non-convex constraints in optimization problems, and it opens the way to solving more challenging optimization problems that occur in many applications. The algorithm is based on solving a sequence of convex objective/feasibility optimization problems obtained by replacing the objective/constraint functions in the original problems with some convex surrogate functions. The CSSCA algorithm allows a wide class of surrogate functions and thus provides many freedoms to design good surrogate functions for specific applications. Moreover, it also facilitates parallel implementation for solving large scale stochastic optimization problems, which arise naturally in today's signal processing such as machine learning and big data analysis. We establish the convergence of CSSCA algorithm with a feasible initial point, and customize the algorithmic framework to solve several important application problems. Simulations show that the CSSCA algorithm can achieve superior performance over existing solutions.

An Alternating Augmented Lagrangian method for constrained nonconvex optimization

ABSTRACTWe consider the problem of minimizing a smooth nonconvex function over a structured convex feasible set, that is, defined by two sets of constraints that are easy to treat when considered separately. In order to exploit the structure of the problem, we define an equivalent formulation by duplicating the variables and we consider the augmented Lagrangian of this latter formulation. Following the idea of the Alternating Direction Method of Multipliers (ADMM), we propose an algorithm where a two-blocks decomposition method is embedded within an augmented Lagrangian framework. The peculiarities of the proposed algorithm are the following: (1) the computation of the exact solution of a possibly nonconvex subproblem is not required; (2) the penalty parameter is iteratively updated once an approximated stationary point of the augmented Lagrangian is determined. Global convergence results are stated under mild assumptions and without requiring convexity of the objective function. Although the primary aim of the paper is theoretical, we perform numerical experiments on a nonconvex problem arising in machine learning, and the obtained results show the practical advantages of the proposed approach with respect to classical ADMM.

An Inexact Bundle Algorithm for Nonconvex Nonsmooth Minimization in Hilbert Space

In this paper a bundle method for nonconvex nonsmooth optimization in infinite-dimensional Hilbert spaces is developed and analyzed. The algorithm requires only inexact function value and subgradie...

A Newton-Based Method for Nonconvex Optimization with Fast Evasion of Saddle Points

Machine learning problems such as neural network training, tensor decomposition, and matrix factorization require local minimization of a nonconvex function. This local minimization is challenged b...

Robust Initialization Estimator and its Application in Solving Quadratic Equations

In many non-convex optimization-based signal recovery tasks, a good initial point is essential for the performance of the optimization process. One seeks to start the point from a small local region surrounding the targeted signal. Then an efficient iterative refinement procedure can help recover the wanted signal. Motivated by this fact, we introduce two efficient and robust estimators to find reasonably good initial points for non-convex phase retrieval algorithms. The proposed estimators can provide high quality initial guesses for phase retrieval even with a number of samples that is close to the information-theoretic limit. The average relative error reduces exponentially as the oversampling ratio grows, which can improve the performance of existing non-convex optimization methods. The experimental results clearly demonstrate the superiority of two introduced estimators, which not only obtain a more accurate estimate of the true solution but also outperform the existing methods in terms of noise robustness when measurements are contaminated with noise.

Stochastic proximal quasi-Newton methods for non-convex composite optimization

ABSTRACTIn this paper, we propose a generic algorithmic framework for stochastic proximal quasi-Newton (SPQN) methods to solve non-convex composite optimization problems. Stochastic second-order information is explored to construct proximal subproblem. Under mild conditions we show the non-asympotic convergence of the proposed algorithm to stationary point of original problems and analyse its computational complexity. Besides, we extend the proximal form of Polyak-Łojasiewicz (PL) inequality to constrained settings and obtain the constrained proximal PL (CP-PL) inequality. Under CP-PL inequality linear convergence rate of the proposed algorithm is achieved. Moreover, we propose a modified self-scaling symmetric rank one incorporated in the framework for SPQN method, which is called stochastic symmetric rank one method. Finally, we report some numerical experiments to reveal the effectiveness of the proposed algorithm.

Inexact Proximal Gradient Methods for Non-Convex and Non-Smooth Optimization

In machine learning research, the proximal gradient methods are popular for solving various optimization problems with non-smooth regularization. Inexact proximal gradient methods are extremely important when exactly solving the proximal operator is time-consuming, or the proximal operator does not have an analytic solution. However, existing inexact proximal gradient methods only consider convex problems. The knowledge of inexact proximal gradient methods in the non-convex setting is very limited. To address this challenge, in this paper, we first propose three inexact proximal gradient algorithms, including the basic version and Nesterov’s accelerated version. After that, we provide the theoretical analysis to the basic and Nesterov’s accelerated versions. The theoretical results show that our inexact proximal gradient algorithms can have the same convergence rates as the ones of exact proximal gradient algorithms in the non-convex setting. Finally, we show the applications of our inexact proximal gradient algorithms on three representative non-convex learning problems. Empirical results confirm the superiority of our new inexact proximal gradient algorithms.

A family of global convergent inexact secant methods for nonconvex constrained optimization

We present a family of new inexact secant methods in association with Armijo line search technique for solving nonconvex constrained optimization. Different from the existing inexact secant methods, the algorithms proposed in this paper need not compute exact directions. By adopting the nonsmooth exact penalty function as the merit function, the global convergence of the proposed algorithms is established under some reasonable conditions. Some numerical results indicate that the proposed algorithms are both feasible and effective.

Local Convergence of the Heavy-Ball Method and iPiano for Non-convex Optimization

A local convergence result for an abstract descent method is proved. The sequence of iterates is attracted by a local (or global) minimum, stays in its neighborhood, and converges within this neighborhood. This result allows algorithms to exploit local properties of the objective function. In particular, the abstract theory in this paper applies to the inertial forward–backward splitting method: iPiano—a generalization of the Heavy-ball method. Moreover, it reveals an equivalence between iPiano and inertial averaged/alternating proximal minimization and projection methods. Key for this equivalence is the attraction to a local minimum within a neighborhood and the fact that, for a prox-regular function, the gradient of the Moreau envelope is locally Lipschitz continuous and expressible in terms of the proximal mapping. In a numerical feasibility problem, the inertial alternating projection method significantly outperforms its non-inertial variants.

An infeasible bundle method for nonconvex constrained optimization with application to semi-infinite programming problems

The main difficulty for solving semi-infinite programming (SIP) problem is precisely that it has infinitely many constraints. By using a maximum function, the SIP problem can be rewritten as a nonconvex nonsmooth constrained optimization (NNCO) problem. Global convergence in most of constrained optimization algorithms has traditionally been enforced by the use of a penalty function or filter strategy. In this paper, we propose an infeasible bundle method for NNCO problem based on the so-called improvement functions, without a penalty function and filter strategy. The method appears to be more direct and easier to implement, in the sense that it is closer in spirit and structure to the well-developed unconstrained bundle methods. Under a special constraint qualification, the sequence generated by this algorithm converges to the KKT point of the NNCO problem as well as the SIP problems. Preliminary numerical results show that this algorithm is robust and efficient for NNCO problems and SIP problems.

Accelerated Methods for NonConvex Optimization

We present an accelerated gradient method for nonconvex optimization problems with Lipschitz continuous first and second derivatives. In a time $O(\epsilon^{-7/4} \log(1/ \epsilon) )$, the method f...

A note on the accelerated proximal gradient method for nonconvex optimization

We improve a recent accelerated proximal gradient (APG) method in [Li, Q., Zhou, Y., Liang, Y. and Varshney, P. K., Convergence analysis of proximal gradient with momentum for nonconvex optimization, in Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, PMLR 70, 2017] for nonconvex optimization by allowing variable stepsizes. We prove the convergence of the APG method for a composite nonconvex optimization problem under the assumption that the composite objective function satisfies the Kurdyka-Łojasiewicz property.

Velocity reconstruction with nonconvex optimization for low-velocity-encoding phase-contrast MRI.

To introduce and demonstrate a nonconvex optimization method for reconstructing velocity data from low-velocity-encoding (Venc ) phase-contrast MRI data. Solving for velocity values from phase-contrast MRI data was formulated as a nonconvex optimization problem. Weighting was added to account for intravoxel dephasing, and a Laplacian-based regularization was used to account for residual velocity aliasing. The reconstruction was tested with two low-Venc schemes: dual-Venc and a multidirectional high-moment encoding. The reconstruction method was tested in a digital simulation, in flow phantoms, and in healthy volunteers (N = 5). The nonconvex-optimization reconstruction velocity error was lower than the conventional reconstruction in simulations (4.6 versus 3.0 cm/s for multidirectional high moment, 8.3 versus 3.8 cm/s for dual-Venc ) and in flow phantoms (23.9 versus 5.9 cm/s for multidirectional high moment, 15.2 versus 6.4 cm/s for dual-Venc ). Qualitative assessment of velocity fields in all experiments, including healthy volunteers, showed decreased apparent noise in the velocity fields and fewer phase wraps. No additional velocity bias in measured velocities was seen in volunteers with the proposed method. The proposed nonconvex-optimization reconstruction method incorporates additional information to solve for velocities when using any type of low-Venc (high-moment) acquisition. The method reduces the amount of residual phase aliasing, and decreases velocity errors that result from intravoxel dephasing. These improvements allow for more robust acquisitions, and for Venc to be lowered 2 to 4 times relative to conventional acquisitions, thereby increasing the velocity-to-noise ratio. Magn Reson Med, 2017. © 2017 International Society for Magnetic Resonance in Medicine. Magn Reson Med 80:42-52, 2018. © 2017 International Society for Magnetic Resonance in Medicine.