Non-convex Optimization Method Research Articles

Universal heavy-ball method for nonconvex optimization under Hölder continuous Hessians

AbstractWe propose a new first-order method for minimizing nonconvex functions with Lipschitz continuous gradients and Hölder continuous Hessians. The proposed algorithm is a heavy-ball method equipped with two particular restart mechanisms. It finds a solution where the gradient norm is less than $$\varepsilon $$ ε in $$O(H_{\nu }^{\frac{1}{2 + 2 \nu }} \varepsilon ^{- \frac{4 + 3 \nu }{2 + 2 \nu }})$$ O ( H ν 1 2 + 2 ν ε - 4 + 3 ν 2 + 2 ν ) function and gradient evaluations, where $$\nu \in [0, 1]$$ ν ∈ [ 0 , 1 ] and $$H_{\nu }$$ H ν are the Hölder exponent and constant, respectively. This complexity result covers the classical bound of $$O(\varepsilon ^{-2})$$ O ( ε - 2 ) for $$\nu = 0$$ ν = 0 and the state-of-the-art bound of $$O(\varepsilon ^{-7/4})$$ O ( ε - 7 / 4 ) for $$\nu = 1$$ ν = 1 . Our algorithm is $$\nu $$ ν -independent and thus universal; it automatically achieves the above complexity bound with the optimal $$\nu \in [0, 1]$$ ν ∈ [ 0 , 1 ] without knowledge of $$H_{\nu }$$ H ν . In addition, the algorithm does not require other problem-dependent parameters as input, including the gradient’s Lipschitz constant or the target accuracy $$\varepsilon $$ ε . Numerical results illustrate that the proposed method is promising.

Sparse linear array synthesis using artificial neural network and nonconvex optimization

AbstractThe concept of sparse array synthesis has attracted considerable attention due to its potential to reduce hardware costs in array design and enhance the efficiency of array aperture utilization. In contrast to conventional artificial neural networks (ANNs), a more efficient encoder–decoding ANN framework is proposed in this letter for designing sparse linear arrays. The decoder is pretrained with a randomly generated training set, enabling its ability to rapidly predictiction of far‐field patterns. This process considers the actual coupling effect among the antenna elements. The output of the encoder is used as the input of the decoder to synthesize the sparse arrays by providing the desired pattern to the encoder. During the training process, a novel ‐norm minimization nonconvex optimization method is integrated into the proposed framework to reduce the number of activated elements. The effectiveness and efficiency of the method are demonstrated through numerical and full‐wave simulations for synthesizing various linear sparse arrays, including large‐scale arrays. In comparison to existing algorithms, our proposed method exhibits superior computational efficiency and predicted accuracy with fewer elements.

Stochastic nested primal-dual method for nonconvex constrained composition optimization

In this paper we study the nonconvex constrained composition optimization, in which the objective contains a composition of two expected-value functions whose accurate information is normally expensive to calculate. We propose a STochastic nEsted Primal-dual (STEP) method for such problems. In each iteration, with an auxiliary variable introduced to track the inner layer function values we compute stochastic gradients of the nested function using a subsampling strategy. To alleviate difficulties caused by possibly nonconvex constraints, we construct a stochastic approximation to the linearized augmented Lagrangian function to update the primal variable, which further motivates to update the dual variable in a weighted-average way. Moreover, to better understand the asymptotic dynamics of the update schemes we consider a deterministic continuous-time system from the perspective of ordinary differential equation (ODE). We analyze the Karush-Kuhn-Tucker measure at the output by the STEP method with constant parameters and establish its iteration and sample complexities to find an ϵ \epsilon -stationary point, ensuring that expected stationarity, feasibility as well as complementary slackness are below accuracy ϵ \epsilon . To leverage the benefit of the (near) initial feasibility in the STEP method, we propose a two-stage framework incorporating a feasibility-seeking phase, aiming to locate a nearly feasible initial point. Moreover, to enhance the adaptivity of the STEP algorithm, we propose an adaptive variant by adaptively adjusting its parameters, along with a complexity analysis. Numerical results on a risk-averse portfolio optimization problem and an orthogonal nonnegative matrix decomposition reveal the effectiveness of the proposed algorithms.

Trust Region Methods for Nonconvex Stochastic Optimization beyond Lipschitz Smoothness

In many important machine learning applications, the standard assumption of having a globally Lipschitz continuous gradient may fail to hold. This paper delves into a more general (L0, L1)-smoothness setting, which gains particular significance within the realms of deep neural networks and distributionally robust optimization (DRO). We demonstrate the significant advantage of trust region methods for stochastic nonconvex optimization under such generalized smoothness assumption. We show that first-order trust region methods can recover the normalized and clipped stochastic gradient as special cases and then provide a unified analysis to show their convergence to first-order stationary conditions. Motivated by the important application of DRO, we propose a generalized high-order smoothness condition, under which second-order trust region methods can achieve a complexity of O(epsilon(-3.5)) for convergence to second-order stationary points. By incorporating variance reduction, the second-order trust region method obtains an even better complexity of O(epsilon(-3)), matching the optimal bound for standard smooth optimization. To our best knowledge, this is the first work to show convergence beyond the first-order stationary condition for generalized smooth optimization. Preliminary experiments show that our proposed algorithms perform favorably compared with existing methods.

Decentralized Gradient-Free Methods for Stochastic Non-smooth Non-convex Optimization

We consider decentralized gradient-free optimization of minimizing Lipschitz continuous functions that satisfy neither smoothness nor convexity assumption. We propose two novel gradient-free algorithms, the Decentralized Gradient-Free Method (DGFM) and its variant, the Decentralized Gradient-Free Method+ (DGFM+). Based on the techniques of randomized smoothing and gradient tracking, DGFM requires the computation of the zeroth-order oracle of a single sample in each iteration, making it less demanding in terms of computational resources for individual computing nodes. Theoretically, DGFM achieves a complexity of O(d^(3/2)δ^(-1)ε^(-4)) for obtaining a (δ,ε)-Goldstein stationary point. DGFM+, an advanced version of DGFM, incorporates variance reduction to further improve the convergence behavior. It samples a mini-batch at each iteration and periodically draws a larger batch of data, which improves the complexity to O(d^(3/2)δ^(-1)ε^(-3)). Moreover, experimental results underscore the empirical advantages of our proposed algorithms when applied to real-world datasets.

Sample-and-Bound for Non-convex Optimization

Standard approaches for global optimization of non-convex functions, such as branch-and-bound, maintain partition trees to systematically prune the domain. The tree size grows exponentially in the number of dimensions. We propose new sampling-based methods for non-convex optimization that adapts Monte Carlo Tree Search (MCTS) to improve efficiency. Instead of the standard use of visitation count in Upper Confidence Bounds, we utilize numerical overapproximations of the objective as an uncertainty metric, and also take into account of sampled estimates of first-order and second-order information. The Monte Carlo tree in our approach avoids the usual fixed combinatorial patterns in growing the tree, and aggressively zooms into the promising regions, while still balancing exploration and exploitation. We evaluate the proposed algorithms on high-dimensional non-convex optimization benchmarks against competitive baselines and analyze the effects of the hyper parameters.

Here comes the SU(N): multivariate quantum gates and gradients

Variational quantum algorithms use non-convex optimization methods to find the optimal parameters for a parametrized quantum circuit in order to solve a computational problem. The choice of the circuit ansatz, which consists of parameterized gates, is crucial to the success of these algorithms. Here, we propose a gate which fully parameterizes the special unitary group SU(N). This gate is generated by a sum of non-commuting operators, and we provide a method for calculating its gradient on quantum hardware. In addition, we provide a theorem for the computational complexity of calculating these gradients by using results from Lie algebra theory. In doing so, we further generalize previous parameter-shift methods. We show that the proposed gate and its optimization satisfy the quantum speed limit, resulting in geodesics on the unitary group. Finally, we give numerical evidence to support the feasibility of our approach and show the advantage of our gate over a standard gate decomposition scheme. In doing so, we show that not only the expressibility of an ansatz matters, but also how it's explicitly parameterized.

An inexact regularized proximal Newton method for nonconvex and nonsmooth optimization

This paper focuses on the minimization of a sum of a twice continuously differentiable function f and a nonsmooth convex function. An inexact regularized proximal Newton method is proposed by an approximation to the Hessian of f involving the ϱ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varrho $$\\end{document}th power of the KKT residual. For ϱ=0\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varrho =0$$\\end{document}, we justify the global convergence of the iterate sequence for the KL objective function and its R-linear convergence rate for the KL objective function of exponent 1/2. For ϱ∈(0,1)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varrho \\in (0,1)$$\\end{document}, by assuming that cluster points satisfy a locally Hölderian error bound of order q on a second-order stationary point set and a local error bound of order q>1+ϱ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$q>1\\!+\\!\\varrho $$\\end{document} on the common stationary point set, respectively, we establish the global convergence of the iterate sequence and its superlinear convergence rate with order depending on q and ϱ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varrho $$\\end{document}. A dual semismooth Newton augmented Lagrangian method is also developed for seeking an inexact minimizer of subproblems. Numerical comparisons with two state-of-the-art methods on ℓ1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\ell _1$$\\end{document}-regularized Student’s t-regressions, group penalized Student’s t-regressions, and nonconvex image restoration confirm the efficiency of the proposed method.

Shape-changing trust-region methods using multipoint symmetric secant matrices

In this work, we consider methods for large-scale and nonconvex unconstrained optimization. We propose a new trust-region method whose subproblem is defined using a so-called ‘shape-changing’ norm together with densely-initialized multipoint symmetric secant (MSS) matrices to approximate the Hessian. Shape-changing norms and dense initializations have been successfully used in the context of traditional quasi-Newton methods, but have yet to be explored in the case of MSS methods. Numerical results suggest that trust-region methods that use densely-initialized MSS matrices together with shape-changing norms outperform MSS with other trust-region methods.

Diffractive optical elements generation by layer-based methods for rapid and high-quality formation of 3D-objects

The article is devoted to the generation of diffractive optical elements and computer holograms for forming three-dimensional images. Possibilities of increasing the speed of diffractive optical elements generation and the quality of reconstructed 3D-objects were investigated. Four methods of optical elements generation were analyzed. The methods use division the 3D-objects into fl at layers. The quality of 3D-object reconstruction and time generation by the methods were assessed. 3D-object reconstruction from generated optical elements was modeled. Optical formation of objects was performed by displaying optical elements onto liquid crystal spatial light modulator. It was found that the best quality of reconstruction was provided by iterative parallel ping-pong and non-convex optimization methods. The optimal ratio of reconstruction quality to generation speed ratio was obtained for the parallel ping-pong method. The possibility of fast formation high-quality three-dimensional scenes consisting of dozens of layers has been demonstrated.

Markov chain stochastic DCA and applications in deep learning with PDEs regularization

This paper addresses a large class of nonsmooth nonconvex stochastic DC (difference-of-convex functions) programs where endogenous uncertainty is involved and i.i.d. (independent and identically distributed) samples are not available. Instead, we assume that it is only possible to access Markov chains whose sequences of distributions converge to the target distributions. This setting is legitimate as Markovian noise arises in many contexts including Bayesian inference, reinforcement learning, and stochastic optimization in high-dimensional or combinatorial spaces. We then design a stochastic algorithm named Markov chain stochastic DCA (MCSDCA) based on DCA (DC algorithm) - a well-known method for nonconvex optimization. We establish the convergence analysis in both asymptotic and nonasymptotic senses. The MCSDCA is then applied to deep learning via PDEs (partial differential equations) regularization, where two realizations of MCSDCA are constructed, namely MCSDCA-odLD and MCSDCA-udLD, based on overdamped and underdamped Langevin dynamics, respectively. Numerical experiments on time series prediction and image classification problems with a variety of neural network topologies show the merits of the proposed methods.

Robustness Meets Low-Rankness: Unified Entropy and Tensor Learning for Multi-View Subspace Clustering

In this paper, we develop the weighted error entropy-regularized tensor learning method for multi-view subspace clustering (WETMSC), which integrates the noise disturbance removal and subspace structure discovery into one unified framework. Unlike most existing methods which focus only on the affinity matrix learning for the subspace discovery by different optimization models and simply assume that the noise is independent and identically distributed (i.i.d.), our WETMSC method adopts the weighted error entropy to characterize the underlying noise by assuming that noise is independent and piecewise identically distributed (i.p.i.d.). Meanwhile, WETMSC constructs the self-representation tensor by storing all self-representation matrices from the view dimension, preserving high-order correlation of views based on the tensor nuclear norm. To solve the proposed nonconvex optimization method, we design a half-quadratic (HQ) additive optimization technology and iteratively solve all subproblems under the alternating direction method of multipliers framework. Extensive comparison studies with state-of-the-art clustering methods on real-world datasets and synthetic noisy datasets demonstrate the ascendancy of the proposed WETMSC method.

Sparse neural network optimization by Simulated Annealing

The over-parameterization of neural networks and the local optimality of backpropagation algorithm have been two major problems associated with deep-learning. In order to reduce the redundancy of neural network parameters, the conventional approach has been to prune branches with small weights. However, this only solves the problem of parameter redundancy, not providing any global optimality guarantees. In this paper, we overturn back-propagation and combine the sparse network optimization problem and the network weight optimization problem using a non-convex optimization method, namely Simulated Annealing. This method can complete network training under the premise of controlling the amount of parameters. Different from simply updating network parameters using gradient descent, our method simultaneously optimizes the topology of the sparse network. With the guarantee of global optimality of Simulated Annealing solution, the performance of the sparse network optimized by our method has exceeded the one trained by backpropagation only.

Self-calibration algorithm for installation angle deviation of bionic polarization compound eyes.

A self-calibration algorithm based on unsupervised optimization for polarizer installation angle deviation is proposed and used in a multi-aperture bionic polarization compound eye system. To simplify calibration operation, under the condition that the calibration-polarized light information is unknown, this algorithm fully exploits redundancy and random polarization information in the scene, and uses a non-convex multi-objective discrete parameter sorting optimization method to achieve angle self-calibration. Compared with ordinary calibration procedures, the algorithm requires less stringent conditions, achieves online calibration and is more accurate. It also can be applied to camera polarization arrays, division-of-focal-plane polarization cameras, and other polarization devices.

Low-Rank tensor completion based on nonconvex regularization

In this paper, we consider the low-rank tensor completion which aims to exactly recover incomplete high-dimensional visual data. Existing studies utilize widely tensor nuclear norm minimization (TNNM), a convex relaxation to tensor-rank minimization (TRM), to solve tensor completion tasks. Nevertheless, TNNM ignores the difference between different tensor singular values induced by the tensor singular value decompositions (t-SVD) and then the obtained solution may be suboptimal. In this paper, we propose a nonconvex minimization approach to solve the tensor completion problem more effectively by adopting a nonconvex regularization to further approximate the tensor-rank. Moreover, alternating direction method of multipliers (ADMM) and iteratively reweighted nuclear norm (IRNN) are adopted to solve the constructed nonconvex models efficiently, and the convergence can also be guaranteed. Finally, we present that the proposed nonconvex optimization methods are suitable for solving other TRM problems induced by any invertible linear transform, such as subspace clustering based on low-rank representation. Extensive experiments on real images and videos validate the superiority of our approach over the state-of-the-art algorithms.

A new non-convex sparse optimization method for image restoration

A new non-convex sparse optimization method for image restoration

A new Lagrangian-based first-order method for nonconvex constrained optimization

In this paper, we introduce a new form of Lagrangian function and propose a simple first-order primal-dual algorithm for solving nonconvex optimization with nonlinear equality constraints. We show that the algorithm generates bounded primal-dual iterates, and establish the convergence to KKT points under standard assumptions. The key features of the proposed method are: (i) it does not require boundedness assumptions on dual iterates generated by the algorithm as well as the set of multipliers; (ii) it is a single-loop algorithm that does not involve any penalty subproblems.

New proximal bundle algorithm based on the gradient sampling method for nonsmooth nonconvex optimization with exact and inexact information

New proximal bundle algorithm based on the gradient sampling method for nonsmooth nonconvex optimization with exact and inexact information

Worst-case evaluation complexity of a quadratic penalty method for nonconvex optimization

This paper addresses the worst-case evaluation complexity of a version of the standard quadratic penalty method for smooth nonconvex optimization problems with constraints. The method analysed allows inexact solution of the subproblems and do not require prior knowledge of the Lipschitz constants related with the problem. When an approximate feasible point is used as starting point, it is shown that the referred method takes at most outer iterations to generate an ϵ-approximate KKT point, where is the first penalty parameter. For equality constrained problems, this bound yields to an evaluation complexity bound of , when and suitable first-order methods are used as inner solvers. For problems having only linear equality constraints, an evaluation complexity bound of is established when appropriate p-order methods () are used as inner solvers. Illustrative numerical results are also presented and corroborate the theoretical predictions.

Conditions for linear convergence of the gradient method for non-convex optimization

In this paper, we derive a new linear convergence rate for the gradient method with fixed step lengths for non-convex smooth optimization problems satisfying the Polyak-Łojasiewicz (PŁ) inequality. We establish that the PŁ inequality is a necessary and sufficient condition for linear convergence to the optimal value for this class of problems. We list some related classes of functions for which the gradient method may enjoy linear convergence rate. Moreover, we investigate their relationship with the PŁ inequality.