Local Superlinear Convergence Research Articles

Approximation Conjugate Gradient Method for Low-Rank Matrix Recovery

Large-scale symmetric and asymmetric matrices have emerged in predicting the relationship between genes and diseases. The emergence of large-scale matrices increases the computational complexity of the problem. Therefore, using low-rank matrices instead of original symmetric and asymmetric matrices can greatly reduce computational complexity. In this paper, we propose an approximation conjugate gradient method for solving the low-rank matrix recovery problem, i.e., the low-rank matrix is obtained to replace the original symmetric and asymmetric matrices such that the approximation error is the smallest. The conjugate gradient search direction is given through matrix addition and matrix multiplication. The new conjugate gradient update parameter is given by the F-norm of matrix and the trace inner product of matrices. The conjugate gradient generated by the algorithm avoids SVD decomposition. The backtracking linear search is used so that the approximation conjugate gradient direction is computed only once, which ensures that the objective function decreases monotonically. The global convergence and local superlinear convergence of the algorithm are given. The numerical results are reported and show the effectiveness of the algorithm.

An inexact variable metric variant of incremental aggregated Forward-Backward method for the large-scale composite optimization problem

ABSTRACT This paper focuses on the large-scale composite optimization problem, which is the sum of a finite number of smooth (possibly nonconvex) functions and a convex (possibly nonsmooth) function. A common method to solve this problem is proximal incremental aggregated gradient (PIAG) method (N. D. Vanli, M. Gurbuzbalaban and A. Ozdaglar, SIAM Journal on Optimization, 2018, 28(2): 1282–1300), which exploits the gradient information from previous iterations to approximate the gradient at the current point. However, as many first-order algorithms, it may suffer from slow convergence. To address this issue, this paper proposes an inexact variable metric variant of incremental aggregated Forward-Backward (iVMFB-IAG) method. This methods incorporates a variable metric strategy to accelerate the PIAG method and introduces an inexact criterion to enhance its practicality. Under the Kurdyka-Łojasiewicz (KL) property and the uniformly bounded positive definiteness of the scaling matrix, the sequence generated by the iVMFB-IAG method converges globally to the critical point of the problem. Additionally, the convergence rate is established under the KL framework. Moreover, under certain restrictive conditions, the method exhibits local superlinear convergence. Through the numerical experiments on image reconstruction applications, we demonstrate the effectiveness of the iVMFB-IAG method by restoring two distinct images.

Incremental Quasi-Newton Methods with Faster Superlinear Convergence Rates

We consider the finite-sum optimization problem, where each component function is strongly convex and has Lipschitz continuous gradient and Hessian. The recently proposed incremental quasi-Newton method is based on BFGS update and achieves a local superlinear convergence rate that is dependent on the condition number of the problem. This paper proposes a more efficient quasi-Newton method by incorporating the symmetric rank-1 update into the incremental framework, which results in the condition-number-free local superlinear convergence rate. Furthermore, we can boost our method by applying the block update on the Hessian approximation, which leads to an even faster local convergence rate. The numerical experiments show the proposed methods significantly outperform the baseline methods.

Communication-efficient distributed cubic Newton with compressed lazy Hessian

Recently, second-order distributed optimization algorithms have been becoming a research hot in distributed learning, due to their faster convergence rate than the first-order algorithms. However, second-order algorithms always suffer from serious communication bottleneck. To conquer such challenge, we propose communication-efficient second-order distributed optimization algorithms in the parameter-server framework, by incorporating cubic Newton methods with compressed lazy Hessian. Specifically, our algorithms require each worker communicate compressed Hessians with the server only at some particular iterations, which can save both communication bits and communication rounds. For non-convex problems, we theoretically prove that our algorithms can reduce the communication cost comparing to the state-of-the-art second-order algorithms, while maintaining the same iteration complexity order O(ϵ−3/2) as the centralized cubic Newton methods. By further using gradient regularization technique, our algorithms can achieve global convergence for convex problems. Moreover, for strongly convex problems, our algorithms can achieve local superlinear convergence rate without any requirement on initial conditions. Finally, numerical experiments are conducted to show the high efficiency of the proposed algorithms.

A Hybrid Genetic Algorithm-Based Parameter Identification Method for Nonlinear Hysteretic System with Experimental Verification

The Bouc–Wen nonlinear hysteretic model is widely used in nonlinear systems, and it is difficult to identify the parameters of Bouc–Wen model because of its high nonlinearity. In the paper, a hybrid genetic algorithm combined with sequential quadratic programming is proposed to solve the problem about the parameter identification of Bouc–Wen model. Sequential quadratic programming method has good local convergence and superlinear convergence rate. Therefore, compared with the genetic algorithm, the proposed identification method can accelerate the convergence speed in parameter recognition of Bouc–Wen model and achieve excellent recognition accuracy. Through the function test and the numerical simulation of the shear structure, it is proved that the proposed method has high convergence speed and accurate identification. In addition, the theory and method proposed in this paper are combined with Bouc–Wen model to identify parameters of overall restoring force model of the energy-dissipation structures, the identification results are in good agreement with the experimental results, verifying reliability and accuracy of the model and robustness of the proposed identification method.

Strong Convergence of a Two-Step Modified Newton Method for Weighted Complementarity Problems

This paper focuses on the weighted complementarity problem (WCP), which is widely used in the fields of economics, sciences and engineering. Not least because of its local superlinear convergence rate, smoothing Newton methods have widespread application in solving various optimization problems. A two-step smoothing Newton method with strong convergence is proposed. With a smoothing complementary function, the WCP is reformulated as a smoothing set of equations and solved by the proposed two-step smoothing Newton method. In each iteration, the new method computes the Newton equation twice, but using the same Jacobian, which can avoid consuming a lot of time in the calculation. To ensure the global convergence, a derivative-free line search rule is inserted. At the same time, we develop a different term in the solution of the smoothing Newton equation, which guarantees the local strong convergence. Under appropriate conditions, the algorithm has at least quadratic or even cubic local convergence. Numerical experiments indicate the stability and effectiveness of the new method. Moreover, compared to the general smoothing Newton method, the two-step smoothing Newton method can significantly improve the computational efficiency without increasing the computational cost.

Optimal Demand Shut-offs of AC Microgrid using AO-SBQP Method

Microgrids are increasingly being utilized to improve the resilience and operational flexibility of power grids, and act as a backup power source during grid outages. However, it necessitates that the microgrid itself could provide power to the critical loads. This paper presents an algorithm named alternating optimization based sequential boolean quadratic programming tailored for solving optimal demand shut-offs problems arising in microgrids. Moreover, we establish local superlinear convergence of the proposed approximate Boolean quadratic programming method over nonconvex problems. In the end, the performance of the proposed method is illustrated on the modified IEEE 30-bus case study.

A Generalized Newton Method for Subgradient Systems

This paper proposes and develops a new Newton-type algorithm to solve subdifferential inclusions defined by subgradients of extended real-valued prox-regular functions. The proposed algorithm is formulated in terms of the second order subdifferential of such functions that enjoy extensive calculus rules and can be efficiently computed for broad classes of extended real-valued functions. Based on this and on the metric regularity and subregularity properties of subgradient mappings, we establish verifiable conditions ensuring the well-posedness of the proposed algorithm and its local superlinear convergence. The obtained results are also new for the class of equations defined by continuously differentiable functions with Lipschitzian gradients ([Formula: see text] functions), which is the underlying case of our consideration. The developed algorithms for prox-regular functions and their extension to a structured class of composite functions are formulated in terms of proximal mappings and forward–backward envelopes. Besides numerous illustrative examples and comparison with known algorithms for [Formula: see text] functions and generalized equations, the paper presents applications of the proposed algorithms to regularized least square problems arising in statistics, machine learning, and related disciplines. Funding: Research of P. D. Khanh is funded by Ho Chi Minh City University of Education Foundation for Science and Technology [Grant CS.2022.19.20TD]. Research of B. Mordukhovich and V. T. Phat was partly supported by the U.S. National Science Foundation [Grants DMS-1808978 and DMS-2204519]. The research of B. Mordukhovich was also supported by the Air Force Office of Scientific Research [Grant 15RT0462] and the Australian Research Council under Discovery Project DP-190100555. This work was supported by the Air Force Office of Scientific Research [Grant 15RT0462].

Variable Metric Method for Unconstrained Multiobjective Optimization Problems

In this paper, we propose a variable metric method for unconstrained multiobjective optimization problems (MOPs). First, a sequence of points is generated using different positive definite matrices in the generic framework. It is proved that accumulation points of the sequence are Pareto critical points. Then, without convexity assumption, strong convergence is established for the proposed method. Moreover, we use a common matrix to approximate the Hessian matrices of all objective functions, along which a new nonmonotone line search technique is proposed to achieve a local superlinear convergence rate. Finally, several numerical results demonstrate the effectiveness of the proposed method.

A semismooth Newton based augmented Lagrangian method for nonsmooth optimization on matrix manifolds

This paper is devoted to studying an augmented Lagrangian method for solving a class of manifold optimization problems, which have nonsmooth objective functions and nonlinear constraints. Under the constant positive linear dependence condition on manifolds, we show that the proposed method converges to a stationary point of the nonsmooth manifold optimization problem. Moreover, we propose a globalized semismooth Newton method to solve the augmented Lagrangian subproblem on manifolds efficiently. The local superlinear convergence of the manifold semismooth Newton method is also established under some suitable conditions. We also prove that the semismoothness on submanifolds can be inherited from that in the ambient manifold. Finally, numerical experiments on compressed modes and (constrained) sparse principal component analysis illustrate the advantages of the proposed method.

A derivative-free modified tensor method with curvilinear linesearch for unconstrained nonlinear programming

In this paper, a random derivative-free modified tensor method with curvilinear linesearch technique is considered for solving nonlinear programming problems. The proposed algorithm is designed to build polynomial interpolation models for the objective function and build the tensor model using the information of the interpolation function. At the same time, we give a new curvilinear tensor step which guarantees the monotonic decrease on the tensor model. The modified tensor step also asymptotically approaches the modified Newton direction as the step length shrinks to zero, and the objective function of problem will be descendent. Under general assumptions, we give the global and local superlinear convergence of the algorithm. Numerical results are recorded, and the compare results with a tensor algorithm without curvilinear linesearch technique and Newton algorithm show that our algorithm is more effectiveness.

Non-asymptotic superlinear convergence of standard quasi-Newton methods

In this paper, we study and prove the non-asymptotic superlinear convergence rate of the Broyden class of quasi-Newton algorithms which includes the Davidon–Fletcher–Powell (DFP) method and the Broyden–Fletcher–Goldfarb–Shanno (BFGS) method. The asymptotic superlinear convergence rate of these quasi-Newton methods has been extensively studied in the literature, but their explicit finite–time local convergence rate is not fully investigated. In this paper, we provide a finite–time (non-asymptotic) convergence analysis for Broyden quasi-Newton algorithms under the assumptions that the objective function is strongly convex, its gradient is Lipschitz continuous, and its Hessian is Lipschitz continuous at the optimal solution. We show that in a local neighborhood of the optimal solution, the iterates generated by both DFP and BFGS converge to the optimal solution at a superlinear rate of (1/k)^{k/2}, where k is the number of iterations. We also prove a similar local superlinear convergence result holds for the case that the objective function is self-concordant. Numerical experiments on several datasets confirm our explicit convergence rate bounds. Our theoretical guarantee is one of the first results that provide a non-asymptotic superlinear convergence rate for quasi-Newton methods.

A global convergent semi-smooth Newton method for semi-linear elliptic optimal control problem

Global convergent semi-smooth Newton (GCSSN) method for L 2 -norm control constrained elliptic optimal control problem with L 1 -control cost is discussed. The first order necessary optimality conditions are analyzed and reduced system is proposed. After finite difference discretization, we propose the global convergent semi-smooth Newton method for discrete reduced system with the nonmonotone line search. The local superlinear convergence rate and convergence are proved theoretically. The demonstrated theoretical properties are verified with numerical results. To illustrate the effectiveness and efficiency, we compare the proposed method with semi-smooth Newton method in the simulation part. • Global convergent semi-smooth Newton method for semi-linear elliptic optimal control problem is proposed. • We obtain the first order optimality necessary condition for optimal control problem. • The convergence properties of algorithm are proven theoretically and verified by simulations.

A Subspace Acceleration Method for Minimization Involving a Group Sparsity-Inducing Regularizer

We consider the problem of minimizing an objective function that is the sum of a convex function and a group sparsity-inducing regularizer. Problems that integrate such regularizers arise in modern machine learning applications, often for the purpose of obtaining models that are easier to interpret and that have higher predictive accuracy. We present a new method for solving such problems that utilize subspace acceleration, domain decomposition, and support identification. Our analysis shows, under common assumptions, that the iterate sequence generated by our framework is globally convergent, converges to an $\epsilon$-approximate solution in at most $O(\epsilon^{-(1+p)})$ (respectively, $O(\epsilon^{-(2+p)})$) iterations for all $\epsilon$ bounded above and large enough (respectively, all $\epsilon$ bounded above) where $p > 0$ is an algorithm parameter, and exhibits superlinear local convergence. Preliminary numerical results for the task of binary classification based on regularized logistic regression show that our approach is efficient and robust, with the ability to outperform a state-of-the-art method.

Cubic Regularized Newton Method for the Saddle Point Models: A Global and Local Convergence Analysis

In this paper, we propose a cubic regularized Newton method for solving the convex-concave minimax saddle point problems. At each iteration, a cubic regularized saddle point subproblem is constructed and solved, which provides a search direction for the iterate. With properly chosen stepsizes, the method is shown to converge to the saddle point with global linear and local superlinear convergence rates, if the saddle point function is gradient Lipschitz and strongly-convex-strongly-concave. In the case that the function is merely convex-concave, we propose a homotopy continuation (or path-following) method. Under a Lipschitz-type error bound condition, we present an iteration complexity bound of \({\mathcal {O}}\left( \ln \left( 1/\epsilon \right) \right) \) to reach an \(\epsilon \)-solution through a homotopy continuation approach, and the iteration complexity bound becomes \({\mathcal {O}}\left( \left( 1/\epsilon \right) ^{\frac{1-\theta }{\theta ^2}}\right) \) under a Hölderian-type error bound condition involving a parameter \(\theta \) (\(0<\theta <1\)).

A dual-mixed approximation for a Huber regularization of generalized p-Stokes viscoplastic flow problems

In this paper, we propose a dual-mixed formulation for stationary viscoplastic flows with yield, such as the Bingham or the Herschel-Bulkley flow. The approach is based on a Huber regularization of the viscosity term and a two-fold saddle point nonlinear operator equation for the resulting weak formulation. We provide the uniqueness of solutions for the continuous formulation and propose a discrete scheme based on Arnold-Falk-Winther finite elements. The discretization scheme yields a system of slantly differentiable nonlinear equations, for which a semismooth Newton algorithm is proposed and implemented. Local superlinear convergence of the method is also proved. Finally, we perform several numerical experiments in two and three dimensions to investigate the behavior and efficiency of the method.

An Efficient Fisher Matrix Approximation Method for Large-Scale Neural Network Optimization.

Although the shapes of the parameters are not crucial for designing first-order optimization methods in large scale empirical risk minimization problems, they have important impact on the size of the matrix to be inverted when developing second-order type methods. In this article, we propose an efficient and novel second-order method based on the parameters in the real matrix space [Formula: see text] and a matrix-product approximate Fisher matrix (MatFisher) by using the products of gradients. The size of the matrix to be inverted is much smaller than that of the Fisher information matrix in the real vector space [Formula: see text]. Moreover, by utilizing the matrix delayed update and the block diagonal approximation techniques, the computational cost can be controlled and is comparable with first-order methods. A global convergence and a superlinear local convergence analysis are established under mild conditions. Numerical results on image classification with ResNet50, quantum chemistry modeling with SchNet, and data-driven partial differential equations solution with PINN illustrate that our method is quite competitive to the state-of-the-art methods.

RFN: A Random-Feature Based Newton Method for Empirical Risk Minimization in Reproducing Kernel Hilbert Spaces

In supervised learning using kernel methods, we often encounter a large-scale finite-sum minimization over a reproducing kernel Hilbert space (RKHS). Large-scale finite-sum problems can be solved using efficient variants of Newton method, where the Hessian is approximated via sub-samples of data. In RKHS, however, the dependence of the penalty function to kernel makes standard sub-sampling approaches inapplicable, since the gram matrix is not readily available in a low-rank form. In this paper, we observe that for this class of problems, one can naturally use kernel approximation to speed up the Newton method. Focusing on randomized features for kernel approximation, we provide a novel second-order algorithm that enjoys local superlinear convergence and global linear convergence (with high probability). We derive the theoretical lower bound for the number of random features required for the approximated Hessian to be close to the true Hessian in the norm sense. Our numerical experiments on real-world data verify the efficiency of our method compared to several benchmarks.

A Class of Inexact Secant Algorithms with Line Search Filter Method for Nonlinear Programming

We propose a class of inexact secant methods in association with the line search filter technique for solving nonlinear equality constrained optimization. Compared with other filter methods that combine the line search method applied in most large-scale optimization problems, the inexact line search filter algorithm is more flexible and realizable. In this paper, we focus on the analysis of the local superlinear convergence rate of the algorithms, while their global convergence properties can be obtained by making an analogy with our previous work. These methods have been implemented in a Matlab code, and detailed numerical results indicate that the proposed algorithms are efficient for 43 problems from the CUTEr test set.

Semismooth Newton-type method for bilevel optimization: global convergence and extensive numerical experiments

We consider the standard optimistic bilevel optimization problem, in particular upper- and lower-level constraints can be coupled. By means of the lower-level value function, the problem is transformed into a single-level optimization problem with a penalization of the value function constraint. For treating the latter problem, we develop a framework that does not rely on the direct computation of the lower-level value function or its derivatives. For each penalty parameter, the framework leads to a semismooth system of equations. This allows us to extend the semismooth Newton method to bilevel optimization. Besides global convergence properties of the method, we focus on achieving local superlinear convergence to a solution of the semismooth system. To this end, we formulate an appropriate CD-regularity assumption and derive sufficient conditions so that it is fulfilled. Moreover, we develop conditions to guarantee that a solution of the semismooth system is a local solution of the bilevel optimization problem. Extensive numerical experiments on 124 examples of nonlinear bilevel optimization problems from the literature show that this approach exhibits a remarkable performance, where only a few penalty parameters need to be considered.