Line Search Strategy Research Articles

Nonmonotone Spectral Analysis for Variational Inclusions

Gradient descent algorithms perform well in convex optimization but can get tied for finding local minima in non-convex optimization. A robust method that combines a spectral approach with nonmonotone line search strategy for solving variational inclusion problems is proposed. Spectral properties using eigenvalues information are used for accelerating the convergence. Nonmonotonic behaviour is exhibited to relax descent property and escape local minima. Nonmonotone spectral conditions leverage adaptive search directions and global convergence for the proposed spectral subgradient algorithm.

Primal-Dual Extrapolation Methods for Monotone Inclusions Under Local Lipschitz Continuity

In this paper, we consider a class of monotone inclusion (MI) problems of finding a zero of the sum of two monotone operators, in which one operator is maximal monotone, whereas the other is locally Lipschitz continuous. We propose primal-dual (PD) extrapolation methods to solve them using a point and operator extrapolation technique, whose parameters are chosen by a backtracking line search scheme. The proposed methods enjoy an operation complexity of [Formula: see text] and [Formula: see text], measured by the number of fundamental operations consisting only of evaluations of one operator and resolvent of the other operator, for finding an ε-residual solution of strongly and nonstrongly MI problems, respectively. The latter complexity significantly improves the previously best operation complexity [Formula: see text]. As a byproduct, complexity results of the primal-dual extrapolation methods are also obtained for finding an ε-KKT or ε-residual solution of convex conic optimization, conic constrained saddle point, and variational inequality problems under local Lipschitz continuity. We provide preliminary numerical results to demonstrate the performance of the proposed methods. Funding: This work was partially supported by the National Science Foundation [Grant IIS-2211491], the Office of Naval Research [Grant N00014-24-1-2702], and the Air Force Office of Scientific Research [Grant FA9550-24-1-0343].

A New Hybrid Descent Algorithm for Large-Scale Nonconvex Optimization and Application to Some Image Restoration Problems

Conjugate gradient methods are widely used and attractive for large-scale unconstrained smooth optimization problems, with simple computation, low memory requirements, and interesting theoretical information on the features of curvature. Based on the strongly convergent property of the Dai–Yuan method and attractive numerical performance of the Hestenes–Stiefel method, a new hybrid descent conjugate gradient method is proposed in this paper. The proposed method satisfies the sufficient descent property independent of the accuracy of the line search strategies. Under the standard conditions, the trust region property and the global convergence are established, respectively. Numerical results of 61 problems with 9 large-scale dimensions and 46 ill-conditioned matrix problems reveal that the proposed method is more effective, robust, and reliable than the other methods. Additionally, the hybrid method also demonstrates reliable results for some image restoration problems.

A new spectral conjugate subgradient method with application in computed tomography image reconstruction

A new spectral conjugate subgradient method is presented to solve nonsmooth unconstrained optimization problems. The method combines the spectral conjugate gradient method for smooth problems with the spectral subgradient method for nonsmooth problems. We study the effect of two different choices of line search, as well as three formulas for determining the conjugate directions. In addition to numerical experiments with standard nonsmooth test problems, we also apply the method to several image reconstruction problems in computed tomography, using total variation regularization. Performance profiles are used to compare the performance of the algorithm using different line search strategies and conjugate directions to that of the original spectral subgradient method. Our results show that the spectral conjugate subgradient algorithm outperforms the original spectral subgradient method, and that the use of the Polak–Ribière formula for conjugate directions provides the best and most robust performance.

An improved descent Perry‐type algorithm for large‐scale unconstrained nonconvex problems and applications to image restoration problems

AbstractConjugate gradient methods are much effective for large‐scale unconstrained optimization problems by their simple computations and low memory requirements. The Perry conjugate gradient method has been considered to be one of the most efficient methods in the context of unconstrained minimization. However, a globally convergent result for general functions has not been established yet. In this paper, an improved three‐term Perry‐type algorithm is proposed which automatically satisfies the sufficient descent property independent of the accuracy of line search strategy. Under the standard Wolfe line search technique and a modified secant condition, the proposed algorithm is globally convergent for general nonlinear functions without convexity assumption. Numerical results compared with the Perry method for stability, two modified Perry‐type conjugate gradient methods and two effective three‐term conjugate gradient methods for large‐scale problems up to 300,000 dimensions indicate that the proposed algorithm is more efficient and reliable than the other methods for the testing problems. Additionally, we also apply it to some image restoration problems.

AN-SPS: adaptive sample size nonmonotone line search spectral projected subgradient method for convex constrained optimization problems

We consider convex optimization problems with a possibly nonsmooth objective function in the form of a mathematical expectation. The proposed framework (AN-SPS) employs Sample Average Approximations (SAA) to approximate the objective function, which is either unavailable or too costly to compute. The sample size is chosen in an adaptive manner, which eventually pushes the SAA error to zero almost surely (a.s.). The search direction is based on a scaled subgradient and a spectral coefficient, both related to the SAA function. The step size is obtained via a nonmonotone line search over a predefined interval, which yields a theoretically sound and practically efficient algorithm. The method retains feasibility by projecting the resulting points onto a feasible set. The a.s. convergence of AN-SPS method is proved without the assumption of a bounded feasible set or bounded iterates. Preliminary numerical results on Hinge loss problems reveal the advantages of the proposed adaptive scheme. In addition, a study of different nonmonotone line search strategies in combination with different spectral coefficients within AN-SPS framework is also conducted, yielding some hints for future work.

Channel Estimation for Underwater Acoustic Communications in Impulsive Noise Environments: A Sparse, Robust, and Efficient Alternating Direction Method of Multipliers-Based Approach

Channel estimation in Underwater Acoustic Communication (UAC) faces significant challenges due to the non-Gaussian, impulsive noise in ocean environments and the inherent high dimensionality of the estimation task. This paper introduces a robust channel estimation algorithm by solving an l1−l1 optimization problem via the Alternating Direction Method of Multipliers (ADMM), effectively exploiting channel sparsity and addressing impulsive noise outliers. A non-monotone backtracking line search strategy is also developed to improve the convergence behavior. The proposed algorithm is low in complexity and has robust performance. Simulation results show that it exhibits a small performance deterioration of less than 1 dB for Channel Impulse Response (CIR) estimation in impulsive noise environments, nearly matching its performance under Additive White Gaussian Noise (AWGN) conditions. For Delay-Doppler (DD) doubly spread channel estimation, it maintains Bit Error Rate (BER) performance comparable to using ground truth channel information in both AWGN and impulsive noise environments. At-sea experimental validations for channel estimation in Orthogonal Frequency Division Multiplexing (OFDM) systems further underscore the fast convergence speed and high estimation accuracy of the proposed method.

An accelerated stochastic extragradient-like algorithm with new stepsize rules for stochastic variational inequalities

In this paper, we devise a stochastic extragradient-like algorithm incorporated with inertial terms, which requires a single projection onto our feasible set and employs a stochastic approximation version of an Armijo-type line search scheme along a feasible direction, for solving pseudomonotone stochastic variational inequalities. In the algorithm, two different stepsize strategies are employed to update steplength sequences without using the prior knowledge of the Lipschitz constant of involved operators. In the process of the stochastic approximation, we iteratively reduce the variance of stochastic errors. The almost sure convergence, the complexity analysis, and rates are provided in a dimensional space under reasonable conditions. Finally, some numerical experiments with graphical illustrations are reported to demonstrate the applicability and the efficiency of our algorithm in comparison with some projection type methods in the literature.

Improved FORM and SORM Based on Improved Modified Symmetric Rank 1 Algorithm and Adaptive Kriging Model

Abstract The first-order reliability method (FORM) is simple and efficient for solving structural reliability problems but may have large errors and converge slowly or even result in divergence when dealing with strongly nonlinear performance functions. For this case, the existing second-order reliability method (SORM) achieves higher computational accuracy but with a consequent decrease in efficiency. To achieve a better balance between accuracy and efficiency, this paper proposes an improved FORM and an improved SORM. First, an improved modified symmetric rank 1 (IMSR1) algorithm, in which the line search strategy for step length is unnecessary, is proposed for iterations of the FORM, and an adaptive Kriging model with a rational update criterion is presented to improve the efficiency of the FORM. Then, an improved FORM with high efficiency and good convergence is proposed. Second, due to the good precision of the adaptive Kriging model at the final design point, the Hessian matrix is available easily without additional computational effort, and an improved SORM with the same efficiency as the improved FORM is presented naturally. Finally, the accuracy, efficiency, and convergence of the proposed methods are verified by numerical and engineering examples.

An Adaptive Low Computational Cost Alternating Direction Method of Multiplier for RELM Large-Scale Distributed Optimization

In a class of large-scale distributed optimization, the calculation of RELM based on the Moore–Penrose inverse matrix is prohibitively expensive, which hinders the formulation of a computationally efficient optimization model. Attempting to improve the model’s convergence performance, this paper proposes a low computing cost Alternating Direction Method of Multipliers (ADMM), where the original update in ADMM is solved inexactly with approximate curvature information. Based on quasi-Newton techniques, the ADMM approach allows us to solve convex optimization with reasonable accuracy and computational effort. By introducing this algorithm into the RELM model, the model fitting problem can be decomposed into a set of subproblems that can be executed in parallel to achieve efficient classification performance. To avoid the storage of expensive Hessian for large problems, BFGS with limited memory is proposed with computational efficiency. And the optimal parameter values of the step-size search method are obtained through Wolfe line search strategy. To demonstrate the superiority of our methods, numerical experiments are conducted on eight real-world datasets. Results on problems arising in machine learning suggest that the proposed method is competitive with other similar methods, both in terms of better computational efficiency as well as accuracy.

Modeling elasticity, similarity, stochasticity, and congestion in a network equilibrium framework using a paired combinatorial weibit choice model

In the traffic assignment problem for predicting traffic flow patterns in a transportation network, it is important to account for route overlap and non-identical perception variance in route choice analysis. In this study, we establish a novel route choice model, named the paired combinatorial weibit (PCW) model, to capture the route overlap and route-specific perception variance. The PCW model retains a closed-form probability solution, which allows the development of an equivalent mathematical programming (MP) formulation for the PCW-based stochastic user equilibrium (PCW-SUE) model. Specifically, we propose two equivalent MP formulations for modeling the fixed demand (FD) and elastic demand (ED), named PCW-SUE-FD and PCW-SUE-ED, respectively. The PCW-SUE-ED model can address the abovementioned two issues in route choice for the FD scheme, but also can consider the effect level-of-service (LOS) in travel choice for the ED scheme. The equivalency and uniqueness of the PCW-SUE-FD and PCW-SUE-ED models are rigorously proved. In addition, a path-based partial linearization algorithm combined with a self-regulated averaging line search strategy is developed to solve the two SUE models. Numerical results are presented to illustrate the features of the PCW-SUE-FD and PCW-SUE-ED models and applicability of the solution algorithm to a real transportation network.

A projection-based hybrid PRP-DY type conjugate gradient algorithm for constrained nonlinear equations with applications

Based on the convex combination technique, we propose a projection-based hybrid conjugate gradient algorithm for solving nonlinear equations with convex constraints in this paper. The conjugate parameter of the proposed algorithm is a convex combination of the modified Polak-Ribière-Polyak and Dai-Yuan type conjugate parameters, and the search direction has the sufficient descent property without the use of a line search strategy. The proposed hybrid algorithm's global convergence is established under appropriate assumptions. The numerical experiments demonstrate that the proposed algorithm is more efficient and competitive than existing methods under some benchmark test problems. Furthermore, it is also extended to solve the sparse signal and impulse noise image restoration problem that arises in compressive sensing.

FedMDFG: Federated Learning with Multi-Gradient Descent and Fair Guidance

Fairness has been considered as a critical problem in federated learning (FL). In this work, we analyze two direct causes of unfairness in FL - an unfair direction and an improper step size when updating the model. To solve these issues, we introduce an effective way to measure fairness of the model through the cosine similarity, and then propose a federated multiple gradient descent algorithm with fair guidance (FedMDFG) to drive the model fairer. We first convert FL into a multi-objective optimization problem (MOP) and design an advanced multiple gradient descent algorithm to calculate a fair descent direction by adding a fair-driven objective to MOP. A low-communication-cost line search strategy is then designed to find a better step size for the model update. We further show the theoretical analysis on how it can enhance fairness and guarantee the convergence. Finally, extensive experiments in several FL scenarios verify that FedMDFG is robust and outperforms the SOTA FL algorithms in convergence and fairness. The source code is available at https://github.com/zibinpan/FedMDFG.

Gradient-based descent linesearch to solve interval-valued optimization problems under gH-differentiability with application to finance

In this article, gradient based descent line search scheme is proposed to solve interval optimization problems under generalized Hukuhara differentiability. The innovation and importance of these concepts are presented from practical and computational point of view. The necessary condition for existence of critical point is presented in inclusion form of interval-valued gradient, without transforming it into real valued function. Suitable efficient descent direction is chosen based on the monotonic property of the interval-valued function and specific interval ordering. Mathematical convergence of the scheme is proved under the assumption of Inexact line search for interval optimization problem. The theoretical developments are implemented with a set of interval test problems in different dimensions. A possible application in finance is provided and solved by our proposed scheme.

A Weibit-Based sequential transit assignment model based on hyperpath graph and generalized extreme value network representation

This study aims to develop a weibit-based stochastic transit assignment model for situations in which passengers’ travel choice strategies are stochastic. The weibit choice model, which was originally developed for the stochastic traffic assignment problem, is adapted using a hyperpath graph network representation. Unlike the traditional logit model, the weibit model can account for the homogeneous perception variance problem. Besides, a joint choice network representation originated from the network generalized extreme value (N-GEV) model, known as the GEV network, is adopted to depict the correlation among the arcs within the hypergraph consisting of all efficient hyperpaths. The GEV network uses a link-based parameter, the degree of membership, to capture the overlap effect, which is thus well incorporated with the link-based transit assignment approach. For implementation, a link-based stochastic network loading approach is proposed for solving the weibit-based transit assignment model in the sequential process. This sequential assignment approach does not need to enumerate and store the transit hyperpaths in the solution process. This model is then extended to a stochastic transit equilibrium model under congestion conditions, which is formulated as a fixed-point problem. Thus, a recent fast line-search scheme, i.e., the Barzilai and Borwein step-size, is adopted for solving the stochastic transit equilibrium model. Numerical examples are provided to illustrate the features of the proposed weibit-based stochastic transit assignment model. A real transit network is used to demonstrate the algorithm efficiency and the detailed results of the weibit-based transit equilibrium model.

A small deformations effective stress model of gradient plasticity phase-field fracture

A variational formulation of small strain ductile fracture, based on a phase-field modeling of crack propagation, is proposed. The formulation is based on an effective stress description of gradient plasticity, combined with an AT1 phase-field model. Starting from established variational statements of finite-step elastoplasticity for generalized standard materials, a mixed variational statement is consistently derived, incorporating in a rigorous way a variational finite-step update for both the elastoplastic and the phase-field dissipations. The complex interaction between ductile and brittle dissipation mechanisms is modeled by assuming a plasticity driven crack propagation model. A non-variational function of the equivalent plastic strain is then introduced to modulate the phase-field dissipation based on the developed plastic strains. Particular care has been devoted to the formulation of a consistent Newton–Raphson scheme for the case of Mises plasticity, with a global return mapping and relative tangent matrix, supplemented by a line-search scheme, for the solution of the gradient elastoplasticity problem for fixed phase field. The resulting algorithm has proved to be very robust and computationally effective. Application to several benchmark tests show the robustness and accuracy of the proposed model.

A modified PRP-type conjugate gradient algorithm with complexity analysis and its application to image restoration problems

In this paper, a modified conjugate gradient method is proposed for nonconvex optimization. This method possesses the sufficient descent property independent of any line search. The global convergence property of the algorithm is established under the Wolfe line search strategy or the Armijo line search condition, respectively. Additionally, the complexity analysis of the proposed algorithm is investigated. To reach the point with the norm of the gradient below ɛ, the worst-case complexity bound matches that of the gradient method. Moreover, the obtained numerical results demonstrate that the modified method is effective for large-scale optimization problems and image restoration problems.

Extending the inverse sequential quasi-Newton method for on-line monitoring and controlling of process conditions in the solidification of alloys

This work introduces the sequential quasi-Newton method (SQNM) to solve the inverse design problem of estimating the transient boundary heat fluxes that produce desired evolutions of the solidus and liquidus isotherms in alloys’ solidification processes. This is an unprecedented application for sequential gradient-based methods. The final goal is monitoring and controlling the evolution of the mushy zone in a near real-time fashion, which is of high technological interest to the foundry industry. We assess the performance of the SQNM in solving several inverse design problems in one- and two-dimensional domains. The method satisfactorily estimates the boundary heat fluxes using relatively few future measurements. However, some instabilities have been detected in the SQNM-predicted heat fluxes. To alleviate this issue, we proposed a modified version of SQNM, say mSQNM, that employs an inexact line search strategy to find the optimal step length during the iterative estimation procedure. We finally show how mSQNM improves the estimation of the boundary heat fluxes.

A class of spectral conjugate gradient methods for Riemannian optimization

Spectral conjugate gradient (SCG) methods are combinations of spectral gradient method and conjugate gradient (CG) methods, which have been well studied in Euclidean space. In this paper, we aim to extend this class of methods to solve optimization problems on Riemannian manifolds. Firstly, we present a Riemannian version of the spectral parameter, which guarantees that the search direction always satisfies the sufficient descent property without the help of any line search strategy. Secondly, we introduce a generic algorithmic framework for the Riemannian SCG methods, in which the selection of the CG parameter is very flexible. Under the Riemannian Wolfe conditions, the global convergence of the proposed algorithmic framework is established whenever the absolute value of the CG parameter is no more than the Riemannian Fletcher–Reeves CG parameter. Finally, some preliminary numerical results are reported and compared with several classical Riemannian CG methods, which show that our new methods are efficient.

Learning rate selection in stochastic gradient methods based on line search strategies

Finite-sum problems appear as the sample average approximation of a stochastic optimization problem and often arise in machine learning applications with large scale data sets. A very popular approach to face finite-sum problems is the stochastic gradient method. It is well known that a proper strategy to select the hyperparameters of this method (i.e. the set of a-priori selected parameters) and, in particular, the learning rate, is needed to guarantee convergence properties and good practical performance. In this paper, we analyse standard and line search based updating rules to fix the learning rate sequence, also in relation to the size of the mini batch chosen to compute the current stochastic gradient. An extensive numerical experimentation is carried out in order to evaluate the effectiveness of the discussed strategies for convex and non-convex finite-sum test problems, highlighting that the line search based methods avoid expensive initial setting of the hyperparameters. The line search based approaches have also been applied to train a Convolutional Neural Network, providing very promising results.