Backtracking Line Search Research Articles

Memoryless quasi-Newton-type methods via some weak secant relations for large-scale unconstrained optimization

This article presents two variants of memoryless quasi-Newton methods with backtracking line search for large-scale unconstrained minimization. These updating methods are derived by means of a least-change updating strategy subjected to some weaker form of secant relation obtained by projecting the secant equation onto the search direction. In such a setting, the search direction can be computed without the need of calculation and storage of matrices. We establish the convergence properties for these methods, and their performance is tested on a large set of test functions by comparing with standard methods of similar computational cost and storage requirement. Our numerical results indicate that significant improvement has been achieved with respect to iteration counts and number of function evaluations.

Primal-Dual Extrapolation Methods for Monotone Inclusions Under Local Lipschitz Continuity

In this paper, we consider a class of monotone inclusion (MI) problems of finding a zero of the sum of two monotone operators, in which one operator is maximal monotone, whereas the other is locally Lipschitz continuous. We propose primal-dual (PD) extrapolation methods to solve them using a point and operator extrapolation technique, whose parameters are chosen by a backtracking line search scheme. The proposed methods enjoy an operation complexity of [Formula: see text] and [Formula: see text], measured by the number of fundamental operations consisting only of evaluations of one operator and resolvent of the other operator, for finding an ε-residual solution of strongly and nonstrongly MI problems, respectively. The latter complexity significantly improves the previously best operation complexity [Formula: see text]. As a byproduct, complexity results of the primal-dual extrapolation methods are also obtained for finding an ε-KKT or ε-residual solution of convex conic optimization, conic constrained saddle point, and variational inequality problems under local Lipschitz continuity. We provide preliminary numerical results to demonstrate the performance of the proposed methods. Funding: This work was partially supported by the National Science Foundation [Grant IIS-2211491], the Office of Naval Research [Grant N00014-24-1-2702], and the Air Force Office of Scientific Research [Grant FA9550-24-1-0343].

LibEMMI_MGFD: A program of marine controlled-source electromagnetic modelling and inversion using frequency-domain multigrid solver

We develop a software package libEMMI_MGFD for 3D frequency-domain marine controlled-source electromagnetic (CSEM) modelling and inversion. It is the first open-source C program tailored for geometrical multigrid (GMG) CSEM simulation. An volumetric anisotropic averaging scheme has been employed to compute effective medium for modelling over uniform and nonuniform grid. The computing coordinate is aligned with acquisition geometry by rotation with the azimuth and dip angles, facilitating the injection of the source and the extraction of data with arbitrary orientations. Efficient nonlinear optimization is achieved using quasi-Newton scheme assisted with bisection backtracking line search. In constructing the modularized Maxwell solver and evaluating the misfit and gradient for 3D CSEM inversion, the reverse communication technique is the key to the compaction of the software while maintaining the computational performance. A number of numeric tests demonstrate the efficiency of the modelling while preserving the solution accuracy. A 3D marine CSEM inversion example has been examined for resistivity imaging. Program summaryProgram Title: libEMMI_MGFDCPC Library link to program files:https://doi.org/10.17632/mpk7xrd38m.1Developer's repository link:https://github.com/yangpl/libEMMI_MGFDLicensing provisions: GNU General Public License v3.0Programming language: C, ShellSoftware dependencies: MPI [1]Nature of problem: 3D Controlled-source electromagnetic modelling and inversion.Solution method: Frequency-domain modelling on staggered grid; Geometrical multigrid (GMG) method as an iterative solver; Quasi-Newton method for nonlinear minimization; Bisection-based backtracking line search.

An effective subgradient algorithm via Mifflin’s line search for nonsmooth nonconvex multiobjective optimization

We propose a descent subgradient algorithm for unconstrained nonsmooth nonconvex multiobjective optimization problems. To find a descent direction, we present an iterative process that efficiently approximates the ɛ-subdifferential of each objective function. To this end, we develop a new variant of Mifflin’s line search in which the subgradients are arbitrary and its finite convergence is proved under a semismooth assumption. To reduce the number of subgradient evaluations, we employ a backtracking line search that identifies the objectives requiring an improvement in the current approximation of the ɛ-subdifferential. Meanwhile, for the remaining objectives, new subgradients are not computed. Unlike bundle-type methods, the proposed approach can handle nonconvexity without the need for algorithmic adjustments. Moreover, the quadratic subproblems have a simple structure, and hence the method is easy to implement. We analyze the global convergence of the proposed method and prove that any accumulation point of the generated sequence satisfies a necessary Pareto optimality condition. Furthermore, our convergence analysis addresses a theoretical challenge in a recently developed subgradient method. Through numerical experiments, we observe the practical capability of the proposed method and evaluate its efficiency when applied to a diverse range of nonsmooth test problems.

Geometry Optimization Algorithms in Conjunction with the Machine Learning Potential ANI-2x Facilitate the Structure-Based Virtual Screening and Binding Mode Prediction.

Structure-based virtual screening utilizes molecular docking to explore and analyze ligand-macromolecule interactions, crucial for identifying and developing potential drug candidates. Although there is availability of several widely used docking programs, the accurate prediction of binding affinity and binding mode still presents challenges. In this study, we introduced a novel protocol that combines our in-house geometry optimization algorithm, the conjugate gradient with backtracking line search (CG-BS), which is capable of restraining and constraining rotatable torsional angles and other geometric parameters with a highly accurate machine learning potential, ANI-2x, renowned for its precise molecular energy predictions reassembling the wB97X/6-31G(d) model. By integrating this protocol with binding pose prediction using the Glide, we conducted additional structural optimization and potential energy prediction on 11 small molecule-macromolecule and 12 peptide-macromolecule systems. We observed that ANI-2x/CG-BS greatly improved the docking power, not only optimizing binding poses more effectively, particularly when the RMSD of the predicted binding pose by Glide exceeded around 5 Å, but also achieving a 26% higher success rate in identifying those native-like binding poses at the top rank compared to Glide docking. As for the scoring and ranking powers, ANI-2x/CG-BS demonstrated an enhanced performance in predicting and ranking hundreds or thousands of ligands over Glide docking. For example, Pearson's and Spearman's correlation coefficients remarkedly increased from 0.24 and 0.14 with Glide docking to 0.85 and 0.69, respectively, with the addition of ANI-2x/CG-BS for optimizing and ranking small molecules binding to the bacterial ribosomal aminoacyl-tRNA receptor. These results suggest that ANI-2x/CG-BS holds considerable potential for being integrated into virtual screening pipelines due to its enhanced docking performance.

Channel Estimation for Underwater Acoustic Communications in Impulsive Noise Environments: A Sparse, Robust, and Efficient Alternating Direction Method of Multipliers-Based Approach

Channel estimation in Underwater Acoustic Communication (UAC) faces significant challenges due to the non-Gaussian, impulsive noise in ocean environments and the inherent high dimensionality of the estimation task. This paper introduces a robust channel estimation algorithm by solving an l1−l1 optimization problem via the Alternating Direction Method of Multipliers (ADMM), effectively exploiting channel sparsity and addressing impulsive noise outliers. A non-monotone backtracking line search strategy is also developed to improve the convergence behavior. The proposed algorithm is low in complexity and has robust performance. Simulation results show that it exhibits a small performance deterioration of less than 1 dB for Channel Impulse Response (CIR) estimation in impulsive noise environments, nearly matching its performance under Additive White Gaussian Noise (AWGN) conditions. For Delay-Doppler (DD) doubly spread channel estimation, it maintains Bit Error Rate (BER) performance comparable to using ground truth channel information in both AWGN and impulsive noise environments. At-sea experimental validations for channel estimation in Orthogonal Frequency Division Multiplexing (OFDM) systems further underscore the fast convergence speed and high estimation accuracy of the proposed method.

An accelerated structured quasi-Newton method with a diagonal second-order Hessian approximation for nonlinear least squares problems

Our study focuses on exploring new variants of the structured quasi-Newton (SQN) method with a secant-like diagonal approximation (SLDA) of the second-order term of Hessian for solving nonlinear least squares (NLS) problems. In addition, an accelerated version of SQN-SLDA referred to as ASQN-SLDA, is also considered. In ASQN-SLDA, we rescale the search direction after the first backtracking linesearch procedure to produce a more aggressive step to increase the objective function value reduction. The concept of the proposed methods is simple and easy to implement. We prove the proposed methods are globally convergent under some appropriate assumptions and report several numerical experiments based on a suite of benchmark problems for NLS. The numerical results show that ASQN-SLDA is more robust than some baseline methods, including SQN-SLDA, the generalized Gauss–Newton (GN), Levenberg–Marquardt update, and the hybrid GN with Broyden–Fletcher–Goldfarb–Shanno (BFGS) method, also called H-GN-BFGS. For computing time, AQN-SLDA outperforms H-GN-BFGS for most test problems, and the speedup is more significant as the problem size increases. Furthermore, due to the trade-off between the number of iterations for convergence and the overhead needed in ASQN-SLDA, the benefit of the acceleration step is more evident for the largest-sized problems.

Modifications for Quasi-Newton Method and Its Spectral Algorithm for Solving Unconstrained Optimization Problems

In this paper, two modifications for spectral quasi-Newton algorithm of type BFGS are imposed. In the first algorithm, named SQNEI, a certain spectral parameter is used in such a step for BFGS algorithm differs from other presented algorithms. The second algorithm, SQNEv-Iv, has both new parameter position and value suggestion. In SQNEI and SQNEv-Iv methods, the parameters are involved in a search direction after an approximated Hessian matrix is updated. It is provided that two methods are effective under some assumptions. Moreover, the sufficient descent property is proved as well as the global and superlinear convergence for SQNEv-Iv and SQNEI. Both of them are superior the standard BFGS (QNBFGS) and previous spectral quasi-Newton (SQNLC). However, SQNEv-Iv is outstanding SQNEI if it is convergent to the solution. This means that, two modified methods are in the race for the more efficiency method in terms less iteration numbers and consuming time in running CPU. Finally, numerical results are presented for the four algorithms by running list of test problems with inexact line search satisfying Armijo condition.

Distribution of Bound Conformations in Conformational Ensembles for X-ray Ligands Predicted by the ANI-2X Machine Learning Potential.

In this study, we systematically studied the energy distribution of bioactive conformations of small molecular ligands in their conformational ensembles using ANI-2X, a machine learning potential, in conjunction with one of our recently developed geometry optimization algorithms, known as a conjugate gradient with backtracking line search (CG-BS). We first evaluated the combination of these methods (ANI-2X/CG-BS) using two molecule sets. For the 231-molecule set, ab initio calculations were performed at both the ωB97X/6-31G(d) and B3LYP-D3BJ/DZVP levels for accuracy comparison, while for the 8,992-molecule set, ab initio calculations were carried out at the B3LYP-D3BJ/DZVP level. For each molecule in the two molecular sets, up to 10 conformations were generated, which diminish the influence of individual outliers on the performance evaluation. Encouraged by the performance of ANI-2x/CG-BS in these evaluations, we calculated the energy distributions using ANI-2x/CG-BS for more than 27,000 ligands in the protein data bank (PDB). Each ligand has at least one conformation bound to a biological molecule, and this ligand conformation is labeled as a bound conformation. Besides the bound conformations, up to 200 conformations were generated using OpenEye's Omega2 software (https://docs.eyesopen.com/applications/ omega/) for each conformation. We performed a statistical analysis of how the bound conformation energies are distributed in the ensembles for 17,197 PDB ligands that have their bound conformation energies within the energy ranges of the Omega2-generated conformation ensembles. We found that half of the ligands have their relative conformation energy lower than 2.91 kcal/mol for the bound conformations in comparison with the global conformations, and about 90% of the bound conformations are within 10 kcal/mol above the global conformation energies. This information is useful to guide the construction of libraries for shape-based virtual screening and to improve the docking algorithm to efficiently sample bound conformations.

Numerical algorithms and simulations of boundary dynamic control for optimal mixing in unsteady Stokes flows

This work develops an efficient and accurate optimization algorithm to study the optimal mixing problem driven by boundary control of unsteady Stokes flows, based on the theoretical foundation laid by Hu and Wu in a series of work. The scalar being mixed is purely advected by the flow and the control is a force exerted tangentially on the domain boundary through the Navier slip conditions. The control design has potential applications in many industrial processes such as rotating wall driven mixing, mircomixers with acoustic waves, and artificial cilia mixing.The numerical algorithms have high complexity, high accuracy demand, and high computing expense, due to the multiscale nature of the mixing problem and the optimization requirements. A crucial problem is the computation of the Gâteaux derivative of the cost functional. To this end, a hybrid approach based on variational formula and finite difference is built with high accuracy and efficiency to treat various types of control input functions. We have experimented with various optimization algorithms including the steepest descent algorithm, the conjugate gradient method and two line search options (backtracking and exact line search). We are able to identify and implement the best combinations.The numerical simulations show that the mixing efficacy is limited when only one single type of control is applied, but can be enhanced when more diverse control types and more time segmentation are utilized. The mix-norm in the optimal mixings decays exponentially. The numerical study in this work demonstrates that boundary control alone could be an effective strategy for mixing in incompressible flows.

A Fast and Simple Modification of Newton’s Method Avoiding Saddle Points

AbstractWe propose in this paper New Q-Newton’s method. The update rule is conceptually very simple, using the projections to the vector subspaces generated by eigenvectors of positive (correspondingly negative) eigenvalues of the Hessian. The main result of this paper roughly says that if a sequence $$\{x_n\}$$ { x n } constructed by the method from a random initial point $$x_0$$ x 0 converges, then the limit point is a critical point and not a saddle point, and the convergence rate is the same as that of Newton’s method. A subsequent work has recently been successful incorporating Backtracking line search to New Q-Newton’s method, thus resolving the global convergence issue observed for some (non-smooth) functions. An application to quickly find zeros of a univariate meromorphic function is discussed, accompanied with an illustration on basins of attraction.

Adaptive Hybrid Mixed Two-Point Step Size Gradient Algorithm for Solving Non-Linear Systems

In this paper, a two-point step-size gradient technique is proposed by which the approximate solutions of a non-linear system are found. The two-point step-size includes two types of parameters deterministic and random. A new adaptive backtracking line search is presented and combined with the two-point step-size gradient to make it globally convergent. The idea of the suggested method depends on imitating the forward difference method by using one point to estimate the values of the gradient vector per iteration where the number of the function evaluation is at most one for each iteration. The global convergence analysis of the proposed method is established under actual and limited conditions. The performance of the proposed method is examined by solving a set of non-linear systems containing high dimensions. The results of the proposed method is compared to the results of a derivative-free three-term conjugate gradient CG method that solves the same test problems. Fair, popular, and sensible evaluation criteria are used for comparisons. The numerical results show that the proposed method has merit and is competitive in all cases and superior in terms of efficiency, reliability, and effectiveness in finding the approximate solution of the non-linear systems.

A novel flow update policy in solving traffic assignment problems: Successive over relaxation iteration method

This paper presents a novel flow update policy, namely the successive over relaxation (SOR) iteration method, which can be implemented in traffic assignment algorithms. Most existing solution algorithms for the user equilibrium traffic assignment problem (UE-TAP) mainly use two flow update policies: Jacobi and Gauss-Seidel iteration methods. The proposed flow update policy SOR can be a more efficient replacement. Following the path-based gradient projection (GP) algorithm, we developed a new method GP-SOR for the UE-TAP. This study first provides the complete procedure of applying the GP-SOR algorithm to solve the UE-TAP. Subsequently, a few properties of the proposed method are rigorously proven. However, empirical tests of the GP-SOR algorithm demonstrate serious oscillations and poor convergence. To cope with this problem, the Armijo Rule is employed to determine the relaxation factor, which substantially improves the convergence of GP-SOR algorithm. The preliminary numerical examples show that the GP-SOR algorithm has speedier convergence compared with the known alternatives, which is reflected by the evident reduction of the computing time and the number of iterations.

Analysis of the convergence of Newton method by finite element simulation with vector hysteresis stop model

PurposeThe goal of this research is to investigate the convergence behavior of the Newton iteration, when solving the nonlinear problem with consideration of hysteresis effects. Incorporating the vector hysteresis model in the magnetic vector potential formulation has encountered difficulties. One of the reasons is that the Newton method is very sensitive regarding the starting point and states distinct requirements for the nonlinear function in terms of monotony and smoothness. The other reason is that the differential reluctivity tensor of the material model is discontinuous due to the properties of the stop operators. In this work, line search methods to overcome these difficulties are discussed.Design/methodology/approachTo stabilize the Newton iteration, line search methods are studied. The first method computes an error-oriented search direction. The second method is based on the Wolfe-Powell rule using the Armijo condition and curvature condition.FindingsIn this paper, the differentiation of the vector stop model, used to evaluate the Jacobian matrix, is studied. Different methods are applied for this nonlinear problem to ensure reliable and stable finite element simulations with consideration of vector hysteresis effects.Originality/valueIn this paper, two different line search Newton methods are applied to solve the magnetic field problems with consideration of vector hysteresis effects and ensure a stable convergence successfully. A comparison of these two methods in terms of robustness and efficiency is presented.

Effectiveness of grid and random approaches for a model parameter vector optimization

Selection of effective initial parameter vectors is important for mathematical models having parameter vectors and differential equations in many science and engineering problems. In this paper, we propose a new mathematical method for an inverse problem of parameter vector optimization. We analyze and compare the effectiveness of grid and random approaches in hyperbox in terms of nonlinear least squares error, maximum improvement factor and number of iterations for an inverse problem of parameter vector optimization in a mathematical model coming from asset flow theory. This analysis is valuable to understand the population dynamics of investors and machine learning applications. For this purpose, we use quasi-Newton (QN) method having the Broyden–Fletcher–Goldfarb–Shanno (BFGS) formula with backtracking line search algorithm to optimize the function F[K̃] for each selected event and initial parameter vector, where F[K̃] represents the sum of exponentially weighted squared differences between the proxy for actual market price values via simulation and the computed market price values. Moreover, we employ Monte Carlo simulations and obtain convergence diagrams. We find that the success of the grid approach is relatively better than that of the random approach based on our simulation data set in the unconstrained optimization problem.

Synthesis of Parameters of Proportionally-Integral and Proportionally-Integral-Differential Controllers for Stationary Linear Objects with Nonzero Initial Conditions

The problem of the synthesis of proportional-integrating (PI) and proportional-integral-differentiating (PID) controllers in a nonstandard formulation is considered. For a linear one-dimensional control object with nonzero initial conditions, it is required to find a controller that is optimal in the sense of a quadratic functional of the state of the object with a regularization additive in control. The synthesis procedure is a solution of the corresponding quadratic optimization problem using a method similar to the conjugate gradient method (the direction at each step is calculated by the conjugate gradient method, and the step length is calculated by the Armijo rule). Numerical examples illustrate the effectiveness of the proposed algorithm in the synthesis of controllers for models of control objects that are common in practice.

A new Newton method for convex optimization problems with singular Hessian matrices

<abstract><p>In this paper, we propose a new Newton method for minimizing convex optimization problems with singular Hessian matrices including the special case that the Hessian matrix of the objective function is singular at any iteration point. The new method we proposed has some updates in the regularized parameter and the search direction. The step size of our method can be obtained by using Armijo backtracking line search. We also prove that the new method has global convergence. Some numerical experimental results show that the new method performs well for solving convex optimization problems whose Hessian matrices of the objective functions are singular everywhere.</p></abstract>

A variable metric and nesterov extrapolated proximal DCA with backtracking for a composite DC program

In this paper, we consider a composite difference-of-convex (DC) program, whose objective function is the sum of a smooth convex function with Lipschitz continuous gradient, a proper closed and convex function, and a continuous concave function. This problem has many applications in machine learning and data science. The proximal DCA (pDCA), a special case of the classical difference-of-convex algorithm (DCA), as well as two Nesterov-type extrapolated DCA – ADCA (Phan et al. IJCAI:1369–1375, 2018) and pDCAe (Wen et al. Comput. Optim. Appl. 69:297–324, 2018) – can solve this problem. The algorithmic stepsizes of pDCA, pDCAe, and ADCA are fixed and determined by estimating a prior the smoothness parameter of the loss function. However, such an estimate may be hard to obtain or poor in some real-world applications. Motivated by this difficulty, we propose a variable metric and Nesterov extrapolated proximal DCA with backtracking (SPDCAe), which combines the backtracking line search procedure (not necessarily monotone) and the Nesterov's extrapolation for potential acceleration; moreover, the variable metric method is incorporated for better local approximation. Numerical simulations on sparse binary logistic regression and compressed sensing with Poisson noise demonstrate the effectiveness of our proposed method.

An approximate Newton-type proximal method using symmetric rank-one updating formula for minimizing the nonsmooth composite functions

Founded upon the scaled memoryless symmetric rank-one updating formula, we propose an approximation of the Newton-type proximal strategy for minimizing the nonsmooth composite functions. More exactly, to approximate the inverse Hessian of the smooth part of the objective function, a symmetric rank-one matrix is employed to straightly compute the search directions for a special category of well-known functions. Convergence of the given algorithm is argued with a nonmonotone backtracking line search adjusted for the corresponding nonsmooth model. Also, its practical advantages are computationally depicted in the two well-known real-world models.

Shape optimization for the strong routing of light in periodic diffraction gratings

In the quest for the development of faster and more reliable technologies, the ability to control the propagation, confinement, and emission of light has become crucial. The design of guide mode resonators and perfect absorbers has proven to be of fundamental importance. In this project, we consider the shape optimization of a periodic dielectric slab aiming at efficient directional routing of light to reproduce similar features of a guide mode resonator. For this, the design objective is to maximize the routing efficiency of an incoming wave. That is, the goal is to promote wave propagation along the periodic slab. A Helmholtz problem with a piecewise constant and periodic refractive index medium models the wave propagation, and an accurate Robin-to-Robin map models an exterior domain. We propose an optimal design strategy that consists of representing the dielectric interface by a finite Fourier formula and using its coefficients as the design variables. Moreover, we use a high order finite element (FE) discretization combined with a bilinear Transfinite Interpolation formula. This setting admits explicit differentiation with respect to the design variables, from where an exact discrete adjoint method computes the sensitivities. We show in detail how the sensitivities are obtained in the quasi-periodic discrete setting. The design strategy employs gradient-based numerical optimization, which consists of a BFGS quasi-Newton method with backtracking line search. As a test case example, we present results for the optimization of a so-called single port perfect absorber. We test our strategy for a variety of incoming wave angles and different polarizations. In all cases, we efficiently reach designs featuring high routing efficiencies that satisfy the required criteria.