Exact Line Search Research Articles

On the Dai–Liao conjugate gradient method for vector optimization

Several conjugate gradient (CG) parameters have led to promising methods for optimization problems. However, some parameters, such as the Dai-Liao (DL+) and modified Polak-Ribière-Polyak (EPRP) methods, have not yet been explored in the vector optimization setting. This paper addresses this gap by proposing the DL+ and EPRP+ methods to vector optimization. We start by developing a self-adjusting DL+ algorithm to find critical points of vector-valued functions within a closed, convex, pointed cone with a nonempty interior. The algorithm is designed to satisfy the sufficient descent condition (SDC) with a Wolfe line search; if this condition is not satisfied, the algorithm adjusts by redefining the DL+ to meet the SDC. We establish the global convergence of this algorithm under the assumption of SDC without requiring regular restarts or convexity assumption on the objective functions. Similarly, we developed the EPRP+ algorithm using the same approach as the DL+ algorithm. Furthermore, under exact line search, the DL+ and EPRP+ methods reduce to Hestenes-Stiefel (HS) and Polak-Ribière-Polyak (PRP) methods, respectively. We present numerical experiments on some selected test problems sourced from the multiobjective optimization literature that demonstrate the effectiveness and practical implementation of the proposed algorithms, highlighting their promising potential.

Another modified version of RMIL conjugate gradient method

Due to their simplicity and global convergence properties, the conjugate gradient (CG) methods are widely used for solving unconstrained optimization problems, especially those of large scale. To establish the global convergence and to obtain better numerical performance in practice, much effort has been devoted to develop new CG methods or even to modify well- known methods. In 2012, Rivaie et al., have proposed a new CG method, called RMIL which has good numerical results and globally convergent under the exact line search. However, in 2016, Dai has pointed out a mistake in the steps of the proof of global convergence of RMIL and hence to guarantee the global convergence he suggested a modified version of RMIL, called RMIL+. In this paper, we present another modified version of RMIL, which is globally convergent via the exact line search. Furthermore, to support the theoretical proof of the global convergence of the modified version in practical computation, a numerical experiment based on comparing it with RMIL, RMIL+, and CG-DESCENT was done.

A modified type of Fletcher-Reeves conjugate gradient method with its global convergence

<p><span>The conjugate gradient methods are one of the most important techniques used to address problems involving minimization or maximization, especially nonlinear optimization problems with no constraints at all. That is because of their simplicity and low memory needed. They can be applied in many areas, such as economics, engineering, neural networks, image restoration, machine learning, and deep learning. The convergence of Fletcher-Reeves (FR) conjugate gradient method has been established under both exact and strong Wolfe line searches. However, it is performance in practice is poor. In this paper, to get good numerical performance from the FR method, a little modification is done. The global convergence of the modified version has been established for general nonlinear functions. Preliminary numerical results show that the modified method is very efficient in terms of number of iterations and CPU time.</span></p>

Linearly convergent adjoint free solution of least squares problems by random descent

We consider the problem of solving linear least squares problems in a framework where only evaluations of the linear map are possible. We derive randomized methods that do not need any other matrix operations than forward evaluations, especially no evaluation of the adjoint map is needed. Our method is motivated by the simple observation that one can get an unbiased estimate of the application of the adjoint. We show convergence of the method and then derive a more efficient method that uses an exact linesearch. This method, called random descent, resembles known methods in other context and has the randomized coordinate descent method as special case. We provide convergence analysis of the random descent method emphasizing the dependence on the underlying distribution of the random vectors. Furthermore we investigate the applicability of the method in the context of ill-posed inverse problems and show that the method can have beneficial properties when the unknown solution is rough. We illustrate the theoretical findings in numerical examples. One particular result is that the random descent method actually outperforms established transposed-free methods (TFQMR and CGS) in examples.

Numerical algorithms and simulations of boundary dynamic control for optimal mixing in unsteady Stokes flows

This work develops an efficient and accurate optimization algorithm to study the optimal mixing problem driven by boundary control of unsteady Stokes flows, based on the theoretical foundation laid by Hu and Wu in a series of work. The scalar being mixed is purely advected by the flow and the control is a force exerted tangentially on the domain boundary through the Navier slip conditions. The control design has potential applications in many industrial processes such as rotating wall driven mixing, mircomixers with acoustic waves, and artificial cilia mixing.The numerical algorithms have high complexity, high accuracy demand, and high computing expense, due to the multiscale nature of the mixing problem and the optimization requirements. A crucial problem is the computation of the Gâteaux derivative of the cost functional. To this end, a hybrid approach based on variational formula and finite difference is built with high accuracy and efficiency to treat various types of control input functions. We have experimented with various optimization algorithms including the steepest descent algorithm, the conjugate gradient method and two line search options (backtracking and exact line search). We are able to identify and implement the best combinations.The numerical simulations show that the mixing efficacy is limited when only one single type of control is applied, but can be enhanced when more diverse control types and more time segmentation are utilized. The mix-norm in the optimal mixings decays exponentially. The numerical study in this work demonstrates that boundary control alone could be an effective strategy for mixing in incompressible flows.

Two global quasi-Newton algorithms for solving matrix polynomial equations

In this article, we globalize the quasi-Newton algorithm proposed in Macías et al. (Appl Math Comput 441:127678, 2023) introducing an exact line search, we propose a polynomial approximation to the merit function and we deduce a sufficient condition for its minimization interval. We use the exact merit function and its approximation to propose two global quasi-Newton algorithms for solving matrix polynomial equations. For each algorithm, we prove that the exact line search does not affect the convergence of quasi-Newton method. In addition, we present comparative numerical tests of the algorithmic proposals in which we also compare with global Newton’s method.

The Convergence Properties of Conjugate Gradient Method Using AMRI Parameter with Exact Line Search

The conjugate gradient method is a simple way to get a solution of optimization problems without constraints. In this work, we offer a conjugate gradient method algorithm using AMRI parameter where the step of length is decided by an exact line search. The proposed algorithm accomplishes the condition of sufficiently descent and globally convergence with several assumptions. Computation results prove that the modified method is effective.

The global convergence of spectral RMIL conjugate gradient method for unconstrained optimization with applications to robotic model and image recovery.

In 2012, Rivaie et al. introduced RMIL conjugate gradient (CG) method which is globally convergent under the exact line search. Later, Dai (2016) pointed out abnormality in the convergence result and thus, imposed certain restricted RMIL CG parameter as a remedy. In this paper, we suggest an efficient RMIL spectral CG method. The remarkable feature of this method is that, the convergence result is free from additional condition usually imposed on RMIL. Subsequently, the search direction is sufficiently descent independent of any line search technique. Thus, numerical experiments on some set of benchmark problems indicate that the method is promising and efficient. Furthermore, the efficiency of the proposed method is demonstrated on applications arising from arm robotic model and image restoration problems.

The Convergent Properties of a New Parameter for Unconstrained Optimization

Because of its simplicity, low memory requirement, low computational cost, and global convergence properties, the Conjugate Gradient (CG) method is the most popular iterative mathematical technique for optimizing both linear and nonlinear systems. Some classical CG methods, however, have drawbacks such as poor global convergence and numerical performance in terms of iterations and function evaluations. To address these shortcomings, researchers proposed new CG parameter variants with efficient numerical results and good convergence properties. We present a new conjugate gradient formula based on the memoryless self-scale DFP quasi-Newton (QN) method in this paper. The proposed new formula fulfills the sufficient descent property and the global convergent condition with any proposed line research. When the exact line search is used, the proposed formula is reduced to the classical HS formula. Finally, we conclude that our proposed method is effective.

Triangularized Orthogonalization-Free Method for Solving Extreme Eigenvalue Problems

A novel orthogonalization-free method together with two specific algorithms is proposed to address extreme eigenvalue problems. On top of gradient-based algorithms, the proposed algorithms modify the multicolumn gradient such that earlier columns are decoupled from later ones. Locally, both algorithms converge linearly with convergence rates depending on eigengaps. Momentum acceleration, exact linesearch, and column locking are incorporated to accelerate algorithms and reduce their computational costs. We demonstrate the efficiency of both algorithms on random matrices with different spectrum distributions and matrices from computational chemistry.

A new direction search of hybrid quasi-Newton

A new hybrid quasi-Newton search direction ( HQN<sup>EI</sup> ) is proposed. It uses the update formula of Broyden–Fletcher–Goldfarb–Shanno (BFGS) with a certain conjugate gradient (CG) parameter by a nested direction. The global convergence analysis and superlinear rate, addtionaly with sufficient descent are proved using exact line search. Finally, the computation comparisons are made with original hybrid parents; BFGS and CG, through the efficiency in terms of iteration numbers and CPU-running time showing the superior of HQN<sup>EI</sup>. Therefore, the results marked preference of HQN<sup>EI</sup> from other two producer algorithms.

New search direction of steepest descent method for solving large linear systems

The steepest descent (SD) method is well-known as the simplest method in optimization. In this paper, we propose a new SD search direction for solving system of linear equations Ax = b. We also prove that the proposed SD method with exact line search satisfies descent condition and possesses global convergence properties. This proposed method is motivated by previous work on the SD method by Zubai’ah-Mustafa-Rivaie-Ismail (ZMRI)[2]. Numerical comparisons with a classical SD algorithm and ZMRI algorithm show that this algorithm is very effective depending on the number of iterations (NOI) and CPU time.

Estimating the seven transformational parameters between two geodetic datums using the steepest descent algorithm of machine learning

This study evaluates the steepest descent algorithm as a tool for root mean square (RMS) error optimization in geodetic reference systems to improve the integrity of transformation. With an initial RMS error estimate of 0.01830m, the negative gradient direction was applied through the steepest optimization leading to a final RMS error estimate of 0.00051m. Using the exact line search mode with a one-point step size of 0.1, we achieved the minimum values in less than sixty iterations, regardless of the slow convergence rate of the steepest descent algorithm.

Analysis of the Frank–Wolfe method for convex composite optimization involving a logarithmically-homogeneous barrier

We present and analyze a new generalized Frank–Wolfe method for the composite optimization problem (P): {min }_{xin {mathbb {R}}^n} ; f(mathsf {A} x) + h(x), where f is a theta -logarithmically-homogeneous self-concordant barrier, mathsf {A} is a linear operator and the function h has a bounded domain but is possibly non-smooth. We show that our generalized Frank–Wolfe method requires O((delta _0 + theta + R_h)ln (delta _0) + (theta + R_h)^2/varepsilon ) iterations to produce an varepsilon -approximate solution, where delta _0 denotes the initial optimality gap and R_h is the variation of h on its domain. This result establishes certain intrinsic connections between theta -logarithmically homogeneous barriers and the Frank–Wolfe method. When specialized to the D-optimal design problem, we essentially recover the complexity obtained by Khachiyan (Math Oper Res 21 (2): 307–320, 1996) using the Frank–Wolfe method with exact line-search. We also study the (Fenchel) dual problem of (P), and we show that our new method is equivalent to an adaptive-step-size mirror descent method applied to the dual problem. This enables us to provide iteration complexity bounds for the mirror descent method despite the fact that the dual objective function is non-Lipschitz and has unbounded domain. In addition, we present computational experiments that point to the potential usefulness of our generalized Frank–Wolfe method on Poisson image de-blurring problems with TV regularization, and on simulated PET problem instances.

Research Spotlights

This issue features three Research Spotlight articles. The first of these is entitled “Variance and Covariance of Distributions on Graphs" and is coauthored by Karel Devriendt, Samuel Martin-Gutierrez, and Renaud Lambiotte. Given a distribution on the graph (that is, a function $p$ from the set of nodes to $[0,1]$ such that the sum of all values is one) the key ingredient in the authors' definition of the variance and covariance is a notion of distance between the nodes of a graph. Although their definitions of (co)variance are valid for different choices of distance, the authors focus on a metric called the effective resistance. The effective resistance resembles geodesic distance in that it reflects the length of the paths between a pair of nodes but differs in that it takes into account all paths, and their lengths, between a node pair. Conveniently, the resulting variance and covariance values can be calculated via evaluation of a quadratic product and matrix trace. In support of the newly introduced measures, the balance of the paper is devoted to the application of the new definitions in practice and to the explanation of the conceptual correspondence between their (co)variance measures and some known graph characteristics. Citing the benefits of the framework induced by their (co)variance measures, the authors leave the reader with suggestions for future application in fields such as neuroscience, economics, and social networks. Competitive tournaments are an integral part of daily life, from schoolyard games to professional sports, to politics and biology. Authors Alexander Strang, Karen C. Abbott, and Peter J. Thomas tackle the difficult problem of quantifying competition in tournament modeling. Their article, “The Network HHD: Quantifying Cyclic Competition in Trait-Performance Models of Tournaments,” outlines how to adapt the discrete Helmholtz--Hodge decomposition (HHD), first introduced in the literature as a method for ranking objects from incomplete and imbalanced data, to study competitive tournaments characterized by intransitivity, the presence of which provides a challenge in ranking. Specifically, an intransitive tournament is one in which there is no clear global ranking of all competitors, and it corresponds to a cycle in the model. The authors show that the HHD arises in the context of representing a generic tournament as a weighted sum of so-called perfectly transitive and perfectly cyclic components. Since a trait-performance model assumes that “the probability that one competitor beats another is a function of their traits” the goal becomes that of identifying which traits and performance functions influence the weights in the HHD. This is addressed with the theoretical results in section 4. Schematics and graphs complement the discussion. Using the code made available by the authors, the interested reader may wish to try an analysis of a tournament for themselves. The third article, “Bilinear Optimal Control of an Advection-Reaction-Diffusion System," authored by Roland Glowinski, Yongcun Song, Xiaoming Yuan, and Hangrui Yue, offers both theoretical and computational insights to solving the mathematical problem specified in the title. Readers may appreciate the space devoted to the motivation and introduction to the problem as well as to the description of the associated difficulties both in deriving existence results and in numerical computation of solutions. Following a few preliminaries, the authors derive a proof of existence of optimal controls for the problem, labeled in the introduction as BCP, without the special case assumptions used elsewhere in the literature. The balance of the article is devoted to issues associated with the numerical implementation. Although their proposed nested conjugate gradient method is straightforward to present on paper, several obstacles arise in the implementation, and the authors tackle each of these in turn. For computational efficiency, for example, an inexact line search is proposed to replace the computationally intractable exact line search; preconditioned CG is employed for two internal subproblems that arise. The choice of the time and space discretizations is explained as well. There are many technical details to consume, but a key point for the readers is well summarized by the authors just before the detailed experiments, namely, “despite its apparent complexity, the nested CG method” given here “is easy to implement" with aspects that are amenable to parallelizability. Misha E. Kilmer Section Editor Misha.Kilmer@tufts.edu

A new modified HS algorithm with strong Powell-Wolfe line search for unconstrained optimization

Optimization is now considered a branch of computational science. This ethos seeks to answer the question «what is best?» by looking at problems where the quality of any answer can be expressed numerically. One of the most well-known methods for solving nonlinear, unrestricted optimization problems is the conjugate gradient (CG) method. The Hestenes and Stiefel (HS-CG) formula is one of the century’s oldest and most effective formulas. When using an exact line search, the HS method achieves global convergence; however, this is not guaranteed when using an inexact line search (ILS). Furthermore, the HS method does not always satisfy the descent property. The goal of this work is to create a new (modified) formula by reformulating the classic parameter HS-CG and adding a new term to the classic HS-CG formula. It is critical that the proposed method generates sufficient descent property (SDP) search direction with Wolfe-Powell line (sWPLS) search at every iteration, and that global convergence property (GCP) for general non-convex functions can be guaranteed. Using the inexact sWPLS, the modified HS-CG (mHS-CG) method has SDP property regardless of line search type and guarantees GCP. When using an sWPLS, the modified formula has the advantage of keeping the modified scalar non-negative sWPLS. This paper is significant in that it quantifies how much better the new modification of the HS performance is when compared to standard HS methods. As a result, numerical experiments between the mHSCG method using the sWPL search and the standard HS optimization problem show that the CG method with the mHSCG conjugate parameter is more robust and effective than the CG method without the mHSCG parameter

Coordinate descent method for solving constraint optimization problems

The study generally aims to find line search technique with exact line search without relying on update or invertible Hessian matrix at every iteration, and theoretical improvement for if H be a symmetric matrix then has n mutually orthogonal eigenvectors.

QPALM: a proximal augmented lagrangian method for nonconvex quadratic programs

We propose QPALM, a nonconvex quadratic programming (QP) solver based on the proximal augmented Lagrangian method. This method solves a sequence of inner subproblems which can be enforced to be strongly convex and which therefore admit a unique solution. The resulting steps are shown to be equivalent to inexact proximal point iterations on the extended-real-valued cost function, which allows for a fairly simple analysis where convergence to a stationary point at an \(R\)-linear rate is shown. The QPALM algorithm solves the subproblems iteratively using semismooth Newton directions and an exact linesearch. The former can be computed efficiently in most iterations by making use of suitable factorization update routines, while the latter requires the zero of a monotone, one-dimensional, piecewise affine function. QPALM is implemented in open-source C code, with tailored linear algebra routines for the factorization in a self-written package LADEL. The resulting implementation is shown to be extremely robust in numerical simulations, solving all of the Maros-Meszaros problems and finding a stationary point for most of the nonconvex QPs in the Cutest test set. Furthermore, it is shown to be competitive against state-of-the-art convex QP solvers in typical QPs arising from application domains such as portfolio optimization and model predictive control. As such, QPALM strikes a unique balance between solving both easy and hard problems efficiently.

On a primal-dual Newton proximal method for convex quadratic programs

This paper introduces QPDO, a primal-dual method for convex quadratic programs which builds upon and weaves together the proximal point algorithm and a damped semismooth Newton method. The outer proximal regularization yields a numerically stable method, and we interpret the proximal operator as the unconstrained minimization of the primal-dual proximal augmented Lagrangian function. This allows the inner Newton scheme to exploit sparse symmetric linear solvers and multi-rank factorization updates. Moreover, the linear systems are always solvable independently from the problem data and exact linesearch can be performed. The proposed method can handle degenerate problems, provides a mechanism for infeasibility detection, and can exploit warm starting, while requiring only convexity. We present details of our open-source C implementation and report on numerical results against state-of-the-art solvers. QPDO proves to be a simple, robust, and efficient numerical method for convex quadratic programming.

A Sufficient Condition for the Global Convergence of Conjugate Gradient Methods for Solving Unconstrained Optimisation Problems

Due to their remarkable convergence properties and performance in practice, conjugate gradient (CG) methods are widely used for solving unconstrained optimisation problems, especially those of large scale. From the 1950s until now, many studies have been carried out to propose new ones to improve existing CG methods. In this paper, we present a condition that guarantees the global convergence of CG methods when they are applied under the exact line search. At the same time, based on this condition, we did a minor modification on the CG methods of Polak-Rebiere-Polyak (PRP) and of Hestenes-Stiefel (HS) to propose new modified methods. Furthermore, to support the theoretical proof of the global convergence of the modified methods in practical computation, a numerical experiment based on comparing the proposed methods with other well-known CG methods was done. It has been found that the new modified methods have the fewest number of iterations and require the shortest time for solving the problems. In addition, they have the highest percentage of the test problems that solved successfully. Hence, we conclude that they can be used successfully for solving unconstrained optimisation problems. KEYWORDS Unconstrained optimisation problems; conjugate gradient methods; exact line search; global convergence