Methods For Unconstrained Optimization Research Articles

Superfast Second-Order Methods for Unconstrained Convex Optimization

In this paper, we present new second-order methods with convergence rate Oleft( k^{-4}right) , where k is the iteration counter. This is faster than the existing lower bound for this type of schemes (Agarwal and Hazan in Proceedings of the 31st conference on learning theory, PMLR, pp. 774–792, 2018; Arjevani and Shiff in Math Program 178(1–2):327–360, 2019), which is Oleft( k^{-7/2} right) . Our progress can be explained by a finer specification of the problem class. The main idea of this approach consists in implementation of the third-order scheme from Nesterov (Math Program 186:157–183, 2021) using the second-order oracle. At each iteration of our method, we solve a nontrivial auxiliary problem by a linearly convergent scheme based on the relative non-degeneracy condition (Bauschke et al. in Math Oper Res 42:330–348, 2016; Lu et al. in SIOPT 28(1):333–354, 2018). During this process, the Hessian of the objective function is computed once, and the gradient is computed Oleft( ln {1 over epsilon }right) times, where epsilon is the desired accuracy of the solution for our problem.

A note on memory-less SR1 and memory-less BFGS methods for large-scale unconstrained optimization

A note on memory-less SR1 and memory-less BFGS methods for large-scale unconstrained optimization

Good Characteristics of The New Spectral Conjugate Gradient Method for Unconstrained Optimization

The spectral conjugate gradient (SCG) method is an effective method to solve large-scale nonlinear unconstrained optimization problems. In this work, we propose a new SCG method in which performance is numerically analyzed. We established the descent property and global convergence conditions based on assumptions through the strongWolfe-Powell line search. Numerical results were performed using benchmark functions widely used in many conventional functions to evaluate the efficiency of the proposed method.Subject Classification: 90C30, 90C06, 65K05, 65K10.

A new hybrid conjugate gradient method of unconstrained optimization methods

In this paper, we present a new hybrid method to solve a nonlinear unconstrained optimization problem by using conjugate gradient, which is a convex combination of Liu–Storey (LS) conjugate gradient method and Hager–Zhang (HZ) conjugate gradient method. This method possesses the sufficient descent property with Strong Wolfe line search and the global convergence with the strong Wolfe line search. In the end of this paper, we illustrate our method by giving some numerical examples.

COMPACT: Concurrent or Ordered Matrix-Based Packing Arrangement Computation Technique

Despite their versatility in treating irregular geometries, the raster methods have received limited attention in solving packing problems involving rotatable objects. In addition, raster approximation allows the use of unique performance metrics and indirect consideration of constraints, which have not been exploited in the literature. This study presents the Concurrent or Ordered Matrix-based Packing Arrangement Computation Technique (COMPACT). The method allows the objects to be rotated by arbitrary angles, unlike the right-angled rotation restrictions imposed in many existing packing optimization studies based on raster methods. The raster approximations are obtained through loop-free operations that improve efficiency. Additionally, a novel performance metric is introduced, which favors efficient filling of the available space by maximizing the overall contact within the domain. Moreover, the objective functions are exploited to discard the overlap and overflow constraints and enable the use of unconstrained optimization methods. The results of the case studies demonstrate the effectiveness of the proposed technique.

A New Spectral Conjugate Gradient Method for Nonlinear Unconstrained Optimization

Theconjugate gradient method is widely used to solve large scale unconstrainedoptimization problems. However, the rate of convergence conjugate gradient method is linearunless it restarted. In this study, we present a new spectral conjugategradient modification formula with restart property obtains the globalconvergence and descent properties.In addition, we proposed a new restart condition for Fletcher-Reeves conjugate gradient formula. The numerical resultsdemonstrated that the modified Fletcher-Reeves parameter and the new CG formulawith their restart conditions are more efficient and robustness than otherconventional methods.

Two-versions of descent conjugate gradient methods for large-scale unconstrained optimization

<p>The conjugate gradient methods are noted to be exceedingly valuable for solving large-scale unconstrained optimization problems since it needn't the storage of matrices. Mostly the parameter conjugate is the focus for conjugate gradient methods. The current paper proposes new methods of parameter of conjugate gradient type to solve problems of large-scale unconstrained optimization. A Hessian approximation in a diagonal matrix form on the basis of second and third-order Taylor series expansion was employed in this study. The sufficient descent property for the proposed algorithm are proved. The new method was converged globally. This new algorithm is found to be competitive to the algorithm of fletcher-reeves (FR) in a number of numerical experiments.</p>

A Linear Hybridization of Dai-Yuan and Hestenes-Stiefel Conjugate Gradient Method for Unconstrained Optimization

A Linear Hybridization of Dai-Yuan and Hestenes-Stiefel Conjugate Gradient Method for Unconstrained Optimization

A variant of Dai-Yuan conjugate gradient method for unconstrained optimization and its application in portfolio selection

The quasi-Newton (QN) method are among the efficient variants of conjugate gradient (CG) method for solving unconstrained optimization problems. The QN method utilizes the gradients of the function while ignoring the available value information at every iteration. In this paper, we extended the Dai-Yuan [39] coefficient in designing a new CG method for large-scale unconstrained optimization problems. An interesting feature of our method is that its algorithm not only uses the available gradient value, but also consider the function value information. The global convergence of the proposed method was established under some suitable Wolfe conditions. Extensive numerical computation have been carried out which show that the average performance of this new algorithm is efficient and promising. In addition, the proposed method was extended to solve practical application problem of portfolio selection.

Adaptive Gradient Methods for Constrained Convex Optimization and Variational Inequalities

We provide new adaptive first-order methods for constrained convex optimization. Our main algorithms AdaACSA and AdaAGD+ are accelerated methods, which are universal in the sense that they achieve nearly-optimal convergence rates for both smooth and non-smooth functions, even when they only have access to stochastic gradients. In addition, they do not require any prior knowledge on how the objective function is parametrized, since they automatically adjust their per-coordinate learning rate. These can be seen as truly accelerated Adagrad methods for constrained optimization. We complement them with a simpler algorithm AdaGrad+ which enjoys the same features, and achieves the standard non-accelerated convergence rate. We also present a set of new results involving adaptive methods for unconstrained optimization and variational inequalities arising from monotone operators.

Generalized Fibonacci Search Method in One-Dimensional Unconstrained Non-Linear Optimization

In this paper, we develop a generalized Fibonacci search method for one-dimensional unconstrained non-linear optimization of unimodal functions. This method uses the idea of the “ratio length of 1” from the golden section search. Our method takes successive lower Fibonacci numbers as the initial ratio and does not specify beforehand, the number of iterations to be used. We evaluated the method using Microsoft Excel with nine one-dimensional benchmark functions. We found that our generalized Fibonacci search method out-performed the golden section and other Fibonacci-type search methods such as the Fibonacci, Lucas and Pell approaches.

A new modification of the quasi-newton method for unconstrained optimization

<p class="MsoNormal" style="text-align: justify;"><span>In this work we propose and analyze a hybrid conjugate gradient (CG) method in which the parameter <!--[if gte mso 9]><xml>
 <o:OLEObject Type="Embed" ProgID="Equation.3" ShapeID="_x0000_i1025"
 DrawAspect="Content" ObjectID="_1674222415">
 </o:OLEObject>
 </xml><![endif]-->is computed as a linear combination between Hager-Zhang [HZ] and Dai-Liao [DL] parameters. We use this proposed method to modify BFGS method and to prove the positive definiteness and QN-conditions of the matrix. Theoretical trils confirm that the new search directions aredescent directions under some conditions, as well as, the new search directions areglobally convergent using strong Wolfe conditions. The numerical experiments show that the proposed method is promising and outperforms alternative similar CG-methods using Dolan-Mor'e performance profile. </span><br /><br /></p>

Stochastic analysis of an adaptive cubic regularization method under inexact gradient evaluations and dynamic Hessian accuracy

We here adapt an extended version of the adaptive cubic regularization method with dynamic inexact Hessian information for nonconvex optimization in Bellavia et al. [Adaptive cubic regularization methods with dynamic inexact hessian information and applications to finite-sum minimization. IMA Journal of Numerical Analysis. 2021;41(1):764–799] to the stochastic optimization setting. While exact function evaluations are still considered, this novel variant inherits the innovative use of adaptive accuracy requirements for Hessian approximations introduced in the just quoted paper and additionally employs inexact computations of the gradient. Without restrictions on the variance of the errors, we assume that these approximations are available within a sufficiently large, but fixed, probability and we extend, in the spirit of Cartis and Scheinberg [Global convergence rate analysis of unconstrained optimization methods based on probabilistic models. Math Program Ser A. 2018;159(2):337–375], the deterministic analysis of the framework to its stochastic counterpart, showing that the expected number of iterations to reach a first-order stationary point matches the well-known worst-case optimal complexity. This is, in fact, still given by , with respect to the first-order ϵ tolerance. Finally, numerical tests on nonconvex finite-sum minimization confirm that using inexact first- and second-order derivatives can be beneficial in terms of the computational savings.

A Modified Hybrid Conjugate Gradient Method for Unconstrained Optimization

The nonlinear conjugate gradient algorithms are a very effective way in solving large-scale unconstrained optimization problems. Based on some famous previous conjugate gradient methods, a modified hybrid conjugate gradient method was proposed. The proposed method can generate decent directions at every iteration independent of any line search. Under the Wolfe line search, the proposed method possesses global convergence. Numerical results show that the modified method is efficient and robust.

Sparse signal reconstruction in compressive sensing via derivative-free iterative method

Finding sparse solutions to under-determined or ill-conditioned equations is a problem that usually arise in compressive sensing. In this article, a derivative-free iterative method is presented for recovering sparse signal in compressive sensing by approximating the solution to a convex constrained nonlinear equation. The proposed method is derived from the modified Polak-Ribiere-Polyak conjugate gradient method for unconstrained optimization. The global convergence is established under mild assumptions. Preliminary numerical results in recovering sparse signal are given to show that the proposed method is efficient.

Derivative-free SMR conjugate gradient method for constraint nonlinear equations

Based on the SMR conjugate gradient method for unconstrained optimization proposed by Mohamed et al. [N. S. Mohamed, M. Mamat, M. Rivaie, S. M. Shaharuddin, Indones. J. Electr. Eng. Comput. Sci., \(\bf 11\) (2018), 1188--1193] and the Solodov and Svaiter projection technique, we propose a derivative-free SMR method for solving nonlinear equations with convex constraints. The proposed method can be viewed as an extension of the SMR method for solving unconstrained optimization. The proposed method can be used to solve large-scale nonlinear equations with convex constraints because of derivative-free and low storage. Under the assumption that the underlying mapping is Lipschitz continuous and satisfies a weaker monotonicity assumption, we prove its global convergence. Preliminary numerical results show that the proposed method is promising.

Fast convex optimization via inertial dynamics combining viscous and Hessian-driven damping with time rescaling

<p style='text-indent:20px;'>In a Hilbert setting, we develop fast methods for convex unconstrained optimization. We rely on the asymptotic behavior of an inertial system combining geometric damping with temporal scaling. The convex function to minimize enters the dynamic via its gradient. The dynamic includes three coefficients varying with time, one is a viscous damping coefficient, the second is attached to the Hessian-driven damping, the third is a time scaling coefficient. We study the convergence rate of the values under general conditions involving the damping and the time scale coefficients. The obtained results are based on a new Lyapunov analysis and they encompass known results on the subject. We pay particular attention to the case of an asymptotically vanishing viscous damping, which is directly related to the accelerated gradient method of Nesterov. The Hessian-driven damping significantly reduces the oscillatory aspects. We obtain an exponential rate of convergence of values without assuming the strong convexity of the objective function. The temporal discretization of these dynamics opens the gate to a large class of inertial optimization algorithms.</p>

A Spectral RMIL+ Conjugate Gradient Method for Unconstrained Optimization With Applications in Portfolio Selection and Motion Control

The Spectral conjugate gradient (SCG) methods are among the efficient variants of CG algorithms which are obtained by combining the spectral gradient parameter and CG parameter. The success of SCG methods relies on effective choices of the step-size $\alpha _{k}$ and the search direction $d_{k}$ . This paper presents an SCG method for unconstrained optimization models. The search directions generated by the new method possess sufficient descent property without the restart condition and independent of the line search procedure used. The global convergence of the new method is proved under the weak Wolfe line search. Preliminary numerical results are presented which show that the method is efficient and promising, particularly for large-scale problems. Also, the method was applied to solve the robotic motion control problem and portfolio selection problem.

Comparative study of reduced-order electrochemical models of the lithium-ion battery

A reliable and efficient lithium-ion battery model is the basis of state estimation and fault diagnosis in the battery management system (BMS). The pseudo-two-dimensional (P2D) mechanism model represented by using partial differential equations has many parameters and high accuracy, but its calculation is time-consuming. The model order should be reduced for the on-board BMS. Here in this work, by using the same model parameters, the P2D model and the order-reduced models including the single particle model (SPM) and the lumped particle model (LPM) are built and their voltage accuracies and operation time are compared with each other. Based on the porous electrode model, the concentrated solution theory, and the assumption of uniform current of the battery, the current density is redistributed by the volume ratio of the solid phase to the liquid phase of the electrode. Furthermore, the overpotential and ohmic resistance caused by the lithium-ion concentration distribution in the liquid phase is deduced to offset the battery voltage bias under a large discharge rate. An unconstrained convex optimization method is established to optimize the concentration difference overpotential of the liquid phase of the P2D model battery, the optimized results of which are used to compensate for the LPM voltage.Under the conditions of the constant current discharge (CCD) of 0.1C — 4C at the ambient temperature, pulse discharge and dynamic stress test (DST), the electrical performance and the operation time of both the LPM and the SPM are compared to those of the P2D model. Some results are validated and given by the model simulation. Firstly, the accuracy of LPM is over 30% higher than those of the SPM at the CCD rates of 0.1 C, 0.5 C and 1 C, and about 30% worse than those of the SPM at the CCD rates of 2 C, 3 C, and 4 C. Secondly, the optimized LPM can produce a voltage in good approximation to the voltage of the P2D model with an absolute relative error of the model voltage below 1.5% whether it is caused by the CCD or the DST. Thirdly, the optimized LPM can run efficiently and the needed calculation time cuts down by 85% and 65% for the P2D and the SPM, respectively. For the real-time applications of lithium-ion batteries, the proposed compensation method by adding the overpotential in the liquid phase can make the LPM produce reliable voltage with shorter operation time than either the SPM or the P2D model.

Sparse signal reconstruction in compressive sensing via derivative-free iterative method

Finding sparse solutions to under-determined or ill-conditioned equations is a problem that usually arise in compressive sensing. In this article, a derivative-free iterative method is presented for recovering sparse signal in compressive sensing by approximating the solution to a convex constrained nonlinear equation. The proposed method is derived from the modified Polak-Ribiere-Polyak conjugate gradient method for unconstrained optimization. The global convergence is established under mild assumptions. Preliminary numerical results in recovering sparse signal are given to show that the proposed method is efficient.