quasi-Newton Optimization Method Research Articles

A new modification of the quasi-newton method for unconstrained optimization

<p class="MsoNormal" style="text-align: justify;"><span>In this work we propose and analyze a hybrid conjugate gradient (CG) method in which the parameter <!--[if gte mso 9]><xml>
 <o:OLEObject Type="Embed" ProgID="Equation.3" ShapeID="_x0000_i1025"
 DrawAspect="Content" ObjectID="_1674222415">
 </o:OLEObject>
 </xml><![endif]-->is computed as a linear combination between Hager-Zhang [HZ] and Dai-Liao [DL] parameters. We use this proposed method to modify BFGS method and to prove the positive definiteness and QN-conditions of the matrix. Theoretical trils confirm that the new search directions aredescent directions under some conditions, as well as, the new search directions areglobally convergent using strong Wolfe conditions. The numerical experiments show that the proposed method is promising and outperforms alternative similar CG-methods using Dolan-Mor'e performance profile. </span><br /><br /></p>

Rates of superlinear convergence for classical quasi-Newton methods

We study the local convergence of classical quasi-Newton methods for nonlinear optimization. Although it was well established a long time ago that asymptotically these methods converge superlinearly, the corresponding rates of convergence still remain unknown. In this paper, we address this problem. We obtain first explicit non-asymptotic rates of superlinear convergence for the standard quasi-Newton methods, which are based on the updating formulas from the convex Broyden class. In particular, for the well-known DFP and BFGS methods, we obtain the rates of the form (frac{n L^2}{mu ^2 k})^{k/2} and (frac{n L}{mu k})^{k/2} respectively, where k is the iteration counter, n is the dimension of the problem, mu is the strong convexity parameter, and L is the Lipschitz constant of the gradient.

The random deterioration rate model with measurement error based on the inverse Gaussian distribution

In this paper, we introduce the random deterioration rate model with measurement error in order to incorporate the variability among different components. The motivation behind the random variable model is to capture the randomness in the individual differences across the population. This model incorporates only sample uncertainty of the degradation, and no temporal variability is included. The measurement error models appear to overcome this problem. The random rate analysis is based on repeated measurements of failure sizes generated by a degradation process over time in a components population. Some characteristics of the random deterioration rate model based on the inverse Gaussian distribution and subject to measurement error, are examined. We carry out simulation studies to (i) assess the performance of the maximum likelihood estimates obtained through the Gaussian quadrature along with Quasi-Newton optimization method; and (ii) examine the effects of model misspecification on the model selection criteria’s performance, as well as on the lifetime prediction’s accuracy and precision. The potentiality of the proposed model is illustrated through two real data sets.

A new Secant-like quasi-Newton method for unconstrained optimization

A new Secant-like quasi-Newton method for unconstrained optimization

Strong Metric (Sub)regularity of Karush–Kuhn–Tucker Mappings for Piecewise Linear-Quadratic Convex-Composite Optimization and the Quadratic Convergence of Newton’s Method

This work concerns the local convergence theory of Newton and quasi-Newton methods for convex-composite optimization: where one minimizes an objective that can be written as the composition of a convex function with one that is continuiously differentiable. We focus on the case in which the convex function is a potentially infinite-valued piecewise linear-quadratic function. Such problems include nonlinear programming, mini-max optimization, and estimation of nonlinear dynamics with non-Gaussian noise as well as many modern approaches to large-scale data analysis and machine learning. Our approach embeds the optimality conditions for convex-composite optimization problems into a generalized equation. We establish conditions for strong metric subregularity and strong metric regularity of the corresponding set-valued mappings. This allows us to extend classical convergence of Newton and quasi-Newton methods to the broader class of nonfinite valued piecewise linear-quadratic convex-composite optimization problems. In particular, we establish local quadratic convergence of the Newton method under conditions that parallel those in nonlinear programming.

Algorithm 1007

QNSTOP consists of serial and parallel (OpenMP) Fortran 2003 codes for the quasi-Newton stochastic optimization method of Castle and Trosset for stochastic search problems. A complete description of QNSTOP for both local search with stochastic objective and global search with “noisy” deterministic objective is given here, to the best of our knowledge, for the first time. For stochastic search problems, some convergence theory exists for particular algorithmic choices and parameter values. Both the parallel driver subroutine, which offers several parallel decomposition strategies, and the serial driver subroutine can be used for local stochastic search or global deterministic search, based on an input switch. Some performance data for computational systems biology problems is given.

Adjoint based state estimation of compressible flow in porous media

Abstract In this paper, we study the state estimation of compressible single phase flow in compressible porous media. The initial pressure distribution is estimated according to discrete adjoint approach based on the collected well pressure data. The first-order Tykhonov regularization method is used to obtain reasonable estimation. By analyzing the optimality condition of estimation problem, the discrete adjoint state equation and discrete adjoint gradient are derived based on the numerical scheme of the continuous equations. A quasi-Newton numerical optimization method related to adjoint gradient is proposed to solve the estimation problem. The estimation results with different regularization coefficients are compared and analyzed by numerical experiments. The deviation between the estimated pressure obtained without regularization and the real pressure is large. Estimation result with smaller deviation and higher smoothness can be obtained through appropriate regularization coefficient. When the observation error is large, the observed values generated by the estimated pressure fit well with the real pressure.

A superlinearly convergent nonmonotone quasi-Newton method for unconstrained multiobjective optimization

ABSTRACT We propose and analyse a nonmonotone quasi-Newton algorithm for unconstrained strongly convex multiobjective optimization. In our method, we allow for the decrease of a convex combination of recent function values. We establish the global convergence and local superlinear rate of convergence under reasonable assumptions. We implement our scheme in the context of BFGS quasi-Newton method for solving unconstrained multiobjective optimization problems. Our numerical results show that the nonmonotone quasi-Newton algorithm uses fewer function evaluations than the monotone quasi-Newton algorithm.

Peak Detection Based on FPGA Using Quasi-Newton Optimization Method for Femtosecond Laser Ranging

Dual femtosecond laser ranging (DFLR) is an enabling absolute distance measurement technique which is advantageous of high measurement precision, fast update rate, and large unambiguous range. Peak detection, which requires repeated online solution of nonlinear curve fitting, is a key module of the DFLR system and its performance affects the accuracy and real-time of the ranging system. In addition, for long baseline measurements based on satellite-borne platforms, the DFLR system has requirements in high integration, low cost, and small size. This paper presents a peak detection implementation on a field-programmable gate array (FPGA) that employs a Broyden-Fletcher-Goldfarb-Shanno (BFGS) method to handle nonlinear curve fitting. FPGA is used to explore the possibilities of parallel architecture for the acceleration of peak detection, and realize the miniaturization of the system. The detailed architecture design of the peak detection module using BFGS method (PD-BFGS) and two hardware structures are proposed. Then, the PD-BFGS module is applied to a DFLR system and evaluated with experiments on the absolute distance measurement. The experimental results indicate that the PD-BFGS based on FPGA effectively reduces the peak detection error by 42.81%, compared with the peak detection module using Caruana's method. For the DFLR system, the ranging error is reduced by 63.63% and the real-time updating of the ranging results is guaranteed.

QUASI NEWTON MULTI STEP n/4 SKIPPING TECHNIQUE FOR OPTIMIZATION OF UNCONSTRAINED NONLINEAR PROBLEMS

Ford and Moghrabi [2] introduced the multi-step quasi-Newton method for optimization, Where Hessian matrix is updated for each iteration, this make it computationally expensive. To overcome this problem Jhon and Nudrat [4] introduced a multi-step skipping technique, which shows a better experimental results. In Ford and Moghrabi, papers series, they introduced some techniques (Accumulative and Fixed-point approaches) to describe the parametric values needed to figure out the distances between many sets of iterations being involved in the recent interpolation curve.In this thesis, we extended the existing three-step Fixed-point skipping approach and examine the sensitivity of this technique. Evidently, experimental result provides, positive improvement with comparison to standard Broyden-Fletcher-GoldfrabShanno (BFGS) method. Result shows comparatively, three-step Fixed-point skipping technique computational saves time than existing single-step BFGS method

Augmented Lagrangian combined to evolutionary heuristic for energy efficiency in OCDMA networks

This paper proposes combining the augmented Lagrangian method (ALM) with evolutionary heuristic methods, as well as quasi-Newton optimization methods applied to the energy efficiency (EE) maximization in the optical code division multiple access (OCDMA) communication network. The particle swarm optimization (PSO) and a hybridization between the PSO and the gravitational search algorithm (GSA) called PSOGSA have been deployed. The ALM structure replaces the objective function and allows a best fit to the problem, and ultimately provide more information about the solution. Numerical results demonstrate the robustness and low-complexity of hybrid ALM-PSO, while the ALM associated with PSOGSA attains robustness at cost of high-complexity. In turn, the usually ALM combined with Broyden-Fletcher-Goldfarb-Shanno (BFGS) method presents convergence for a restrict scenarios, failing to perform suitably for networks with large numbers of users.

Determining optimal policies for sugarcane harvesting in Thailand using bi-objective and quasi-Newton optimization methods

In Thailand, the harvesting season for sugarcane usually begins in November and ends the following May. At the beginning of each harvesting season, the Royal Thai government sets the price of two types of sugarcane, namely fresh and fired, based on sweetness (sugar content) and gross weight of sugarcane delivered to the sugar factories. The aim of the present research is to determine optimal harvesting policies for the two types of sugarcane in sugarcane producing regions of Thailand in order to maximize revenue and minimize harvesting cost. In this paper, a harvesting policy is defined as the amount of each type of sugarcane harvested and delivered to the sugar factories during each 15-day period of a harvesting season. Two optimization methods have been used to solve this optimization problem, namely the ε-constraints method and a quasi-Newton optimization method. In the ε-constraints method, the problem is considered as a bi-objective optimization problem with the main objective being to determine harvesting policies that maximize the total revenue subject to upper bounds on the harvesting cost. In the quasi-Newton method, the aim is to determine the harvesting policy which gives maximum profit to the farmers subject to constraints on the maximum amount that can be cut in a 15-day period. The methods are used to determine optimal harvesting policies for the north, central, east, and north-east regions of Thailand for harvesting seasons 2012/13, 2013/14, and 2014/15 based on the data obtained from the Ministry of Industry and the Ministry of Agriculture and Co-operatives of the Royal Thai government.

Analytical level set fabrication constraints for inverse design

Inverse design methods produce nanophotonic devices with arbitrary geometries that show high efficiencies as well as novel functionalities. Ensuring fabricability during optimization of these unrestricted device geometries is a major challenge for these design methods. In this work, we construct a fabrication constraint penalty function for level set geometry representations of these devices. This analytical penalty function limits both the gap size and boundary curvature of a device. We incorporate this penalty in a fully automated optical design flow using a quasi-Newton optimization method. The performance of our design method is evaluated by designing a series of waveguide demultiplexers (WDM) and mode converters with various footprints and minimum feature sizes. Finally, we design and experimentally characterize three WDMs with a 80 nm, 120 nm and 160 nm feature size.

A New Transformed Biggs 's Self-Scaling Quasi-Newton Method for Optimization

A New Transformed Biggs 's Self-Scaling Quasi-Newton Method for Optimization

A modified quasi-Newton equation in the quasi-Newton methods for optimization

A modified quasi-Newton equation in the quasi-Newton methods for optimization

A Molecular Dynamics Approach for Optimizing Beam Intensities in IMPT Treatment Planning

This paper uses a molecular dynamics (MD) method for intensity-modulated proton therapy treatment planning optimization to overcome the problem of gradient-based optimization methods such as quasi-Newton, which is sensitive to starting conditions and is easily trapped in local minima. We implemented a molecular dynamics (MD) method and a quasi-Newton method for plan optimization. Three types of cancer cases, prostate, head-and-neck and lung cancer, were tested with three starting different initial conditions. Overall, the MD method consistently resulted in solutions with lower objective function values (OFVs) compared to those from the quasi-Newton algorithm. Furthermore, the MD method converged on the same OFV regardless of its initial starting points used for the prostate cancer case.

Laplace-domain waveform inversion using the l-BFGS method

ABSTRACTWe examine the performance of the limited-memory Broyden–Fletcher–Goldfarb–Shanno (l-BFGS) optimization method for full waveform inversions in the Laplace domain. Full waveform inversion methods generally adopt gradient-based optimization schemes for efficiency. Most Laplace-domain full waveform inversion methods published to date use the preconditioned steepest descent (PSD) method with the diagonal elements of the pseudo-Hessian as the preconditioner. These methods use a small fixed step length for stable convergence. The l-BFGS method is a quasi-Newton optimization method that efficiently approximates the inverse of the Hessian. We compare Laplace-domain inversion results from the PSD method with those from the l-BFGS method. Numerical examples using the SEG/EAGE salt model and the Pluto model demonstrate that the l-BFGS method with the Wolfe line search yields smaller cost function values than the PSD method.

A robust intensity modulated proton therapy optimizer based on Monte Carlo dose calculation.

Accuracy of dose calculation models and robustness under various uncertainties are key factors influencing the quality of intensity modulated proton therapy (IMPT) plans. To mitigate the effects of uncertainties and to improve the dose calculation accuracy, an all-scenario robust IMPT optimization based on accurate Monte Carlo (MC) dose calculation was developed. In the all-scenario robust IMPT optimization, dose volume histograms (DVHs) were computed for the nominal case and for each uncertainty scenario. All scenarios were weighted by DVH values dynamically throughout optimization iterations. In contrast, probabilistic approach weighted scenarios with fixed scenario weights and worst case optimizations picked one single scenario - the worst scenario for each iteration. Uncertainties in patient setup and proton range were considered in all clinical cases studied. Graphics processing unit (GPU) computation was employed to reduce the computational time in both the MC dose calculation and optimization stages. A previously published adaptive quasi-Newton method for proton optimization was extended to include robustness. To validate the all-scenario algorithm extension, it was compared with the single scenario optimization target volume (OTV) based approach in clinical cases of three different disease sites. Additional comparisons with worst case optimization methods were conducted to evaluate the performance of the all-scenario robust optimization against other robust optimizations. The all-scenario robust IMPT optimization spared organs at risk (OARs) better than the OTV-based method while maintaining target coverage and improving the robustness of targets and OARs. Compared with composite and voxel-wise worst case optimization, the all-scenario robust optimization converged faster, and arrived at solutions of tighter DVH robustness spread, better target coverage and lower OAR dose. An all-scenario robust IMPT treatment planning system was developed using an adaptive quasi-Newton optimization method. The optimization system was GPU-accelerated and based on MC dose calculation. Improved performance was observed in clinical cases when compared with worst case optimization methods.

Fast Neural Network Training on FPGA Using Quasi-Newton Optimization Method

In this brief, a customized and pipelined hardware implementation of the quasi-Newton (QN) method on field-programmable gate array (FPGA) is proposed for fast artificial neural networks onsite training, targeting at the embedded applications. The architecture is scalable to cope with different neural network sizes while it supports batch-mode training. Experimental results demonstrate the superior performance and power efficiency of the proposed implementation over CPU, graphics processing unit, and FPGA QN implementations.

A hybrid method of three-term conjugate gradient method and memoryless quasi-Newton method for unconstrained optimization

Memoryless quasi-Newton methods are studied for solving large-scale unconstrained optimization problems. Nakayama et al. (2017) proposed a memoryless quasi-Newton method based on the spectral-scaling Broyden family and showed that the method satisfies the sufficient descent condition and converges globally. To relax the conditions on parameters in the method, we apply the modification technique by Kou and Dai (2015) to the method of Nakayama et al., and we give a hybrid method of the three-term conjugate gradient method and the memoryless quasi-Newton method based on the spectral-scaling Broyden family. We show that our method satisfies the sufficient descent condition, and we prove that the method converges globally. Furthermore, we give a concrete choice of parameters for our method. Finally, some numerical results are given.