quasi-Newton Research Articles

New quasi-Newton methods via higher order tensor models

Many researches attempt to improve the efficiency of the usual quasi-Newton (QN) methods by accelerating the performance of the algorithm without causing more storage demand. They aim to employ more available information from the function values and gradient to approximate the curvature of the objective function. In this paper we derive a new QN method of this type using a fourth order tensor model and show that it is superior with respect to the prior modification of Wei et al. (2006) [4]. Convergence analysis gives the local convergence property of this method and numerical results show the advantage of the modified QN method.

Improved Hessian approximation with modified secant equations for symmetric rank-one method

Symmetric rank-one (SR1) is one of the competitive formulas among the quasi-Newton (QN) methods. In this paper, we propose some modified SR1 updates based on the modified secant equations, which use both gradient and function information. Furthermore, to avoid the loss of positive definiteness and zero denominators of the new SR1 updates, we apply a restart procedure to this update. Three new algorithms are given to improve the Hessian approximation with modified secant equations for the SR1 method. Numerical results show that the proposed algorithms are very encouraging and the advantage of the proposed algorithms over the standard SR1 and BFGS updates is clearly observed.

Improved Quasi-Newton Adaptive-Filtering Algorithm

An improved quasi-Newton (QN) algorithm that performs data-selective adaptation is proposed whereby the weight vector and the inverse of the input-signal autocorrelation matrix are updated only when the <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a priori</i> error exceeds a prespecified error bound. The proposed algorithm also incorporates an improved estimator of the inverse of the autocorrelation matrix. With these modifications, the proposed QN algorithm takes significantly fewer updates to converge and yields a reduced steady-state misalignment relative to a known QN algorithm proposed recently. These features of the proposed QN algorithm are demonstrated through extensive simulations. Simulations also show that the proposed QN algorithm, like the known QN algorithm, is quite robust with respect to roundoff errors introduced in fixed-point implementations.

Resolution and uniqueness of estimated parameters of a model of thin filament regulation in solution

The estimation of chemical kinetic rate constants for any non-trivial model is complex due to the nonlinear effects of second order chemical reactions. We developed an algorithm to accomplish this goal based on the Damped Least Squares (DLS) inversion method and then tested the effectiveness of this method on the McKillop–Geeves (MG) model of thin filament regulation. The kinetics of MG model is defined by a set of nonlinear ordinary differential equations (ODEs) that predict the evolution of troponin–tropomyosin–actin and actin–myosin states. The values of the rate constants are estimated by integrating these ODEs numerically and fitting them to a series of stopped-flow pyrene fluorescence transients of myosin-S1 fragment binding to regulated actin in solution. The accuracy and robustness of the estimated rate constants are evaluated for DLS and two other methods, namely quasi-Newton (QN) and simulated annealing (SA). The comparison of these methods revealed that SA provides the best estimates of the model parameters because of its global optimization scheme. However it converges slowly and does quantify the uniqueness of the estimated parameters. On the other hand the QN method converges rapidly but only if the initial guess of the parameters is close to the optimum values, otherwise it diverges. Overall, the DLS method proves to be the most convenient method. It converges fast and was able to provide excellent estimates of kinetic parameters. Furthermore, DLS provides the model resolution matrix, which quantifies the interdependence of model parameters thereby evaluating the uniqueness of their estimated values. This property is essential for estimating of the dependence of the model parameters on experimental conditions (e.g. Ca 2+ concentration) when it is assessed from noisy experimental data such as pyrene fluorescence from stopped-flow transients. The advantages of the DLS method observed in this study should be further examined in other physicochemical systems to firmly establish the observed effectiveness of DSL vs. the other parameter estimation methods.

Memoryless Modified Symmetric Rank-One Method for Large-Scale Unconstrained Optimization

Problem statement: Memoryless QN methods have been regarded effective techniques for solving large-scale problems that can be considered as one step limited memory QN methods. In this study, we present a scaled memoryless modified Symmetric Rank-One (SR1) algorithm and investigate the numerical performance of the proposed algorithm for solving large-scale unconstrained optimization problems. Approach: The basic idea is to apply the modified Quasi-Newton (QN)equations, which uses both the gradients and the function values in two successive points in the frame of the scaled memoryless SR1 update, in which the modified SR1 update is reset, at every iteration, to the positive multiple of the identity matrix. The scaling of the identity is chosen such that the positive definiteness of the memoryless modified SR1 update is preserved. Results: Under some suitable conditions, the global convergence and rate of convergence are established. Computational results, for a test set consisting of 73 unconstrained optimization problems, show that the proposed algorithm is very encouraging. Conclusion/Recommendations: In this study a memoryless QN method developed for solving large-scale unconstrained optimization problems, in which the SR1 update based on the modified QN equation have applied. An important feature of the proposed method is that it preserves positive definiteness of the updates. The presented method owns global and R-linear convergence.Numerical results showed that the proposed method is encouraging comparing with the methods MMBFGS and FRCG.

H∞ bounds for quasi-Newton adaptive algorithm

The upper and lower bounds for the H ∞ norm of the quasi-Newton (QN) family of adaptive filtering algorithms are obtained. A simulation study of the behaviour of the bounds with respect to the iteration order and the weight-vector dimension for a practical input data model reveals that the QN family is quite robust against modelling errors and errors due to the uncertainty in the initial weight-vector assumption.

L-Broyden methods: a generalization of the L-BFGS method to the limited-memory Broyden family

The paper derives a multiplicative update equation for the convex Broyden family of quasi-Newton (QN) updates. The well-known multiplicative Broyden-fletcher-Goldfarb-Shanno (BFGS) update is a special case of this. Using a self-scaling parameter, the formula is extended to the SR1 update. It is shown that for each Broyden update, a pair of multiplicative update formulae can be defined (coincident in the case of Davidon-fletcher-Powell (DFP)). The multiplicative updates are used to define limited-memory QN algorithms, denoted the L-Broyden methods, of which L-BFGS is a special case. Numerical results show that the L-Broyden methods are competitive with extensions of the variable storage conjugate gradients limited memory QN method to other Broyden updates, but that L-BFGS with Shanno scaling remains the most efficient and reliable method in the L-Broyden family.

MULTILAYER PERCEPTRON NEURAL ANALYSIS OF EDGE COUPLED AND CONDUCTOR-BACKED EDGE COUPLED COPLANAR WAVEGUIDES

In recent years, Computer Aided Design (CAD) based on Artiflcial Neural Networks (ANNs) have been introduced for microwave modeling, simulation and optimization. In this paper, the characteristic parameters of edge coupled and conductor-backed edge coupled Coplanar Waveguides have been determined with the use of ANN model. Eight learning algorithms, Levenberg-Marquart (LM), Bayesian Regularization (BR), Quasi-Newton (QN), Scaled Conjugate Gradient (SCG), Conjugate Gradient of Fletcher-Powell (CGF), Resilient Propagation (RP), Conjugate Gradient back-propagation with Polak-Ribiere (CGP) and Gradient Descent (GD) are used to train the Multi-Layer Perceptron Neural Networks (MLPNNs). The results of neural models presented in this paper are compared with the results of Conformal Mapping Technique (CMT). The neural results are in very good agreement with the CMT results. When the performances of neural models are compared with each other, the best results are obtained from the neural networks trained by LM and BR algorithms.

NEURAL MODEL FOR CIRCULAR-SHAPED MICROSHIELD AND CONDUCTOR-BACKED COPLANAR WAVEGUIDE

A Computer Aided Design (CAD) approach based on Artiflcial Neural Networks (ANN's) is successfully introduced to determine the characteristic parameters of Circular-shaped Microshield and Conductor-backed Coplanar Waveguide (CMCB-CPW). ANN's have been promising tools for many applications and recently ANN has been introduced to microwave modeling, simulation and optimization. The Multi Layered Perceptron (MLP) neural network used in this work were trained with Levenberg-Marquart (LM), Bayesian regularization (BR), Quasi-Newton (QN), Scaled Conjugate gradient (SCG), Conjugate gradient of Fletcher-Powell (CGF) and Conjugate Gradient backpropagation with Polak-Ribiere (CGP) learning algorithms. This has facilitated the usage of ANN models. The notable beneflts are simplicity & accurate determination of the characteristic parameters of CMCBCPW's. The greatest advantage is lengthy formulas can be dispensed with.

Flow Regime Classification Using Artificial Neural Network Trained on Electrical Capacitance Tomography Sensor Data

The main goal of the presented work is to analyse the performance of the Multi-Layer Perceptron (MLP) neural network for flow regime classification based on sets of simulated Electrical Capacitance Tomography (ECT) data. Normalised ECT data have been used to separately train several MLPs employing various commonly used back-propagation learning algorithms, namely the Levenberg-Marquardt (LM), Quasi-Newton (QN) and Resilient-Backpropagation (RP), to classify the gas-oil flow regimes. The performances of the MLPs have been analysed based on their correct classification percentage (CCP). The results demonstrate the feasibility of using MLP, and the superiority of LM algorithm for flow regime classification based on ECT data.

NEURAL MODELS FOR THE ELLIPTIC- AND CIRCULAR-SHAPED MICROSHIELD LINES

This article presents a new approach based on artificial neural networks (ANNs) to calculate the characteristic parameters of elliptic and circular-shaped microshield lines. Six learning algorithms, bayesian regularization (BR), Levenberg-Marquardt (LM), quasi- Newton (QN), scaled conjugate gradient (SCG), resilient propagation (RP), and conjugate gradient of Fletcher-Reeves (CGF), are used to train the ANNs. The neural results are in very good agreement with the results reported elsewhere. When the performances of neural models are compared with each other, the best and worst results are obtained from the ANNs trained by the BR and CGF algorithms, respectively.

Nonlinear predictive control to track deviated power of an identified NNARX model of a hydro plant

This paper presents the performance study of predictive control approach in application to hydro plant. The tracking on deviated power as reference signal for identified neural network nonlinear autoregressive with exogenous signal (NNARX) hydro plant model is studied. A detailed plant dynamics constituting various hydro-components and -structures are considered for identification of its corresponding NNARX model. In obtaining an appropriate NNARX model structure, the plant is simulated on random load disturbance variation with input as random gate position and output as deviated power. With the identified model, a nonlinear predictive control (NPC) strategy is applied using Levenberg–Marquardt (LM) and Quasi-Newton (QN) algorithms for optimization of control performance index (CPI). The study also describes a control approach involving extraction of linear model from the neural network (NN) model, based on instantaneous linearization theory. Its tracking performance is compared with NPC on different nature of reference signals. These control approaches are applied so as to cause model output track on various deviated power as a reference signal.

NEURAL MODELS FOR THE BROADSIDE-COUPLED V-SHAPED MICROSHIELD COPLANAR WAVEGUIDES

This article presents a new approach based on multilayered perceptron neural networks (MLPNNs) to calculate the odd-and even-mode characteristic impedances and effective permittivities of the broadside-coupled V-shaped microshield coplanar waveguides (BC-VSMCPWs). Six learning algorithms, bayesian regulation (BR), Levenberg-Marquardt (LM), quasi-Newton (QN), scaled conjugate gradient (SCG), resilient propagation (RP), and conjugate gradient of Fletcher-Powell (CGF), are used to train the MLPNNs. The neural results are in very good agreement with the results reported elsewhere. When the performances of neural models are compared with each other, the best and worst results are obtained from the MLPNNs trained by the BR and CGF algorithms, respectively.

Electromagnetic device optimization by hybrid evolution strategy approaches

PurposeThis paper aims to show on a widely used benchmark problem that chaotic sequences can improve the search ability of evolution strategies (ES).Design/methodology/approachThe Lozi map is used to generate new individuals in the framework of ES algorithms. A quasi‐Newton (QN) method is also used within the iterative loop to improve the solution's quality locally.FindingsIt is shown that the combined use of chaotic sequences and QN methods can provide high‐quality solutions with small standard deviation on the selected benchmark problem.Research limitations/implicationsAlthough the benchmark is considered to be representative of typical electromagnetic problems, different test cases may give less satisfactory results.Practical implicationsThe proposed approach appears to be an efficient general purpose optimizer for electromagnetic design problems.Originality/valueThis paper introduces the use of chaotic sequences in the area of electromagnetic design optimization.

Estimation of Dye Configuration from Conventional Chiroptical Spectra of Porphyrin Integrates: Combination of Exciton Theory with Monte Carlo Molecular Structural Simulation

Dye integrates (arrays and aggregates) are the subject of current interest in photochemical devices. However, they are in general not suitable for X-ray crystallography because of their poor crystallinity. Here, we improved a simple method of estimating dye configurations in porphyrin integrates from their visible absorption (AB) and circular dichroism (CD) spectra. For this purpose, we calculate the dipolar and optical rotatory strengths of an integrate on the basis of the exciton theory for a given porphyrin configuration, generate the theoretical AB and CD spectra of the dye integrate using a phenomenological line shape function, a Voigt function with an adjustable line width, and optimize the configuration by minimizing the square sum (S) of the difference between the observed and calculated spectra. We adopted two optimization methods to achieve a least-squares fit between the calculated and observed spectra: the Metropolis Monte Carlo (MC) method, which incorporates S into the molecular force-field energy as a constraint, and the quasi-Newton (QN) method, which numerically minimizes S and uses no molecular force field. To check the feasibility of these methods, we simulated the AB and CD spectra of Tröger's base and meso-meso-linked porphyrins using the QN program, then compared the dye configurations with their X-ray structures. The calculated dye configuration of Tröger's base porphyrin is sufficiently in agreement with that of the X-ray structure (RMSD=0.21 A for the ZnS4 center), whereas that of meso-meso-linked porphyrin was not. These results were discussed in terms of charge transfer between two porphyrins. Finally, we applied the QN and MC methods to the structural estimation of a newly prepared peptide-linked bis(porphyrin) Boc-(PorZn,S)2-OBut. The best configurations estimated by these two methods were sufficiently in agreement with each other.

NEURAL MODELS FOR COPLANAR STRIP LINE SYNTHESIS

Simple and accurate models based on artificial neural networks (ANNs) are presented to accurately determine the physical dimensions of coplanar strip lines (CPSs). Five learning algorithms, Levenberg-Marquardt (LM), bayesian regularization (BR), quasiNewton (QN), conjugate gradient with Fletcher (CGF), and scaled conjugate gradient (SCG), are used to train the neural models. The neural results are compared with the results of the quasi-static analysis and the synthesis formulas available in the literature. The accuracy of the neural model trained by LM algorithm is found to be better than 0.24% for 10614 CPS samples.

Neural models for quasi-static analysis of conventional and supported coplanar waveguides

Neural models based on multilayered perceptrons (MLPs) for computing the effective permittivities and the characteristic impedances of both the conventional coplanar waveguides (CCPWs) and the supported coplanar waveguides (SCPWs) are presented. Six learning algorithms, Levenberg–Marquardt (LM), Bayesian regularization (BR), quasi-Newton (QN), scaled conjugate gradient (SCG), conjugate gradient of Fletcher–Powell (CGF), and resilient propagation (RP), are used to train the MLPs. The results of neural models presented in this paper are compared with the results of the experimental works, the conformal mapping technique (CMT), the spectral domain approach (SDA), and three commercial electromagnetic simulators such as IE3D, CAPIND2D, and MMICTL. The neural results are in very good agreement with the theoretical and experimental results. When the performances of neural models are compared with each other, the best result is obtained from the MLPs trained by the LM algorithm.

A limited-memory quasi-Newton inversion for 1D magnetotellurics

We apply a limited-memory quasi-Newton (QN) method to the 1D magnetotelluric (MT) inverse problem. Using this method we invert a realistic synthetic MT impedance data set calculated for a layered earth model. The calculation of gradients based on the adjoint method speeds up the inverse problem solution many times. In addition, regularization stabilizes the QN inversion result and a few correction pairs are sufficient to produce reasonable results. Comparison with the L-BFGS-B algorithm shows similar convergence rates. This study is a first step towards the solution of large-scale electromagnetic problems, with a full treatment of the 3D conductivity structure of the earth.

Evaluation of optimization techniques for parameter estimation: Application to ethanol fermentation considering the effect of temperature

Optimization techniques are evaluated to estimate the kinetic model parameters of batch fermentation process for ethanol production using Saccharomyces cerevisiae. Batch experimental observations at five temperatures (28, 31, 34, 37 and 40 °C) are used to formulate the parameter estimation problem. The potential of Quasi-Newton (QN) and Real-Coded Genetic Algorithm (RGA) to solve the estimation problem is considered to find out the optimal solution. Subsequently, the optimized parameters ( μ max, X max, P max, Y x and Y p x ) were characterized by correlation functions assuming temperature dependence. The kinetic models optimized by QN and RGA describe satisfactorily the batch fermentation process as demonstrated by the experimental results.

QN Inversion of Large-scale MT Data

A limited memory quasi-Newton (QN) method with simple bounds is applied to a 1-D magnetotel- luric (MT) problem. The method is used to invert a realistic synthetic MT impedance dataset, calculated for a layered earth model. An adjoint method is employed to calculate the gradients and to speed up the inverse problem solution. In addition, it is shown that regularization stabilizes the QN inversion result. We demonstrate that only a few correction pairs are enough to produce reasonable results. Comparison with inversion based on known L-BFGS-B optimization algorithm shows similar convergence rates. The study presented is a first step towards the solution of large-scale electromagnetic problems with a full 3-D conductivity structure of the Earth.