quasi-Newton Optimization Method Research Articles

Oil removal efficiency forecast of a Dissolved Air Flotation (DAF) reduced scale prototype using the dimensionless number of Damköhler

The increasingly automated industrial processes are responsible for a large part of the generation of oily effluents, which present great difficulty of treatment due to the complexity of removal of fine particles of oil. The application of biosurfactants and the dimensionless number of Damköhler (Da) in a prototype of dissolved air flotation (DAF) allowed predictions of efficiency of oil removal in oily waters. Two different biosurfactants were used in order to confirm a first-order kinetic law for the DAF process in the water-oil separation. The kinetic behavior of the DAF process was associated by analogy with a perfect blending reactor (Continued Stirred Tank Reactor – CSTR). In this way, a laboratory float was used in order to allow experiments with the biosurfactants produced by the bacteria Bacillus sp. and Pseudomonas aeruginosa. A first-order kinetic model was fitted to the experimental data, using Quasi-Newton numerical optimization methods The application of the dimensionless number of Damköhler, together with the use of the biosurfactants, allowed prediction of the oil removal efficiency, around 90%. It has been observed that it is also possible to predict the flotation chamber volume based on a desired value of removal efficiency.

A direct search quasi-Newton method for nonsmooth unconstrained optimization

A direct search quasi-Newton algorithm is presented for local minimization of Lipschitz continuous black-box functions. The method estimates the gradient via central differences using a maximal frame around each iterate. When nonsmoothness prevents progress, a global direction search is used to locate a descent direction. Almost sure convergence to Clarke stationary point(s) is shown, where convergence is independent of the accuracy of the gradient estimates. Numerical results show that the method is effective in practice. doi:10.1017/S1446181117000323

Quasi-Newton methods for multiobjective optimization problems

This work is an attempt to develop multiobjective versions of some well-known single objective quasi-Newton methods, including BFGS, self-scaling BFGS (SS-BFGS), and the Huang BFGS (H-BFGS). A comprehensive and comparative study of these methods is presented in this paper. The Armijo line search is used for the implementation of these methods. The numerical results show that the Armijo rule does not work the same way for the multiobjective case as for the single objective case, because, in this case, it imposes a large computational effort and significantly decreases the speed of convergence in contrast to the single objective case. Hence, we consider two cases of all multi-objective versions of quasi-Newton methods: in the presence of the Armijo line search and in the absence of any line search. Moreover, the convergence of these methods without using any line search under some mild conditions is shown. Also, by introducing a multiobjective subproblem for finding the quasi-Newton multiobjective search direction, a simple representation of the Karush–Kuhn–Tucker conditions is derived. The H-BFGS quasi-Newton multiobjective optimization method provides a higher-order accuracy in approximating the second order curvature of the problem functions than the BFGS and SS-BFGS methods. Thus, this method has some benefits compared to the other methods as shown in the numerical results. All mentioned methods proposed in this paper are evaluated and compared with each other in different aspects. To do so, some well-known test problems and performance assessment criteria are employed. Moreover, these methods are compared with each other with regard to the expended CPU time, the number of iterations, and the number of function evaluations.

Formula omitted] optimization for multigrid fluorescence tomography of small animals using simplified spherical harmonics

We present the first Lq−Lp optimization scheme for fluorescence tomographic imaging. This is then applied to small animal imaging. Fluorescence tomography is an ill-posed, and in full generality, a nonlinear problem that seeks to image the 3D concentration distribution of a fluorescent agent inside a biological tissue. Standard candidates for regularization to deal with the ill-posedness of the image reconstruction problem include L1 and L2 regularization. In this work, a general Lq−Lp regularization framework (Lq discrepancy function - Lp regularization term) is introduced for fluorescence tomographic imaging. A method to calculate the gradient for this general framework is developed which allows evaluating the performance of different cost functions/regularization schemes in solving the fluorescence tomographic problem. The simplified spherical harmonics approximation is used to accurately model light propagation inside the tissue. Furthermore, a multigrid mesh is utilized to decrease the dimension of the inverse problem and reduce the computational cost of the solution. The inverse problem is solved iteratively using an lm-BFGS quasi-Newton optimization method. The simulations are performed under different scenarios of noisy measurements. These are carried out on the Digimouse numerical mouse model with the kidney being the target organ. The evaluation of the reconstructed images is performed both qualitatively and quantitatively using several metrics including QR, RMSE, CNR, and TVE under rigorous conditions. The best reconstruction results under different scenarios are obtained with an L1.5−L1 scheme with premature termination of the optimization process. This is in contrast to approaches commonly found in the literature relying on L2−L2 schemes.

Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization

In this paper we study stochastic quasi-Newton methods for nonconvex stochastic optimization, where we assume that noisy information about the gradients of the objective function is available via a stochastic first-order oracle ($\mathcal{SFO}$). We propose a general framework for such methods, for which we prove almost sure convergence to stationary points and analyze its worst-case iteration complexity. When a randomly chosen iterate is returned as the output of such an algorithm, we prove that in the worst case, the $\mathcal{SFO}$-calls complexity is $O(\epsilon^{-2})$ to ensure that the expectation of the squared norm of the gradient is smaller than the given accuracy tolerance $\epsilon$. We also propose a specific algorithm, namely, a stochastic damped limited-memory BFGS (SdLBFGS) method, that falls under the proposed framework. Moreover, we incorporate the stochastic variance reduced gradient variance reduction technique into the proposed SdLBFGS method and analyze its $\mathcal{SFO}$-calls complexity. Numerical results on a nonconvex binary classification problem using a support vector machine and a multiclass classification problem using neural networks are reported.

FPGA-based acceleration of Davidon-Fletcher-Powell quasi-Newton optimization method

Quasi-Newton methods are the most widely used methods to find local maxima and minima of functions in various engineering practices. However, they involve a large amount of matrix and vector operations, which are computationally intensive and require a long processing time. Recently, with the increasing density and arithmetic cores, field programmable gate array (FPGA) has become an attractive alternative to the acceleration of scientific computation. This paper aims to accelerate Davidon-Fletcher-Powell quasi-Newton (DFP-QN) method by proposing a customized and pipelined hardware implementation on FPGAs. Experimental results demonstrate that compared with a software implementation, a speed-up of up to 17 times can be achieved by the proposed hardware implementation.

Angle Estimation and Self-calibration Method for Bistatic MIMO Radar with Transmit and Receive Array Errors

The mutual coupling, gain-phase error and sensor position error of the transmit and receive arrays would significantly degrade the performance of high-resolution direction of departure and direction of arrival estimation algorithms. By applying two well-calibrated instrumental sensors in both transmit and receive arrays, a novel angle estimation and self-calibration method, which considers the combined influences of the above three array errors, is proposed for bistatic MIMO radar. We show that the integrated array errors can be translated to angularly dependent gain-phase error. Then, a reduced dimensional method is used to estimate the angles and the angularly dependent gain-phase error coefficients. Based on the estimated angularly dependent gain-phase error coefficients, the nonlinear least squares optimization model for the three array errors are established and a quasi-Newton optimization method is applied to estimate the mutual coupling matrix, gain-phase error and sensors position errors of the transmit and receive arrays, respectively. Our simulation results corroborate our analysis.

An operational dynamical neuro-forecasting model for hydrological disasters

In the last decades, artificial neural network has been increasingly applied in hydrological modeling given its potential to process the complex nonlinear relationships of the associated physical-environmental variables and produce a suitable solution (for instance, a forecasting model) in a relatively short time. In this scope, this work reports the design methodology and the operational results obtained with an artificial neural network-based model developed to forecast, with 2 h in advance, the level of a river in the mountainous region of Rio de Janeiro state in Brazil. This is an area susceptible to natural disasters with recent records of floods and landslides that caused environmental and socio-economic damage of large proportions. The proposed neural network uses an innovative learning algorithm (the quasi-Newton optimization method is applied to the slopes of each hidden activation function) and, as input features, values of rainfall and river level data collected from 8 monitoring stations located on studied watershed between 2013 and 2014. The results of the neural model, with NASH index greater than 0.86, are promising making possible its operational use on an issuing flood alerts system.

New data assimilation system DNDAS for high-dimensional models**Project supported by the National Natural Science Foundation of China (Grant Nos. 41475094 and 41375113).

The tangent linear (TL) models and adjoint (AD) models have brought great difficulties for the development of variational data assimilation system. It might be impossible to develop them perfectly without great efforts, either by hand, or by automatic differentiation tools. In order to break these limitations, a new data assimilation system, dual-number data assimilation system (DNDAS), is designed based on the dual-number automatic differentiation principles. We investigate the performance of DNDAS with two different optimization schemes and subsequently give a discussion on whether DNDAS is appropriate for high-dimensional forecast models. The new data assimilation system can avoid the complicated reverse integration of the adjoint model, and it only needs the forward integration in the dual-number space to obtain the cost function and its gradient vector concurrently. To verify the correctness and effectiveness of DNDAS, we implemented DNDAS on a simple ordinary differential model and the Lorenz-63 model with different optimization methods. We then concentrate on the adaptability of DNDAS to the Lorenz-96 model with high-dimensional state variables. The results indicate that whether the system is simple or nonlinear, DNDAS can accurately reconstruct the initial condition for the forecast model and has a strong anti-noise characteristic. Given adequate computing resource, the quasi-Newton optimization method performs better than the conjugate gradient method in DNDAS.

Computing Planar and Spherical Choreographies

An algorithm is presented for numerical computation of choreographies in the plane in a Newtonian potential and on the sphere in a cotangent potential. It is based on stereographic projection, approximation by trigonometric polynomials, and quasi-Newton and Newton optimization methods with exact gradient and exact Hessian matrix. New choreographies on the sphere are presented.

Optimal closure of system actuators for transient control: an analytical approach

The effect of downstream valve closure scheduling was analyzed to find optimal closure parameters that lead to the minimal maximum pressure head in the water distribution system. Several valve closure strategies were explored, combining the known valve performance curve (change in flow as a function of change in valve's opened area) with unknown valve closure curve (change in valve's opened area as a function of time). Second-order polynomial curve, power function curve, and piece-wise linear curve were implemented and compared. Genetic algorithm and quasi-Newton (QN) optimization methods were applied. The methodology was tested for three networks, including looped gravitational and pressurized networks. The results demonstrate that flexible multi-parametric valve closure curve and QN optimization method are more effective in minimizing the maximum pressure head in the system.

On the efficiency of gradient based optimization algorithms for DNS-based optimal control in a turbulent channel flow

We analyze the performance of different limited-memory quasi-Newton methods for unconstrained DNS-based optimization. Optimization based on Direct Numerical Simulation (DNS) of turbulent flows is extremely expensive, as functional and gradient evaluations require the simulation of Navier–Stokes and adjoint Navier–Stokes equations with high space and time resolution. Nowadays, simple and robust nonlinear conjugate gradient methods are generally used for DNS-based optimal control, as they do not require much memory overhead in a large control space. In the current study, we investigate the use of quasi-Newton methods instead. They combine a cheap approximation of the Hessian to improve step direction and step length, leading to faster convergence of the optimization. Since control spaces are often large in DNS-based optimization, we investigate only limited-memory quasi-Newton methods. Three methods are studied, i.e., the discrete truncated Newton method, the limited-memory BFGS method, and the damped L-BFGS method. The latter method is designed for constrained optimization, but can also address unconstrained problems. Furthermore, the damped L-BFGS method only requires the Armijo condition in the line search, not the Wolfe conditions, limiting expensive functional and gradient evaluations. We investigate the combination of the three quasi-Newton methods with three different line-search methods either based on bisection, quadratic interpolation, or cubic interpolation. Initially, all possible combinations are evaluated in a test problem that is based on the extended Rosenbrock function. The three best performing methods are further tested in two different DNS-based optimal control cases in a turbulent channel flow at Reτ=180. This reveals that the damped L-BFGS method in combination with a cubic line search performs best, closely followed by classical L-BFGS with cubic line search. Though the damped L-BFGS often requires a few more iterations to reach convergence, this is compensated by a more cost effective line search, with fewer functional and gradient evaluations. Moreover, compared to the conjugate-gradient method, damped L-BFGS speeds up convergence by a factor of four.

Barzilai and Borwein’s method for multiobjective optimization problems

The present study is an attempt to extend Barzilai and Borwein's method for dealing with unconstrained single objective optimization problems to multiobjective ones. As compared with Newton, Quasi-Newton and steepest descent multi-objective optimization methods, Barzilai and Borwein multiobjective optimization (BBMO) method requires simple and quick calculations in that it makes no use of the line search methods like the Armijo rule that necessitates function evaluations at each iteration. It goes without saying that the innovative aspect of the current study is due to the use of no function evaluations in comparison with other multi-objective optimization non-parametric methods (e.g. Newton, Quasi-Newton and steepest descent methods, to name a few) that have been investigated so far. Also, the convergence of the BBMO method for the objective functions assumed to be twice continuously differentiable has been proved. MATLAB software was utilized to implement the BBMO method, and the results were compared with the other methods mentioned earlier. Using some performance assessment, the quality of nondominated frontier of BBMO was analogized to above mentioned methods. In addition, the approximate nondominated frontiers gained from the methods were compared with the exact nondominated frontier for some problems. Also, performance profiles are considered to visualize numerical results presented in tables.

A dynamic XFEM formulation for crack identification

Nelder-Mead (NM) and Quasi-Newton (QN) optimization methods are used for the numerical solution of crack identification problems in elastody- namics. Fracture is modeled by the eXtended Finite Element Method. The Newmark-b method with Rayleigh damping is employed for the time integra- tion. The effects of various dynamical test loads on the crack identification are investigated. For a time- harmonic excitation with a single frequency and a short-duration signal measured along part of the external boundary, the crack is detected through the solution of an inverse time-dependent problem. Com- pared to the static load, we show that the dynamic loads are more effective for crack detection problems. Moreover, we tested different dynamic loads and find that NM method works more efficient under the harmonic load than the pounding load while the QN method achieves almost the same results for both load types.

Experimental and numerical study of cavitation inception phenomenon in diesel injector nozzles

In this paper the experimental and numerical methods were combined to investigate the critical conditions of cavitation inception in diesel injector nozzles. It was found that at the beginning of cavitation inception, the discharge coefficient presents a slight increase and then a special turning point appears which can therefore be taken as the critical point for cavitation inception. Effects of needle lift, length–diameter ratio, orifice inclination angle, back pressure, and scaled-up times on critical cavitation number were studied through the experimental method. Due to the difficulties in manufacturing, influence of inlet rounding radius was investigated by using the numerical simulation method. Based on all the experimental and numerical simulation data, a correlation of critical cavitation number with above 6 geometrical and dynamic parameters for nozzle flow was developed through statistical analysis using the Quasi-Newton (BFGS) and Universal Global Optimization (UGO) methods. The relative error between predicted results and experimental data is less than 7%. These results indicate that the derived expression has good accuracy to predict the start of cavitation.

Interpretation of disturbed data in thermal response tests using the infinite line source model and numerical parameter estimation method

Effective ground thermal conductivity and borehole thermal resistance, which are key parameters in the design of borehole heat exchangers (BHEs), are often determined on the basis of in-situ thermal response tests (TRTs). However, many disturbance factors can affect the accuracy of a TRT, e.g., voltage fluctuations from the power grid and oscillating external environments where a TRT rig is installed. Interpretation of TRT data is often done using the infinite line source (ILS) model, combined with the sequential plot method, because it is not only simple but also provides additional information about the estimation behavior and convergence. However, estimation behavior using the sequential method tends to fluctuate over time because the constant heat flux assumption is always violated as a result of the disturbance factors. As an alternative, a temporal superposition applied analytical model can be used in a recursive curve fitting manner, but this method cannot provide the additional information that sequential method can. In this study, as a solution for interpreting disturbed TRT data and to utilize additional information from the sequential plot method, we proposed an alternative method using a temporal superposition applied ILS model combined with the quasi-Newton optimization method. To verify the effectiveness, the proposed method was applied to in-situ TRTs and the results were compared with those from the conventional method in terms of the estimation stability and convergence speed. The results showed that, compared to the conventional sequential method using the ILS model, the proposed method yielded standard deviations for the effective thermal conductivity and borehole thermal resistance that were at least six times and four times lower, respectively. Moreover, the proposed method was able to achieve about four times faster convergence speeds.

A Broyden Class of Quasi-Newton Methods for Riemannian Optimization

This paper develops and analyzes a generalization of the Broyden class of quasi-Newton methods to the problem of minimizing a smooth objective function $f$ on a Riemannian manifold. A condition on vector transport and retraction that guarantees convergence and facilitates efficient computation is derived. Experimental evidence is presented demonstrating the value of the extension to the Riemannian Broyden class through superior performance for some problems compared to existing Riemannian BFGS methods, in particular those that depend on differentiated retraction.

Three-dimensional electrical conductivity structure beneath Australia from inversion of geomagnetic observatory data: evidence for lateral variations in transition-zone temperature, water content and melt

In this paper, we report the 3-D electrical conductivity distribution beneath the Australian continent in the depth range 410–1600 km, which we have imaged by inverting C-response estimates from a regional network of geomagnetic observatories. The inversion scheme is based on a quasi-Newton optimization method while the forward algorithm relies on an integral-equation approach. To properly account for the ocean effect in responses at coastal observatories we included a high-resolution (1° × 1°) fixed thin laterally varying surface conductance layer. As starting model in the inversion we considered a laboratory-based 3-D conductivity model of the region obtained from seismic surface wave data and thermodynamic modelling. This model provides a good fit to observed C-response estimates supporting its choice as initial model. The most striking feature of the obtained 3-D model is a high-conductivity anomaly in the lower part of the mantle transition zone (MTZ; 520–660 km depth) beneath southeastern Australia implying considerable lateral as radial heterogeneity in the conductivity structure. The high-conductivity region appears to be 0.5–1 log units more conductive than previous global and other regionalized 1-D models. Further analysis using laboratory-based conductivity models combined with thermochemical phase equilibrium computations shows that the strong conductivity anomaly implies water contents of around 0.1 wt per cent in the upper part and >0.4 wt per cent in the lower part of the MTZ. This implies a large MTZ water reservoir that likely totals one to three times that which currently resides in the oceans. The amount of water in the lower MTZ appears to exceed the experimentally determined water storage capacity of the main lower MTZ mineral ringwoodite, which, as a result, undergoes dehydration-induced partial melting. Including contributions to conductivity from a thin melt layer (20 km thick) located in the mid-MTZ increases conductivity locally in the melt layer to ∼1 S m−1, that is, about 0.5 log units more conductive than the average surrounding mantle. This provides an adequate explanation for the strong conductivity anomalies observed beneath part of the continent and points to lateral variations in melt in the MTZ.

Determination of the 3-D distribution of electrical conductivity in Earth’s mantle from Swarm satellite data: Frequency domain approach based on inversion of induced coefficients

Mapping the three-dimensional (3-D) electrical conductivity of Earth’s mantle has been identified as one of the primary scientific objectives for the Swarm satellite mission. We present a 3-D frequency domain inversion scheme to recover mantle conductivity from satellite magnetic data. The scheme is based on an inversion of time spectra of internal (induced) spherical harmonic coefficients of the magnetic potential due to magnetospheric sources. Time series of internal and external (inducing) coefficients, whose determination is a prerequisite for this formulation, will be available as a Swarm Level-2 data product. An iterative gradient-type (quasi-Newton) optimization method is chosen to solve our 3-D non-linear inverse problem. In order to make the inversion tractable, we elaborate an adjoint approach for a fast and robust calculation of the data misfit gradient. We verify our approach with synthetic, but realistic time spectra of internal coefficients, obtained by simulating induction due to a realistic magnetospheric source in a 3-D conductivity model of the Earth. In these model studies, both shape and conductivity of a large-scale conductivity anomaly in the mid-mantle are recovered very well. The inversion scheme also shows to be robust with respect to noise and is therefore ready to process Swarm data.

Chaos optimization algorithms based on chaotic maps with different probability distribution and search speed for global optimization

Chaos optimization algorithms (COAs) usually utilize the chaotic map like Logistic map to generate the pseudo-random numbers mapped as the design variables for global optimization. Many existing researches indicated that COA can more easily escape from the local minima than classical stochastic optimization algorithms. This paper reveals the inherent mechanism of high efficiency and superior performance of COA, from a new perspective of both the probability distribution property and search speed of chaotic sequences generated by different chaotic maps. The statistical property and search speed of chaotic sequences are represented by the probability density function (PDF) and the Lyapunov exponent, respectively. Meanwhile, the computational performances of hybrid chaos-BFGS algorithms based on eight one-dimensional chaotic maps with different PDF and Lyapunov exponents are compared, in which BFGS is a quasi-Newton method for local optimization. Moreover, several multimodal benchmark examples illustrate that, the probability distribution property and search speed of chaotic sequences from different chaotic maps significantly affect the global searching capability and optimization efficiency of COA. To achieve the high efficiency of COA, it is recommended to adopt the appropriate chaotic map generating the desired chaotic sequences with uniform or nearly uniform probability distribution and large Lyapunov exponent.