BFGS Research Articles

Convergence Analysis of an Improved BFGS Method and Its Application in the Muskingum Model

The BFGS method is one of the most effective quasi-Newton algorithms for minimization-optimization problems. In this paper, an improved BFGS method with a modified weak Wolfe–Powell line search technique is used to solve convex minimization problems and its convergence analysis is established. Seventy-four academic test problems and the Muskingum model are implemented in the numerical experiment. The numerical results show that our algorithm is comparable to the usual BFGS algorithm in terms of the number of iterations and the time consumed, which indicates our algorithm is effective and reliable.

Using nonlinear functions to approximate a new quasi-Newton method for unconstrained optimization problems

In order to get a higher order accuracy of approximating the Hessian matrix of the objective function, we use the chain rule and propose two modified secant equations. An interesting property of the proposed methods is that these utilize information from two most recent steps where the usual secant equation uses only the latest step information. The other point of interest to one of the proposed methods is that it makes use of both gradient and function value information. We show that the modified BFGS methods based on the new secant equations are globally convergent. The presented experimental results illustrate that the proposed methods are efficient.

Computing the matrix geometric mean: Riemannian versus Euclidean conditioning, implementation techniques, and a Riemannian BFGS method

SummaryThis paper addresses the problem of computing the Riemannian center of mass of a collection of symmetric positive definite matrices. We show in detail that the condition number of the Riemannian Hessian of the underlying optimization problem is never very ill conditioned in practice, which explains why the Riemannian steepest descent approach has been observed to perform well. We also show theoretically and empirically that this property is not shared by the Euclidean Hessian. We then present a limited‐memory Riemannian BFGS method to handle this computational task. We also provide methods to produce efficient numerical representations of geometric objects that are required for Riemannian optimization methods on the manifold of symmetric positive definite matrices. Through empirical results and a computational complexity analysis, we demonstrate the robust behavior of the limited‐memory Riemannian BFGS method and the efficiency of our implementation when compared to state‐of‐the‐art algorithms.

A new quasi-newton equation on the gradient methods for optimization minimization problem

<p><span>The quasi-Newton equation is the very foundation of an assortment of </span><span>the quasi-Newton methods for optimization minimization problem. In this paper, we deriving a new quasi-Newton equation based on the second-order Taylor’s series expansion. The global convergence is established underneath suitable conditions and numerical results are reported to show that the given algorithm is more effective than those of the normal BFGS method.</span></p>

Structure, electronic and magnetic properties of 2D Graphene-Molybdenum diSulphide (G-MoS2) Heterostructure (HS) with vacancy defects at Mo sites

First-principles calculations have been performed to study Structure, Electronic and Magnetic properties of 2D Graphene-Molybdenum diSulphide G-MoS2 heterostructure (HS) materials with vacancy defects at Mo sites. All the calculations have been performed within the framework of the first-principles Density Functional Theory (DFT) under the Perdew-Burke-Ernzerhof (PBE) form of the Generalized Gradient Approximation (GGA). The ven der Waals (vdW) interactions have been taken into account (considered via Grimme’s model) by DFT-D2 approach. The G-MoS2 heterostructure without Mo sites defects and with Mo sites defects have been optimized using BFGS methods implemented in Quantum ESPRESSO. Our investigations show that the defects materials are more compact than the pure material. Further, we have studied the Electronic and Magnetic properties of pure and defects G-MoS2 heterostructures from the band structure calculations and DOS-PDOS analysis. The results show that the pure and defects heterostructures are metallic in nature and they can be useful for low power consumption electronic devices. From the band structures calculation, it is seen that Fermi energy level shifted toward the valence band in single Mo atom and two Mo atoms vacancy defects both the cases means p-type Schottky defects. The analysis of the magnetic properties show that the nonmagnetic pure G-MoS2 heterostructure changes to magnetic materials by Mo sites defects. It is also found that total magnetism values of C-1Mo, L-1Mo and 2Mo atoms vacancy defects have 2.61 (Bohr-Mag/cell), 2.65 (Bohr-Mag/cell) and 5.25 (Bohr-Mag/cell) respectively and 2p orbital of C atoms, 3p orbital of S atoms and 4d orbital of Mo atoms have main contributions to the magnetism in all three configurations.

Advanced orthogonal moth flame optimization with Broyden–Fletcher–Goldfarb–Shanno algorithm: Framework and real-world problems

As a typical emergent swarm intelligence algorithm, Moth-Flame Optimization (MFO) has been created to deal with global optimization problems. Since the introduction, it has been applied to various optimization problems. However, MFO may have the trouble of getting into the local best, and the convergence rate cannot be satisfying when handling the high-dimensional and some multimodal problems. In this work, an enhanced MFO integrated with orthogonal learning (OL) and Broyden-Fletcher-Goldfarb-Shanno (BFGS), which we called BFGSOLMFO, is proposed to alleviate the stagnation shortcomings and accelerate the performance of well-regarded MFO. In the BFGSOLMFO, OL is used to construct a better candidate solution for each moth and then guide the whole population to a reasonable potential area. Meanwhile, in each iteration, after the evolution of population finished and the global optima are obtainable, BFGS is employed to further excavate the potential of the global best moth in the current population. With the aim of evaluating the efficacy of the BFGSOLMFO, first of all, the IEEE CEC2014 benchmark set is utilized to measure the performance in solving function optimizations with high-dimensional and multimodal characteristics. Both sets of the IEEE CEC2011 real-world benchmark problems and the three constrained engineering optimization problems are adopted to estimate the performance of BFGSOLMFO in tackling practical scenarios. In all the experiments, the developed BFGSOLMFO is compared with state-of-the-art advanced algorithms. Experimental results and statistical tests demonstrate that the proposed method outperforms the basic MFO and a comprehensive set of advanced algorithms.

A double parameter self-scaling memoryless BFGS method for unconstrained optimization

A double parameter self-scaling memoryless BFGS method for unconstrained optimization is presented. In this method, the first two terms of the self-scaling memoryless BFGS matrix are scaled with a positive parameter, while the third one is scaled with another positive parameter. The first parameter scaling the first two terms is determined to cluster the eigenvalues of the memoryless BFGS matrix. The second parameter scaling the third term is computed as a preconditioner to the Hessian of the minimizing function combined with the minimization of the conjugacy condition to shift the large eigenvalues of the self-scaling memoryless BFGS matrix to the left. The stepsize is determined by the Wolfe line search conditions. The global convergence of this method is proved, assuming that the minimizing function is uniformly convex. The preliminary computational experiments on a set of 80 unconstrained optimization test functions show that this algorithm is more efficient and more robust than the self-scaling BFGS updates by Oren and Luenberger and by Oren and Spedicato. Subject to the CPU time metric, CG-DESCENT is top performer. Comparisons with L-BFGS show that our algorithm is more efficient.

Hybrid reconstruction method for multispectral bioluminescence tomography with log-sum regularization.

Bioluminescence tomography (BLT) has important applications in the in vivo visualization of a pathological process for preclinical studies. However, the reconstruction of BLT is severely ill-posed. To recover the bioluminescence source stably and efficiently, we use a log-sum regularization term in the objective function and utilize a hybrid optimization algorithm for solving the nonconvex regularized problems (HONOR). The hybrid optimization scheme of HONOR merges second-order information and first-order information to reconstruction by choosing either the quasi-Newton (QN) or gradient descent step at each iteration. The QN step uses the limited-memory Broyden-Fletcher-Goldfarb-Shanno algorithm (L-BFGS) to acquire second-order information. Simulations and in vivo experiments based on multispectral measurements demonstrated the remarkable performance of the proposed hybrid method in the sparse reconstruction of BLT.

Diagonal Approximation of the Hessian by Finite Differences for Unconstrained Optimization

A new quasi-Newton method with a diagonal updating matrix is suggested, where the diagonal elements are determined by forward or by central finite differences. The search direction is a direction of sufficient descent. The algorithm is equipped with an acceleration scheme. The convergence of the algorithm is linear. The preliminary computational experiments use a set of 75 unconstrained optimization test problems classified in five groups according to the structure of their Hessian: diagonal, block-diagonal, band (tri- or penta-diagonal), sparse and dense. Subject to the CPU time metric, intensive numerical experiments show that, for problems with Hessian in a diagonal, block-diagonal or band structure, the algorithm with diagonal approximation of the Hessian by finite differences is top performer versus the well-established algorithms: the steepest descent and the Broyden–Fletcher–Goldfarb–Shanno. On the other hand, as a by-product of this numerical study, we show that the Broyden–Fletcher–Goldfarb–Shanno algorithm is faster for problems with sparse Hessian, followed by problems with dense Hessian.

In Silico Prediction of Critical Micelle Concentration (CMC) of Classic and Extended Anionic Surfactants from Their Molecular Structural Descriptors

CMC is an important parameter for the characterization of surfactants. Compared to other properties, the CMC can be correlated with surfactants performance characteristics on an industrial scale. In this investigation, QSPR models were established to identify the relation between the molecular structures and the critical micelle concentration (CMC) of 50 anionic surfactants employing four molecular structural descriptors. Three regression methods were chosen in this work to develop robust predictive models, namely multilayer perceptron–artificial neural network (MLP/ANN), multiple linear regressions, and partial least square approach. To establish the reliability and the robustness of the developed QSPR models, all available validation strategies were adopted. The best results $$ \left( {\overline{{r_{m}^{2} }} = 0.87;Q_{\text{LOO}}^{2} = 0.93;Q_{F1}^{2} = 0.95;\Delta r_{m}^{2} = 0.15} \right) $$ were obtained for MLP/ANN with a 4-3-1 artificial neural network model trained with the Broyden–Fletcher–Goldfarb–Shanno algorithm. In this study, it is observed that electronic properties, structure and size of the molecule, as well as the number of atoms in the longest aliphatic chain play major roles in the development of the CMC model of anionic surfactants.

DUPLICATE: Advanced Orthogonal Moth Flame Optimization with Broyden–Fletcher–Goldfarb–Shanno Algorithm: Framework and Real-world Problems

The Publisher regrets that this article is an accidental duplication of an article that has already been published, http://dx.doi.org/10.1016/j.endend.2014.06.001. The duplicate article has therefore been withdrawn. The full Elsevier Policy on Article Withdrawal can be found at http://www.elsevier.com/locate/withdrawalpolicy.

The time response of plasmonic sensors due to binary adsorption: analytical versus numerical modeling

In order to allow for multiscale modeling of complex systems, we focus on various approaches to modeling binary adsorption. We consider multiple methods of modeling the temporal response of general plasmonic sensors. We start from the analytical approach. The kinetics of adsorption and desorption is modeled both as a first order reaction and as a second-order reaction. The criteria for their validity and the choice between them in the case of two-component adsorption are established. Due to the nonlinearities of the second-order reactions and the lack of their analytical solutions, computer-aided modeling is considered next, also in multiple ways: the employment of numerical solvers, fitting of experimental results, the stochastic simulation algorithms and the employment of artificial neural networks (ANN). The examples we present illustrate the advantages and disadvantages of the particular approaches. The goal is to aid the concurrent multiscale modeling of adsorption-based devices. Machine learning in ANN performed here is used to estimate the equilibrium values of adsorbed quantities. The obtained results show that to train an ANN for the estimation of the equilibrium adsorption quantities the Levenberg–Marquardt and the Bayesian regularization algorithms are less efficient than the quasi-Newton BFGS (Broyden–Fletcher–Goldfarb–Shanno) algorithm.

Symmetrical Rank-Three Vectorized Loading Scores Quasi-Newton for Identification of Hydrogeological Parameters and Spatiotemporal Recharges

In a multi-layered groundwater model, achieving accurate spatiotemporal identification and solving the ill-posed problem is the vital topic for model calibration. This study proposes a symmetry rank three vectorized loading scores (SR3 VLS) quasi-Newton algorithm by modifying the Levenberg–Marquardt algorithm and importing a rank three structure from Broyden–Fletcher–Goldfarb–Shanno algorithm for identification of hydrogeological parameters and spatiotemporal recharge simultaneously. To accelerate directional convergence and approach a global optimum, this study uses a vectorized limited switchable step size in the transmissive groundwater inverse problem. The Hessian approximation rank three uses high and low-rank factor loading scores analyzed from simulated storage fluctuation between adjacent iterations for calculation and matrix correction. Two numerical experiments were designed to validate the proposing algorithm, showing the SR3 VLS quasi-Newton reduced the error percentages of the identified parameters by 1.63% and 9.65% compared to the Jacobian quasi-Newton. The proposing method is applied to the Chou-Shui River alluvial fan groundwater system in Taiwan. Results show that the simulated storage error decreased rapidly in six iterations, and has good head convergence as small as 0.11% with a root-mean-square-error (RMSE) of 0.134 m, indicating that the proposing algorithm reduces the computational cost to converge to the true solution.

Multi-label Log-Loss function using L-BFGS for document categorization

Text mining, which fundamentally involves quantitative tactics to analyze textual data, can be used for discovering knowledge and to achieve scholarly research goals. For large-scale data such as corpus text, intelligent learning methods have been effectively approached. In this paper, an artificial neural network with a quasi-Newton updating procedure is presented for multi-label multi-class text classification. This numerical unconstrained training technique, the Multi-Label extension of Log-Loss function using in Limited-memory Broyden–Fletcher–Goldfarb–Shanno algorithm (ML4BFGS), provides a noteworthy opportunity for text mining and leads to a significant improvement in text classification performances. The ML4BFGS training approach is applied to allocate some (one or multi) of the classes to each corresponding sentence from different available labels. We evaluate this method on English translations of the Holy Quran. These religious texts have been chosen for experiments of this manuscript because each verse (sentence) usually has multiple labels (topics) and different translations of each verse should have the same labels. Experimental results show that ML4BFGS is talented for multi-label multi-class classification in the Quranic corpus. Evaluation criteria of some advanced updating methods such as ITCG, BFGS, L-BFGS-B, L3BFGS as well as some other multi-label approaches such as ML-k-NN, and well-known SVM are compared with the proposed ML4BFGS and the outcomes are fully-described in this study. The performance measures including the Hamming loss, recall, precision, and F1 score show that the ML4BFGS achieves the best results in extracting related classes for each verse, while the proposed network takes the least epochs compared to the other training approach for completing learning or training phase. Simultaneously, the elapsed time for ML4BFGS is just 78% (in seconds) of the best experience of this term. Compared with the applicability of some state-of-the-art algorithms, ML4BFGS has a less computational cost, faster convergence rate, and much accuracy in corpus analysis.

A modified three-term conjugate gradient method for large –scale optimization

We propose a three-term conjugate gradient method in this paper . The basic idea is to exploit the good properties of the BFGS update. Quasi – Newton method lies a good efficient numerical computational, so we suggested to be based on BFGS method. However, the descent condition and the global convergent is proven under Wolfe condition. The new algorithm is very effective e for solving the large – scale unconstrained optimization problem.

A bilevel learning approach for optimal observation placement in variational data assimilation

In this paper we propose a bilevel optimization approach for the placement of space and time observations in variational data assimilation problems. Within the framework of supervised learning, we consider a bilevel problem where the lower-level task is the variational reconstruction of the initial condition of a semilinear system, and the upper-level problem solves the optimal placement with help of a sparsity inducing function. Due to the pointwise nature of the observations, an optimality system with regular Borel measures is obtained as necessary optimality condition for the lower-level problem. The latter is then considered as constraint for the upper-level instance, yielding an optimization problem constrained by a multi-state system with measures. We demonstrate existence of Lagrange multipliers and derive a necessary optimality system characterizing the optimal solutions of the bilevel problem. The numerical solution is carried out also on two levels. The lower-level problem is solved using a standard BFGS method, while the upper-level one is solved by means of a projected BFGS algorithm, based on the estimation of -active sets. Finally some numerical experiments are presented to illustrate the main features of our approach.

An advanced active set L-BFGS algorithm for training weight-constrained neural networks

In this work, a new advanced active set limited memory BFGS (Broyden–Fletcher–Goldfarb–Shanno) algorithm is proposed for efficiently training weight-constrained neural networks, called AA-L-BFGS. The proposed algorithm possesses the significant property of approximating the curvature of the error function with high-order accuracy by utilizing the theoretically advanced secant condition proposed by Livieris and Pintelas (Appl Math Comput 221:491–502, 2013). Moreover, the global convergence of the proposed algorithm is established provided that the line search satisfies the modified Armijo condition. The presented numerical experiments illustrate the efficiency of the proposed AA-L-BFGS, providing empirical evidence that it significantly accelerates the convergence of the training process.

Supervised learning in Hamiltonian reconstruction from local measurements on eigenstates

Reconstructing a system Hamiltonian through measurements on its eigenstates is an important inverse problem in quantum physics. Recently, it was shown that generic many-body local Hamiltonians can be recovered by local measurements without knowing the values of the correlation functions. In this work, we discuss this problem in more depth for different systems and apply supervised learning method via neural networks to solve it. For low-lying eigenstates, the inverse problem is well-posed, neural networks turn out to be efficient and scalable even with a shallow network and a small data set. For middle-lying eigenstates, the problem is ill-posed, we present a modified method based on transfer learning accordingly. Neural networks can also efficiently generate appropriate initial points for numerical optimization based on the BFGS method.

Limited memory BFGS algorithm for the matrix approximation problem in Frobenius norm

This paper proposes an L-BFGS algorithm based on the active set technique to solve the matrix approximation problem: given a symmetric matrix, find a nearest approximation matrix in the sense of Frobenius norm to make it satisfy some linear equalities, inequalities and a positive semidefinite constraint. The problem is a convex optimization problem whose dual problem is a nonlinear convex optimization problem with non-negative constraints. Under the Slater constraint qualification, it has zero duality gap with the dual problem. To handle large-scale dual problem, we make use of the active set technique to estimate the active constraints, and then the L-BFGS method is used to accelerate free variables. The global convergence of the proposed algorithm is established under certain conditions. Finally, we conduct some preliminary numerical experiments, and compare the L-BFGS method with the inexact smoothing Newton method, the projected BFGS method, the alternating direction method and the two-metric projection method based on the L-BFGS. The numerical results show that our algorithm has some advantages in terms of CPU time when a large number of linear constraints are involved.

Predictive modeling and response analysis of spent catalyst bioleaching using artificial neural network

Bioleaching of spent catalyst using acidophiles usually leads to lower Mo extraction than other metals. To address this problem, Mo leaching has been examined using alkaline chemicals or other microorganisms. In this study, experimental data of Mo bioleaching from spent catalyst using Escherichia coli was collected and modeled with Artificial Neural Network (ANN). Variation in metal extraction with change in particle size, pulp density, temperature, and residence time, was examined using response modeling. ANN was trained and optimized using Limited-memory Broyden–Fletcher–Goldfarb–Shanno algorithm for varying hidden layers and different activation functions. The training was done with tenfold cross-validation to avoid overfitting on the training data. The best network configuration was identified using Coefficient of Determination (R2 score) and evaluated on a test dataset. A network with logistic activation function and hidden layer configuration with two layers was found most effective with an R2 score of 0.988.