Limited Memory BFGS Algorithm Research Articles

A Stochastic Inertial Limited Memory BFGS Algorithm Based on the Mathematical Model of Rural Pipeline Network and its Application in Machine Learning

Abstract Stochastic algorithms are critical in addressing complex rural pipe networks and non-convex stochastic optimization problems. With the development of artificial intelligence, large-scale optimization problems that cannot be solved effectively by traditional optimization methods have emerged. Therefore, stochastic optimization algorithms are rapidly developing in the field of optimization. This paper introduces an inertial extrapolation stochastic BFGS (IESBFGS) algorithm, an innovative amalgamation of the inertial extrapolation technique and the finite memory quasi-Newton algorithm to solve nonconvex stochastic optimization problems. Firstly, the inertial extrapolation technique is employed to track the iteration point to the optimal x-value. Second, it is combined with a finite-memory proposed Newton algorithm thereby increasing the convergence speed. Then, the superiority of IESBFGS is verified by comparing the experimental performance with other better algorithms on machine learning SVM model and ERM model. Finally, it is shown that the algorithm offers good prospects for solving nonconvex problems.

Prediction of adsorption performance of ZIF-67 for malachite green based on artificial neural network using L-BFGS algorithm

Given the necessity and urgency in removing organic pollutants such as malachite green (MG) from the environment, it is vital to screen high-capacity adsorbents using artificial neural network (ANN) methods quickly and accurately. In this study, a series of ZIF-67 were synthesized, which adsorption properties for organic pollutants, especially MG, were systematically evaluated and determined as 241.720 mg g−1 (25 ℃, 2 h). The adsorption process was more consistent with pseudo-second-order kinetics and Langmuir adsorption isotherm, which correlation coefficients were 0.995 and 0.997, respectively. The chemisorption mechanism was considered to be π-π stacking interaction between imidazole and aromatic ring. Then, a Python-based neural network model using the Limited-memory BFGS algorithm was constructed by collecting the crucial structural parameters of ZIF-67 and the experimental data of batch adsorption. The model, optimized extensively, outperformed similar Matlab-based ANN with a coefficient of determination of 0.9882 and mean square error of 0.0009 in predicting ZIF-67 adsorption of MG. Furthermore, the model demonstrated a good generalization ability in the predictive training of other organic pollutants. In brief, ANN was successfully separated from the Matlab platform, providing a robust framework for high-precision prediction of organic pollutants and guiding the synthesis of adsorbents.

Inner product preconditioned trust-region methods for frequency-domain full waveform inversion

Full waveform inversion is a seismic imaging method which requires solving a large-scale minimization problem, typically through local optimization techniques. Most local optimization methods can basically be built up from two choices: the update direction and the strategy to control its length. In the context of full waveform inversion, this strategy is very often a line search. We here propose to use instead a trust-region method, in combination with non-standard inner products which act as preconditioners. More specifically, a line search and several trust-region variants of the steepest descent, the limited memory BFGS algorithm and the inexact Newton method are presented and compared. A strong emphasis is given to the inner product choice. For example, its link with preconditioning the update direction and its implication in the trust-region constraint are highlighted. A first numerical test is performed on a 2D synthetic model then a second configuration, containing two close reflectors, is studied. The latter configuration is known to be challenging because of multiple reflections. Based on these two case studies, the importance of an appropriate inner product choice is highlighted and the best trust-region method is selected and compared to the line search method. In particular we were able to demonstrate that using an appropriate inner product greatly improves the convergence of all the presented methods and that inexact Newton methods should be combined with trust-region methods to increase their convergence speed.

A conditional random field framework for language process in product review mining

The Part-Of-Speech tagging is widely used in the natural language process. There are many statistical approaches in this area. The most popular one is Hidden Markov Model. In this paper, an alternative approach, linear-chain Conditional Random Fields, is introduced. The Conditional Random Fields is a factor graph approach that can naturally incorporate arbitrary, non-independent features of the input without conditional independence among the features or distributional assumptions of inputs. This paper applied the Conditional Random Fields for the car review word Part-Of-Speech tagging and then the feature extraction, which can be used as an input to an opinion mining system. To reduce the computational time, we also proposed applying the Limited-memory BFGS algorithm to train the Conditional Random Fields. Furthermore, this paper evaluated the Conditional Random Fields and the classical graph approach using the car review dataset to demonstrate that the Conditional Random Fields have a more robust result with a smaller training dataset.

An Active Set Limited Memory BFGS Algorithm for Machine Learning

In this paper, a stochastic quasi-Newton algorithm for nonconvex stochastic optimization is presented. It is derived from a classical modified BFGS formula. The update formula can be extended to the framework of limited memory scheme. Numerical experiments on some problems in machine learning are given. The results show that the proposed algorithm has great prospects.

Fast non-uniform Fourier transform based regularization for sparse-view large-size CT reconstruction

<p style='text-indent:20px;'>Spare-view CT imaging is advantageous to decrease the radiation exposure, acquisition time and computational cost, but suffers from severe streak noise in reconstruction if the classical filter back projection method is employed. Although a few compressed sensing based algorithms have recently been proposed to remedy the insufficiency of projections and have achieved remarkable improvement in reconstruction quality, they face computational challenges for large-scale CT images (e.g., larger than 2000℅2000 pixels). In this paper, we present a fast non-uniform Fourier transform based reconstruction method, targeting at under-sampling high resolution Synchrotron-based micro-CT imaging. The proposed method manipulates the Fourier slice theorem to avoid the involvement of large-scale system matrices, and the reconstruction process is performed in the Fourier domain. With a total variation penalty term, the proposed method can be formulated into an unconstrained minimization problem, which is able to be efficiently solved by the limited-memory BFGS algorithm. Moreover, direct non-uniform Fourier transform is computationally costly, so the developed NUFFT algorithm is adopted to approximate it with little loss of quality. Numerical simulation is implemented to compare the proposed method with some other competing approaches, and then real data obtained from the Australia Synchrotron facility are tested to demonstrate the practical applications of the proposed approach. In short, the significance of the proposed approach includes (1) that it can handle high-resolution CT images with millions of pixels while several other contemporary methods fail; (2) that it can achieve much better reconstruction quality than other methods when the projections are insufficient.</p>

Application of sequential quadratic programming based on active set method in cleaner production

On the platform of general chemical process simulation software (it was named Optimization Engineer, OPEN), a general optimization algorithm for chemical process simulation is developed using C + + code. The algorithm is based on sequential quadratic programming (SQP). We adopt the activity set algorithm and the rotation axis algorithm to generate the activity set to solve the quadratic programming sub-problem. The active set method can simplify the number of constraints and speed up the calculation. At the same time, we used limited memory BFGS algorithm (L-BFGS) to simplify the solution of second derivative matrix. The special matrix storage mode of L-BFGS algorithm can save the storage space and speed up the computing efficiency. We use exact penalty function and traditional step-size rule in the algorithm. These two methods can ensure the convergence of the algorithm, a more correct search direction and suitable search step. The example shows that the advanced optimization function can meet the requirements of General Chemical Process Calculation. The number of iterations can reduce by about 6.0%. The computation time can reduce by about 6.5%. We combined this algorithm with chemical simulation technology to develop the optimization function of chemical engineering simulation. This optimization function can play an important role in the process optimization calculation aiming at energy saving and green production.

Data-driven multichannel poststack seismic impedance inversion via patch-ordering regularization

Seismic acoustic impedance inversion plays an important role in reservoir prediction. However, single-trace inversion methods often suffer from spatial discontinuities and instability due to poor-quality seismic records with spatially variable signal-to-noise ratios or missing traces. The specified hyperparameters for seismic inversion cannot be suitable to all seismic traces and subsurface structures. In addition, conventional multichannel inversion imposes lateral continuity with a prespecified mathematical model. However, the inversion results constrained with specified lateral regularization are inferior when the subsurface situations violate the hypothesis. A data-driven multichannel acoustic impedance inversion method with patch-ordering regularization is introduced, in which the spatial correlation of seismic reflection is used. The method decomposes the seismic profile into patches and constructs the patch-ordering matrix based on the similarity among seismic patches to record the impedance structural extension. So the patch-ordering matrix can record the spatial extension of the acoustic impedance. Then, a simple regularization with difference operators of varying weights can reduce the random noise presented in the inverted impedance profile, stabilize the inversion result, and enhance the spatial continuity of the layer extension. The objective function for multichannel poststack seismic impedance inversion can be constructed by integrating the observed seismic record and the spatial continuity in the form of patch-ordering regularization, and it can be solved effectively with the limited-memory BFGS algorithm. The synthetic and field data tests illustrate the improvement of accuracy and lateral continuity of inverted results with our method, compared to conventional model-based inversion results.

Limited memory BFGS algorithm for the matrix approximation problem in Frobenius norm

This paper proposes an L-BFGS algorithm based on the active set technique to solve the matrix approximation problem: given a symmetric matrix, find a nearest approximation matrix in the sense of Frobenius norm to make it satisfy some linear equalities, inequalities and a positive semidefinite constraint. The problem is a convex optimization problem whose dual problem is a nonlinear convex optimization problem with non-negative constraints. Under the Slater constraint qualification, it has zero duality gap with the dual problem. To handle large-scale dual problem, we make use of the active set technique to estimate the active constraints, and then the L-BFGS method is used to accelerate free variables. The global convergence of the proposed algorithm is established under certain conditions. Finally, we conduct some preliminary numerical experiments, and compare the L-BFGS method with the inexact smoothing Newton method, the projected BFGS method, the alternating direction method and the two-metric projection method based on the L-BFGS. The numerical results show that our algorithm has some advantages in terms of CPU time when a large number of linear constraints are involved.

A New Diagonal Quasi-Newton Updating Method With Scaled Forward Finite Differences Directional Derivative for Unconstrained Optimization

A new diagonal quasi-Newton updating algorithm for unconstrained optimization is presented. The elements of the diagonal matrix approximating the Hessian are determined as scaled forward finite differences directional derivatives of the components of the gradient. Under mild classical assumptions, the convergence of the algorithm is proved to be linear. Numerical experiments with 80 unconstrained optimization test problems, of different structures and complexities, as well as five applications from MINPACK-2 collection, prove that the suggested algorithm is more efficient and more robust than the quasi-Newton diagonal algorithm retaining only the diagonal elements of the BFGS update, than the weak quasi-Newton diagonal algorithm, than the quasi-Cauchy diagonal algorithm, than the diagonal approximation of the Hessian by the least-change secant updating strategy and minimizing the trace of the matrix, than the Cauchy with Oren and Luenberger scaling algorithm in its complementary form (i.e. the Barzilai-Borwein algorithm), than the steepest descent algorithm, and than the classical BFGS algorithm. However, our algorithm is inferior to the limited memory BFGS algorithm (L-BFGS).

A L-BFGS Based Learning Algorithm for Complex-Valued Feedforward Neural Networks

In this paper, a new learning algorithm is proposed for complex-valued feedforward neural networks (CVFNNs). The basic idea of this algorithm is that the descent directions of the cost function with respect to complex-valued parameters are calculated by limited-memory BFGS algorithm and the learning step is determined by Armijo line search method. Since the approximation of Hessian matrix is calculated by utilizing the information of the latest several iterations, the memory efficiency is improved. To keep away from the saturated ranges of activation functions, some gain parameters are adjusted together with weights and biases. Compared with some existing learning algorithms for CVFNNs, the convergence speed is faster and a deeper minima of the cost function can be reached by the developed algorithm. In addition, the effects of initial values of weights and biases on the efficiency and convergence speed of the learning algorithm are analyzed. The performance of the proposed algorithm is evaluated in comparison with some existing classifiers on a variety of benchmark classification problems. Experimental results show that better performance is achieved by our algorithm with relatively compact network structure.

Seismic waveform tomography with shot-encoding using a restarted L-BFGS algorithm.

In seismic waveform tomography, or full-waveform inversion (FWI), one effective strategy used to reduce the computational cost is shot-encoding, which encodes all shots randomly and sums them into one super shot to significantly reduce the number of wavefield simulations in the inversion. However, this process will induce instability in the iterative inversion regardless of whether it uses a robust limited-memory BFGS (L-BFGS) algorithm. The restarted L-BFGS algorithm proposed here is both stable and efficient. This breakthrough ensures, for the first time, the applicability of advanced FWI methods to three-dimensional seismic field data. In a standard L-BFGS algorithm, if the shot-encoding remains unchanged, it will generate a crosstalk effect between different shots. This crosstalk effect can only be suppressed by employing sufficient randomness in the shot-encoding. Therefore, the implementation of the L-BFGS algorithm is restarted at every segment. Each segment consists of a number of iterations; the first few iterations use an invariant encoding, while the remainder use random re-coding. This restarted L-BFGS algorithm balances the computational efficiency of shot-encoding, the convergence stability of the L-BFGS algorithm, and the inversion quality characteristic of random encoding in FWI.

Proximal Limited-Memory Quasi-Newton Methods for Scenario-based Stochastic Optimal Control

Stochastic optimal control problems are typically of rather large scale involving millions of decision variables, but possess a certain structure which can be exploited by first-order methods such as forward-backward splitting and the alternating direction method of multipliers (ADMM). In this paper, we use the forward-backward envelope, a real-valued continuously differentiable penalty function, to recast the dual of the original nonsmooth problem as an unconstrained problem which we solve via the limited-memory BFGS algorithm. We show that the proposed method leads to a significant improvement of the convergence rate without increasing much the computational cost per iteration.

A limited memory BFGS algorithm for non-convex minimization with applications in matrix largest eigenvalue problem

This study aims to present a limited memory BFGS algorithm to solve non-convex minimization problems, and then use it to find the largest eigenvalue of a real symmetric positive definite matrix. The proposed algorithm is based on the modified secant equation, which is used to the limited memory BFGS method without more storage or arithmetic operations. The proposed method uses an Armijo line search and converges to a critical point without convexity assumption on the objective function. More importantly, we do extensive experiments to compute the largest eigenvalue of the symmetric positive definite matrix of order up to 54,929 from the UF sparse matrix collection, and do performance comparisons with EIGS (a Matlab implementation for computing the first finite number of eigenvalues with largest magnitude). Although the proposed algorithm converges to a critical point, not a global minimum theoretically, the compared results demonstrate that it works well, and usually finds the largest eigenvalue of medium accuracy.

Quasi-Newton methods based on ordinary differential equation approach for unconstrained nonlinear optimization

In this paper, we propose a hybrid ODE-based quasi-Newton (QN) method for unconstrained optimization problems, which combines the idea of low-order implicit Runge–Kutta (RK) techniques for gradient systems with the QN type updates of the Jacobian matrix such as the symmetric rank-one (SR1) update. The main idea of this approach is to associate a QN matrix to approximate numerically the Jacobian matrix in the gradient system. Fundamentally this is a gradient system based on the first order optimality conditions of the optimization problem. To further extend the methods for solving large-scale problems, a feature incorporated to the proposed methods is that a limited memory setting for matrix–vector multiplications is used, thus avoiding the computational and storage issues when computing Jacobian information. Under suitable assumptions, global convergence of the proposed method is proved. Practical insights on the effectiveness of these approaches on a set of test functions are given by a numerical comparison with that of the limited memory BFGS algorithm (L-BFGS) and conjugate gradient algorithm (CG).

Restarted LBFGS Algorithm for Power Amplifier Predistortion

For the linearization of power amplifiers, a digital pre-distorter is proposed to compensate nonlinear distortion in this paper. The pre-distorter is realized by Volterra series, and a restarted limited memory BFGS(LBFGS) algorithm is proposed for training the nonlinear pre-distorter. Training of Volterra series can be switched between gradient descend algorithm and quasi Newton algorithm by setting a switching condition, and LBFGS algorithm can be used for high dimensional iterative matrix of BFGS algorithm. These make the new algorithm much more stable with a fast convergence rate. The pre-distorter is used for OFDM power amplifiers and it high efficiency is proved by simulation results. Moreover, the convergence rate of the proposed algorithm is greatly faster than that of the gradient algorithm.

Estimation of Open Boundary Conditions for an Internal Tidal Model with Adjoint Method: A Comparative Study on Optimization Methods

Based on an internal tidal model, the practical performances of the limited-memory BFGS (L-BFGS) method and two gradient descent (GD) methods (the normal one with Wolfe’s line search and the simplified one) are investigated computationally through a series of ideal experiments in which the open boundary conditions (OBCs) are inverted by assimilating the interior observations with the adjoint method. In the case that the observations closer to the unknown boundary are included for assimilation, the L-BFGS method performs the best. As compared with the simplified GD method, the normal one really uses less iteration to reach a satisfactory solution, but its advantage over the simplified one is much smaller than expected. In the case that only the observations that are further from the unknown boundary are assimilated, the simplified GD method performs the best instead, whereas the performances of the other two methods are not satisfactory. The advanced L-BFGS algorithm and Wolfe’s line search still need to be improved when applied to the practical cases. The simplified GD method, which is controllable and easy to implement, should be regarded seriously as a choice, especially when the classical advanced optimization techniques fail or perform poorly.

Limited memory BFGS method with backtracking for symmetric nonlinear equations

A limited memory BFGS algorithm is presented for solving symmetric nonlinear equations. The global convergence of the given method is established under mild conditions. Numerical results show that the limited memory BFGS method is more interesting than the normal BFGS method for the given problems.

An active set limited memory BFGS algorithm for bound constrained optimization

In this paper, an active set limited BFGS algorithm is proposed for bound constrained optimization. The global convergence will be established under some suitable conditions. Numerical results show that the given method is effective.

An active set limited memory BFGS algorithm for large-scale bound constrained optimization

An active set limited memory BFGS algorithm for large-scale bound constrained optimization is proposed. The active sets are estimated by an identification technique. The search direction is determined by a lower dimensional system of linear equations in free subspace. The implementations of the method on CUTE test problems are described, which show the efficiency of the proposed algorithm.