L-BFGS Research Articles

Gradient based invasive weed optimization algorithm for the training of deep neural network

Stacked Sparse Auto-Encoder (SSAE) is well known hierarchical deep neural networks for simulating the deep architecture of mammal brain. SSAE can be trained in a greedy layer-wise manner by using methods based on gradient such as Limited memory BFGS (LBFGS). However, methods based on gradient have many disadvantages. The main disadvantage is that they are sensitive to the initial value. In this paper, a meta-heuristic algorithm based on gradient, referred to GCIWOSS, is used to optimize the weights and biases of SSAE. Chaos strategy is firstly used to initial the population of IWO and then a new selection strategy is adopted with the purpose of improving the diversity of population and increasing the global exploration ability. The improved IWO is preparing for the following exploitation based on gradient to avoid falling into local optimal values. In the experiments, the proposed algorithm is proven to be effective in extracting features from different image datasets, compared with the LBFGS and several other feature learning models.

Compact modeling of metal-oxide TFTs based on artificial neural network and improved particle swarm optimization

The application of artificial neural network (ANN) can give a very accurate and fast model for semiconductor devices used in circuit simulations. In this paper, we have applied multi-layer perceptron (MLP) neural network based on limited memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) method to model the flexible metal-oxide thin-film transistors (TFTs). An improved particle swarm optimization (PSO) is employed to find suitable initial parameters for the ANN model, which consists of a centroid opposition-based learning algorithm and a mutation strategy based on Euclidean distance to enhance the searching ability further. This hybrid modeling routine can improve the accuracy of predictions of both the I–V and small signal parameters ( $$g_d$$ , $$g_m$$ , etc.) characteristics, which are in good agreement with experimental data and fully demonstrate the validity of the proposed model. Furthermore, the model is implemented into a simulator with Verilog-A. The circuit-level tests of TFT show that the ANN compact model with PSO enables accurate performance estimation of metal-oxide TFT circuits.

APPLICATION OF THE SPH EQUIVALENCE TECHNIQUE TO THE CABRI FULL CORE IN NON-FUNDAMENTAL MODE

We developed a SPH equivalence technique in non-fundamental mode condition between a CABRI full-core model solved with the method of characteristics (MOC) in 2D and a simplified full-core model solved with the simplified P3 (SP3) method, linear anisotropic sources and discretized with Raviart-Thomas finite elements over a pure Cartesian mesh. The MOC and SP3 calculations are performed with DRAGON5 and DONJON5 codes, respectively. A three-parameter database is generated by DRAGON5 and is interpolated in DONJON5 as a function of the core condition. An objective function is set as the root mean square (RMS) error (MOC-SP3 discrepancy) on absorption distribution and leakage rates defined over the macro-geometry in DONJON5. Our algorithm is a quasi-Newtonian gradient search based on the Limited memory Broyden-Fletcher-Goldfarb-Shanno (LBFGS) method. Numerical results are presented with Hafnium bars withdrawn or inserted.

Nonlinear Conjugate Gradient Methods for PDE Constrained Shape Optimization Based on Steklov--Poincaré-Type Metrics

Shape optimization based on shape calculus has received a lot of attention in recent years, particularly regarding the development, analysis, and modification of efficient optimization algorithms. In this paper we propose and investigate nonlinear conjugate gradient methods based on Steklov-Poincar\'e-type metrics for the solution of shape optimization problems constrained by partial differential equations. We embed these methods into a general algorithmic framework for gradient-based shape optimization methods and discuss the numerical discretization of the algorithms. We numerically compare the proposed nonlinear conjugate gradient methods to the already established gradient descent and limited memory BFGS methods for shape optimization on several benchmark problems. The results show that the proposed nonlinear conjugate gradient methods perform well in practice and that they are an efficient and attractive addition to already established gradient-based shape optimization algorithms.

Fast and robust algorithm for energy minimization of spin systems applied in an analysis of high temperature spin configurations in terms of skyrmion density

An algorithm for the minimization of the energy of magnetic systems is presented and applied to the analysis of thermal configurations of a ferromagnet to identify inherent structures, i.e. the nearest local energy minima, as a function of temperature. Over a rather narrow temperature interval, skyrmions appear and reach a high temperature limit for the skyrmion density. In addition, the performance of the algorithm is further demonstrated in a self-consistent field calculation of a skyrmion in an itinerant magnet. The algorithm is based on a geometric approach in which the curvature of the spherical domain is taken into account and as a result the length of the magnetic moments is preserved in every iteration. In the limit of infinitesimal rotations, the minimization path coincides with that obtained using damped spin dynamics while the use of limited-memory quasi-newton minimization algorithms, such as the limited-memory Broyden–Fletcher–Goldfarb–Shanno (LBFGS) algorithm, significantly accelerates the convergence.

A Stochastic Quasi-Newton Method for Large-Scale Nonconvex Optimization With Applications.

Ensuring the positive definiteness and avoiding ill conditioning of the Hessian update in the stochastic Broyden-Fletcher-Goldfarb-Shanno (BFGS) method are significant in solving nonconvex problems. This article proposes a novel stochastic version of a damped and regularized BFGS method for addressing the above problems. While the proposed regularized strategy helps to prevent the BFGS matrix from being close to singularity, the new damped parameter further ensures the positivity of the product of correction pairs. To alleviate the computational cost of the stochastic limited memory BFGS (LBFGS) updates and to improve its robustness, the curvature information is updated using the averaged iterate at spaced intervals. The effectiveness of the proposed method is evaluated through the logistic regression and Bayesian logistic regression problems in machine learning. Numerical experiments are conducted by using both synthetic data set and several real data sets. The results show that the proposed method generally outperforms the stochastic damped LBFGS (SdLBFGS) method. In particular, for problems with small sample sizes, our method has shown superior performance and is capable of mitigating ill-conditioned problems. Furthermore, our method is more robust to the variations of the batch size and memory size than the SdLBFGS method.

ML-descent: An optimization algorithm for full-waveform inversion using machine learning

Full-waveform inversion (FWI) is a nonlinear optimization problem, and a typical optimization algorithm such as the nonlinear conjugate gradient or limited-memory Broyden-Fletcher-Goldfarb-Shanno (LBFGS) would iteratively update the model mainly along the gradient-descent direction of the misfit function or a slight modification of it. Based on the concept of meta-learning, rather than using a hand-designed optimization algorithm, we have trained the machine (represented by a neural network) to learn an optimization algorithm, entitled the “ML-descent,” and apply it in FWI. Using a recurrent neural network (RNN), we use the gradient of the misfit function as the input, and the hidden states in the RNN incorporate the history information of the gradient similar to an LBFGS algorithm. However, unlike the fixed form of the LBFGS algorithm, the machine-learning (ML) version evolves in response to the gradient. The loss function for training is formulated as a weighted summation of the L2 norm of the data residuals in the original inverse problem. As with any well-defined nonlinear inverse problem, the optimization can be locally approximated by a linear convex problem; thus, to accelerate the training, we train the neural network by minimizing randomly generated quadratic functions instead of performing time-consuming FWIs. To further improve the accuracy and robustness, we use a variational autoencoder that projects and represents the model in latent space. We use the Marmousi and the overthrust examples to demonstrate that the ML-descent method shows faster convergence and outperforms conventional optimization algorithms. The energy in the deeper part of the models can be recovered by the ML-descent even when the pseudoinverse of the Hessian is not incorporated in the FWI update.

Virtual semi-active damping learning control for robot manipulators interacting with unknown environment

Position controllers are used for free motion, whereas force controllers are used for constrained motion of robotic manipulators. The hybrid controller switches between position and force control modes depending on whether the manipulator is in contact with the environment. To improve production efficiency, the velocity of contact between the manipulator and environment is not set to zero. However, the high impact force due to the nonzero contact velocity might damage the environment surface or manipulators. In this article, we propose a virtual semi-active damping learning method to suppress force overshoot without decreasing the contact velocity. Virtual semi-active damping is adjusted according to the manipulator position in force control. The limited-memory BFGS method is used to obtain the ideal impedance model for the unknown environment. By minimizing the defined cost function, we get the desired interaction performance. The correctness and effectiveness of the proposed method are verified by conducting simulations and experiments.

DoNOF: An open-source implementation of natural-orbital-functional-based methods for quantum chemistry

The natural orbital functional theory (NOFT) has emerged as an alternative formalism to both density functional (DF) and wavefunction methods. In NOFT, the electronic structure is described in terms of the natural orbitals (NOs) and their occupation numbers (ONs). The approximate NOFs have proven to be more accurate than those of the density for systems with a significant multiconfigurational character, on one side, and scale better with the number of basis functions than correlated wavefunction methods, on the other side. A challenging task in NOFT is to efficiently perform orbital optimization. In this article we present DoNOF, our open source implementation based on diagonalizations that allows to obtain the resulting orbitals automatically orthogonal. The one-particle reduced-density matrix (1RDM) of the ensemble of pure-spin states provides the proper description of spin multiplets. The capabilities of the code are tested on the water molecule, namely, geometry optimization, natural and canonical representations of molecular orbitals, ionization potential, and electric moments. In DoNOF, the electron-pair-based NOFs developed in our group PNOF5, PNOF7 and PNOF7s are implemented. These JKL-only NOFs take into account most non-dynamic effects plus intrapair-dynamic electron correlation, but lack a significant part of interpair-dynamic correlation. Correlation corrections are estimated by the single-reference NOF-MP2 method that simultaneously calculates static and dynamic electron correlations taking as a reference the Slater determinant formed with the NOs of a previous PNOF calculation. The NOF-MP2 method is used to analyze the potential energy surface (PES) and the binding energy for the symmetric dissociation of the water molecule, and compare it with accurate wavefunction-based methods.

Optimization Modeling and Simulating of the Stationary Wigner Inflow Boundary Value Problem

The stationary Wigner inflow boundary value problem (SWIBVP) is modeled as an optimization problem by using the idea of shooting method in this paper. To remove the singularity at $$v=0$$ , we consider a regularized SWIBVP, where a regularization constraint is considered along with the original SWIBVP, and a modified optimization problem is established for it. A shooting algorithm is proposed to solve the two optimization problems, involving the limited-memory BFGS (L-BFGS) algorithm as the optimization solver. Numerical results show that solving the optimization problems with respect to the SWIBVP with the shooting algorithm is as effective as solving the SWIBVP with Frensley’s numerical method (Frensley in Phys Rev B 36:1570–1580, 1987). Furthermore, the modified optimization problem gets rid of the singularity at $$v=0$$ , and preserves symmetry of the Wigner function, which implies the optimization modeling with respect to the regularized SWIBVP is successful.

Evolutionary full-waveform inversion

SUMMARY We present a new approach to full-waveform inversion (FWI) that enables the assimilation of data sets that expand over time without the need to reinvert all data. This evolutionary inversion rests on a reinterpretation of stochastic Limited-memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS), which randomly exploits redundancies to achieve convergence without ever considering the data set as a whole. Specifically for seismological applications, we consider a dynamic mini-batch stochastic L-BFGS, where the size of mini-batches adapts to the number of sources needed to approximate the complete gradient. As an illustration we present an evolutionary FWI for upper-mantle structure beneath Africa. Starting from a 1-D model and data recorded until 1995, we sequentially add contemporary data into an ongoing inversion, showing how (i) new events can be added without compromising convergence, (ii) a consistent measure of misfit can be maintained and (iii) the model evolves over times as a function of data coverage. Though applied retrospectively in this example, our method constitutes a possible approach to the continuous assimilation of seismic data volumes that often tend to grow exponentially.

Robust statistical phase-diversity method for high-accuracy wavefront sensing

Phase diversity phase retrieval (PDPR) has been a popular technique for quantitatively measuring wavefront errors of optical imaging systems by extracting the phase information from several designated intensity measurements. As the problem is inverse and non-convex in general, the accuracy and robustness of most such algorithms rely greatly on the initial conditions. In this work, we propose a new strategy that combines Limited-Memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) with the initial points generated by k-means clustering method and three various channels to improve the overall performance. Experimental results show that, for 500 different phase aberrations with root mean square (RMS) value bounded within [0.2λ, 0.3λ], the minimum, the maximum and the mean RMS residual errors reach 0.017λ, 0.066λ and 0.039λ, respectively, and 84.8% of the RMS residual errors are less than 0.05λ. We have further investigated and analyzed the proposed method in details to quantitatively demonstrate its performance: statistical results reveal that our proposed PDPR with k-means clustering enhanced method has excellent robustness in terms of initial points and other influential factors, and the accuracy can outperform its counterpart methods such as classic L-BFGS and modified BFGS.

Joint PP and PS Pre-stack Seismic Inversion for Stratified Models Based on the Propagator Matrix Forward Engine

Pre-stack seismic inversion of the P- and S-wave velocities and bulk density is important in seismic exploration for evaluating lithological units and fluid properties. Generally, this inversion is based on ray-tracing modeling, which introduces errors and requires substantial pre-processing for stratified models due to its oversimplified single-interface assumption. To overcome those problems, we propose a pre-stack inversion method, using wave-equation-based modeling as a forward engine. Most wave-equation-based pre-stack inversions are based on the reflectivity method and adopt nonlinear optimization algorithms, although accurate, but computationally expensive. Hence, we use a fast propagator matrix (PM) method valid for layered media. To improve the stability and accuracy, the PP data inversion is extended to joint PP and PS PM-based inversion (JPMI). A linear inversion scheme is adopted to reduce the computational cost, and the Frechet derivatives are computed accordingly. Moreover, to obtain an optimal model solution, the L-BFGS (Limited-memory Broyden–Fletcher–Goldfarb–Shanno) optimization algorithm and L-curve criterion, an adaptive regularization parameter acquisition method, are implemented. A posterior probability analysis shows that the method has a higher parameter sensitivity than the joint exact Zoeppritz-based inversion and gives better estimations than the single-data inversion. We discuss the effects of dataset weight, internal multi-reflections, time window setting, noise level and initial model by using model tests. Synthetic and real-data examples demonstrate that the algorithm is better than the single PP inversion in terms of stability and accuracy, especially for S-wave velocity and density estimations.

Training Variational Networks With Multidomain Simulations: Speed-of-Sound Image Reconstruction.

Speed-of-sound (SoS) has been shown as a potential biomarker for breast cancer imaging, successfully differentiating malignant tumors from benign ones. SoS images can be reconstructed from time-of-flight measurements from ultrasound images acquired using conventional handheld ultrasound transducers. Variational networks (VNs) have recently been shown to be a potential learning-based approach for optimizing inverse problems in image reconstruction. Despite earlier promising results, these methods, however, do not generalize well from simulated to acquired data, due to the domain shift. In this work, we present for the first time a VN solution for a pulse-echo SoS image reconstruction problem using diverging waves with conventional transducers and single-sided tissue access. This is made possible by incorporating simulations with varying complexity into training. We use loop unrolling of gradient descent with momentum, with an exponentially weighted loss of outputs at each unrolled iteration in order to regularize the training. We learn norms as activation functions regularized to have smooth forms for robustness to input distribution variations. We evaluate reconstruction quality on the ray-based and full-wave simulations as well as on the tissue-mimicking phantom data, in comparison with a classical iterative [limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS)] optimization of this image reconstruction problem. We show that the proposed regularization techniques combined with multisource domain training yield substantial improvements in the domain adaptation capabilities of VN, reducing the median root mean squared error (RMSE) by 54% on a wave-based simulation data set compared to the baseline VN. We also show that on data acquired from a tissue-mimicking breast phantom, the proposed VN provides improved reconstruction in 12 ms.

A Frequency-Domain Quasi-Newton-Based Biparameter Synchronous Imaging Scheme for Ground-Penetrating Radar With Applications in Full Waveform Inversion

Full waveform inversion (FWI) of ground-penetrating radar (GPR) data is becoming a promising technique to facilitate the interpretation of surface-GPR data and the mapping of the subsurface. However, more general FWIs still require a sufficient amount of RAM memory, and it is difficult to produce an accurate and representative reconstruction result due to a large amount of the Hessian matrix calculations and singular value decomposition (SVD). In this article, we developed a novel full-waveform approach of the limited-memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) algorithm for surface-GPR data that is based on a quasi-Newton framework and the total variation (TV) regularization. The proposed approach uses an L-BFGS algorithm and combines a scale-transformation regularization technique to mitigate the ill-posed problem of inversion, which can impose a biparameter preinformation constraint to ensure the stability of inversion, and adaptive regularization weights are applied to improve the convergence efficiency of inversion. To demonstrate the novelty and effectiveness of the proposed scheme, we tested our FWI algorithm using synthetic data and in-site field data. In the testing, we focus on analyzing the influence of different aspects of the FWI results, including different scale factors, regularization weights, inversion strategies, acquisition configurations, initial models, and the noisy data set. In particular, the FWI experiment is performed to demonstrate the applicability of the proposed algorithm. The results show that the proposed algorithm can effectively reconstruct the biparameter near the subsurface with high accuracy, which makes our approach very attractive for attribute analysis applications and makes the surface-GPR FWI commercially viable.

Three-dimensional magnetotelluric inversion using L-BFGS

The gradient-based optimization methods are preferable for the large-scale three-dimensional (3D) magnetotelluric (MT) inverse problem. Compared with the popular nonlinear conjugate gradient (NLCG) method, however, the limited-memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) method is less adopted. This paper aims to implement a L-BFGS-based inversion algorithm for the 3D MT problem. And we develop our code on top of the ModEM package, which is highly extensible and popular among the MT community. To accelerate the convergence speed, the preconditioning technique by the affine linear transformation of the original model parameters is used. Two modifications of the conventional L-BFGS algorithm are also made to get a comparable convergence rate with the NLCG method. The impacts of the preconditioner parameters, the regularization parameters, the starting model, etc., on the inversion are evaluated by synthetic examples for both L-BFGS and NLCG methods. And the real MT Kayabe dataset is also inverted by the inversion algorithms. The synthetic tests show that through our L-BFGS inversion algorithm the similar resistivity models can be obtained with that from the NLCG method. For the real data inversion, the L-BFGS method performs more efficiently and reasonable results could be obtained by less iterations of the inversion process than the NLCG method. Thus, we suggest the common usage of the L-BFGS method for the 3D MT inverse problem.

Efficient optimization method for finding minimum energy paths of magnetic transitions

Efficient algorithms for the calculation of minimum energy paths of magnetic transitions are implemented within the geodesic nudged elastic band (GNEB) approach. While an objective function is not available for GNEB and a traditional line search can, therefore, not be performed, the use of limited memory Broyden–Fletcher–Goldfarb–Shanno (LBFGS) and conjugate gradient algorithms in conjunction with orthogonal spin optimization (OSO) approach is shown to greatly outperform the previously used velocity projection and dissipative Landau–Lifschitz dynamics optimization methods. The implementation makes use of energy weighted springs for the distribution of the discretization points along the path and this is found to improve performance significantly. The various methods are applied to several test problems using a Heisenberg-type Hamiltonian, extended in some cases to include Dzyaloshinskii–Moriya and exchange interactions beyond nearest neighbours. Minimum energy paths are found for magnetization reversals in a nano-island, collapse of skyrmions in two-dimensional layers and annihilation of a chiral bobber near the surface of a three-dimensional magnet. The LBFGS-OSO method is found to outperform the dynamics based approaches by up to a factor of 8 in some cases.

A proximal ADMM with the Broyden family for convex optimization problems

Alternating direction methods of multipliers (ADMM) have been well studied and effectively used in various application fields. The classical ADMM must solve two subproblems exactly at each iteration. To overcome the difficulty of computing the exact solution of the subproblems, some proximal terms are added to the subproblems. Recently, {{a special proximal ADMM has been studied}} whose regularized matrix in the proximal term is generated by the BFGS update (or limited memory BFGS) at every iteration for a structured quadratic optimization problem. {{The numerical experiments also showed}} that the numbers of iterations were almost same as those by the exact ADMM. In this paper, we propose such a proximal ADMM for more general convex optimization problems, and extend the proximal term by the Broyden family update. We also show the convergence of the proposed method under standard assumptions.

A limited memory BFGS subspace algorithm for bound constrained nonsmooth problems

The subspace technique has been widely used to solve unconstrained/constrained optimization problems and there exist many results that have been obtained. In this paper, a subspace algorithm combining with limited memory BFGS update is proposed for large-scale nonsmooth optimization problems with box-constrained conditions. This algorithm can ensure that all iteration points are feasible and the sequence of objective functions is decreasing. Moreover, rapid changes in the active set are allowed. The global convergence is established under some suitable conditions. Numerical results show that this method is very effective for large-scale nonsmooth box-constrained optimization, where the largest dimension of the test problems is 11,000 variables.

Hybrid modelling routine for metal‐oxide TFTs based on particle swarm optimisation and artificial neural network

An effective and robust hybrid algorithm consisting of particle swarm optimisation (PSO) and limited memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) method based on artificial neural network (ANN) is proposed for modelling flexible metal-oxide thin-film transistors (TFTs). The L-BFGS method as an optimiser is exploited to update the parameters of ANN and speed up the training process. A mutation strategy for PSO is derived to enhance the searching ability further. With the great global searching ability, PSO is implemented to find a hopeful initial position in solution space for the next ANN model. The simulation result shows a high accuracy not only in I–V curve fitting but also in small-signal parameter ( g m , g d , etc.) predictions, which have not been exposed in the training process. The measured DC characteristics of In–Zn–O TFTs are used to verify the proposed ANN model, which has the benefits of rapid fitting from the L-BFGS algorithm and universal searching ability from PSO.