Stochastic quasi-Newton Method Research Articles

A single timescale stochastic quasi-Newton method for stochastic optimization

In this paper, we propose a single timescale stochastic quasi-Newton method for solving the stochastic optimization problems. The objective function of the problem is a composition of two smooth functions and their derivatives are not available. The algorithm sets to approximate sequences to estimate the gradient of the composite objective function and the inner function. The matrix correction parameters are given in BFGS update form for avoiding the assumption that Hessian matrix of objective is positive definite. We show the global convergence of the algorithm. The algorithm achieves the complexity to find an approximate stationary point and ensure that the expectation of the squared norm of the gradient is smaller than the given accuracy tolerance ϵ. The numerical results of nonconvex binary classification problem using the support vector machine and a multicall classification problem using neural networks are reported to show the effectiveness of the algorithm.

Hydropower Planning in Combination with Batteries and Solar Energy

Battery storage is an important factor for power systems made up of renewable energy sources. Technologies for battery storage are crucial to accelerating the transition from fossil fuels to renewable energy. Between responding to electricity demand and using renewable energy sources, battery storage devices will become increasingly important. The aim of this study is to examine how battery storage affects a power system consisting of solar and hydroelectric energy and to draw conclusions about whether energy storage recommends a power system. The method involves designing a model of eight real cascade hydropower power plants and solving an optimization problem. This power system model is based on existing hydroelectric power plants powered by solar energy and batteries in the Turkish cities of Yozgat and Tokat. A case study with four different battery capacities in the system was carried out to assess the implications of energy storage in the power system. The stochastic nonlinear optimization problem was modeled for 72 h and solved with the MATLAB programming tool. The stochastic Quasi-Newton method performs very well in hybrid renewable problems arising from large-scale machine learning. When solar energy and batteries were added to the system, the maximum installed wind power was found to be 2 MW and 3.6 MW, respectively. In terms of profit and hydropower planning, a medium-proportion battery was found to be the most suitable. Increased variability in hydropower generation results from the installation of an energy storage system.

Variance-Reduced Stochastic Quasi-Newton Methods for Decentralized Learning

In this work, we investigate stochastic quasi-Newton methods for minimizing a finite sum of cost functions over a decentralized network. In Part I, we develop a general algorithmic framework that incorporates stochastic quasi-Newton approximations with variance reduction so as to achieve fast convergence. At each time each node constructs a local, inexact quasi-Newton direction that asymptotically approaches the global, exact one. To be specific, (i) A local gradient approximation is constructed by using dynamic average consensus to track the average of variance-reduced local stochastic gradients over the entire network; (ii) A local Hessian inverse approximation is assumed to be positive definite with bounded eigenvalues, and how to construct it to satisfy these assumptions will be given in Part II. Compared to the existing decentralized stochastic first-order methods, the proposed general framework introduces the second-order curvature information without incurring extra sampling or communication. With a fixed step size, we establish the conditions under which the proposed general framework linearly converges to an exact optimal solution.

An Improved Technique for Pneumonia Infected Patients Image Recognition Based on Combination Algorithm of Smooth Generalized Pinball SVM and Variational Autoencoders

We present a method based on combining a smooth generalized pinball support vector machine (SVM) and variational autoencoders (VAEs) in chest X-ray (CXR) images. We incorporate generalized pinball into the SVM model to address the hinge loss function’s noise sensitivity and resampling instability. Since the generalized pinball loss is a non-differentiable function, the objective function is non-differentiable. To solve the non-differentiable term in the model’s objective function, we construct a smooth approximation function for the generalized pinball loss function so that the model in the original space can be solved directly. More specifically, using the proposed smoothing functions, we employ the smoothing stochastic quasi-Newton method to solve the primal problem. Moreover, we have introduced verified theorems related to our approaches, and the theoretical convergence of our approaches has been proposed. Numerical comparisons between parameter values of several iteration objective functions are also provided to test the model. Experiments with several benchmark datasets show that our method has the best prediction accuracy 64% and 57% from all of binary datasets and multiclass datasets, respectively. Moreover, our method has the best prediction of Matthews correlation coefficient (MCC) 64% from all of datasets.

AdaCN: An Adaptive Cubic Newton Method for Nonconvex Stochastic Optimization.

In this work, we introduce AdaCN, a novel adaptive cubic Newton method for nonconvex stochastic optimization. AdaCN dynamically captures the curvature of the loss landscape by diagonally approximated Hessian plus the norm of difference between previous two estimates. It only requires at most first order gradients and updates with linear complexity for both time and memory. In order to reduce the variance introduced by the stochastic nature of the problem, AdaCN hires the first and second moment to implement and exponential moving average on iteratively updated stochastic gradients and approximated stochastic Hessians, respectively. We validate AdaCN in extensive experiments, showing that it outperforms other stochastic first order methods (including SGD, Adam, and AdaBound) and stochastic quasi-Newton method (i.e., Apollo), in terms of both convergence speed and generalization performance.

Asynchronous parallel stochastic Quasi-Newton methods

Although first-order stochastic algorithms, such as stochastic gradient descent, have been the main force to scale up machine learning models, such as deep neural nets, the second-order quasi-Newton methods start to draw attention due to their effectiveness in dealing with ill-conditioned optimization problems. The L-BFGS method is one of the most widely used quasi-Newton methods. We propose an asynchronous parallel algorithm for stochastic quasi-Newton (AsySQN) method. Unlike prior attempts, which parallelize only the calculation for gradient or the two-loop recursion of L-BFGS, our algorithm is the first one that truly parallelizes L-BFGS with a convergence guarantee. Adopting the variance reduction technique, a prior stochastic L-BFGS, which has not been designed for parallel computing, reaches a linear convergence rate. We prove that our asynchronous parallel scheme maintains the same linear convergence rate but achieves significant speedup. Empirical evaluations in both simulations and benchmark datasets demonstrate the speedup in comparison with the non-parallel stochastic L-BFGS, as well as the better performance than first-order methods in solving ill-conditioned problems.

Stochastic Quasi-Newton Methods

Large-scale data science trains models for data sets containing massive numbers of samples. Training is often formulated as the solution of empirical risk minimization problems that are optimization programs whose complexity scales with the number of elements in the data set. Stochastic optimization methods overcome this challenge, but they come with their own set of limitations. This article discusses recent developments to accelerate the convergence of stochastic optimization through the exploitation of second-order information. This is achieved with stochastic variants of quasi-Newton methods that approximate the curvature of the objective function using stochastic gradient information. The reasons for why this leads to faster convergence are discussed along with the introduction of an incremental method that exploits memory to achieve a superlinear convergence rate. This is the best-known convergence rate for a stochastic optimization method. Stochastic quasi-Newton methods are applied to several problems, including prediction of the click-through rate of an advertisement displayed in response to a specific search engine query by a specific visitor. Experimental evaluations showcase reductions in overall computation time relative to stochastic gradient descent algorithms.

On Stochastic and Deterministic Quasi-Newton Methods for Nonstrongly Convex Optimization: Asymptotic Convergence and Rate Analysis

Motivated by applications arising from large-scale optimization and machine learning, we consider stochastic quasi-Newton (SQN) methods for solving unconstrained convex optimization problems. Much ...

Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization

In this paper we study stochastic quasi-Newton methods for nonconvex stochastic optimization, where we assume that noisy information about the gradients of the objective function is available via a stochastic first-order oracle ($\mathcal{SFO}$). We propose a general framework for such methods, for which we prove almost sure convergence to stationary points and analyze its worst-case iteration complexity. When a randomly chosen iterate is returned as the output of such an algorithm, we prove that in the worst case, the $\mathcal{SFO}$-calls complexity is $O(\epsilon^{-2})$ to ensure that the expectation of the squared norm of the gradient is smaller than the given accuracy tolerance $\epsilon$. We also propose a specific algorithm, namely, a stochastic damped limited-memory BFGS (SdLBFGS) method, that falls under the proposed framework. Moreover, we incorporate the stochastic variance reduced gradient variance reduction technique into the proposed SdLBFGS method and analyze its $\mathcal{SFO}$-calls complexity. Numerical results on a nonconvex binary classification problem using a support vector machine and a multiclass classification problem using neural networks are reported.

A Stochastic Quasi-Newton Method for Large-Scale Optimization

The question of how to incorporate curvature information into stochastic approximation methods is challenging. The direct application of classical quasi-Newton updating techniques for deterministic optimization leads to noisy curvature estimates that have harmful effects on the robustness of the iteration. In this paper, we propose a stochastic quasi-Newton method that is efficient, robust, and scalable. It employs the classical BFGS update formula in its limited memory form, and is based on the observation that it is beneficial to collect curvature information pointwise, and at spaced intervals. One way to do this is through (subsampled) Hessian-vector products. This technique differs from the classical approach that would compute differences of gradients at every iteration, and where controlling the quality of the curvature estimates can be difficult. We present numerical results on problems arising in machine learning that suggest that the proposed method shows much promise.

Stochastic quasi-Newton method: Application to minimal model for proteins

Knowledge of protein folding pathways and inherent structures is of utmost importance for our understanding of biological function, including the rational design of drugs and future treatments against protein misfolds. Computational approaches have now reached the stage where they can assess folding properties and provide data that is complementary to or even inaccessible by experimental imaging techniques. Minimal models of proteins, which make possible the simulation of protein folding dynamics by (systematic) coarse graining, have provided understanding in terms of descriptors for folding, folding kinetics, and folded states. Here we focus on the efficiency of equilibration on the coarse-grained level. In particular, we applied a new regularized stochastic quasi-Newton (S-QN) method, developed for accelerated configurational space sampling while maintaining thermodynamic consistency, to analyze the folding pathway and inherent structures of a selected protein, where regularization was introduced to improve stability. The adaptive compound mobility matrix B in S-QN, determined by a factorized secant update, gives rise to an automated scaling of all modes in the protein, in particular an acceleration of protein domain dynamics or principal modes and a slowing down of fast modes or "soft" bond constraints, similar to lincs/shake algorithms, when compared to conventional Langevin dynamics. We used and analyzed a two-step strategy. Owing to the enhanced sampling properties of S-QN and increased barrier crossing at high temperatures (in reduced units), a hierarchy of inherent protein structures is first efficiently determined by applying S-QN for a single initial structure and T=1>T(θ), where T(θ) is the collapse temperature. Second, S-QN simulations for several initial structures at very low temperature (T=0.01<T(F), where T(F) is the folding temperature), when the protein is known to fold but conventional Langevin dynamics experiences critical slowing down, were applied to determine the protein domain dynamics (collective modes) toward folded states, including the native state. This general treatment is efficient and directly applicable to other coarse-grained proteins.

Stochastic quasi-Newton molecular simulations

We report a new and efficient factorized algorithm for the determination of the adaptive compound mobility matrix B in a stochastic quasi-Newton method (S-QN) that does not require additional potential evaluations. For one-dimensional and two-dimensional test systems, we previously showed that S-QN gives rise to efficient configurational space sampling with good thermodynamic consistency [C. D. Chau, G. J. A. Sevink, and J. G. E. M. Fraaije, J. Chem. Phys. 128, 244110 (2008)]. Potential applications of S-QN are quite ambitious, and include structure optimization, analysis of correlations and automated extraction of cooperative modes. However, the potential can only be fully exploited if the computational and memory requirements of the original algorithm are significantly reduced. In this paper, we consider a factorized mobility matrix B=JJ(T) and focus on the nontrivial fundamentals of an efficient algorithm for updating the noise multiplier J . The new algorithm requires O(n2) multiplications per time step instead of the O(n3) multiplications in the original scheme due to Choleski decomposition. In a recursive form, the update scheme circumvents matrix storage and enables limited-memory implementation, in the spirit of the well-known limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) method, allowing for a further reduction of the computational effort to O(n). We analyze in detail the performance of the factorized (FSU) and limited-memory (L-FSU) algorithms in terms of convergence and (multiscale) sampling, for an elementary but relevant system that involves multiple time and length scales. Finally, we use this analysis to formulate conditions for the simulation of the complex high-dimensional potential energy landscapes of interest.

A stochastic quasi-Newton method for simulation response optimization

Simulation response optimization has wide applications for management of systems that are so complicated that the performance can only be evaluated by using simulation. This paper modifies the quasi-Newton method used in deterministic optimization to suit the stochastic environment in simulation response optimization. The basic idea is to use the estimated subgradient calculated from different replications and a metric matrix updated from the Broyden–Fletcher–Goldfarb–Shanno (BFGS) formula to yield a quasi-Newton search direction. To avoid misjudging the minimal point, in both the line search and the quasi-Newton iterations, due to the stochastic nature, a t-test instead of a simple comparison of the mean responses is performed. It is proved that the resulting stochastic quasi-Newton algorithm is able to generate a sequence that converges to the optimal point, under certain conditions. Empirical results from a four-station queueing problem and an ( s, S) inventory problem indicate that this method is able to find the optimal solutions in a statistical sense. Moreover, this method is robust with respect to the number of replications conducted at each trial point.

Strong Consistency and Asymptotic Efficiency of Stochastic Quasi-Newton Methods — Theory and Applications

Abstract A stochastic approximation procedure for the solution of a nonlinear algebraic equation in several variables is generated on the basis of a quasi-Newton ideology. The Jacobian matrix is estimated by least squares. Strong consistency with 1-ϵ probability is shown by a nonstandard application of Ljung's scheme.