Stochastic Gradient Descent Method Research Articles

Variance-Reduced Accelerated First-Order Methods: Central Limit Theorems and Confidence Statements

In this paper, we consider a strongly convex stochastic optimization problem and propose three classes of variable sample-size stochastic first-order methods: (i) the standard stochastic gradient descent method, (ii) its accelerated variant, and (iii) the stochastic heavy-ball method. In each scheme, the exact gradients are approximated by averaging across an increasing batch size of sampled gradients. We prove that when the sample size increases at a geometric rate, the generated estimates converge in mean to the optimal solution at an analogous geometric rate for schemes (i)–(iii). Based on this result, we provide central limit statements, whereby it is shown that the rescaled estimation errors converge in distribution to a normal distribution with the associated covariance matrix dependent on the Hessian matrix, the covariance of the gradient noise, and the step length. If the sample size increases at a polynomial rate, we show that the estimation errors decay at a corresponding polynomial rate and establish the associated central limit theorems (CLTs). Under certain conditions, we discuss how both the algorithms and the associated limit theorems may be extended to constrained and nonsmooth regimes. Finally, we provide an avenue to construct confidence regions for the optimal solution based on the established CLTs and test the theoretical findings on a stochastic parameter estimation problem. Funding: This work was supported by the Office of Naval Research [Grant N00014-22-1-2589] and Basic Energy Sciences [Grant DE-SC0023303]. The work of J. Lei was partially supported by the National Natural Science Foundation of China [Grants 72271187 and 62088101]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/moor.2021.0068 .

Imaging extended single crystal lattice distortion fields with multi-peak Bragg ptychography

Recent advances in phase-retrieval-based x-ray imaging methods have demonstrated the ability to reconstruct 3D distortion vector fields within a nanocrystal by using coherent diffraction information from multiple crystal Bragg reflections. However, these works do not provide a solution to the challenges encountered in imaging lattice distortions in crystals with significant defect content that result in phase wrapping. Moreover, these methods only apply to isolated crystals smaller than the x-ray illumination, and therefore cannot be used for imaging of distortions in extended crystals. We introduce multi-peak Bragg ptychography which addresses both challenges via an optimization framework that combines stochastic gradient descent and phase unwrapping methods for robust image reconstruction of lattice distortions and defects in extended crystals. Our work uses modern automatic differentiation toolsets so that the method is easy to extend to other settings and easy to implement in high-performance computers. This work is particularly timely given the broad interest in using the increased coherent flux in fourth-generation synchrotrons for innovative material research.

An Optimized Uncertainty-Aware Training Framework for Neural Networks.

Uncertainty quantification (UQ) for predictions generated by neural networks (NNs) is of vital importance in safety-critical applications. An ideal model is supposed to generate low uncertainty for correct predictions and high uncertainty for incorrect predictions. The main focus of state-of-the-art training algorithms is to optimize the NN parameters to improve the accuracy-related metrics. Training based on uncertainty metrics has been fully ignored or overlooked in the literature. This article introduces a novel uncertainty-aware training algorithm for classification tasks. A novel predictive uncertainty estimate-based objective function is defined and optimized using the stochastic gradient descent method. This new multiobjective loss function covers both accuracy and uncertainty accuracy (UA) simultaneously during training. The performance of the proposed training framework is compared from different aspects with other UQ techniques for different benchmarks. The obtained results demonstrate the effectiveness of the proposed framework for developing the NN models capable of generating reliable uncertainty estimates.

Efficient mini-batch stochastic gradient descent with Centroidal Voronoi Tessellation for PDE-constrained optimization under uncertainty

The study of optimal control problems under uncertainty plays an important role in scientific numerical simulations. This class of optimization problems is frequently utilized in engineering, biology and finance. Although stochastic gradient descent is a well-known stochastic optimization technique for solving the approximate optimal control problem, it often exhibits a slow convergence rate due to the inherent variance in gradient approximation. To address this issue, we propose a more efficient stochastic optimization algorithm based on Centroidal Voronoi Tessellation sampling. This strategy reduces the error in gradient estimation compared to the conventional mini-batch stochastic gradient descent method. Our approach involves dividing the whole snapshot set into several Voronoi cells with low variance and extracting samples with good uniformity from each region to construct an unbiased estimation of the full gradient. Our method can economically and stably approximate numerical optimal control functions even with a fixed step size. Numerical results demonstrate that the proposed method is a reliable algorithm with great potential for applications in the field of optimization.

Real-time Welding Machine Status Recognition using Embedded Perceptron Color Classification Model

This paper aims to utilize the perceptron model in machine learning to achieve precise classification of the working status of welding machines by analyzing image data. We collected static images of welding and non-welding operations, extracted the R, G, B color values of the images, and labeled them accordingly (+1 for welding status, -1 for idle status). By employing the stochastic gradient descent method for training the perceptron model, we obtained the weights w and bias b of the model under the condition that all data points in the training set are correctly classified. The model's recognition accuracy was validated on the test set. Ultimately, we applied this model to the STM32f103zet6 microcontroller chip and successfully achieved real-time recognition of the welding machine's operational status during the welding process via a camera. Through testing, the classification accuracy reached 83.6%. Further optimization of classification criteria can be combined with relevant expert suggestions, providing a reference solution for monitoring and identifying the working status of welding machines in steel pipe production.

A Nonparametric Learning Algorithm for a Stochastic Multi-echelon Inventory Problem

We consider a periodic-review single-product multi-echelon inventory problem with instantaneous replenishment. In each period, the decision-maker makes ordering decisions for all echelons. Any unsatisfied demand is back-ordered, and any excess inventory is carried to the next period. In contrast to the classic inventory literature, we assume that the information of the demand distribution is not known a priori, and the decision-maker observes demand realizations over the planning horizon. We propose a nonparametric algorithm that generates a sequence of adaptive ordering decisions based on the stochastic gradient descent method. We compare the [Formula: see text]-period cost of our algorithm to the clairvoyant, who knows the underlying demand distribution in advance, and we prove that the expected [Formula: see text]-period regret is at most [Formula: see text], matching a lower bound for this problem.

Parallel Fractional Stochastic Gradient Descent With Adaptive Learning for Recommender Systems

The structural change toward the digital transformation of online sales elevates the importance of parallel processing techniques in recommender systems, particularly in the pandemic and post-pandemic era. Matrix factorization (MF) is a popular and scalable approach in collaborative filtering (CF) to predict user preferences in recommender systems. Researchers apply Stochastic Gradient Descent (SGD) as one of the most famous optimization techniques for MF. Paralleling SGD methods help address Big Data challenges due to the wide range of products and the sparsity in user ratings. However, these methods’ convergence rate and accuracy are affected by the dependency between the user and item latent factors, specifically in large-scale problems. Besides, the performance is sensitive to the applied learning rates. This paper proposes a new parallel method to remove dependencies and boost speed-up by using fractional calculus to improve accuracy and convergence rate. We also apply adaptive learning rates to enhance the performance of our proposed method. The proposed method is based on Compute Unified Device Architecture (CUDA) platform. We evaluate the performance of our proposed method using real-world data and compare the results with the close baselines. The results show that our method can obtain high accuracy and convergence rate in addition to high parallelism.

Feature-Based Inventory Control with Censored Demand

Problem definition: We study stochastic periodic-review inventory systems with lost sales, where the decision maker has no access to the true demand distribution a priori and can only observe historical sales data (referred to as censored demand) and feature information about the demand. In an inventory system, excess demand is unobservable because of inventory constraints, and sales data alone cannot fully recover the true demand. Meanwhile, feature information about the demand is abundant to assist inventory decisions. We incorporate features for inventory systems with censored demand. Methodology/results: We propose two feature-based inventory algorithms called the feature-based adaptive inventory algorithm and the dynamic shrinkage algorithm. Both algorithms are based on the stochastic gradient descent method. We measure the performance of the proposed algorithms through the average expected regret in finite periods: that is, the difference between the cost of our algorithms and that of a clairvoyant optimal policy with access to information, which is acting optimally. We show that the average expected cost incurred under both algorithms converges to the clairvoyant optimal cost at the rate of [Formula: see text] for the perishable inventory case and [Formula: see text] for the nonperishable inventory case. The feature-based adaptive inventory algorithm results in high volatility in the stochastic gradients, which hampers the initial performance of regret. The dynamic shrinkage algorithm uses a shrinkage parameter to adjust the gradients, which significantly improves the initial performance. Managerial implications: This paper considers feature information. The idea of dynamic shrinkage for the stochastic gradient descent method builds on a fundamental insight known as the bias-variance trade-off. Our research shows the importance of incorporating the bias-variance in a dynamic environment for inventory systems with feature information. Funding: W. T. Huh acknowledges support from the NSERC Discovery Grants [Grant RGPIN 2020-04213] and the Canada Research Chair Program. The work of Y. Rong was supported by the National Natural Science Foundation of China [Grants 72025201, 72331006, and 72221001]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/msom.2021.0135 .

Enhanced Support Vector Machine Methods Using Stochastic Gradient Descent and Its Application to Heart Disease Dataset

Support Vector Machines (SVMs) are supervised learning models used to examine data sets in order to classify or predict dependent variables. SVM is typically used for classification by determining the best hyperplane between two classes. However, working with huge datasets can lead to a number of problems, including time-consuming and inefficient solutions. This research updates the SVM by employing a stochastic gradient descent method. The new approach, the extended stochastic gradient descent SVM (ESGD-SVM), was tested on two simulation datasets. The proposed method was compared with other classification approaches such as logistic regression, naive model, K Nearest Neighbors and Random Forest. The results show that the ESGD-SVM has a very high accuracy and is quite robust. ESGD-SVM is used to analyze the heart disease dataset downloaded from Harvard Dataverse. The entire analysis was performed using the program R version 4.3.

Design control and management of intelligent and autonomous nanorobots with artificial intelligence for Prevention and monitoring of blood related diseases

The effective management of human bloodstream remains to be the prime focus for the clinicians over years and it impose greater challenges when it comes to real-time solution. In particular managing hypoxemia and detection of blood clots is most pertinent. One major challenge faced is the existence of limited training data generated from real-world scenarios. On the other hand, creating an efficient model is often time consuming and expensive. This paper focusses on effective convergence of artificial intelligence and nanorobotics technologies to design and implement autonomous intelligent nanorobots to deal with blood related diseases. The major contribution of the research is two-fold, first we define an efficient architecture of the nanorobotics system with appropriate design parameter. Next, we develop a novel semi-supervised learning model using stochastic gradient descent method and kernel space that efficiently control and manage the nanorobots and helps in earlier prognosis and treatment of blood related diseases. The proposed model is novel and efficient as it enables working at nanoscale, providing resourceful understanding on physical and chemical properties associated with human body. The use of artificial intelligence techniques further makes the system to work more intelligently and independently. COSMOL with integrated MATLAB environment is used for experimental setup and simulation. MNIST dataset is compared with online RP tree method and other conventional batch related techniques. The performance analysis is compared based on performance, error rates and risk related factors. The proposed approach provides significant improvement in terms of performance with minimal error rate and improved accuracy measures.

Exploration of the application of optimisation algorithm using stochastic gradient descent method in satellite resource allocation

Exploration of the application of optimisation algorithm using stochastic gradient descent method in satellite resource allocation

Selective Memory Recursive Least Squares: Recast Forgetting Into Memory in RBF Neural Network-Based Real-Time Learning.

In radial basis function neural network (RBFNN)-based real-time learning tasks, forgetting mechanisms are widely used such that the neural network can keep its sensitivity to new data. However, with forgetting mechanisms, some useful knowledge will get lost simply because they are learned a long time ago, which we refer to as the passive knowledge forgetting phenomenon. To address this problem, this article proposes a real-time training method named selective memory recursive least squares (SMRLS) in which the classical forgetting mechanisms are recast into a memory mechanism. Different from the forgetting mechanism, which mainly evaluates the importance of samples according to the time when samples are collected, the memory mechanism evaluates the importance of samples through both temporal and spatial distribution of samples. With SMRLS, the input space of the RBFNN is evenly divided into a finite number of partitions, and a synthesized objective function is developed using synthesized samples from each partition. In addition to the current approximation error, the neural network also updates its weights according to the recorded data from the partition being visited. Compared with classical training methods including the forgetting factor recursive least squares (FFRLS) and stochastic gradient descent (SGD) methods, SMRLS achieves improved learning speed and generalization capability, which are demonstrated by corresponding simulation results.

A Loss Differentiation Method Based on Heterogeneous Ensemble Learning Model for Low Earth Orbit Satellite Networks.

In light of the high bit error rate in satellite network links, the traditional transmission control protocol (TCP) fails to distinguish between congestion and wireless losses, and existing loss differentiation methods lack heterogeneous ensemble learning models, especially feature selection for loss differentiation, individual classifier selection methods, effective ensemble strategies, etc. A loss differentiation method based on heterogeneous ensemble learning (LDM-HEL) for low-Earth-orbit (LEO) satellite networks is proposed. This method utilizes the Relief and mutual information algorithms for selecting loss differentiation features and employs the least-squares support vector machine, decision tree, logistic regression, and K-nearest neighbor as individual learners. An ensemble strategy is designed using the stochastic gradient descent method to optimize the weights of individual learners. Simulation results demonstrate that the proposed LDM-HEL achieves higher accuracy rate, recall rate, and F1-score in the simulation scenario, and significantly improves throughput performance when applied to TCP. Compared with the integrated model LDM-satellite, the above indexes can be improved by 4.37%, 4.55%, 4.87%, and 9.28%, respectively.

Data Augmentation for Bayesian Deep Learning

Deep Learning (DL) methods have emerged as one of the most powerful tools for functional approximation and prediction. While the representation properties of DL have been well studied, uncertainty quantification remains challenging and largely unexplored. Data augmentation techniques are a natural approach to provide uncertainty quantification and to incorporate stochastic Monte Carlo search into stochastic gradient descent (SGD) methods. The purpose of our paper is to show that training DL architectures with data augmentation leads to efficiency gains. We use the theory of scale mixtures of normals to derive data augmentation strategies for deep learning. This allows variants of the expectation-maximization and MCMC algorithms to be brought to bear on these high dimensional nonlinear deep learning models. To demonstrate our methodology, we develop data augmentation algorithms for a variety of commonly used activation functions: logit, ReLU, leaky ReLU and SVM. Our methodology is compared to traditional stochastic gradient descent with back-propagation. Our optimization procedure leads to a version of iteratively re-weighted least squares and can be implemented at scale with accelerated linear algebra methods providing substantial improvement in speed. We illustrate our methodology on a number of standard datasets. Finally, we conclude with directions for future research.

Continuous‐time stochastic gradient descent for optimizing over the stationary distribution of stochastic differential equations

AbstractWe develop a new continuous‐time stochastic gradient descent method for optimizing over the stationary distribution of stochastic differential equation (SDE) models. The algorithm continuously updates the SDE model's parameters using an estimate for the gradient of the stationary distribution. The gradient estimate is simultaneously updated using forward propagation of the SDE state derivatives, asymptotically converging to the direction of steepest descent. We rigorously prove convergence of the online forward propagation algorithm for linear SDE models (i.e., the multidimensional Ornstein–Uhlenbeck process) and present its numerical results for nonlinear examples. The proof requires analysis of the fluctuations of the parameter evolution around the direction of steepest descent. Bounds on the fluctuations are challenging to obtain due to the online nature of the algorithm (e.g., the stationary distribution will continuously change as the parameters change). We prove bounds for the solutions of a new class of Poisson partial differential equations (PDEs), which are then used to analyze the parameter fluctuations in the algorithm. Our algorithm is applicable to a range of mathematical finance applications involving statistical calibration of SDE models and stochastic optimal control for long time horizons where ergodicity of the data and stochastic process is a suitable modeling framework. Numerical examples explore these potential applications, including learning a neural network control for high‐dimensional optimal control of SDEs and training stochastic point process models of limit order book events.

Multi-objective Stochastic Gradient Based ADR Mechanism for Throughput and Latency Optimization in LoRaWAN

Background: In Long Range Wide Area Networks (LoRaWAN), the goal of Adaptive Data Rate (ADR) is to allocate resources to End Devices (ED) like Transmission Power (TP) and Spreading Factor (SF). The EDs are designed in a way that they can choose optimal configuration resource parameters from a set of LoRa physical layer parameters. The SF parameter has to be chosen correctly, as an incorrect one may cause collisions and interference if multiple nodes have the same SF. This paper focuses on throughput and latency optimization using an effective ADR mechanism for LoRaWAN-based IoT networks. Objective: The objective of this study is to maximize the total throughput. SF should be used by multiple nodes as it will have less Time on Air (ToA), but it may cause collision, contention, and co-spreading factor interference problems. The idea is to find an optimal SF allocation to end devices and the optimal number of total devices using the same SF to avoid collision and interference. Methods: This paper proposes a multi-objective stochastic gradient descent method to solve the constrained optimization problem for optimizing throughput and latency. Results: This work compares throughput and latency results for the static, quasi-static, and dynamic environments. Trade-offs between latency and throughput for the simulated scenarios are also presented. Conclusion: The simulation results show that the throughput obtained using this technique is higher than the naive ADR approach and the existing gradient descent methods.

Maximum Complex Correntropy Criterion Adaptive IIR Filtering Based on Gauss-Newton Approach

Adaptive infinite impulse response (IIR) filters have received noticeable consideration for its lower computational complexity compared with adaptive finite impulse response (FIR) filters recently. At present, the adaptive IIR algorithms are almost using the well-established stochastic gradient descent method, but the convergence rate is less than satisfactory. In this paper, to pursue the fast convergence speed of Gauss-Newton (GN) method, we present a GN algorithm for adaptive IIR filters with complex coefficients based on the complex correntropy measure. Besides, the steady-state excess mean square error is provided. Simulations are performed to show the eximious performance of this algorithm.

Learning to Learn Variational Quantum Algorithm.

Variational quantum algorithms (VQAs) use classical computers as the quantum outer loop optimizer and update the circuit parameters to obtain an approximate ground state. In this article, we present a meta-learning variational quantum algorithm (meta-VQA) by recurrent unit, which uses a technique called "meta-learner." Motivated by the hybrid quantum-classical algorithms, we train classical recurrent units to assist quantum computing, learning to find approximate optima in the parameter landscape. Here, aiming to reduce the sampling number more efficiently, we use the quantum stochastic gradient descent method and introduce the adaptive learning rate. Finally, we deploy on the TensorFlow Quantum processor within approximate quantum optimization for the Ising model and variational quantum eigensolver for molecular hydrogen (H2), lithium hydride (LiH), and helium hydride cation (HeH+). Our algorithm can be expanded to larger system sizes and problem instances, which have higher performance on near-term processors.

RP-SWSGD: Design of sliding window stochastic gradient descent method with user’s ratings pattern for recommender systems

RP-SWSGD: Design of sliding window stochastic gradient descent method with user’s ratings pattern for recommender systems

Powered stochastic optimization with hypergradient descent for large-scale learning systems

Stochastic optimization (SO) algorithms based on the Powerball function, namely powered stochastic optimization (PoweredSO) algorithms, have been confirmed, effectively, and demonstrated great potential in the context of large-scale optimization and machine learning tasks. Nevertheless, the issue of how to determine the learning rate for PoweredSO is a challenge and still unsolved problem. In this paper, we propose a class of adaptive PoweredSO approaches that are efficient, scalable and robust. It takes advantage of the hypergradient descent (HD) technique to automatically acquire an online learning rate for PoweredSO-like methods. In the first part, we study the behavior of the canonical PoweredSO algorithm, the Powerball stochastic gradient descent (pbSGD) method, with HD. The existing PoweredSO algorithms also suffer from the high variance because they take the similar algorithmic framework to SO algorithms, arising from sampling tactics. Therefore, the second portion develops an adaptive powered variance-reduced optimization method via utilizing both variance-reduced technique and HD. Moreover, we present the convergence analysis of the proposed algorithms and explore their iteration complexity on non-convex cases. Numerical experiments are conducted on machine learning tasks, verifying the superior performance over modern SO algorithms.