Stochastic Gradient Descent Approach Research Articles

Optimization by linear kinetic equations and mean-field Langevin dynamics

One of the most striking examples of the close connections between global optimization processes and statistical physics is the simulated annealing method, inspired by the famous Monte Carlo algorithm devised by Metropolis et al. in the middle of last century. In this paper, we show how the tools of linear kinetic theory allow the description of this gradient-free algorithm from the perspective of statistical physics and how convergence to the global minimum can be related to classical entropy inequalities. This analysis highlights the strong link between linear Boltzmann equations and stochastic optimization methods governed by Markov processes. Thanks to this formalism, we can establish the connections between the simulated annealing process and the corresponding mean-field Langevin dynamics characterized by a stochastic gradient descent approach. Generalizations to other selection strategies in simulated annealing that avoid the acceptance–rejection dynamic are also provided.

A metaheuristic-optimization-based neural network for icing prediction on transmission lines

Ice accretion on overhead transmission line systems is a leading cause of power outages and can lead to substantial economic losses in northern regions. Therefore, accurately and rapidly predicting ice accretion on power lines is crucial for ensuring the safe operation of the power grid. This study introduces a machine learning method for predicting the ice-to-liquid ratio (ILR), an important parameter for assessing ice accretion efficiency. While estimating ILR is vital for operational forecasting, many existing ice accretion models do not include this capability. A feedforward neural network (FFNN) trained with stochastic gradient descent and various metaheuristic optimizers - specifically particle swarm optimization, grey wolf optimizer, whale optimizer, and slime mold optimizer - is employed to forecast hourly ILR. Environmental data required for training and testing the FFNN model were obtained from the Automated Surface Observing System (ASOS). A global sensitivity analysis using the Sobol index, evaluated via the coefficients of a polynomial chaos expansion, was conducted to identify the most influential input parameters. The results indicate that only four input parameters significantly contribute to the variance in the response: precipitation, temperature, dew point temperature, and wind speed. Furthermore, the FFNN model trained with metaheuristic optimizers outperformed the stochastic gradient descent approach. With the predicted ILR, ice accumulation can be easily calculated as the product of ILR and the amount of liquid precipitation depth.

LSTM network optimization and task network construction based on heuristic algorithm

This work aims to advance the security management of complex networks to better align with evolving societal needs. The work employs the Ant Colony Optimization algorithm in conjunction with Long Short-Term Memory neural networks to reconstruct and optimize task networks derived from time series data. Additionally, a trend-based noise smoothing scheme is introduced to mitigate data noise effectively. The approach entails a thorough analysis of historical data, followed by applying trend-based noise smoothing, rendering the processed data more scientifically robust. Subsequently, the network reconstruction problem for time series data originating from one-dimensional dynamic equations is addressed using an algorithm based on the principles of Stochastic Gradient Descent (SGD). This algorithm decomposes time series data into smaller samples and yields optimal learning outcomes in conjunction with an adaptive learning rate SGD approach. Experimental results corroborate the remarkable fidelity of the weight matrix reconstructed by this algorithm to the true weight matrix. Moreover, the algorithm exhibits efficient convergence with increasing data volume, manifesting shorter time requirements per iteration while ensuring the attainment of optimal solutions. When the sample size remains constant, the algorithm’s execution time is directly proportional to the square of the number of nodes. Conversely, as the sample size scales, the SGD algorithm capitalizes on the availability of more information, resulting in improved learning outcomes. Notably, when the noise standard deviation is 0.01, models predicated on SGD and the Least-Squares Method (LSM) demonstrate reduced errors compared to instances with a noise standard deviation of 0.1, highlighting the sensitivity of LSM to noise. The proposed methodology offers valuable insights for advancing research in complex network studies.

Performance Comparison of LMS and RLS Algorithms for Ambient Noise Attenuation

The aim of this study is to implement two different types of adaptive algorithms for the noise cancellation. The study explores the well-known least mean squares (LMS) adaptive algorithm, which is based on stochastic gradient descent approach, and its performances in terms of noise attenuation level and swiftness in active noise control (ANC). Another algorithm is considered in this investigation based upon the use of the least squares estimation (LSE), commonly named, the recursive least squares algorithm (RLS), and will be compared to the LMS. In order to evaluate the potential of each one, a few simulations are achieved. The numerical experiments are performed by using several real recordings of different environment noises tested on the two proposed adaptive algorithms. A comparison is emphasized regarding noise suppression ability and convergence speed, by implementing both adaptive algorithms on the same noise sources. From this numerical study, the RLS algorithm reveals a faster convergence speed and better control performances than the LMS algorithm.

Federated quanvolutional neural network: a new paradigm for collaborative quantum learning

In recent years, the concept of federated machine learning has been actively driven by scientists to ease the privacy concerns of data owners. Currently, the combination of machine learning and quantum computing technologies is a hot industry topic and is positioned to be a major disruptor. It has become an effective new tool for reshaping several industries ranging from healthcare to finance. Data sharing poses a significant hurdle for large-scale machine learning in numerous industries. It is a natural goal to study the advanced quantum computing ecosystem, which will be comprised of heterogeneous federated resources. In this work, the problem of data governance and privacy is handled by developing a quantum federated learning approach, that can be efficiently executed on quantum hardware in the noisy intermediate-scale quantum era. We present the federated hybrid quantum–classical algorithm called a quanvolutional neural network with distributed training on different sites without exchanging data. The hybrid algorithm requires small quantum circuits to produce meaningful features for image classification tasks, which makes it ideal for near-term quantum computing. The primary goal of this work is to evaluate the potential benefits of hybrid quantum–classical and classical-quantum convolutional neural networks on non-independently and non-identically partitioned (Non-IID) and real-world data partitioned datasets among several healthcare institutions/clients. We investigated the performance of a collaborative quanvolutional neural network on two medical machine learning datasets, COVID-19 and MedNIST. Extensive experiments are carried out to validate the robustness and feasibility of the proposed quantum federated learning framework. Our findings demonstrate a decrease of 2%–39% times in necessary communication rounds compared to the federated stochastic gradient descent approach. The hybrid federated framework maintained a high classification testing accuracy and generalizability, even in scenarios where the medical data is unevenly distributed among clients.

Sigma-point and stochastic gradient descent approach to solving global self-optimizing controlled variables

Sigma-point and stochastic gradient descent approach to solving global self-optimizing controlled variables

Graph-Based Algorithm Unfolding for Energy-Aware Power Allocation in Wireless Networks

We develop a novel graph-based trainable framework to maximize the weighted sum energy efficiency (WSEE) for power allocation in wireless communication networks. To address the non-convex nature of the problem, the proposed method consists of modular structures inspired by a classical iterative suboptimal approach and enhanced with learnable components. More precisely, we propose a deep unfolding of the successive concave approximation (SCA) method. In our unfolded SCA (USCA) framework, the originally preset parameters are now learnable via graph convolutional neural networks (GCNs) that directly exploit multi-user channel state information as the underlying graph adjacency matrix. We show the permutation equivariance of the proposed architecture, which is a desirable property for models applied to wireless network data. The USCA framework is trained through a stochastic gradient descent approach using a progressive training strategy. The unsupervised loss is carefully devised to feature the monotonic property of the objective under maximum power constraints. Comprehensive numerical results demonstrate its generalizability across different network topologies of varying size, density, and channel distribution. Thorough comparisons illustrate the improved performance and robustness of USCA over state-of-the-art benchmarks.

Structure optimization with stochastic density functional theory.

Linear-scaling techniques for Kohn-Sham density functional theory are essential to describe the ground state properties of extended systems. Still, these techniques often rely on the localization of the density matrix or accurate embedding approaches, limiting their applicability. In contrast, stochastic density functional theory (sDFT) achieves linear- and sub-linear scaling by statistically sampling the ground state density without relying on embedding or imposing localization. In return, ground state observables, such as the forces on the nuclei, fluctuate in sDFT, making optimizing the nuclear structure a highly non-trivial problem. In this work, we combine the most recent noise-reduction schemes for sDFT with stochastic optimization algorithms to perform structure optimization within sDFT. We compare the performance of the stochastic gradient descent approach and its variations (stochastic gradient descent with momentum) with stochastic optimization techniques that rely on the Hessian, such as the stochastic Broyden-Fletcher-Goldfarb-Shanno algorithm. We further provide a detailed assessment of the computational efficiency and its dependence on the optimization parameters of each method for determining the ground state structure of bulk silicon with varying supercell dimensions.

Strategic Generation Investment in Energy Markets: A Multiparametric Programming Approach

An investor has to carefully select the location and size of new generation units it intends to build, since adding capacity in a market affects the profit from units this investor may already own. To capture this closed-loop characteristic, strategic investment (SI) of generation can be posed as a bilevel optimization. By analytically studying a small market, we first show that its objective function can be non-convex and discontinuous. Realizing that existing mixed-integer problem formulations become impractical for larger markets and number of instances, this work put forth two SI solvers: a grid search to handle setups where the candidate investment locations are few, and a stochastic gradient descent approach for otherwise. Both solvers leverage powerful results of multiparametric programming (MPP), each in a unique way. The grid search entails finding the primal/dual solutions for a large number of optimal power flow (OPF) problems, which nonetheless can be efficiently computed several at once thanks to the properties of MPP. The same properties facilitate the rapid calculation of gradients in a mini-batch fashion, thus accelerating the implementation of a stochastic (sub)-gradient descent search. Tests on the IEEE 30- and 118-bus systems using real-world data corroborate the advantages of the novel solvers.

Differentiable predictive control: Deep learning alternative to explicit model predictive control for unknown nonlinear systems

We present differentiable predictive control (DPC) as a deep learning-based alternative to the explicit model predictive control (MPC) for unknown nonlinear systems. In the DPC framework, a neural state-space model is learned from time-series measurements of the system dynamics. The neural control policy is then optimized via stochastic gradient descent approach by differentiating the MPC loss function through the closed-loop system dynamics model. The proposed DPC method learns model-based control policies with state and input constraints, while supporting time-varying references and constraints. In embedded implementation using a Raspberry-Pi platform, we experimentally demonstrate that it is possible to train constrained control policies purely based on the measurements of the unknown nonlinear system. We compare the control performance of the DPC method against explicit MPC and report efficiency gains in online computational demands, memory requirements, policy complexity, and construction time. In particular, we show that our method scales linearly compared to exponential scalability of the explicit MPC solved via multiparametric programming.

An efficient algorithm for data parallelism based on stochastic optimization

Deep neural network models can achieve greater performance in numerous machine learning tasks by raising the depth of the model and the amount of training data samples. However, these essential procedures will proportionally raise the cost of training deep neural network models. Accelerating the training process of deep neural network models in a distributed computing environment has become the most often utilized strategy for developers in order to better cope with a huge quantity of training overhead. The current deep neural network model is the stochastic gradient descent (SGD) technique. It is one of the most widely used training techniques in network models, although it is prone to gradient obsolescence during parallelization, which impacts the overall convergence. The majority of present solutions are geared at high-performance nodes with minor performance changes. Few studies have taken into account the cluster environment in high-performance computing (HPC), where the performance of each node varies substantially. A dynamic batch size stochastic gradient descent approach based on performance-aware technology is suggested to address the aforesaid difficulties (DBS-SGD). By assessing the processing capacity of each node, this method dynamically allocates the minibatch of each node, guaranteeing that the update time of each iteration between nodes is essentially the same, lowering the average gradient of the node. The suggested approach may successfully solve the asynchronous update strategy’s gradient outdated problem. The Mnist and cifar10 are two widely used image classification benchmarks, that are employed as training data sets, and the approach is compared with the asynchronous stochastic gradient descent (ASGD) technique. The experimental findings demonstrate that the proposed algorithm has better performance as compared with existing algorithms.

A stochastic gradient descent approach with partitioned-truncated singular value decomposition for large-scale inverse problems of magnetic modulus data

We propose a stochastic gradient descent approach with partitioned-truncated singular value decomposition (SVD) for large-scale inverse problems of magnetic modulus data. Motivated by a uniqueness theorem in gravity inverse problem and realizing the similarity between gravity and magnetic inverse problems, we propose to solve the level-set function modeling the volume susceptibility distribution from the nonlinear magnetic modulus data. To deal with large-scale data, we employ a mini-batch stochastic gradient descent approach with random reshuffling when solving the optimization problem of the inverse problem. We propose a stepsize rule for the stochastic gradient descent according to the Courant–Friedrichs–Lewy condition of the evolution equation. In addition, we develop a partitioned-truncated SVD algorithm for the linear part of the inverse problem in the context of stochastic gradient descent. Numerical examples illustrate the efficacy of the proposed method, which turns out to have the capability of efficiently processing large-scale measurement data for the magnetic inverse problem. A possible generalization to the inverse problem of deep neural network is discussed at the end.

Efficient Parameter Estimation of Truncated Boolean Product Distributions

We study the problem of estimating the parameters of a Boolean product distribution in d dimensions, when the samples are truncated by a set \(S \subseteq \{0, 1\}^d\) accessible through a membership oracle. This is the first time that the computational and statistical complexity of learning from truncated samples is considered in a discrete setting. We introduce a natural notion of fatness of the truncation set S, under which truncated samples reveal enough information about the true distribution. We show that if the truncation set is sufficiently fat, samples from the true distribution can be generated from truncated samples. A stunning consequence is that virtually any statistical task (e.g., learning in total variation distance, parameter estimation, uniformity or identity testing) that can be performed efficiently for Boolean product distributions, can also be performed from truncated samples, with a small increase in sample complexity. We generalize our approach to ranking distributions over d alternatives, where we show how fatness implies efficient parameter estimation of Mallows models from truncated samples. Exploring the limits of learning discrete models from truncated samples, we identify three natural conditions that are necessary for efficient identifiability: (i) the truncation set S should be rich enough; (ii) S should be accessible through membership queries; and (iii) the truncation by S should leave enough randomness in all directions. By carefully adapting the Stochastic Gradient Descent approach of (Daskalakis et al., FOCS 2018), we show that these conditions are also sufficient for efficient learning of truncated Boolean product distributions.

Decentralized Federated Learning With Unreliable Communications

Decentralized federated learning, inherited from decentralized learning, enables the edge devices to collaborate on model training in a peer-to-peer manner without the assistance of a server. However, existing decentralized learning frameworks usually assume perfect communication among devices, where they can reliably exchange messages, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , gradients or parameters. But the real-world communication networks are prone to packet loss and transmission errors. Transmission reliability comes with a price. The commonly-used solution is to adopt a reliable transportation layer protocol, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , transmission control protocol (TCP), which however leads to significant communication overhead and reduces connectivity among devices that can be supported. For a communication network with a lightweight and unreliable communication protocol, user datagram protocol (UDP), we propose a robust decentralized stochastic gradient descent (SGD) approach, called Soft-DSGD, to address the unreliability issue. Soft-DSGD updates the model parameters with partially received messages and optimizes the mixing weights according to the link reliability matrix of communication links. We prove that the proposed decentralized training system, even with unreliable communications, can still achieve the same asymptotic convergence rate as vanilla decentralized SGD with perfect communications. Moreover, numerical results confirm the proposed approach can leverage all available unreliable communication links to speed up convergence.

Identification Method for Cone Yarn Based on the Improved Faster R-CNN Model

To solve the problems of high labor intensity, low efficiency, and frequent errors in the manual identification of cone yarn types, in this study five kinds of cone yarn were taken as the research objects, and an identification method for cone yarn based on the improved Faster R-CNN model was proposed. In total, 2750 images were collected of cone yarn samples in real of textile industry environments, then data enhancement was performed after marking the targets. The ResNet50 model with strong representation ability was used as the feature network to replace the VGG16 backbone network in the original Faster R-CNN model to extract the features of the cone yarn dataset. Training was performed with a stochastic gradient descent approach to obtain an optimally weighted file to predict the categories of cone yarn. Using the same training samples and environmental settings, we compared the method proposed in this paper with two mainstream target detection algorithms, YOLOv3 + DarkNet-53 and Faster R-CNN + VGG16. The results showed that the Faster R-CNN + ResNet50 algorithm had the highest mean average precision rate for the five types of cone yarn at 99.95%, as compared with the YOLOv3 + DarkNet-53 algorithm with a mean average precision rate that was 2.24% higher and the Faster R-CNN + VGG16 algorithm with a mean average precision that was 1.19% higher. Regarding cone yarn defects, shielding, and wear, the Faster R-CNN + ResNet50 algorithm can correctly identify these issues without misdetection occurring, with an average precision rate greater than 99.91%.

Reliability-based topology optimization using stochastic gradients

This paper addresses the computational challenges in reliability-based topology optimization (RBTO) of structures associated with the estimation of statistics of the objective and constraints using standard sampling methods. The aim is to overcome the accuracy issues of traditional methods that rely on approximating the limit-state function. Herein, we present a stochastic gradient-based approach, where we estimate the probability of failure at every few optimization iterations using an efficient sampling strategy. To estimate the gradients of the failure probability with respect to the design parameters, we apply Bayes’ rule wherein we assume a parametric exponential model for the probability density function of the design parameters conditioned on the failure. The design parameters and the parameters of this probability density function are updated using a stochastic gradient descent approach requiring only a small, e.g., $${\mathcal {O}}(1)$$ , number of random samples per iteration, thus, leading to considerable reduction of the computational cost as compared to standard RBTO techniques. We illustrate the proposed approach with a benchmark example that has an analytical solution as well as two widely used problems in structural topology optimization. These examples illustrate the efficacy of the approach in producing reliable designs.

Gradient-Based Training of Gaussian Mixture Models for High-Dimensional Streaming Data

We present an approach for efficiently training Gaussian Mixture Model (GMM) by Stochastic Gradient Descent (SGD) with non-stationary, high-dimensional streaming data. Our training scheme does not require data-driven parameter initialization (e.g., k-means) and can thus be trained based on a random initial state. Furthermore, the approach allows mini-batch sizes as low as 1, which are typical for streaming-data settings. Major problems in such settings are undesirable local optima during early training phases and numerical instabilities due to high data dimensionalities. We introduce an adaptive annealing procedure to address the first problem, whereas numerical instabilities are eliminated by an exponential-free approximation to the standard GMM log-likelihood. Experiments on a variety of visual and non-visual benchmarks show that our SGD approach can be trained completely without, for instance, k-means based centroid initialization. It also compares favorably to an online variant of Expectation-Maximization (EM)—stochastic EM (sEM), which it outperforms by a large margin for very high-dimensional data.

SSGD: A Safe and Efficient Method of Gradient Descent

With the vigorous development of artificial intelligence technology, various engineering technology applications have been implemented one after another. The gradient descent method plays an important role in solving various optimization problems, due to its simple structure, good stability, and easy implementation. However, in multinode machine learning system, the gradients usually need to be shared, which will cause privacy leakage, because attackers can infer training data with the gradient information. In this paper, to prevent gradient leakage while keeping the accuracy of the model, we propose the super stochastic gradient descent approach to update parameters by concealing the modulus length of gradient vectors and converting it or them into a unit vector. Furthermore, we analyze the security of super stochastic gradient descent approach and demonstrate that our algorithm can defend against the attacks on the gradient. Experiment results show that our approach is obviously superior to prevalent gradient descent approaches in terms of accuracy, robustness, and adaptability to large-scale batches. Interestingly, our algorithm can also resist model poisoning attacks to a certain extent.

First-Order Newton-Type Estimator for Distributed Estimation and Inference

This article studies distributed estimation and inference for a general statistical problem with a convex loss that could be nondifferentiable. For the purpose of efficient computation, we restrict ourselves to stochastic first-order optimization, which enjoys low per-iteration complexity. To motivate the proposed method, we first investigate the theoretical properties of a straightforward divide-and-conquer stochastic gradient descent approach. Our theory shows that there is a restriction on the number of machines and this restriction becomes more stringent when the dimension p is large. To overcome this limitation, this article proposes a new multi-round distributed estimation procedure that approximates the Newton step only using stochastic subgradient. The key component in our method is the proposal of a computationally efficient estimator of , where is the population Hessian matrix and w is any given vector. Instead of estimating (or ) that usually requires the second-order differentiability of the loss, the proposed first-order Newton-type estimator (FONE) directly estimates the vector of interest as a whole and is applicable to nondifferentiable losses. Our estimator also facilitates the inference for the empirical risk minimizer. It turns out that the key term in the limiting covariance has the form of , which can be estimated by FONE.

An Improvement of Stochastic Gradient Descent Approach for Mean-Variance Portfolio Optimization Problem

In this paper, the current variant technique of the stochastic gradient descent (SGD) approach, namely, the adaptive moment estimation (Adam) approach, is improved by adding the standard error in the updating rule. The aim is to fasten the convergence rate of the Adam algorithm. This improvement is termed as Adam with standard error (AdamSE) algorithm. On the other hand, the mean-variance portfolio optimization model is formulated from the historical data of the rate of return of the S&amp;P 500 stock, 10-year Treasury bond, and money market. The application of SGD, Adam, adaptive moment estimation with maximum (AdaMax), Nesterov-accelerated adaptive moment estimation (Nadam), AMSGrad, and AdamSE algorithms to solve the mean-variance portfolio optimization problem is further investigated. During the calculation procedure, the iterative solution converges to the optimal portfolio solution. It is noticed that the AdamSE algorithm has the smallest iteration number. The results show that the rate of convergence of the Adam algorithm is significantly enhanced by using the AdamSE algorithm. In conclusion, the efficiency of the improved Adam algorithm using the standard error has been expressed. Furthermore, the applicability of SGD, Adam, AdaMax, Nadam, AMSGrad, and AdamSE algorithms in solving the mean-variance portfolio optimization problem is validated.