Stochastic Gradient Algorithm Research Articles

Convergence of the Distributed SG Algorithm Under Cooperative Excitation Condition.

In this article, a distributed stochastic gradient (SG) algorithm is proposed where the estimators are aimed to collectively estimate an unknown time-invariant parameter from a set of noisy measurements obtained by distributed sensors. The proposed distributed SG algorithm combines the consensus strategy of the estimation of neighbors with the diffusion of regression vectors. For the theoretical investigation of the proposed algorithm, the main challenge lies in analyzing the influence of the Laplacian matrix on the state transition matrix and the properties of the product of nonindependent and nonstationary random matrices. Some analysis techniques such as graph theory and martingale theory are used to deal with the above issues. A cooperative excitation condition is introduced, under which the convergence of the distributed SG algorithm can be obtained without relying on the independency or stationarity assumptions of regression vectors which are commonly used in the existing literature. Furthermore, the convergence rate of the algorithm can be established. Finally, we show that all the sensors can cooperate to fulfill the estimation task even though any individual sensor cannot by a simulation example.

Parameter identification algorithm for ship manoeuvrability and wave peak model based multi-innovation stochastic gradient algorithm use data filtering technique

This paper addresses the issue of identifying ship motion parameters and wave peak frequency. Utilising the Euler discretisation principle, we establish a discrete-time auto-regressive moving average model with exogenous input (ARMAX) for the ship-wave system. Furthermore, we develop a filtering-based stochastic gradient algorithm for the system by applying filtering techniques and auxiliary model identification idea. A filtering-based multi-innovation stochastic gradient algorithm, utilizing the multi-innovation identification theory, was developed to enhance the convergence rate and accuracy of parameter identification. This approach was found to be more effective than the filtering-based stochastic gradient algorithm. Simulation results validate the efficacy of the proposed algorithm in parameter identification.

Online Distributed Stochastic Gradient Algorithm for Nonconvex Optimization With Compressed Communication

Online Distributed Stochastic Gradient Algorithm for Nonconvex Optimization With Compressed Communication

Sparse optimization guided pruning for neural networks

Neural network pruning is a critical field aimed at reducing the infrastructure costs of neural networks by removing parameters. Traditional methods follow a fixed paradigm including pretraining, pruning, and fine-tuning. Despite the close relationship among these three stages, most pruning methods treat them as independent processes. In this paper, we propose a novel two-stage pruning method, which includes pretraining a network that is instructive for subsequent pruning, and a unified optimization model that integrates pruning and fine-tuning. Specifically, in the first stage, we design a group sparse regularized model for pretraining. This model not only safeguards the network from irreversible damage but also offers valuable insights for the pruning process. In the second stage, we introduce an element-wise sparse regularization into pruning model. This model enables us to pinpoint sparse weights more precisely than pretrained network. It automatically derives effective pruning criteria, and omits the step of fine-tuning. To implement the two-stage process in practice, we utilize stochastic gradient algorithm for the pretraining and design a threshold algorithm for pruning stage. Extensive experiments confirm the competitive performance of our proposed method in terms of both accuracy and memory cost when compared to various benchmarks. Furthermore, ablation experiments validate the effectiveness of the proposed pretraining model’s guidance for the pruning process.

Robust and compact maximum margin clustering for high-dimensional data

In the field of machine learning, clustering has become an increasingly popular research topic due to its critical importance. Many clustering algorithms have been proposed utilizing a variety of approaches. This study focuses on clustering of high-dimensional data using the maximum margin clustering approach. In this paper, two methods are introduced: The first method employs the classical maximum margin clustering approach, which separates data into two clusters with the greatest margin between them. The second method takes cluster compactness into account and searches for two parallel hyperplanes that best fit to the cluster samples while also being as far apart from each other as possible. Additionally, robust variants of these clustering methods are introduced to handle outliers and noise within the data samples. The stochastic gradient algorithm is used to solve the resulting optimization problems, enabling all proposed clustering methods to scale well with large-scale data. Experimental results demonstrate that the proposed methods are more effective than existing maximum margin clustering methods, particularly in high-dimensional clustering problems, highlighting the efficacy of the proposed methods.

Identification of Multi-Innovation Stochastic Gradients with Maximum Likelihood Algorithm Based on Ship Maneuverability and Wave Peak Models

This paper investigates the problem of real-time parameter identification for ship maneuvering parameters and wave peak frequency in an ocean environment. Based on the idea of Euler discretion, a combined model of ship maneuvering and wave peak frequency (ship–wave) is made a discretion, and a discrete-time auto-regressive moving-average model with exogenous input (ARMAX) is derived for parameter identification. Based on the ideas of stochastic gradient identification and multi-innovation theory, a multi-innovation stochastic gradient (MI-SG) algorithm is derived for parameter identification of the ship–wave discretion model. Maximum likelihood theory is introduced to propose a maximum likelihood-based multi-innovation stochastic gradient (ML-MI-SG) algorithm. Compared to the MI-SG algorithm, the ML-MI-SG algorithm shows improvements in both parameter identification accuracy and identification convergence speed. Simulation results verify the effectiveness of the proposed algorithm.

Asynchronous Parallel Large-Scale Gaussian Process Regression.

Gaussian process regression (GPR) is an important nonparametric learning method in machine learning research with many real-world applications. It is well known that training large-scale GPR is a challenging task due to the required heavy computational cost and large volume memory. To address this challenging problem, in this article, we propose an asynchronous doubly stochastic gradient algorithm to handle the large-scale training of GPR. We formulate the GPR to a convex optimization problem, i.e., kernel ridge regression. After that, in order to efficiently solve this convex kernel problem, we first use the random feature mapping method to approximate the kernel model and then utilize two unbiased stochastic approximations, i.e., stochastic variance reduced gradient and stochastic coordinate descent, to update the solution asynchronously and in parallel. In this way, our algorithm scales well in both sample size and dimensionality, and speeds up the training computation. More importantly, we prove that our algorithm has a global linear convergence rate. Our experimental results on eight large-scale benchmark datasets with both regression and classification tasks show that the proposed algorithm outperforms the existing state-of-the-art GPR methods.

New Scalable and Efficient Online Pairwise Learning Algorithm.

Pairwise learning is an important machine-learning topic with many practical applications. An online algorithm is the first choice for processing streaming data and is preferred for handling large-scale pairwise learning problems. However, existing online pairwise learning algorithms are not scalable and efficient enough for large-scale high-dimensional data, because they were designed based on singly stochastic gradients. To address this challenging problem, in this article, we propose a dynamic doubly stochastic gradient algorithm (D2SG) for online pairwise learning. Especially, only the time and space complexities of O (d) are needed for incorporating a new sample, where d is the dimensionality of data. This means that our D2SG is much faster and more scalable than the existing online pairwise learning algorithms while the statistical accuracy can be guaranteed through our rigorous theoretical analysis under standard assumptions. The experimental results on a variety of real-world datasets not only confirm the theoretical result of our new D2SG algorithm, but also show that D2SG has better efficiency and scalability than the existing online pairwise learning algorithms.

Convergence analysis of a synchronous gradient estimation scheme for time-varying parameter systems

For the problem of time-varying parameter estimation and its convergence, this paper proposes an invariant matrix to represent the changing laws of time-varying parameters, which suggests a novel expression to describe the parameters by employing finite terms. A joint recursive least squares algorithm is derived by minimizing the criterion function associated with the unknown invariant matrix. For a class of time-varying systems, a joint stochastic gradient algorithm with the computational efficiency is obtained by analogy to achieve the estimation of invariant matrix and system parameters. The convergence analysis is explored to demonstrate that the time-varying parameter estimation error is bounded under the persistent excitation conditions. The simulation results also validate the effectiveness of the proposed synchronous parameter estimation approach.

An off-policy multi-agent stochastic policy gradient algorithm for cooperative continuous control

Multi-agent reinforcement learning (MARL) algorithms based on trust regions (TR) have achieved significant success in numerous cooperative multi-agent tasks. These algorithms restrain the Kullback–Leibler (KL) divergence (i.e., TR constraint) between the current and new policies to avoid aggressive update steps and improve learning performance. However, the majority of existing TR-based MARL algorithms are on-policy, meaning that they require new data sampled by current policies for training and cannot utilize off-policy (or historical) data, leading to low sample efficiency. This study aims to enhance the data efficiency of TR-based learning methods. To achieve this, an approximation of the original objective function is designed. In addition, it is proven that as long as the update size of the policy (measured by the KL divergence) is restricted, optimizing the designed objective function using historical data can guarantee the monotonic improvement of the original target. Building on the designed objective, a practical off-policy multi-agent stochastic policy gradient algorithm is proposed within the framework of centralized training with decentralized execution (CTDE). Additionally, policy entropy is integrated into the reward to promote exploration, and consequently, improve stability. Comprehensive experiments are conducted on a representative benchmark for multi-agent MuJoCo (MAMuJoCo), which offers a range of challenging tasks in cooperative continuous multi-agent control. The results demonstrate that the proposed algorithm outperforms all other existing algorithms by a significant margin.

Auxiliary model-based hierarchical stochastic gradient methods for multiple-input multiple-output systems

This paper presents a hierarchical identification model for multiple-input multiple-output (MIMO) systems. An auxiliary model-based hierarchical stochastic gradient (AM-HSG) algorithm is derived by means of the auxiliary model identification idea and the hierarchical identification principle. Furthermore, an auxiliary model-based hierarchical multi-innovation stochastic gradient (AM-HMISG) algorithm is derived by utilizing the multi-innovation identification theory. In order to compare the computational efficiency of the AM-HSG and AM-HMISG algorithms for identifying MIMO systems, this paper gives the existing traditional gradient algorithms and discusses the complexity of these algorithms in detail. Finally, the simulation example tests the effectiveness of all four algorithms.

Research on partition control of direct air‐cooling condenser based on multivariable Hammerstein CARMA model predictive control

AbstractPartition control of the fan array within a direct air‐cooling condenser (DACC) is recognized as a highly effective energy‐saving measure and a means to mitigate the adverse effects of ambient winds. This study focuses on the development of a dynamic partition model, which establishes the relationship between mass flow rate and fan array speed under windy conditions. The model is grounded in multivariable Hammerstein controlled autoregressive moving average (H‐CARMA) systems. Additionally, an innovative extended stochastic gradient algorithm, founded on the hierarchical identification principle, is introduced to estimate the model's unknown parameters. The recursive least square is employed as a benchmark to verify the accuracy and efficiency of the approach. Subsequently, leveraging the identified model, the model predictive controller based on the differential evolution method is designed to derive the optimal control law. Finally, simulation test and analysis are carried out to testify the effectiveness of our control strategy. The experiment result shows that the dynamic characteristics of the system have been significantly improved and the fan power consumption is reduced by 3.5% compared to the PID controller.

SARAH-M: A fast stochastic recursive gradient descent algorithm via momentum

As a simple but effective way, the momentum method has been widely adopted in stochastic optimization algorithms for large-scale machine learning problems and the success of stochastic optimization with the momentum term for many applications in machine learning and other related areas has been reported everywhere. However, the understanding of how the momentum improves the performance of modern variance reduced stochastic gradient algorithms, e.g., the stochastic dual coordinate ascent average gradient (SDCA) method, the stochastically controlled stochastic gradient (SCSG) method, the stochastic recursive gradient algorithm (SARAH), etc., is still limited. To tackle this issue, this work studies the performance of SARAH with the momentum term theoretically and empirically, and develops a novel variance reduced stochastic gradient algorithm, termed as SARAH-M. We rigorously prove that SARAH-M attains a linear rate of convergence for minimizing the strongly convex function. We further propose an adaptive SARAH-M method (abbreviated as AdaSARAH-M) by incorporating the random Barzilai–Borwein (RBB) technique into SARAH-M, which provides an easy way to determine the step size for the original SARAH-M algorithm. The theoretical analysis that shows AdaSARAH-M with a linear convergence speed is also provided. Moreover, we show that the complexity of the proposed algorithms can outperform modern stochastic optimization algorithms. Finally, the numerical results, compared with state-of-the-art algorithms on benchmarking machine learning problems, verify the efficacy of the momentum in variance reduced stochastic gradient algorithms.

Stochastic differential equation approximations of generative adversarial network training and its long-run behavior

AbstractThis paper analyzes the training process of generative adversarial networks (GANs) via stochastic differential equations (SDEs). It first establishes SDE approximations for the training of GANs under stochastic gradient algorithms, with precise error bound analysis. It then describes the long-run behavior of GAN training via the invariant measures of its SDE approximations under proper conditions. This work builds a theoretical foundation for GAN training and provides analytical tools to study its evolution and stability.

A Stochastic Algorithm Based on Cable-stayed Bridges Model for Stochastic Optimization in Machine Learning

This paper takes the economic cost of road design as the objective function to study the optimization of road construction cost. It is well-know that stochastic optimization is widely regarded as one of the most important and difficult problems in optimization and machine learning. It is noted that various methods for stochastic optimization problems are sensitive to the learning rate (step-size), which is usually underestimated. Therefore, two new stochastic conjugate gradient algorithms are proposed in this paper. We adapt two modified line search techniques to set the stochastic optimization. One uses the modified Armijo line search technology, the other uses the modified weak Wolfe-Powell line search technology. The new search direction has the properties of sufficient descent and trust region, which makes the establishment of global convergence of the two algorithms very understandable and clear. Finally, numerical experiments show that the two new stochastic conjugate gradient methods are competitive compared with other classical methods.

A key-term separation based least square method for Hammerstein SOC estimation model

In this paper, a Hammerstein state of charge (SOC) estimation model for lithium-ion batteries with two inputs and single output is constructed to map the relation between battery SOC and its discharge current/voltage, and a key-term separation based least square (KT-LS) algorithm is studied to identify the model parameters so as to identify the battery SOC. The thought is, by adopting the key-term separation principle, to decompose the key-term from the linear element and to substitute the expression of the nonlinear element only into the decomposed key-term, thus to express the model output as a linear auto-regression form about parameters. Then the least square algorithm is performed to optimize parameters in the reformed auto-regression form of the Hammerstein SOC system. The advantages of the explored the KT-LS algorithm contains the least parameters required, thus the calculation burden of the KT-LS algorithm is small compared with the over-parameterized model based algorithm. In the simulation experiment part, the performance of the KT-LS algorithm under different conditions are analyzed under UDDS and LA92 conditions.; the performance comparison between the KT-LS algorithm and the key-term separation based stochastic gradient (KT-SG) algorithm is carried out; the performance of the KT-LS algorithm based Hammerstein model method is compared with that of the back propagation (BP) neural network, the Long Short Term Memory (LSTM) neural network and the Extended Kalman filter (EKF) method. The simulation results show that the KT-LS algorithm based Hammerstein SOC model can effectively estimate the SOC of lithium-ion batteries.

Methodics and tools of cough sound processing on basic of neural net

The purpose of the article is to analyze the methods and means of processing cough sounds to detect lung diseases, as well as to describe the developed system for classifying and detecting cough sounds based on a deep neural network. Four types of machine learning and the use of convolutional neural network (CNN) are considered. Hypermarkets of CNN are given. Varieties of machine learning based on the CNN are discussed. The analysis of works on the methodology and means of processing cough sounds based on the CNN with the reduction of the means used and the accuracy of recognition is carried out. Details of machine learning using the environmental sound classification 50 (ESC-50) dataset are discussed. To recognize COVID-19 cough, a classifier was analyzed using CNN as a machine learning model. The proposed CNN system is designed to classify and detect cough sounds based on ESC-50. After selecting a set of sound classification data, four stages are described: extraction of features from audio files, labeling, training, testing. The ESC-50 used for the study was downloaded from the Kaggle website. Python libraries and modules related to deep learning and data science were used to implement the project: NumPy, Librosa, Matplotlib, Hickle, Sci-Kit Learn, Keras. The implemented network used a stochastic gradient algorithm. Several volunteers recorded their voices while coughing using their smartphones and it was assured to record their voices in a public environment to introduce noise to the sounds, in addition to some audio files that were downloaded online. The results showed an average accuracy of 85.37 %, precision of 78.8 % and a recall record of 91.9 %.

Maximum likelihood based multi‐innovation stochastic gradient identification algorithms for bilinear stochastic systems with ARMA noise

SummaryThis paper considers the parameter estimation problem for bilinear stochastic systems with autoregressive moving average (ARMA) noise using the stochastic gradient method. First, the identification model is derived by eliminating the state variables. Based on the obtained identification model, a multi‐innovation generalized extended stochastic gradient (MI‐GESG) algorithm is proposed using the multi‐innovation identification theory. Furthermore, to enhance the parameter estimation accuracy, a maximum likelihood based MI‐GESG (ML‐MI‐GESG) algorithm is developed by using the maximum likelihood identification principle. Finally, an illustrative simulation example is provided to testify the proposed algorithms. The simulation results show the effectiveness of the proposed algorithms for identifying bilinear systems.

Parameter Estimation of Fractional Wiener Systems with the Application of Photovoltaic Cell Models

Fractional differential equations are used to construct mathematical models and can describe the characteristics of real systems. In this paper, the parameter estimation problem of a fractional Wiener system is studied by designing linear filters which can obtain smaller tunable parameters and maintain the stability of the parameters in any case. To improve the identification performance of the stochastic gradient algorithm, this paper derives two modified stochastic gradient algorithms for the fractional nonlinear Wiener systems with colored noise. By introducing the forgetting factor, a forgetting factor stochastic gradient algorithm is deduced to improve the convergence rate. To achieve more efficient and accurate algorithms, we propose a multi-innovation forgetting factor stochastic gradient algorithm by means of the multi-innovation theory, which expands the scalar innovation into the innovation vector. To test the developed algorithms, a fractional-order dynamic photovoltaic model is employed in the simulation, and the dynamic elements of this photovoltaic model are estimated using the modified algorithms. Concurrently, a numerical example is given, and the simulation results verify the feasibility and effectiveness of the proposed procedures.

Hybrid SGD algorithms to solve stochastic composite optimization problems with application in sparse portfolio selection problems

In this paper, we study stochastic composite problems where the objective can be the composition of an outer single-valued function and an inner vector-valued mapping. In this stochastic composite optimization, the inner mapping can be expressed as an expectation over random component mappings. In this study, we propose two algorithms to address the generality and possible singularities of this problem and bound their sample complexities for finding an ϵ-stationary point. The first algorithm is the prox-linear hybrid stochastic gradient algorithm, which may achieve sample complexities of O(ϵ2τ−5/2) and O(ϵτ−3/2) for the component mappings and their Jacobians respectively, where τ∈[0,1]. The second algorithm is the normalized proximal hybrid stochastic gradient algorithm, which takes advantage of the special structure of the regularizer. This algorithm may achieve sample complexities of O(ϵ2τ−4) for both the component mappings and the Jacobians, where τ∈[5/4,7/4]. Numerical experiments prove that the two proposed algorithms are quite competitive with other existing algorithms. A real-life application in sparse portfolio selection problems is also promising.