Traditional Stochastic Gradient Descent Research Articles

Decentralized Sensor Scheduling, Bandwidth Allocation, and Dynamic Quantization for FL Under Hybrid Data Partitioning

Considering the wide application of multiple types of sensors with diversified data sensing and collection capabilities, we focus on the resulting hybrid data partitioning among the local data set distributed at the edge sensors, especially the practical training implementation of federated learning (FL) under such a setting, where the neural network (NN) is trained collaboratively without requiring the sensors to share their data. Different from the conventional FL schemes, since each local sensor now only has partial data samples with type-specific features, the traditional stochastic gradient descent (SGD)-based training method cannot be directly utilized due to the intertype and intratype data coupling. To address this issue, we first transform the training problem into the primal–dual domain utilizing the corresponding Lagrangian and propose a stochastic primal-descent dual-ascent training method with a two-side residual feedback mechanism. Such a method can be implemented in a scalable way and compensate for the data distortion and loss caused by the practical transmission noise. Furthermore, a decentralized joint scheduling, bandwidth allocation, and dynamic quantization policy is proposed by analyzing the performance at each training iteration and the consumed transmission resources. The proposed method is adaptive to not only the channel state information (CSI) but also the instantaneous gradient importance and dynamic gradient statistics. The closed-form convergence analysis is provided, and the simulation experiments illustrate the superior performance of the proposed scheme.

Approximation of regression-based fault minimization for network traffic

This research associates three distinct approaches for computer network traffic prediction. They are the traditional stochastic gradient descent (SGD) using a few random samplings instead of the complete dataset for each iterative calculation, the gradient descent algorithm (GDA) which is a well-known optimization approach in Deep Learning, and the proposed method. The network traffic is computed from the traffic load (data and multimedia) of the computer network nodes via the Internet. It is apparent that the SGD is a modest iteration but can conclude suboptimal solutions. The GDA is a complicated one, can function more accurate than the SGD but difficult to manipulate parameters, such as the learning rate, the dataset granularity, and the loss function. Network traffic estimation helps improve performance and lower costs for various applications, such as an adaptive rate control, load balancing, the quality of service (QoS), fair bandwidth allocation, and anomaly detection. The proposed method confirms optimal values out of parameters using simulation to compute the minimum figure of specified loss function in each iteration.

Asynchronous Proximal Stochastic Gradient Algorithm for Composition Optimization Problems

In machine learning research, many emerging applications can be (re)formulated as the composition optimization problem with nonsmooth regularization penalty. To solve this problem, traditional stochastic gradient descent (SGD) algorithm and its variants either have low convergence rate or are computationally expensive. Recently, several stochastic composition gradient algorithms have been proposed, however, these methods are still inefficient and not scalable to large-scale composition optimization problem instances. To address these challenges, we propose an asynchronous parallel algorithm, named Async-ProxSCVR, which effectively combines asynchronous parallel implementation and variance reduction method. We prove that the algorithm admits the fastest convergence rate for both strongly convex and general nonconvex cases. Furthermore, we analyze the query complexity of the proposed algorithm and prove that linear speedup is accessible when we increase the number of processors. Finally, we evaluate our algorithm Async-ProxSCVR on two representative composition optimization problems including value function evaluation in reinforcement learning and sparse mean-variance optimization problem. Experimental results show that the algorithm achieves significant speedups and is much faster than existing compared methods.