Convergence Of Learning Algorithm Research Articles

A Large-Scale Study of Activation Functions in Modern Deep Neural Network Architectures for Efficient Convergence

Activation functions play an important role in the convergence of learning algorithms based on neural networks. Theyprovide neural networks with nonlinear ability and the possibility to fit in any complex data. However, no deep study exists in theliterature on the comportment of activation functions in modern architecture. Therefore, in this research, we compare the 18 most used activation functions on multiple datasets (CIFAR-10, CIFAR-100, CALTECH-256) using 4 different models (EfficientNet,ResNet, a variation of ResNet using the bag of tricks, and MobileNet V3). Furthermore, we explore the shape of the losslandscape of those different architectures with various activation functions. Lastly, based on the result of our experimentation,we introduce a new locally quadratic activation function namely Hytana alongside one variation Parametric Hytana whichoutperforms common activation functions and address the dying ReLU problem.

Semi-Synchronous Federated Learning Protocol With Dynamic Aggregation in Internet of Vehicles

In an Internet of Vehicle (IoV) system, federated learning (FL) is a new approach to process real-time vehicle data in a distributed way, which can improve the driving experience and service quality. However, due to the high mobility and uncertainty of vehicles, the existing federated learning protocols are difficult to meet the full requirements from an IoV system, such as efficient resource allocation, high precision learning, and fast convergence of learning algorithm. To solve the above problems, in this paper, we propose a semi-synchronous federated learning (Semi-SynFed) protocol, to improve the performance of machine learning at Internet of Vehicles. In Semi-SynFed, we first select appropriate nodes to participate the aggregation by its computing capacity, network capacity and learning value of training samples. Meanwhile, the dynamic waiting time technique is designed to adjust the server waiting time at each round dynamically, which makes federated learning process work much more efficient. Finally, a dynamic aggregation scheme is designed to aggregate model parameters in an asynchronous way. To evaluate the reliability of Semi-SynFed, we establish a simulation of IoVs based on the real world data. In the comparative experiments, the results show that the Semi-SynFed outperforms the existing federated learning protocols in terms of convergence speed and resource consumption.

Methodology formation of the training sample short-term forecasting electricity load for an energy supply company using data mining technologies

For effective management of the power system operation mode the predictive information about hourly electrical load of all consumers is required. Forecasting errors, as a rule, lead to a decline of the technological and economic indicators of the power system operation, due to unreasonable changes of the generating equipment operating mode, as well as the selection of a non-optimal scheme of electrical networks. This article is devoted to improving the accuracy of short-term load forecasting of delivery points cluster of energy sales company with the use artificial neural networks. One of the most important conditions for achieving high prediction accuracy is the quality of the data sample required for training and testing neural network algorithms for short-term loading forecast. The proposed methodology is based on the authors’ analysis uses the factors influencing the hourly power loading. The proposed methodology is based on the authors’ analysis uses the factors influencing the hourly power loading as well as methods for improving the convergence of learning algorithms for artificial neural networks.

Cover and iterative learning control for and decryption in secure communication

Typical cover techniques adopted in the conventional secure communication schemes are the additive masking and modulation by multiplication. In order to enhance security, in this paper, the new masking and decryption methods were given by iterative learning control algorithm for secure communication. In order to enhance security, the paper proposed the nonlinear masking method to apply to the traditional items and used the iterative learning control algorithm to decrypt. The algorithm reconstructed the information signal completely and analyzed the robustness and convergence of learning algorithm about the initial error and output error. The convergence conditions were given and the simulations shown in the algorithm can reconstruct the signal in secure communication.

Some Features of Neural Networks as Nonlinearly Parameterized Models of Unknown Systems Using an Online Learning Algorithm

This paper deals with deriving the properties of updated neural network model that is exploited to identify an unknown nonlinear system via the standard gradient learning algorithm. The convergence of this algorithm for online training the three-layer neural networks in stochastic environment is studied. A special case where an unknown nonlinearity can exactly be approximated by some neural network with a nonlinear activation function for its output layer is considered. To analyze the asymptotic behavior of the learning processes, the so-called Lyapunov-like approach is utilized. As the Lyapunov function, the expected value of the square of approximation error depending on network parameters is chosen. Within this approach, sufficient conditions guaranteeing the convergence of learning algorithm with probability 1 are derived. Simulation results are presented to support the theoretical analysis.

Load Balancing in Heterogeneous Networks Based on Distributed Learning in Near-Potential Games

We present a novel approach for distributed load balancing in heterogeneous networks that use cell range expansion (CRE) for user association and almost blank subframe (ABS) for interference management. First, we formulate the problem as a minimization of an $\alpha$ -fairness objective function with load and outage constraints. Depending on $\alpha$ , different objectives in terms of network performance or fairness can be achieved. Next, we model the interactions among the base stations for load balancing as a near-potential game, in which the potential function is the $\alpha$ -fairness function. The optimal pure Nash equilibrium (PNE) of the game is found by using distributed learning algorithms. We propose log-linear and binary log-linear learning algorithms for complete and partial information settings, respectively. We give a detailed proof of convergence of learning algorithms for a near-potential game. We provide sufficient conditions under which the learning algorithms converge to the optimal PNE. By running extensive simulations, we show that the proposed algorithms converge within few hundreds of iterations. The convergence speed in the case of partial information setting is comparable to that of the complete information setting. Finally, we show that outage can be controlled and a better load balancing can be achieved by introducing ABS.

Convergence of Learning Algorithms in Neural Networks for Adaptive Identification of Nonlinearly Parameterized Systems

This paper deals with studying the asymptotical properties of neural networks used for the adaptive identification of nonlinearly parameterized system. To update the neural network's parameters, simple online gradient type learning algorithm is employed. A distinguishing feature of this algorithm is that its step size remains constant both in non-stochastic case and in stochastic case, and the learning set is infinite. Based on the Lyapunov-like concept, sufficient conditions guaranteeing the convergence of the learning algorithms are derived. Simulations are presented to support the theoretical results.

Fuzzy wavelet neural network based on fuzzy clustering and gradient techniques for time series prediction

This paper presents the development of fuzzy wavelet neural network system for time series prediction that combines the advantages of fuzzy systems and wavelet neural network. The structure of fuzzy wavelet neural network (FWNN) is proposed, and its learning algorithm is derived. The proposed network is constructed on the base of a set of TSK fuzzy rules that includes a wavelet function in the consequent part of each rule. A fuzzy c-means clustering algorithm is implemented to generate the rules, that is the structure of FWNN prediction model, automatically, and the gradient-learning algorithm is used for parameter identification. The use of fuzzy c-means clustering algorithm with the gradient algorithm allows to improve convergence of learning algorithm. FWNN is used for modeling and prediction of complex time series and prediction of foreign-exchange rates. Exchange rates are dynamic process that changes every day and have high-order nonlinearity. The statistical data for the last 2 years are used for the development of FWNN prediction model. Effectiveness of the proposed system is evaluated with the results obtained from the simulation of FWNN-based systems and with the comparative simulation results of previous related models.

Convergence of learning algorithms without a projection facility

Drawing upon recent contributions in the statistical literature, we present a new result on the convergence of recursive, stochastic algorithms which can be applied to economic models with learning. The formal result provides probability bounds for convergence which can be used to describe the local stability under learning of rational expectations equilibria. Our treatment also generalizes previous results by permitting the state variable to follow a nonlinear Markov process. Economic applications include the stability of noisy sunspot equilibria.

Learning control algorithms for tracking "slowly" varying trajectories

To date, most of the available results in learning control have been utilized in applications where a robot is required to execute the same motion over and over again, with a certain periodicity. This is due to the requirement that all learning algorithms assume that a desired output is given a priori over the time duration t in [0,T]. For applications where the desired outputs are assumed to change "slowly", we present a D-type, PD-type, and PID-type learning algorithms. At each iteration we assume that the system outputs and desired trajectories are contaminated with measurement noise, the system state contains disturbances, and errors are present during reinitialization. These algorithms are shown to be robust and convergent under certain conditions. In theory, the uniform convergence of learning algorithms is achieved as the number of iterations tends to infinity. However, in practice we desire to stop the process after a minimum number of iterations such that the trajectory errors are less than a desired tolerance bound. We present a methodology which is devoted to alleviate the difficulty of determining a priori the controller parameters such that the speed of convergence is improved. In particular, for systems with the property that the product matrix of the input and output coupling matrices, CB, is not full rank. Numerical examples are given to illustrate the results.

Convergence of learning algorithms with constant learning rates

The behavior of neural network learning algorithms with a small, constant learning rate, epsilon, in stationary, random input environments is investigated. It is rigorously established that the sequence of weight estimates can be approximated by a certain ordinary differential equation, in the sense of weak convergence of random processes as epsilon tends to zero. As applications, backpropagation in feedforward architectures and some feature extraction algorithms are studied in more detail.