Neural Network Training Algorithm Research Articles

SkipNet: an adaptive neural network equalization algorithm for future passive optical networking

In this paper, we propose an original adaptive neural network equalizer (NNE) algorithm named SkipNet, which is suitable for rapid training on a packet-by-packet basis for burst-mode non-linear equalization in upstream PON transmission. SkipNet uses the simple LMS algorithm and avoids complex neural network training algorithms such as backpropagation and mini-batch training. We demonstrate SkipNet on captured continuous mode 100 Gbit/s PAM4 signals using an SOA preamplifier to achieve the challenging 29 dB PON optical loss budget. The adaptive SkipNet equalizer is shown to overcome combinations of severe SOA patterning effects and fiber dispersion impairments to achieve >29dB dynamic range back-to-back and >22.9dB dynamic range for up to 81.6 ps/nm accumulated dispersion. It can adapt in as little as 250 training symbols to each impairment scenario, which is equivalent to existing FFE/DFE solutions, while matching the non-linear performance of previously proposed static NNE solutions. To the best of our knowledge, SkipNet is the first ever adaptive NNE framework that can realistically be trained and adapted on a packet-by-packet basis and within strict PON packet preamble lengths.

Speed and accuracy investigations of neural network algorithms for ionospheric modelling at an equatorial region

Neural networks are very efficient tools for modeling, including ionospheric modeling. The training algorithm is important for achieving the optimum performance of the trained network. This research is therefore meant to evaluate and compare the performances of ten neural network training algorithms based on their prediction accuracies, and the duration/times taken by each of the algorithms to establish the optimum result. The neural networks were trained using electron density measurements by Radio Occultation (RO) technique from the Constellation Observing System for Meteorology, Ionosphere, and Climate (COSMIC) satellites. Data for the period 2006 through 2021 was used. The networks were trained using about 2.9 million data points collected from the Kano region, Nigeria (5-degree rectangular region around geographic: 12.00° N, 8.59° E) after performing data quality control. The training algorithms considered in the work include: Bayesian Regularization (BR); Broyden-Fletcher-Goldfarb-Shanno Quasi-Newton (BFG); Conjugate Gradient with Powell/Beale (CGB); Fletcher-Reeves Conjugate Gradient (CGF); Gradient descent with momentum and adaptive learning rate (GDX); Levenberg Marquardt (LM); One Step Secant (OSS); Polak-Ribiére Conjugate Gradient (CGP); Resilient Backpropagation (RP); and Scaled Conjugate Gradient (SCG). The results showed that the BR and the LM algorithms gave the best performances in minimizing the errors of prediction (the mean RMSEs are respectively 112 and 114 ×103 electrons/cm3), but the RP algorithm, which came third in terms of accuracy, was significantly faster than both the LM and BR algorithms. The worst-performing algorithm in terms of accuracy was the GDX algorithm, although it was the fastest algorithm. The BFG algorithm was the worst-performing algorithm in terms of a combination of speed and accuracy. The developed neural network model was validated using ionosonde electron density measurements obtained from Ilorin, Nigeria (geographic: 8.5° N, 4.5° E; geomagnetic: 1.8° S). A comparison of the neural network, the NeQuick, and the IRI model predictions relative to the ionosonde measurements indicate that the neural network model was the best-performing model; the NN model predictions minimized the mean absolute errors (MAEs) in ∼44% of 399 ionosonde profiles investigated, the IRI model did so in ∼32%, and the NeQuick did so in ∼24%. The MAEs of the NeQuick however exhibited the best (least) variance. In overall, the NN model gave the least (best) mean of the MAEs (∼73 × 103 cm−3), compared to ∼82 × 103 cm−3 given by both the NeQuick and the IRI models, further supporting the idea that neural networks are excellent for present-day ionospheric modeling.

МЕТОД ОБУЧЕНИЯ ИНТЕЛЛЕКТУАЛЬНОГО АГЕНТА С ПОМОЩЬЮ СЕТЕЙ DOUBLE DQN, ПУТЕВЫХ ТОЧЕК И ФУНКЦИИ ВОЗНАГРАЖДЕНИЯ

The problems of increasing the control efficiency of autonomous vehicles are considered. The problem of reducing the required computational resources for the intelligent vehicle control module is highlighted. A neural network training algorithm for Double DQN architecture with modified reward functions is proposed. The basis of the proposed solution is the use of lane segmentation, reward function and the use of additional waypoints in training. A software model has been developed and simulation of the learning process has been performed. The results obtained from a comparative analysis with known solutions show a stable increase in episode duration, and effective training in a realistic urban simulation. The study points to the possibility of reducing the need for high computing power, which will enable the use of central processing units (CPUs) for basic functions of unmanned vehicles instead of graphics processing units (GPUs).

A Comparative Study of Artificial Neural Network Training Algorithms to Predict Photovoltaic Module Output Power

Introduction: Solar energy is a crucial component and contributes a large portion of renewable resources, the demand for which has recently increased. During previous years, it was difficult to predict the amount of energy obtained from photovoltaic systems. The angle of inclination of the solar panel, the amount of radiation, the speed of the wind, the amount of humidity, and the temperature of the weather are major factors that effectively affect the production of electrical energy. Scientists have used many strategies to predict the power generated by PV modules accurately, but each method has different pros and cons. Method: This study tested three training algorithms for artificial neural networks (ANN): scaled conjugate gradient (SCG), Levenberg Marquardt (LM), and Bayesian regularization (BR), determining which one performed best in terms of prediction speed and accuracy. Twenty-eight thousand two hundred ninety-six samples of experimental data for primary influencing environmental factors were fed into the artificial neural network, which consists of 15 hidden layers. Before training the network, we preprocessed the data to remove factors that have a secondary effect. Result: The analytical results showed that the artificial neural network trained according to the LM algorithm is the best in terms of accuracy and speed of predicting the resulting photovoltaic energy. Conclusion: The results showed that although the regression evaluation and MSE values for the LM, SCG, and BR algorithms are close (98.129%, 0.0622, 97.849%, 0.0587, 98.151%, and 0.0585, respectively), the LM training algorithm is the best in terms of speed of calculation and display of results.

Tuning parameters of deep neural network training algorithms pays off: a computational study

AbstractThe paper aims to investigate the impact of the optimization algorithms on the training of deep neural networks with an eye to the interaction between the optimizer and the generalization performance. In particular, we aim to analyze the behavior of state-of-the-art optimization algorithms in relationship to their hyperparameters setting to detect robustness with respect to the choice of a certain starting point in ending on different local solutions. We conduct extensive computational experiments using nine open-source optimization algorithms to train deep Convolutional Neural Network architectures on an image multi-class classification task. Precisely, we consider several architectures by changing the number of layers and neurons per layer, to evaluate the impact of different width and depth structures on the computational optimization performance. We show that the optimizers often return different local solutions and highlight the strong correlation between the quality of the solution found and the generalization capability of the trained network. We also discuss the role of hyperparameters tuning and show how a tuned hyperparameters setting can be re-used for the same task on different problems achieving better efficiency and generalization performance than a default setting.

Speech Recognition Classification Leveraging a Spiking Neural Network Training Algorithm Based on Tempotron

Speech Recognition Classification Leveraging a Spiking Neural Network Training Algorithm Based on Tempotron

Social entertainment robot based on neural network algorithm in personalized music course simulation

This study focuses on the simulation of personalized music courses using social entertainment robots based on neural network algorithms, and introduces the current research status of music processing algorithms based on neural networks. A detailed explanation was provided on the training methods for neural network algorithms, including BP neural network training and music generation algorithms. Research was conducted on activation functions, music output detection, and loss function detection. A virtual music course system framework has been designed to provide students with personalized learning experiences. In a social entertainment robot based on neural network algorithms, a complete control node system was reconstructed, aiming to achieve interactive and entertainment effects of music courses through sensing technology. The control node system includes multiple key components, and the robot is able to perceive and recognize the actions, facial expressions, and sounds of students. Social entertainment robots have intelligent classification and recommendation functions, and this algorithm can intelligently classify and recommend music resources based on students’ music preferences, levels, and needs. By analyzing the personal preferences and learning progress of students, robots can provide customized music course content and learning materials for each student, improving learning effectiveness and personal interests.

Fast-M Adversarial Training Algorithm for Deep Neural Networks

Fast-M Adversarial Training Algorithm for Deep Neural Networks

ATM Forecasting using Optimized Artificial Neural Networks

Abstract: Running out of money at an ATM or other place results in higher costs for unscheduled currency supplies and less income from lost surcharge fees. However, banks are unable to invest this non-earning assets to produce interest revenue due to oversupply of currency. ATM forecasting automation is therefore urgently needed. Artificial neural networks are utilized as a forecasting method in this study because they function well for automated prediction tasks. The most significant algorithm for neural network training is propagation in reverse. However, it is prone to become stuck in local minima, which might result in incorrect answers. Thereby, in order to hybridize with artificial neural networks, a few global search and optimization strategies were needed. Genetic algorithms that mimic the idea of natural growth are one such method. Therefore, a hybrid intelligent system that combines genetic algorithms and neural networks made up of neurons is presented for ATM forecasting in this study. The suggested mixed approach functioned better than the current back propagation-based system, according to the results

Mathematical Modeling on a Physics-Informed Radial Basis Function Network

The article is devoted to approximate methods for solving differential equations. An approach based on neural networks with radial basis functions is presented. Neural network training algorithms adapted to radial basis function networks are proposed, in particular adaptations of the Nesterov and Levenberg-Marquardt algorithms. The effectiveness of the proposed algorithms is demonstrated for solving model problems of function approximation, differential equations, direct and inverse boundary value problems, and modeling processes in piecewise homogeneous media.

ВИБІР НЕЙРОМЕРЕЖЕВОГО КЛАСИФІКАТОРА ДЛЯ ВИРІШЕННЯ ЗАВДАННЯ АНАЛІЗУ СИГНАЛІВ РЕБ ТА ШТУЧНИХ ЗАВАД

At present, the task of ensuring high-quality control of unmanned aerial vehicles and the transmission of reliable information in a complex interference environment is relevant. This task becomes especially important when intentional interference is used in the form of countermeasures. Various methods are used to detect and localize powerful sources of parasitic signals, including: time domain methods: signal spectrum analysis, correlation function analysis, phase shift analysis; frequency domain methods: signal amplitude response analysis, signal phase response analysis, power spectral density analysis; spatial domain methods: antenna directivity analysis, signal strength analysis at different points in space. Analyzing signals affected by EW and artificial interference is a challenging task. Traditional signal analysis methods are often ineffective for this task because they cannot take into account the complex characteristics of EW and artificial interference. Neural network classifiers are a promising tool for analyzing signals with EW and artificial interference. Neural networks can learn from large datasets and detect complex patterns that cannot be detected by traditional methods. This article describes the process of selecting a neural network classifier for analyzing EW and artificial interference signals. Various factors that should be considered when choosing a neural network architecture, training algorithm, and network parameters are discussed). The paper presents an analytical review of neural network classifiers, their varieties and features, and selects the optimal one for solving the task.

Do we need a more sophisticated multilayer artificial neural network to compute roughness coefficient?

Artificial neural networks (ANNs) are one of the most rapidly growing fields of soft computing. Along with deep learning, they are currently the most widely used machine learning techniques. Artificial neural networks are especially suitable for problem-solving where a researcher deals with incomplete data sets and no algorithms or specific sets of rules to be followed.This article deals with a case of comparison of several modifications of neural networks that may be applied to compute Chézy’s roughness coefficient. Neural network modelling is often started with one hidden layer. Having even one hidden layer, a neural network presents a powerful computing system to give good results. If it is necessary, the number of hidden layers may increase. Usually, two or three hidden layers of neurons are used. Diverse activation functions may also apply. The article aims to explore the necessity of developing sophisticated multilayer artificial neural networks to compute Chézy’s roughness coefficient.Under the study, the following modifications of the neural network computing Chézy’s roughness coefficient were considered and analysed: (1) Application of two hidden layers of neurons; (2) Application of three hidden layers of neurons; (3) Use of a dropout algorithm for training neural networks by randomly dropping units during training to prevent their co-adaptation; (4) Apart from the sigmoid (logistic) activation function, the use of other artificial neuron transfer functions – hyperbolic tangent (tanh) and rectifying activation function (ReLU).The training and testing of the considered neural network options were carried out using the actual hydro-morphological and hydrological data related to the channel section on the Dnieper River (downstream of Kyiv), the Desna River section near Chernihiv, and the Pripyat River section near the town of Turiv. The Python object-oriented programming environment was applied to build and train the neural networks. The test results confirm the acceptability and sufficiency of computing the Chézy roughness coefficient using the ANN of direct propagation with one hidden layer and a sigmoid logistic activation function. The formation of a qualitative set of training data, as well as data arrangement and choosing a relevant computing model based on empirical knowledge, are, as concluded, among more actual issues than creating more sophisticated neural networks.

Desire backpropagation: A lightweight training algorithm for multi-layer spiking neural networks based on spike-timing-dependent plasticity

Spiking neural networks (SNNs) are a viable alternative to conventional artificial neural networks when resource efficiency and computational complexity are of importance. A major advantage of SNNs is their binary information transfer through spike trains which eliminates multiplication operations. The training of SNNs has, however, been a challenge, since neuron models are non-differentiable and traditional gradient-based backpropagation algorithms cannot be applied directly. Furthermore, spike-timing-dependent plasticity (STDP), albeit being a spike-based learning rule, updates weights locally and does not optimize for the output error of the network. We present desire backpropagation, a method to derive the desired spike activity of all neurons, including the hidden ones, from the output error. By incorporating this desire value into the local STDP weight update, we can efficiently capture the neuron dynamics while minimizing the global error and attaining a high classification accuracy. That makes desire backpropagation a spike-based supervised learning rule. We trained three-layer networks to classify MNIST and Fashion-MNIST images and reached an accuracy of 98.41% and 87.56%, respectively. In addition, by eliminating a multiplication during the backward pass, we reduce computational complexity and balance arithmetic resources between forward and backward pass, making desire backpropagation a candidate for training on low-resource devices.

A review of practice of using evolutionary algorithms for neural network synthesis and training

The object of this research is the application of evolutionary algorithms for the synthesis and training of neural networks. The paper aims to select and review the existing experience on using evolutionary algorithms as competitive methods to conventional approaches in neural network training and creation, and to evaluate such existing solutions for further development of this field. The essence of the obtained results lies in the successful application of genetic algorithms in conjunction with neural networks to optimize parameters, architecture, and weight coefficients of the networks. The genetic algorithms allowed improving the performance and accuracy of neural networks, especially in cases where backpropagation algorithms faced difficulties in finding optimal solutions. These results can be attributed to the fact that genetic algorithms are efficient methods for global optimization in parameter space. They help avoid local minima and discover more reliable and stable solutions. The obtained findings can be practically utilized to enhance the performance and quality of neural networks in various classification and prediction tasks. The use of genetic algorithms enables the selection of optimal weight coefficients, network connections, and identification of significant features from the dataset. However, they come with the limitation of additional time costs for evaluating the entire population according to the selection criteria. It is worth noting that the application of genetic algorithms is not a universal method for all tasks, and the algorithm parameters should be individually tuned for each specific problem. Further research could focus on refining the combination methods of genetic algorithms and neural networks, as well as exploring their application in new domains and tasks.

An Overview of Speech Enhancement Based on Deep Learning Techniques

Recent years have seen a significant amount of studies in the area of speech enhancement. This review looks at several speech improvement methods as well as Deep Neural Network (DNN) functions in speech enhancement. Speech transmissions are frequently distorted by ambient noise, background noise, and reverberations. There are processing methods, such as Short-time Fourier Transform, Short-time Autocorrelation, and Short-time Energy (STE), that can be used to enhance speech. To reduce speech noise, features such as the Mel-Frequency Cepstral Coefficients (MFCCs), Logarithmic Power Spectrum (LPS), and Gammatone Frequency Cepstral Coefficients (GFCCs) can be retrieved and input to a DNN. DNN is essential to speech improvement since it builds models using a lot of training data and evaluates the efficacy of the enhanced speech using certain performance metrics. Since the beginning of deep learning publications in 1993, a variety of speech enhancement methods have been examined in this study. This review provides a thorough examination of the several neural network topologies, training algorithms, activation functions, training targets, acoustic features, and databases that were employed for the job of speech enhancement and were gathered from various articles published between 1993 and 2022.

Hybrid deep learning model for efficient state of charge estimation of Li-ion batteries in electric vehicles

State of charge (SoC) estimation is critical for the safe and efficient operation of electric vehicles (EVs). This work proposes a hybrid multi-layer deep neural network (HMDNN)-based approach for SoC estimation in EVs. This HMDNN uses Mountain Gazelle Optimizer (MGO) as a training algorithm for the deep neural network. Our method leverages the intrinsic relationship between the SoC and the voltage/current measurements of the EV battery to accurately estimate the SoC in real time. We evaluate our approach on a large dataset of real-world EV charging data and demonstrate its effectiveness in comparison to traditional SoC estimation methods. Four diverse Li-ion battery datasets of electric vehicles are employed which are the dynamic stress test (DST), Beijing dynamic stress test (BJDST), federal urban driving schedule (FUDS), and highway driving schedule (US06) with different temperatures of 0oC,25oC,45oC. The comparison is made with Mayfly Optimization Algorithm based DNN, Particle Swarm Optimization based DNN and Back-Propagation based DNN. The evaluation indices used are normalized mean square error (NMSE), root mean square error (RMSE), mean absolute error (MAE), and relative error (RE). The proposed algorithm achieves 0.1% NMSE and 0.3% RMSE on average on all datasets, which validates the effective performance of the proposed model. The results show that the proposed neural network-based approach can achieve higher accuracy and faster convergence than existing methods. This can enable more efficient EV operation and improved battery life.

Functional extreme learning machine.

Extreme learning machine (ELM) is a training algorithm for single hidden layer feedforward neural network (SLFN), which converges much faster than traditional methods and yields promising performance. However, the ELM also has some shortcomings, such as structure selection, overfitting and low generalization performance. This article a new functional neuron (FN) model is proposed, we takes functional neurons as the basic unit, and uses functional equation solving theory to guide the modeling process of FELM, a new functional extreme learning machine (FELM) model theory is proposed. The FELM implements learning by adjusting the coefficients of the basis function in neurons. At the same time, a simple, iterative-free and high-precision fast parameter learning algorithm is proposed. The standard data sets UCI and StatLib are selected for regression problems, and compared with the ELM, support vector machine (SVM) and other algorithms, the experimental results show that the FELM achieves better performance.

Memristor Crossbar Circuits Implementing Equilibrium Propagation for On-Device Learning.

Equilibrium propagation (EP) has been proposed recently as a new neural network training algorithm based on a local learning concept, where only local information is used to calculate the weight update of the neural network. Despite the advantages of local learning, numerical iteration for solving the EP dynamic equations makes the EP algorithm less practical for realizing edge intelligence hardware. Some analog circuits have been suggested to solve the EP dynamic equations physically, not numerically, using the original EP algorithm. However, there are still a few problems in terms of circuit implementation: for example, the need for storing the free-phase solution and the lack of essential peripheral circuits for calculating and updating synaptic weights. Therefore, in this paper, a new analog circuit technique is proposed to realize the EP algorithm in practical and implementable hardware. This work has two major contributions in achieving this objective. First, the free-phase and nudge-phase solutions are calculated by the proposed analog circuits simultaneously, not at different times. With this process, analog voltage memories or digital memories with converting circuits between digital and analog domains for storing the free-phase solution temporarily can be eliminated in the proposed EP circuit. Second, a simple EP learning rule relying on a fixed amount of conductance change per programming pulse is newly proposed and implemented in peripheral circuits. The modified EP learning rule can make the weight update circuit practical and implementable without requiring the use of a complicated program verification scheme. The proposed memristor conductance update circuit is simulated and verified for training synaptic weights on memristor crossbars. The simulation results showed that the proposed EP circuit could be used for realizing on-device learning in edge intelligence hardware.

A 24.1 TOPS/W mixed-signal BNN processor in 28-nm CMOS

ABSTRACT A mixed-signal binarized neural network (BNN) processor for the MNIST image classification is demonstrated based on analogue circuit networks. BNN algorithm for training neural networks with binary weights and activations reduces power consumption and memory size. This algorithm is a design to perform core operations of multi-layer perceptron (MLP) to reduce complexity and power consumption using analogue circuits. The mixed-signal BNN processor employs a current mirror neuron to perform multiply-and-accumulate (MAC) operations and sign activation functions. The near-threshold current mirror neuron that computes the key operations of the BNN algorithm is used to achieve low power consumption. The design occupies 0.065 mm2 in 28-nm CMOS with 560B of on-chip SRAM. The 28-nm CMOS test-chip achieves energy efficiency of 24.1 TOPS/W and 94% accuracy on the MNIST image classification. This design with binary weights and activations exhibits only 5% degradation in accuracy compared with the models with floating-point precision.

A new training algorithm for long short-term memory artificial neural network based on particle swarm optimization

A new training algorithm for long short-term memory artificial neural network based on particle swarm optimization