Neural Network Training Algorithm Research Articles

Social entertainment robot based on neural network algorithm in personalized music course simulation

This study focuses on the simulation of personalized music courses using social entertainment robots based on neural network algorithms, and introduces the current research status of music processing algorithms based on neural networks. A detailed explanation was provided on the training methods for neural network algorithms, including BP neural network training and music generation algorithms. Research was conducted on activation functions, music output detection, and loss function detection. A virtual music course system framework has been designed to provide students with personalized learning experiences. In a social entertainment robot based on neural network algorithms, a complete control node system was reconstructed, aiming to achieve interactive and entertainment effects of music courses through sensing technology. The control node system includes multiple key components, and the robot is able to perceive and recognize the actions, facial expressions, and sounds of students. Social entertainment robots have intelligent classification and recommendation functions, and this algorithm can intelligently classify and recommend music resources based on students’ music preferences, levels, and needs. By analyzing the personal preferences and learning progress of students, robots can provide customized music course content and learning materials for each student, improving learning effectiveness and personal interests.

Fast-M Adversarial Training Algorithm for Deep Neural Networks

Fast-M Adversarial Training Algorithm for Deep Neural Networks

ATM Forecasting using Optimized Artificial Neural Networks

Abstract: Running out of money at an ATM or other place results in higher costs for unscheduled currency supplies and less income from lost surcharge fees. However, banks are unable to invest this non-earning assets to produce interest revenue due to oversupply of currency. ATM forecasting automation is therefore urgently needed. Artificial neural networks are utilized as a forecasting method in this study because they function well for automated prediction tasks. The most significant algorithm for neural network training is propagation in reverse. However, it is prone to become stuck in local minima, which might result in incorrect answers. Thereby, in order to hybridize with artificial neural networks, a few global search and optimization strategies were needed. Genetic algorithms that mimic the idea of natural growth are one such method. Therefore, a hybrid intelligent system that combines genetic algorithms and neural networks made up of neurons is presented for ATM forecasting in this study. The suggested mixed approach functioned better than the current back propagation-based system, according to the results

Mathematical Modeling on a Physics-Informed Radial Basis Function Network

The article is devoted to approximate methods for solving differential equations. An approach based on neural networks with radial basis functions is presented. Neural network training algorithms adapted to radial basis function networks are proposed, in particular adaptations of the Nesterov and Levenberg-Marquardt algorithms. The effectiveness of the proposed algorithms is demonstrated for solving model problems of function approximation, differential equations, direct and inverse boundary value problems, and modeling processes in piecewise homogeneous media.

Do we need a more sophisticated multilayer artificial neural network to compute roughness coefficient?

Artificial neural networks (ANNs) are one of the most rapidly growing fields of soft computing. Along with deep learning, they are currently the most widely used machine learning techniques. Artificial neural networks are especially suitable for problem-solving where a researcher deals with incomplete data sets and no algorithms or specific sets of rules to be followed.This article deals with a case of comparison of several modifications of neural networks that may be applied to compute Chézy’s roughness coefficient. Neural network modelling is often started with one hidden layer. Having even one hidden layer, a neural network presents a powerful computing system to give good results. If it is necessary, the number of hidden layers may increase. Usually, two or three hidden layers of neurons are used. Diverse activation functions may also apply. The article aims to explore the necessity of developing sophisticated multilayer artificial neural networks to compute Chézy’s roughness coefficient.Under the study, the following modifications of the neural network computing Chézy’s roughness coefficient were considered and analysed: (1) Application of two hidden layers of neurons; (2) Application of three hidden layers of neurons; (3) Use of a dropout algorithm for training neural networks by randomly dropping units during training to prevent their co-adaptation; (4) Apart from the sigmoid (logistic) activation function, the use of other artificial neuron transfer functions – hyperbolic tangent (tanh) and rectifying activation function (ReLU).The training and testing of the considered neural network options were carried out using the actual hydro-morphological and hydrological data related to the channel section on the Dnieper River (downstream of Kyiv), the Desna River section near Chernihiv, and the Pripyat River section near the town of Turiv. The Python object-oriented programming environment was applied to build and train the neural networks. The test results confirm the acceptability and sufficiency of computing the Chézy roughness coefficient using the ANN of direct propagation with one hidden layer and a sigmoid logistic activation function. The formation of a qualitative set of training data, as well as data arrangement and choosing a relevant computing model based on empirical knowledge, are, as concluded, among more actual issues than creating more sophisticated neural networks.

Desire backpropagation: A lightweight training algorithm for multi-layer spiking neural networks based on spike-timing-dependent plasticity

Spiking neural networks (SNNs) are a viable alternative to conventional artificial neural networks when resource efficiency and computational complexity are of importance. A major advantage of SNNs is their binary information transfer through spike trains which eliminates multiplication operations. The training of SNNs has, however, been a challenge, since neuron models are non-differentiable and traditional gradient-based backpropagation algorithms cannot be applied directly. Furthermore, spike-timing-dependent plasticity (STDP), albeit being a spike-based learning rule, updates weights locally and does not optimize for the output error of the network. We present desire backpropagation, a method to derive the desired spike activity of all neurons, including the hidden ones, from the output error. By incorporating this desire value into the local STDP weight update, we can efficiently capture the neuron dynamics while minimizing the global error and attaining a high classification accuracy. That makes desire backpropagation a spike-based supervised learning rule. We trained three-layer networks to classify MNIST and Fashion-MNIST images and reached an accuracy of 98.41% and 87.56%, respectively. In addition, by eliminating a multiplication during the backward pass, we reduce computational complexity and balance arithmetic resources between forward and backward pass, making desire backpropagation a candidate for training on low-resource devices.

A review of practice of using evolutionary algorithms for neural network synthesis and training

The object of this research is the application of evolutionary algorithms for the synthesis and training of neural networks. The paper aims to select and review the existing experience on using evolutionary algorithms as competitive methods to conventional approaches in neural network training and creation, and to evaluate such existing solutions for further development of this field. The essence of the obtained results lies in the successful application of genetic algorithms in conjunction with neural networks to optimize parameters, architecture, and weight coefficients of the networks. The genetic algorithms allowed improving the performance and accuracy of neural networks, especially in cases where backpropagation algorithms faced difficulties in finding optimal solutions. These results can be attributed to the fact that genetic algorithms are efficient methods for global optimization in parameter space. They help avoid local minima and discover more reliable and stable solutions. The obtained findings can be practically utilized to enhance the performance and quality of neural networks in various classification and prediction tasks. The use of genetic algorithms enables the selection of optimal weight coefficients, network connections, and identification of significant features from the dataset. However, they come with the limitation of additional time costs for evaluating the entire population according to the selection criteria. It is worth noting that the application of genetic algorithms is not a universal method for all tasks, and the algorithm parameters should be individually tuned for each specific problem. Further research could focus on refining the combination methods of genetic algorithms and neural networks, as well as exploring their application in new domains and tasks.

An Overview of Speech Enhancement Based on Deep Learning Techniques

Recent years have seen a significant amount of studies in the area of speech enhancement. This review looks at several speech improvement methods as well as Deep Neural Network (DNN) functions in speech enhancement. Speech transmissions are frequently distorted by ambient noise, background noise, and reverberations. There are processing methods, such as Short-time Fourier Transform, Short-time Autocorrelation, and Short-time Energy (STE), that can be used to enhance speech. To reduce speech noise, features such as the Mel-Frequency Cepstral Coefficients (MFCCs), Logarithmic Power Spectrum (LPS), and Gammatone Frequency Cepstral Coefficients (GFCCs) can be retrieved and input to a DNN. DNN is essential to speech improvement since it builds models using a lot of training data and evaluates the efficacy of the enhanced speech using certain performance metrics. Since the beginning of deep learning publications in 1993, a variety of speech enhancement methods have been examined in this study. This review provides a thorough examination of the several neural network topologies, training algorithms, activation functions, training targets, acoustic features, and databases that were employed for the job of speech enhancement and were gathered from various articles published between 1993 and 2022.

Hybrid deep learning model for efficient state of charge estimation of Li-ion batteries in electric vehicles

State of charge (SoC) estimation is critical for the safe and efficient operation of electric vehicles (EVs). This work proposes a hybrid multi-layer deep neural network (HMDNN)-based approach for SoC estimation in EVs. This HMDNN uses Mountain Gazelle Optimizer (MGO) as a training algorithm for the deep neural network. Our method leverages the intrinsic relationship between the SoC and the voltage/current measurements of the EV battery to accurately estimate the SoC in real time. We evaluate our approach on a large dataset of real-world EV charging data and demonstrate its effectiveness in comparison to traditional SoC estimation methods. Four diverse Li-ion battery datasets of electric vehicles are employed which are the dynamic stress test (DST), Beijing dynamic stress test (BJDST), federal urban driving schedule (FUDS), and highway driving schedule (US06) with different temperatures of 0oC,25oC,45oC. The comparison is made with Mayfly Optimization Algorithm based DNN, Particle Swarm Optimization based DNN and Back-Propagation based DNN. The evaluation indices used are normalized mean square error (NMSE), root mean square error (RMSE), mean absolute error (MAE), and relative error (RE). The proposed algorithm achieves 0.1% NMSE and 0.3% RMSE on average on all datasets, which validates the effective performance of the proposed model. The results show that the proposed neural network-based approach can achieve higher accuracy and faster convergence than existing methods. This can enable more efficient EV operation and improved battery life.

Functional extreme learning machine.

Extreme learning machine (ELM) is a training algorithm for single hidden layer feedforward neural network (SLFN), which converges much faster than traditional methods and yields promising performance. However, the ELM also has some shortcomings, such as structure selection, overfitting and low generalization performance. This article a new functional neuron (FN) model is proposed, we takes functional neurons as the basic unit, and uses functional equation solving theory to guide the modeling process of FELM, a new functional extreme learning machine (FELM) model theory is proposed. The FELM implements learning by adjusting the coefficients of the basis function in neurons. At the same time, a simple, iterative-free and high-precision fast parameter learning algorithm is proposed. The standard data sets UCI and StatLib are selected for regression problems, and compared with the ELM, support vector machine (SVM) and other algorithms, the experimental results show that the FELM achieves better performance.

Memristor Crossbar Circuits Implementing Equilibrium Propagation for On-Device Learning.

Equilibrium propagation (EP) has been proposed recently as a new neural network training algorithm based on a local learning concept, where only local information is used to calculate the weight update of the neural network. Despite the advantages of local learning, numerical iteration for solving the EP dynamic equations makes the EP algorithm less practical for realizing edge intelligence hardware. Some analog circuits have been suggested to solve the EP dynamic equations physically, not numerically, using the original EP algorithm. However, there are still a few problems in terms of circuit implementation: for example, the need for storing the free-phase solution and the lack of essential peripheral circuits for calculating and updating synaptic weights. Therefore, in this paper, a new analog circuit technique is proposed to realize the EP algorithm in practical and implementable hardware. This work has two major contributions in achieving this objective. First, the free-phase and nudge-phase solutions are calculated by the proposed analog circuits simultaneously, not at different times. With this process, analog voltage memories or digital memories with converting circuits between digital and analog domains for storing the free-phase solution temporarily can be eliminated in the proposed EP circuit. Second, a simple EP learning rule relying on a fixed amount of conductance change per programming pulse is newly proposed and implemented in peripheral circuits. The modified EP learning rule can make the weight update circuit practical and implementable without requiring the use of a complicated program verification scheme. The proposed memristor conductance update circuit is simulated and verified for training synaptic weights on memristor crossbars. The simulation results showed that the proposed EP circuit could be used for realizing on-device learning in edge intelligence hardware.

A 24.1 TOPS/W mixed-signal BNN processor in 28-nm CMOS

ABSTRACT A mixed-signal binarized neural network (BNN) processor for the MNIST image classification is demonstrated based on analogue circuit networks. BNN algorithm for training neural networks with binary weights and activations reduces power consumption and memory size. This algorithm is a design to perform core operations of multi-layer perceptron (MLP) to reduce complexity and power consumption using analogue circuits. The mixed-signal BNN processor employs a current mirror neuron to perform multiply-and-accumulate (MAC) operations and sign activation functions. The near-threshold current mirror neuron that computes the key operations of the BNN algorithm is used to achieve low power consumption. The design occupies 0.065 mm2 in 28-nm CMOS with 560B of on-chip SRAM. The 28-nm CMOS test-chip achieves energy efficiency of 24.1 TOPS/W and 94% accuracy on the MNIST image classification. This design with binary weights and activations exhibits only 5% degradation in accuracy compared with the models with floating-point precision.

A new training algorithm for long short-term memory artificial neural network based on particle swarm optimization

A new training algorithm for long short-term memory artificial neural network based on particle swarm optimization

Digital financial asset price fluctuation forecasting in digital economy era using blockchain information: A reconstructed dynamic-bound Levenberg–Marquardt neural-network approach

Digital financial assets such as cryptocurrency are playing an increasingly crucial role in the digital economy era. Cryptocurrency is characterized by significant volatility and asset price fluctuations in the short term. Therefore, the development of an accurate and technologically reliable forecasting approach is important. For accurately predicting the closing price of the cryptocurrency, as a representative digital financial asset, we developed a reconstructed dynamic-bound Levenberg–Marquardt neural network (R-DB-LM-NN) architecture and a corresponding neural-network training algorithm with a moving-boundary mechanism for evaluating the correctness of each descent direction. We used a high-frequency blockchain information dataset for training and prediction. The dynamic bound was introduced to increase the step size so that the neural network could effectively cross the local minimum and to avoid interrupting the neural-network iteration process. Then, we built a high-frequency encrypted digital currency blockchain information dataset. Experiments confirmed that the proposed architecture and algorithm are superior to traditional neural-network machine-learning methods, such as artificial neural networks, and deep-learning methods, such as long short-term memory and convolutional neural networks, with regard to prediction performance. Finally, the implications of the study and limitations of the proposed approach are discussed, along with the extension of the approach to other time series research domains, for researchers and practitioners.

Nonlinear adaptive tracking control of piezoelectric bimorph actuator using hybrid modeling approach

Piezoelectric bimorph actuator has the advantages of small size, fast response speed and high displacement accuracy, but its inherent hysteresis nonlinearity seriously affect the control accuracy and stability of the system. The dead-zone operator was incorporated into classical Prandtl–Ishlinskii model to enable the description of asymmetric hysteresis of piezoelectric bimorph actuator. A hybrid model approach was developed with neural network and improved Prandtl–Ishlinskii model, and it has the advantages of a neural network with ready-made training algorithms and improve the Prandtl–Ishlinskii (PI) model to describe the asymmetric hysteresis. The adaptive control method was derived from training algorithm of neural network, which can update the weight parameters of Play operator and Dead-zone operator in real time. Comparing the results without control, the RMSE of displacement error decreases by 61.35% with classic model, and decreases by 82.93% with hybrid model and proposed adaptive tracking control. Experimental results show that the proposed hybrid model and adaptive control approach can more effectively compensate the hysteresis of piezoelectric bimorph.

Teaching schoolchildren in the basics of artificial intelligence technologies in conditions of additional education

The article is devoted to the description of methodological approaches to teaching schoolchildren the basics of artificial intelligence based on the technology of organizing project and educational and research activities to achieve meta-subject results of education, the formation of systemic representations and the experience of their application in real life. The experience of organizing courses within the framework of additional education for students of 10–11th grades of secondary school on the use of neural networks in solving practice-oriented problems is presented. The block structure of the content of the courses is given, which is based on the phased implementation of the idea of the future project. A detailed description of the content of each block is given with a demonstration of a presentation of complex theoretical material that is accessible to the understanding of schoolchildren. In particular, recommendations are given for the study of sequential and recurrent models of artificial neural networks, fully connected and convolutional layers, object detectors, as well as an algorithm for training neural networks by backpropagation of errors. Data acquisition tools for training artificial neural networks are described. An algorithm for writing a program for training a neural network to recognize handwritten digits of the MNIST set is characterized. Examples of projects for the use of neural networks, carried out by students together with the teacher as part of the study of the proposed courses, are given.The article is addressed to teachers of additional education, informatics teachers, teachers and students of universities — everyone who is interested in the use of artificial neural networks in solving practical problems and methods of teaching schoolchildren in project activity.

Metaheuristic Algorithms to Enhance the Performance of a Feedforward Neural Network in Addressing Missing Hourly Precipitation

This research study investigates the implementation of three metaheuristic algorithms, namely, Grey Wolf Optimizer (GWO), Multi-Verse Optimizer (MVO), and Moth-Flame Optimisation (MFO), for coupling with a Feedforward Neural Network (FNN) in addressing missing hourly rainfall observations, while overcoming the limitation of conventional training algorithm of artificial neural network that often traps in local optima. The proposed GWOFNN, MVOFNN, and MFOFNN were compared against the conventional Levenberg Marquardt Feedforward Neural Network (LMFNN) in addressing the artificially introduced missing hourly rainfall records of Kuching Third Mile Station. The findings show that the proposed approaches are superior to LMFNN in predicting the 20% hourly rainfall observations in terms of mean absolute error (MAE) and coefficient of correlation (r). The best performance ANN model is GWOFNN, followed with MVOFNN, MFOFNN and lastly LMFNN.

A Binary-Activation, Multi-Level Weight RNN and Training Algorithm for ADC-/DAC-Free and Noise-Resilient Processing-in-Memory Inference With eNVM

We propose a new algorithm for training neural networks with binary activations and multi-level weights, which enables efficient processing-in-memory circuits with embedded nonvolatile memories (eNVM). Binary activations obviate costly DACs and ADCs. Multi-level weights leverage multi-level eNVM cells. Compared to existing algorithms, our method not only works for feed-forward networks (e.g., fully-connected and convolutional), but also achieves higher accuracy and noise resilience for recurrent networks. In particular, we present an RNN-based trigger-word detection PIM accelerator, with detailed hardware noise models and circuit co-design techniques, and validate our algorithm's high inference accuracy and robustness against a variety of real hardware non-idealities.

A generic resource augmentation architecture for efficient mobile communication

SummaryMobile devices play a vital role in people's day‐to‐day activities and are essential for communication, accessing Internet resources, entertainment, and so forth. In most cases, the performance of mobile applications is restricted due to the inadequate resources of mobile specifications such as controlled battery power, predetermined storage, and limited processing competence. Thus to improve the processing efficiency of resource‐constrained mobile devices, this research proposes a model developed on offloading technique based on the recent literature observations which transfer the resource‐consuming tasks from mobile devices to proximate computing entities. This generic architecture aiming for resource augmentation is formulated with five fundamental components including a mobile device, offloading engine, resource augmentation engine, scheduler, and synchronizer. The offloading engine identifies the resource‐intensive tasks that need to be executed on external entities using soft computing techniques such as neural network training algorithms, fuzzy logic, and neuro‐fuzzy logic processing. The augmentation engine chooses feasible edge computing entities such as Arduino, Raspberry PI controller, and fog devices. The scheduler prioritizes the resource‐consuming tasks for offloading based on their importance. The synchronizer coordinates all the central components by updating the execution status of tasks at regular intervals of time. Further extension of this research work, based on the proposed architecture three resource augmentation frameworks are developed namely (a) an adaptive proximate computing framework (b) a web service‐based IoT framework, and (c) a neuro‐fuzzy hybrid framework. Moreover, these frameworks are realized through the development of resource‐efficient intelligent edge computing IoT mobile apps namely (a) an inventory monitoring app (b) a lecture recording app, and (c) an intelligent transport app. The performance of these three resource augmentation frameworks is analyzed through the simulation results and it illustrates that the neuro‐fuzzy hybrid framework outperforms the other two. The outcome of the proposed research contributions improves the effectiveness of mobile communication by providing seamless services to mobile users regardless of their resource‐limited devices. Besides the devised edge computing architecture supports mobile application developers in the design and deployment of resource‐rich mobile applications in low‐specification mobile devices. Also, there is a strong scope for implementing the proposed generic architecture as an industrial smart product with adequate hardware and software configurations.

Optimal parameter design of a slot jet impingement/microchannel heat sink base on multi-objective optimization algorithm

Jet impingement and microchannel cooling as effective cooling methods have been widely employed for thermal management of high heat flux electronics. In this work, a novel hybrid slot jet impingement/microchannel heat sink, combining the advantages of two cooling technologies, was proposed to achieve vertical flushing of the jet and timely discharge of the waste fluid. To further enhance the cooling performance, a 3-D numerical simulation was applied to evaluate the impact of inlet and outlet branch passage inclination ratios (Hin,1/Hin,2 and Hout,1/Hout,2), channel unit width (W1), channel width ratio (φ) and channel height (H3) on heat sink thermal-hydraulic characteristics. According to the parametric analysis, the variations in the inlet and outlet branch passage inclination ratios have significant effect on the mass flow distribution among the heat transfer channels. Afterward, to acquire the optimal compromise solution for practical applications, the artificial neural network training and multi-objective optimization algorithms, i.e. multi-objective particle swarm optimization (MOPSO) and multi-objective genetic algorithm (MOGA), are utilized to optimize the geometrical parameters, i.e. W1, φ and H3. During the optimization, the average heated surface temperature (T¯heat) and the total pressure drop (ΔP) are two conflicting objectives to evaluate the heat transfer and resistance characteristics. Finally, an optimal compromise solution (W1=1.03mm, φ=0.58 and H3=2.44mm) is chosen from the Pareto front by using the classical decision-making algorithm, i.e. TOPSIS. Compared with one objective minimization in the Pareto front, the optimal compromise solution has a sound balance between heat transfer and resistance performance. Additionally, for the optimal compromise solution, with an inlet velocity of 1.5 m/s and a heat flux of 200 W/cm2, the maximum and average temperature of the heated surface does not exceed 73 °C and 67 °C, respectively, and the total pressure drop is only 5.7 kPa.