Back-propagation Training Method Research Articles

A reconfigurable FPGA-based spiking neural network accelerator

The spiking neural network (SNN) is suitable for the intelligent edge computing applications because of its low-power characteristic. This work designs a reconfigurable spiking neural network accelerator supporting the spatiotemporal backpropagation (STBP) training method. The reconfigurable architecture is proposed between the spatial convolution module and the temporal accumulation module of the SNN accelerator. A sparse zero-hopping mechanism is designed to exploit the input sparsity of SNN datasets, and a mask mechanism is introduced between the forward inference computation and the backward training computation to exploit the output sparsity. During the training process, the peak and average performances of the SNN accelerator are 5.57 TOPS and 4.96 TOPS respectively, the power consumption is 6.124 W and the energy efficiency is 0.81 TOPS/W. The peak and average performances of the SNN accelerator are 5.98 TOPS and 5.14 TOPS respectively, the power consumption is 6.943 W and the energy efficiency is 0.74 TOPS/W, during the inference process.

DT-SCNN: dual-threshold spiking convolutional neural network with fewer operations and memory access for edge applications.

The spiking convolutional neural network (SCNN) is a kind of spiking neural network (SNN) with high accuracy for visual tasks and power efficiency on neuromorphic hardware, which is attractive for edge applications. However, it is challenging to implement SCNNs on resource-constrained edge devices because of the large number of convolutional operations and membrane potential (Vm) storage needed. Previous works have focused on timestep reduction, network pruning, and network quantization to realize SCNN implementation on edge devices. However, they overlooked similarities between spiking feature maps (SFmaps), which contain significant redundancy and cause unnecessary computation and storage. This work proposes a dual-threshold spiking convolutional neural network (DT-SCNN) to decrease the number of operations and memory access by utilizing similarities between SFmaps. The DT-SCNN employs dual firing thresholds to derive two similar SFmaps from one Vm map, reducing the number of convolutional operations and decreasing the volume of Vms and convolutional weights by half. We propose a variant spatio-temporal back propagation (STBP) training method with a two-stage strategy to train DT-SCNNs to decrease the inference timestep to 1. The experimental results show that the dual-thresholds mechanism achieves a 50% reduction in operations and data storage for the convolutional layers compared to conventional SCNNs while achieving not more than a 0.4% accuracy loss on the CIFAR10, MNIST, and Fashion MNIST datasets. Due to the lightweight network and single timestep inference, the DT-SCNN has the least number of operations compared to previous works, paving the way for low-latency and power-efficient edge applications.

Automatic Calibration of Microscopic Traffic Simulation Models Using Artificial Neural Networks

Traffic simulations are valuable tools for urban mobility planning and operation, particularly in large cities. Simulation-based microscopic models have enabled traffic engineers to understand local transit and transport behaviors more deeply and manage urban mobility. However, for the simulations to be effective, the transport network and user behavior parameters must be calibrated to mirror real scenarios. In general, calibration is performed manually by traffic engineers who use their knowledge and experience to adjust the parameters of the simulator. Unfortunately, there is still no systematic and automatic process for calibrating traffic simulation networks, although some methods have been proposed in the literature. This study proposes a methodology that facilitates the calibration process, where an artificial neural network (ANN) is trained to learn the behavior of the transport network of interest. The ANN used is the Multi-Layer Perceptron (MLP), trained with back-propagation methods. Based on this learning procedure, the neural network can select the optimized values of the simulation parameters that best mimic the traffic conditions of interest. Experiments considered two microscopic models of traffic and two psychophysical models (Wiedemann 74 and Wiedemann 99). The microscopic traffic models are located in the metropolitan region of São Paulo, Brazil. Moreover, we tested the different configurations of the MLP (layers and numbers of neurons) as well as several variations of the backpropagation training method: Stochastic Gradient Descent (SGD), Adam, Adagrad, Adadelta, Adamax, and Nadam. The results of the experiments show that the proposed methodology is accurate and efficient, leading to calibration with a correlation coefficient greater than 0.8, when the calibrated parameters generate more visible effects on the road network, such as travel times, vehicle counts, and average speeds. For the psychophysical parameters, in the most simplified model (W74), the correlation coefficient was greater than 0.7. The advantage of using ANN for the automatic calibration of simulation parameters is that it allows traffic engineers to carry out comprehensive studies on a large number of future scenarios, such as at different times of the day, as well as on different days of the week and months of the year.

Human Walking Gait Classification Utilizing an Artificial Neural Network for the Ergonomics Study of Lower Limb Prosthetics

Prosthetics and orthotics research, studies, and technologies have been evolving through the years. According to World Health Organization (WHO) data, it is estimated that, globally, 35–40 million people require prosthetics and orthotics usage in daily life. Prosthetics and orthotics demand is increasing due to certain factors. One of the factors is vascular-related disease, which leads to amputation. Prosthetic usage can increase an amputee’s quality of life. Therefore, studies of the ergonomic design of prosthetics are important. The ergonomic factor in design delivers prosthetic products that are comfortable for daily use. One way to incorporate the ergonomic design of prosthetics is by studying the human walking gait. This paper presents a multiclassification of human walking gait based on electromyography (EMG) signals using a machine learning method. An EMG sensor was attached to the bicep femoris longus and gastrocnemius lateral head to acquire the EMG signal. The experiment was conducted by volunteers during normal walking activity at various speeds and the movements were segmented as initial contact, which was labeled as initial gait; loading response to the terminal stance, which was labeled as mid-gait; and pre-swing to terminal swing, which was labeled as final gait. The EMG signal was then characterized using an artificial neural network (ANN) and compared to six training accuracy methods, i.e., the Levenberg–Marquardt backpropagation training algorithm, quasi-Newton training method, Bayesian regulation backpropagation training method, gradient descent backpropagation, gradient descent with adaptive learning rate backpropagation, and one-step secant backpropagation. The machine learning study performed well in the classification of three classes of human walking gait with an overall accuracy (training, testing, and validation) of 96% for Levenberg–Marquardt backpropagation. The gait data will be used to explore the design of lower limb prosthetics in future research.

GFDM TRANSCEIVER BASED ON ANN CHANNEL ESTIMATION

Generalized frequency division multiplexing (GFDM) is one of the candidate waveforms beyond 5G for wireless communication. Channel estimation is challenging in wireless transmission because of inherent interference. It is based on a pilot signal that uses Least Square (LS) and Normalized Least Mean Square (NLMS) to perform the estimation. This paper used the Artificial Neural Network (ANN) as a channel estimation method, considered a novel estimation process for the GFDM transceiver. The channel estimation based on ANN depends on the data set generated from LS estimation, which considers the proposed method is optimized to LS based on the backpropagation neural network. The ANN algorithm considered a fitness function to estimate the channel. Levenberg-Marquardt backpropagation (LM), Bayesian Regularization backpropagation (BR), and Scaled Conjugate Gradient backpropagation (SCG) training methods training by using the Matlab toolbox used to perform the estimation. BR training method gives the best performance than LS and other training methods at 22 neurons in hidden layers, which at 20dB give BER = 0.0369 that enhanced over LS by 0.08.

Sign Language Gesture Recognition and Classification Based on Event Camera with Spiking Neural Networks

Sign language recognition has been utilized in human–machine interactions, improving the lives of people with speech impairments or who rely on nonverbal instructions. Thanks to its higher temporal resolution, less visual redundancy information and lower energy consumption, the use of an event camera with a new dynamic vision sensor (DVS) shows promise with regard to sign language recognition with robot perception and intelligent control. Although previous work has focused on event camera-based, simple gesture datasets, such as DVS128Gesture, event camera gesture datasets inspired by sign language are critical, which poses a great impediment to the development of event camera-based sign language recognition. An effective method to extract spatio-temporal features from event data is significantly desired. Firstly, the event-based sign language gesture datasets are proposed and the data have two sources: traditional sign language videos to event stream (DVS_Sign_v2e) and DAVIS346 (DVS_Sign). In the present dataset, data are divided into five classification, verbs, quantifiers, position, things and people, adapting to actual scenarios where robots provide instruction or assistance. Sign language classification is demonstrated in spike neuron networks with a spatio-temporal back-propagation training method, leading to the best recognition accuracy of 77%. This work paves the way for the combination of event camera-based sign language gesture recognition and robotic perception for the future intelligent systems.

Prediction of operating parameters and output power of ducted wind turbine using artificial neural networks

The performance of a ducted wind turbine was simulated in this study utilizing an artificial neural network under various duct operating conditions. Ducted wind turbines have been identified as one of the cleanest and most effective future renewable energy possibilities. Different methods have been used throughout time to exploit the flow stream power successfully. However, the Ducted Wind Turbine System is one method that is still in its early phases. The operation of the ducted wind turbine is analyzed in this research under different wind conditions. The ANN algorithm is recommended for generating turbine power at various wind speeds. The proposed neural network model is shown to perform better for estimating turbine power curves. The ANN models used the free-stream wind speed, wind speed in the throat section, turbulence intensity, and wind power in the intake part of the duct as input datasets and the output power of wind in the throat section as output datasets. At the same time, a three-layer backpropagation training method from the Multilayer Perceptron algorithm approach, a Levenberg–Marquardt, was selected as the best ANN. It was evaluated by comparing to a Radial-based functions approach with a single hidden layer. The optimum number of neurons in the MLP method’s first and second hidden layers, as well as the RBF method’s single hidden layer, has been determined to be 55 and 70, respectively. At 77 and 70 epochs, the best MSE evaluation efficiency of MLP and RBF networks was estimated to be 0. 000103 and 0. 000353, respectively.

Application of Artificial Neural Networks in Analysis of Time-Variable Optical Reflectance Spectra in Digital Light Projection Spectroscopy

This article undertakes the subject matter of applying artificial neural networks to analyze optical reflectance spectra of objects exhibiting a change of optical properties in the domain of time. A compact Digital Light Projection NIRscan Nano Evaluation Module spectrometer was used to record spectra. Due to the miniature spectrometer’s size and its simplicity of measurement, it can be used to conduct tests outside of a laboratory. A series of plant-derived objects were used as test subjects with rapidly changing optical properties in the presented research cycle. The application of artificial neural networks made it possible to determine the aging time of plants with a relatively low mean squared error, reaching 0.56 h for the Levenberg–Marquardt backpropagation training method. The results of the other ten training methods for artificial neural networks have been included in the paper.

Quality Evaluation Method of a Mathematics Teaching Model Reform Based on an Improved Genetic Algorithm

The poor comprehensiveness of the evaluation indexes of quality evaluation methods for the traditional college mathematics teaching model reform results in low accuracy of the evaluation outcomes. In this paper, aiming at this problem, a quality evaluation method for the college mathematics teaching model reform, based on the genetic algorithm, is proposed. The simulated annealing algorithm uses the weighted comprehensive objective evaluation multiobjective optimization effect that can effectively improve the accuracy of the evaluation results. In the training process, the gradient descent back-propagation training method is used to obtain new weights for the quality evaluation of college mathematics teaching mode reforms and to score various indicators and evaluate the indicators. The mean value of the outcomes is the result of mathematics teaching quality evaluation. The experimental results show that the training error of the convolutional network of the proposed method is significantly small. Based on the genetic algorithm that improves the convolutional network training process, the obtained quality evaluation outcomes are higher in accuracy, better in goodness of fitness function, and considerably lower than other state-of-the-art methods. We observed that the improved genetic algorithm has a more than 90% goodness of fit and the error is significantly lower, that is, 0.01 to 0.04, than the classical genetic algorithm.

Investigation of dissimilar laser welding of stainless steel 304 and copper using the artificial neural network model

In this study, to accurately predict the temperature and melting ratio at low time and cost, the process of dissimilar laser welding of stainless steel 304 and copper was simulated based on artificial neural network (ANN). Among various ANN models, the Bayesian regulation backpropagation training method was utilized to model the current problem. This method was used considering the two temperatures of copper and steel and the two melting ratios of steel and copper as the four outputs, and the four parameters, pulse width, pulse frequency, welding speed, and focal length, as the inputs. According to the results, regression values had a good accuracy in all cases and the histogram diagrams indicated that the error distribution was mainly concentrated at the center; in other words, the major errors of the network were not very large. It was also observed that the error concerning the trained neural networks was acceptable in the experiment phase. Finally, this neural network could be used as a numerical model to estimate the four outputs of steel temperature, copper temperature, steel melting ratio, and copper melting ratio for all input values of pulse width, pulse frequency, welding speed, and focal length in the studied range, without any need to rerun the experiment.

Adaptive Single Neuron Anti-Windup PID Controller Based on the Extended Kalman Filter Algorithm

In this paper, an adaptive single neuron Proportional–Integral–Derivative (PID) controller based on the extended Kalman filter (EKF) training algorithm is proposed. The use of EKF training allows online training with faster learning and convergence speeds than backpropagation training method. Moreover, the propose adaptive PID approach includes a back-calculation anti-windup scheme to deal with windup effects, which is a common problem in PID controllers. The performance of the proposed approach is shown by presenting both simulation and experimental tests, giving results that are comparable to similar and more complex implementations. Tests are performed for a four wheeled omnidirectional mobile robot. Tests show the superiority of the proposed adaptive PID controller over the conventional PID and other adaptive neural PID approaches. Experimental tests are performed on a KUKA® Youbot® omnidirectional platform.

Prediction of the Slump of Fly Ash Blended Concrete Based on Various Numerical Models

Slump is a fundamental engineering property of fly ash blended concrete. This study proposes various numerical models, such as multiple linear regression (MLR), artificial neural network (ANN), and genetic algorithm assisted artificial neural network (GA-ANN) for estimating the slump. First, the concrete slump is regressed as a multiple linear equation of water-to-binder ratio, water content, sand ratio, fly ash replacement ratio, air-entraining agent content, and superplasticizer contents. The correlation coefficient of the MLR method is 0.63. Second, the ANN model is set up, which consists of an input layer, a hidden layer, and output layer. Based on the backpropagation (BP) training method, the optimized network is found. The correlation coefficient between analysis results and experimental results of ANN model is 0.78. Third, GA is used to assist in the optimization process of ANN. The initial value of the hidden layer of ANN is generated by using a genetic algorithm (GA). The correlation coefficient of GAANN integrated model is 0.85. GA-ANN integrated model can make more accurate prediction results than MLR model and ANN model.

Hybrid Wolf-Bat Algorithm for Optimization of Connection Weights in Multi-layer Perceptron

In a neural network, the weights act as parameters to determine the output(s) from a set of inputs. The weights are used to find the activation values of nodes of a layer from the values of the previous layer. Finding the ideal set of these weights for training a Multi-layer Perceptron neural network such that it minimizes the classification error is a widely known optimization problem. The presented article proposes a Hybrid Wolf-Bat algorithm, a novel optimization algorithm, as a solution to solve the discussed problem. The proposed algorithm is a hybrid of two already existing nature-inspired algorithms, Grey Wolf Optimization algorithm and Bat algorithm. The novel introduced approach is tested on ten different datasets of the medical field, obtained from the UCI machine learning repository. The performance of the proposed algorithm is compared with the recently developed nature-inspired algorithms: Grey Wolf Optimization algorithm, Cuckoo Search, Bat Algorithm, and Whale Optimization Algorithm, along with the standard Back-propagation training method available in the literature. The obtained results demonstrate that the proposed method outperforms other bio-inspired algorithms in terms of both speed of convergence and accuracy.

Calibration Method of Magnetometer Based on BP Neural Network

Due to the influence of processing technology and environmental factors, there are errors in attitude measurement with the three-axis magnetometer, and the change of parameters during the operation of the magnetometer in orbit will have a great impact on the measurement accuracy. This paper studies the calibration method of magnetometer based on BP neural network, which reduces the influence of model error on calibration accuracy. Firstly, the error model of the magnetometer and the structural characteristics of the BP neural network are analyzed. Secondly, the number of hidden layers and hidden nodes is optimized. To avoid the problem of slow convergence and low accuracy of basic BP algorithm, this paper uses the Levenberg Marquardt backpropagation training method to improve the training speed and prediction accuracy and realizes the on-orbit calibration of magnetometer through online training of the neural network. Finally, the effectiveness of the method is verified by numerical simulation. The results show that the neural network designed in this paper can effectively reduce the measurement error of magnetometer, while the online training can effectively reduce the error caused by the change of magnetometer parameters, and reduce the measurement error of magnetometer to less than 10 nT.

Micro-seismic event detection and location in underground mines by using Convolutional Neural Networks (CNN) and deep learning

Recent years have witnessed a clear trend to develop deeper and longer tunnels to meet the growing needs of mining. Micro-seismic events location is vital for predicting and avoiding the traditional mine disasters induced by high stress concentration, such as rock burst, roof caving, water inrush and slope landslide. Deep learning has become a research hotspot within the field of artificial intelligence in recent years, which has achieved significant progresses and applications in the areas of image recognition, speech recognition, language processing and computer vision. The biggest difference between the deep learning and the traditional back propagation training method is that the deep learning can automatically and independently learn the characteristics of a large amount of data without human intervention. This paper uses Convolutional Neural Network (CNN) and deep learning techniques to develop a method for identifying the Time Delay of Arrival (TDOA) and subsequently the source location of micro-seismic events in underground mines. The power spectrum and phase spectrum of cross wavelet transform calculated from the recorded seismic waves due to micro-seismic events are used as inputs to CNN. The amplitude and phase information of the cross wavelet transform power spectrum are parameters that are used without manual manipulation to build the complex mapping to predict TDOA by deep learning network. Experimental data from the in-field blast tests and simulation tests show that the proposed approach can well identify TDOA and hence detect the event source locations of the field blasting tests. It is demonstrated that the proposed approach with the CNN and deep learning techniques gives more accurate micro seismic source identifications with the recorded noisy waveforms from in-situ blast tests, as compared to several typical existing methods.

Selective combination in multiple neural networks prediction using independent component regression approach

<p>Biological processes are highly nonlinear in nature and difficult to represent accurately by simple mathematical models. However, this problem can be solved by using neural network. Neural network is a prominent modeling tool especially when it comes to intricate process such as biological process. In this paper, a multiple single hidden layer with ten hidden neurons Feedforward Artificial Neural Network (FANN) was used to model the complex and dynamic relationships between the input (dilution rate, D) and outputs (conversion, y and dimensionless temperature value, θ) for the reactive biological process. Levenberg-Marquardt Backpropagation training method was used. The multiple neural networks predicted outputs were then combined through three different methods which area simple averaging, Principal Component Regression (PCR) and Independent Component Regression (ICR). Multiple neural networks which were created by the bootstrap approach help improved single neural network performance as well as the model robustness for nonlinear process modeling. Comparison was made between the three methods. The result showed that ICR is slightly superior between the three methods especially in noise level 1,2 and 3, however ICR slightly suffer in noise level 4 and 5. This is due to the independent component regression used the latent factors and non-Gaussian distribution of y and θ values for the combination.</p><p>Chemical Engineering Research Bulletin 19(2017) 12-19</p>

Neural-network-based analysis of EEG data using the neuromorphic TrueNorth chip for brain-machine interfaces

Electroencephalography (EEG) is a noninvasive way to record brain activity by means of measuring electrical fields arising from neural activation. Being relatively inexpensive, safe, and readily available, EEG-based techniques have been studied as potential methods for controlling brain-machine interfaces. Previous attempts to analyze EEG signals have focused on well-characterized sensorimotor data features. However, the brain-machine interface field seems to have stagnated in improving motor decoding using this method. One way to overcome this hurdle is to use neural-network-based classification methods to analyze brain-activity data. In this paper, we describe the novel neural networks we created for analyzing existing EEG data. Although these neural networks were programmed, trained, and tested in a conventional central processing unit or graphics processing unit environment, their novelty lies in their full compatibility with IBM's recently introduced ultralow power, neuromorphic TrueNorth chip infrastructure, thus, constituting the analytical units in the next generation of neurobionic mobile devices. We report on the development of a new EEG signal classifier built on a spiking neural network that runs on the TrueNorth platform. Using a modified back-propagation training method that employs trinary weights, we demonstrate state-of-the-art classification accuracy.

Deep belief network based electricity load forecasting: An analysis of Macedonian case

A number of recent studies use deep belief networks (DBN) with a great success in various applications such as image classification and speech recognition. In this paper, a DBN made up from multiple layers of restricted Boltzmann machines is used for electricity load forecasting. The layer-by-layer unsupervised training procedure is followed by fine-tuning of the parameters by using a supervised back-propagation training method. Our DBN model was applied to short-term electricity load forecasting based on the Macedonian hourly electricity consumption data in the period 2008–2014. The obtained results are not only compared with the latest actual data, but furthermore, they are compared with the predicted data obtained from a typical feed-forward multi-layer perceptron neural network and with the forecasted data provided by the Macedonian system operator (MEPSO). The comparisons show that the applied model is not only suitable for hourly electricity load forecasting of the Macedonian electric power system, it actually provides superior results than the ones obtained using traditional methods. The mean absolute percentage error (MAPE) is reduced by up to 8.6% when using DBN, compared to the MEPSO data for the 24-h ahead forecasting, and the MAPE for daily peak forecasting is reduced by up to 21%.

Enhanced Software Effort Estimation Using Multi Layered Feed Forward Artificial Neural Network Technique

Software Effort Estimation models are hot topic of study over 3 decades. Several models have been developed in these decades. Providing accurate estimations of software is still very challenging. The major reason for such disappointments in projects are because of inaccurate software development norms; effort estimation is one such practice. Dynamically fluctuating environment of technology in software development industry make effort estimation further perplexing. One of the most commonly used algorithmic model for estimating effort in industry is COCOMO. Capability of machine learning particularly Artificial Neural Networks is to adjust a complex set of bond among the various independent and dependent variables. The paper proposes usage of ANN (Artificial Neural Network) based model technologically advanced using Multi Layered Feed Forward Neural Network which is given training with Back Propagation training method. COCOMO data-set is accustomed to test and train the network. Mean-Square-Error (MSE) and Mean Magnitude of Relative-Error (MMRE) are used as performance measurement indices. The experiment outputs suggest that the suggested model can provide better results and accurately forecast the software development effort.

An ANN-PSO-based model to predict fault-prone modules in software

Fault-prone module identification in software helps software developers to allocate effort and resources more efficiently during software testing process. In this paper, the fault-prone software modules are identified, making use of existing reduced software metrics. Different methods have been used to reduce dimension of software metrics and taken as input of ANN-based models where the ANN is trained using back propagation algorithm. The back propagation algorithm suffers from local optima problem and, in order to avoid this problem, a global optimisation algorithm such as Particle Swarm Optimisation (PSO) algorithm has been used to train the ANN in this paper. An ANN-based model trained using PSO (ANN-PSO) has been proposed in this paper to identify the fault-prone modules in software. The reduced software metrics from different methods have been taken as input of the proposed ANN-PSO approach to determine prediction accuracy. A comparative experimental study has been performed on different data sets to show the effectiveness of the proposed ANN-PSO approach. The experimental results show that the proposed model has better prediction accuracy than the ANN-based models trained using the conventional back propagation training method.