Conventional Backpropagation Algorithm Research Articles

Backpropagation in hyperbolic chaos via adjoint shadowing

To generalise the backpropagation method to both discrete-time and continuous-time hyperbolic chaos, we introduce the adjoint shadowing operator S acting on covector fields. We show that S can be equivalently defined as: S is the adjoint of the linear shadowing operator S; S is given by a ‘split then propagate’ expansion formula; S(ω) is the only bounded inhomogeneous adjoint solution of ω. By (a), S adjointly expresses the shadowing contribution, a significant part of the linear response, where the linear response is the derivative of the long-time statistics with respect to system parameters. By (b), S also expresses the other part of the linear response, the unstable contribution. By (c), S can be efficiently computed by the nonintrusive shadowing algorithm in Ni and Talnikar (2019 J. Comput. Phys. 395 690–709), which is similar to the conventional backpropagation algorithm. For continuous-time cases, we additionally show that the linear response admits a well-defined decomposition into shadowing and unstable contributions.

Evolutionary optimization framework to train multilayer perceptrons for engineering applications.

Training neural networks by using conventional supervised backpropagation algorithms is a challenging task. This is due to significant limitations, such as the risk for local minimum stagnation in the loss landscape of neural networks. That may prevent the network from finding the global minimum of its loss function and therefore slow its convergence speed. Another challenge is the vanishing and exploding gradients that may happen when the gradients of the loss function of the model become either infinitesimally small or unmanageably large during the training. That also hinders the convergence of the neural models. On the other hand, the traditional gradient-based algorithms necessitate the pre-selection of learning parameters such as the learning rates, activation function, batch size, stopping criteria, and others. Recent research has shown the potential of evolutionary optimization algorithms to address most of those challenges in optimizing the overall performance of neural networks. In this research, we introduce and validate an evolutionary optimization framework to train multilayer perceptrons, which are simple feedforward neural networks. The suggested framework uses the recently proposed evolutionary cooperative optimization algorithm, namely, the dynamic group-based cooperative optimizer. The ability of this optimizer to solve a wide range of real optimization problems motivated our research group to benchmark its performance in training multilayer perceptron models. We validated the proposed optimization framework on a set of five datasets for engineering applications, and we compared its performance against the conventional backpropagation algorithm and other commonly used evolutionary optimization algorithms. The simulations showed the competitive performance of the proposed framework for most examined datasets in terms of overall performance and convergence. For three benchmarking datasets, the proposed framework provided increases of 2.7%, 4.83%, and 5.13% over the performance of the second best-performing optimizers, respectively.

Geometric Constellation Shaping for Fiber-Optic Channels via End-to-End Learning

End-to-end learning has become a popular method to optimize a constellation shape of a communication system. When the channel model is differentiable, end-to-end learning can be applied with conventional backpropagation algorithm for optimization of the shape. A variety of optimization algorithms have also been developed for end-to-end learning over a non-differentiable channel model. In this paper, we compare gradient-free optimization method based on the cubature Kalman filter, model-free optimization and backpropagation for end-to-end learning on a fiber-optic channel modeled by the split-step Fourier method. The results indicate that the gradient-free optimization algorithms provide a decent replacement to backpropagation in terms of performance at the expense of computational complexity. Furthermore, the quantization problem of finite bit resolution of the digital-to-analog and analog-to-digital converters is addressed and its impact on geometrically shaped constellations is analysed. Here, the results show that when optimizing a constellation with respect to mutual information, a minimum number of quantization levels is required to achieve shaping gain. For generalized mutual information, the gain is maintained throughout all of the considered quantization levels. Also, the results implied that the autoencoder can adapt the constellation size to the given channel conditions.

Optimal parameters selection of back propagation algorithm in the feedforward neural network

The back propagation (BP) is one of the most widely used algorithms in the feedforward neural network (FNN), but selecting the non-optimal weights and thresholds may induce the slow convergence and local optimization. In this work, we propose an improved BP algorithm in which the loss function is constructed based on a small scale of data to seek the optimal weights and thresholds. The iteration process of the improved BP algorithm can be accelerated and the solutions will not be trapped into the local optimum. Solutions corresponding to the increasing scales of data are convergent which demonstrates the convergence of the parameters selection. Numerical simulations of elasticity and graphic reconstruction and repair confirm that the improved BP algorithm can achieve much faster convergence and higher efficiency than the conventional BP algorithm. Moreover, different initial values leading to the same optimal parameters indicates that the improved algorithm can achieve the global optimization during the learning process.

In-Depth Case Study on Artificial Neural Network Weights Optimization Using Meta-Heuristic and Heuristic Algorithmic Approach

The Meta-heuristic and Heuristic algorithms that have been introduced for deep neural network optimization is in this paper. Artificial Intelligence, and also the most used Deep Learning methods are all growing in popularity these days, thus we need faster optimization strategies for finding the results of future activities. Neural Network Optimization with Particle Swarm Optimization, Backpropagation (BP), Resilient Propagation (Rprop), and Genetic Algorithm (GA) is used for numerical analysis of different datasets and comparing each other to find out which algorithms work better for finding optimal solutions by reducing training loss. Genetic algorithm and also bio-inspired Particle Swarm Optimization is introduced in this paper. Besides, Resilient Propagation and Conventional Backpropagation algorithms which are application-specific algorithms have also been introduced. Meta-heuristic algorithms GA and PSO are a higher-level formula and problem-independent technique that may be used to a diverse number of challenges. The characteristic of Heuristic algorithms has extremely specific features that vary depending on the problem. The conventional Backpropagation (BP) based optimization, the Particle Swarm Optimization methodology, and Resilient Propagation (Rprop) are all fully presented, and how to apply these procedures in Artificial Deep Neural networks Optimization is also thoroughly described. Applied numerical simulation over several datasets proves that the Meta-heuristic algorithm Particle Swarm Optimization and also Genetic Algorithm performs better than the conventional heuristic algorithm like Backpropagation and Resilient Propagation.

Introduction of a novel evolutionary neural network for evaluating the compressive strength of concretes: A case of Rice Husk Ash concrete

The construction industry is facing challenges from the hazardous nature of Ordinary Portland Cement (OPC) production as one of the main contributors to global warming and CO2 emission. Given its increasing demand, the need to replace OPC with sustainable alternatives for green concrete production is essential. The promising results of using Rice Husk Ash (RHA) in concrete mixtures as a supplementary cementitious material (SCM) have attracted great attention. However, conventional laboratory procedures for the analysis of concrete properties like Compressive Strength (CS) are laborious and expensive. Therefore, developing a reliable and accurate model for its prediction can save time and reduce operational costs. In this study, the Artificial Neural Network (ANN) technique was hybridized with a robust optimization technique called Particle Swarm Optimization algorithm with Two Differential mutations (PSOTD) to predict the CS of RHA concrete. To compare the results of the PSOTD over classical optimization techniques, three other algorithms, including the best reported conventional Back-Propagation algorithm (i.e., Levenberg-Marquardt (LM)), Differential Evolution (DE), and Particle Swarm Optimization (PSO), were employed. For the training dataset, the coefficients of determination obtained for the ANN-LM, ANN-DE, ANN-PSO, and ANN-PSOTD were 0.9797, 0.9389, 0.9462, and 0.9803, while for the testing dataset, the values were 0.9682, 0.9198, 0.9223, and 0.9697, respectively. Overall, the error measures show the superiority of the ANN-PSOTD in predicting the CS of RHA concretes. To further enhance the prediction accuracy and reliability, the hybrid model of ANN-LM and ANN-PSOTD (i.e., ANN-PSOTD-LM) was also proposed. However, there is a trade-off between the computational time and the accuracy of models when the ANN-PSOTD-LM technique is chosen.

Many-Body Neural Network-Based Force Field for Structure-Based Coarse-Graining of Water.

High-fidelity results from atomistic simulations can only be obtained by using accurate force-field (FF) parameters. Although empirical FFs are commonly used in the modeling of atomistic systems due to their simplicity, they have many limitations inherent in the crude approximations associated with their analytical form. Recent advances in neural network-based FFs have led to more accurate FFs by using symmetry functions or full many-body expansions. However, this approach leads to several issues including the arbitrariness of the symmetry functions, and the intangible and uninterpretable interactions which are only known once the positions of all atoms are set. More importantly, training is another bottleneck, as high-quality force and energy information is required, which is usually not accessible from experimental data. To solve these issues within the context of structure-based coarse-graining methods, we switch in this work to a local-search method to target the reference structure instead of using conventional backpropagation algorithms used to target the forces and energies of the reference structure. Our FF is decomposed into two-, three-, and higher-order terms, where each term is modeled with a separate neural network. To show the versatility of our method, we study four different systems, namely, Stillinger-Weber particles as an atomistic case and three water models, namely SPC/E, MB-pol, and ab initio, as coarse-graining cases. We show the successful application of our approach, by reproducing structural properties of different water models, followed by providing insight into the role of two-and three-body interactions. The results of all models indicate that the double-well isotropic pair potential, the signature of water-like behavior in an isotropic system, vanishes upon inclusion of the three-body interaction, showing dominance of the three-body interaction over the two-body interaction in water-like behavior with the single-well isotropic pair potential.

On-chip trainable hardware-based deep Q-networks approximating a backpropagation algorithm

Reinforcement learning (RL) using deep Q-networks (DQNs) has shown performance beyond the human level in a number of complex problems. In addition, many studies have focused on bio-inspired hardware-based spiking neural networks (SNNs) given the capabilities of these technologies to realize both parallel operation and low power consumption. Here, we propose an on-chip training method for DQNs applicable to hardware-based SNNs. Because the conventional backpropagation (BP) algorithm is approximated, a performance evaluation based on two simple games shows that the proposed system achieves performance similar to that of a software-based system. The proposed training method can minimize memory usage and reduce power consumption and area occupation levels. In particular, for simple problems, the memory dependency can be significantly reduced given that high performance is achieved without using replay memory. Furthermore, we investigate the effect of the nonlinearity characteristics and two types of variation of non-ideal synaptic devices on the performance outcomes. In this work, thin-film transistor (TFT)-type flash memory cells are used as synaptic devices. A simulation is also conducted using fully connected neural network with non-leaky integrated-and-fire (I&F) neurons. The proposed system shows strong immunity to device variations because an on-chip training scheme is adopted.

Migration of Ground Penetrating Radar With Antenna Radiation Pattern Correction

Migration can reconstruct the geometric structure of a subsurface object from the ground penetrating radar (GPR) data. However, a GPR antenna is usually simplified as an ideal point/line source of normal migration algorithms, which ignore the influence of the antenna radiation pattern in subsurface soil. In this letter, the back-propagation algorithm is corrected with the analytical half-space far-field radiation pattern of an infinite line source. The superiority of this modified migration algorithm is verified through numerical, laboratory, and field experiments. The results show that the undesired diffractive artifacts at the target edges can be suppressed, while reserving the reflection amplitude in the migrated images with antenna pattern correction, compared with the conventional back-propagation and Kirchhoff algorithms.

Fast and improved backpropagation learning of multi‐layer artificial neural network using adaptive activation function

AbstractIn the conventional backpropagation (BP) learning algorithm used for the training of the connecting weights of the artificial neural network (ANN), a fixed slope−based sigmoidal activation function is used. This limitation leads to slower training of the network because only the weights of different layers are adjusted using the conventional BP algorithm. To accelerate the rate of convergence during the training phase of the ANN, in addition to updates of weights, the slope of the sigmoid function associated with artificial neuron can also be adjusted by using a newly developed learning rule. To achieve this objective, in this paper, new BP learning rules for slope adjustment of the activation function associated with the neurons have been derived. The combined rules both for connecting weights and slopes of sigmoid functions are then applied to the ANN structure to achieve faster training. In addition, two benchmark problems: classification and nonlinear system identification are solved using the trained ANN. The results of simulation‐based experiments demonstrate that, in general, the proposed new BP learning rules for slope and weight adjustments of ANN provide superior convergence performance during the training phase as well as improved performance in terms of root mean square error and mean absolute deviation for classification and nonlinear system identification problems.

Tensor Deep Learning Model for Heterogeneous Data Fusion in Internet of Things

With the rapid evolvement of the Internet and data acquisition technology as well as the continuous advancement of science and technology, the amount of data in many fields has reached the level of terabyte or petabyte and most data collection comes from the Internet of Things (IoT). The rapid advancement of IoT big data has provided valuable opportunities for the development of people in all areas of society. At the same time, it has also brought severe challenges to various types of current information processing systems. Effectively using the big data technology, discovering the hidden laws in big data, tapping the potential value of big data, and predicting the development trend of things to allocate resources more reasonably will promote the overall development of society. However, most of the IoT big data are presented as heterogeneous data, with high dimensions, different forms of expression, and a lot of redundant information. The current machine learning model works in vector space, which makes it impossible to gain big data features because vectors cannot simulate the highly nonlinear distribution of IoT big data. This paper presents a deep learning calculation model called tensor deep learning (TDL), which further improves big data feature learning and high-level feature fusion. It uses tensors to model the complexity of multisource heterogeneous data and extends the vector space data to the tensor space, when feature extraction in the tensor space is included. To fully understand the underlying data distribution, the tensor distance is adopted as the average square sum error term of the output layer reconstruction error. Based on the conventional back-propagation algorithm, this study proposes a high-order back-propagation algorithm to extend the data from the linear space to multiple linear space and train the parameters of the proposed model. Then, to evaluate its performance, the proposed TDL model is compared with the stacked auto encoder and the multimodal deep learning model. Furthermore, experiments are performed on two representative datasets, namely CUAVE and STL-10. Experimental results show that the proposed model not only excels in heterogeneous data fusion but also provides a higher recognition accuracy than the conventional deep learning model or the multimodal learning model for big data.

Prediction of concrete strength in presence of furnace slag and fly ash using Hybrid ANN-GA (Artificial Neural Network-Genetic Algorithm)

Mineral admixtures have been widely used to produce concrete. Pozzolans have been utilized as partially replacement for Portland cement or blended cement in concrete based on the materials' properties and the concrete's desired effects. Several environmental problems associated with producing cement have led to partial replacement of cement with other pozzolans. Furnace slag and fly ash are two of the pozzolans which can be appropriately used as partial replacements for cement in concrete. However, replacing cement with these materials results in significant changes in the mechanical properties of concrete, more specifically, compressive strength. This paper aims to intelligently predict the compressive strength of concretes incorporating furnace slag and fly ash as partial replacements for cement. For this purpose, a database containing 1030 data sets with nine inputs (concrete mix design and age of concrete) and one output (the compressive strength) was collected. Instead of absolute values of inputs, their proportions were used. A hybrid artificial neural network-genetic algorithm (ANN-GA) was employed as a novel approach to conducting the study. The performance of the ANN-GA model is evaluated by another artificial neural network (ANN), which was developed and tuned via a conventional backpropagation (BP) algorithm. Results showed that not only an ANN-GA model can be developed and appropriately used for the compressive strength prediction of concrete but also it can lead to superior results in comparison with an ANN-BP model.

Particle Swarm Optimization With Probability Sequence for Global Optimization

Particle Swarm Optimization (PSO) has been frequently employed to solve diversified optimization problems. Choosing initial placement for population plays an important role in meta-heuristic methods since they can significantly converge. In this study, probability distribution has been introduced to enhance the diversity of swarm and convergence speed. Population initialization method based on uniform distribution is normally used when there is no preceding knowledge available regarding the candidate solution. In this paper, a new approach to initialize population is proposed using probability sequence Weibull marked as (WI-PSO) that applies the probability distribution to generate numbers at random locations for swarm initialization. The proposed method (WI-PSO) is tested on sixteen well-known uni-modal and multi-modal benchmark test functions broadly adopted by the research community and its encouraging performance is investigated and compared with the Exponential distribution based PSO (E-PSO), Beta distribution based PSO (BT-PSO), Gamma distribution based PSO (GA-PSO) and Log-normal distribution based PSO (LN-PSO). Artificial Neural Networks (ANNs) have become the most powerful tool for classification of complex benchmark problems. We have experimented the proposed method (WI-PSO) for weight optimization of a feed-forward neural network to ensure its purity and have compared with conventional back-propagation algorithm (BPA), E-PSO, BT-PSO, GA-PSO and LN-PSO. Due to flexible behaviour in the degree of freedom, the experimental results infer the perfection and dominance of the Weibull based population initialization. The result exhibits the anticipation of influence exerted by the proposed technique on all sixteen objective functions and eight real-world benchmark data sets.

Efficient Spiking Neural Networks With Logarithmic Temporal Coding

A Spiking Neural Network (SNN) can be trained indirectly by first training an Artificial Neural Network (ANN) with the conventional backpropagation algorithm, then converting it into an SNN. The conventional rate-coding method for SNNs uses the number of spikes to encode magnitude of an activation value, and may be computationally inefficient due to the large number of spikes. Temporal-coding is typically more efficient by leveraging the timing of spikes to encode information. In this paper, we present Logarithmic Temporal Coding (LTC), where the number of spikes used to encode an activation value grows logarithmically with the activation value; and the accompanying Exponentiate-and-Fire (EF) spiking neuron model, which only involves efficient bit-shift and addition operations. Moreover, we improve the training process of ANN to compensate for approximation errors due to LTC. Experimental results indicate that the resulting SNN achieves competitive performance at significantly lower computational cost than related work.

Flower Pollination Neural Network For Heart Disease Classification

Heart Disease are among the leading cause of death worldwide. The application of artificial neural network as decision support tool for heart disease detection have been previously proposed. However, artificial neural network using conventional back propagation algorithm for error minimization and these algorithm tend to stuck at local minima. This paper proposed the use of flower pollination algorithm as a substitute to conventional back propagation algorithm for error minimization. Heart disease dataset obtain from UCI machine learning repository is used to evaluate the performance of the proposed framework. The results show that the proposed flower pollination neural network able to produce higher classification accuracy compared to the conventional back propagation neural network algorithm.

Flowshop scheduling with artificial neural networks

For effective modelling of flowshop scheduling problems, artificial neural networks (ANNs), due to their robustness, parallelism and predictive ability have been successfully used by researchers. These studies reveal that the ANNs trained with conventional back propagation (CBP) algorithm utilising the gradient descent method are commonly applied to model and solve flowshop scheduling problems. However, the existing scheduling literature has not explored the suitability of some improved neural network training algorithms, such as gradient descent with adaptive learning (GDAL), Boyden, Fletcher, Goldfarb and Shanno updated Quasi-Newton (UQ-N), and Levenberg-Marquardt (L-M) algorithms to solve the flowshop scheduling problem. In this article, we investigate the use of these training algorithms as competitive neural network learning tools to minimise makepsan in a flowshop. Based on training and testing measures, overall results from extensive computational experiments demonstrate that, in terms of the solution quality and computational effort required, the L-M algorithm performs the best followed by the UQ-N algorithm, GDAL algorithm, and the CBP algorithm. These computational results also reveal that the average percent deviation of the makespan from its best solution obtained by using the ANN trained with the L-M algorithm is the least among all examined approaches for the benchmark problem instances.

Fine-Grained Vehicle Classification With Channel Max Pooling Modified CNNs

Convolutional neural networks (CNNs) have recently shown excellent performance on the task of fine-grained vehicle classification, where the motivation is to identify the fine-grained categories of the given vehicles. Generally speaking, the main motivation of the conventional back-propagation algorithm is to optimize the loss function. The algorithm itself does not guarantee if the extracted features are discriminative for the task of classification. Intuitively, if we can learn more discriminative features with a relatively small number of feature maps, the generalization ability of the CNNs will be significantly improved. Therefore, we propose a channel max pooling (CMP) scheme, where a new layer is inserted between the fully connected layers and the convolutional layers. The proposed CMP scheme divides the feature maps into to several sub-groups. Then, it compresses the feature maps within each sub-group into a new one. The compression is carried out by selecting the maximum value among the same locations from different feature maps. Moreover, the proposed CMP layer has the advantage that it can reduce the number of parameters via reducing the number of channels in the CNNs. Experimental results on two fine-grained vehicle datasets demonstrate that the CMP modified CNNs can improve the classification accuracies on the task of fine-grained vehicle classification while a massive amount of parameters are reduced. Moreover, it has competitive performance when comparing with the-state-of-the-art methods.

Evolutionary Deep Learning with Extended Kalman Filter for Effective Prediction Modeling and Efficient Data Assimilation

With increasing concerns about infrastructure sustainability, ubiquitous sensing is an integral part of smart infrastructure in the context of smart cities. It generates large data sets containing hidden patterns and intelligence, which must be effectively extracted to derive actionable wisdom to support decision-making. Thus, it is imperative to develop intelligent data analytics to extract intelligence from data. Various data analysis methods have been developed in recent decades, but the lack of robustness and data assimilation features prevents the previously developed methods from yielding adequately accurate results for time-variant data sets over a long duration. This paper proposes an improved deep belief network (DBN), a deep machine learning model, which is integrated with genetic algorithms (GAs) and the extended Kalman filter (EKF) for effective predictive modeling and efficient data assimilation. The proposed method uses a genetic algorithm to optimize the configuration of the DBN for the given problem. Then the DBN is trained in two steps, namely pretraining layer by layer and fine-tuning with either a conventional back propagation (BP) algorithm, namely BP-DBN, or an EKF that is generalized with a new algorithm for calculating the Jacobian matrix for many-layer DBNs, namely EKF-DBN, which was tested together with BP-DBN and a recurrence neural network (RNN) on three real cases with and without data assimilation. The comparison results showed that the EKF-DBN outperforms BP-DBN and RNN in both computational efficiency and accuracy for predictive modeling. In addition, EKF-DBN generates the error covariance matrix that enables the calculation of prediction confidence interval. This can be used to detect the anomalies in a real system.

Optimal Parameter Selection Using Three-term Back Propagation Algorithm for Data Classification

The back propagation (BP) algorithm is the most popular supervised learning method for multi-layered feed forward Neural Network. It has been successfully deployed in numerous practical problems and disciplines. Regardless of its popularity, BP is still known for some major drawbacks such as easily getting stuck in local minima and slow convergence; since, it uses Gradient Descent (GD) method to learn the network. Over the years, many improved modifications of the BP learning algorithm have been made by researchers but the local minima problem remains unresolved. Therefore, to resolve the inherent problems of BP algorithm, this paper proposed BPGD-A3T algorithm where the approach introduces three adaptive parameters which are gain, momentum and learning rate in BP. The performance of the proposed BPGD-A3T algorithm is then compared with BPGD two term parameters (BPGD-2T), BP with adaptive gain (BPGD-AG) and conventional BP algorithm (BPGD) by means of simulations on classification datasets. The simulation results show that the proposed BPGD-A3T shows better performance and performed highest accuracy for all dataset as compared to other.

Forward-Looking Radar Imaging: A Comparison of Two Data Processing Strategies

This paper aims at comparing the conventional backpropagation algorithm and a microwave tomographic approach in the framework of forward-looking ground-penetrating radar imaging. The reconstruction capabilities of both methods are investigated in terms of achievable resolution limits for a sensing configuration similar to the Synchronous Impulse Reconstruction radar developed at the U.S. Army Research Laboratory. Furthermore, reconstruction results obtained by processing full-wave simulated data are shown with the goal of comparing the capability of both methods to detect buried and surface targets in scenarios emulating realistic operation conditions.