Training Multilayer Neural Networks Research Articles

Hardware implementation of backpropagation using progressive gradient descent for in situ training of multilayer neural networks.

Neural network training can be slow and energy-expensive due to the frequent transfer of weight data between digital memory and processing units. Neuromorphic systems can accelerate neural networks by performing multiply-accumulate operations in parallel using nonvolatile analog memory. However, executing the widely used backpropagation training algorithm in multilayer neural networks requires information-and therefore storage-of the partial derivatives of the weight values preventing suitable and scalable implementation in hardware. Here, we propose a hardware implementation of the backpropagation algorithm that progressively updates each layer using in situ stochastic gradient descent, avoiding this storage requirement. We experimentally demonstrate the in situ error calculation and the proposed progressive backpropagation method in a multilayer hardware-implemented neural network. We confirm identical learning characteristics and classification performance compared to conventional backpropagation in software. We show that our approach can be scaled to large and deep neural networks, enabling highly efficient training of advanced artificial intelligence computing systems.

Inferential control strategies using neural soft sensor in a high purity distillation column

High-purity distillation columns are processes in which you want to minimally increase the purity of a key component by separation. These processes are sensitive to disturbances, where small changes in the feed flowrate stream cause drastic changes in product compositions. Furthermore, when one wants to apply traditional control and optimization techniques to these processes, some difficulties have to be faced: the process is generally non-linear and has a long response time, there are many immeasurable disturbances, it is difficult to keep the process in steady-state, and the bottom and top compositions are highly coupled. Therefore, this paper presents a methodology for the development of soft sensors (SS), here in applied to an industrial 1,2-Dichloroethane separation plant. The process was simulated and validated with real data from the industrial plant. In the step second an algorithm based on multivariate statistics was developed for the selection of inputs SS variables. The configuration for training multilayer artificial neural network (ANN's) was optimize, accounting for the ANN's prediction capacity of responses and the rejection of disturbances inserted in the process. Thereafter, a temperature control was implemented to maintain the impurity compositions within specifications but failed due to the small temperature variations in the high purity column. To overcome these difficulty three inferential control strategies (impurity composition estimated by the SS) were implemented: a classic feedback control, a cascade control, and a ratio-cascade control. All control strategies acted to minimize the process disturbances effects. However, the inferential ratio-cascade control shown to be the most robust, because of the lowest integral error criteria value and percentage overshoot.

Training multi-layer spiking neural networks with plastic synaptic weights and delays.

Spiking neural networks are usually considered as the third generation of neural networks, which hold the potential of ultra-low power consumption on corresponding hardware platforms and are very suitable for temporal information processing. However, how to efficiently train the spiking neural networks remains an open question, and most existing learning methods only consider the plasticity of synaptic weights. In this paper, we proposed a new supervised learning algorithm for multiple-layer spiking neural networks based on the typical SpikeProp method. In the proposed method, both the synaptic weights and delays are considered as adjustable parameters to improve both the biological plausibility and the learning performance. In addition, the proposed method inherits the advantages of SpikeProp, which can make full use of the temporal information of spikes. Various experiments are conducted to verify the performance of the proposed method, and the results demonstrate that the proposed method achieves a competitive learning performance compared with the existing related works. Finally, the differences between the proposed method and the existing mainstream multi-layer training algorithms are discussed.

Detection of Breast Cancer using Deep Learning Techniques

Because of the current population boom in health research, early sickness diagnosis has become a vital concern. As the population expands, the risk of dying from breast cancer rises dramatically. Breast carcinoma has been identified whenever the second most dangerous of the previously described malignancies. The researcher automated illness detection system assists medical practitioners in disease diagnosis, provides consistent, effective, and punctual intervention, and reduces the risk of death. Any disease that is diagnosed appropriately and promptly may be treated with minimal human intervention. An overwhelming majority of people are unaware of their illness until it becomes chronic. It increases the world mortality rate. Breast carcinoma has emerged as one of the increasingly rare diseases that may be treated if detected early enough and before it spreads to other regions of the body. Breast carcinoma constitutes one of the most frequent malignancies in women globally, and early identification is critical for improving survival and treatment success. Breast cancer detection technologies in areas like mammography and ultrasound have limits outside in the sense of preciseness as well as sensitivity. Deep learning algorithms have begun to emerge as a potential strategy for enhancing the degree of certainty and efficiency belonging to breast cancer diagnosis in recent years. Deep learning is an artificial intelligence area that focuses down training multi-layer neural networks to gain knowledge of and extract complicated patterns from big datasets. Researchers have developed sophisticated models suited to successfully diagnosing breast cancer from several medical imaging modalities, which might involve mammograms, MRI scans, additionally histopathological images, by utilizing the power throughout deep learning algorithms. Breast carcinoma detection is an important subject of study with significant public health implications. Deep learning techniques, a subset of computational neuroscience (AI), demonstrate excellent results in identifying and identifying cases of breast cancer. Deep learning breast cancer detection technologies have significant research repercussions due to the fact that they enable early diagnosis, enhance exactness, automate screening processes, give personalized treatment, and together with expand healthcare services to underserved areas. Persevered research in this area has the potential to change breast cancer diagnostics, resulting in better patient outcomes and, eventually, lifesaving. In this research we will be using The Weighted Product Model. The Weighted Product Model (WPM) represents a decision-making approach that uses numerous criteria to evaluate and rank options. It applies a multiple-criteria analysis approach that considers the value or weight assigned to every criterion as well as the effectiveness or score residing in every possible outcome on those criteria. Taken of Alternative Parameters SVM, Random Forest, Logistic Regression, KNN, Naive Bayes. Taken of Evaluation Parameters Accuracy, Recall, Precision, FI-Score, ROC AUC. As per Weighted Normalized Decision Matrix we get to know that SVM got more value were Random Forest, Logistic Regression, Naive Bayes got less value. From the above results I conclude that as per Weighted Normalized Decision Matrix we get to know that SVM got more value than others.

Detecting fake news with deep neural networks

The raise of social media and the easy access to obtain and spread news has increased the probability of various news dissemination. However, an undetermined factor in usage scenario is the authenticity of spreading news, which has posed detrimental effects on individuals and society in current scenario. It is imperative to develop proper mechanism for fake news detection. These years, with the sharp development of deep learning, referring to the method of training multi-layer artificial neural networks, some researchers have utilized deep learning in fake news detection and classification. Although many studies have achieved a preeminent level and high accuracy, scarcity of review work regarding fake news detection with deep learning is still an important problem. This work urges to derive a comprehensive review of fake news detection and classification techniques for further study in the aspect, and mechanism in various studies for sake of raising accuracy. This work also figure out some possible challenges with existing limitation based on previous research.

Training Multilayer Neural Network Based on Optimal Control Theory for Limited Computational Resources

Backpropagation (BP)-based gradient descent is the general approach to train a neural network with a multilayer perceptron. However, BP is inherently slow in learning, and it sometimes traps at local minima, mainly due to a constant learning rate. This pre-fixed learning rate regularly leads the BP network towards an unsuccessful stochastic steepest descent. Therefore, to overcome the limitation of BP, this work addresses an improved method of training the neural network based on optimal control (OC) theory. State equations in optimal control represent the BP neural network’s weights and biases. Meanwhile, the learning rate is treated as the input control that adapts during the neural training process. The effectiveness of the proposed algorithm is evaluated on several logic gates models such as XOR, AND, and OR, as well as the full adder model. Simulation results demonstrate that the proposed algorithm outperforms the conventional method in terms of improved accuracy in output with a shorter time in training. The training via OC also reduces the local minima trap. The proposed algorithm is almost 40% faster than the steepest descent method, with a marginally improved accuracy of approximately 60%. Consequently, the proposed algorithm is suitable to be applied on devices with limited computation resources, since the proposed algorithm is less complex, thus lowering the circuit’s power consumption.

Identification of Non-Stationary Objects Based on Training a Neural Network and Tuning the Parameters of a Generalized Model

Methods and algorithms have been developed for identifying non-stationary objects of various types using statistical, dynamic, neural network models, which are taken into account when solving problems of conditions of a priori insufficiency, uncertainty, low reliability of data. Mechanisms are proposed that provide effective identification based on combining the features of dynamic models with the properties of random time series. The possibilities of algorithms based on mechanisms that use statistical, dynamic, specific data characteristics, as well as the properties of self-adaptation, approximation, organization, self-learning of neural networks have been expanded. A generalized function identification algorithm has been developed and its functions have been expanded by adaptive segmentation of time series, setting the informative interval of element values, the size of the training set, training multilayer neural networks, database, and knowledge base. The training algorithms for a three-layer neural network are modified based on the mechanisms for regulating interneuronal connections in layers, weight coefficients of neurons, variable activation functions, network architecture, and superposition of continuous input-output dependencies. A software package for identifying random time series in the C++ language in the CUDA parallel computing environment has been developed to predict the annual power consumption of the industrial zone of the Samarkand region using software tools for data preprocessing, filtering, smoothing; determining the boundaries of the informative interval of time series elements.

Sonar Data Classification Using Neural Network Trained by Hybrid Dragonfly and Chimp Optimization Algorithms

This paper proposes a hybrid Dragonfly Algorithm (DA) for training Multi-Layer Perceptron Neural Network (MLP NN) to design the classifier for solving complicated problems and distinguishing the real target from liars’ targets in sonar applications. Due to improve the cost computation and reducing the waste of time, a modified low-cost DA is designed for evaluation. To assess the accuracy of the technique, some well-known meta-heuristic trainers include Chimp Optimization Algorithm, Gravitational Search Algorithm (GSA), DA, and Particle Swarm Optimization (PSO) compared to show the accuracy of similar algorithms. DA and ChoA algorithms have remarkable features and a hybrid algorithm of them is proposed. The proposed classifier has acceptable performance, and two standard benchmark datasets are used to evaluate performance. The results show that the modified hybrid DA-ChoA has 15% less time-consuming and 4% better performance rather than the original dragonfly method.

Learning curves of generic features maps for realistic datasets with a teacher-student model* *This article is an updated version of: Loureiro B, Gerbelot C, Cui H, Goldt S, Krzakala F, Mezard M and Zdeborová L 2021 Learning curves of generic features maps for realistic datasets with a teacher-student model Advances in Neural Information Processing Systems vol 34 ed M Ranzato,

Teacher-student models provide a framework in which the typical-case performance of high-dimensional supervised learning can be described in closed form. The assumptions of Gaussian i.i.d. input data underlying the canonical teacher-student model may, however, be perceived as too restrictive to capture the behaviour of realistic data sets. In this paper, we introduce a Gaussian covariate generalisation of the model where the teacher and student can act on different spaces, generated with fixed, but generic feature maps. While still solvable in a closed form, this generalization is able to capture the learning curves for a broad range of realistic data sets, thus redeeming the potential of the teacher-student framework. Our contribution is then two-fold: first, we prove a rigorous formula for the asymptotic training loss and generalisation error. Second, we present a number of situations where the learning curve of the model captures the one of a realistic data set learned with kernel regression and classification, with out-of-the-box feature maps such as random projections or scattering transforms, or with pre-learned ones—such as the features learned by training multi-layer neural networks. We discuss both the power and the limitations of the framework.

Ensembles of Biologically Inspired Optimization Algorithms for Training Multilayer Perceptron Neural Networks

Artificial neural networks have proven to be effective in a wide range of fields, providing solutions to various problems. Training artificial neural networks using evolutionary algorithms is known as neuroevolution. The idea of finding not only the optimal weights and biases of a neural network but also its architecture has drawn the attention of many researchers. In this paper, we use different biologically inspired optimization algorithms to train multilayer perceptron neural networks for generating regression models. Specifically, our contribution involves analyzing and finding a strategy for combining several algorithms into a hybrid ensemble optimizer, which we apply for the optimization of a fully connected neural network. The goal is to obtain good regression models for studying and making predictions for the process of free radical polymerization of methyl methacrylate (MMA). In the first step, we use a search procedure to find the best parameter values for seven biologically inspired optimization algorithms. In the second step, we use a subset of the best-performing algorithms and improve the search capability by combining the chosen algorithms into an ensemble of optimizers. We propose three ensemble strategies that do not involve changes in the logic of optimization algorithms: hybrid cascade, hybrid single elite solution, and hybrid multiple elite solutions. The proposed strategies inherit the advantages of each individual optimizer and have faster convergence at a computational effort very similar to an individual optimizer. Our experimental results show that the hybrid multiple elite strategy ultimately produces neural networks which constitute the most dependable regression models for the aforementioned process.

Deep learning via message passing algorithms based on belief propagation

Message-passing algorithms based on the belief propagation (BP) equations constitute a well-known distributed computational scheme. They yield exact marginals on tree-like graphical models and have also proven to be effective in many problems defined on loopy graphs, from inference to optimization, from signal processing to clustering. The BP-based schemes are fundamentally different from stochastic gradient descent (SGD), on which the current success of deep networks is based. In this paper, we present and adapt to mini-batch training on GPUs a family of BP-based message-passing algorithms with a reinforcement term that biases distributions towards locally entropic solutions. These algorithms are capable of training multi-layer neural networks with performance comparable to SGD heuristics in a diverse set of experiments on natural datasets including multi-class image classification and continual learning, while being capable of yielding improved performances on sparse networks. Furthermore, they allow to make approximate Bayesian predictions that have higher accuracy than point-wise ones.

Pattern recognition and diagnosis of short and open circuit faults inverter in induction motor drive using neural networks

Nowadays, feeding induction motors with voltage source inverters under faulty conditions is a major challenge. For this reason, electrical systems must be well thought out to provide good diagnostics for these elements. Consequently, the early detection of faults is very important to establish strategies that allow us to control the operation and take preventive measures to avoid frequent failures. Our aim in this paper is to train multilayer neural networks using features extracted from currents and voltages measurements to detect and classify open and short-circuit switch faults in source voltage inverters. Simulation results show that instead of using several types of features extracted from measurements of several signal cycles as in previous works, a two-component feature obtained from one cycle is sufficient to obtain an excellent accuracy. The normalized mean Clark currents and the power spectrum using the fast Fourier transform have been used as features for open switches and short-circuit faults respectively.

Comparative Evaluation of Predicting Energy Consumption of Absorption Heat Pump with Multilayer Shallow Neural Network Training Algorithms

The performance of various multilayer neural network algorithms to predict the energy consumption of an absorption chiller in an air conditioning system under the same conditions was compared and evaluated in this study. Each prediction model was created using 12 representative multilayer shallow neural network algorithms. As training data, about a month of actual operation data during the heating period was used, and the predictive performance of 12 algorithms according to the training size was evaluated. The prediction results indicate that the error rates using the measured values are 0.09% minimum, 5.76% maximum, and 1.94 standard deviation (SD) for the Levenberg–Marquardt backpropagation model and 0.41% minimum, 5.05% maximum, and 1.68 SD for the Bayesian regularization backpropagation model. The conjugate gradient with Polak–Ribiére updates backpropagation model yielded lower values than the other two models, with 0.31% minimum, 5.73% maximum, and 1.76 SD. Based on the results for the predictive performance evaluation index, CvRMSE, all other models (conjugate gradient with Fletcher–Reeves updates backpropagation, one-step secant backpropagation, gradient descent with momentum and adaptive learning rate backpropagation, gradient descent with momentum backpropagation) except for the gradient descent backpropagation model yielded results that satisfy ASHRAE (American Society of Heating, Refrigerating and Air-Conditioning Engineers) Guideline 14. The results of this study confirm that the prediction performance may differ for each multilayer neural network training algorithm. Therefore, selecting the appropriate model to fit the characteristics of a specific project is essential.

Approaches to the choice of tools for constructing a methodology for learning to work with Big Data

The article actualizes the need for teaching students to work with Big Data technologies. Big Data is a promising and fundamental industry that requires a large number of qualified specialists in various fields. The aim of the work is to describe the concept of determining a set of hardware, software, algorithmic and methodological tools (taking into account the contingent of students and the capabilities of the educational institution) for building a methodology for teaching a discipline related to the study of Big Data processing methods. There are two main sectors of stakeholders who need specialists in the field of Big Data. A detailed comparative analysis of software solutions that support Big Data processing is carried out. The article describes the methodology for constructing a course for teaching students technologies for processing and analyzing Big Data. A plan for organizing a lecture course and laboratory practice with consideration of subtasks is proposed for students to perform during training. The composition and methodology of independent work of students in the discipline related to the study of Big Data, using a learning management system such as Moodle, are discussed. An example of implementing data processing by means of the RapidMiner Studio package using a multi-layer neural network training algorithm using the error back propagation method is presented.

On the application of neural network technologies for control problems in cognitive transportation systems

In the work, the functions of cognition are brought in line with the aspects of the functioning of cognitive transport systems and technologies. We consider the mathematical apparatus of cognitive technologies – neural network control systems, based on trained multi-layer neural networks of direct action in the class of adaptive control systems for dynamic objects. The possibility and variants of their inclusion in the control system of the transport system are clarified. On the basis of the considered approach, a country road recognition system based on a convolutional neural network has been developed, which will automatically segment rural roads using satellite images and build a road network diagram to analyze the spatial development of transport networks and to control unmanned vehicles. The development of a neural network based on U-net was carried out in Python 3x. The training set consists of 880 images prepared by hand mark up. The accuracy of the developed model when testing on prepared samples was 64%. According to the results of the study, conclusions were drawn and prospects for further functional development of the developed tools were determined.

Supervised Learning With First-to-Spike Decoding in Multilayer Spiking Neural Networks.

Experimental studies support the notion of spike-based neuronal information processing in the brain, with neural circuits exhibiting a wide range of temporally-based coding strategies to rapidly and efficiently represent sensory stimuli. Accordingly, it would be desirable to apply spike-based computation to tackling real-world challenges, and in particular transferring such theory to neuromorphic systems for low-power embedded applications. Motivated by this, we propose a new supervised learning method that can train multilayer spiking neural networks to solve classification problems based on a rapid, first-to-spike decoding strategy. The proposed learning rule supports multiple spikes fired by stochastic hidden neurons, and yet is stable by relying on first-spike responses generated by a deterministic output layer. In addition to this, we also explore several distinct, spike-based encoding strategies in order to form compact representations of presented input data. We demonstrate the classification performance of the learning rule as applied to several benchmark datasets, including MNIST. The learning rule is capable of generalizing from the data, and is successful even when used with constrained network architectures containing few input and hidden layer neurons. Furthermore, we highlight a novel encoding strategy, termed “scanline encoding,” that can transform image data into compact spatiotemporal patterns for subsequent network processing. Designing constrained, but optimized, network structures and performing input dimensionality reduction has strong implications for neuromorphic applications.

Industrial Robot Trajectory Tracking Control Using Multi-Layer Neural Networks Trained by Iterative Learning Control

Fast and precise robot motion is needed in many industrial applications. Most industrial robot motion controllers allow externally commanded motion profiles, but the trajectory tracking performance is affected by the robot dynamics and joint servo controllers, to which users have no direct access and about which they have little information. The performance is further compromised by time delays in transmitting the external command as a setpoint to the inner control loop. This paper presents an approach for combining neural networks and iterative learning controls to improve the trajectory tracking performance for a multi-axis articulated industrial robot. For a given desired trajectory, the external command is iteratively refined using a high-fidelity dynamical simulator to compensate for the robot inner-loop dynamics. These desired trajectories and the corresponding refined input trajectories are then used to train multi-layer neural networks to emulate the dynamical inverse of the nonlinear inner-loop dynamics. We show that with a sufficiently rich training set, the trained neural networks generalize well to trajectories beyond the training set as tested in the simulator. In applying the trained neural networks to a physical robot, the tracking performance still improves but not as much as in the simulator. We show that transfer learning effectively bridges the gap between simulation and the physical robot. Finally, we test the trained neural networks on other robot models in simulation and demonstrate the possibility of a general purpose network. Development and evaluation of this methodology are based on the ABB IRB6640-180 industrial robot and ABB RobotStudio software packages.

An effective SteinGLM initialization scheme for training multi-layer feedforward sigmoidal neural networks

Network initialization is the first and critical step for training neural networks. In this paper, we propose a novel network initialization scheme based on the celebrated Stein’s identity. By viewing multi-layer feedforward sigmoidal neural networks as cascades of multi-index models, the projection weights to the first hidden layer are initialized using eigenvectors of the cross-moment matrix between the input’s second-order score function and the response. The input data is then forward propagated to the next layer and such a procedure can be repeated until all the hidden layers are initialized. Finally, the weights for the output layer are initialized by generalized linear modeling. Such a proposed SteinGLM method is shown through extensive numerical results to be much faster and more accurate than other popular methods commonly used for training neural networks.

Efficient Training of Multi-Layer Neural Networks to Achieve Faster Validation

Artificial neural networks (ANNs) are one of the hottest topics in computer science and artificial intelligence due to their potential and advantages in analyzing real-world problems in various disciplines, including but not limited to physics, biology, chemistry, and engineering. However, ANNs lack several key characteristics of biological neural networks, such as sparsity, scale-freeness, and small-worldness. The concept of sparse and scale-free neural networks has been introduced to fill this gap. Network sparsity is implemented by removing weak weights between neurons during the learning process and replacing them with random weights. When the network is initialized, the neural network is fully connected, which means the number of weights is four times the number of neurons. In this study, considering that a biological neural network has some degree of initial sparsity, we design an ANN with a prescribed level of initial sparsity. The neural network is tested on handwritten digits, Arabic characters, CIFAR-10, and Reuters newswire topics. Simulations show that it is possible to reduce the number of weights by up to 50% without losing prediction accuracy. Moreover, in both cases, the testing time is dramatically reduced compared with fully connected ANNs.

A neural adaptive level set method for wildland forest fire tracking

Tracking of smoke and fire in videos can provide helpful regional measures to evaluate precisely damages caused by fires. In security applications, real-time video segmentation of both fire and smoke regions represents a crucial operation to avoid disaster. In this paper, we propose a robust tracking method for fire regions in forest wildfire videos using neural pixel classification approach combined with a non-linear adaptive level set method based on the Bayesian rule. Firstly, an estimation function is built with chromatic and statistical features using linear discriminant analysis and a trained multilayer neural network in order to get a preliminary fire localisation in each frame. This function is used to compute initial curve and the level set evolution parameters providing fast refined fire segmentation in each processed frame. The experimental results of the proposed method prove its accuracy and robustness when tested on different varieties of wildfire-smoke scenarios.