Backpropagation Rule Research Articles

Frequency-Domain and Spatial-Domain MLMVN-Based Convolutional Neural Networks

This paper presents a detailed analysis of a convolutional neural network based on multi-valued neurons (CNNMVN) and a fully connected multilayer neural network based on multi-valued neurons (MLMVN), employed here as a convolutional neural network in the frequency domain. We begin by providing an overview of the fundamental concepts underlying CNNMVN, focusing on the organization of convolutional layers and the CNNMVN learning algorithm. The error backpropagation rule for this network is justified and presented in detail. Subsequently, we consider how MLMVN can be used as a convolutional neural network in the frequency domain. It is shown that each neuron in the first hidden layer of MLMVN may work as a frequency-domain convolutional kernel, utilizing the Convolution Theorem. Essentially, these neurons create Fourier transforms of the feature maps that would have resulted from the convolutions in the spatial domain performed in regular convolutional neural networks. Furthermore, we discuss optimization techniques for both networks and compare the resulting convolutions to explore which features they extract from images. Finally, we present experimental results showing that both approaches can achieve high accuracy in image recognition.

Machines of Finite Depth: Towards a Formalization of Neural Networks

We provide a unifying framework where artificial neural networks and their architectures can be formally described as particular cases of a general mathematical construction---machines of finite depth. Unlike neural networks, machines have a precise definition, from which several properties follow naturally. Machines of finite depth are modular (they can be combined), efficiently computable, and differentiable. The backward pass of a machine is again a machine and can be computed without overhead using the same procedure as the forward pass. We prove this statement theoretically and practically via a unified implementation that generalizes several classical architectures---dense, convolutional, and recurrent neural networks with a rich shortcut structure---and their respective backpropagation rules.

Oriented Localization of Surgical Tools by Location Encoding.

Surgical tool localization is the foundation to a series of advanced surgical functions e.g. image guided surgical navigation. For precise scenarios like surgical tool localization, sophisticated tools and sensitive tissues can be quite close. This requires a higher localization accuracy than general object localization. And it is also meaningful to know the orientation of tools. To achieve these, this paper proposes a Compressive Sensing based Location Encoding scheme, which formulates the task of surgical tool localization in pixel space into a task of vector regression in encoding space. Furthermore with this scheme, the method is able to capture orientation of surgical tools rather than simply outputting horizontal bounding boxes. To prevent gradient vanishing, a novel back-propagation rule for sparse reconstruction is derived. The back-propagation rule is applicable to different implementations of sparse reconstruction and renders the entire network end-to-end trainable. Finally, the proposed approach gives more accurate bounding boxes as well as capturing the orientation of tools, and achieves state-of-the-art performance compared with 9 competitive both oriented and non-oriented localization methods on a mainstream surgical image dataset: m2cai16-tool-locations. A range of experiments support our claim that regression in CSLE space performs better than traditionally detecting bounding boxes in pixel space.

Compact Neural Architecture Designs by Tensor Representations

We propose a framework of tensorial neural networks (TNNs) extending existing linear layers on low-order tensors to multilinear operations on higher-order tensors. TNNs have three advantages over existing networks: First, TNNs naturally apply to higher-order data without flattening, which preserves their multi-dimensional structures. Second, compressing a pre-trained network into a TNN results in a model with similar expressive power but fewer parameters. Finally, TNNs interpret advanced compact designs of network architectures, such as bottleneck modules and interleaved group convolutions. To learn TNNs, we derive their backpropagation rules using a novel suite of generalized tensor algebra. With backpropagation, we can either learn TNNs from scratch or pre-trained models using knowledge distillation. Experiments on VGG, ResNet, and Wide-ResNet demonstrate that TNNs outperform the state-of-the-art low-rank methods on a wide range of backbone networks and datasets.

A Visual Designer of Layer‐wise Relevance Propagation Models

AbstractLayer‐wise Relevance Propagation (LRP) is an emerging and widely‐used method for interpreting the prediction results of convolutional neural networks (CNN). LRP developers often select and employ different relevance backpropagation rules and parameters, to compute relevance scores on input images. However, there exists no obvious solution to define a “best” LRP model. A satisfied model is highly reliant on pertinent images and designers' goals. We develop a visual model designer, named as VisLRPDesigner, to overcome the challenges in the design and use of LRP models. Various LRP rules are unified into an integrated framework with an intuitive workflow of parameter setup. VisLRPDesigner thus allows users to interactively configure and compare LRP models. It also facilitates relevance‐based visual analysis with two important functions: relevance‐based pixel flipping and neuron ablation. Several use cases illustrate the benefits of VisLRPDesigner. The usability and limitation of the visual designer is evaluated by LRP users.

Crowd Scene Analysis Encounters High Density and Scale Variation.

Crowd scene analysis receives growing attention due to its wide applications. Grasping the accurate crowd location is important for identifying high-risk regions. In this article, we propose a Compressed Sensing based Output Encoding (CSOE) scheme, which casts detecting pixel coordinates of small objects into a task of signal regression in encoding signal space. To prevent gradient vanishing, we derive our own sparse reconstruction backpropagation rule that is adaptive to distinct implementations of sparse reconstruction and makes the whole model end-to-end trainable. With the support of CSOE and the backpropagation rule, the proposed method shows more robustness to deep model training error, which is especially harmful to crowd counting and localization. The proposed method achieves state-of-the-art performance across four mainstream datasets, especially achieves excellent results in highly crowded scenes. A series of analysis and experiments support our claim that regression in CSOE space is better than traditionally detecting coordinates of small objects in pixel space for highly crowded scenes.

Graph prolongation convolutional networks: explicitly multiscale machine learning on graphs with applications to modeling of cytoskeleton

We define a novel type of ensemble graph convolutional network (GCN) model. Using optimized linear projection operators to map between spatial scales of graph, this ensemble model learns to aggregate information from each scale for its final prediction. We calculate these linear projection operators as the infima of an objective function relating the structure matrices used for each GCN. Equipped with these projections, our model (a Graph Prolongation-Convolutional Network) outperforms other GCN ensemble models at predicting the potential energy of monomer subunits in a coarse-grained mechanochemical simulation of microtubule bending. We demonstrate these performance gains by measuring an estimate of the Floating Point OPerations spent to train each model, as well as wall-clock time. Because our model learns at multiple scales, it is possible to train at each scale according to a predetermined schedule of coarse vs. fine training. We examine several such schedules adapted from the algebraic multigrid literature, and quantify the computational benefit of each. We also compare this model to another model which features an optimized coarsening of the input graph. Finally, we derive backpropagation rules for the input of our network model with respect to its output, and discuss how our method may be extended to very large graphs.

Yarn Tensile Properties Modeling Using Artificial Intelligence

Yarn quality is an important factor that influences the subsequent products quality. Employing artificial intelligence technologies can lead to more objective yarn testing systems and higher product specifications that meet the demands of the manufacturing and end-user. This work was carried out in order to employ image processing and artificial neural networks for modeling yarn tenacity and elongation% for different yarn types using a feasible method in cost and time. To evaluate yarn parameters, yarn samples were collected from the yarn testing lab and wound around a blackboard using the appearance tester. Sample images were captured and preprocessed and vectors were defined and used as network inputs. Feed-forward neural networks trained with the back-propagation rule were employed. Two systems were developed; the first one was used to assess the elongation% and tenacity of cotton yarns and the second was utilized for blended yarn parameters evaluation. By applying image enhancement combined with a multilayer neural network, better results were obtained for estimating different yarn parameters. Therefore, a modeling system can be implemented successfully. It was also proved that the system can be applied for diverse yarn types and different tested parameters. This research is an attempt to model several yarn properties using a cost effective system that employs machine vision and learning. In addition, utilizing these techniques can improve properties assessment and overall quality control.

Neural network based non‐invasive method to detect anemia from images of eye conjunctiva

AbstractDetection of anemia can be done by examining the hemoglobin concentration level in the blood using complete blood count, which is an invasive, time‐consuming, and costly technique. Preliminary methods for detecting anemia include examining the color of the palpebral conjunctiva, which is a non‐invasive method, but color perception may vary from person to person. This study aims to develop a computerized non‐invasive technique for anemia detection. We propose a novel machine learning model using the artificial neural network to detect anemic patients from the images of eye conjunctiva. Since limited and small dataset has been used in the earlier approaches, this may cause over fitting of the model. We have improved the number of available training images using image augmentation techniques. To standardize a non‐invasive method, we have used computer vision algorithms for preprocessing and feature extraction. This article derives the backpropagation rules mathematically for adjusting the weights for the proposed neural network model. After hyper parameter tuning and using the mathematically derived backpropagation rules, the model was able to achieve the best accuracy of 97.00% with sensitivity 99.21% and specificity 95.42% on the created dataset.

Training Convolutional Neural Networks and Compressed Sensing End-to-End for Microscopy Cell Detection.

Automated cell detection and localization from microscopy images are significant tasks in biomedical research and clinical practice. In this paper, we design a new cell detection and localization algorithm that combines deep convolutional neural network (CNN) and compressed sensing (CS) or sparse coding (SC) for end-to-end training. We also derive, for the first time, a backpropagation rule, which is applicable to train any algorithm that implements a sparse code recovery layer. The key innovation behind our algorithm is that the cell detection task is structured as a point object detection task in computer vision, where the cell centers (i.e., point objects) occupy only a tiny fraction of the total number of pixels in an image. Thus, we can apply compressed sensing (or equivalently SC) to compactly represent a variable number of cells in a projected space. Subsequently, CNN regresses this compressed vector from the input microscopy image. The SC/CS recovery algorithm ( L 1 optimization) can then recover sparse cell locations from the output of CNN. We train this entire processing pipeline end-to-end and demonstrate that end-to-end training improves accuracy over a training paradigm that treats CNN and CS-recovery layers separately. We have validated our algorithm on five benchmark datasets with excellent results.

Backpropagation algorithm with fractional derivatives

The paper presents a model of a neural network with a novel backpropagation rule, which uses a fractional order derivative mechanism. Using the Grunwald Letnikow definition of the discrete approximation of the fractional derivative, the author proposed the smooth modeling of the transition functions of a single neuron. On this basis, a new concept of a modified backpropagation algorithm was proposed that uses the fractional derivative mechanism both for modeling the dynamics of individual neurons and for minimizing the error function. The description of the signal flow through the neural network and the mechanism of smooth shape control of the activation functions of individual neurons are given. The model of minimization of the error function is presented, which takes into account the possibility of changes in the characteristics of individual neurons. For the proposed network model, example courses of the learning processes are presented, which prove the convergence of the learning process for different shapes of the transition function. The proposed algorithm allows the learning process to be conducted with a smooth modification of the shape of the transition function without the need for modifying the IT model of the designed neural network. The proposed network model is a new tool that can be used in signal classification tasks.

A comparative prediction for tensile properties of ternary blended open-end rotor yarns using regression and neural network models

This study focused on predicting tensile properties of PES/CV/PAN blended Open-End Rotor yarns. The effective factors were fiber blend ratios (six stages from 0 to 100%), linear density (three count levels), mixing method (carding machine and drawframe), and number of passages in drawframe (one and two times) as production parameters. We performed a stepwise multiple linear regression (MLR) analysis and established an artificial neural network (ANN) model that trained with backpropagation rule as Levenberg–Marquardt. Then, we conducted a comparative analysis for both models in terms of prediction performance. As a result, ANN has given a slightly better prediction values than MLR for breaking strength but significantly better prediction values for breaking elongation.

Event-Driven Random Back-Propagation: Enabling Neuromorphic Deep Learning Machines.

An ongoing challenge in neuromorphic computing is to devise general and computationally efficient models of inference and learning which are compatible with the spatial and temporal constraints of the brain. One increasingly popular and successful approach is to take inspiration from inference and learning algorithms used in deep neural networks. However, the workhorse of deep learning, the gradient descent Gradient Back Propagation (BP) rule, often relies on the immediate availability of network-wide information stored with high-precision memory during learning, and precise operations that are difficult to realize in neuromorphic hardware. Remarkably, recent work showed that exact backpropagated gradients are not essential for learning deep representations. Building on these results, we demonstrate an event-driven random BP (eRBP) rule that uses an error-modulated synaptic plasticity for learning deep representations. Using a two-compartment Leaky Integrate & Fire (I&F) neuron, the rule requires only one addition and two comparisons for each synaptic weight, making it very suitable for implementation in digital or mixed-signal neuromorphic hardware. Our results show that using eRBP, deep representations are rapidly learned, achieving classification accuracies on permutation invariant datasets comparable to those obtained in artificial neural network simulations on GPUs, while being robust to neural and synaptic state quantizations during learning.

Competitive co-evolution of multi-layer perceptron classifiers

This paper analyses the competitive approach to the co-evolutionary training of multi-layer perceptron classifiers. Two algorithms were tested: the first opposes a population of classifiers to a population of training patterns; the second pits a population of classifiers against a population of subsets of training patterns. The classifiers are regarded as predators that need to ‘capture’ (correctly categorise) the prey (training patterns). Success for the predators is measured on their ability to capture prey. Success for the prey is measured on their ability to escape predation (be misclassified). The aim of the procedure is to create an evolutionary tug-of-war between the best classifiers and the most difficult data samples, increasing the efficiency and accuracy of the learning process. The two co-evolutionary algorithms were tested on a number of well-known benchmarks and on several artificial data sets modelling different kinds of common classification problems such as overlapping data categories, noisy training inputs, and unbalanced data classes. The performance of the co-evolutionary methods was compared with that of two traditional training techniques: the standard backpropagation rule and a conventional evolutionary algorithm. The co-evolutionary procedures achieved top accuracy in all classification problems. They particularly excelled on data sets containing noisy training inputs, where they outperformed the backpropagation rule, and on tasks involving unbalanced data classes, where they outperformed both backpropagation and the conventional evolutionary algorithm. Compared to the standard evolutionary algorithm, the co-evolutionary procedures were able to obtain similar or superior learning accuracies, whilst needing considerably less presentations of the training patterns. This economy in the use of training patterns translated into significant savings in computational overheads and algorithms running time.

Transfer of conflict and cooperation from experienced games to new games: a connectionist model of learning

The question of whether, and if so how, learning can be transfered from previously experienced games to novel games has recently attracted the attention of the experimental game theory literature. Existing research presumes that learning operates over actions, beliefs or decision rules. This study instead uses a connectionist approach that learns a direct mapping from game payoffs to a probability distribution over own actions. Learning is operationalized as a backpropagation rule that adjusts the weights of feedforward neural networks in the direction of increasing the probability of an agent playing a myopic best response to the last game played. One advantage of this approach is that it expands the scope of the model to any possible n × n normal-form game allowing for a comprehensive model of transfer of learning. Agents are exposed to games drawn from one of seven classes of games with significantly different strategic characteristics and then forced to play games from previously unseen classes. I find significant transfer of learning, i.e., behavior that is path-dependent, or conditional on the previously seen games. Cooperation is more pronounced in new games when agents are previously exposed to games where the incentive to cooperate is stronger than the incentive to compete, i.e., when individual incentives are aligned. Prior exposure to Prisoner's dilemma, zero-sum and discoordination games led to a significant decrease in realized payoffs for all the game classes under investigation. A distinction is made between superficial and deep transfer of learning both—the former is driven by superficial payoff similarities between games, the latter by differences in the incentive structures or strategic implications of the games. I examine whether agents learn to play the Nash equilibria of games, how they select amongst multiple equilibria, and whether they transfer Nash equilibrium behavior to unseen games. Sufficient exposure to a strategically heterogeneous set of games is found to be a necessary condition for deep learning (and transfer) across game classes. Paradoxically, superficial transfer of learning is shown to lead to better outcomes than deep transfer for a wide range of game classes. The simulation results corroborate important experimental findings with human subjects, and make several novel predictions that can be tested experimentally.

Neural Network based Plant Identification using Leaf Characteristics Fusion

computerized method for recognizing plant leaf based on their images is proposed. Plant classification is based on leaf identification which has broad application on prospective in medicine and agriculture. Plant leaf images corresponding to six plant types are taken using a digital camera which are examined using three different modeling techniques, first based on Multi Layer Perceptron (MLP) Neural network and second on Normalized Cubic Spline Feed Forward Neural network (NCS-FNN) and third on proposed NCS-FNN for real data. Correlation based feature selection (CFS) is considered to produce a ranked list of attributes. Matlab is used to extract the leaf features such as edge and texture. Edge and texture are the important visual attribute which can be used to describe the pixel organization in an image. Further to increase the accuracy in NCS-FNN the neural network is trained using a back propagation rule by back propagating errors and changing weights of node. The dataset consists of 197 images which are divided into six classes.

Nonlinear optimisation using directional step lengths based on RPROP

A search method based on the backpropagation rule commonly used for training neural networks is proposed here for the optimisation of smooth nonlinear functions. The use of the Resilient backPROPagation (RPROP) heuristic rule for local minimisation is described. The details of employing the directional step length determined by RPROP along with a simple restarting scheme are provided. In the approach proposed here direct use of the directional step determined by the heuristic without using any line search conditions takes place. The overall algorithm has been tested on a number of benchmark functions found in the literature with very positive results. The test problems’ dimension ranges from 100 to 50,000. The results obtained show that the suggested search direction method results to a highly efficient algorithm suitable for large scale optimisation.

Probabilistic Simulation Approach for Distributed Generation (DG) Capacity Evaluation using Artificial Neural Network Representation of Load Duration Curve

This paper describes a methodology for adequacy evaluation of capacity of a distribution system having distributed generation (DG). The load model used is a load duration curve and is represented in closed from using back-propagation rule based ANN. Monte Carlo simulation has been used to evaluate probability of failure and expected demand not supplied (EDNS). Discrete and continuous probability distribution functions have been used to represent DG and distribution substation capacities. Procedure has been explained using a sample composite distribution system. Normal distribution function has been considered for the capacities of distribution substations.

Automatic design of control systems for robot manipulators using the bees algorithm

This paper proves the capability of the bees algorithm to solve complex parameter optimization problems for robot manipulator control. Two applications are presented. The first case considers the modelling of the inverse kinematics of an articulated robot arm using neural networks. The weights of the connections between the nodes need to be set so as to minimize the difference between the neural network model and the desired behaviour. In the proposed example, the bees algorithm is used to train three multilayer perceptrons to learn the inverse kinematics of the joints of a three-link manipulator. The second case considers the design of a hierarchical proportional–integral–derivative (PID) controller for a flexible single-link robot manipulator. The six gains of the PID controller need to be optimized so as to minimize positional inaccuracies and vibrations. Experimental tests demonstrated the validity of the proposed approach. In the first case, the bees algorithm proved very effective at optimizing the neural network models. Compared with the results obtained employing the standard back-propagation rule and an evolutionary algorithm, the bees algorithm obtained superior results in terms of training accuracy and robustness. In the second case, the proposed method demonstrated remarkable efficiency and consistency in the tuning of the PID controller parameters. In 50 independent optimization trials, the PID controllers designed using the bees algorithm consistently outperformed a robot controller designed using a standard manual technique.

Active Noise Control Algorithm Based on a Neural Network and Nonlinear Input-Output System Identification Model

The development of digital signal processors and the increase in their computing capabilities bring opportunities to employ algorithms with multiple variable parameters in active noise control systems. Of particular interest are the algorithms based on artificial neural networks. This paper presents an active noise control algorithm based on a neural network and a nonlinear input-output system identification model. The purpose of the algorithm is an active noise control system with a nonlinear primary path. The algorithm uses the NARMAX system identification model. The neural network employed in the proposed algorithm is a multilayer perceptron. The error backpropagation rule with adaptive learning rate is employed to update the weight of the neural network. The performance of the proposed algorithm has been tested by numerical simulations. Results for narrow-band input signals and nonlinear primary path are presented below.