Learning In Feedforward Neural Networks Research Articles

Topology of Learning in Feedforward Neural Networks.

Understanding how neural networks learn remains one of the central challenges in machine learning research. From random at the start of training, the weights of a neural network evolve in such a way as to be able to perform a variety of tasks, such as classifying images. Here, we study the emergence of structure in the weights by applying methods from topological data analysis. We train simple feedforward neural networks on the MNIST data set and monitor the evolution of the weights. When initialized to zero, the weights follow trajectories that branch off recurrently, thus generating trees that describe the growth of the effective capacity of each layer. When initialized to tiny random values, the weights evolve smoothly along 2-D surfaces. We show that natural coordinates on these learning surfaces correspond to important factors of variation.

An Intelligent Task Scheduling Mechanism for Autonomous Vehicles via Deep Learning

With the rapid development of the Internet of Things (IoT) and artificial intelligence, autonomous vehicles have received much attention in recent years. Safe driving is one of the essential concerns of self-driving cars. The main problem in providing better safe driving requires an efficient inference system for real-time task management and autonomous control. Due to limited battery life and computing power, reducing execution time and resource consumption can be a daunting process. This paper addressed these challenges and developed an intelligent task management system for IoT-based autonomous vehicles. For each task processing, a supervised resource predictor is invoked for optimal hardware cluster selection. Tasks are executed based on the earliest hyper period first (EHF) scheduler to achieve optimal task error rate and schedule length performance. The single-layer feedforward neural network (SLFN) and lightweight learning approaches are designed to distribute each task to the appropriate processor based on their emergency and CPU utilization. We developed this intelligent task management module in python and experimentally tested it on multicore SoCs (Odroid Xu4 and NVIDIA Jetson embedded platforms). Connected Autonomous Vehicles (CAV) and Internet of Medical Things (IoMT) benchmarks are used for training and testing purposes. The proposed modules are validated by observing the task miss rate, resource utilization, and energy consumption metrics compared with state-of-art heuristics. SLFN-EHF task scheduler achieved better results in an average of 98% accuracy, and in an average of 20–27% reduced in execution time and 32–45% in task miss rate metric than conventional methods.

Deciphering anomalous heterogeneous intracellular transport with neural networks.

Intracellular transport is predominantly heterogeneous in both time and space, exhibiting varying non-Brownian behavior. Characterization of this movement through averaging methods over an ensemble of trajectories or over the course of a single trajectory often fails to capture this heterogeneity. Here, we developed a deep learning feedforward neural network trained on fractional Brownian motion, providing a novel, accurate and efficient method for resolving heterogeneous behavior of intracellular transport in space and time. The neural network requires significantly fewer data points compared to established methods. This enables robust estimation of Hurst exponents for very short time series data, making possible direct, dynamic segmentation and analysis of experimental tracks of rapidly moving cellular structures such as endosomes and lysosomes. By using this analysis, fractional Brownian motion with a stochastic Hurst exponent was used to interpret, for the first time, anomalous intracellular dynamics, revealing unexpected differences in behavior between closely related endocytic organelles.

Performance evaluation of pattern recognition networks using electromyography signal and time-domain features for the classification of hand gestures.

The problem of classifying individual finger movements of one hand is focused in this article. The input electromyography signal is processed and eight time-domain features are extracted for classifying hand gestures. The classified finger movements are thumb, middle, index, little, ring, hand close, thumb index, thumb ring, thumb little and thumb middle and the hand grasps are palmar class, spherical class, hook class, cylindrical class, tip class and lateral class. Four state-of-the-art classifiers namely feed forward artificial neural network, cascaded feed forward artificial neural network, deep learning neural network and support vector machine are selected for this work to classify the finger movements and hand grasps using the extracted time-domain features. The experimental results show that the artificial neural network classifier is stabilized at 6 epochs for finger movement dataset and at 4 epochs for hand grasps dataset with low mean square error. However, the support vector machine classifier attains the maximum accuracy of 97.3077% for finger movement dataset and 98.875% for hand grasp dataset which is significantly greater than feed forward artificial neural network, cascaded feed forward artificial neural network and deep learning neural network classifiers.

Forecasting of water level in multiple temperate lakes using machine learning models

Due to global climate change and growing population, fresh water resources are becoming more vulnerable to pollution. Protecting fresh water resources, especially lakes and the associated environment, is one of the key challenges faced by policy makers and water managers. Lake water level is an important physical indicator of lakes, and its fluctuation may significantly impact lake ecosystems. Therefore, reliable forecasting of lake water level is vital for a proper assessment of the health of lake ecosystems and their management. In this study, two machine learning models, including feed forward neural network (FFNN) and Deep Learning (DL) technique, were used to predict monthly lake water level. The two models were employed for one month ahead forecasting of lake water level in 69 temperate lakes in Poland. The results show that both the FFNN and the DL models performed generally well for forecasting of lake water level of the 69 lakes, with only marginal differences. The results also indicate that the DL model did not show significant superiority over the traditional FFNN model; indeed, the FFNN model slightly outperformed the DL model for 33 of the 69 lakes. These results seem to suggest that traditional machine learning models may just be sufficient for forecasting of lake water level when they are properly trained. The outcomes of the present study have important implications for water level forecasting and water resources management of lakes, especially from the perspective of machine learning models and their complexities.

Learning in Feedforward Neural Networks Accelerated by Transfer Entropy.

Current neural networks architectures are many times harder to train because of the increasing size and complexity of the used datasets. Our objective is to design more efficient training algorithms utilizing causal relationships inferred from neural networks. The transfer entropy (TE) was initially introduced as an information transfer measure used to quantify the statistical coherence between events (time series). Later, it was related to causality, even if they are not the same. There are only few papers reporting applications of causality or TE in neural networks. Our contribution is an information-theoretical method for analyzing information transfer between the nodes of feedforward neural networks. The information transfer is measured by the TE of feedback neural connections. Intuitively, TE measures the relevance of a connection in the network and the feedback amplifies this connection. We introduce a backpropagation type training algorithm that uses TE feedback connections to improve its performance.

Machine learning facilitated business intelligence (Part II)

PurposeThe purpose of this paper is three-fold: to review the categories explaining mainly optimization algorithms (techniques) in that needed to improve the generalization performance and learning speed of the Feedforward Neural Network (FNN); to discover the change in research trends by analyzing all six categories (i.e. gradient learning algorithms for network training, gradient free learning algorithms, optimization algorithms for learning rate, bias and variance (underfitting and overfitting) minimization algorithms, constructive topology neural networks, metaheuristic search algorithms) collectively; and recommend new research directions for researchers and facilitate users to understand algorithms real-world applications in solving complex management, engineering and health sciences problems.Design/methodology/approachThe FNN has gained much attention from researchers to make a more informed decision in the last few decades. The literature survey is focused on the learning algorithms and the optimization techniques proposed in the last three decades. This paper (Part II) is an extension of Part I. For the sake of simplicity, the paper entitled “Machine learning facilitated business intelligence (Part I): Neural networks learning algorithms and applications” is referred to as Part I. To make the study consistent with Part I, the approach and survey methodology in this paper are kept similar to those in Part I.FindingsCombining the work performed in Part I, the authors studied a total of 80 articles through popular keywords searching. The FNN learning algorithms and optimization techniques identified in the selected literature are classified into six categories based on their problem identification, mathematical model, technical reasoning and proposed solution. Previously, in Part I, the two categories focusing on the learning algorithms (i.e. gradient learning algorithms for network training, gradient free learning algorithms) are reviewed with their real-world applications in management, engineering, and health sciences. Therefore, in the current paper, Part II, the remaining four categories, exploring optimization techniques (i.e. optimization algorithms for learning rate, bias and variance (underfitting and overfitting) minimization algorithms, constructive topology neural networks, metaheuristic search algorithms) are studied in detail. The algorithm explanation is made enriched by discussing their technical merits, limitations, and applications in their respective categories. Finally, the authors recommend future new research directions which can contribute to strengthening the literature.Research limitations/implicationsThe FNN contributions are rapidly increasing because of its ability to make reliably informed decisions. Like learning algorithms, reviewed in Part I, the focus is to enrich the comprehensive study by reviewing remaining categories focusing on the optimization techniques. However, future efforts may be needed to incorporate other algorithms into identified six categories or suggest new category to continuously monitor the shift in the research trends.Practical implicationsThe authors studied the shift in research trend for three decades by collectively analyzing the learning algorithms and optimization techniques with their applications. This may help researchers to identify future research gaps to improve the generalization performance and learning speed, and user to understand the applications areas of the FNN. For instance, research contribution in FNN in the last three decades has changed from complex gradient-based algorithms to gradient free algorithms, trial and error hidden units fixed topology approach to cascade topology, hyperparameters initial guess to analytically calculation and converging algorithms at a global minimum rather than the local minimum.Originality/valueThe existing literature surveys include comparative study of the algorithms, identifying algorithms application areas and focusing on specific techniques in that it may not be able to identify algorithms categories, a shift in research trends over time, application area frequently analyzed, common research gaps and collective future directions. Part I and II attempts to overcome the existing literature surveys limitations by classifying articles into six categories covering a wide range of algorithm proposed to improve the FNN generalization performance and convergence rate. The classification of algorithms into six categories helps to analyze the shift in research trend which makes the classification scheme significant and innovative.

GlacierNet: A Deep-Learning Approach for Debris-Covered Glacier Mapping

Rising global temperatures over the past decades is directly affecting glacier dynamics. To understand glacier fluctuations and document regional glacier-state trends, glacier-boundary detection is necessary. Debris-covered glacier (DCG) mapping, however, is notoriously difficult using conventional geospatial technology methods. Therefore, in this research for automated DCG mapping, we evaluate the utility of a convolutional neural network (CNN), which is a deep learning feed-forward neural network. The CNN inputs include Landsat satellite images, an Advanced Land Observation Satellite (ALOS) digital elevation model (DEM) and DEM-derived land-surface parameters. Our CNN based deep-learning approach named GlacierNet was designed by appropriately choosing the type, number and size of layers and filters, and encoder depth based on the properties of the input data, CNN segmentation process and empirical results. The GlacierNet was then trained using input data and corresponding glacier boundaries from the Global Land Ice Measurements from Space (GLIMS) database, and tested on glaciers in the Karakoram and Nepal Himalaya. Our results show proof-of-concept that GlacierNet reasonably identifies the boundaries of DCGs with a relatively high degree of accuracy, and that morphometric parameters improves boundary detection.

Prediction of occupancy level and energy consumption in office building using blind system identification and neural networks

Occupancy behaviour plays an important role in energy consumption in buildings. Currently, the shallow understanding of occupancy has led to a considerable performance gap between predicted and measured energy use. This paper presents an approach to estimate the occupancy based on blind system identification (BSI), and a prediction model of electricity consumption by an air-conditioning system is developed and reported based on an artificial neural network with the BSI estimation of the number of occupants as an input. This starts from the identification of indoor CO2 dynamics derived from the mass-conservation law and venting levels. The unknown parameters, including the occupancy and model parameters, are estimated by using a frequentist maximum-likelihood algorithm and Bayesian estimation. The second phase is to establish the prediction model of the electricity consumption of the air-conditioning system by using a feed-forward neural network (FFNN) and extreme learning machine (ELM), as well as ensemble models. To analyse some aspects of the benchmark test for identifying the effect of structure parameters and input-selection alternatives, three studies are conducted on (1) the effect of predictor selection based on principal component analysis, (2) the effect of the estimated occupancy as the supplementary input, and (3) the effect of the neural network ensemble. The result shows that the occupancy number, as the input, is able to improve the accuracy in predicting energy consumption using a neural-network model.

A Note on Liao’s Recurrent Neural-Network Learning for Discrete Multi-stage Optimal Control Problems

The roots of neural-network backpropagation (BP) may be traced back to classical optimal-control gradient procedures developed in early 1960s. Hence, BP can directly apply to a general discrete N-stage optimal control problem that consists of N stage costs plus a terminal state cost. In this journal (Liao in Neural Process Lett 10:195–200, 1999), given such a multi-stage optimal control problem, Liao has turned it into a problem involving a terminal state cost only (via classical transformation), and then claimed that BP on the transformed problem leads to new recurrent neural network learning. The purpose of this paper is three-fold: First, the classical terminal-cost transformation yields no particular benefit for BP. Second, two simulation examples (with and without time lag) demonstrated by Liao can be regarded naturally as deep feed-forward neural-network learning rather than as recurrent neural-network learning from the perspective of classical optimal-control gradient methods. Third, BP can readily deal with a general history-dependent optimal control problem (e.g., involving time-lagged state and control variables) owing to Dreyfus’s 1973 extension of BP. Throughout the paper, we highlight systematic BP derivations by employing the recurrence relation of nominal cost-to-go action-value functions based on the stage-wise concept of dynamic programming.

Deep feedforward neural network learning using Local Binary Patterns histograms for outdoor object categorization

Deep feedforward neural network learning using Local Binary Patterns histograms for outdoor object categorization

DeepRec: A deep neural network approach to recommendation with item embedding and weighted loss function

Traditional collaborative filtering techniques suffer from the data sparsity problem in practice. That is, only a small proportion of all items in the recommender system occur in a user's rated item list. However, in order to retrieve items meeting a user's interest, all possible candidate items should be investigated. To address this problem, this paper proposes a recommendation approach called DeepRec, based on feedforward deep neural network learning with item embedding and weighted loss function. Specifically, item embedding learns numerical vectors for item representation, and weighted loss function balances popularity and novelty of recommended items. Moreover, it introduces two strategies, i.e. sampling by random (Ran-Strategy) and sampling by distribution (Pro-Strategy), to leave one item as output and the remaining as input from each user's historically rated item list. Max-pooling and average-pooling are employed to combine individual item vectors to derive users’ input vectors for feedforward deep neural network learning. Experiments on the App dataset and the Last.fm dataset demonstrate that the proposed DeepRec approach is superior to state-of-the-art techniques in recommending Apps and songs in terms of accuracy and diversity as well as complexity.

Restricted Boltzmann machine to determine the input weights for extreme learning machines

The Extreme Learning Machine (ELM) is a single-hidden layer feedforward neural network (SLFN) learning algorithm that can learn effectively and quickly. The ELM training phase assigns the input weights and bias randomly and does not change them in the whole process. Although the network works well, the random weights in the input layer may affect the algorithm performance. Therefore, we propose a new approach to determine the input weights and bias for the ELM using the restricted Boltzmann machine (RBM), which we call RBM-ELM. We compare our new approach to the well-known ELM-AE and to the ELM-RO, a state of the art algorithm to select the input weights for the ELM. The experimental results show that the RBM-ELM achieves a better performance than the ELM and outperforms the ELM-AE and ELM-RO.

ASD+M: Automatic parameter tuning in stochastic optimization and on-line learning

In this paper the classic momentum algorithm for stochastic optimization is considered. A method is introduced that adjusts coefficients for this algorithm during its operation. The method does not depend on any preliminary knowledge of the optimization problem. In the experimental study, the method is applied to on-line learning in feed-forward neural networks, including deep auto-encoders, and outperforms any fixed coefficients. The method eliminates coefficients that are difficult to determine, with profound influence on performance. While the method itself has some coefficients, they are ease to determine and sensitivity of performance to them is low. Consequently, the method makes on-line learning a practically parameter-free process and broadens the area of potential application of this technology.

Partial Learning Machine: A New Learning Scheme for Feed Forward Neural Networks

As for the traditional Feed Forward Neural Network (FNN) learning methods such as Back-Propagation (BP) algorithms, the training methods waste much time with poor generalization ability. And for another learning scheme of FNN, Extreme Learning Machine (ELM), the training speed is increased dramatically with the cost of low accuracy and large fluctuation. In this paper, a new machine learning scheme named Partial Learning Machine (PLM) is proposed based on heuristic adjustment to general problems. The experimental results show that the training time of PLM is less than that of BP with better generalization ability. Moreover, PLM has less fluctuation and higher learning precision than ELM.

Distributed learning for feedforward neural networks with random weights using an event-triggered communication scheme

This paper aims to develop distributed learning algorithms for feedforward neural networks with random weights (FNNRWs) by using event-triggered communication schemes. Based on this scheme, the communication process of each agent is driven by a trigger condition where the agents exchange information in an asynchronous manner, only when it is crucially required. To this end, the centralized FNNRW problem is cast as a set of distributed subproblems with consensus constrains imposed on the desired parameters and solved following the discrete-time zero-gradient-sum (ZGS) strategy. An event-triggered communication scheme is introduced to the ZGS-based FNNRW algorithm in order to avoid unnecessary transmission costs. This is particularly useful for the case when communication resource is limited. It is proved that the proposed event-triggered approach is exponentially convergent if the design parameter is chosen properly under strongly connected and weight-balanced agent interactions. Two numerical simulation examples are provided to verify the effectiveness of the proposed algorithm.

Continuous Online Sequence Learning with an Unsupervised Neural Network Model.

The ability to recognize and predict temporal sequences of sensory inputs is vital for survival in natural environments. Based on many known properties of cortical neurons, hierarchical temporal memory (HTM) sequence memory recently has been proposed as a theoretical framework for sequence learning in the cortex. In this letter, we analyze properties of HTM sequence memory and apply it to sequence learning and prediction problems with streaming data. We show the model is able to continuously learn a large number of variable order temporal sequences using an unsupervised Hebbian-like learning rule. The sparse temporal codes formed by the model can robustly handle branching temporal sequences by maintaining multiple predictions until there is sufficient disambiguating evidence. We compare the HTM sequence memory with other sequence learning algorithms, including statistical methods-autoregressive integrated moving average; feedforward neural networks-time delay neural network and online sequential extreme learning machine; and recurrent neural networks-long short-term memory and echo-state networks on sequence prediction problems with both artificial and real-world data. The HTM model achieves comparable accuracy to other state-of-the-art algorithms. The model also exhibits properties that are critical for sequence learning, including continuous online learning, the ability to handle multiple predictions and branching sequences with high-order statistics, robustness to sensor noise and fault tolerance, and good performance without task-specific hyperparameter tuning. Therefore, the HTM sequence memory not only advances our understanding of how the brain may solve the sequence learning problem but is also applicable to real-world sequence learning problems from continuous data streams.

Biomimetic Hybrid Feedback Feedforward Neural-Network Learning Control

This brief presents a biomimetic hybrid feedback feedforward neural-network learning control (NNLC) strategy inspired by the human motor learning control mechanism for a class of uncertain nonlinear systems. The control structure includes a proportional-derivative controller acting as a feedback servo machine and a radial-basis-function (RBF) NN acting as a feedforward predictive machine. Under the sufficient constraints on control parameters, the closed-loop system achieves semiglobal practical exponential stability, such that an accurate NN approximation is guaranteed in a local region along recurrent reference trajectories. Compared with the existing NNLC methods, the novelties of the proposed method include: 1) the implementation of an adaptive NN control to guarantee plant states being recurrent is not needed, since recurrent reference signals rather than plant states are utilized as NN inputs, which greatly simplifies the analysis and synthesis of the NNLC and 2) the domain of NN approximation can be determined a priori by the given reference signals, which leads to an easy construction of the RBF-NNs. Simulation results have verified the effectiveness of this approach.

Is extreme learning machine feasible? A theoretical assessment (part II).

An extreme learning machine (ELM) can be regarded as a two-stage feed-forward neural network (FNN) learning system that randomly assigns the connections with and within hidden neurons in the first stage and tunes the connections with output neurons in the second stage. Therefore, ELM training is essentially a linear learning problem, which significantly reduces the computational burden. Numerous applications show that such a computation burden reduction does not degrade the generalization capability. It has, however, been open that whether this is true in theory. The aim of this paper is to study the theoretical feasibility of ELM by analyzing the pros and cons of ELM. In the previous part of this topic, we pointed out that via appropriately selected activation functions, ELM does not degrade the generalization capability in the sense of expectation. In this paper, we launch the study in a different direction and show that the randomness of ELM also leads to certain negative consequences. On one hand, we find that the randomness causes an additional uncertainty problem of ELM, both in approximation and learning. On the other hand, we theoretically justify that there also exist activation functions such that the corresponding ELM degrades the generalization capability. In particular, we prove that the generalization capability of ELM with Gaussian kernel is essentially worse than that of FNN with Gaussian kernel. To facilitate the use of ELM, we also provide a remedy to such a degradation. We find that the well-developed coefficient regularization technique can essentially improve the generalization capability. The obtained results reveal the essential characteristic of ELM in a certain sense and give theoretical guidance concerning how to use ELM.

Robust sequential learning of feedforward neural networks in the presence of heavy-tailed noise

Feedforward neural networks (FFNN) are among the most used neural networks for modeling of various nonlinear problems in engineering. In sequential and especially real time processing all neural networks models fail when faced with outliers. Outliers are found across a wide range of engineering problems. Recent research results in the field have shown that to avoid overfitting or divergence of the model, new approach is needed especially if FFNN is to run sequentially or in real time. To accommodate limitations of FFNN when training data contains a certain number of outliers, this paper presents new learning algorithm based on improvement of conventional extended Kalman filter (EKF). Extended Kalman filter robust to outliers (EKF-OR) is probabilistic generative model in which measurement noise covariance is not constant; the sequence of noise measurement covariance is modeled as stochastic process over the set of symmetric positive-definite matrices in which prior is modeled as inverse Wishart distribution. In each iteration EKF-OR simultaneously estimates noise estimates and current best estimate of FFNN parameters. Bayesian framework enables one to mathematically derive expressions, while analytical intractability of the Bayes’ update step is solved by using structured variational approximation. All mathematical expressions in the paper are derived using the first principles. Extensive experimental study shows that FFNN trained with developed learning algorithm, achieves low prediction error and good generalization quality regardless of outliers’ presence in training data.