Problem In Artificial Neural Networks Research Articles

Generic optimization by fast chaotic exploration and slow feedback fixation

Living systems adapt to various environmental conditions by changing their internal states through processes such as gene expression and epigenetic modification. In this paper, we propose a generic mechanism for optimization that combines fast oscillatory dynamics with a slower feedback fixation process. Through extensive model simulations, we demonstrate that the fast chaotic dynamics serve as a global search for optimal states, which are then fixed by the slower dynamics. This mechanism becomes more effective as the number of elements is increased. We also discuss the potential relevance of this optimization mechanism to problems in artificial neural networks.

Inferring the location of neurons within an artificial network from their activity

Inferring the connectivity of biological neural networks from neural activation data is an open problem. We propose that the analogous problem in artificial neural networks is more amenable to study and may illuminate the biological case. Here, we study the specific problem of assigning artificial neurons to locations in a network of known architecture, specifically the LeNet image classifier. We evaluate a supervised learning approach based on features derived from the eigenvectors of the activation correlation matrix. Experiments highlighted that for an image dataset to be effective for accurate localisation, it should fully activate the network and contain minimal confounding correlations. No single image dataset was found that resulted in perfect assignment, however perfect assignment was achieved using a concatenation of features from multiple image datasets.

A Novel Condition Monitoring Method of Wind Turbines Based on GMDH Neural Network

The safety of power transmission systems in wind turbines is crucial to the wind turbine’s stable operation and has attracted a great deal of attention in condition monitoring of wind farms. Many different intelligent condition monitoring schemes have been developed to detect the occurrence of defects via supervisory control and data acquisition (SCADA) data, which is the most commonly applied condition monitoring system in wind turbines. Normally, artificial neural networks are applied to establish prediction models of the wind turbine condition monitoring. In this paper, an alternative and cost-effective methodology has been proposed, based on the group method of data handling (GMDH) neural network. GMDH is a kind of computer-based mathematical modelling and structural identification algorithm. GMDH neural networks can automatically organize neural network architecture by heuristic self-organization methods and determine structural parameters, such as the number of layers, the number of neurons in hidden layers, and useful input variables. Furthermore, GMDH neural network can avoid over-fitting problems, which is a ubiquitous problem in artificial neural networks. The effectiveness and performance of the proposed method are validated in the case studies.

Multi-objective neural network model selection with a graph-based large margin approach

This work presents a new decision-making strategy for multi-objective learning problem of artificial neural networks (ANN). The proposed decision-maker searches for the solution that minimizes a margin-based validation error amongst Pareto set solutions. The proposal is based on a geometric approximation to find the large margin (distance) of separation among the classes. Several benchmarks commonly available in the literature were used for testing. The obtained results showed that the proposal is more efficient in controlling the generalization capacity of neural models than other learning machines. It yields smooth (noise robustness) and well-fitted models straightforwardly, i.e., without the necessity of parameter set definition in advance or validation data use, as often required by learning machines.

Can sleep protect memories from catastrophic forgetting?

Continual learning remains an unsolved problem in artificial neural networks. The brain has evolved mechanisms to prevent catastrophic forgetting of old knowledge during new training. Building upon data suggesting the importance of sleep in learning and memory, we tested a hypothesis that sleep protects old memories from being forgotten after new learning. In the thalamocortical model, training a new memory interfered with previously learned old memories leading to degradation and forgetting of the old memory traces. Simulating sleep after new learning reversed the damage and enhanced old and new memories. We found that when a new memory competed for previously allocated neuronal/synaptic resources, sleep replay changed the synaptic footprint of the old memory to allow overlapping neuronal populations to store multiple memories. Our study predicts that memory storage is dynamic, and sleep enables continual learning by combining consolidation of new memory traces with reconsolidation of old memory traces to minimize interference.

Resilient back-propagation approach in small-world feed-forward neural network topology based on Newman–Watts algorithm

The scientific researches are focused on network topologies and training algorithms fields because they reduce overfitting problem in artificial neural networks. In this context, we showed in our previous work that Newman–Watts small-world feed-forward artificial neural networks present better classification and prediction performance than conventional feed-forward artificial neural networks. In this study, we investigate the effects of the Resilient back-propagation algorithm on SW network topology and propose a Resilient Newman–Watts small-world feed-forward artificial neural network model by assuming fixed initial topological conditions. We find that Resilient small-world network further reduces overfitting and further increases the network performance when compared to the conventional feed-forward artificial neural networks. Furthermore, it is shown that the proposed network model does not increase the algorithmic complexity as per other models. The obtained results imply that the proposed model can contribute to the solving of overfitting problem encountered in both deep neural networks and conventional artificial neural networks.

APPROACH TO THE SYNTHESIS OF NEURAL NETWORK STRUCTURE DURING CLASSIFICATION

Evaluating the number of hidden neurons necessary for solving of pattern recognition and classification tasks is one of the key problems in artificial neural networks. Multilayer perceptron is the most useful artificial neural network to estimate the functional structure in classification. In this paper, we show that artificial neural network with a two hidden layer feed forward neural network with d inputs, d neurons in the first hidden layer, 2d+2 neurons in the second hidden layer, k outputs and with a sigmoidal infinitely differentiable function can solve classification and pattern problems with arbitrary accuracy. This result can be applied to design pattern recognition and classification models with optimal structure in the number of hidden neurons and hidden layers. The experimental results over well-known benchmark datasets show that the convergence and the accuracy of the proposed model of artificial neural network are acceptable. Findings in this paper are experimentally analyzed on four different datasets from machine learning repository.

Multi-criteria decision making based architecture selection for single-hidden layer feedforward neural networks

Architecture selection is a fundamental problem in artificial neural networks, which could be treated as a decision making process that evaluates, ranks, and makes choices from a set of network structures. Traditional methods evaluate a network structure by designing a criterion based on a validation model or an error bound model. On one hand, the time complexity of a validation model is usually high; on the other hand, different validation models or error bound models may lead to different (even conflicting) results, which post challenges to the traditional single criterion-based architecture selection methods. In the area of decision making, many problems employed multiple criteria since the performance is better than using a single criterion. In this paper, we propose a multi-criteria decision making based architecture selection algorithm for single-hidden layer feedforward neural networks trained by extreme learning machine. Two criteria are incorporated into the selection process, i.e., training accuracy and the Q-value estimated by the localized generalization error model. The training accuracy reflects the capability of the model on correctly categorizing the known samples, and the Q-value estimated by localized generalization error model reflects the size of the neighbourhood of training samples in which the model can predict unseen samples with confidence. By achieving a trade-off between these two criteria, a new architecture selection algorithm is proposed. Experimental comparisons demonstrate the feasibility and effectiveness of the proposed method.

Efficient methods of initializing neuron weights in self-organizing networks implemented in hardware

In this paper, we focus on the topic of an efficient initialization of neuron weights, which is one of key problems in artificial neural networks (ANNs). This problem is important in ANNs implemented as Application Specific Integrated Circuits (ASICs), in which the number of the weights is relatively large. When ANNs are implemented in software, the weights can be easily modified. In contrast, in neural networks realized as ASICs in which due to parallel data processing each neuron is realized as a separate circuit, it is necessary to provide programming and addressing lines to each memory cell containing a weight. This causes a substantial increase in the complexity of such systems. In this study, we performed comprehensive investigations, in which we simulated the training process of the Self-Organizing ANN with different initialization scenarios. The aim of these investigations was to find simple and efficient initialization procedures that lead to optimal learning process for a broad spectrum of values of other network parameters.The investigations have shown that Self-Organizing Maps (SOMs) in many situations may be trained without any initialization (with zeroed weights). This is possible due to the neighborhood mechanism that to some degree stimulates the neurons belonging to the SOM. We present selected results of several thousands simulations for different topologies of the SOM, for different neighborhood functions and two distance measures between the learning patterns and neurons in the input data space. Simulations were performed for initial values of the weights equal to zero, for small values (up to 1% of full scale range) and for neurons randomly distributed over the overall input data space. The results in most cases are comparable that allows to reduce the complexity of the SOM implemented in the CMOS technology.

Comparing large number of metaheuristics for artificial neural networks training to predict water temperature in a natural river

Nature-inspired metaheuristics found various applications in different fields of science, including the problem of artificial neural networks (ANN) training. However, very versatile opinions regarding the performance of metaheuristics applied to ANN training may be found in the literature.Both nature-inspired metaheuristics and ANNs are widely applied to various geophysical and environmental problems. Among them the water temperature forecasting in a natural river, especially in colder climate zones where the seasonality plays important role, is of great importance, as water temperature has strong impact on aquatic life and chemistry. As the impact of possible future climate change on water temperature is not trivial, models are needed to allow projection of streamwater temperature based on simple hydro-meteorological variables.In this paper the detailed comparison of the performance of nature-inspired optimization methods and Levenberg–Marquardt (LM) algorithm in ANNs training is performed, based on the case study of water temperature forecasting in a natural stream, namely Biala Tarnowska river in southern Poland. Over 50 variants of 22 various metaheuristics, including a large number of Differential Evolution, as well as some Particle Swarm Optimization, Evolution Strategies, multialgorithms and Direct Search methods are compared with LM algorithm on ANN training for the described case study. The impact of population size and some control parameters of particular metaheuristics on the ANN training performance are verified. It is found that despite widely claimed large improvement in nature-inspired methods during last years, the vast majority of them are still outperformed by LM algorithm on the selected problem. The only methods that, based on this case study, seem competitive to LM algorithm in terms of the final performance (but not speed) are Differential Evolution algorithms that benefit from the concept of Global and Local neighborhood-based mutation operators. The streamwater forecasting performance of the neural networks is adequate, the major prediction errors are related to the river freezing and melting processes that occur during winter in the mountainous catchment under study.

Longterm forecasting of solid waste generation by the artificial neural networks

AbstractThis study presents a new approach—preprocessing for reaching the stationary chain in time series—to unravel the interpolating problem of artificial neural networks (ANN) for long‐term prediction of solid waste generation (SWG). To evaluate the accuracy of the prediction by ANN, comparison between the results of the multivariate regression model and ANN is performed. Monthly time series datasets, by the yrs 2000–2010, for the city of Mashhad, are used to simulate the generated solid waste. Different socioeconomic and environmental factors are assessed, and the most effective ones are used as input variables. The projections of these explanatory variables are used in the estimated model to predict the generated solid waste values through the yr 2032. Ultimately, various structures of ANN models are examined to select the best result based on the performance indices criteria. Research findings clearly indicate that such a new approach can be a practical method for long‐term prediction by ANNs. Furthermore, it can reduce uncertainties and lead to noticeable increase in the accuracy of the long‐term forecasting. Results indicate that multilayer perception approach has more advantages in comparison with traditional methods in predicting the municipal SWG. © 2011 American Institute of Chemical Engineers Environ Prog, 2011

Noise injection for training artificial neural networks: A comparison with weight decay and early stopping

The purpose of this study was to investigate the effect of a noise injection method on the "overfitting" problem of artificial neural networks (ANNs) in two-class classification tasks. The authors compared ANNs trained with noise injection to ANNs trained with two other methods for avoiding overfitting: weight decay and early stopping. They also evaluated an automatic algorithm for selecting the magnitude of the noise injection. They performed simulation studies of an exclusive-or classification task with training datasets of 50, 100, and 200 cases (half normal and half abnormal) and an independent testing dataset of 2000 cases. They also compared the methods using a breast ultrasound dataset of 1126 cases. For simulated training datasets of 50 cases, the area under the receiver operating characteristic curve (AUC) was greater (by 0.03) when training with noise injection than when training without any regularization, and the improvement was greater than those from weight decay and early stopping (both of 0.02). For training datasets of 100 cases, noise injection and weight decay yielded similar increases in the AUC (0.02), whereas early stopping produced a smaller increase (0.01). For training datasets of 200 cases, the increases in the AUC were negligibly small for all methods (0.005). For the ultrasound dataset, noise injection had a greater average AUC than ANNs trained without regularization and a slightly greater average AUC than ANNs trained with weight decay. These results indicate that training ANNs with noise injection can reduce overfitting to a greater degree than early stopping and to a similar degree as weight decay.

An alternative evaluation of FMEA: Fuzzy ART algorithm

AbstractFailure Mode and Effects Analysis (FMEA) is a technique used in the manufacturing industry to improve production quality and productivity. It is a method that evaluates possible failures in the system, design, process or service. It aims to continuously improve and decrease these kinds of failure modes. Adaptive Resonance Theory (ART) is one of the learning algorithms without consultants, which are developed for clustering problems in artificial neural networks. In the FMEA method, every failure mode in the system is analyzed according to severity, occurrence and detection. Then, risk priority number (RPN) is acquired by multiplication of these three factors and the necessary failures are improved with respect to the determined threshold value. In addition, there exist many shortcomings of the traditional FMEA method, which affect its efficiency and thus limit its realization. To respond to these difficulties, this study introduces the method named Fuzzy Adaptive Resonance Theory (Fuzzy ART), one of the ART networks, to evaluate RPN in FMEA. Copyright © 2008 John Wiley & Sons, Ltd.

Visualization of neural-network gaps based on error analysis

Presented in this paper is a new methodology for detection of neural-network gaps (NNG's) based on error analysis and the visualization that is applicable for n-dimensional I/O domain. The generalization problem in artificial neural networks (ANN) training is analyzed and the concept of NNG's is introduced. The NNG's are highly undesirable in ANN generalization and methods for detecting, analyzing, and eliminating them are necessary. Previous methods for NNG detection, based on two-dimensional (2-D) and three-dimensional (3-D) visualization, were not applicable for ANN's with more than three inputs. Experiments demonstrate advantages of this new methodology, which allows better understanding of the NNG phenomena using a quantitative approach.

Solvable models of layered neural networks based on their differential structure

We introduce the notion of solvable models of artificial neural networks, based on the theory of ordinary differential equations. It is shown that a solvable, three layer, neural network can be realized as a solution of an ordinary differential equation. Several neural networks in standard use are shown to be solvable. This leads to a new, two-step, non-recursive learning paradigm: estimate the differential equation which the target function satisfies “approximately”, and then approximate the target function in the solution space of that differential equation. It is shown experimentally that the proposed algorithm is useful for analyzing the generalization problem in artificial neural networks. Connections with wavelet analysis are also pointed out.

Consolidation in Neural Networks and in the Sleeping Brain

In this paper we explore the topic of the consolidation of information in neural network learning. One problem in particular has limited the ability of a broad range of neural networks to perform ongoing learning and consolidation. This is 'catastrophic forgetting', the tendency for new information, when it is learned, to disrupt old information. We will review and slightly extend the rehearsal and pseudorehearsal solutions to the catastrophic forgetting problem presented in Robins (1995). The main focus of this paper is to then relate these mechanisms to the consolidation processes which have been proposed in the psychological literature regarding sleep. We suggest that the catastrophic forgetting problem in artificial neural networks (ANNs) is a problem that has actually occurred in the evolution of the mammalian brain, and that the pseudorehearsal solution to the problem in ANNs is functionally equivalent to the sleep consolidation solution adopted by the brain. Finally, we review related work by McClelland et al. (1995) and propose a tentative model of learning and sleep that emphasizes consolidation mechanisms and the role of the hippocampus.