Multilayer Perceptron Training Research Articles

Considering supply risk for supplier selection using an integrated framework of data envelopment analysis and neural networks

Article history: Received October202012 Received in revised format December 28 2012 Accepted December29 2012 Available online 2January 2013 For many years, supplier selection as an important multi-criteria decision has attracted both the researchers and practitioners. During recent years, high incidence of natural disasters, terrorism attacks, labor strikes and other kinds of risks, also known as disruptions, indicates the vulnerability of procurement process to these unpredicted events. In this study, a new framework is introduced to select suppliers while considering the supply risks. In the proposed framework, an expert determines the reliabilities of procurement elements (i.e., production, transportation and communication) based on some proposed risk factors. Then, a trained Multi-Layer Perceptron (MLP) network plays the role of the expert opinion for estimating the reliability scores. In addition to reliabilities, the Data Envelopment Analysis (DEA) is used to take into account the conventional selection criteria: price, delivery, quality and capacity. A set of Pareto-optimal suppliers is obtained from the combination of efficiencies and reliability scores. Finally, the decision maker chooses between the non-dominated suppliers. Obtained experiment results indicate the effectiveness of the proposed framework. © 2013 Growing Science Ltd. All rights reserved

A MUTATION PARTICLE SWARM OPTIMIZATION ALGORITHM FOR MULTILAYER PERCEPTRON TRAINING WITH APPLICATIONS

Particle swarm optimization (PSO), a prevalent optimization algorithm, has been successfully applied to various fields of science and engineering. However, PSO still suffers from some problems such as premature convergence. To solve these problems, we propose a mutation PSO (MPSO) in this paper. Compared with the traditional PSO, there are two main improvements of the proposed MPSO. First, a new particle update rule is explored. The new rule updates a particle's position according to not only its best known position and the global best known position of the swarm, but also a number of other particles' best known positions. The second improvement is that a mutation operator is employed. Mutation operator is used to avoid premature convergence. The MPSO is utilized to train a multilayer perceptron (MLP). The MLP trained by MPSO is finally applied to two classification problems: Iris flower classification and scene classification. For comparison purposes, traditional PSO, genetic algorithm (GA), and back-propagation (BP) are also investigated. Experimental results demonstrate the superior performance of the proposed MPSO for MLP training.

A NEW MODIFIED PARTICLE SWARM OPTIMIZATION FOR MULTILAYER PERCEPTION TRAINING WITH APPLICATIONS

Particle swarm optimization (PSO) is one of the important evolutionary algorithms. However, the traditional PSO suffers from the premature convergence problem. In view of this, a new PSO, named mutation PSO (MPSO), is proposed in this paper. The proposed MPSO not only makes use of a mutation operator to update particles/individuals, which was originally designed for genetic algorithm (GA). But also a new weighted update rule is proposed for MPSO to produce the new swarm. Then we use the proposed MPSO to train multilayer perceptron (MLP) with two tasks: curve fitting and classification. In particular, the performance investigation is concentrated on scene classification. For a comparison purpose, MLPs trained using the error backpropagation (BP), traditional PSO and GA are also investigated. The advantages and disadvantages of these algorithms are also analyzed. Experimental results show that the proposed MPSO outperforms than other algorithms for the training of an MLP.

MLP training in a self-organizing state space model using unscented Kalman particle filter

Many Bayesian learning approaches to the multi-layer perceptron (MLP) parameter optimization have been proposed such as the extended Kalman filter (EKF). This paper uses the unscented Kalman particle filter (UPF) to train the MLP in a self-organizing state space (SOSS) model. This involves forming augmented state vectors consisting of all parameters (the weights of the MLP) and outputs. The UPF is used to sequentially update the true system states and high dimensional parameters that are inherent to the SOSS model for the MLP simultaneously. Simulation results show that the new method performs better than traditional optimization methods.

An ant colony optimisation and Nelder-Mead simplex hybrid algorithm for training neural networks: an application to bankruptcy prediction in banks

In this paper, we propose the application of a hybrid meta-heuristic, viz., ant colony optimisation-Nelder-Mead simplex hybrid (ACONM) for training multi-layer perceptron MLP instead of the back-propagation algorithm to solve classification problems. The resulting network is called ACONM-NN. In the ACONM, the ACO part is employed to find a promising region in the search space and then the simplex search is applied to quickly converge to the optimum solution. The influence of the parameters on the performance of the algorithm is also investigated. The effectiveness of the ACONM-NN is tested on the three benchmark datasets viz., Iris, Wine and Wisconsin Breast cancer and bankruptcy prediction problem in banks. The bankruptcy datasets analysed are that of Spanish, Turkish and US banks. Throughout the study we performed ten-fold cross validation. Based on the experiments, it is clear that the proposed ACONM-NN outperformed many other approaches in terms of accuracy.

A SCALABLE NEURAL-NETWORK MODULAR-ARRAY ARCHITECTURE FOR REAL-TIME MULTI-PARAMETER DAMAGE DETECTION IN PLATE STRUCTURES USING SINGLE SENSOR OUTPUT

A scalable modular neural network array architecture has been proposed for real time damage detection in plate like structures for structural health monitoring applications. Damages in a plate like structure are simulated using finite element method of numeric system simulation. Various damage states are numerically simulated by varying Young's modulus of the material at various locations of the structure. Transient vibratory loads are applied at one end of the beam and picked at the other end by means of point sensors. The vibration signals thus obtained are then filtered and subjected to wavelet transform (WT) based multi resolution analysis (MRA) to extract features and identify them. The redundant features are removed and only the principal features are retained using principal component analysis (PCA). A large database of principal features (the feature base) corresponding to different damage scenarios is created. This feature base is used to train individual multi layer perceptron (MLP) networks to identify different parameters of the damage such as location and extent (Young's modulus). Individually trained MLP units are then organized and connected in parallel so that different damage parameters can be identified almost simultaneously, on being fed with new signal feature vectors. For a given case, damage classification success rate has been found to be encouraging. The main feature of this implementation is that it is scalable. That is, any number of trained MLP units capable of identifying a certain parameter of damages can be integrated into the architecture and theoretically it will take almost the same time to identify various damage parameters irrespective of their numbers.

Flow-based anomaly detection in high-speed links using modified GSA-optimized neural network

Ever growing Internet causes the availability of information. However, it also provides a suitable space for malicious activities, so security is crucial in this virtual environment. The network intrusion detection system (NIDS) is a popular tool to counter attacks against computer networks. This valuable tool can be realized using machine learning methods and intrusion datasets. Traditional datasets are usually packet-based in which all network packets are analyzed for intrusion detection in a time-consuming process. On the other hand, the recent spread of 1–10-Gbps-technologies have clearly pointed out that scalability is a growing problem. In this way, flow-based solutions can help to solve the problem by reduction of data and processing time, opening the way to high-speed detection on large infrastructures. Besides, NIDS should be capable of detecting new malicious activities. Artificial neural network-based NIDSs can detect unseen attacks, so a multi-layer perceptron (MLP) neural classifier is used in this study to distinguish benign and malicious traffic in a flow-based NIDS. In this way, a modified gravitational search algorithm (MGSA), as a modern heuristic technique, is employed to optimize the interconnection weights of the neural anomaly detector. The proposed scheme is trained using an enhanced version of the first labeled flow-based dataset for intrusion detection introduced in 2009. In addition, the particle swarm optimization (PSO) algorithm and traditional error back-propagation (EBP) algorithm are employed to train MLP, so performance comparison becomes possible. The experimental results based on the actual network data show that the MGSA-optimized neural anomaly detector is effective for monitoring abnormal traffic flows in the gigabytes traffic environment, and the accuracy is about 97.8 %.

Performance Analysis of Neural Networks Training using Real Coded Genetic Algorithm

perceptrons (MLPs) are widely used for pattern classification and regression problems. Backpropagation (BP) algorithm is known technique in the training of multilayer perceptrons. However for its optimum training convergence, the learning and momentum parameters need to be tuned on trial and error method. Further, sometimes the backpropagation algorithm fails to achieve global convergence. To alleviate these problems we suggest a genetic algorithm based training for MLP network. Both binary coded and real coded genetic algorithm are used and a comparative training performance analysis has been studied. It is observed from simulation results that both the schemes outperform backpropagation algorithm and achieve global convergence. Further the real coded GA based training shows a faster convergence than binary coded GA based training. For simulation datasets are taken from UCI based machine learning repository. Hence real-coded genetic algorithm finds an alternative for back propagation based training algorithm.

Dynamic nonlinear state-space model with a neural network via improved sequential learning algorithm for an online real-time hydrological modeling

Summary This paper proposes a dynamic nonlinear state-space model with a neural network that uses a sequential learning algorithm capable of online simulation, in which the model predicts and adapts to the arrival of each new item of hydrological data in a sequential manner (as opposed to a ‘batch’), thereby enabling online real-time hydrological modeling. The improved sequential extended Kalman filtering (EKF) learning algorithm is developed to train multi-layer perceptron (MLP) neural networks, and is known as the MLP-EKF method with updating of noise covariance (MLP-EKFQ). It is herein proposed to allow the evolution of the weight of a neural network sequentially in time while also computing the noise covariance matrices of the EKF algorithm automatically by maximizing the evidence density function with respect to the noise covariance matrices. The proposed MLP-EKFQ was used to develop an online real-time warning system to predict river temperatures affected by the discharge of cooling water 1 km downstream of a thermal power station, from real-time to 2 h ahead, sequentially on the arrival of each new item of hydrological, meteorological, and power station operational data at 10 min intervals. It is demonstrated that the proposed MLP-EKFQ is superior in terms of both model performance and computational efficiency to those models that adopt a batch learning algorithm such as a multi-layer perceptron (MLP) system trained using the back-prorogation learning algorithm (MLP-BP), or an adaptive neural-fuzzy inference system (ANFIS). Due to its computational efficiency, its online simulation capability, and the high levels of accuracy achieved by the proposed MLP-EKFQ method, there is a great deal of potential for its use as an online dynamic hydrological modeling tool that may be suitable for a variety of complex dynamic and/or real-time tasks.

Training Multi-Layer Perceptrons with the Unscented Kalman Particle Filter

Many Bayesian learning approaches to multi-layer perceptrons (MLPs) parameters optimization have been proposed such as the extended Kalman filter (EKF). In this paper, a sequential approach is applied to train the MLPs. Based on the particle filter, the approach named unscented Kalman particle filter (UPF) uses the unscented Kalman filter as proposal distribution to generate the importance sampling density. The UPF are devised to deal with the high dimensional parameter space that is inherent to neural network models. Simulation results show that the new algorithm performs better than traditional optimization methods such as the extended Kalman filter.

A Comparative Study of Nonlinear Time-Varying Process Modeling Techniques: Application to Chemical Reactor

This paper proposes the design and a comparative study of two nonlinear systems modeling techniques. These two approaches are developed to address a class of nonlinear systems with time-varying parameter. The first is a Radial Basis Function (RBF) neural networks and the second is a Multi Layer Perceptron (MLP). The MLP model consists of an input layer, an output layer and usually one or more hidden layers. However, training MLP network based on back propagation learning is computationally expensive. In this paper, an RBF network is called. The parameters of the RBF model are optimized by two methods: the Gradient Descent (GD) method and Genetic Algorithms (GA). However, the MLP model is optimized by the Gradient Descent method. The performance of both models are evaluated first by using a numerical simulation and second by handling a chemical process known as the Continuous Stirred Tank Reactor CSTR. It has been shown that in both validation operations the results were successful. The optimized RBF model by Genetic Algorithms gave the best results.

Artificial Neural Network-Based Multiobjective Optimization of Mechanical Alloying Process for Synthesizing of Metal Matrix Nanocomposite Powder

The aim of this article was to optimize the mechanical alloying process for synthesizing of Al-8vol%SiC nanocomposite powders through an artificial neural network based on multiobjective optimization procedure. First, a suitable trained multi-layer perceptron (MLP) neural network was established for modeling purpose. Process variables as inputs of the network included milling time, milling speed, and balls to powders weight ratio. Parameters of the nanocomposite as outputs of the network were the crystallite size and the lattice strain of the aluminum matrix. The optimization was carried out by using two methods: gradient descent and pattern search. The aim of the optimization was to determine the minimum crystallite size and the maximum lattice strain of the aluminum matrix that could be obtained by regulating the mechanical alloying process variables. The response surfaces and the contour plots showed that the combination of the artificial neural network (ANN) and the optimization procedure were able to optimize the mechanical alloying process to synthesize Al-8vol%SiC nanocomposite.

Prevention of transient instability employing rules based on back propagation based ANN for series compensation

Power system preventive control is dominated by generation rescheduling. Other preventive control actions are rarely mentioned in the literature. This paper presents a transient stability preventive control design that takes series compensation into account. The design methodology is aided with the ‘IF–THEN’ rules extracted from a trained multilayer perceptron (MLP) artificial neural network (ANN). The proposed method can add more degrees of freedom by incorporation of series compensation, and have additional flexibility in deploying preventive control actions. Two preventive control schemes are presented and applied to 39-bus power systems. Extensive computational studies have demonstrated the effectiveness of the proposal.

Improving long-range hydrological forecasts with extended Kalman filters

There is a continuing effort to advance the skill of long-range hydrological forecasts to support water resources decision making. The present study investigates the potential of an extended Kalman filter approach to perform supervised training of a recurrent multilayer perceptron (RMLP) to forecast up to 12-month-ahead lake water levels and streamflows in Canada. The performance of the RMLP was compared with the conventional multilayer perceptron (MLP) using suites of diagnostic measures. The results of the forecasting experiment showed that the RMLP model was able to provide a robust modelling framework capable of describing complex dynamics of the hydrological processes, thereby yielding more accurate and realistic forecasts than the MLP model. The performance of the method in the present study is very promising; however, further investigation is required to ascertain the versatility of the approach in characterizing different water resources and environmental problems. Citation Muluye, G. Y. (2011) Improving long-range hydrological forecasts with extended Kalman filters. Hydrol. Sci. J. 56(7), 1118–1128.

Development and performance evaluation of neural network classifiers for Indian internet shoppers

The rapid growth of usage of internet has paved the way towards the use of online shopping. Consumers’ behavior is one of the significant aspects that is considered by the service providers for the improvement of various services. Consumers are generally satisfied if their needs are fulfilled. In this paper an in depth investigation is made on the behavior of Indian consumers towards online shopping. Factor analysis is carried out to extract significant factors that affect online shopping of Indian consumers and these consumers are clustered based on their behavior, towards online shopping using hierarchical clustering. Employing the results of clustering in training of multilayer perceptron (MLP), functional link artificial neural network (FLANN) and radial basis function (RBF) networks efficient classifier models are developed. The performance of these classifiers are evaluated and compared with those obtained by conventional statistical based discriminant analysis. The simulation study demonstrates that the RBF network provides best classification performance of internet shoppers compared to those given by the FLANN, MLP and discriminant analysis based methods. The simulation study on the impact of different combination of inputs demonstrates that demographic input has least effect on classification performance. On the other hand the combination of psychological and cultural inputs play the most significant role in classification followed by psychological and then cultural inputs alone.

Identifying individuality and variability in team tactics by means of statistical shape analysis and multilayer perceptrons

Offensive and defensive systems of play represent important aspects of team sports. They include the players’ positions at certain situations during a match, i.e., when players have to be on specific positions on the court. Patterns of play emerge based on the formations of the players on the court. Recognition of these patterns is important to react adequately and to adjust own strategies to the opponent. Furthermore, the ability to apply variable patterns of play seems to be promising since they make it harder for the opponent to adjust.The purpose of this study is to identify different team tactical patterns in volleyball and to analyze differences in variability. Overall 120 standard situations of six national teams in women’s volleyball are analyzed during a world championship tournament. Twenty situations from each national team are chosen, including the base defence position (start configuration) and the two players block with middle back deep (end configuration). The shapes of the defence formations at the start and end configurations during the defence of each national team as well as the variability of these defence formations are statistically analyzed. Furthermore these shapes data are used to train multilayer perceptrons in order to test whether artificial neural networks can recognize the teams by their tactical patterns.Results show significant differences between the national teams in both the base defence position at the start and the two players block with middle back deep at the end of the standard defence situation. Furthermore, the national teams show significant differences in variability of the defence systems and start-positions are more variable than the end-positions. Multilayer perceptrons are able to recognize the teams at an average of 98.5%.It is concluded that defence systems in team sports are highly individual at a competitive level and variable even in standard situations. Artificial neural networks can be used to recognize teams by the shapes of the players’ configurations. These findings support the concept that tactics and strategy have to be adapted for the team and need to be flexible in order to be successful.

Objective Functions of Online Weight Noise Injection Training Algorithms for MLPs

Injecting weight noise during training has been a simple strategy to improve the fault tolerance of multilayer perceptrons (MLPs) for almost two decades, and several online training algorithms have been proposed in this regard. However, there are some misconceptions about the objective functions being minimized by these algorithms. Some existing results misinterpret that the prediction error of a trained MLP affected by weight noise is equivalent to the objective function of a weight noise injection algorithm. In this brief, we would like to clarify these misconceptions. Two weight noise injection scenarios will be considered: one is based on additive weight noise injection and the other is based on multiplicative weight noise injection. To avoid the misconceptions, we use their mean updating equations to analyze the objective functions. For injecting additive weight noise during training, we show that the true objective function is identical to the prediction error of a faulty MLP whose weights are affected by additive weight noise. It consists of the conventional mean square error and a smoothing regularizer. For injecting multiplicative weight noise during training, we show that the objective function is different from the prediction error of a faulty MLP whose weights are affected by multiplicative weight noise. With our results, some existing misconceptions regarding MLP training with weight noise injection can now be resolved.

Extremal Optimization Combined with LM Gradient Search for MLP Network Learning

Gradient search based neural network training algorithm may suffer from local optimum, poor generalization and slow convergence. In this study, a novel Memetic Algorithm based hybrid method with the integration of “extremal optimization” and “Levenberg–Marquardt” is proposed to train multilayer perceptron (MLP) networks. Inheriting the advantages of the two approaches, the proposed “EO-LM” method can avoid local minima and improve MLP network learning performance in generalization capability and computation efficiency. The experimental tests on two benchmark problems and an application example for the end-point-prediction of basic oxygen furnace in steelmaking show the effectiveness of the proposed EO-LM algorithm.

The efficient incorporation of MLP features into automatic speech recognition systems

In recent years, the use of Multi-Layer Perceptron (MLP) derived acoustic features has become increasingly popular in automatic speech recognition systems. These features are typically used in combination with standard short-term spectral-based features, and have been found to yield consistent performance improvements. However there are a number of design decisions and issues associated with the use of MLP features for state-of-the-art speech recognition systems. Two modifications to the standard training/adaptation procedures are described in this work. First, the paper examines how MLP features, and the associated acoustic models, can be trained efficiently on large training corpora using discriminative training techniques. An approach that combines multiple individual MLPs is proposed, and this reduces the time needed to train MLPs on large amounts of data. In addition, to further speed up discriminative training, a lattice re-use method is proposed. The paper also examines how systems with MLP features can be adapted to a particular speakers, or acoustic environments. In contrast to previous work (where standard HMM adaptation schemes are used), linear input network adaptation is investigated. System performance is investigated within a multi-pass adaptation/combination framework. This allows the performance gains of individual techniques to be evaluated at various stages, as well as the impact in combination with other sub-systems. All the approaches considered in this paper are evaluated on an Arabic large vocabulary speech recognition task which includes both Broadcast News and Broadcast Conversation test data.

Using Neural Network with Speaker Applications

In Automatic Speech Recognition (ASR) the non-linear data projection provided by a one hidden layer Multilayer Perceptron (MLP), trained to recognize phonemes, and has previous experiments to provide feature enhancement substantially increased ASR performance, especially in noise. Previous attempts to apply an analogous approach to speaker identification have not succeeded in improving performance, except by combining MLP processed features with other features. We present test results for the TIMIT database which show that the advantage of MLP preprocessing for open set speaker identification increases with the number of speakers used to train the MLP and that improved identification is obtained as this number increases beyond sixty. We also present a method for selecting the speakers used for MLP training which further improves identification performance.