Types Of Activation Functions Research Articles

The Application of Artificial Neural Networks in Predicting Children’s Giftedness

Artificial neural networks have a wide use in the prediction and classification of different variables, but their application in the area of educational psychology is still relatively rare. The aim of this study was to examine the accuracy of artificial neural networks in predicting students’ general giftedness. The participants were 221 fourth grade students from one Croatian elementary school. The input variables for artificial neural networks were teachers’ and peers’ nominations, school grades, earlier school readiness assessment and parents’ education. The output variable was the result on the Standard Progressive Matrices (Raven, 1994), according to which students were classified as gifted or non-gifted. We tested two artificial neural networks’ algorithms: multilayer perceptron and radial basis function. Within each algorithm, a number of different types of activation functions were tested. 80% of the sample was used for training the network and the remaining 20% to test the network. For a criterion according to which students were classified as gifted if their result on the Standard Progressive Matrices was in the 95th centile or above, the best model was obtained by the hyperbolic tangent multilayer perceptron, which had a high accuracy of 100% of correctly classified non-gifted students and 75% correctly classified gifted students in the test sample. When the criterion was the 90th centile or above, the best model was also obtained by the hyperbolic tangent multilayer perceptron, but the accuracy was lower: 94.7% in the classification of non-gifted students and 66.7% in the classification of gifted students. The study has shown artificial neural networks’ potential in this area, which should be further explored. Keywords: gifted students, identification of gifted students, artificial neural networks

Multistability analysis of a general class of recurrent neural networks with non-monotonic activation functions and time-varying delays

This paper addresses the multistability for a general class of recurrent neural networks with time-varying delays. Without assuming the linearity or monotonicity of the activation functions, several new sufficient conditions are obtained to ensure the existence of (2K+1)n equilibrium points and the exponential stability of (K+1)n equilibrium points among them for n-neuron neural networks, where K is a positive integer and determined by the type of activation functions and the parameters of neural network jointly. The obtained results generalize and improve the earlier publications. Furthermore, the attraction basins of these exponentially stable equilibrium points are estimated. It is revealed that the attraction basins of these exponentially stable equilibrium points can be larger than their originally partitioned subsets. Finally, three illustrative numerical examples show the effectiveness of theoretical results.

A novel activation function for multilayer feed-forward neural networks

Traditional activation functions such as hyperbolic tangent and logistic sigmoid have seen frequent use historically in artificial neural networks. However, nowadays, in practice, they have fallen out of favor, undoubtedly due to the gap in performance observed in recognition and classification tasks when compared to their well-known counterparts such as rectified linear or maxout. In this paper, we introduce a simple, new type of activation function for multilayer feed-forward architectures. Unlike other approaches where new activation functions have been designed by discarding many of the mainstays of traditional activation function design, our proposed function relies on them and therefore shares most of the properties found in traditional activation functions. Nevertheless, our activation function differs from traditional activation functions on two major points: its asymptote and global extremum. Defining a function which enjoys the property of having a global maximum and minimum, turned out to be critical during our design-process since we believe it is one of the main reasons behind the gap observed in performance between traditional activation functions and their recently introduced counterparts. We evaluate the effectiveness of the proposed activation function on four commonly used datasets, namely, MNIST, CIFAR-10, CIFAR-100, and the Pang and Lee's movie review. Experimental results demonstrate that the proposed function can effectively be applied across various datasets where our accuracy, given the same network topology, is competitive with the state-of-the-art. In particular, the proposed activation function outperforms the state-of-the-art methods on the MNIST dataset.

Comparison of Different Neural Network Training Algorithms for Wind Velocity Forecasting

In this paper the wind speed is predicted by the use of data provided from the Mehrabad meteorological station located in Tehran, Iran, Collected between 2003 and 2008. A comprehensive analogy study is presented on Comparison of various Back Propagation neural networks methods in wind velocity forecasting. Four types of activation functions, namely, BFGS quasi-Newton, Bayesian regularized, Levenberg-Marquardt, and conjugate gradient algorithm, werestudied. The data was investigated by correlation coefficient and characterizing the amount of dependency between the wind speed and other input data. The meteorological parameters (pressure, direction, temperature and humidity) were used as input data, while the wind velocity is used as the output of the network.The results demonstrate that for the similar wind dataset, Bayesian Regularized algorithm can accurately predict compared with other method. In addition, choosing the type of activation function is dependent on the amount of input data, which should be acceptably large.

The Application of the Artificial Neural Network Ensemble Model for Simulating Streamflow

This study applies the ensemble method with artificial neural network (ANN) for simulating daily discharge. The study area is the Bocheong-cheon watershed, located in the central part of South Korea. The ANN outputs were generated by randomly sampling the initial weights, the nodes of hidden layer, and the training data. The correlation and principal component analysis are used for identifying the input vectors for ANN model. The impact affecting on the performance and the uncertainty of ANN ensemble model is investigated by considering the length of training data, the type of activation functions, the number of hidden nodes and training methods. The results indicate that the performance and uncertainty of the ANN ensemble model are affected by various elements related to ANN model development. The ensemble method applied in this case study provides satisfactory results for the stream flow modelling and is useful for assessing the uncertainty of ANN model. However, the computational efficiency of the ensemble method is not effective and might be problematic for an application that needs a rapid simulation.

An efficient hybrid kernel extreme learning machine approach for early diagnosis of Parkinson׳s disease

In this paper, we explore the potential of extreme learning machine (ELM) and kernel ELM (KELM) for early diagnosis of Parkinson’s disease (PD). In the proposed method, the key parameters including the number of hidden neuron and type of activation function in ELM, and the constant parameter C and kernel parameter γ in KELM are investigated in detail. With the obtained optimal parameters, ELM and KELM manage to train the optimal predictive models for PD diagnosis. In order to further improve the performance of ELM and KELM models, feature selection techniques are implemented prior to the construction of the classification models. The effectiveness of the proposed method has been rigorously evaluated against the PD data set in terms of classification accuracy, sensitivity, specificity and the area under the ROC (receiver operating characteristic) curve (AUC). Compared to the existing methods in previous studies, the proposed method has achieved very promising classification accuracy via 10-fold cross-validation (CV) analysis, with the highest accuracy of 96.47% and average accuracy of 95.97% over 10 runs of 10-fold CV.

Global Lagrange stability for neutral-type Cohen–Grossberg BAM neural networks with mixed time-varying delays

The paper discusses global Lagrange stability for a class of Cohen–Grossberg BAM neural networks of neutral-type with multiple time-varying and finite distributed delays by constructing appropriate Lyapunov-like functions. To this end, we first establish a new differential integral inequality for non-autonomous Cohen–Grossberg BAM neural networks with finite distributed delays. By using the new inequality and employing some other inequality techniques, we analyze two different types of activation functions which include both bounded and unbounded activation functions. Some easily verifiable criteria are obtained for global exponential attractive sets in which all trajectories converge. These results can also be applied to analyze monostable as well as multistable and more extensive neural networks due to making no assumptions on the number of equilibria. Meanwhile, the results obtained in this paper are more general and challenging than that of the existing references. Finally, some examples with numerical simulations are given and analyzed to verify our results.

Lagrange exponential stability for fuzzy Cohen–Grossberg neural networks with time-varying delays

This paper focuses on the globally exponential stability in Lagrange sense for Takagi–Sugeno (T–S) fuzzy Cohen–Grossberg neural networks with time-varying delays. By employing Lyapunov method and delay inequality technique, we analyze two different types of activation functions which include both Lipschitz function and general activation functions, several easily verifiable sufficient criteria about linear matrix inequality form are obtained to guarantee the Lagrange exponential stability of Cohen–Grossberg neural networks with time varying delays which are represented by T–S fuzzy models. Meanwhile, the estimations of the globally exponentially attractive sets are given. Here, the existence and uniqueness of the equilibrium points need not be considered. Finally, two numerical examples with simulations are given to illustrate the effectiveness of the theoretical results.

Review of Environmental and Hydrological Systems Modeling by A. W. JayawardenaCRC Press Taylor & Francis Group, Boca Raton, FL 33487; 2014; ISBN 978-0-415-46532-8; 516 pp.; $99.95.

Review of <i>Environmental and Hydrological Systems Modeling</i> by A. W. JayawardenaCRC Press Taylor &amp; Francis Group, Boca Raton, FL 33487; 2014; ISBN 978-0-415-46532-8; 516 pp.; $99.95.

CLASSIFICATION OF DIGITAL IMAGES USING FUSION ELEVATED ORDER CLASSIFIER IN WAVELET NEURAL NETWORK

The revival of wavelet neural networks obtained an extensive use in digital image processing. The shape representation, classification and detection play a very important role in the image analysis. Boosted Greedy Sparse Linear Discriminate Analysis (BGSLDA) trains the cascade level of detection in an efficient manner. With the application of reweighting concept and deployment of class-reparability criterion, lesser search was made on more efficient weak classifiers. At the same time, Multi-Scale Histogram of Oriented Gradients (MS-HOG) method removes the confined portions of images. MS-HOG algorithm includes the advanced recognition scenarios such as rotations transportations on multiple objects but does not perform effective feature classification. To overcome the drawbacks in classification of higher order units, Fusion Elevated Order Classifier (FEOC) method is introduced. FEOC contains a different fusion of high order units to deal with diverse datasets by making changes in the order of units with parametric considerations. FEOC uses a prominent value of input neurons for better fitting properties resulting in a higher level of learning parameters (i.e.,) weights. FEOC method features are reduced using feature subset collection method. However, elevation mechanisms are significantly applied to the neuron, neuron activation function type and finally in the higher order types of neural network with the functions of adaptive in nature. FEOC have evaluated sigma-pi network representing both the Elevated order Processing Unit (EPU) and pi-sigma network. The experimental performance of Fusion Elevated Order Classifier in the wavelet neural network is evaluated against BGSLDA and MS-HOG using Statlog (Landsat Satellite) Data Set from UCI repository. FEOC performed in MATLAB with factors such as classification accuracy rate, false positive error, computational cost, memory consumption, response time and higher order classifier rate.

Stability analysis in Lagrange sense for a class of BAM neural networks of neutral type with multiple time-varying delays

In this paper, Lagrange stability for a class of neutral type BAM neural networks with multiple time-varying and bounded or unbounded delays is studied. Various different types of activation functions are considered, including general bounded and unbounded activation functions. By constructing appropriate Lyapunov functions and applying inequality techniques, some easily verifiable algebraic criteria for the ultimate boundedness and globally attractive sets of neutral type BAM neural networks are obtained. Meanwhile, the estimations of the positive invariant set, globally attractive set and globally exponentially attractive set are given. In particular, the estimations of the globally exponentially attractive sets with respect to the partial variables and the globally attractive sets of the systems are derived. These results here generalize and improve the earlier publications and can be applied to monostable and multistable neural networks as well as chaos control and chaos synchronization. Finally, some examples with numerical simulations are also given to demonstrate our results. HighlightsWe discuss Lagrange stability of neutral type BAM neural networks.We give the detailed estimations for the globally attractive sets.The delays are multi-time-varying bounded or unbounded but not distributed delays.The results here generalize and improve the earlier publications.The two sufficient conditions of poly-stability in Lagrange sense are also given.

Nonlinearly Activated Neural Network for Solving Time-Varying Complex Sylvester Equation

The Sylvester equation is often encountered in mathematics and control theory. For the general time-invariant Sylvester equation problem, which is defined in the domain of complex numbers, the Bartels-Stewart algorithm and its extensions are effective and widely used with an O(n³) time complexity. When applied to solving the time-varying Sylvester equation, the computation burden increases intensively with the decrease of sampling period and cannot satisfy continuous realtime calculation requirements. For the special case of the general Sylvester equation problem defined in the domain of real numbers, gradient-based recurrent neural networks are able to solve the time-varying Sylvester equation in real time, but there always exists an estimation error while a recently proposed recurrent neural network by Zhang et al [this type of neural network is called Zhang neural network (ZNN)] converges to the solution ideally. The advancements in complex-valued neural networks cast light to extend the existing real-valued ZNN for solving the time-varying real-valued Sylvester equation to its counterpart in the domain of complex numbers. In this paper, a complex-valued ZNN for solving the complex-valued Sylvester equation problem is investigated and the global convergence of the neural network is proven with the proposed nonlinear complex-valued activation functions. Moreover, a special type of activation function with a core function, called sign-bi-power function, is proven to enable the ZNN to converge in finite time, which further enhances its advantage in online processing. In this case, the upper bound of the convergence time is also derived analytically. Simulations are performed to evaluate and compare the performance of the neural network with different parameters and activation functions. Both theoretical analysis and numerical simulations validate the effectiveness of the proposed method.

Predicting expected progeny difference for marbling score in Angus cattle using artificial neural networks and Bayesian regression models

BackgroundArtificial neural networks (ANN) mimic the function of the human brain and are capable of performing massively parallel computations for data processing and knowledge representation. ANN can capture nonlinear relationships between predictors and responses and can adaptively learn complex functional forms, in particular, for situations where conventional regression models are ineffective. In a previous study, ANN with Bayesian regularization outperformed a benchmark linear model when predicting milk yield in dairy cattle or grain yield of wheat. Although breeding values rely on the assumption of additive inheritance, the predictive capabilities of ANN are of interest from the perspective of their potential to increase the accuracy of prediction of molecular breeding values used for genomic selection. This motivated the present study, in which the aim was to investigate the accuracy of ANN when predicting the expected progeny difference (EPD) of marbling score in Angus cattle. Various ANN architectures were explored, which involved two training algorithms, two types of activation functions, and from 1 to 4 neurons in hidden layers. For comparison, BayesCπ models were used to select a subset of optimal markers (referred to as feature selection), under the assumption of additive inheritance, and then the marker effects were estimated using BayesCπ with π set equal to zero. This procedure is referred to as BayesCpC and was implemented on a high-throughput computing cluster.ResultsThe ANN with Bayesian regularization method performed equally well for prediction of EPD as BayesCpC, based on prediction accuracy and sum of squared errors. With the 3K-SNP panel, for example, prediction accuracy was 0.776 using BayesCpC, and ranged from 0.776 to 0.807 using BRANN. With the selected 700-SNP panel, prediction accuracy was 0.863 for BayesCpC and ranged from 0.842 to 0.858 for BRANN. However, prediction accuracy for the ANN with scaled conjugate gradient back-propagation was lower, ranging from 0.653 to 0.689 with the 3K-SNP panel, and from 0.743 to 0.793 with the selected 700-SNP panel.ConclusionsANN with Bayesian regularization performed as well as linear Bayesian regression models in predicting additive genetic values, supporting the idea that ANN are useful as universal approximators of functions of interest in breeding contexts.

A CLUSTERING-BASED FUZZY WAVELET NEURAL NETWORK MODEL FOR SHORT-TERM LOAD FORECASTING

Load forecasting is a critical element of power system operation, involving prediction of the future level of demand to serve as the basis for supply and demand planning. This paper presents the development of a novel clustering-based fuzzy wavelet neural network (CB-FWNN) model and validates its prediction on the short-term electric load forecasting of the Power System of the Greek Island of Crete. The proposed model is obtained from the traditional Takagi-Sugeno-Kang fuzzy system by replacing the THEN part of fuzzy rules with a "multiplication" wavelet neural network (MWNN). Multidimensional Gaussian type of activation functions have been used in the IF part of the fuzzyrules. A Fuzzy Subtractive Clustering scheme is employed as a pre-processing technique to find out the initial set and adequate number of clusters and ultimately the number of multiplication nodes in MWNN, while Gaussian Mixture Models with the Expectation Maximization algorithm are utilized for the definition of the multidimensional Gaussians. The results corresponding to the minimum and maximum power load indicate that the proposed load forecasting model provides significantly accurate forecasts, compared to conventional neural networks models.

Empirical model for the prediction of lateral stability coefficient for un-trenched submarine pipes based on self-evolving neural network (SEANN)

The breakout failure mechanism of un-trenched submarine pipelines subject to current-induced hydrodynamic loading involves complex interactions between the pipe and the surrounding soil. Methods commonly used in conventional stability analyses of pipelines do not adequately consider several important factors, such as the effects of seabed slope and pipe anchorages. In this paper, a self-evolving artificial neural network (SEANN) is used to develop an empirical relation between the coefficient of ultimate lateral soil resistance to breakout, the properties of the pipe, and the supporting seabed. Additional parameters such as the angle of inclination of the seabed to the horizontal and the pipe's rolling restraint condition are also included as variables affecting the resistance coefficient. The results of the model validation with previous experimental data demonstrate the abilities of the proposed SEANN. Its performance was found to surpass that of BPN networks with various types of activation functions. Compared with the method described in the DNV manual, the proposed model provides better predictions of the ultimate soil resistance.

A growing and pruning sequential learning algorithm of hyper basis function neural network for function approximation

Radial basis function (RBF) neural network is constructed of certain number of RBF neurons, and these networks are among the most used neural networks for modeling of various nonlinear problems in engineering. Conventional RBF neuron is usually based on Gaussian type of activation function with single width for each activation function. This feature restricts neuron performance for modeling the complex nonlinear problems. To accommodate limitation of a single scale, this paper presents neural network with similar but yet different activation function—hyper basis function (HBF). The HBF allows different scaling of input dimensions to provide better generalization property when dealing with complex nonlinear problems in engineering practice. The HBF is based on generalization of Gaussian type of neuron that applies Mahalanobis-like distance as a distance metrics between input training sample and prototype vector. Compared to the RBF, the HBF neuron has more parameters to optimize, but HBF neural network needs less number of HBF neurons to memorize relationship between input and output sets in order to achieve good generalization property. However, recent research results of HBF neural network performance have shown that optimal way of constructing this type of neural network is needed; this paper addresses this issue and modifies sequential learning algorithm for HBF neural network that exploits the concept of neuron’s significance and allows growing and pruning of HBF neuron during learning process. Extensive experimental study shows that HBF neural network, trained with developed learning algorithm, achieves lower prediction error and more compact neural network.

A fast approach for detection of erythemato-squamous diseases based on extreme learning machine with maximum relevance minimum redundancy feature selection

In this paper, a novel hybrid method, which integrates an effective filter maximum relevance minimum redundancy (MRMR) and a fast classifier extreme learning machine (ELM), has been introduced for diagnosing erythemato-squamous (ES) diseases. In the proposed method, MRMR is employed as a feature selection tool for dimensionality reduction in order to further improve the diagnostic accuracy of the ELM classifier. The impact of the type of activation functions, the number of hidden neurons and the size of the feature subsets on the performance of ELM have been investigated in detail. The effectiveness of the proposed method has been rigorously evaluated against the ES disease dataset, a benchmark dataset, from UCI machine learning database in terms of classification accuracy. Experimental results have demonstrated that our method has achieved the best classification accuracy of 98.89% and an average accuracy of 98.55% via 10-fold cross-validation technique. The proposed method might serve as a new candidate of powerful methods for diagnosing ES diseases.

Horseshoe Chaos in a 3D Neural Network with Different Activation Functions

This paper studies a small neural network with three neurons. First, the activation function takes the sign function. Although the network is a simple hybrid system with all subsystems being exponentially stable, we find that it can exhibit very complex dynamics such as limit cycles and chaos. Since the sign function is a limit case of sigmoidal functions, we find that chaos robustly exists with some different activation functions, which implies that such chaos in this network is more related to its weight matrix than the type of activation functions. For chaos, we present a rigorous computer-assisted study by virtue of topological horseshoe theory.

Modeling of Drying Process of Bittim Nuts (pistacia terebinthus) in a Fixed Bed Dryer System by Using Extreme Learning Machine

Abstract Artificial neural networks (ANNs) have been widely used in modeling of various systems. Training of ANNs is commonly performed by backpropagation based on a gradient-based learning rule. However, it is well-known that such learning rule has several shortcomings such as slow convergence and training failures. This paper proposes a modeling technique based on Extreme Learning Machine (ELM) eliminating disadvantages of backpropagation based on a gradient-based learning rule for the drying of bittim (pistacia terebinthus). The samples for ELM based model are obtained by experimental studies. In experimental studies, the sample mass loss rate as a function time was investigated in different air velocities (0.5 and 1 m/s) and air temperatures (40, 60 and 80°C) in a designed dryer system. The obtained samples from experiments are used for training and testing of ELM. Further, some parameters of ELM such as type of activation function and the number of hidden neurons are set to obtain the best possible modelling results. The obtained prediction results show that ELM algorithm with tangent sigmoid activation function and 20 hidden neurons is appeared to be most optimal topology since maximum R2 and minimum rms (0.0500) and cov (0.2256) values are obtained. Thus, it is concluded that ELM can be used as an effective modelling tool in the drying of bittim (pistacia terebinthus) in fixed bed dryer system.

An artificial immune-based hybrid multi-layer feedforward neural network for predicting grid-connected photovoltaic system output

This paper presents a Hybrid Multi-Layer Feedforward Neural Network (HMLFNN) technique for predicting the output from a grid-connected photovoltaic (GCPV) system. In the proposed HMLFNN, the Artificial Immune System (AIS) was selected as the optimizer for the training process of the Multi-Layer Feedforward Neural Network (MLFNN). AIS was used to optimize the number of neurons in the hidden layer, the learning rate, the momentum rate, the type of activation function and the learning algorithm. In addition, the MLFNN utilized solar irradiance (SI) and module temperature (MT) as its inputs and kWh energy as its output. When compared with the classically trained MLFNN, the proposed HMLFNN was found to be superior in terms of having shortest computation time and lower prediction error.