Types Of Activation Functions Research Articles

Prediction of ship fuel consumption by using an artificial neural network

A smart ship collects various data with large volume, such as voyage, machinery, and weather data. Thus, big data analysis for smart ships is an important technology that can be widely applied to improve ship maintenance, operational efficiency, and equipment life management. In this study, an accurate regression model for the fuel consumption of the main engine by using an artificial neural network (ANN) was proposed by big data analysis including data collection, clustering, compression, and expansion. To obtain an accurate regression model, various numbers of hidden layers and neurons and different types of activation functions were tested in the ANN, and their effects on the accuracy and efficiency of the regression analysis were studied. The proposed regression model using ANN is a more accurate and efficient model to predict the fuel consumption of the main engine than polynomial regression and support vector machine.

Classification of cerebral microbleeds based on fully-optimized convolutional neural network

Cerebral microbleeds are important biomarkers of many cerebrovascular diseases and cognitive dysfunctions. Their distribution patterns can indicate some underlying aetiologies. Hitherto, few researches tried to detect cerebral microbleeds accurately and automatically. Some improvements have been achieved via traditional machine learning methods. In this paper, we proposed a method based on convolutional neural network (CNN) for further improving the performance. Firstly, sliding neighborhood processing method was applied to generate the input and target datasets based on 10 3D brain images of cerebral autosomal-dominant arteriopathy with subcortical infarcts and Leukoencephalopathy scanned by susceptibility-weighted imaging (SWI). Then, CNN was used to classify the cerebral microbleeds. To exert the full-power of convolutional neural network, almost all hyperparameters of CNN structure that could affect the performance were tested, such as the number of layers, type of activation function, pooling method, and filter size. A fully-optimized convolutional neural network structure for cerebral microbleeds classification was obtained. It performed better than four existed state-of-the-art approaches with a sensitivity of 99.74%, a specificity of 96.89% and an accuracy of 98.32%.

Dirichlet sınır değer koşullarına sahip adi diferansiyel denklemlerin nümerik çözümleri için basit tekrarlayan sinir ağları

In this study, we consider Dirichlet Boundary Value Problems (DBVPs) for Ordinary Differential Equations (ODEs) to illustrate the general procedure of obtaining numerical solutions using simple Recurrent Neural Networks (RNNs).  Different types of both linear and nonlinear activation functions are used in the neural network.  The network is trained by Particle Swarm Optimization (PSO) method, and cross validation approach is performed to tune the arbitrary parameters of neural nets.  The exact solutions and the obtained neural net solutions, regarding with the types of activation functions, are compared to determine the efficiency of using RNNs in solving the problem.  In all cases, the exact solutions are confronted with those obtained from RNNs in the context of absolute errors and average mean squared errors (MSEs) with standard deviations.

A Prediction Methodology of Energy Consumption Based on Deep Extreme Learning Machine and Comparative Analysis in Residential Buildings

In this paper, we have proposed a methodology for energy consumption prediction in residential buildings. The proposed method consists of four different layers, namely data acquisition, preprocessing, prediction, and performance evaluation. For experimental analysis, we have collected real data from four multi-storied residential building. The collected data are provided as input for the acquisition layer. In the pre-processing layer, several data cleaning and preprocessing schemes were deployed to remove abnormalities from the data. In the prediction layer, we have used the deep extreme learning machine (DELM) for energy consumption prediction. Further, we have also used the adaptive neuro-fuzzy inference system (ANFIS) and artificial neural network (ANN) in the prediction layer. In the DELM different numbers of hidden layers, different hidden neurons, and various types of activation functions have been used to achieve the optimal structure of DELM for energy consumption prediction. Similarly, in the ANN, we have employed a different combination of hidden neurons with different types of activation functions to get the optimal structure of ANN. To obtain the optimal structure of ANFIS, we have employed a different number and type of membership functions. In the performance evaluation layer for the comparative analysis of three prediction algorithms, we have used the mean absolute error (MAE), root mean square error (RMSE) and mean absolute percentage error (MAPE). The results indicate that the performance of DELM is far better than ANN and ANFIS for one-week and one-month hourly energy prediction on the given data.

Effect of Successive Convolution Layers to Detect Gender

Image classification can be defined as one of the most important tasks in the area of machine learning. Recently, deep neural networks, especially deep convolution networks, have participated greatly in end-to-end learning which reduce need for human designed features in the image recognition like Convolution Neural Network. It is offers the computation models which are made up of several processing layers for learning data representations with several abstraction levels. In this work, a pre-trained deep CNN is utilized according to some parameters like filter size, no of convolution, pooling, fully connected and type of activation function which includes 300 images for training and predict 100 image gender using probability measures. Results in Classification and precision accuracy equal to 0.68 and 0.3225 respectively.

Design and Analysis of Sentiment Classification Model for Korean Music Reviews based on Convolutional Neural Networks

최근 텍스트 데이터의 감정 분류에 컨볼루션 신경망을 적용한 결과 효율적인 성능을 나타내고 있음을 여러 연구에서 보이고 있다. 본 연구에서는 한글로 작성된 음악 리뷰의 감정을 분류하는 컨볼루션 신경망 기반의 모델을 설계하고, 설계한 모델의 성능을 분석한다. 일반적으로 문자로 이루어진 음악 리뷰는 신경망 기반의 모델에서 사용할 수 있도록 벡터로 변환되어야 하는데, 본 연구에서는 룩업 테이블을 사용하여 문자를 벡터로 변환하였고, 전체 음악 리뷰의 80%는 학습 데이터, 20%는 평가 데이터로 구분하여 사용하였다. 또한 감정 분류 모델은 한 개의 컨볼루션 층과 두 개의 완전 연결 층을 갖도록 설계하였다. 그리고 이들을 사용하여 네 가지 활성화 함수와 함께 음악 리뷰의 감정 분류를 수행한 결과, Leaky ReLU 함수를 사용하였을 때 분류 정확도가 가장 높았으며, SELU 함수를 사용하였을 때 학습 속도가 가장 빠르고 학습이 안정적으로 이루어지는 모습을 확인하였다. 또한 학습이 거듭될수록 감정 분류 정확도는 활성화 함수의 종류에 상관없이 높은 수준의 정확도로 수렴함을 보였고, 이러한 실험 결과들은 컨볼루션 신경망을 사용하여 텍스트 데이터의 감정을 분류하는 연구에 기여할 수 있을 것으로 본다.Recently, several researches have shown that sentiment analysis of text data using convolutional neural networks shows efficient performance. In this study, we design a model based on convolutional neural networks that classifies the sentiment of music reviews written in Korean and analyze the performance of the designed model. Generally, a reviews composed of letters should be converted to vectors for use in a neural network based model. In this study, we converted letters to vectors using a lookup table. 80% of the reviews were used as learning data and 20% as evaluation data. The model is designed to have one convolution layer and two fully connected layers. The results showed that classification accuracy was the highest when Leaky ReLU was used, and training speed was the fastest and learning was stable when SELU was used. In addition, as the learning progressed, the accuracy converged to a high level irrespective of the type of activation function, and these experimental results can contribute to the sentiment classification of text data using convolutional neural networks.

Global Lagrange Stability for Takagi-Sugeno Fuzzy Cohen-Grossberg BAM Neural Networks with Time-varying Delays

This paper concerns the globally exponential stability in Lagrange sense for Takagi-Sugeno (T-S) fuzzy Cohen-Grossberg BAM neural networks with time-varying delays. Based on the Lyapunov functional method and inequality techniques, two different types of activation functions which include both Lipschitz function and general activation functions are analyzed. Several sufficient conditions in linear matrix inequality form are derived to guarantee the Lagrange exponential stability of Cohen-Grossberg BAM neural networks with time-varying delays which are represented by T-S fuzzy models. Finally, simulation results demonstrate the effectiveness of the theoretical results.

Effective lossy and lossless color image compression with Multilayer Perceptron

This paper presents the effective lossy and lossless color image compression algorithm with Multilayer perceptron. The parallel structure of neural network and the concept of image compression combined to yield a better reconstructed image with constant bit rate and less computation complexity. Original color image component has been divided into 8x8 blocks. The discrete cosine transform (DCT) applied on each block for lossy compression or discrete wavelet transform (DWT) applied for lossless image compression. The output coefficient values have been normalized by using mod function. These normalized vectors have been passed to Multilayer Perceptron (MLP). This proposed method implements the Back propagation neural network (BPNN) which is suitable for compression process with less convergence time. Performance of the proposed compression work is evaluated based on three ways. First one compared the performance of lossy and lossless compression with BPNN. Second one, evaluated based on different sized hidden layers and proved that increased neurons in hidden layer has been preserved the brightness of an image. Third, the evaluation based on three different types of activation function and the result shows that each function has its own merit. Proposed algorithm has been competed with existing JPEG color compression algorithm based on PSNR measurement. Resultant value denotes that the proposed method well performed to produce the better reconstructed image with PSNR value approximately increased by 21.62%.

A novel type of activation function in artificial neural networks: Trained activation function

Determining optimal activation function in artificial neural networks is an important issue because it is directly linked with obtained success rates. But, unfortunately, there is not any way to determine them analytically, optimal activation function is generally determined by trials or tuning. This paper addresses, a simpler and a more effective approach to determine optimal activation function. In this approach, which can be called as trained activation function, an activation function was trained for each particular neuron by linear regression. This training process was done based on the training dataset, which consists the sums of inputs of each neuron in the hidden layer and desired outputs. By this way, a different activation function was generated for each neuron in the hidden layer. This approach was employed in random weight artificial neural network (RWN) and validated by 50 benchmark datasets. Achieved success rates by RWN that used trained activation functions were higher than obtained success rates by RWN that used traditional activation functions. Obtained results showed that proposed approach is a successful, simple and an effective way to determine optimal activation function instead of trials or tuning in both randomized single and multilayer ANNs.

Multilayer perceptron architecture optimization using parallel computing techniques.

The objective of this research was to develop a methodology for optimizing multilayer-perceptron-type neural networks by evaluating the effects of three neural architecture parameters, namely, number of hidden layers (HL), neurons per hidden layer (NHL), and activation function type (AF), on the sum of squares error (SSE). The data for the study were obtained from quality parameters (physicochemical and microbiological) of milk samples. Architectures or combinations were organized in groups (G1, G2, and G3) generated upon interspersing one, two, and three layers. Within each group, the networks had three neurons in the input layer, six neurons in the output layer, three to twenty-seven NHL, and three AF (tan-sig, log-sig, and linear) types. The number of architectures was determined using three factorial-type experimental designs, which reached 63, 2 187, and 50 049 combinations for G1, G2 and G3, respectively. Using MATLAB 2015a, a logical sequence was designed and implemented for constructing, training, and evaluating multilayer-perceptron-type neural networks using parallel computing techniques. The results show that HL and NHL have a statistically relevant effect on SSE, and from two hidden layers, AF also has a significant effect; thus, both AF and NHL can be evaluated to determine the optimal combination per group. Moreover, in the three study groups, it is observed that there is an inverse relationship between the number of processors and the total optimization time.

Some Tricks in Parameter Selection for Extreme Learning Machine

Extreme learning machine (ELM) is a widely used neural network with random weights (NNRW), which has made great contributions to many fields. However, the relationship between the parameters and the performance of ELM has not been fully investigated yet, i.e. the impact of the number of hidden layer nodes, the randomization range of the weights between the input layer and hidden layer, the randomization range of the threshold of hidden nodes, and the type of activation functions. In this paper, eight benchmark functions are used to study this relationship. We have some interesting findings, such as more hidden layer nodes cannot guarantee the best performance of ELM, the empirical randomization range of the hidden weights (i.e., [-1, 1]) and the empirical randomization range of the threshold of hidden nodes (i.e., [0, 1]) may not lead to the optimal performance of ELM models, and ELM with sigmoid as the activation function always achieves better performance on some regression problems than ELM with tribas as the activation function. We hope the findings from our work could provide a useful guidance for researchers to select right parameters for ELM.

Implementation of Fixed-point Neuron Models with Threshold, Ramp and Sigmoid Activation Functions

This paper presents the hardware implementation of single-neuron models with three types of activation functions using fixed-point data format on Field Programmable Gate Arrays (FPGA). Activation function defines the transfer behavior of a neuron model and consequently the Artificial Neural Network (ANN) constructed using it. This paper compared single neuron models designed with bipolar ramp, threshold and sigmoid activation functions. It is also demonstrated that the FPGA hardware implementation performance can be significantly improved by using 16-bit fixed-point data format instead of 32-bit floating-point data format for the neuron model with sigmoid activation function.

Ultrasound-based differentiation of malignant and benign thyroid Nodules: An extreme learning machine approach

Background and objectivesIt is important to be able to accurately distinguish between benign and malignant thyroid nodules in order to make appropriate clinical decisions. The purpose of this study was to improve the effectiveness and efficiency for discriminating the malignant from benign thyroid cancers based on the Ultrasonography (US) features. MethodsThere were 114 benign nodules in 106 patients (82 women and 24 men) and 89 malignant nodules in 81 patients (69 women and 12 men) included in this study. The potential of extreme learning machine (ELM) has been explored for the first time to discriminate malignant and benign thyroid nodules based on the sonographic features in ultrasound images. The influence of two key parameters (the number of hidden neurons and type of activation function) on the performance of ELM was investigated. The relationship between feature subsets obtained by the feature selection method and the classification performance of ELM was also examined. A real-life dataset was used to evaluate the effectiveness of the proposed method in terms of classification accuracy, sensitivity, specificity, and area under the ROC (receiver operating characteristic) curve (AUC). ResultsThe results demonstrate that there are significant differences between the malignant and benign thyroid nodules (p-value<0.01), the most discriminative features are echogenicity, calcification, margin, composition and shape. Compared with other methods, the proposed method not only has achieved very promising classification accuracy via 10-fold cross-validation (CV) scheme, but also greatly reduced the computational cost compared to other counterparts. The proposed ELM-based approach achieves 87.72% ACC, 0.8672 AUC, 78.89% sensitivity, and 94.55% specificity. ConclusionsBased on the empirical analysis, the proposed ELM-based approach for thyroid cancer detection has promising potential in clinical use, and it can be of assistance as an optional tool for the clinicians.

Robust Finite-Time Stabilization of Fractional-Order Neural Networks With Discontinuous and Continuous Activation Functions Under Uncertainty.

This paper is concerned with robust finite-time stabilization for a class of fractional-order neural networks (FNNs) with two types of activation functions (i.e., discontinuous and continuous activation function) under uncertainty. It is worth noting that there exist few results about FNNs with discontinuous activation functions, which is mainly because classical solutions and theories of differential equations cannot be applied in this case. Especially, there is no relevant finite-time stabilization research for such system, and this paper makes up for the gap. The existence of global solution under the framework of Filippov for such system is guaranteed by limiting discontinuous activation functions. According to set-valued analysis and Kakutani's fixed point theorem, we obtain the existence of equilibrium point. In particular, based on differential inclusion theory and fractional Lyapunov stability theory, several new sufficient conditions are given to ensure finite-time stabilization via a novel discontinuous controller, and the upper bound of the settling time for stabilization is estimated. In addition, we analyze the finite-time stabilization of FNNs with Lipschitz-continuous activation functions under uncertainty. The results of this paper improve corresponding ones of integer-order neural networks with discontinuous and continuous activation functions. Finally, three numerical examples are given to show the effectiveness of the theoretical results.

Mean-square global exponential stability in Lagrange sense for delayed recurrent neural networks with Markovian switching

The paper discusses the mean-square global exponential stability in Lagrange sense for delayed recurrent neural networks (DRNNs) with Markovian switching. Two different types of activation functions are considered, which include both bounded and unbounded activation functions. By using the vector Lyapunov function and stochastic analysis technique, we establish two L-operator differential inequalities. By employing the two L-operator differential inequalities and vector Lyapunov functions methods, we provide easily verifiable criteria for Lagrange stability in mean-square sense of DRNNs with Markovian switching. Finally, two numerical examples are given to illustrate the efficiency of the derived results.

Quaternion neuro-fuzzy learning algorithm for generation of fuzzy rules

The conventional neuro-fuzzy learning algorithm with Gaussian membership functions based on the gradient descent method can generate or tune fuzzy rules. However, increasing the number of inputs greatly increases the number of parameters. Thus, representing fuzzy rule tables is difficult, learning time increases, and learning accuracy decreases. To overcome these problems, we have developed a complex-valued neuro-fuzzy learning algorithm that extends the neuro-fuzzy learning algorithm to complex numbers. In the method, the inputs, antecedent membership functions, and consequent singletons are complex numbers, but the outputs are real. For converting complex to real numbers, we proposed two types of activation function. The complex-valued method reduced the parameter numbers and showed better learning accuracy. In this paper, we extend the method to the quaternion domain. In the quaternion neuro-fuzzy learning algorithm, the inputs, antecedent membership functions, and consequent singletons are quaternion, and the outputs are real. For parameter tuning, we derived the quaternion back propagation of quaternion neural networks that outputs real values for quaternion-valued inputs. The quaternion back propagation shows better learning convergence and accuracy than the conventional back propagation, and the tuning process is more complex, although it benefits from the quaternion back propagation. The method assigns a four-dimensional real number to one real and three imaginary parts of a quaternion number, which is used as a single quaternion input. This process greatly reduces the number of tuned parameters, leading to better learning than the conventional method. We compare the proposed and conventional methods using several function identification problems, and show that the proposed method outperforms its counterpart, making it a useful tool for learning in a fuzzy system model. In the best cases, the number of epochs was reduced to one-fortieth and the error to one-thirtieth of those in the conventional method.

Complex Neural Network Models for Time-Varying Drazin Inverse.

Two complex Zhang neural network (ZNN) models for computing the Drazin inverse of arbitrary time-varying complex square matrix are presented. The design of these neural networks is based on corresponding matrix-valued error functions arising from the limit representations of the Drazin inverse. Two types of activation functions, appropriate for handling complex matrices, are exploited to develop each of these networks. Theoretical results of convergence analysis are presented to show the desirable properties of the proposed complex-valued ZNN models. Numerical results further demonstrate the effectiveness of the proposed models.

The Application of Artificial Neural Networks in Predicting Children’s Giftedness

Artificial neural networks have a wide use in the prediction and classification of different variables, but their application in the area of educational psychology is still relatively rare. The aim of this study was to examine the accuracy of artificial neural networks in predicting students’ general giftedness. The participants were 221 fourth grade students from one Croatian elementary school. The input variables for artificial neural networks were teachers’ and peers’ nominations, school grades, earlier school readiness assessment and parents’ education. The output variable was the result on the Standard Progressive Matrices (Raven, 1994), according to which students were classified as gifted or non-gifted. We tested two artificial neural networks’ algorithms: multilayer perceptron and radial basis function. Within each algorithm, a number of different types of activation functions were tested. 80% of the sample was used for training the network and the remaining 20% to test the network. For a criterion according to which students were classified as gifted if their result on the Standard Progressive Matrices was in the 95th centile or above, the best model was obtained by the hyperbolic tangent multilayer perceptron, which had a high accuracy of 100% of correctly classified non-gifted students and 75% correctly classified gifted students in the test sample. When the criterion was the 90th centile or above, the best model was also obtained by the hyperbolic tangent multilayer perceptron, but the accuracy was lower: 94.7% in the classification of non-gifted students and 66.7% in the classification of gifted students. The study has shown artificial neural networks’ potential in this area, which should be further explored. Keywords: gifted students, identification of gifted students, artificial neural networks

Multistability analysis of a general class of recurrent neural networks with non-monotonic activation functions and time-varying delays

This paper addresses the multistability for a general class of recurrent neural networks with time-varying delays. Without assuming the linearity or monotonicity of the activation functions, several new sufficient conditions are obtained to ensure the existence of (2K+1)n equilibrium points and the exponential stability of (K+1)n equilibrium points among them for n-neuron neural networks, where K is a positive integer and determined by the type of activation functions and the parameters of neural network jointly. The obtained results generalize and improve the earlier publications. Furthermore, the attraction basins of these exponentially stable equilibrium points are estimated. It is revealed that the attraction basins of these exponentially stable equilibrium points can be larger than their originally partitioned subsets. Finally, three illustrative numerical examples show the effectiveness of theoretical results.

A novel activation function for multilayer feed-forward neural networks

Traditional activation functions such as hyperbolic tangent and logistic sigmoid have seen frequent use historically in artificial neural networks. However, nowadays, in practice, they have fallen out of favor, undoubtedly due to the gap in performance observed in recognition and classification tasks when compared to their well-known counterparts such as rectified linear or maxout. In this paper, we introduce a simple, new type of activation function for multilayer feed-forward architectures. Unlike other approaches where new activation functions have been designed by discarding many of the mainstays of traditional activation function design, our proposed function relies on them and therefore shares most of the properties found in traditional activation functions. Nevertheless, our activation function differs from traditional activation functions on two major points: its asymptote and global extremum. Defining a function which enjoys the property of having a global maximum and minimum, turned out to be critical during our design-process since we believe it is one of the main reasons behind the gap observed in performance between traditional activation functions and their recently introduced counterparts. We evaluate the effectiveness of the proposed activation function on four commonly used datasets, namely, MNIST, CIFAR-10, CIFAR-100, and the Pang and Lee's movie review. Experimental results demonstrate that the proposed function can effectively be applied across various datasets where our accuracy, given the same network topology, is competitive with the state-of-the-art. In particular, the proposed activation function outperforms the state-of-the-art methods on the MNIST dataset.