Activation Function Of Neural Network Research Articles

Periodic Function as Activation Function for Neural Networks

In this paper, we explore the periodic function as alternative activation function for neural network. Previously sigmoid function is used as standard activation function for neuron and now linear rectifier function are used. Even max-out function can be learned as a general form convex activation function. We explore the possibility in the other direction, where we use periodic function as activation function. We expect network with less layer and less neuron can capture the target distribution. The experiments verify our expectation and show that period function can act as an alternative activation function.

CTL-DNNet: Effective Circular Traffic Light Recognition with a Deep Neural Network

This paper proposed an algorithm for a deep learning network for identifying circular traffic lights (CTL-DNNet). The sample labeling process uses translation to increase the number of positive samples, and the similarity is calculated to reduce the number of negative samples, thereby reducing overfitting. We use a dataset of approximately 370[Formula: see text]000 samples, with approximately 20[Formula: see text]000 positive samples and approximately 350[Formula: see text]000 negative samples. The datasets are generated from images taken at the Beijing Garden Expo. To obtain a very robust method for the detection of traffic lights, we use different layers, different cost functions and different activation functions of the depth neural network for training and comparison. Our algorithm has evaluated autonomous vehicles in varying illumination and gets the result with high accuracy and robustness. The experimental results show that CTL-DNNet is effective at recognizing road traffic lights in the Beijing Garden Expo area.

مدل‌سازی برخی از خصوصیات شیمیایی روغن سیاه‌دانه تحت تاثیر پیش‌تیمار مایکروویو و سرعت دورانی پرس مارپیچی

Black Cumin seed (Nigella sativa L.) as one of the novel edible oil resources used commonly these days for seasoning in food product industries and with considerable medicinal properties and high nutritional impacts has been noticed. Oil extraction by pressing method as an approach compared to other methods including solvent extraction is more faster, safer and cheaper. In the oil extraction process, the preparation of the seeds is a substantial stage for obtaining oil with high quality and efficiency. Microwaves are electromagnetic waves that have a frequency ranged from 300 MHz to 300 GHz with corresponding wave lengths ranged from 1 mm to 1 m.On the other hand the artificial neural network as a powerful predictive tool ina wide scale of process parameters has been studied on an industrial scale in this research in order to achieve a simple, rapid, precise as well as effective model in the oil extraction of Nigella sativa L. Materials and methods: In the present study Black Cumin seed after preparation including cleaning and passing resistant test against air and moisture penetration have been shifted and preserved in a plastic bag until the experiments. Then, they have been pre-treated with microwave within different processing times (90, 180 and 270 S) and powers (180, 540, and 900 W). Afterwards, seeds’ oil have been extracted by screw rotational speed levels approach (11, 34 and 57 rpm), then different selected parameters including extraction efficiency, oil acidity value, colour and oxidative stability have been detected. To predict the alterations trend, the artificial neural network (ANN) design in MATLAB R2013a software has been used. Results and Discussion: According to MSE and R2 values presented in these tables, feed forward neural network with transfer function sigmoid hyperbolic tangent and Levenberg- Marquardt learning algorithm with topology of 3-10-5 (input layer with 3 neurons– a hidden layer with 10 neurons – output layer with 5 neurons) were selected as the optimal neural network with R2 more than 0.995 and MSE equal to 0.0005. Also, the results of the optimized and selected models were evaluated and these models with high correlation coefficients (over 0.949), were able to predict the changes' trend. According to the complexity and multiplicity of the effective factors in food industry processes and the results of this research, the neural network can be introduced as an acceptable model for modeling these processes. By determined the activation function in neural networks which was a function of sigmoid hyperbolic tangent in this study and also, with having the amounts of weight and bias, the connections created by the neuro-fuzzy model can be extracted. By defining this simple created mathematical equation, in a computer software such as Excel, we can have a useful, simple and accurate program for predicting the desired parameters in the process of oil extraction by using microwave pre-treatment. Due to high accuracy of neural model we can trust the prediction of these models with high confidence, and this model can be used to optimize and control the process, which can lead to saving in energy and time, and on the other hand, can create a better final product.

Deep Convolutional Neural Networks for Predominant Instrument Recognition in Polyphonic Music

Identifying musical instruments in polyphonic music recordings is a challenging but important problem in the field of music information retrieval. It enables music search by instrument, helps recognize musical genres, or can make music transcription easier and more accurate. In this paper, we present a convolutional neural network framework for predominant instrument recognition in real-world polyphonic music. We train our network from fixed-length music excerpts with a single-labeled predominant instrument and estimate an arbitrary number of predominant instruments from an audio signal with a variable length. To obtain the audio-excerpt-wise result, we aggregate multiple outputs from sliding windows over the test audio. In doing so, we investigated two different aggregation methods: one takes the class-wise average followed by normalization, and the other perform temporally local class-wise max-pooling on the output probability prior to averaging and normalization steps to minimize the effect of averaging process suppresses the activation of sporadically appearing instruments. In addition, we conducted extensive experiments on several important factors that affect the performance, including analysis window size, identification threshold, and activation functions for neural networks to find the optimal set of parameters. Our analysis on the instrument-wise performance found that the onset type is a critical factor for recall and precision of each instrument. Using a dataset of 10k audio excerpts from 11 instruments for evaluation, we found that convolutional neural networks are more robust than conventional methods that exploit spectral features and source separation with support vector machines. Experimental results showed that the proposed convolutional network architecture obtained an F1 measure of 0.619 for micro and 0.513 for macro, respectively, achieving 23.1% and 18.8% in performance improvement compared with the state-of-the-art algorithm.

Speech Processing for Text Independent Amharic Language Dialect Recognition

<span style="color: #666666; font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 11.2px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; display: inline !important; float: none;">Dialect is a difference of verbal communication spoken by people from a particular society or geographic area so the paper focuses on Amharic language dialect recognition. In this paper, the authors have used backpropagation artificial neural network, VQ(vector quantization), (Gaussian Mixture Models) and a combination of GMM and backpropagation artificial neural network for classifying dialects of Amharic language speakers. In this research, a total of 100 speakers for each group of dialects are considered each having about 10 seconds duration is collected. The feature vectors of Mel frequency cepstral coefficients (MFCC) had been used to recognize the dialects of speakers. In this research paper the recognition model that uses a tanh activation function have a better result instead of using the Logistic Sigmoid activation function in backpropagation artificial neural network. After conducting the above experiments 95.7% accuracy achieved when GMM and backpropagation artificial neural network with tanh activation function are combined.</span>

Multidimensional wavelet neural networks Based on polynomial powers of sigmoid

Wavelet functions have been used as the activation function in feed forward neural networks. An abundance of R&D has been produced on wavelet neural network area. Some successful algorithms and applications in wavelet neural network have been developed and reported in the literature. However, most of the aforementioned reports impose many restrictions in the classical back propagation algorithm, such as low dimensionality, tensor product of wavelets, parameters initialization, and, in general, the output is one dimensional, etc. In order to remove some of these restrictions, a family of polynomial wavelets generated from powers of sigmoid functions is presented. We described how a multidimensional wavelet neural networks based on these functions can be constructed, trained and applied in pattern recognition tasks. As examples of applications for the method proposed a framework for face verfication is presented.

New Algebraic Activation Function for Multi-Layered Feed Forward Neural Networks

ABSTRACTNumber of attempts have been made in the past to improve the generalization performance of artificial neural networks (ANN). Internal adjustment in ANN deals with tuning of various parameters like learning rate, activation function, etc. Majority of activation functions that exist in the literature are transcendental in nature. In this paper, a novel parametric algebraic activation (PAA) function has been proposed. The function PAA is a generalized function of which Elliott activation function is a special case. PAA is a family of S-shaped curves and satisfies all the important properties of activation functions. This activation is employed on resilient propagation algorithm (RPROP) learning algorithm. Comparative performance evaluation with the widely known activation functions has been carried out on various benchmark datasets taken from University of California Irvine (UCI) machine learning repository. Comparative performance evaluation in terms of number of epochs and testing error of the proposed PAA has been made with the standard activation functions. It has been observed that decrease in number of epochs and testing error for the proposed PAA in RPROP is highly statistically significant for most of the UCI datasets when compared with standard activation functions. Thus incorporating PAA in RPROP can make it more powerful for classification.

Modelling of mass transfer kinetic in osmotic dehydration of kiwifruit

Abstract Osmotic dehydration characteristics of kiwifruit were predicted by different activation functions of an artificial neural network. Osmotic solution concentration (y1), osmotic solution temperature (y2), and immersion time (y3) were considered as the input parameters and solid gain value (x1) and water loss value (x2) were selected as the outlet parameters of the network. The result showed that logarithm sigmoid activation function has greater performance than tangent hyperbolic activation function for the prediction of osmotic dehydration parameters of kiwifruit. The minimum mean relative error for the solid gain and water loss parameters with one hidden layer and 19 nods were 0.00574 and 0.0062% for logarithm sigmoid activation function, respectively, which introduced logarithm sigmoid function as a more appropriate tool in the prediction of the osmotic dehydration of kiwifruit slices. As a result, it is concluded that this network is capable in the prediction of solid gain and water loss parameters (responses) with the correlation coefficient values of 0.986 and 0.989, respectively.

A novel activation function for multilayer feed-forward neural networks

Traditional activation functions such as hyperbolic tangent and logistic sigmoid have seen frequent use historically in artificial neural networks. However, nowadays, in practice, they have fallen out of favor, undoubtedly due to the gap in performance observed in recognition and classification tasks when compared to their well-known counterparts such as rectified linear or maxout. In this paper, we introduce a simple, new type of activation function for multilayer feed-forward architectures. Unlike other approaches where new activation functions have been designed by discarding many of the mainstays of traditional activation function design, our proposed function relies on them and therefore shares most of the properties found in traditional activation functions. Nevertheless, our activation function differs from traditional activation functions on two major points: its asymptote and global extremum. Defining a function which enjoys the property of having a global maximum and minimum, turned out to be critical during our design-process since we believe it is one of the main reasons behind the gap observed in performance between traditional activation functions and their recently introduced counterparts. We evaluate the effectiveness of the proposed activation function on four commonly used datasets, namely, MNIST, CIFAR-10, CIFAR-100, and the Pang and Lee's movie review. Experimental results demonstrate that the proposed function can effectively be applied across various datasets where our accuracy, given the same network topology, is competitive with the state-of-the-art. In particular, the proposed activation function outperforms the state-of-the-art methods on the MNIST dataset.

Speeding up Deep Neural Networks in Speech Recognition with Piecewise Quantized Sigmoidal Activation Function

This paper proposes a new quantization framework on activation function of deep neural networks (DNN). We implement fixed-point DNN by quantizing the activations into powers-of-two integers. The costly multiplication operations in using DNN can be replaced with low-cost bit-shifts to massively save computations. Thus, applying DNN-based speech recognition on embedded systems becomes much easier. Experiments show that the proposed method leads to no performance degradation.

Navigation of autonomous mobile robot using different activation functions of wavelet neural network

Abstract An autonomous mobile robot is a robot which can move and act autonomously without the help of human assistance. Navigation problem of mobile robot in unknown environment is an interesting research area. This is a problem of deducing a path for the robot from its initial position to a given goal position without collision with the obstacles. Different methods such as fuzzy logic, neural networks etc. are used to find collision free path for mobile robot. This paper examines behavior of path planning of mobile robot using three activation functions of wavelet neural network i.e. Mexican Hat, Gaussian and Morlet wavelet functions by MATLAB. The simulation result shows that WNN has faster learning speed with respect to traditional artificial neural network.

Deep Neural Networks with Multistate Activation Functions.

We propose multistate activation functions (MSAFs) for deep neural networks (DNNs). These MSAFs are new kinds of activation functions which are capable of representing more than two states, including the N-order MSAFs and the symmetrical MSAF. DNNs with these MSAFs can be trained via conventional Stochastic Gradient Descent (SGD) as well as mean-normalised SGD. We also discuss how these MSAFs perform when used to resolve classification problems. Experimental results on the TIMIT corpus reveal that, on speech recognition tasks, DNNs with MSAFs perform better than the conventional DNNs, getting a relative improvement of 5.60% on phoneme error rates. Further experiments also reveal that mean-normalised SGD facilitates the training processes of DNNs with MSAFs, especially when being with large training sets. The models can also be directly trained without pretraining when the training set is sufficiently large, which results in a considerable relative improvement of 5.82% on word error rates.

Optical Computer Mouse Referenced Calibration of an Inertial Measurement Unit for Use in Unmanned Underwater Vehicles

In this study an inertial measurement unit (IMU) used in unmanned underwater vehicles has been taken into consideration.The main objective of this study is to improvethe measurements obtained from an IMU used in the position detection by minimizing the effect of its static and dynamic errors on the output. To enhance the IMU data optical computer mouse (OCM) is proposed as calibrator. The data received from the OCM is used to train an artificial neural network (ANN) which would improve the IMU outputs by trying to estimate the reference data from the actual sensor outputs. The ANN performance is compared with that of classic low pass filtering methods to provide a relative performance criterion. The ANN trained with OCM data has given satisfactory results. During the training of ANNs the effects of several parameters such as neural network architecture, activation functions, training algorithm, layer and cell number have been investigated. Thus, the results, findings and insights obtained in this study can be applied in research areas where this kind of nonlinear estimators are used.

Emulation of Spline Networks Through Approximation of Polynomials and Step Function of Neural Networks with Cosine Modulated Symmetric Exponential Function

This study proves that the neural network, using the cosine modulated symmetric exponential function which is a non-monotonic function, can emulate the spline networks by approximating the polynomials and step functions. This means that the network with cosine modulated symmetric exponential function is equivalent to the spline networks. It is also equivalent to the neural network, which uses sigmoidal, hyperbolic tangent, and Gaussian activation function, as proved by DasGupta and Schnitger. In the multi-network structure, the cosine modulated symmetric exponential function has the capability to make more local hills than any other functions. In the neural network that uses this function, it has the capability to quickly localize the input space pattern, even though it makes a fewer number of layers. On the other hand, the monotonic function needs a greater number of layers to make these local hills. Therefore, in the training for the pattern classification of the neural network, we need a greater number of units and epochs. This is connected to the training speed of the neural network for the pattern classification, which also indicates the capabilities of the network. For the capacity test of the pattern classification in the cosine modulated symmetric exponential function, we have used the Cascade-Correlation neural network. Cascade-Correlation is a supervised learning algorithm that automatically determines the size and topology of the network. The Cascade-Correlation adds new hidden units one by one and creates a multi-layer structure in which each unit is in a hidden layer. In this experiment, the two benchmark problems have been used: one is the iris plant classification problem; the other is the tic-tac-toe endgame problem. The results are compared with those obtained with other activation functions. In this experiment, the evaluation items, such as the number of epochs, produced hidden units, listing of the run time, and the average crossings per second of ten trials on the training set of the problem, have been compared. For instance, In the experiment of the iris plants classification problem, the CosExp function has recorded about 53 % of the average epochs number when compared to the sigmoid function, and for the number of the hidden units, it is approximately 54 %. In the tic-tac-toe problem experiment, the average number of epochs and hidden units produced during the process has been reduced approximately by one thirds more with the CosExp function than with other activation functions. Accordingly, learning has been improved three times faster. The results of the experiments show that performance can be improved very significantly by using the cosine modulated symmetric exponential function as the activation function in neural networks with a predetermined set of parameters.

Research on Intelligent Optimization Control Method for Oil Pumping

In the oil production process, some oil pumps exist in the light load running and empty pumping problems, which result in the waste of electric energy. In order to realize the oil pumping energy saving optimal control by adopting oil pumping start-stop intermittent control scheme, an intelligent energy saving optimal control system based on genetic algorithm and wavelet neural network is proposed in this paper. The Morlet wavelet is adopted as the activation function of neural network and builds the wavelet neural networkandthe structure and parameters of the wavelet neural network which are chromosome encoded, and the genetic algorithm is used to optimize the connection weights and the scale parameters in order to improve the generalization ability and the approximation ability of neural networks. The parameters of effective oil pumping energy saving optimal control are measured using multisensors. The system is used in the oil production plant; the test data show that the oil pumping energy conservation effect is obvious.

Stability analysis and output feedback control for a class of switched nonlinear systems

The problem of stability analysis and output feedback control for a class of switched nonlinear systems is addressed in this paper. The subsystems are from the class of so-called Φ-systems (Φ-systems). The standard saturation and the hyperbolic tangent (popular activation function in neural networks) are examples of this type of nonlinearity. The discrete-time recurrent artificial neural network is a special case of Φ-systems. Furthermore, results related to this class of nonlinear systems have potential applications in the classical problems related to uncertain nonlinearities such as Lure systems. Two linear matrix inequality-based (LMI) sufficient conditions for asymptotic stability are proposed for these switched nonlinear systems. These conditions are analogous counterparts for switched linear systems which are shown to be easily verifiable and suitable for design problems. Two conditions for output feedback controller synthesis based on the proposed stability conditions are presented. The results are further illustrated by numerical examples.

Application of Neural Networks in Computer Security

This contribution is focused on the insurance of control system data communication via neural network technologies in connection with classical methods used in expert systems. The solution proposed defines a way of data element identification in transfer networks, solves the transformation of their parameters for neural network input and defines the type and architecture of a suitable neural network. This is supported by experiments with various architecture types and neural network activation functions and followed by subsequent real environment tests. A functional system proposal with possible practical application is the result.

An Open Flow Controller Based on the Intercerebral Neural Network for Media Independent Handover

This paper presents a novel open flow controller based on the intercerebral neural network for the media independent handover over wireless operation. This controller makes a binary decision based on the link control parameters obtained instantaneously from the trained neural network, which in turn depends on the instantaneous mobile speed interaction with the wireless link performance parameters. These parameters of packet loss, round trip time, radio signal strength, bandwidth, data cost and vehicle speed are typically different from network to network. We use a mutated particle swarm optimization to train the fundamental controller equation for the media independent handover that are mixed with the radial and sigmoid activation functions of the intercerebral neural network. Simulations based on the field experiment test data show that the system using the swarm intelligence algorithm is practical for the improvement of the overall secure networking experience for the fast delivery.

Global Exponential Stability Analysis of Dynamic Neural Networks with Distributed Delays

In this paper, the existence, uniqueness and globally exponential stability of the equilibrium point of a dynamic neural network with distributed delays were studied without assumption of boundedness and differentiability of activation functions. Sufficient criteria for existence, uniqueness and global exponential stability of the equilibrium point of such neural networks were obtained based on the knowledge of M-matrix, topology and Lyapunov stability theory. A test matrix was constructed by the weight matrix and the conditions satisfying activation functions of the neural networks. A neural network has a unique equilibrium point and is globally exponential stable if the test matrix is an M-matrix. Since the criterion is independent of the delays and simplifies the calculation, it is easy to test the conditions of the criterion in practice.

Combination of supervised and unsupervised learning for training the activation functions of neural networks

Standard feedforward neural networks benefit from the nice theoretical properties of mixtures of sigmoid activation functions, but they may fail in several practical learning tasks. These tasks would be better faced by relying on a more appropriate, problem-specific basis of activation functions. The paper presents a connectionist model which exploits adaptive activation functions. Each hidden unit in the network is associated with a specific pair (f(·),p(·)), where f(·) is the activation function and p(·) is the likelihood of the unit being relevant to the computation of the network output over the current input. The function f(·) is optimized in a supervised manner, while p(·) is realized via a statistical parametric model learned through unsupervised (or, partially supervised) estimation. Since f(·) and p(·) influence each other’s learning process, the overall machine is implicitly a co-trained coupled model and, in turn, a flexible, non-standard neural architecture. Feasibility of the approach is corroborated by empirical evidence yielded by computer simulations involving regression and classification tasks.