Kinds Of Activation Functions Research Articles

Application of Weighted Combinations of Activation Functions to Defect Prediction in Software Development

Software defect prediction (SDP) method aims to identify potential bugs in programs or defective modules in software projects. SDP method can greatly help developers allocate needed testing- and debugging-efforts. Presently, SDP is typically divided into two procedures: extracting features from source code and building a classification model using machine learning methods, such as support vector machine, decision tree, and neural networks, to build a classifier for defect prediction. However, there are still some limitations for SDP. For example, machine learning models require a fixed input size, but the size of each program is mostly inconsistent. Activation functions play an essential role in the training of artificial neural networks, but every kind of activation function has its own particular strengths and inherent constraints. The main purpose of this article is to propose a general framework of combining different activation functions with given weights to improve the effectiveness of SDP. We construct 41 kinds of defect prediction models by deep belief networks (DBNs) built with double-weighted or triple-weighted combination of six most commonly used activation functions to improve the predictions. In the experiment of this study, some real data from open-source projects are selected to evaluate the performance of our proposed weighted combination methods. It is found experimentally that the weighted combinations methods can enhance the accuracy of SDP. It is also noticed that our proposed weighted combination scheme is not restricted to the DBN or a particular kind of activation functions.

Data-driven soliton solutions and parameters discovery of the coupled nonlinear wave equations via a deep learning method

In this paper, we extend the physics-informed neural networks (PINNs) to learn data-driven solutions and discover the coefficients of multi-component and high-dimensional coupled nonlinear partial differential equations (cNPDEs). In the forward problems, the data-driven solutions include “bell”-type (dark and bright) and “anti-kink”-type solitons of the generalized Hirota–Satsuma coupled KdV equation are investigated and reconstructed. Moreover, some special shape one-soliton and two-soliton solutions such as “Kink”-shaped, “U”-shaped, “Y”-shaped, and “X”-shaped of a (2+1)-dimensional cNPDE are learned and simulated as well. Particularly, some factors affecting the performance of PINNs are investigated, such as the structure of neural networks, the kinds of activation functions and optimizers, as well as the number of collocation points and iterations. For the inverse problems, the coefficients of the equations are successfully identified and driven by various kinds of solution data. The impact of the number and moment of observations on the model is discussed while the network’s robustness is validated by introducing mensurable initial noises to the training data. Numerical experiments not only show that the propagation direction and dynamical behaviors of the cNPDE can be well uncovered by utilizing the deep learning method, but also exhibit the advantage of finding equations with multiple parameters.

Multistability analysis of delayed recurrent neural networks with a class of piecewise nonlinear activation functions

This paper studies the multistability of delayed recurrent neural networks (DRNNs) with a class of piecewise nonlinear activation functions. The coexistence as well as the stability of multiple equilibrium points (EPs) of DRNNs are proved. With the Brouwer’s fixed point theorem as well as the Lagrange mean value theorem, it is obtained that under some conditions, the n-neuron DRNNs with the proposed activation function can have at least 5n EPs and 3n of them are locally stable. Compared with the DRNNs with sigmoidal activation functions, DRNNs with this kind of activation function can have more total EPs and more locally stable EPs. It implies that when designing DRNNs with the proposed activation function to apply in associative memory, it can have an even larger storage capacity. Furthermore, it is obtained that there exists a relationship between the number of the total EPs/stable EPs and the frequency of the sinusoidal function in the proposed activation function. Last, the above obtained results are extended to a more general case. It is shown that, DRNNs with the extended activation function can have (2k+1)n EPs, (k+1)n of which are locally stable, therein k is closely related to the frequency of the sinusoidal function in the extended activation function. Two simulation examples are given to verify the correctness of the theoretical results.

Multistability analysis of state-dependent switched Hopfield neural networks with the Gaussian-wavelet-type activation function

In multistability analysis, the Gaussian-wavelet-type activation function is shown to have better properties by comparison with sigmoidal functions, saturated functions and Mexican-hat-type functions. State-dependent switched Hopfield neural network (SSHNN) is expected to display even richer dynamical behaviors in contrast with conventional Hopfield neural networks (HNNs). Considering these two reasons, this paper studies the multistability of SSHNNs with the Gaussian-wavelet-type activation function. Some sufficient conditions for the coexistence as well as the stability of multiple equilibria of SSHNNs are derived. It is obtained that SSHNNs with the Gaussian-wavelet-type activation function can have at least 7n or 6n equilibria, of which 4n or 5n are locally stable (LS). We find that, compared with conventional HNNs with the Gaussian-wavelet-type activation function or SSHNNs with other kinds of activation functions, SSHNNs with the Gaussian-wavelet-type activation functions can have more LS equilibria. It implies that SSHNNs with the Gaussian-wavelet-type activation functions have even larger storage capacity and have overwhelming superiority in associative memory applications. Lastly, some simulation results are given to verify the correctness of the theoretical results.

Multistability of Hopfield neural networks with a designed discontinuous sawtooth-type activation function

In this paper, a class of discontinuous sawtooth-type activation function is designed and the multistability of Hopfield neural networks (HNNs) with such kind of activation function is studied. By virtue of the Brouwer’s fixed point theorem and the property of strictly diagonally dominant matrix (SDDM), some sufficient conditions are presented to ensure that the n-neuron HNN can have at least 7n equilibria, among which 4n equilibria are locally exponentially stable and the remaining 7n-4n equilibria are unstable. Then, the obtained results are extended to a more general case. We continue to increase the number of the peaks of the sawtooth-type activation function and we find that the n-neuron HNN can have (2k+3)n equilibria, (k+2)n of them are locally exponentially stable and the remaining equilibria are unstable. Therein, k denotes the total number of the peaks in the designed activation function. That is to say, there is a quantitative relationship between the number of the peaks and the number of the equilibria. It implies that one can improve the storage capacity of a HNN by increasing the number of the peaks of the activation function in theory and in practice. To some extent, this method is convenient and flexible. Compared with the existing results, HNN with the designed sawtooth-type activation function can have more total equilibria as well as more locally stable equilibria. Finally, two examples are presented to demonstrate the validity of the obtained results.

Signal processing algorithm for neural networks with integrodifferential splines as an activation function and its particular case of image classification

Classic neural networks suppose trainable parameters to include just weights of neurons. This paper proposes parabolic integrodifferential splines (ID-splines), developed by author, as a new kind of activation function (AF) for neural networks, where ID-splines coefficients are also trainable parameters. Parameters of ID-spline AF together with weights of neurons are vary during the training in order to minimize the loss function thus reducing the training time and increasing the operation speed of the neural network. The newly developed algorithm enables software implementation of the ID-spline AF as a tool for neural networks construction, training and operation. It is proposed to use the same ID-spline AF for neurons in the same layer, but different for different layers. In this case, the parameters of the ID-spline AF for a particular layer change during the training process independently of the activation functions (AFs) of other network layers. In order to comply with the continuity condition for the derivative of the parabolic ID-spline on the interval (x x0, n) , its parameters fi (i= 0,...,n) should be calculated using the tridiagonal system of linear algebraic equations: To solve the system it is necessary to use two more equations arising from the boundary conditions for specific problems. For exam- ple the values of the grid function (if they are known) in the points (x x0, n) may be used for solving the system above: f f x0 = ( 0) , f f xn = ( n) . The parameters Iii+1 (i= 0,...,n−1 ) are used as trainable parameters of neural networks. The grid boundaries and spacing of the nodes of ID-spline AF are best chosen experimentally. The optimal selection of grid nodes allows improving the quality of results produced by the neural network. The formula for a parabolic ID-spline is such that the complexity of the calculations does not depend on whether the grid of nodes is uniform or non-uniform. An experimental comparison of the results of image classification from the popular FashionMNIST dataset by convolutional neural 0, x< 0 networks with the ID-spline AFs and the well-known ReLUx( ) =AF was carried out. The results reveal that the usage x x, ≥ 0 of the ID-spline AFs provides better accuracy of neural network operation than the ReLU AF. The training time for two convolutional layers network with two ID-spline AFs is just about 2 times longer than with two instances of ReLU AF. Doubling of the training time due to complexity of the ID-spline formula is the acceptable price for significantly better accuracy of the network. Wherein the difference of an operation speed of the networks with ID-spline and ReLU AFs will be negligible. The use of trainable ID-spline AFs makes it possible to simplify the architecture of neural networks without losing their efficiency. The modification of the well-known neural networks (ResNet etc.) by replacing traditional AFs with ID-spline AFs is a promising approach to increase the neural network operation accuracy. In a majority of cases, such a substitution does not require to train the network from scratch because it allows to use pre-trained on large datasets neuron weights supplied by standard software libraries for neural network construction thus substantially shortening training time.

Novel methods to finite-time Mittag-Leffler synchronization problem of fractional-order quaternion-valued neural networks

This paper proposes two methods to investigate the problem of finite-time Mittag-Leffler synchronization for the systems of fractional-order quaternion-valued neural networks (FQVNNs) with two kinds of activation functions, respectively. Generally, the first method mainly reflects in the new establishment of Lyapunov-Krasovskii functionals (LKFs) and the novel application of a new fractional-order derivative inequality which contains and exploits the wider coefficients with more values. Meanwhile, the second one is embodied in the comprehensive development of both the norm comparison rules and the generalized Gronwall-Bellman inequality with the help of Laplace transform of Mittag-Leffler function. Thanks to the above two methods, the flexible synchronization criteria are easily and separately obtained for the studied four systems of FQVNNs with general activation functions and linear threshold ones. Finally, two numerical simulations are given to demonstrate the feasibility and effectiveness of the newly proposed approaches.

Monostability and Multistability for Almost-Periodic Solutions of Fractional-Order Neural Networks With Unsaturating Piecewise Linear Activation Functions

Since the unsaturating activation function is unbounded, more complex dynamics may exist in neural networks with this kind of activation function. In this article, monostability and multistability results of almost-periodic solutions are developed for fractional-order neural networks with unsaturating piecewise linear activation functions. Some globally Mittag-Leffler attractive sets are given, and the existence of globally Mittag-Leffler stable almost-periodic solution is demonstrated by using Ascoli-Arzela theorem. In particular, some sufficient conditions are provided to ascertain the multistability of almost-periodic solutions based on locally positively invariant set. It shows that there exists an almost-periodic solution in each positively invariant set, and all trajectories converge to this periodic trajectory in that rectangular area. Two illustrative examples are provided to demonstrate the effectiveness of the proposed sufficient criteria.

Global Mittag-Leffler stability analysis of impulsive fractional-order complex-valued BAM neural networks with time varying delays

This paper investigates the stability of impulsive fractional-order complex-valued BAM neural networks with time varying delays. As the extension of fractional-order real-valued BAM neural networks, fractional-order complex-valued BAM neural networks have complex-valued states, synaptic weights, and the activation functions. Two different kinds of activation functions are considered, along with popular bounded and Lipschitz-kind activation functions. By using Lyapunov function and Homomorphic mapping theorem, sufficient conditions for the existence of unique equilibrium and global asymptotic stability of complex-valued systems are derived. In derivation we separated nonlinear complex-valued activation functions into real and imaginary parts. Moreover, Mittag-Leffler stability for BAM neural networks(BAMNNs) have been proposed when the nonlinear complex activation functions are bounded. Simulation results are presented to prove the efficiency of the obtained methods.

Lagrange stability of delayed switched inertial neural networks

As an extension of single/multiple stability notion, Lagrange stability is considered here. A class of switched inertial neural networks (SINNs) is studied on both continuous-time and discrete-time domain. Two kinds of activation functions are evolved for the network. Through characteristic function approach, matrix measure strategy, and theory of time scales, Lagrange stability of delayed continuous-/discrete-time SINNs with lurie-type and bounded-type activation functions are addressed, respectively. The results also show that the criteria corresponding to discrete-time network approach to those corresponding to continuous-time one as the graininess function tends to zero. Two examples are given to show the effectiveness of the main results.

Convergence and Robustness Analysis of the Exponential-Type Varying Gain Recurrent Neural Network for Solving Matrix-Type Linear Time-Varying Equation

To solve matrix-type linear time-varying equation more efficiently, a novel exponential-type varying gain recurrent neural network (EVG-RNN) is proposed in this paper. Being distinguished from the traditional fixed-parameter gain recurrent neural network (FG-RNN), the proposed EVG-RNN is derived from a vector- or matrix-based unbounded error function by a varying-parameter neural dynamic approach. With four different kinds of activation functions, the super-exponential convergence performance of EVG-RNN is proved theoretically in details, of which the error convergence rate is much faster than that of FG-RNN. In addition, mathematics proves that the computation errors of EVG-RNN can converge to zero, and it possesses the capability of restraining external interference. Finally, series of computer simulations verify and illustrate the better performance of convergence and robustness of EVG-RNN than that of FG-RNN and FTZNN when solving the identical linear time-varying equation.

Convolutional neural network with nonlinear competitive units

Convolutional Neural Network (CNN) has been an important breakthrough in pattern recognition in recent years. Nevertheless, with the increase in complexity, CNN becomes more difficult to train. To alleviate the problem of training difficulties, we propose a novel nonlinear unit, called Nonlinear Competitive Unit (NCU). By comparing the elements from different network layers and selecting the larger signals element-wisely, it can not only strengthen feature propagation but also accelerate the convergence of CNN. This unit can be regarded as a feature fusion method as well as a kind of activation function. We evaluate our NCU-based models for face verification task and visual classification task on four benchmark datasets. The experimental results demonstrate the superior performance of our models over many state-of-the-art methods, which shows the advantage and potential of the NCU in networks.

Complex-valued neural network topology and learning applied for identification and control of nonlinear systems

In this paper we present a Complex-Valued Recurrent Neural Network (CVRNN), trained with a recursive Levenberg-Marquardt (LM) learning algorithm in the complex domain, applying it to the problem of dynamic system identification, and to an adaptive neural control scheme of a nonlinear oscillatory plant. This methodology is applied to two different CVRNN topologies with different kinds of activation functions. Furthermore, we applied the CVRNN identification and control for a particular case of a nonlinear, oscillatory mechanical plant to validate the performance of the adaptive neural controller using the LM algorithm developed throughout this work, compared to a complex-valued Backpropagation learning algorithm. The obtained comparative simulation results using a flexible robot arm gives a good performance of the derived control schemes. The results show some priority of the recursive LM learning over the BP learning, and the use of constructed activation functions in the neural network topology.

Застосування інтелектуальної діагностичної інфраструктури для управління кібербезпекою. Частина 2. Підтримка життєвого циклу системи кіберзахисту

How use of structural features in the construction of hybrid models allows supporting of models adjustment and adaptation to problem-subject environment was considered in the article. The following features were attributed to structural ones: the type of learning algorithm; kind of activation function; the number of layers of the neural network; type of neurons; way of spreading information in neural networks; method of evaluating and interpreting the results of the neural network; the format of fuzzy inference rules; fuzzification and defuzzification method; way to implement the operations of fuzzy implication and logical operations NOT, AND, OR; kind of used genetic operators and the target functions, etc. We propose to use a neural network approach as a basis for the decision of difficulty tasks using decision-support systems. Its effectiveness can be enhanced by: prior training or adjustment of individual neural modules for solvable problem; incorporation of knowledge about the peculiarities of the domain in the hierarchical (multilayer) neural networks structure; application of basic types of hybrid models in which neural network communicates with other information technologies.

Застосування інтелектуальної діагностичної інфраструктури для управління кібербезпекою

How use of structural features in the construction of hybrid models allows supporting of models adjustment and adaptation to problem-subject environment was considered in the article. The following features were attributed to structural ones: the type of learning algorithm; kind of activation function; the number of layers of the neural network; type of neurons; way of spreading information in neural networks; method of evaluating and interpreting the results of the neural network; the format of fuzzy inference rules; fuzzification and defuzzification method; way to implement the operations of fuzzy implication and logical operations NOT, AND, OR; kind of used genetic operators and the target functions, etc. We propose to use a neural network approach as a basis for the decision of difficulty tasks using decision-support systems. Its effectiveness can be enhanced by: prior training or adjustment of individual neural modules for solvable problem; incorporation of knowledge about the peculiarities of the domain in the hierarchical (multilayer) neural networks structure; application of basic types of hybrid models in which neural network communicates with other information technologies.

Modeling Hysteretic Deteriorating Behavior Using Generalized Prandtl Neural Network

In this paper, a new kind of activation function using a particular combination of stop and play operators is proposed and used in a feedforward neural network to improve its learning capability in the identification of nonlinear hysteretic material behavior with both stiffness and strength degradation. The new neuron and neural network are referred to as a deteriorating stop and generalized Prandtl neural network, respectively. To show the generality of the proposed neural network, it is trained on several data sets generated by various mathematical models of material hysteresis with and without deterioration as well as on a set of experimental data with very high nonlinearity, including severe damage. In each case, the training is successful, and the generalized Prandtl neural network response precision is very high. Also, using the proposed neural network, a neuro-modeler is designed and used in the dynamic analysis of a one-story shear frame under seismic loads with severe damage. A comparison of the results shows that the generalized Prandtl neural network type of the neuro-modeler is more successful than the previously proposed Prandtl neural network type.

Deep Neural Networks with Multistate Activation Functions.

We propose multistate activation functions (MSAFs) for deep neural networks (DNNs). These MSAFs are new kinds of activation functions which are capable of representing more than two states, including the N-order MSAFs and the symmetrical MSAF. DNNs with these MSAFs can be trained via conventional Stochastic Gradient Descent (SGD) as well as mean-normalised SGD. We also discuss how these MSAFs perform when used to resolve classification problems. Experimental results on the TIMIT corpus reveal that, on speech recognition tasks, DNNs with MSAFs perform better than the conventional DNNs, getting a relative improvement of 5.60% on phoneme error rates. Further experiments also reveal that mean-normalised SGD facilitates the training processes of DNNs with MSAFs, especially when being with large training sets. The models can also be directly trained without pretraining when the training set is sufficiently large, which results in a considerable relative improvement of 5.82% on word error rates.

An oscillation bound of the generalization performance of extreme learning machine and corresponding analysis

Extreme Learning Machine (ELM), proposed by Huang et al. in 2004 for the first time, performs better than traditional learning machines such as BP networks and SVM in some applications. This paper attempts to give an oscillation bound of the generalization performance of ELM and a reason why ELM is not sensitive to the number of hidden nodes, which are essential open problems proposed by Huang et al. in 2011. The derivation of the bound is in the framework of statistical learning theory and under the assumption that the expectation of the ELM kernel exists. It turns out that our bound is consistent with the experimental results about ELM obtained before and predicts that overfitting can be avoided even when the number of hidden nodes approaches infinity. The prediction is confirmed by our experiments on 15 data sets using one kind of activation function with every parameter independently drawn from the same Guasssian distribution, which satisfies the assumption above. The experiments also showed that when the number of hidden nodes approaches infinity, the ELM kernel with the activation is insensitive to the kernel parameter.

SOFT COMPUTING SINGLE HIDDEN LAYER MODELS FOR SHELF LIFE PREDICTION OF BURFI

ABSTRACT Burfi is an extremely popular sweetmeat, which is prepared by desiccating the standardized water buf-falo milk. Soft computing feedforward single layer models were developed for predicting the shelf life of burfi stored at 30oC. The data of the product relating to moisture, titratable acidity, free fatty acids, tyrosine, and peroxide value were used as input variables, and the overall acceptability score as out-put variable. The results showed excellent agreement between the experimental and the predicted da-ta, suggesting that the developed soft computing model can alternatively be used for predicting the shelf life of burfi. KEY WORDS Keeping quality; Forecasting; Instant foods; Layering; Milk; Instantizing; Amino acids; Desserts; Fatty acids. Artificial Neural Network (ANN) models are mathematical and algorithmic software mod-els inspired by biological neural network. An ANN model is interconnected group of nodes, parallel to the vast network of neurons in the hu-man brain. It consists of interconnected group of artificial neurons and processes information using a connectionist approach to computation. In most cases, ANN model is an adaptive system that changes its structure based on external or internal information that flows through the network dur-ing the learning phase. ANN models are non-linear statistical data modelling tools. They can be used to model complex relationships between inputs and outputs or to find patterns inherent in the data. In other words, the application of ANN models is a method of data analysis that is de-signed to imitate the workings of the human brain. They emulate the way in which arrays of neurons most likely function in biological learn-ing and memory. ANN models differ from clas-sical computer programs in that they ‘‘learn’’ or are ‘‘taught’’ from a set of examples rather than simply being programmed to give a correct an-swer. Information is encoded in the strength of the network’s ‘‘synaptic’’ connections. It has been established that ANN is fully equipped to predict the shelf stability and safety of food products in general, and dairy products in par-ticular, as ANN model has the ability to learn from examples and relearn when new data are utilized (Vallejo-Cordoba et al.,1995). Single layer perceptron network consists of a single layer of output nodes; the inputs are fed directly to the outputs via a series of weights. In this way it can be considered the simplest kind of feed-forward network. The sum of the products of the weights and the inputs is calculated in each node, and if the value is above some threshold (typically 0) the neuron fires and takes the acti-vated value (typically 1); otherwise it takes the deactivated value (typically -1). Neurons with this kind of activation function are also called artificial neurons or linear threshold units. In the literature the term perceptron often refers to net-works consisting of just one of these units. A similar neuron was described by Warren McCul-loch and Walter Pitts in the 1940s (Wikipedia ANN Website, 2011). In Indian subcontinent burfi is extremely popular milk based sweetmeat, which is prepared by desiccating standardized water buffalo milk. Its importance can be gauged from the fact that no festival, get-together, marriage or birthday party is considered complete unless it is served. Several varieties of burfi such as coconut burfi, chocolate burfi, cashew nut burfi, almond burfi, pistachio burfi, cardamom burfi and plain burfi

An LMI Approach to Stability Analysis of Reaction–Diffusion Cohen–Grossberg Neural Networks Concerning Dirichlet Boundary Conditions and Distributed Delays

The global asymptotic stability problem for a class of reaction-diffusion Cohen-Grossberg neural networks with both time-varying delay and infinitely distributed delay is investigated under Dirichlet boundary conditions. Instead of using the M-matrix method and the algebraic inequality method, under some suitable assumptions and using a matrix decomposition method, we adopt the linear matrix inequality method to propose two sufficient stability conditions for the concerned neural networks with Dirichlet boundary conditions and different kinds of activation functions, respectively. The obtained results are easy to check and improve upon the existing stability results. Two examples are given to demonstrate the effectiveness of the obtained results.