Monotonous Activation Function Research Articles

Robustness Verification of Swish Neural Networks Embedded in Autonomous Driving Systems

With the applications of deep learning in safety-critical domains such as autonomous driving systems gaining ground, it demands rigorous verification to guarantee the safety and reliability of corresponding systems. As the intelligent component in such systems, neural networks (NNs) must be robust in that their outputs are not affected by minor perturbation to inputs. Many research studies have shown that formal methods are effective ways to the robustness verification of NNs. However, most of the existing approaches are focused on NNs that contain monotonic activation functions, such as ReLU, Tanh, and Sigmoid. In this work, we propose an approach to verify the robustness of NNs with the nonmonotonic activation function called Swish. Such networks have been proved to have a better performance on image classification than other NNs. In our approach, we turn the robustness verification problem into a constraint-solving problem using the linear approximation technique. We first model the affine function of an NN into a linear constraint model. Then, for nonlinear activation functions, we leverage an efficient approximation strategy to linearly approximate them. Finally, we utilize the constraint solver gurobi to solve the model, which reveals that the model satisfies the robustness property. We develop a prototype tool and evaluate it with open-sourced NNs. Experimental results showed the effectiveness and efficiency of our approach.

HcLSH: A Novel Non-Linear Monotonic Activation Function for Deep Learning Methods

HcLSH: A Novel Non-Linear Monotonic Activation Function for Deep Learning Methods

Methods and Means for Small Dynamic Objects Recognition and Tracking

A literature analysis has shown that object search, recognition, and tracking systems are becoming increasingly popular. However, such systems do not achieve high practical results in analyzing small moving living objects ranging from 8 to 14 mm. This article examines methods and tools for recognizing and tracking the class of small moving objects, such as ants. To fulfill those aims, a customized You Only Look Once Ants Recognition (YOLO_AR) Convolutional Neural Network (CNN) has been trained to recognize Messor Structor ants in the laboratory using the LabelImg object marker tool. The proposed model is an extension of the You Only Look Once v4 (Yolov4) 512 × 512 model with an additional Self Regularized Non–Monotonic (Mish) activation function. Additionally, the scalable solution for continuous object recognizing and tracking was implemented. This solution is based on the OpenDatacam system, with extended Object Tracking modules that allow for tracking and counting objects that have crossed the custom boundary line. During the study, the methods of the alignment algorithm for finding the trajectory of moving objects were modified. I discovered that the Hungarian algorithm showed better results in tracking small objects than the K–D dimensional tree (k-d tree) matching algorithm used in OpenDataCam. Remarkably, such an algorithm showed better results with the implemented YOLO_AR model due to the lack of False Positives (FP). Therefore, I provided a new tracker module with a Hungarian matching algorithm verified on the Multiple Object Tracking (MOT) benchmark. Furthermore, additional customization parameters for object recognition and tracking results parsing and filtering were added, like boundary angle threshold (BAT) and past frames trajectory prediction (PFTP). Experimental tests confirmed the results of the study on a mobile device. During the experiment, parameters such as the quality of recognition and tracking of moving objects, the PFTP and BAT, and the configuration parameters of the neural network and boundary line model were analyzed. The results showed an increased tracking accuracy with the proposed methods by 50%. The study results confirmed the relevance of the topic and the effectiveness of the implemented methods and tools.

Multiple bifurcations and periodic coexistence in a delayed Hopfield two-neural system with a monotonic activation function

In this paper, we consider a delayed Hopfield two-neural system with a monotonic activation function and find the periodic coexistence by bifurcation analysis. Firstly, we obtain the pitchfork bifurcation of the trivial equilibrium employing the central manifold and normal form methods. The neural system exhibits two pitchfork bifurcations near the trivial equilibrium. Then, analyzing the characteristic equation of the nontrivial equilibrium, we illustrate the saddle-node bifurcation of the nontrivial equilibria. The system exhibits the multi-coexistences of the stable and unstable equilibria. Further, we illustrate the plane regions of parameters having different numbers of equilibria. To obtain a time delay in neural system dynamics, we present the stability analysis and find the periodic orbit. The system exhibits stability switching by the Hopf bifurcation curves. Finally, the dynamic behaviors near the Hopf–Hopf bifurcation point are presented. The system exhibits coexistence of multiple periodic orbits with different frequencies.

Multistability of Fractional-Order Recurrent Neural Networks With Discontinuous and Nonmonotonic Activation Functions

The coexistence of multiple stable equilibria in recurrent neural networks is an important dynamic characteristic for associative memory and other applications. In this paper, the existence and local Mittag-Leffler stability of multiple equilibria are investigated for a class of fractional-order recurrent neural networks with discontinuous and nonmonotonic activation functions. By using Brouwer's fixed point theory, several conditions are established to ensure the existence of 5 n equilibria, in which all the coemponents of 4 n equilibria are located in the continuous intervals of the activation functions. and some of the components of 5 n - 4 n equilibria are located at some discontinuous points of the activation functions. The introduction of discontinuous activation functions makes the neural networks have more equilibria than those with continuous activation functions. Furthermore, some criteria are proposed to ensure local Mittag-Leffler stability of 3 n equilibria. The introduction of nonmonotonic activation functions makes the neural networks have more stable equilibria than those with monotonic activation functions. Two examples are given to illustrate the effectiveness of the results.

Frequency conversion sinusoidal chaotic neural network and its application

The optimization performance of transiently chaotic neural network (TCNN) is affected by various factors such as chaotic characteristic, model parameters, and annealing function, and its capacity of global optimization is limited. It is demonstrated that the non-monotonic activation function can generate richer chaotic characteristic than the monotonic activation function in the TCNN model. Besides, the activation function involving neurobiological mechanism can not only reflect the rich brain activity in brain waves, but also enhance the non-linear dynamic characteristic, which may further improve the global optimization ability. Hence, a novel chaotic neuron model is proposed with the non-monotonic activation function based on the neurobiological mechanisms from the electroencephalogram. The electroencephalogram consists of five brain waves (i.e., , , , , and waves) which are defined by the quality and intensity of brain waves with different frequency bands ranging from 0.5 Hz to 100 Hz. The brain wave with a higher frequency and a lower amplitude represents a more active brain. Researches demonstrate that the five brain waves can be simplified into sinusoidal waves with different frequencies. Hence, a frequency conversion sinusoidal (FCS) function which has the consistent frequency range and features with brain waves is designed based on the above neurobiological mechanisms. Then a novel chaotic neuron model with non-monotonic activation function which is composed of the FCS function and sigmoid function, is proposed for richer chaotic dynamic characteristic. The reversed bifurcation and the Lyapunov exponent of the chaotic neuron are given and the dynamic system is analyzed, indicating that the proposed FCS neuron model owns richer chaotic dynamic characteristic than transiently chaotic neuron model due to its special non-monotonic activation function. Based on the neuron model, a novel transiently-chaotic neural networkfrequency conversion sinusoidal chaotic neural network (FCSCNN) is constructed and the basis of model parameter selection is provided as well. To validate the effectiveness of the proposed model, the FCSCNN is applied to nonlinear function optimization and 10-city, 30-city, 75-city traveling salesman problem. The experimental results show that 1) the FCSCNN has a good performance under the condition of moderate a, smaller cA(0) and 2(0); 2) on the basis of the appropriate model parameters, the FCSCNN has better global optimization ability and optimization accuracy than Hopfield neural network, TCNN, improved-TCNN due to its richer chaotic characteristic in complicated combinational optimization problem, especially in middle and large scale problem.

Multistability of memristive Cohen–Grossberg neural networks with non-monotonic piecewise linear activation functions and time-varying delays

The problem of coexistence and dynamical behaviors of multiple equilibrium points is addressed for a class of memristive Cohen–Grossberg neural networks with non-monotonic piecewise linear activation functions and time-varying delays. By virtue of the fixed point theorem, nonsmooth analysis theory and other analytical tools, some sufficient conditions are established to guarantee that such n-dimensional memristive Cohen–Grossberg neural networks can have 5n equilibrium points, among which 3n equilibrium points are locally exponentially stable. It is shown that greater storage capacity can be achieved by neural networks with the non-monotonic activation functions introduced herein than the ones with Mexican-hat-type activation function. In addition, unlike most existing multistability results of neural networks with monotonic activation functions, those obtained 3n locally stable equilibrium points are located both in saturated regions and unsaturated regions. The theoretical findings are verified by an illustrative example with computer simulations.

A general insight into the effect of neuron structure on classification

This paper gives a general insight into how the neuron structure in a multilayer perceptron (MLP) can affect the ability of neurons to deal with classification. Most of the common neuron structures are based on monotonic activation functions and linear input mappings. In comparison, the proposed neuron structure utilizes a nonmonotonic activation function and/or a nonlinear input mapping to increase the power of a neuron. An MLP of these high power neurons usually requires a less number of hidden nodes than conventional MLP for solving classification problems. The fewer number of neurons is equivalent to the smaller number of network weights that must be optimally determined by a learning algorithm. The performance of learning algorithm is usually improved by reducing the number of weights, i.e., the dimension of the search space. This usually helps the learning algorithm to escape local optimums, and also, the convergence speed of the algorithm is increased regardless of which algorithm is used for learning. Several 2-dimensional examples are provided manually to visualize how the number of neurons can be reduced by choosing an appropriate neuron structure. Moreover, to show the efficiency of the proposed scheme in solving real-world classification problems, the Iris data classification problem is solved using an MLP whose neurons are equipped by nonmonotonic activation functions, and the result is compared with two well-known monotonic activation functions.

Large Memory Capacity in Chaotic Artificial Neural Networks: A View of the Anti-Integrable Limit

In the literature, it was reported that the chaotic artificial neural network model with sinusoidal activation functions possesses a large memory capacity as well as a remarkable ability of retrieving the stored patterns, better than the conventional chaotic model with only monotonic activation functions such as sigmoidal functions. This paper, from the viewpoint of the anti-integrable limit, elucidates the mechanism inducing the superiority of the model with periodic activation functions that includes sinusoidal functions. Particularly, by virtue of the anti-integrable limit technique, this paper shows that any finite-dimensional neural network model with periodic activation functions and properly selected parameters has much more abundant chaotic dynamics that truly determine the model's memory capacity and pattern-retrieval ability. To some extent, this paper mathematically and numerically demonstrates that an appropriate choice of the activation functions and control scheme can lead to a large memory capacity and better pattern-retrieval ability of the artificial neural network models.

On Stability of Neural Networks by a Lyapunov Functional-Based Approach

In this paper, a new Lyapunov functional-based method is proposed for the stability analysis of delayed cellular neural networks (DCNN). Global exponential stability conditions are obtained for the general DCNN, the Hopfield neural networks (HNNs), and delayed HNNs with monotonic nondecreasing and nonconstant activation functions

Autocorrelation associative memory using refractory period of neurons

AbstractIn autocorrelation associative memory, a monotonic activation function is used from the viewpoint of network stability. The problems arise that the memory capacity is reduced and false memory can be generated, although the structure is simplified. This paper proposes a new memory model in which the associative ability is improved, while the conventional monotonic activation function is used in the autocorrelation associative memory model. In the proposed model, the refractory period is defined in the neuron, and the same convergence of the network as in the Hopfield model is always guaranteed. In the case of the discrete‐time model, the proposed model with the refractory period T = 1 is equivalent to the case of a ternary nonmonotonic activation function. In other words, the proposed model can be regarded as a model including the nonmonotonic activation function model. The following results are shown by numerical experiment. As a discrete‐time discrete‐value model, the proposed model has a higher associative ability than the Hopfield model. As a continuous‐time continuous‐value model, when the refractory period is long, the associative ability is greater than that of the Hopfield model guaranteeing the same convergence. When the refractory period is short, the associative ability is further improved. © 2004 Wiley Periodicals, Inc. Electron Comm Jpn Pt 2, 87(8): 41–52, 2004; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ecjb.20107

Back propagation learning with periodic chaos neurons

AbstractThis paper proposes a back propagation method using a periodic activation function for a hierarchical neural network and examines its learning capability. As a specific learning problem, the paper addresses the n‐bit parity problem and conducts a learning process by using a three‐layer neural network. The computer simulation results indicate an increased learning coefficient range achieving a 100% learning success rate with an improved speed of learning compared with a neural network model using a monotonic activation function. In addition, the new method lessens the influence of the initial value that is set at the beginning of a learning process, and reduces the degree of dependency on parameters. © 2002 Wiley Periodicals, Inc. Electron Comm Jpn Pt 3, 86(3): 11–19, 2003; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ecjc.1136

Statistical Properties of Chaos Associative Memory

In this paper we shall investigate the statistical property and the memory capacity of the chaotic autoassociation memory. The present artificial neuron model is properly characterized in terms of a time-dependent sinusoidal activation function to involve a transient chaotic dynamics as well as the energy steepest descent strategy. It is elucidated that the present neural network has a remarkable retrieval ability beyond the conventional models with such a monotonous activation function as sigmoidal one. This advantage is found to result from the property of the analogue periodic mapping accompanied with a chaotic behaviour of the neurons as well as the symmetry of the dynamic equation which may be shown in the invariant measure determined by the Frobenius-Perron equation.

Use of periodic and monotonic activation functions in multilayer feedforward neural networks trained by extended Kalman filter algorithm

The authors investigate the convergence and pruning performance of multilayer feedforward neural networks with different types of neuronal activation functions in solving various problems. Three types of activation functions are adopted in the network, namely, the traditional sigmoid function, the sinusoidal function and a periodic function that can be considered as a combination of the first two functions. To speed up the learning, as well as to reduce the network size, the extended Kalman filter (EKF) algorithm conjunct with a pruning method is used to train the network. The corresponding networks are applied to solve five typical problems, namely, 4-point XOR logic function, parity generation, handwritten digit recognition, piecewise linear function approximation and sunspot series prediction. Simulation results show that periodic activation functions perform better than monotonic ones in solving multicluster classification problems. Moreover, the combined periodic activation function is found to possess the fast convergence and multicluster classification capabilities of the sinusoidal activation function while keeping the robustness property of the sigmoid function required in the modelling of unknown systems.

Repeated least squares with inversion and its application in identifying linear distributed-parameter systems

In the paper an approach to a certain class of the nonlinear parameter estimation problem is proposed, which is, in particular, applicable to distributed-parameter systems described by elliptic partial differential equations. The approach exploits the special structure of nonlinear dependence, which allows the least-squares algorithm to be applied twice, together with the inversion of a nonlinear characteristic. One can roughly say that the class of considered systems can be described by a feedforward neural net with two hidden layers and monotonic activation functions. In the language of neural nets, the estimation problem can be interpreted as a partial inversion of the net, that is finding part of its inputs from a learning sequence. Simulations confirm that the approach is useful and much simpler than a direct iteration minimization of the sum of squares.

A Chaos Associative Model with a Sinusoidal Activation Function

In this paper, we shall propose a novel chaos neural network model applied to the chaotic autoassociation memory. The present artificial neuron model is properly characterized in terms of a time-dependent periodic activation function to involve chaotic dynamics as well as the energy steepest descent strategy. It is elucidated that the present neural network has a remarkable ability of dynamic memory retrieval beyond the conventional models with the nonmonotonous activation function as well as a monotonous activation function as the sigmoidal one. This advantage is found to result from the property of the analogue periodic mapping accompanied with a chaotic behaviour of the neurons. It is also concluded that the present analogue neuron model, with the periodicity control, has an apparently large memory capacity in comparison with the previously proposed association models.

Chaos Associative Memory with a Periodic Activation Function

In this paper we shall propose a novel chaos neural network model applied to the chaotic autoassociation memory. The present artificial neuron model is properly characterized in terms of a time-dependent periodic activation function to involve a chaotic dynamics as well as the energy steepest descent strategy. It is elucidated that the present neural network has a remarkable ability of the dynamic memory retrievals beyond the conventional models with the nonmonotonous activation function as well as such a monotonous activation function as sigmoidal one. This advantage is found to result from the property of the analogue periodic mapping accompanied with a chaotic behaviour of the neurons. It is also concluded that the present analogue neuron model with the periodicity control has an apparently large memory capacity in comparison with the previously proposed association models.

Multi-threshold neurones perceptron

Minsky and Papert (Perceptons: An Introduction to Computational Geometry, MIT Press: Cambridge, MA, 1969) show that a two-layer perceptron with monotonic activation function cannot solve the Xor problem. We present a two-layer perceptron with non-monotonic activation function which can separate non-linearly separated sets of data. This kind of perceptron is the applied to the Xor problem, parity problem and a pattern recognition problem. Learning strategy is detailed, and performance is evaluated.