General Activation Functions Research Articles

Multistability of recurrent neural networks with general periodic activation functions and unbounded time-varying delays

This paper investigates the multistability of recurrent neural networks (RNNs) with unbounded time-varying delays whose activation functions are general periodic functions. The activation function can be linear, nonlinear, or have multiple corner points, as long as it satisfies Lipschitz continuous condition. According to the characteristics of the parameters of the RNNs and the state space division method, the number of equilibrium points (EPs) of the RNNs is split into three categories, which can be unique, finite, or countable infinite. Some sufficient conditions for determining the number of EPs are presented, the criterion of asymptotically stable EPs is deduced, and the attraction basins of stable EPs are estimated. Furthermore, the multistability results in this paper are an extension of some previous results. The theoretical results are validated using three numerical simulation examples.

Normal Approximation of Random Gaussian Neural Networks

In this paper, we provide explicit upper bounds on some distances between the (law of the) output of a random Gaussian neural network and (the law of) a random Gaussian vector. Our main results concern deep random Gaussian neural networks with a rather general activation function. The upper bounds show how the widths of the layers, the activation function, and other architecture parameters affect the Gaussian approximation of the output. Our techniques, relying on Stein’s method and integration by parts formulas for the Gaussian law, yield estimates on distances that are indeed integral probability metrics and include the convex distance. This latter metric is defined by testing against indicator functions of measurable convex sets and so allows for accurate estimates of the probability that the output is localized in some region of the space, which is an aspect of a significant interest both from a practitioner’s and a theorist’s perspective. We illustrated our results by some numerical examples. Funding: This research was supported by the European Union’s Horizon 2020 research project WARIFA under grant agreement no. 101017385, by the PRIN project 2022 “Variational Analysis of Complex Systems in Materials Science, Physics and Biology” (CUP B53D23009290006), and by the INdAM project “Modelli ed Algoritmi per dati ad elevata dimensionalità” (CUP E53C23001670001).

GEPAF: A non-monotonic generalized activation function in neural network for improving prediction with diverse data distributions characteristics

The world today has made prescriptive analytics that uses data-driven insights to guide future actions. The distribution of data, however, differs depending on the scenario, making it difficult to interpret and comprehend the data efficiently. Different neural network models are used to solve this, taking inspiration from the complex network architecture in the human brain. The activation function is crucial in introducing non-linearity to process data gradients effectively. Although popular activation functions such as ReLU, Sigmoid, Swish, and Tanh have advantages and disadvantages, they may struggle to adapt to diverse data characteristics. A generalized activation function named the Generalized Exponential Parametric Activation Function (GEPAF) is proposed to address this issue. This function consists of three parameters expressed: α, which stands for a differencing factor similar to the mean; σ, which stands for a variance to control distribution spread; and p, which is a power factor that improves flexibility; all these parameters are present in the exponent. When p=2, the activation function resembles a Gaussian function. Initially, this paper describes the mathematical derivation and validation of the properties of this function mathematically and graphically. After this, the GEPAF function is practically implemented in real-world supply chain datasets. One dataset features a small sample size but exhibits high variance, while the other shows significant variance with a moderate amount of data. An LSTM network processes the dataset for sales and profit prediction. The suggested function performs better than popular activation functions when a comparative analysis of the activation function is performed, showing at least 30% improvement in regression evaluation metrics and better loss decay characteristics.

On H∞\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${H}_\\infty $$\\end{document} Finite-Time Boundedness and Finite-Time Stability for Discrete-Time Neural Networks with Leakage Time-Varying Delay

This paper investigates the problems of H∞\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$H_\\infty $$\\end{document} finite-time boundedness (FTB) and finite-time stability (FTS) for discrete-time neural networks (NNs) with both leakage delay and discrete delay, as well as different generalized activation functions. To this end, we construct suitable Lyapunov–Krasovskii (L–K) functionals and apply the extended reciprocally convex approach to derive delay-dependent criteria for the addressed problems. The obtained conditions are expressed as linear matrix inequalities (LMIs), which can be efficiently solved by the LMI Toolbox in Matlab. The paper also admits of two numerical examples to point up the advantages and reliability of the proposed method.

μ-stability and instability of multiple equilibrium points in delayed neural networks with general discontinuous activation functions

This paper proposes a general type of discontinuous activation functions (AFs) and studies the μ-stability of multiple equilibrium points (EPs) in delayed neural networks (DNNs). By judging 4n algebraic inequalities and the nonsingularity of an M-matrix, DNNs with the general discontinuous AFs can be shown to have 5n EPs, therein 3n EPs are locally μ-stable. Moreover, these 3n EPs are located at the points of continuity of the AF. Compared with the existing general continuous AF, DNNs with the general discontinuous AF can have more total/stable EPs. Hence, DNNs with the general discontinuous AF could store a much larger number of memory patterns if they are applied to associative memory. In addition, the attraction basin (AB) of each stable EP in DNNs is estimated. Two numerical examples are shown to testify the validity of the obtained results.

Multistability of Almost Periodic Solutions for Fuzzy Competitive NNs With Time-Varying Delays.

In this article, the multistability problem of almost periodic solutions of fuzzy competitive neural networks (FCNNs) with time-varying delays is investigated. Considering more general activation functions, which are nonmonotonic and nonlinear, and incorporating the almost periodic property of the parameters in FCNNs, sufficient conditions for the multistability of almost periodic solutions are given. ∏r=1n(Lr+1) stable almost periodic solutions are obtained, where Lr depends on the geometric features of the activation functions, which enriches and extends the research on multistability in fuzzy systems. Furthermore, the extended domain of attraction based on the original state space is presented. Finally, numerical simulations are provided to verify the conclusions of this article.

Stability Analysis of Quaternion-Valued Neutral Neural Networks with Generalized Activation Functions

Stability Analysis of Quaternion-Valued Neutral Neural Networks with Generalized Activation Functions

Training Provably Robust Models by Polyhedral Envelope Regularization.

Training certifiable neural networks enables us to obtain models with robustness guarantees against adversarial attacks. In this work, we introduce a framework to obtain a provable adversarial-free region in the neighborhood of the input data by a polyhedral envelope, which yields more fine-grained certified robustness than existing methods. We further introduce polyhedral envelope regularization (PER) to encourage larger adversarial-free regions and thus improve the provable robustness of the models. We demonstrate the flexibility and effectiveness of our framework on standard benchmarks; it applies to networks of different architectures and with general activation functions. Compared with state of the art, PER has negligible computational overhead; it achieves better robustness guarantees and accuracy on the clean data in various settings.

Latent code-based fusion: A Volterra neural network approach

We propose a deep structure encoder using Volterra Neural Networks (VNNs) to seek a latent representation of multi-modal data whose features are jointly captured by a union of subspaces. The so-called self-representation embedding of the latent codes leads to a simplified fusion which is driven by a similarly constructed decoding. The Volterra Filter architecture achieved reduction in parameter complexity is primarily due to controlled non-linearities being introduced by the higher-order convolutions in lieu of generalized activation functions. Experimental results on two different datasets have shown a significant improvement in the clustering performance for VNNs auto-encoder over conventional Convolutional Neural Networks (CNNs) auto-encoder. In addition, we also show that the proposed approach demonstrates a much-improved sample complexity over CNN-based auto-encoder with a robust classification performance.

Finite-Time and Fixed-Time Synchronization of Quaternion-Valued Neural Networks With/Without Mixed Delays: An Improved One-Norm Method.

In this article, the finite-time synchronization (FTSYN) of a class of quaternion-valued neural networks (QVNNs) with discrete and distributed time delays is studied. Furthermore, the FTSYN and fixed-time synchronization (FIXSYN) of the QVNNs without time delay are investigated. Different from the existing results, which used decomposition techniques, by introducing an improved one-norm, we use a direct analytical method to study the synchronization problems. Incidentally, several properties of one-norm of the quaternion are analyzed, and then, three effective controllers are proposed to synchronize the drive and response QVNNs within a finite time or fixed time. Moreover, efficient criteria are proposed to guarantee that the synchronization of QVNNs with or without mixed time delays can be realized within a finite and fixed time interval, respectively. In addition, the settling times are reckoned. Compared with the existing work, our advantages are mainly reflected in the simpler Lyapunov analytical process and more general activation function. Finally, the validity and practicability of the conclusions are illustrated via four numerical examples.

Synchronization of Uncertain Neural Networks with Additive Time-Varying Delays and General Activation Function

Synchronization of Uncertain Neural Networks with Additive Time-Varying Delays and General Activation Function

A novel finite-time complex-valued zeoring neural network for solving time-varying complex-valued Sylvester equation

A novel finite-time complex-valued zeroing neural network (FTCVZNN) for solving time-varying Sylvester equation is proposed and investigated. Asymptotic stability analysis of this network is examined with any general activation function satisfying a condition or with an odd monotonically increasing activation function. So far, finite-time model studies have been investigated for the upper bound time of convergence using a linear activation function with design formula for the derivative of the error or with variations of sign-bi-power activation functions to zeroing neural networks. A function adaptive coefficient for sign-bi-power activation function (FA-CSBP) is introduced and examined for faster convergence. An upper bound on convergence time is derived with the two components in the function adaptive coefficients of sign-bi-power activation function. Numerical simulation results demonstrate that the FTCVZNN with function adaptive coefficient for sign-bi-power activation function is faster than applying a sign-bi-power activation function to the zeroing neural network (ZNN) and the other finite-time complex-valued models for the selected example problems.

Finite-time stabilization of quaternion-valued neural networks with time delays: An implicit function method

In this study, we propose a non-decomposing method, the implicit Lyapunov function method, for the finite-time stabilization of a class of quaternion-valued neural networks (QVNNs) with time delays. Our approach has certain advantages, including more relaxed constraints on time delays, more flexible controllers without sign functions, and more general activation functions. Specifically, the derivative of an implicit Lyapunov function with quaternions is analyzed. Some properties of the implicit Lyapunov function, such as positive definiteness, monotonicity, and radial unboundedness, are deduced. With regard to two control schemes, two sets of sufficient conditions are provided for the finite-time stabilization of the QVNN, and an improved adaptive control strategy is designed to enhance stabilization efficiency. Two numerical examples illustrate the correctness, applicability and effectiveness of the method.

Extended analysis on the global Mittag-Leffler synchronization problem for fractional-order octonion-valued BAM neural networks

In this paper, a new case of neural networks called fractional-order octonion-valued bidirectional associative memory neural networks (FOOVBAMNNs) is established. First, the higher dimensional models are formulated for FOOVBAMNNs with general activation functions and the special linear threshold ones, respectively. On one hand, employing Cayley–Dichson construction in octonion multiplication which is essentially neither commutative nor associative, the system of FOOVBAMNNs is divided into four fractional-order complex-valued ones. On the other hand, Caputo fractional derivative’s character and BAM’s interactive feature are also properly dealt with. Second, the general criteria are obtained by the new design of LKFs, the application of the related inequalities and the construction of the linear feedback controllers for the global Mittag-Leffler synchronization problem of FOOVBAMNNs. Finally, we present two numerical examples to show the realizability and progress of the derived results.

An ESETM based robust synchronizing control on master-slave neural network with multiple time-varying delays

This paper focuses on the synchronization issue of master-slave neural network with multiple time-varying delays by way of an enhanced stretchy event-triggered mechanism (ESETM). Firstly, multiple time-varying delays are considered to express the generalization and complexity of system behavior. Secondly, the ESETM is designed by employing an internal dynamic variable and a modified adaptive function, which can provide the expected robustness performance during synchronization on the premise of saving limited network resources. Thirdly, less conservative stabilization criteria are derived in terms of an asymmetric Lyapunov-Krasovskii functional (LKF) candidate. In addition, Wirtinger-based integral inequality and reciprocally convex combination are utilized to estimate derivative of the constructed LKF. Meanwhile, a general activation function method is developed to introduce more information of system states on cross terms of neuron activation function to further reduce the conservatism. Finally, two numerical examples are illustrated to demonstrate the merits and effectiveness of the development.

Global [formula omitted]-stabilization of quaternion-valued inertial BAM neural networks with time-varying delays via time-delayed impulsive control

This article deals with global μ-stabilization of quaternion-valued inertial bidirectional associative memory neural networks (QVIBAMNNs) with time-varying delays via time-delayed impulsive control. Firstly, the existence and uniqueness of the equilibrium point are proved by the homeomorphism mapping method. Then, based on an improved differential inequality, sufficient conditions for global μ-stabilization of QVIBAMNNs via time-delayed impulsive control are obtained. Significantly, the activation function in this paper is an extension of Lipschitz condition, which is superior to general quaternion-valued activation function. And quaternion is not required to be decomposed into real or complex values in this paper. Ultimately, some numerical simulations have proved the feasibility of the theorems.

Noise-Tolerant Zeroing Neural Dynamics for Solving Hybrid Multilayered Time-Varying Linear Equation System

Resource allocation problem in a wireless sensor network can be formulated as time-varying optimization, which can be further converted as time-varying linear equation system (TVLES). Hybrid multilayered time-varying linear equation system (HMTVLES) involving hybrid multilayers and time-variation characteristic is a complicated and challenging problem. Recently it has been solved by zeroing neural dynamics (ZND) method under ideal conditions, i.e., without noises. However, noises are ubiquitous, immanent and unavoidable in real-time systems. In this work, we propose a noise-tolerant zeroing neural dynamics (NTZND) model for solving HMTVLES. It can deal with different kinds of noises such as constant noise, linear-increasing noise, and random noise. Theoretical analyses guarantee the precision of NTZND model in the presence of different kinds of noises. In addition, a general NTZND model is proposed based on general activation function. Besides, classical ZND method and gradient neural dynamics (GND) method are also investigated and compared. Numerical experimental results are presented to verify the theoretical results of proposed models.

Information theoretic limits of learning a sparse rule

We consider generalized linear models in regimes where the number of nonzero components of the signal and accessible data points are sublinear with respect to the size of the signal. We prove a variational formula for the asymptotic mutual information per sample when the system size grows to infinity. This result allows us to derive an expression for the minimum mean-square error (MMSE) of the Bayesian estimator when the signal entries have a discrete distribution with finite support. We find that, for such signals and suitable vanishing scalings of the sparsity and sampling rate, the MMSE is nonincreasing piecewise constant. In specific instances the MMSE even displays an all-or-nothing phase transition, that is, the MMSE sharply jumps from its maximum value to zero at a critical sampling rate. The all-or-nothing phenomenon has previously been shown to occur in high-dimensional linear regression. Our analysis goes beyond the linear case and applies to learning the weights of a perceptron with general activation function in a teacher-student scenario. In particular, we discuss an all-or-nothing phenomenon for the generalization error with a sublinear set of training examples.

Discovering Parametric Activation Functions

Recent studies have shown that the choice of activation function can significantly affect the performance of deep learning networks. However, the benefits of novel activation functions have been inconsistent and task dependent, and therefore the rectified linear unit (ReLU) is still the most commonly used. This paper proposes a technique for customizing activation functions automatically, resulting in reliable improvements in performance. Evolutionary search is used to discover the general form of the function, and gradient descent to optimize its parameters for different parts of the network and over the learning process. Experiments with four different neural network architectures on the CIFAR-10 and CIFAR-100 image classification datasets show that this approach is effective. It discovers both general activation functions and specialized functions for different architectures, consistently improving accuracy over ReLU and other activation functions by significant margins. The approach can therefore be used as an automated optimization step in applying deep learning to new tasks.

Novel trial functions and rogue waves of generalized breaking soliton equation via bilinear neural network method

In this work, some new test functions are constructed by setting generalized activation functions in different artificial network models. Bilinear neural network method is introduced to solve the explicit solution of a generalized breaking soliton equation. Rogue waves of generalized breaking soliton equation are obtained by symbolic computing technology and displayed intuitively with the help of Maple software.