Piecewise Linear Activation Functions Research Articles

Piecewise Linear Transformation – Propagating Aleatoric Uncertainty in Neural Networks

Real-world data typically exhibit aleatoric uncertainty which has to be considered during data-driven decision-making to assess the confidence of the decision provided by machine learning models. To propagate aleatoric uncertainty represented by probability distributions (PDs) through neural networks (NNs), both sampling-based and function approximation-based methods have been proposed. However, these methods suffer from significant approximation errors and are not able to accurately represent predictive uncertainty in the NN output. In this paper, we present a novel method, Piecewise Linear Transformation (PLT), for propagating PDs through NNs with piecewise linear activation functions (e.g., ReLU NNs). PLT does not require sampling or specific assumptions about the PDs. Instead, it harnesses the piecewise linear structure of such NNs to determine the propagated PD in the output space. In this way, PLT supports the accurate quantification of predictive uncertainty based on the criterion exactness of the propagated PD. We assess this exactness in theory by showing error bounds for our propagated PD. Further, our experimental evaluation validates that PLT outperforms competing methods on publicly available real-world classification and regression datasets regarding exactness. Thus, the PDs propagated by PLT allow to assess the uncertainty of the provided decisions, offering valuable support.

Smoothing piecewise linear activation functions based on mollified square root functions

Smoothing piecewise linear activation functions based on mollified square root functions

Extending Neural Network Verification to a Larger Family of Piece-wise Linear Activation Functions

Extending Neural Network Verification to a Larger Family of Piece-wise Linear Activation Functions

Leveraging Satisfiability Modulo Theory Solvers for Verification of Neural Networks in Predictive Maintenance Applications

Interest in machine learning and neural networks has increased significantly in recent years. However, their applications are limited in safety-critical domains due to the lack of formal guarantees on their reliability and behavior. This paper shows recent advances in satisfiability modulo theory solvers used in the context of the verification of neural networks with piece-wise linear and transcendental activation functions. An experimental analysis is conducted using neural networks trained on a real-world predictive maintenance dataset. This study contributes to the research on enhancing the safety and reliability of neural networks through formal verification, enabling their deployment in safety-critical domains.

Activation Function Effects and Simplified Implementation for Hopfield Neural Network

The activation function is crucial in the Hopfield neural network (HNN) to restrict the input–output relation of each neuron. The physical realizability and simplicity of the hardware circuit of activation function are beneficial to promote the practical engineering application of the HNN. However, the HNN commonly used hyperbolic tangent activation function involves a complex hardware circuit implementation. This paper discusses a piecewise-linear activation function (PWL-AF) with simplified circuit implementation and a tri-neuron small-world HNN is built as a paradigm. The hardware implementation circuit of the HNN is greatly simplified, benefited from the PWL-AF with a simple analog circuit. Meanwhile, the dynamics related to the PWL-AF and initial conditions are numerically explored. The numerical results demonstrate that the PWL-AF-based HNN can produce dynamical behaviors like the HNN based on the hyperbolic tangent activation function. Nevertheless, the multistability with up to six kinds of coexisting multiple attractors emerged because of the PWL-AF breakpoint. This can give more flexible and potential aspects in multistability-based engineering applications. Especially, the PWL-AF breakpoint value simultaneously acts as the offset booster and amplitude controller in regulating the offset boosting and amplitude rescaling of neuron states. Afterwards, an analog circuit with three straightforward operational amplifiers (op-amp)-based circuit modules is designed for the PWL-AF, and a PCB-based analog circuit is thereby implemented for the tri-neuron small-world HNN. The hardware experiments agree with the numerical simulations, implying the feasibility of the PWL-AF simplification for the HNN.

Design of continuous-time recurrent neural networks with piecewise-linear activation function for generation of prescribed sequences of bipolar vectors

A recurrent neural network (RNN) can generate a sequence of patterns as the temporal evolution of the output vector. This paper focuses on a continuous-time RNN model with a piecewise-linear activation function that has neither external inputs nor hidden neurons, and studies the problem of finding the parameters of the model so that it generates a given sequence of bipolar vectors. First, a sufficient condition for the model to generate the desired sequence is derived, which is expressed as a system of linear inequalities in the parameters. Next, three approaches to finding solutions of the system of linear inequalities are proposed: One is formulated as a convex quadratic programming problem and others are linear programming problems. Then, two types of sequences of bipolar vectors that can be generated by the model are presented. Finally, the case where the model generates a periodic sequence of bipolar vectors is considered, and a sufficient condition for the trajectory of the state vector to converge to a limit cycle is provided.

Memristor synapse-coupled piecewise-linear simplified Hopfield neural network: Dynamics analysis and circuit implementation

Electromagnetic induction current is generated between the adjacent neurons in neural network caused by the existence of membrane potential difference. Memristor is the fourth fundamental electric element, which can mimic the behavior of neural synapses and simulate the electromagnetic induction effect. In this paper, in order to simplify practical implementation, the commonly used hyperbolic tangent activation function of Hopfield neural network (HNN) is replaced by a piecewise-linear function, and a simple bi-neuron-based memristor synapse-coupled HNN is proposed. Theoretical analysis and numerical simulation results illustrate that the memsirstive HNN possesses five equilibria including one unstable saddle-focus, two unstable saddle points, and two stable node points (or node-foci). Local attraction basins and phase plane plots show that the memristive HNN model behaves multistability of coexisting chaos, periodic limit cycles, and stable point attractors. Various system dynamical behaviors affected by parameters of the piecewise-linear activation function, memristor coupling strength, and initial conditions of the neurons are investigated by numerical simulations. Furthermore, an analog circuit of the memristive HNN model is simply designed, which is easier for hardware implementation as the piecewise-linear activation function can be implemented by simple op-amp-based module. Finally, the experimental results verify the correctness of the design and analyses.

Specific chemical bioexergy and microwave-assisted torrefaction optimization via statistical and artificial intelligence approaches

The artificial neural network (ANN) in artificial intelligence (AI) is a computational model that portrays how nerve cells (neurons) work in the human brain. Meanwhile, specific chemical bioexergy (SCB) is a vital indicator to provide essential information on how high the actual energy may be contained within biomass fuel (biofuel). In this study, the Taguchi method, analysis of variance (ANOVA), and artificial neural network (AI-ANN) are utilized to predict the SCB of biofuels from the spent mushroom substrate (SMS) torrefaction via microwave-assisted heating (MAH). In Taguchi’s orthogonal array, washing, catalyst, and power operating parameters are considered. Acid and water washings lower the SMS’s ash content by 58.45 % and 36.29 %, respectively. The optimum conditions for combining with acid washing, a catalyst with higher Fe2O3 (35 %), and microwave power 540 W render the highest total SCB in biofuels (biochar + bio-oil) of 47.90 MJ·kg−1, which is close to the SCB of crude oil derivatives (41–49 MJ·kg−1). The enhancement of biomass’s SCB value from optimal torrefaction approximately 3 folds (256.30 %) from 12.84 (raw) to 47.90 MJ·kg−1. The ANN model with an architecture of 1 hidden layer (sigmoid activation function) with 3 neurons and the output layer (piecewise linear activation function) with a quick propagation algorithm for the training process of all layers poses excellent prediction with high accuracy R2 = 1. This result demonstrates that ANN with the designed scheme is suitable for predicting the SCB of SMS-derived biofuels.

Complex network dynamics of a memristor neuron model with piecewise linear activation function

Complex network dynamics of a memristor neuron model with piecewise linear activation function

Analysis on the Number of Linear Regions of Piecewise Linear Neural Networks.

Deep neural networks (DNNs) are shown to be excellent solutions to staggering and sophisticated problems in machine learning. A key reason for their success is due to the strong expressive power of function representation. For piecewise linear neural networks (PLNNs), the number of linear regions is a natural measure of their expressive power since it characterizes the number of linear pieces available to model complex patterns. In this article, we theoretically analyze the expressive power of PLNNs by counting and bounding the number of linear regions. We first refine the existing upper and lower bounds on the number of linear regions of PLNNs with rectified linear units (ReLU PLNNs). Next, we extend the analysis to PLNNs with general piecewise linear (PWL) activation functions and derive the exact maximum number of linear regions of single-layer PLNNs. Moreover, the upper and lower bounds on the number of linear regions of multilayer PLNNs are obtained, both of which scale polynomially with the number of neurons at each layer and pieces of PWL activation function but exponentially with the number of layers. This key property enables deep PLNNs with complex activation functions to outperform their shallow counterparts when computing highly complex and structured functions, which, to some extent, explains the performance improvement of deep PLNNs in classification and function fitting.

Verification of piecewise deep neural networks: a star set approach with zonotope pre-filter

Abstract Verification has emerged as a means to provide formal guarantees on learning-based systems incorporating neural network before using them in safety-critical applications. This paper proposes a new verification approach for deep neural networks (DNNs) with piecewise linear activation functions using reachability analysis. The core of our approach is a collection of reachability algorithms using star sets (or shortly, stars), an effective symbolic representation of high-dimensional polytopes. The star-based reachability algorithms compute the output reachable sets of a network with a given input set before using them for verification. For a neural network with piecewise linear activation functions, our approach can construct both exact and over-approximate reachable sets of the neural network. To enhance the scalability of our approach, a star set is equipped with an outer-zonotope (a zonotope over-approximation of the star set) to quickly estimate the lower and upper bounds of an input set at a specific neuron to determine if splitting occurs at that neuron. This zonotope pre-filtering step reduces significantly the number of linear programming optimization problems that must be solved in the analysis, and leads to a reduction in computation time, which enhances the scalability of the star set approach. Our reachability algorithms are implemented in a software prototype called the neural network verification tool, and can be applied to problems analyzing the robustness of machine learning methods, such as safety and robustness verification of DNNs. Our experiments show that our approach can achieve runtimes twenty to 1400 times faster than Reluplex, a satisfiability modulo theory-based approach. Our star set approach is also less conservative than other recent zonotope and abstract domain approaches.

Tropical Geometry and Machine Learning

Tropical geometry is a relatively recent field in mathematics and computer science, combining elements of algebraic geometry and polyhedral geometry. The scalar arithmetic of its analytic part preexisted in the form of max-plus and min-plus semiring arithmetic used in finite automata, nonlinear image processing, convex analysis, nonlinear control, optimization, and idempotent mathematics. Tropical geometry recently emerged in the analysis and extension of several classes of problems and systems in both classical machine learning and deep learning. Three such areas include: 1) deep neural networks with piecewise linear (PWL) activation functions; 2) probabilistic graphical models; and 3) nonlinear regression with PWL functions. In this article, we first summarize introductory ideas and objects of tropical geometry, providing a theoretical framework for both the max-plus algebra that underlies tropical geometry and its extensions to general max algebras. This unifies scalar and vector/signal operations over a class of nonlinear spaces, called weighted lattices, and allows us to provide optimal solutions for algebraic equations used in tropical geometry and generalize tropical geometric objects. Then, we survey the state of the art and recent progress in the aforementioned areas. First, we illustrate a purely geometric approach for studying the representation power of neural networks with PWL activations. Then, we review the tropical geometric analysis of parametric statistical models, such as HMMs; later, we focus on the Viterbi algorithm and related methods for weighted finite-state transducers and provide compact and elegant representations via their formal tropical modeling. Finally, we provide optimal solutions and an efficient algorithm for the convex regression problem, using concepts and tools from tropical geometry and max-plus algebra. Throughout this article, we also outline problems and future directions in machine learning that can benefit from the tropical-geometric point of view.

Multistability of state-dependent switching neural networks with discontinuous nonmonotonic piecewise linear activation functions

This paper presents the theoretical results on the multistability of state-dependent switching neural networks with discontinuous nonmonotonic piecewise linear activation functions. For n-neurons switching model, this paper shows that neural networks have 7n equilibrium points, 6n of which are located at the continuous points of activation functions and others are located at the discontinuous points of activation functions. Among these equilibrium points, 4n or 5n are stable and others are unstable, which depend on the relationship between the switching threshold and the discontinuous points of the activation functions. Compared with existing results, this paper reveals that switching threshold and discontinuous character are crucial in increasing the number of equilibrium points. Two examples are presented to verify the theoretical results.

Learning Deep Sparse Regularizers With Applications to Multi-View Clustering and Semi-Supervised Classification.

Sparsity-constrained optimization problems are common in machine learning, such as sparse coding, low-rank minimization and compressive sensing. However, most of previous studies focused on constructing various hand-crafted sparse regularizers, while little work was devoted to learning adaptive sparse regularizers from given input data for specific tasks. In this paper, we propose a deep sparse regularizer learning model that learns data-driven sparse regularizers adaptively. Via the proximal gradient algorithm, we find that the sparse regularizer learning is equivalent to learning a parameterized activation function. This encourages us to learn sparse regularizers in the deep learning framework. Therefore, we build a neural network composed of multiple blocks, each being differentiable and reusable. All blocks contain learnable piecewise linear activation functions which correspond to the sparse regularizer to be learned. Furthermore, the proposed model is trained with back propagation, and all parameters in this model are learned end-to-end. We apply our framework to multi-view clustering and semi-supervised classification tasks to learn a latent compact representation. Experimental results demonstrate the superiority of the proposed framework over state-of-the-art multi-view learning models.

Multistability of delayed neural networks with monotonically nondecreasing linear activation function

This paper investigates the multistability on a class of delayed neural networks with monotonically nondecreasing linear activation function. For n state neurons with 2m+1 piecewise linear activation function, we prove the neural networks have (2m+1)n equilibrium points, (m+1)n of which are locally exponentially stable. Different from the traditional multistability analysis method such as fixed point theory, this paper only uses the character of activation functions, and can also handle the neural networks with disturbance term. The research results of this paper generalize the previous related research works and are easy to test. A numerical example is given to shown the effectiveness of our theoretical results.

On Sharpness of Error Bounds for Univariate Approximation by Single Hidden Layer Feedforward Neural Networks

A new non-linear variant of a quantitative extension of the uniform boundedness principle is used to show sharpness of error bounds for univariate approximation by sums of sigmoid and ReLU functions. Single hidden layer feedforward neural networks with one input node perform such operations. Errors of best approximation can be expressed using moduli of smoothness of the function to be approximated (i.e., to be learned). In this context, the quantitative extension of the uniform boundedness principle indeed allows to construct counterexamples that show approximation rates to be best possible. Approximation errors do not belong to the little-o class of given bounds. By choosing piecewise linear activation functions, the discussed problem becomes free knot spline approximation. Results of the present paper also hold for non-polynomial (and not piecewise defined) activation functions like inverse tangent. Based on Vapnik–Chervonenkis dimension, first results are shown for the logistic function.

Monostability and Multistability for Almost-Periodic Solutions of Fractional-Order Neural Networks With Unsaturating Piecewise Linear Activation Functions

Since the unsaturating activation function is unbounded, more complex dynamics may exist in neural networks with this kind of activation function. In this article, monostability and multistability results of almost-periodic solutions are developed for fractional-order neural networks with unsaturating piecewise linear activation functions. Some globally Mittag-Leffler attractive sets are given, and the existence of globally Mittag-Leffler stable almost-periodic solution is demonstrated by using Ascoli-Arzela theorem. In particular, some sufficient conditions are provided to ascertain the multistability of almost-periodic solutions based on locally positively invariant set. It shows that there exists an almost-periodic solution in each positively invariant set, and all trajectories converge to this periodic trajectory in that rectangular area. Two illustrative examples are provided to demonstrate the effectiveness of the proposed sufficient criteria.

Deep Learning Classifier with Piecewise Linear Activation Function: An Empirical Evaluation with Intraday Financial Data

Price movement predictions of financial instruments using traditional time-series models with a predefined mathematical structure are common, but this method restricts their ability to learn latent patterns in the data. In recent times, artificial neural networks (ANN) have been able to learn complex hidden patterns from financial datasets using a highly nonlinear architecture. However, most experiments with neural networks require a lot of time to search a suitable network and subsequently train the network. The authors have developed a deep multilayer perceptron (MLP) classifier with a zero-centered piecewise linear unit activation that yielded better classification performance according to their accuracy metric and required consistently less training time compared to a similar MLP network with leaky rectified linear unit and tanh activation functions. The authors illustrate their technique with a large high-frequency dataset on selected bank shares from the Indian stock market. The authors also discuss the theoretical properties and advantages of their proposal. TOPICS:Statistical methods, simulations, big data/machine learning Key Findings • Time-series models have been used extensively to understand stock price movements. They are constrained by a predefined structure. Artificial neural network (ANNs), due to their universal approximation properties, offer more flexibility; however, they suffer from long training times. • In this article, the authors introduce a zero-centered piecewise linear unit (PLU), a hybrid activation function derived from tanh and leaky rectified linear unit (ReLU) functions that exhibits the zero-centered property from tanh and the nonzero gradient property from leaky ReLU. • The authors developed a multilayer perceptron (MLP) classifier with PLU activation, which yielded better classification performance according to the accuracy metric and required consistently less training time compared to a similar MLP network with leaky ReLU and tanh activation.

Multistability of switched neural networks with sigmoidal activation functions under state-dependent switching

This paper presents theoretical results on the multistability of switched neural networks with commonly used sigmoidal activation functions under state-dependent switching. The multistability analysis with such an activation function is difficult because state–space partition is not as straightforward as that with piecewise-linear activations. Sufficient conditions are derived for ascertaining the existence and stability of multiple equilibria. It is shown that the number of stable equilibria of an n-neuron switched neural networks is up to 3n under given conditions. In contrast to existing multistability results with piecewise-linear activation functions, the results herein are also applicable to the equilibria at switching points. Four examples are discussed to substantiate the theoretical results.

Multistability and attraction basins of discrete-time neural networks with nonmonotonic piecewise linear activation functions

This paper is concerned with multistability and attraction basins of discrete-time neural networks with nonmonotonic piecewise linear activation functions. Under some reasonable conditions, the addressed networks have (2m+1)n equilibrium points. (m+1)n of which are locally asymptotically stable, and the others are unstable. The attraction basins of the locally asymptotically stable equilibrium points are given in the form of hyperspherical regions. These results here, which include existence, uniqueness, locally asymptotical stability, instability and attraction basins of the multiple equilibrium points, generalize and improve the earlier publications. Finally, an illustrative example with numerical simulation is given to show the feasibility and the effectiveness of the theoretical results. The theoretical results and illustrative example indicate that the activation functions improve the storage capacity of neural networks significantly.