Activation Function Of Neural Network Research Articles

Learning Specialized Activation Functions for Physics-Informed Neural Networks

Learning Specialized Activation Functions for Physics-Informed Neural Networks

Model reference control by recurrent neural network built with paraconsistent neurons for trajectory tracking of a rotary inverted pendulum

This investigation presents a recurrent paraconsistent neural network (RPNN), as the main element of the model reference control (MRC) strategy for the rotary inverted pendulum (RIP). The RIP characteristics, such as nonlinearity, two-degree-of-freedom (2DoF) motion, and under-actuated system, make it an ideal device to apply and test the RPNN. The designed paraconsistent neural model reference controller (PNMRC) uses three RPNNs: two of them to model the arm and pendulum angles and the third one to control the system while tracking a reference trajectory. The hidden neurons of the RPNN use the paraconsistent annotated logic by 2-value annotations (PAL2v) rules as an activation function. PAL2v, as a member of the paraconsistent logics family, deals with uncertain and contradictory data, representing a potentially robust alternative to applications of artificial neural networks in control. The PAL2v neuron is detailed and compared with other activation functions in recurrent neural networks (RNN). With real-time experiments, the PNMRC strategy is compared with classical control methodology, presenting excellent performance.

Memristor synapse-coupled piecewise-linear simplified Hopfield neural network: Dynamics analysis and circuit implementation

Electromagnetic induction current is generated between the adjacent neurons in neural network caused by the existence of membrane potential difference. Memristor is the fourth fundamental electric element, which can mimic the behavior of neural synapses and simulate the electromagnetic induction effect. In this paper, in order to simplify practical implementation, the commonly used hyperbolic tangent activation function of Hopfield neural network (HNN) is replaced by a piecewise-linear function, and a simple bi-neuron-based memristor synapse-coupled HNN is proposed. Theoretical analysis and numerical simulation results illustrate that the memsirstive HNN possesses five equilibria including one unstable saddle-focus, two unstable saddle points, and two stable node points (or node-foci). Local attraction basins and phase plane plots show that the memristive HNN model behaves multistability of coexisting chaos, periodic limit cycles, and stable point attractors. Various system dynamical behaviors affected by parameters of the piecewise-linear activation function, memristor coupling strength, and initial conditions of the neurons are investigated by numerical simulations. Furthermore, an analog circuit of the memristive HNN model is simply designed, which is easier for hardware implementation as the piecewise-linear activation function can be implemented by simple op-amp-based module. Finally, the experimental results verify the correctness of the design and analyses.

An Adaptive Offset Activation Function for CNN Image Classification Tasks

The performance of the activation function in convolutional neural networks is directly related to the model’s image classification accuracy. The rectified linear unit (ReLU) activation function has been extensively used in image classification models but has significant shortcomings, including low classification accuracy. The performance of a series of parametric activation functions has made parameter addition a popular research avenue for improving the performance of activation functions in recent years, and excellent progress has been achieved. Existing parametric activation functions often focus on assigning a different slope to the negative part of the activation function and still involve the negative value alone in the activation function calculation, without considering the impact of linking the negative value to the positive value on the performance of the activation function. As a result, this work proposes a novel parametric right-shift activation function, the adaptive offset activation function (AOAF). By inserting an adaptive parameter (the mean value of the input feature tensor) and two custom ReLU parameters, the negative parameters previously driven to zero by ReLU can be turned into positive parameters with lower weight and participate in CNN feature extraction. We compared the performance of the suggested activation function to the performance of a selection of typical activation functions using four distinct public datasets. Compared with ReLU, the average classification accuracy of our proposed activation function improved by 3.82%, 0.6%, 1.02%, and 4.8% for the four datasets, respectively.

Continuous Proteretic Hopfield Neural Network in Walsh-based Distributed Storage

A neural network is a series of algorithms that endeavors to recognize underlying relationships in a data set through a process that mimics the operations of a human brain. It may be used in many real-life applications such as speech and voice recognition, eCommerce, cybersecurity, etc. The activation function is one of the vital parts of neural networks, which affects the performance of neural network applications. Moreover, the choice of the activation function depends on the type of problem we want to solve. This paper presents a specific neuron activation function based on a property found in physical systems, “proteretic”. This prosperity is applied to the Walsh-based distributed memory. We show that using the proteretic function with the Walsh-based memory enhances the storage performance by making the network converge faster than the other neural network activation functions.

Reconfigurable nonlinear photonic activation function for photonic neural network based on non-volatile opto-resistive RAM switch

Photonic neural network has been sought as an alternative solution to surpass the efficiency and speed bottlenecks of electronic neural network. Despite that the integrated Mach–Zehnder Interferometer (MZI) mesh can perform vector-matrix multiplication in photonic neural network, a programmable in-situ nonlinear activation function has not been proposed to date, suppressing further advancement of photonic neural network. Here, we demonstrate an efficient in-situ nonlinear accelerator comprising a unique solution-processed two-dimensional (2D) MoS2 Opto-Resistive RAM Switch (ORS), which exhibits tunable nonlinear resistance switching that allow us to introduce nonlinearity to the photonic neuron which overcomes the linear voltage-power relationship of typical photonic components. Our reconfigurable scheme enables implementation of a wide variety of nonlinear responses. Furthermore, we confirm its feasibility and capability for MNIST handwritten digit recognition, achieving a high accuracy of 91.6%. Our accelerator constitutes a major step towards the realization of in-situ photonic neural network and pave the way for the integration of photonic integrated circuits (PIC).

A Modular Approximation Methodology for Efficient Fixed-Point Hardware Implementation of the Sigmoid Function

The sigmoid function is a widely used nonlinear activation function in neural networks. In this article, we present a modular approximation methodology for efficient fixed-point hardware implementation of the sigmoid function. Our design consists of three modules: piecewise linear (PWL) approximation as the initial solution, Taylor series approximation of the exponential function, and Newton–Raphson method-based approximation as the final solution. Its modularity enables the designer to flexibly choose the most appropriate approximation method for each module separately. Performance evaluation results indicate that our work strikes an appropriate balance among the objectives of approximation accuracy, hardware resource utilization, and performance.

A New Fault Diagnosis Method for Unbalanced Data Based on 1DCNN and L2-SVM

In general, the measured health condition data from rolling bearings usually exhibit imbalanced distribution. However, traditional intelligent fault diagnosis methods usually assume that the data categories are balanced. To improve the diagnosis accuracy of unbalanced datasets, a new fault diagnosis method for unbalanced data based on 1DCNN and L2-SVM is proposed in this paper. Firstly, to prevent the minority class samples from being heavily suppressed by the rectified linear unit (ReLU) activation function in the traditional convolutional neural network (CNN), ReLU is improved by linear and scaled exponential linear units (SELUs). Secondly, to solve the problem where the cross-entropy loss treats all input samples equally, it is replaced by the L2-support vector machine (L2-SVM) loss. Furthermore, a dynamic adjustment parameter is introduced to assign less misclassification cost to the majority of class samples. Finally, we add a new modulation factor that reduces the weight of more distinguishable samples to generate more focus on training indiscernible samples. The proposed method is carried out on two kinds of bearing datasets. The experimental results illustrate a significant improvement in recognition accuracy and the higher diagnosis performance of the model when dealing with unbalanced data compared with other intelligent methods.

Choice of Activation Function in Convolutional Neural Networks for Person Re-Identification in Video Surveillance Systems

Choice of Activation Function in Convolutional Neural Networks for Person Re-Identification in Video Surveillance Systems

A novel scaled-gamma-tanh (SGT) activation function in 3D CNN applied for MRI classification

Activation functions in the neural network are responsible for ‘firing’ the nodes in it. In a deep neural network they ‘activate’ the features to reduce feature redundancy and learn the complex pattern by adding non-linearity in the network to learn task-specific goals. In this paper, we propose a simple and interesting activation function based on the combination of scaled gamma correction and hyperbolic tangent function, which we call Scaled Gamma Tanh (SGT) activation. The proposed activation function is applied in two steps, first is the calculation of gamma version as y = f(x) = axα for x < 0 and y = f(x) = bxβ for x ≥ 0, second is obtaining the squashed value as z = tanh(y). The variables a and b are user-defined constant values whereas alpha and beta are channel-based learnable parameters. We analyzed the behavior of the proposed SGT activation function against other popular activation functions like ReLU, Leaky-ReLU, and tanh along with their role to confront vanishing/exploding gradient problems. For this, we implemented the SGT activation functions in a 3D Convolutional neural network (CNN) for the classification of magnetic resonance imaging (MRIs). More importantly to support our proposed idea we have presented a thorough analysis via histogram of inputs and outputs in activation layers along with weights/bias plot and t-SNE (t-Distributed Stochastic Neighbor Embedding) projection of fully connected layer for the trained CNN models. Our results in MRI classification show SGT outperforms standard ReLU and tanh activation in all cases i.e., final validation accuracy, final validation loss, test accuracy, Cohen’s kappa score, and Precision.

Sigmoid and Beyond: Algebraic Activation Functions for Artificial Neural Networks Based on Solutions of a Riccati Equation

Activation functions play a key role in neural networks, as they significantly affect the training process and the network’s performance. Based on the solution of a certain ordinary differential equation of the Riccati type, this work proposes an alternative generalized adaptive solution to the fixed sigmoid, which is called “generalized Riccati activation” (GRA). The proposed GRA function was employed on the output layer of an artificial neural network with a single hidden layer that consisted of eight neurons. The performance of the neural network was evaluated on a binary and a multiclass classification problem using different combinations of activation functions in the input/output layers. The results demonstrated that the swish/GRA combination yields higher accuracy than any other combination of activation functions. This benefit in terms of accuracy could be critical for certain domains, such as healthcare and smart grids, where AI-assisted decisions are becoming essential.

Optimization of Microchannels and Application of Basic Activation Functions of Deep Neural Network for Accuracy Analysis of Microfluidic Parameter Data.

The fabrication of microflow channels with high accuracy in terms of the optimization of the proposed designs, minimization of surface roughness, and flow control of microfluidic parameters is challenging when evaluating the performance of microfluidic systems. The use of conventional input devices, such as peristaltic pumps and digital pressure pumps, to evaluate the flow control of such parameters cannot confirm a wide range of data analysis with higher accuracy because of their operational drawbacks. In this study, we optimized the circular and rectangular-shaped microflow channels of a 100 μm microfluidic chip using a three-dimensional simulation tool, and analyzed concentration profiles of different regions of the microflow channels. Then, we applied a deep learning (DL) algorithm for the dense layers of the rectified linear unit (ReLU), Leaky ReLU, and Swish activation functions to train and test 1600 experimental and interpolation of data samples which obtained from the microfluidic chip. Moreover, using the same DL algorithm, we configured three models for each of these three functions by changing the internal middle layers of these models. As a result, we obtained a total of 9 average accuracy values of ReLU, Leaky ReLU, and Swish functions for a defined threshold value of using the trial-and-error method. We applied single-to-five-fold cross-validation technique of deep neural network to avoid overfitting and reduce noises from data-set to evaluate better average accuracy of data of microfluidic parameters.

Comparison of Different Convolutional Neural Network Activation Functions and Methods for Building Ensembles for Small to Midsize Medical Data Sets.

CNNs and other deep learners are now state-of-the-art in medical imaging research. However, the small sample size of many medical data sets dampens performance and results in overfitting. In some medical areas, it is simply too labor-intensive and expensive to amass images numbering in the hundreds of thousands. Building Deep CNN ensembles of pre-trained CNNs is one powerful method for overcoming this problem. Ensembles combine the outputs of multiple classifiers to improve performance. This method relies on the introduction of diversity, which can be introduced on many levels in the classification workflow. A recent ensembling method that has shown promise is to vary the activation functions in a set of CNNs or within different layers of a single CNN. This study aims to examine the performance of both methods using a large set of twenty activations functions, six of which are presented here for the first time: 2D Mexican ReLU, TanELU, MeLU + GaLU, Symmetric MeLU, Symmetric GaLU, and Flexible MeLU. The proposed method was tested on fifteen medical data sets representing various classification tasks. The best performing ensemble combined two well-known CNNs (VGG16 and ResNet50) whose standard ReLU activation layers were randomly replaced with another. Results demonstrate the superiority in performance of this approach.

On the Implementation of Fixed-Point Exponential Function for Machine Learning and Signal- Processing Accelerators

The natural exponential function is widely used in modeling many engineering and scientific systems. It is also an integral part of many neural network activation function such as sigmoid, tanh, ELU, RBF etc. Dedicated hardware accelerator and processors are designed for faster execution of such applications. Such accelerators can immensely benefit from an optimal implementation of exponential function. This can be achieved for most applications with the knowledge that the exponential function for a negative domain (R -) is more widely used than the positive domain (R+). This paper presents an optimized implementation of exponential function for variable precision fixed point negative input. The implementation presented here significantly reduces the number of multipliers and adders. This is further optimized using mixed world-length implementation for the series expansion. The reduction in area and power consumption is more than 30% and 50% respectively over previous equivalent method.

Hyperverlet: A Symplectic Hypersolver for Hamiltonian Systems

Hamiltonian systems represent an important class of dynamical systems such as pendulums, molecular dynamics, and cosmic systems. The choice of solvers is significant to the accuracy when simulating Hamiltonian systems, where symplectic solvers show great significance. Recent advances in neural network-based hypersolvers, though achieve competitive results, still lack the symplecity necessary for reliable simulations, especially over long time horizons. To alleviate this, we introduce Hyperverlet, a new hypersolver composing the traditional, symplectic velocity Verlet and symplectic neural network-based solvers. More specifically, we propose a parameterization of symplectic neural networks and prove that hyperbolic tangent is r-finite expanding the set of allowable activation functions for symplectic neural networks, improving the accuracy. Extensive experiments on a spring-mass and a pendulum system justify the design choices and suggest that Hyperverlet outperforms both traditional solvers and hypersolvers.

SpikeConverter: An Efficient Conversion Framework Zipping the Gap between Artificial Neural Networks and Spiking Neural Networks

Spiking Neural Networks (SNNs) have recently attracted enormous research interest since their event-driven and brain-inspired structure enables low-power computation. In image recognition tasks, the best results are achieved by SNN so far utilizing ANN-SNN conversion methods that replace activation functions in artificial neural networks~(ANNs) with integrate-and-fire neurons. Compared to source ANNs, converted SNNs usually suffer from accuracy loss and require a considerable number of time steps to achieve competitive accuracy. We find that the performance degradation of converted SNN stems from the fact that the information capacity of spike trains in transferred networks is smaller than that of activation values in source ANN, resulting in less information being passed during SNN inference. To better correlate ANN and SNN for better performance, we propose a conversion framework to mitigate the gap between the activation value of source ANN and the generated spike train of target SNN. The conversion framework originates from exploring an identical relation in the conversion and exploits temporal separation scheme and novel neuron model for the relation to hold. We demonstrate almost lossless ANN-SNN conversion using SpikeConverter for VGG-16, ResNet-20/34, and MobileNet-v2 SNNs on challenging datasets including CIFAR-10, CIFAR-100, and ImageNet. Our results also show that SpikeConverter achieves the abovementioned accuracy across different network architectures and datasets using 32X - 512X fewer inference time-steps than state-of-the-art ANN-SNN conversion methods.

An Efficient Asymmetric Nonlinear Activation Function for Deep Neural Networks

As a key step to endow the neural network with nonlinear factors, the activation function is crucial to the performance of the network. This paper proposes an Efficient Asymmetric Nonlinear Activation Function (EANAF) for deep neural networks. Compared with existing activation functions, the proposed EANAF requires less computational effort, and it is self-regularized, asymmetric and non-monotonic. These desired characteristics facilitate the outstanding performance of the proposed EANAF. To demonstrate the effectiveness of this function in the field of object detection, the proposed activation function is compared with several state-of-the-art activation functions on the typical backbone networks such as ResNet and DSPDarkNet. The experimental results demonstrate the superior performance of the proposed EANAF.

The Sigmoid Neural Network Activation Function and its Connections to Airy’s and the Nield-Kuznetsov Functions

Analysis and solution of Airy’s inhomogeneous equation, when its forcing function is the sigmoid neural network activation function, are provided in this work. Relationship between the Nield-Kuznetsov, the Scorer, the sigmoid, the polylogarithm and Airy’s functions are established. Solutions to initial and boundary value problems, when the sigmoid function is involved, are obtained. Computations were carried out using Wolfram Alpha.

Adaptive hybrid activation function for deep neural networks

The adaptive hybrid activation function (AHAF) is proposed that combines the properties of the rectifier units and the squashing functions. The proposed function can be used as a drop-in replacement for ReLU, SiL and Swish activations for deep neural networks and can evolve to one of such functions during the training. The effectiveness of the function was evaluated on the image classification task using the Fashion-MNIST and CIFAR-10 datasets. The evaluation shows that the neural networks with AHAF activations achieve better classification accuracy comparing to their base implementations that use ReLU and SiL. A double-stage parameter tuning process for training the neural networks with AHAF is proposed. The proposed approach is sufficiently simple from the implementation standpoint and provides high performance for the neural network training process.

Significance of activation functions in developing an online classifier for semiconductor defect detection

In anomaly detection problems for advanced semiconductor devices, non-visual defects occur frequently. Machine learning (ML) algorithms have the advantage of identifying such defects. However, in this real-world problem, data comes sequentially in a streaming fashion, thus, we may not have sufficient data to train an ML model in batch mode. In such a scenario, online ML models are useful to detect defects immediately since they work in a single-pass mode. Besides, when data is collected from more realistic non-stationary monitoring environments, online ML models with evolving architecture are more practical. Thus, evolving and online ML models are developed in this work to detect defects in technology computer-aided design (TCAD)-based digital twin model of advanced nano-scaled semiconductor devices such as a fin field-effect transistor (FinFET) and a gate-all-around field-effect transistor (GAA-FET). Activation functions (AFs) in deep neural networks (DNNs) and membership functions (MFs) in neuro-fuzzy systems (NFSs) play an important role in the performance of those ML models. This work focuses on analyzing the effects of various AFs/MFs in our developed online ML models while detecting defects in real-world nano-scaled semiconductor devices, where significant training samples are not available. From various semiconductor datasets having fewer samples, it has been observed that the proposed evolving neuro-fuzzy system (ENFS) with Leaky-ReLU MF performs better (improvement in the range of 1.9% to 30.8% considering overall classification accuracy) than the other DNN or ENFS-based online ML models. Having an evolving architecture and online learning mechanism, besides anomaly detection, the proposed model’s performance has also been evaluated for handling large data streams problems with concept drift. The performance of the proposed method has been compared with some recently developed baselines under the prequential test-then-train protocol. The classification rates of the proposed method has an improvement in the range of 1.1% to 65.9% than the existing methods. The code of this work has been made publicly available at https://github.com/MdFerdaus/LREC.