Activation Function Of Neural Network Research Articles

Comparison of Different Convolutional Neural Network Activation Functions and Methods for Building Ensembles for Small to Midsize Medical Data Sets.

CNNs and other deep learners are now state-of-the-art in medical imaging research. However, the small sample size of many medical data sets dampens performance and results in overfitting. In some medical areas, it is simply too labor-intensive and expensive to amass images numbering in the hundreds of thousands. Building Deep CNN ensembles of pre-trained CNNs is one powerful method for overcoming this problem. Ensembles combine the outputs of multiple classifiers to improve performance. This method relies on the introduction of diversity, which can be introduced on many levels in the classification workflow. A recent ensembling method that has shown promise is to vary the activation functions in a set of CNNs or within different layers of a single CNN. This study aims to examine the performance of both methods using a large set of twenty activations functions, six of which are presented here for the first time: 2D Mexican ReLU, TanELU, MeLU + GaLU, Symmetric MeLU, Symmetric GaLU, and Flexible MeLU. The proposed method was tested on fifteen medical data sets representing various classification tasks. The best performing ensemble combined two well-known CNNs (VGG16 and ResNet50) whose standard ReLU activation layers were randomly replaced with another. Results demonstrate the superiority in performance of this approach.

On the Implementation of Fixed-Point Exponential Function for Machine Learning and Signal- Processing Accelerators

The natural exponential function is widely used in modeling many engineering and scientific systems. It is also an integral part of many neural network activation function such as sigmoid, tanh, ELU, RBF etc. Dedicated hardware accelerator and processors are designed for faster execution of such applications. Such accelerators can immensely benefit from an optimal implementation of exponential function. This can be achieved for most applications with the knowledge that the exponential function for a negative domain (R -) is more widely used than the positive domain (R+). This paper presents an optimized implementation of exponential function for variable precision fixed point negative input. The implementation presented here significantly reduces the number of multipliers and adders. This is further optimized using mixed world-length implementation for the series expansion. The reduction in area and power consumption is more than 30% and 50% respectively over previous equivalent method.

Hyperverlet: A Symplectic Hypersolver for Hamiltonian Systems

Hamiltonian systems represent an important class of dynamical systems such as pendulums, molecular dynamics, and cosmic systems. The choice of solvers is significant to the accuracy when simulating Hamiltonian systems, where symplectic solvers show great significance. Recent advances in neural network-based hypersolvers, though achieve competitive results, still lack the symplecity necessary for reliable simulations, especially over long time horizons. To alleviate this, we introduce Hyperverlet, a new hypersolver composing the traditional, symplectic velocity Verlet and symplectic neural network-based solvers. More specifically, we propose a parameterization of symplectic neural networks and prove that hyperbolic tangent is r-finite expanding the set of allowable activation functions for symplectic neural networks, improving the accuracy. Extensive experiments on a spring-mass and a pendulum system justify the design choices and suggest that Hyperverlet outperforms both traditional solvers and hypersolvers.

SpikeConverter: An Efficient Conversion Framework Zipping the Gap between Artificial Neural Networks and Spiking Neural Networks

Spiking Neural Networks (SNNs) have recently attracted enormous research interest since their event-driven and brain-inspired structure enables low-power computation. In image recognition tasks, the best results are achieved by SNN so far utilizing ANN-SNN conversion methods that replace activation functions in artificial neural networks~(ANNs) with integrate-and-fire neurons. Compared to source ANNs, converted SNNs usually suffer from accuracy loss and require a considerable number of time steps to achieve competitive accuracy. We find that the performance degradation of converted SNN stems from the fact that the information capacity of spike trains in transferred networks is smaller than that of activation values in source ANN, resulting in less information being passed during SNN inference. To better correlate ANN and SNN for better performance, we propose a conversion framework to mitigate the gap between the activation value of source ANN and the generated spike train of target SNN. The conversion framework originates from exploring an identical relation in the conversion and exploits temporal separation scheme and novel neuron model for the relation to hold. We demonstrate almost lossless ANN-SNN conversion using SpikeConverter for VGG-16, ResNet-20/34, and MobileNet-v2 SNNs on challenging datasets including CIFAR-10, CIFAR-100, and ImageNet. Our results also show that SpikeConverter achieves the abovementioned accuracy across different network architectures and datasets using 32X - 512X fewer inference time-steps than state-of-the-art ANN-SNN conversion methods.

An Efficient Asymmetric Nonlinear Activation Function for Deep Neural Networks

As a key step to endow the neural network with nonlinear factors, the activation function is crucial to the performance of the network. This paper proposes an Efficient Asymmetric Nonlinear Activation Function (EANAF) for deep neural networks. Compared with existing activation functions, the proposed EANAF requires less computational effort, and it is self-regularized, asymmetric and non-monotonic. These desired characteristics facilitate the outstanding performance of the proposed EANAF. To demonstrate the effectiveness of this function in the field of object detection, the proposed activation function is compared with several state-of-the-art activation functions on the typical backbone networks such as ResNet and DSPDarkNet. The experimental results demonstrate the superior performance of the proposed EANAF.

The Sigmoid Neural Network Activation Function and its Connections to Airy’s and the Nield-Kuznetsov Functions

Analysis and solution of Airy’s inhomogeneous equation, when its forcing function is the sigmoid neural network activation function, are provided in this work. Relationship between the Nield-Kuznetsov, the Scorer, the sigmoid, the polylogarithm and Airy’s functions are established. Solutions to initial and boundary value problems, when the sigmoid function is involved, are obtained. Computations were carried out using Wolfram Alpha.

Adaptive hybrid activation function for deep neural networks

The adaptive hybrid activation function (AHAF) is proposed that combines the properties of the rectifier units and the squashing functions. The proposed function can be used as a drop-in replacement for ReLU, SiL and Swish activations for deep neural networks and can evolve to one of such functions during the training. The effectiveness of the function was evaluated on the image classification task using the Fashion-MNIST and CIFAR-10 datasets. The evaluation shows that the neural networks with AHAF activations achieve better classification accuracy comparing to their base implementations that use ReLU and SiL. A double-stage parameter tuning process for training the neural networks with AHAF is proposed. The proposed approach is sufficiently simple from the implementation standpoint and provides high performance for the neural network training process.

Significance of activation functions in developing an online classifier for semiconductor defect detection

In anomaly detection problems for advanced semiconductor devices, non-visual defects occur frequently. Machine learning (ML) algorithms have the advantage of identifying such defects. However, in this real-world problem, data comes sequentially in a streaming fashion, thus, we may not have sufficient data to train an ML model in batch mode. In such a scenario, online ML models are useful to detect defects immediately since they work in a single-pass mode. Besides, when data is collected from more realistic non-stationary monitoring environments, online ML models with evolving architecture are more practical. Thus, evolving and online ML models are developed in this work to detect defects in technology computer-aided design (TCAD)-based digital twin model of advanced nano-scaled semiconductor devices such as a fin field-effect transistor (FinFET) and a gate-all-around field-effect transistor (GAA-FET). Activation functions (AFs) in deep neural networks (DNNs) and membership functions (MFs) in neuro-fuzzy systems (NFSs) play an important role in the performance of those ML models. This work focuses on analyzing the effects of various AFs/MFs in our developed online ML models while detecting defects in real-world nano-scaled semiconductor devices, where significant training samples are not available. From various semiconductor datasets having fewer samples, it has been observed that the proposed evolving neuro-fuzzy system (ENFS) with Leaky-ReLU MF performs better (improvement in the range of 1.9% to 30.8% considering overall classification accuracy) than the other DNN or ENFS-based online ML models. Having an evolving architecture and online learning mechanism, besides anomaly detection, the proposed model’s performance has also been evaluated for handling large data streams problems with concept drift. The performance of the proposed method has been compared with some recently developed baselines under the prequential test-then-train protocol. The classification rates of the proposed method has an improvement in the range of 1.1% to 65.9% than the existing methods. The code of this work has been made publicly available at https://github.com/MdFerdaus/LREC.

The Limits of SEMA on Distinguishing Similar Activation Functions of Embedded Deep Neural Networks

Artificial intelligence (AI) is progressing rapidly, and in this trend, edge AI has been researched intensively. However, much less work has been performed around the security of edge AI. Machine learning models are a mass of intellectual property, and an optimized network is very valuable. Trained machine learning models need to be black boxes as well because they may give away information about the training data to the outside world. As selecting the appropriate activation functions to enable fast training of accurate deep neural networks is an active area of research, it is important to conceal the information of the activation functions used in a neural network architecture as well. There has been research on the use of physical attacks such as the side-channel attack (SCA) in areas other than cryptography. The SCA is highly effective against edge artificial intelligence due to its property of the device computing close to the user. We studied a previously proposed method to retrieve the activation functions of a black box neural network implemented on an edge device by using simple electromagnetic analysis (SEMA) and improved the signal processing procedure for further noisy measurements. The SEMA attack identifies activation functions by directly observing distinctive electromagnetic (EM) traces that correspond to the operations in the activation function. This method requires few executions and inputs and also has little implementation dependency on the activation functions. We distinguished eight similar activation functions with EM measurements and examined the versatility and limits of this attack. In this work, the machine learning architecture is a multilayer perceptron, evaluated on an Arduino Uno.

Deep Neural Network with Adaptive Parametric Rectified Linear Units and its Fast Learning

The adaptive parametric rectified linear unit (AdPReLU) as an activation function of the deep neural network is proposed in the article. The main benefit of the proposed system is adjusted activation function whose parameters are tuning parallel with synaptic weights in online mode. The algorithm of the simultaneous learning of all neurons parameters with AdPReLU and the modified backpropagation procedure based on this algorithm is introduced. The approach under consideration permits to reduce volume of the training data set and increase tuning speed of the DNN with AdPReLU. The proposed approach could be applied in the deep convolutional neural networks (CNN) in conditions of the small value of training data sets and additional requirements for system performance. The main feature of DNN under consideration is possibility to tune not only synaptic weights but the parameters of activation function too. The effectiveness of this approach is proved by experimental modeling.

Programmable low-threshold optical nonlinear activation functions for photonic neural networks.

We experimentally demonstrate two types of programmable, low-threshold, optically controlled nonlinear activation functions, which are challenging to realize in photonic neural networks (PNNs). These devices rely on on-chip integrated Ge-Si photoelectric detectors and silicon electro-optical switches, and they generate rectified linear unit (ReLU) or sigmoid functions with arbitrary slopes without additional electrical processing. Both devices function at an extremely low threshold of 0.2 mW. The embedding of these nonlinear activation functions into convolutional neural networks facilitates the attainment of high inference accuracies of up to 95% when applied to Modified National Institute of Standards and Technology (MNIST) handwritten digit-classification tasks. The devices are suitable for low-power PNNs with an arbitrary number of propagation layers in photonic-computing chips.

Comparison of the Representational Power of Random Forests, Binary Decision Diagrams, and Neural Networks.

In this letter, we compare the representational power of random forests, binary decision diagrams (BDDs), and neural networks in terms of the number of nodes. We assume that an axis-aligned function on a single variable is assigned to each edge in random forests and BDDs, and the activation functions of neural networks are sigmoid, rectified linear unit, or similar functions. Based on existing studies, we show that for any random forest, there exists an equivalent depth-3 neural network with a linear number of nodes. We also show that for any BDD with balanced width, there exists an equivalent shallow depth neural network with a polynomial number of nodes. These results suggest that even shallow neural networks have the same or higher representation power than deep random forests and deep BDDs. We also show that in some cases, an exponential number of nodes are required to express a given random forest by a random forest with a much fewer number of trees, which suggests that many trees are required for random forests to represent some specific knowledge efficiently.

Extend mixed models to multilayer neural networks for genomic prediction including intermediate omics data.

With the growing amount and diversity of intermediate omics data complementary to genomics (e.g. DNA methylation, gene expression, and protein abundance), there is a need to develop methods to incorporate intermediate omics data into conventional genomic evaluation. The omics data help decode the multiple layers of regulation from genotypes to phenotypes, thus forms a connected multilayer network naturally. We developed a new method named NN-MM to model the multiple layers of regulation from genotypes to intermediate omics features, then to phenotypes, by extending conventional linear mixed models ("MM") to multilayer artificial neural networks ("NN"). NN-MM incorporates intermediate omics features by adding middle layers between genotypes and phenotypes. Linear mixed models (e.g. pedigree-based BLUP, GBLUP, Bayesian Alphabet, single-step GBLUP, or single-step Bayesian Alphabet) can be used to sample marker effects or genetic values on intermediate omics features, and activation functions in neural networks are used to capture the nonlinear relationships between intermediate omics features and phenotypes. NN-MM had significantly better prediction performance than the recently proposed single-step approach for genomic prediction with intermediate omics data. Compared to the single-step approach, NN-MM can handle various patterns of missing omics measures and allows nonlinear relationships between intermediate omics features and phenotypes. NN-MM has been implemented in an open-source package called "JWAS".

A gradient-type noise-tolerant finite-time neural network for convex optimization

As one of powerful methods for solving optimization problems, the gradient-type neural network has attracted the attention of many scholars. Up to now, there are few studies on finite-time convergence and the ability of noise tolerance of gradient-type networks. Motivated by the superior performance of various activation functions and advantages of gradient-type networks for solving constrained optimization, this paper aims to introduce a suitable activation function in a gradient-type neural network to solve convex optimization. It is shown that the state solution of the network converges to an optimal solution of the original convex optimization and the convergence time is finite under mild conditions. The capability of the network to suppress bounded noises is also analyzed theoretically. Superior to existing models for constrained optimization, the resulting network model not only has fast convergence but also can tolerate additive noises. Some numerical results are reported, including the results for smooth and nonsmooth problems, to show that the presented network is effective for convex constrained problems in the absence and presence of additive noises.

Rational activation functions in neural network with uniform norm based loss function and its application in classification

Rational activation functions in neural network with uniform norm based loss function and its application in classification

Improved Learning by Using a Modified Activation Function of a Convolutional Neural Network in Image Classification

Improved Learning by Using a Modified Activation Function of a Convolutional Neural Network in Image Classification

Image Verification and Emotion Detection using Effective Modelling Techniques

The feelings expressed on the face reflect the manner of thinking and provide useful insights of happenings inside the brain. Face Detection enables us to identify a face. Recognizing the facial expressions for different emotions is to familiarize the machine with human like capacity to perceive and identify human feelings, which involves classifying the given input images of the face into one of the seven classes which is achieved by building a multi class classifier. The proposed methodology is based on convolutional neural organizations and works on 48x48 pixel-based grayscale images. The proposed model is tested on various images and gives the best accuracy when compared with existing functionalities. It detects faces in images, recognizes them and identifies emotions and shows improved performance because of data augmentation. The model is experimented with varying depths and pooling layers. The best results are obtained sequential model of six layers of Convolutional Neural Network and softmax activation function applied to last layer. The approach works for real time data taken from videos or photos.

An ITO–graphene heterojunction integrated absorption modulator on Si-photonics for neuromorphic nonlinear activation

The high demand for machine intelligence of doubling every three months is driving novel hardware solutions beyond charging of electrical wires, given a resurrection to application specific integrated circuit (ASIC)-based accelerators. These innovations include photonic-based ASICs (P-ASICs) due to prospects of performing optical linear (and also nonlinear) operations, such as multiply–accumulate for vector matrix multiplications or convolutions, without iterative architectures. Such photonic linear algebra enables picosecond delay when photonic integrated circuits are utilized via “on-the-fly” mathematics. However, the neuron’s full function includes providing a nonlinear activation function, known as thresholding, to enable decision making on inferred data. Many P-ASIC solutions perform this nonlinearity in the electronic domain, which brings challenges in terms of data throughput and delay, thus breaking the optical link and introducing increased system complexity via domain crossings. This work follows the notion of utilizing enhanced light–matter interactions to provide efficient, compact, and engineerable electro-optic neuron nonlinearity. Here, we introduce and demonstrate a novel electro-optic device to engineer the shape of this optical nonlinearity to resemble a leaky rectifying linear unit—the most commonly used nonlinear activation function in neural networks. We combine the counter-directional transfer functions from heterostructures made out of two electro-optic materials to design a diode-like nonlinear response of the device. Integrating this nonlinearity into a photonic neural network, we show how the electrostatics of this thresholder’s gating junction improves machine learning inference accuracy and the energy efficiency of the neural network.

A Mixed-Integer linear programming based training and feature selection method for artificial neural networks using piece-wise linear approximations

Artificial Neural Networks (ANNs) may suffer from suboptimal training and test performance related issues not only because of the presence of high number of features with low statistical contributions but also due to their non-convex nature. This study develops piecewise-linear formulations for the efficient approximation of the non-convex activation and objective functions in artificial neural networks for optimal, global and simultaneous training and feature selection in regression problems. Such formulations include binary variables to account for the existence of the features and piecewise-linear approximations, which in turn, after one exact linearization step, calls for solving a mixed-integer linear programming problem with a global optimum guarantee because of convexity. Suggested formulation is implemented on two industrial case studies. Results show that efficient approximations are obtained through the usage of the method with only a few number of breakpoints. Significant feature space reduction is observed bringing about notable improvement in test accuracy.

Resource efficient activation functions for neural network accelerators

Implementations of machine learning models in resource-limited embedded systems are becoming highly desired. This has led to a need for resource-efficient building blocks for computing the mathematical operations required for neural network training and inferencing. Efficient activation functions for low-end hardware devices with limited hardware capabilities are important. In this work, we present a method for generating symmetric and asymmetric activation functions for deep and convolution neural networks. Furthermore, we propose a solution that simultaneously computes a symmetric activation function with an integrated scaling functionality for Long Short Term Memory (LSTM) models. This effectively eliminates two of the three element-wise multipliers in an LSTM cell. Also, this built-in scaling requires no additional computation time because it is integrated within the computation of the symmetric non-linear mapping. This approach replaces the need to compute several Tanh activation functions and element-wise multipliers separately. A resource-efficient approximate multiplier is also proposed to eliminate the third element-wise multiplier and potentially replace all the resource-hungry multipliers. The digital implementation of the proposed method is highly amenable to parallelization and is extremely resource-efficient. We record an area-saving on field-programmable gate arrays with different precision. Our proposal’s formulaic equivalents are also computationally fast on CPU-based engines. On an embedded ARM processor, our method achieves a speedup of at least 4.37× for the proposed functions. We show that LSTMs with our method can achieve up to 3.5× resource footprint saving when compared to the hard activation implementation. We demonstrate that our method achieves competitive results with negligible loss of performance.