Activation Function Of Neural Network Research Articles

10 GHz Optoelectronic Circuit With Variable Sigmoid Shaped Activation Function for Photonic Neural Networks

10 GHz Optoelectronic Circuit With Variable Sigmoid Shaped Activation Function for Photonic Neural Networks

Simple yet effective adaptive activation functions for physics-informed neural networks

Physics-informed neural networks (PINNs) gained widespread advancements in solving differential equations, where the performance tightly hinges on the choice of activation functions that are inefficient when selected manually. To tackle this issue, we propose two straightforward yet powerful adaptive activation functions: a weighted average function that adjusts activation functions by directly manipulating their weights, and a L2-normalization function that compresses learnable parameters. These methods ensure a consistent sum of weights for each activation function, thereby enhancing optimization efficiency. We assess the performance of these approaches across a range of differential equation problems, encompassing Poisson equation, Wave equation, Burgers equation, Navier-Stokes equation, and linear/nonlinear solid mechanics problems. Through comparisons with exact solutions, we demonstrate significant improvements in convergence rate and solution accuracy. Our results underscore the efficacy of these techniques, providing a simple yet promising pathway for augmenting PINN performance across diverse applications. The source codes and software implementation are available at https://github.com/jzhange/AAF-for-PINNs.

Unification of popular artificial neural network activation functions

Unification of popular artificial neural network activation functions

A novel variable selection algorithm based on neural network for near-infrared spectral modeling

BackgroundPartial least squares (PLS) is a widely used technique for modeling spectral data. Researchers have developed numerous PLS-based variable selection algorithms to enhance model predictive ability and interpretability. In recent years, as neural network technology has advanced, these algorithms have been increasingly applied to spectral data modeling. However, current research on neural network modeling tends to prioritize network structure over variable selection. ResultsOur study introduces a neural network-based variable selection algorithm called VSNN (Variable Selection based on Neural Network). By iteratively eliminating unimportant variables using an exponentially decreasing function (EDF), the algorithm achieves the selection of variables in spectral data. VSNN can easily integrate different types of neural networks. In this study, we analyzed the impact of neural network types, activation functions, and variable importance vectors on algorithm performance. We tested the algorithm on four datasets: corn moisture, corn oil, tablets, and meat. The results indicate that VSNN significantly enhances the predictive ability of the model compared to partial least squares (PLS), neural networks (NN), and Joint Mutual Information Maximisation (JMIM). Specifically, non-linear activation functions markedly improve performance on non-linear meat datasets. Compared to PLS, the Root Mean Square Error of Prediction (RMSEP) values for the four datasets—corn moisture, corn oil, tablets, and meat—decreased from 0.0409, 0.0728, 3.97, and 3.2 to 0.002, 0.0236, 3.12, and 0.36, respectively, after applying the VSNN variable selection algorithm. SignificanceVSNN can serve as a versatile framework to enhance variable selection, modeling, and predictive performance by adapting neural network types and variable importance evaluation indicators. As machine learning technology advances, the strength of VSNN is poised to increase. This study highlights the potential of VSNN as an effective algorithmic framework for variable selection in spectroscopy applications.

Constructed encoded data based coded distributed DNN training for edge computing scenario

Deep learning in unmanned aerial vehicle (UAV) deployment encounters two problems: 1, One UAV may fail to store and execute large Deep Neural Network (DNN) model. 2, One UAV fails to accomplish real time services. One possible solution is the group of UAVs (nodes) collectively generate swarm intelligence in the form of edge computing scenario, namely in the distributed computing (DC) mode with clever arrangement, say Coded Distributed Computing (CDC). In CDC systems, the redundant computation introduced by linear coding can compensate stragglers. However, since linear property cannot pass the nonlinear activation function in Deep Neural Network (DNN) training, coding/decoding for CDC need to be applied layer by layer, which slows down the training. To avoid layer-by-layer coding/decoding, we propose a novel DNN training scheme based on constructing encoded data. This construction process lies before the training process (can be done before training without any impact on training efficiency). Based on both the original data and the newly constructed encoded data, the training phase can take advantage of the (n,k) property (Wait for the first k returned data) and hence improve the training speed. The training process does not require encoding/decoding operations, and hence significantly improves the training speed. Experimental results show that the training scheme based on constructed encoded data can achieve prediction accuracy approximating that of the centralized one and significantly reduce the latency compared to the layer-by-layer linear encoding and decoding scheme.

GEPAF: A non-monotonic generalized activation function in neural network for improving prediction with diverse data distributions characteristics

The world today has made prescriptive analytics that uses data-driven insights to guide future actions. The distribution of data, however, differs depending on the scenario, making it difficult to interpret and comprehend the data efficiently. Different neural network models are used to solve this, taking inspiration from the complex network architecture in the human brain. The activation function is crucial in introducing non-linearity to process data gradients effectively. Although popular activation functions such as ReLU, Sigmoid, Swish, and Tanh have advantages and disadvantages, they may struggle to adapt to diverse data characteristics. A generalized activation function named the Generalized Exponential Parametric Activation Function (GEPAF) is proposed to address this issue. This function consists of three parameters expressed: α, which stands for a differencing factor similar to the mean; σ, which stands for a variance to control distribution spread; and p, which is a power factor that improves flexibility; all these parameters are present in the exponent. When p=2, the activation function resembles a Gaussian function. Initially, this paper describes the mathematical derivation and validation of the properties of this function mathematically and graphically. After this, the GEPAF function is practically implemented in real-world supply chain datasets. One dataset features a small sample size but exhibits high variance, while the other shows significant variance with a moderate amount of data. An LSTM network processes the dataset for sales and profit prediction. The suggested function performs better than popular activation functions when a comparative analysis of the activation function is performed, showing at least 30% improvement in regression evaluation metrics and better loss decay characteristics.

Dynamics analysis and FPGA implementation of discrete memristive cellular neural network with heterogeneous activation functions

The activation function, as an important component of artificial neural networks, endows neural networks with rich dynamical phenomena by virtue of its nonlinear properties. However, especially for cellular neural networks (CNNs), the exploration of neural networks consisting of heterogeneous activation functions has not been sufficient. In this article, we introduce heterogeneous activation functions in discrete cellular neural network (DCNN) for the first time. In order to enhance the dynamics of the original DCNN, we propose our discrete memristive cellular neural network (DMCNN) based on a memristor with a sinusoidal function. Using various methods of dynamical characterization, such as Lyapunov exponential analysis, bifurcation diagrams and spectral entropy, the rich dynamical behaviour of the proposed model is comprehensively investigated, including hyperchaos, transient chaos, high spectral entropy and multiple types of coexisting attractors in the proposed model. Last but not least, the hardware platform of the proposed model is implemented by field programmable gate array (FPGA), and the dynamics of the proposed model is verified by a combination of software simulation and hardware experiments. The new exploration of DMCNN with heterogeneous activation functions in this article lays the foundation for further research into neural networks with complex dynamical behaviour.

Degree of Lp Approximation Using Activated Singular Integrals

In this article we present the Lp, p≥1, approximation properties of activated singular integral operators over the real line. We establish their approximation to the unit operator with rates. The kernels here come from neural network activation functions and we employ the related density functions. The derived inequalities use the high order Lp modulus of smoothness.

Comparative Performance of Different Convolutional Neural Network Activation Functions on Image Classification

Comparative Performance of Different Convolutional Neural Network Activation Functions on Image Classification

Adaptive Morphing Activation Function for Neural Networks

Adaptive Morphing Activation Function for Neural Networks

A reconfigurable multi-precision quantization-aware nonlinear activation function hardware module for DNNs

In recent years, the increasing variety of nonlinear activation functions (NAFs) in deep neural networks (DNNs) has led to higher computational demands. However, hardware implementation faces challenges such as lack of flexibility, high hardware cost, and limited accuracy. This paper proposes a highly flexible and low-cost hardware solution for implementing activation functions to overcome these issues. Based on the piecewise linear (PWL) approximation method, our method supports NAFs with different accuracy configurations through a customized implementation strategy to meet the requirements in different scenario applications. In this paper, the symmetry of the activation function is investigated, and incorporate curve translation preprocessing and data quantization to significantly reduce hardware storage costs. The modular hardware architecture proposed in this study supports NAFs of multiple accuracies, which is suitable for designing deep learning neural network accelerators in various scenarios, avoiding the need to design dedicated hardware circuits for the activation function layer and enhances circuit design efficiency. The proposed hardware architecture is validated on the Xilinx XC7Z010 development board. The experimental results show that the average absolute error (AAE) is reduced by about 35.6 % at a clock frequency of 312.5 MHz. Additionally, the accuracy loss of the model is maximized to −0.684 % after replacing the activation layer function of DNNs under the PyTorch framework.

Computational Investigation of the Potential and Limitations of Machine Learning with Neural Network Circuits Based on Synaptic Transistors.

Synaptic transistors have been proposed to implement neuron activation functions of neural networks (NNs). While promising to enable compact, fast, inexpensive, and energy-efficient dedicated NN circuits, they also have limitations compared to digital NNs (realized as codes for digital processors), including shape choices of the activation function using particular types of transistor implementation, and instabilities due to noise and other factors present in analog circuits. We present a computational study of the effects of these factors on NN performance and find that, while accuracy competitive with traditional NNs can be realized for many applications, there is high sensitivity to the instability in the shape of the activation function, suggesting that, when highly accurate NNs are required, high-precision circuitry should be developed beyond what has been reported for synaptic transistors to date.

Quantitative Uniform Approximation by Activated Singular Operators

In this article, we study the approximation properties of activated singular integral operators over the real line. We establish their convergence to the unit operator with rates. The kernels here derive from neural network activation functions and their corresponding density functions. The estimates are mostly sharp, and they are pointwise and uniform. The derived inequalities involve the higher order modulus of smoothness.

A Neural-Network-Based Watermarking Method Approximating JPEG Quantization.

We propose a neural-network-based watermarking method that introduces the quantized activation function that approximates the quantization of JPEG compression. Many neural-network-based watermarking methods have been proposed. Conventional methods have acquired robustness against various attacks by introducing an attack simulation layer between the embedding network and the extraction network. The quantization process of JPEG compression is replaced by the noise addition process in the attack layer of conventional methods. In this paper, we propose a quantized activation function that can simulate the JPEG quantization standard as it is in order to improve the robustness against the JPEG compression. Our quantized activation function consists of several hyperbolic tangent functions and is applied as an activation function for neural networks. Our network was introduced in the attack layer of ReDMark proposed by Ahmadi et al. to compare it with their method. That is, the embedding and extraction networks had the same structure. We compared the usual JPEG compressed images and the images applying the quantized activation function. The results showed that a network with quantized activation functions can approximate JPEG compression with high accuracy. We also compared the bit error rate (BER) of estimated watermarks generated by our network with those generated by ReDMark. We found that our network was able to produce estimated watermarks with lower BERs than those of ReDMark. Therefore, our network outperformed the conventional method with respect to image quality and BER.

Comparative study of integer-order and fractional-order artificial neural networks: Application for mathematical function generation

This paper investigates the impact of fractional derivatives on the activation functions of an artificial neural network (ANN). Based on the results and analysis, a three-layer backpropagation neural network model with fractional and integer derivatives employed in the activation function and fractional gradient descent method backpropagation learning algorithm has been proposed. Specifically, three perceptrons have been proposed based on fractional and integer derivatives applied to activation functions and learning algorithms. They are fractional derivative activation function (FDAF) perceptron, integer derivative activation function (IDAF)-fractional derivative learning algorithm (FDLA) perceptron, and fractional derivative learning algorithm (FDLA) perceptron. The Riemann-Liouville (RL) fractional derivative, Grunwald-Letnikov (GL) derivative, Caputo-Fabrizio (CF) fractional derivative, Caputo (C) fractional derivative, and Atangana-Baleanu (AB) fractional derivative have been employed. The impact of these derivatives’ fractional-order (FO) is investigated in a wide range from 0.1−0.9 on testing mean square error (MSE) and time required to train the perceptron. FO-based and IO-based perceptrons are compared with the help of performance metrics such as testing MSE and the time required to train the models. The training and testing simulation results illustrate that Caputo-Fabrizio derivative-based perceptrons outperform other fractional derivatives. Also, the performance of the FO-based perceptron is better in terms of the least MSE.

An Improved Activation Function in Convolution Neural Network to Estimate the Hazardous Air Pollutant Based on Images

An Improved Activation Function in Convolution Neural Network to Estimate the Hazardous Air Pollutant Based on Images

Activation Functions for Neural Networks: Application and Performance-based Comparison

Past decade has seen explosive growth of Deep Learning (DL) algorithms based on Artificial Neural Networks (ANNs) and its applications in vast emerging domains to solve real world complex problems. The DL architecture uses Activation Functions (AFs), to perform the task of finding relationship between the input feature and the output. Essential building blocks of any ANN are AFs which bring the required non-linearity of the output in the Output layer of network. Layers of ANNs are combinations of linear and nonlinear AFs. Most extensively used AFs are Sigmoid, Hyperbolic Tangent (Tanh), Rectified Linear Unit (ReLU) etc to name a few. Choosing an AF for a particular AF depends on various factors such as Nature of Application, Design of ANN, Optimizers used in the network, Complexity of Data etc. This paper presents a survey on most widely used AFs along with the important consideration while selecting an AF on a specific problem domain. A broad guideline on selecting an AF based on the literature survey has been presented to help researchers in employing suitable AF in their problem domain.

Anti-derivatives approximator for enhancing physics-informed neural networks

This study presents a novel strategy for constructing an approximator for arbitrary univariate functions. The proposed approximation utilizes the anti-derivatives of a Fourier series expansion for the presumed piecewise function, resulting in a remarkable feature that enables the simultaneous approximation of an arbitrary function and its anti-derivatives. These anti-derivatives can be employed to discover solution curves for systems of ordinary differential equations based on an optimization scheme, even in the presence of chaotic dynamics. Additionally, the anti-derivatives approximator is extended as an adaptive activation function for physics-informed neural networks, leveraging the high-order differentiability of the anti-derivatives. Systematic experiments have demonstrated the outstanding merits of the proposed anti-derivatives-based approximator, including its ability to construct regression models for volatile data and their anti-derivatives, solve differential equations, and enhance the capabilities of physics-informed neural networks.

An Innovative Deep Learning Approach for Image Semantic and Instance Segmentation

In this study, we propose a segmentation model based on convolutional neural networks (CNNs) to address image segmentation challenges in computer vision. Prior to designing the model, the activation function and other modules of the convolutional neural network were optimized to meet specific requirements. The segmentation task was transformed into binary classification problem to simplify network calculations and improve efficiency. Additionally, the model utilized a mask map obtained from the semantic segmentation model to aid in instance segmentation. Class activation technology was introduced to extract feature mapping maps. The corresponding thermal maps were obtained to achieve target instance segmentation. To further validate the effectiveness of the segmentation model, simulation experiments were conducted on semantic segmentation and instance segmentation respectively. The results show that the accuracy of the basic semantic segmentation model reached 87.58%, while the average accuracy of the entire class of the optimized instance segmentation model reached 97.9%. Therefore, the research and design of image segmentation models demonstrate high accuracy and good robustness.

ReActHE: A homomorphic encryption friendly deep neural network for privacy-preserving biomedical prediction

The growing distribution of deep learning models to individuals’ devices on sensitive healthcare data introduces challenging privacy and security problems when computation is being operated on an untrusted server. Homomorphic encryption (HE) is one of the appropriate cryptographic techniques to provide secure machine learning computation by directly computing over encrypted data, so that allows the data owner and model owner to outsource processing of sensitive information to an untrusted server without leaking any information about the data. However, most current HE schemes only support limited arithmetic operations, which significantly hinder their applications to implement a secure deep learning algorithm, especially on the nonlinear activation function of a deep neural network. In this paper, we develop a novel HE-friendly deep neural network, named REsidue ACTivation HE (ReActHE), to implement a precise and privacy-preserving algorithm with a non-approximating HE scheme on the activation function. We consider a residue activation strategy with a scaled power activation function in a deep neural network for HE-friendly nonlinear activation. Moreover, we propose a residue activation network structure to constrain the latent space in the training process to alleviate the optimization difficulty. We comprehensively evaluate the proposed ReActHE method using various biomedical datasets and widely-used image datasets. Our results demonstrate that ReActHE outperforms other alternative solutions to secure machine learning with HE and achieves low approximation errors in classification and regression tasks.