Sigmoid Activation Function Research Articles

Speeding up quantum perceptron via shortcuts to adiabaticity

The quantum perceptron is a fundamental building block for quantum machine learning. This is a multidisciplinary field that incorporates abilities of quantum computing, such as state superposition and entanglement, to classical machine learning schemes. Motivated by the techniques of shortcuts to adiabaticity, we propose a speed-up quantum perceptron where a control field on the perceptron is inversely engineered leading to a rapid nonlinear response with a sigmoid activation function. This results in faster overall perceptron performance compared to quasi-adiabatic protocols, as well as in enhanced robustness against imperfections in the controls.

Global Lagrange stability analysis of retarded SICNNs

Abstract The stability in the Lagrange sense for cellular neural networks (CNNs) has proven to be one of the effective tools to study multi-stable dynamics of the neural networks. In this article, rather than studying the existence and Lyapunov stability of an equilibrium point we investigate multi-stable dynamics of shunting inhibitory cellular neural networks (SICNNs) with time-varying delays and coefficients. This is the first paper that addresses the Lagrange stability for SICNNs. By constructing proper Lyapunov functions and using inequality techniques, we analyze three different types of activation functions, namely, bounded, sigmoid and Lipschitz-like type activation functions. New delay-dependent sufficient criteria are derived to ensure the global Lagrange stability for SICNNs. Furthermore, globally exponentially attractive sets are given for the different activation functions. Finally, an illustrating example with numerical simulations is given to support the theoretical results.

RSigELU: A nonlinear activation function for deep neural networks

In deep learning models, the inputs to the network are processed using activation functions to generate the output corresponding to these inputs. Deep learning models are of particular importance in analyzing big data with numerous parameters and forecasting and are useful for image processing, natural language processing, object recognition, and financial forecasting. Sigmoid and tangent activation functions, which are traditional activation functions, are widely used in deep learning models. However, the sigmoid and tangent activation functions face the vanishing gradient problem. In order to overcome this problem, the ReLU activation function and its derivatives were proposed in the literature. However, there is a negative region problem in these activation functions. In this study, novel RSigELU activation functions, such as single-parameter RSigELU (RSigELUS) and double-parameter (RSigELUD), which are a combination of ReLU, sigmoid, and ELU activation functions, were proposed. The proposed RSigELUS and RSigELUD activation functions can overcome the vanishing gradient and negative region problems and can be effective in the positive, negative, and linear activation regions. Performance evaluation of the proposed RSigELU activation functions was performed on the MNIST, Fashion MNIST, CIFAR-10, and IMDb Movie benchmark datasets. Experimental evaluations showed that the proposed activation functions perform better than other activation functions.

Prediction of Band Gap Energy of Doped Graphitic Carbon Nitride Using Genetic Algorithm-Based Support Vector Regression and Extreme Learning Machine

Graphitic carbon nitride is a stable and distinct two dimensional carbon-based polymeric semiconductor with remarkable potentials in organic pollutants degradation, chemical sensors, the reduction of CO2, water splitting and other photocatalytic applications. Efficient utilization of this material is hampered by the nature of its band gap and the rapid recombination of electron-hole pairs. Heteroatom incorporation due to doping alters the symmetry of the semiconductor and has been among the adopted strategies to tailor the band gap for enhancing the visible-light harvesting capacity of the material. Electron modulation and enhancement of reaction active sites due to doping as evident from the change in specific surface area of doped graphitic carbon nitride is employed in this work for modeling the associated band gap using hybrid genetic algorithm-based support vector regression (GSVR) and extreme learning machine (ELM). The developed GSVR performs better than ELM-SINE (with sine activation function), ELM-TRANBAS (with triangular basis activation function) and ELM-SIG (with sigmoid activation function) model with performance enhancement of 69.92%, 73.59% and 73.67%, respectively, on the basis of root mean square error as a measure of performance. The four developed models are also compared using correlation coefficient and mean absolute error while the developed GSVR demonstrates a high degree of precision and robustness. The excellent generalization and predictive strength of the developed models would ultimately facilitate quick determination of the band gap of doped graphitic carbon nitride and enhance its visible-light harvesting capacity for various photocatalytic applications.

Comment toxicity detection via a multichannel convolutional bidirectional gated recurrent unit

Recently, toxicity identification has become the most serious problem in online communities and social networking sites. Therefore, an automatic toxic identification system needs to be developed for preventing and limiting users from these online environments. In this paper, we present a multichannel convolutional bidirectional gated recurrent unit (MCBiGRU) for detecting toxic comments in a multilabel environment. The proposed model generates word vectors using pre-trained word embeddings. Moreover, this hybrid model extracts local features with many filters and different kernel sizes to model input words with long term dependency. We then integrate multiple channels with a fully connected layer, normalization layer, and an output layer with a sigmoid activation function for predicting multilabel categories. The experimental results indicate that the proposed MCBiGRU model outperforms in terms of multilabel metrics.

Synthesis of ranking functions via DNN

We propose a new approach to synthesis of non-polynomial ranking functions for loops via deep neural network(DNN). Firstly, we construct a ranking function template by DNN structure. And then the coefficients of the template can be learned by the train-set we construct to get a candidate ranking function, which is transcendental and non-polynomial because of the existence of sigmoid activation function in the neural network. Finally, the candidate ranking function will be verified to show if it is a real ranking function. Most of the existing methods focus on linear or polynomial ranking functions, and are limited to verification tools, while in this paper we make progress regarding the synthesis of non-polynomial ranking functions and new verification method for candidate ranking functions. The experimental results show us that for some of loops from other work, we can find their ranking functions efficiently. Moreover, for some loops having multi-phase ranking functions obtained by existing methods, our method can directly detect their global ranking functions. Especially, our method can also detect the global ranking functions for some loops with transcendental terms, which cannot be dealt with by existing methods.

Asymptotic Expansion for Neural Network Operators of the Kantorovich Type and High Order of Approximation

In this paper, we study the rate of pointwise approximation for the neural network operators of the Kantorovich type. This result is obtained proving a certain asymptotic expansion for the above operators and then by establishing a Voronovskaja type formula. A central role in the above resuts is played by the truncated algebraic moments of the density functions generated by suitable sigmoidal functions. Furthermore, to improve the rate of convergence, we consider finite linear combinations of the above neural network type operators, and also in the latter case, we obtain a Voronovskaja type theorem. Finally, concrete examples of sigmoidal activation functions have been deeply discussed, together with the case of rectified linear unit (ReLu) activation function, very used in connection with deep neural networks.

CORR Synthesis: When Should the Orthopaedic Surgeon Use Artificial Intelligence, Machine Learning, and Deep Learning?

CORR Synthesis: When Should the Orthopaedic Surgeon Use Artificial Intelligence, Machine Learning, and Deep Learning?

Using independently recurrent networks for reinforcement learning based unsupervised video summarization

Sigmoid and hyperbolic activation functions in long short-term memory (LSTM) and gated recurrent unit (GRU) based models used in recent studies on video summarization, may cause gradient decay over layers. Moreover, interpreting and developing network models are difficult because of entanglement of neurons on recurrent neural network (RNN). To solve these issues, in this study, we propose a method that uses deep reinforcement learning together with independently recurrent neural networks (IndRNN) for unsupervised video summarization. In this method, Leaky Rectified Linear Unit (Leaky ReLU) is used as an activation function to deal with decaying gradient and dying neuron problems. The model, which does not rely on any labels or user interaction, is designed with a reward function that jointly accounts for uniformity, diversity and representativeness of generated summaries. In this way, our model can create summaries as uniform as possible, has more layers and can be trained with more steps without having any problem related to gradients. Based on the experiments conducted on two benchmark datasets, we observe that, compared to the state-of-the-art methods on video summarization task, better summarization performance can be obtained.

Intelligent adaptive optimal control using incremental model-based global dual heuristic programming subject to partial observability

The scarcity of information regarding dynamics and full-state feedback increases the demand for a model-free control technique that can cope with partial observability. To deal with the absence of prior knowledge of system dynamics and perfect measurements, this paper develops a novel intelligent control scheme by combining global dual heuristic programming with an incremental model-based identifier. An augmented system consisting of the unknown nonlinear plant and unknown varying references is identified online using a locally linear regression technique. The actor–critic is implemented using artificial neural networks, and the actuator saturation constraint is addressed by exploiting a symmetrical sigmoid activation function in the output layer of the actor network. Numerical experiments are conducted by applying the proposed method to online adaptive optimal control tasks of an aerospace system. The results reveal that the developed method can deal with partial observability with performance comparable to the full-state feedback control, while outperforming the global model-based method in stability and adaptability.

A Controlled Accuracy-Based Recursive Algorithm for Approximation of Sigmoid Activation

The sigmoid activation function is widely used in the development of neural networks. It is implemented in hardware development using an approximation scheme. In this paper, a piecewise linear (PWL) approximation scheme is developed as an accuracy controlled recursive algorithm. It uses a maximum allowable error as an input parameter and the least square method for finding the PWL function. The algorithm’s accuracy is calculated with an absolute average and maximum error and compared to the state of the art. The approximation scheme’s hardware implementation complexity is compared in terms of the number of PWL functions required. Results show that the proposed algorithm requires comparatively fewer PWL functions for the approximate sigmoid function for the specified accuracy.

Modeling cyclic volatile methylsiloxanes removal efficiency from wastewater by ZnO-coated aluminum anode using artificial neural networks

Usage of cyclic volatile methyl siloxanes (cVMSs) in the industrial process is unavoidable due to their superior properties; however, it is hazardous to human health. Photocatalytic zinc oxide coated aluminum anode is used to degrade the cVMSs in wastewater. In this work, we investigated the relationship among degradation process parameters such as current density (4–20 mA/cm2), initial pH (5–9), plate distance (8–24 cm), UV intensity (0–120 W), and reaction time (30–100 min) vis-a-vis cVMSs removal efficiency by using data-driven artificial neural networks(ANN) model. The ANN model was trained using a backpropagation algorithm with the sigmoid activation function between input, hidden, and the output layers. Two hidden layers with eight neurons in each layer presented the minimum average training error (0.24) and higher (0.99) correlation coefficient values (both Pearson’s r and Adj. R2) as compared with the conventional regression model. The effect and relationship between the parameters and cVMSs removal efficiency were analyzed by single, two variable sensitivity analysis, qualitative and quantitative estimation.

Computational approach to clinical diagnosis of diabetes disease: a comparative study

Diabetes is one of the most prevalent non-communicable diseases and is the 6th leading cause of death worldwide. It’s a chronic metabolic disorder which has no cure, however, it is a highly treatable condition, if diagnosed and managed on time to avoid its complications. This paper explores and compares various machine learning (ML) approaches that can help in determining the risk of diabetes at an early stage and aid in improving the medical diagnosis of diabetes. The paper considers two real-world datasets one is a diabetic clinical dataset (DCA) collected from a medical practitioner in the state of Assam, India during the year 2017–2018 and other is public PIMA Indian diabetic dataset. To analyze the various machine learning techniques on DCA and PIMA Indian diabetic datasets for the classification of diabetic and non-diabetic patients, different classifiers like perceptron, Gaussian process, linear discriminant analysis, quadratic discriminant analysis, statistical gradient descent, ridge regression classifier, support vector machines, k-nearest neighbors, decision tree, naïve Bayes, logistic regression, random forest and ELM for multiquadric, RBF, sigmoid activation functions are used. The results of numerical experiments suggested that logistic regression yields better performance in comparison to the other techniques.

Reliability of Neural Networks Based on Spintronic Neurons

Spintronic technology is emerging as a direction for the hardware implementation of neurons and synapses of neuromorphic architectures. In particular, a single spintronic device can be used to implement the nonlinear activation function of neurons. Here, we propose how to implement spintronic neurons with a sigmoidal and ReLU-like activation functions. We then perform a numerical experiment showing the robustness of neural networks made by spintronic neurons all having different activation functions to emulate device-to-device variations in a possible hardware implementation of the network. Therefore, we consider a vanilla neural network implemented to recognize the categories of the Mixed National Institute of Standards and Technology database, and we show an average accuracy of 98.87 % in the test dataset which is very close to the 98.89% as obtained for the ideal case (all neurons have the same sigmoid activation function). Similar results are also obtained with neurons having a ReLU-like activation function.

A Lightweight and Real-Time Hardware Architecture for Interference Detection and Mitigation of Time Synchronization Attacks Based on MLP Neural Networks

Stationary GPS receivers provide time information for critical infrastructures, such as phasor measurement units (PMUs), communication networks, and financial systems. Therefore, they are prone to a specific type of spoofing attack called time synchronization attack (TSA), which affects time information such as clock offset and clock drift. The receiver’s position remains constant during the attack; hence, attack detection and mitigation are challenging. Various countermeasures have been suggested to mitigate TSA effects. However, they are mainly software-based and are exploited to protect software implemented software-defined radios (SDRs). In this research, two hardware protection approaches are contributed for hardware-based SDRs based on multi-layer perceptron neural network (MLP NN) with sigmoid activation function. The most challenging part of MPL NN implementation is the activation function approximation. Therefore, two lightweight architectures are proposed for sigmoid function implementation. Linear approximation and look-up table (LA-LUT) and piece-wise linear approximation (PLA) are exploited for this task. The synthesis results demonstrate that the PLA approach has a slightly higher resource utilization in comparison to LA-LUT, while this method is more accurate. The mean squared error (MSE) of the PLA approach is equal to 0.019, which is 57% better than the LA-LUT approach with an MSE of 0.033. Furthermore, the designs are evaluated by two conventional types of TSA. According to the results, both methods are lightweight, and they only consume less than 0.3% of slice registers, 5% of slice LUTs, and 8% of DSP48E1Ss. Furthermore, they are real-time, and can mitigate the attack consequences; however, the PLA architecture has a better performance compared to LA-LUT.

Comparison of Multivariate Regression Models and Artificial Neural Networks for Prediction Highway Traffic Accidents in Spain: A Case Study

In recent years Spain shows the great reduction in the accident rate that has been achieved and the improvement of the behavior of road users, despite this, there is still a need to improve many areas. In 2016 for the first time since the last 13 years, the number of deaths increased by 7% concerning to the previous year. In this paper, analysis, and prediction of road traffic accidents (RTAs) of high accident locations highways in Spain, were undertaken using Artificial Neural Networks (ANNs), which can be used for policymakers, this paper contributes to the area of transportation safety and researchers. ANN is a powerful technique that has demonstrated considerable success in analyzing historical data to forecast future trends.There are many ANN models for predicting the number of accidents on highways that were developed using 4 years of data for accident counts on the Spain freeway roads from 2014 to 2017. The best ANN model was selected for this task and the model variables involved highway sections, years, section length (km), annual average daily traffic (AADT), the average horizontal curve radius, Slope gradient, traffic accidents with the number of heavy vehicles. In the ANN model development, the sigmoid activation function was employed with the Levenberg-Marquardt algorithm and the different number of neurons.The model results indicate the estimated traffic accidents, based on appropriate data are close enough to actual traffic accidents and so are dependable to forecast traffic accidents in Spain. However, it demonstrates that ANNs provide a potentially powerful tool in analyzing and predicting traffic accidents. The performance of the model was in comparison to the multivariate regression model developed for the same purpose. The results prove that the ANN model stronger forecasted model which produced estimates fairly close to forecast future highway traffic accidents with Spanish conditions.

Activation Function: Cell Recognition Based on YoLov5s/m

Activation functions play a critical role in neural networks. The paper mainly studies activation functions with four activation functions that were the selection for reference and comparison. The Mish activation function was expending as the Mish_PLUS activation function, the Sigmoid activation function, and the Tanh were combined to obtain a new Sigmoid_Tanh activation function. We used the recently popular YoLov5s and YoLov5m as the basic structure of the neural network. The function realized in this article was the recognition function of red blood cells, white blood cells, and platelets. Through the role and comparison of different activation functions in the neural network structure, the test results show that, in this paper, the training precision curve under the Sigmoid_Tanh activation function was better than that under the action of other activation functions. That means that the accuracy of cell recognition under the activation function was higher.

Using artificial neural networks of the type extreme learning machine for the modelling and prediction of the temperature in the head the column. Case of a C6H11-CH3 distillation column

This study establishes a reliable model based on artificial neural networks of the ELM (extreme learning machine) type. The use of this model must reflect the true behavior of the process under its normal operating conditions and enable distinguishing a normal mode from an abnormal one. The installation in this study is a continuous distillation column for methylcyclohexane from a toluene/methylcyclohexane mixture, which mass composition has been defined as 23% methylcyclohexane. The ANN-ELM model was applied to a database of 1000 samples. All the relevant inputs of the model are defined by the inputs of the continuous distillation column during its normal operation, namely: the heating power, the preheating power, the reflux rate, the feed rate, the pressure drops and the preheating temperature. On the other hand, the model’s output is defined by the output of the continuous distillation column during its normal operation, namely the temperature at the head of the column. The ELM-type neural architecture obtained during the learning phase was tested on a 30% of the database. The results showed very good forecast accuracy using the ELM model. The low RMSE value (RMSE = 0.0168) was recorded during the test phase when the number of neurons in the hidden layer becomes 30. The prediction of the temperature at the head of the column by the ANN-ELM model has achieved its most accurate performance when the sigmoid activation function was adopted with a number of neurons in the hidden layer of 30. Moreover, the correlation coefficients were very close to unity during the test phase (R = 0.9345). The ANN-ELM prediction model is best suited for modeling the normal mode with variable operating points of the automated continuous distillation column and can be used online for the detection and diagnosis of malfunctions in this type of operation installation.

Performance Analysis of Activation Functions for Wheat Crop Yield Prediction

Activation function (AF) plays a vital role in the neural network. This research study focuses on the performance evaluation of the various AFs in multilayer perceptron (MLP) algorithm for the wheat crop yield prediction at the regional level. Nowadays researchers are publishing the new AFs and also proposing the improvement in existing AFs for more accurate results. Sigmoid AF is the default function used in the MLP algorithm. In this research we have applied the various AFs in MLP using the WEKA java libraries and performance analysis has been carried out for the selected districts of the Gujarat state. Based on the mean absolute percentage error (MAPE) performance measure output is discussed. Therefore scientists and researchers can select the appropriate AF to improve the prediction accuracy and hence improve the performance of the network as per the research data.Neural Network, Activation Function (AF), MLP, Yield Prediction

COVID-19 Time Series Prediction

The Artificial Neural Network (ANN) is a computer technique that uses a mathematical model to represent a simpler form of the biologic neural structure. It is formed by many processing units and its intelligent behavior comes from the iterations between these units. One application of the ANN is for time series prediction algorithms, where the network learns the behavior of time dependent data and it is able to predict future values. In this work, the ANN is applied in predicting the number of COVID-19 confirmed cases and deaths and also the future seven days for the time series of Brazil, Portugal and the United States. From the simulations it is possible to conclude that the prediction of confirmed cases and deaths from COVID-19 have been successfully made by the ANN. Overall, the ANN with a specific test set had a Mean Squared Error (MSE) 50% higher than the ANN with a random test set. The combination of the sigmoidal and linear activation functions and the Levenberg-Marquardt training function had the lowest MSE for all cases.