Types Of Activation Functions Research Articles

An overview of activation functions used in Neuronal Networks: Application for plant disease diagnosis

Deep learning techniques, particularly Convolutional Neural Networks (CNNs), have led to significant progress in image processing. Many applications in automatic identification of plant diseases have been developed. This work adopts a new approach that focuses on studying a relevant parameter that make a significant impact on the performance of CNNs, namely, the variants of activation function, particularly the most famous used functions and their influence on the model’s performance and accuracy. We will also present the different types of activation functions, which are also called transfer functions. Then, and through a case study application to the plant disease detection, we will have the opportunity to compare the results of these different functions with a graphical presentation using evaluation metrics, such as accuracy functions and loss functions as Binary Cross-Entropy. The training of the models was carried out using a free accessible database of 20,639 photographs, taken both in the laboratory and in real conditions from the crop fields. The data includes three plant species in fifteen distinct classes of combinations [plant, disease], including some healthy plants.

Performance of HAF in Attention-BiLSTM for predicting the quality of automobile ancillary suppliers

Deep learning plays an important role in Machine learning and Artificial Neural Networks. Deep learning resembles the working of human brain in data processing for detecting objects and making decisions. Training and predicting data could happen through various deep learning algorithms available. One such algorithm is Bidirectional LSTM which could be used for classification and prediction of data in machine learning. Attention layer is a cognitive process in neural networks that gives attention and weights towards certain important features. The activation functions are mathematical equations used in neural networks to learn a complex problem which determines the output. Activation functions decide which feature to be activated and define the output. There are 7 types of Activation functions, they are tanh, sigmoid, relu,Leaky ReLU, Parametric ReLU and softmax. The proposed novel work uses Modified tanH with ReLU activation functions to analyze the performance of Auto Ancillary suppliers. Automobile Ancillary spare parts are one of the core industries in Indian Economy. Suppliers of Auto Ancillary Manufacturers provide the raw materials to the product Manufacturers which should be good in quality with reasonable cost and delivered on time.

A comparison of activation functions in multilayer neural network for predicting the production and consumption of electricity power

Predicting electricity power is an important task, which helps power utilities in improving their systems’ performance in terms of effectiveness, productivity, management and control. Several researches had introduced this task using three main models: engineering, statistical and artificial intelligence. Based on the experiments, which used artificial intelligence models, multilayer neural networks model has proven its success in predicting many evaluation datasets. However, the performance of this model depends mainly on the type of activation function. Therefore, this paper introduces an experimental study for investigating the performance of the multilayer neural networks model with respect to different activation functions and different depths of hidden layers. The experiments in this paper cover the comparison among eleven activation functions using four benchmark electricity datasets. The activation functions under examination are sigmoid, hyperbolic tangent, SoftSign, SoftPlus, ReLU, Leak ReLU, Gaussian, ELU, SELU, Swish and Adjust-Swish. Experimental results show that ReLU and Leak ReLU activation functions outperform their counterparts in all datasets.

A generalized varying-parameter recurrent neural network for super solution of quadratic programming problem

To obtain superior convergent speed, a generalized varying-parameter recurrent neural network (GVP-RNN) is established and analyzed for solving quadratic programming (QP) problems. Different from the classical fix-parameter neural network, such as Zhang neural network (ZNN) and finite-time convergent differential neural network (FT-CDNN), GVP-RNN dynamics can achieve exponential convergent results and possess high precision. By Lagrange theorem, QP problems can be transformed to a time-varying matrix equation, which is prepared for the GVP-RNN model. Theoretical analysis has shown the GVP-RNN model takes a super-exponential convergence comparing the corresponding simulated results of ZNN and FT-CDNN. Simulation comparisons are presented to evaluate the theoretical performance of the GVP-RNN with three types of activation functions. Furthermore, the proposed GVP-RNN model is analyzed and applied to control kinematic trajectory of redundant manipulators.

Pattern Recognition of Development Stage of Creepage Discharge of Oil–Paper Insulation under AC–DC Combined Voltage Based on OS-ELM

The recognition of the creepage discharge development process of oil–paper insulation under AC–DC combined voltage is the basis for fault monitoring and diagnosis of converter transformers; however, only a few related studies are available. In this study, the AC–DC combined voltage with a ratio of 1:1 was used to develop a recognition method for the creepage discharge development process of an oil–paper insulation under a cylinder–plate electrode structure. First, the pulse current method was used to collect the discharge signals in the creepage discharge development process. Then, 24 characteristic parameters were extracted from four types of creepage discharge characteristic spectra after dimensionality reduction. Finally, based on the online sequential extreme learning machine (OS-ELM) algorithm, these characteristic parameters were used to recognize the development stage of the creepage discharge of the oil–paper insulation. The results showed that when the size of the sample training set used in the OS-ELM algorithm is close to the number of hidden layer neurons, a high recognition accuracy can be obtained, and the type of activation function has little influence on the recognition accuracy. Four stages of the creepage discharge development process were recognized using the OS-ELM algorithm; the trend was the same as that of the characteristic parameters of the entire creepage discharge development process. The recognition accuracy was 91.4%. The algorithm has a high computing speed and high accuracy and can train data in batches. Therefore, it can be widely used in the field of online monitoring and evaluation of electrical equipment status.

Neural networks with finite-time convergence for solving time-varying linear complementarity problem

Time-varying linear complementarity problem (TLCP) has received a great deal of attention due to its broad variety of scientific and engineering applications. Several efficient Zhang neural networks are introduced for solving TLCP in this paper. Theoretical analysis shows that the related error function of the model proposed in this paper eventually tends to zero. The state convergence time periods of those Zhang neural networks with three types of activation functions are proved to be finite and can be quantitatively estimated by using some given parameters. Further, it is shown that the proposed neural network is of noise-tolerance, which means the neural network is more appropriate for a wider application. Moreover, in order to implement neural network numerically, a related discrete-time version is also studied. Finally, numerical simulations confirm the analysis of the proposed models concretely.

Performance of HAF in Attention-BiLSTM for predicting the quality of automobile ancillary suppliers

Deep learning plays an important role in Machine learning and Artificial Neural Networks. Deep learning resembles the working of human brain in data processing for detecting objects and making decisions. Training and predicting data could happen through various deep learning algorithms available. One such algorithm is Bidirectional LSTM which could be used for classification and prediction of data in machine learning. Attention layer is a cognitive process in neural networks that gives attention and weights towards certain important features. The activation functions are mathematical equations used in neural networks to learn a complex problem which determines the output. Activation functions decide which feature to be activated and define the output. There are 7 types of Activation functions, they are tanh, sigmoid, relu,Leaky ReLU, Parametric ReLU and softmax. The proposed novel work uses Modified tanH with ReLU activation functions to analyze the performance of Auto Ancillary suppliers. Automobile Ancillary spare parts are one of the core industries in Indian Economy. Suppliers of Auto Ancillary Manufacturers provide the raw materials to the product Manufacturers which should be good in quality with reasonable cost and delivered on time.

Improving the Performance of Deep Neural Networks Using Two Proposed Activation Functions

In artificial neural networks, activation functions play a significant role in the learning process. Choosing the proper activation function is a major factor in achieving a successful learning performance. Many activation functions are sufficient universal approximators, but their performance is lacking. Thus, many efforts have been directed toward activation functions to improve the learning performance of artificial neural networks. However, the learning process involves many challenges, such as saturation, dying, and exploding/vanishing the gradient problems. The contribution of this work resides in several axes. First, we introduce two novel activation functions: absolute linear units and inverse polynomial linear units. Both activation functions are augmented by an adjustable parameter that controls the slope of the gradient. Second, we present a comprehensive study and a taxonomy of various types of activation functions. Third, we conduct a broad range of experiments on several deep neural architecture models with consideration of network type and depth. Fourth, we evaluate the proposed activation functions' performance in image and text classification tasks. For this purpose, several public benchmark datasets are utilized to evaluate and compare the performance of the proposed functions with that of a group of common activation functions. Finally, we deeply analyze the impact of several common activation functions on deep network architectures. Results reveal that the proposed functions outperform most of the popular activation functions in several benchmarks. The statistical study of the overall experiments on both classification categories indicates that the proposed activation functions are robust and superior among all the competitive activation functions in terms of average accuracy.

Diagnosis of Lung Cancer Disease Based on Back-Propagation Artificial Neural Network Algorithm

Early stage detection of lung cancer is important for successful controlling of the diseases, also to offer additional chance to the patients in order to survive. So , algorithms that are related with computer vision and Image processing are extremely important for early medical diagnosis of lung cancer. In current work () computed tomography scan images were collected from several patients Classification was done using Back Propagation Artificial Neural Network ( ).It is considered as a powerful artificially intelligent technique with training rule for optimization to update the weights of the overall connections in order to determine the abnormal image. Several pre-processing operations and morphologic techniques were introduced to improve the condition of the image and make it suitable for detection cancer.Histogram and () Gray Level Co-occurrence Matrix were applied toget best features extraction analysis from lung image.Three types of activation functions(trainlm ,trainbr ,traingd) were used which gives a significant accuracy for detecting cancer in scan lung image related to the suggested algorithm. Best results were obtained with accuracy rate 95.9 % in trainlm activation function.. Graphic User Interface ( ) was displaying to show the final diagnosis for lung.

L_p_ Approximation by ReLU Neural Networks

We know that we can use the neural networks for the approximation of functions for many types of activation functions. Here, we treat only neural networks with simple and particular activation function called rectified linear units (ReLU). The main aim of this paper is to introduce a type of constructive universal approximation theorem and estimate the error of the universal approximation. We will obtain optimal approximation if we have a basis independent of the target function. We prove a type of Debao Chen's theorem for approximation.

An artificial neural network for analysis of ionograms obtained by ionosonde at the Ukrainian Antarctic Akademik Vernadsky station

The article presents the developed artificial neural network for F2 ionosphere layer traces scaling on ionograms obtained using the IPS-42 ionosonde installed at the Ukrainian Antarctic Akademik Vernadsky station. The parameters of the IPS-42 ionosonde and the features of the data obtained with it, in particular the format of the output files, are presented. The advantages of using an artificial neural network for identification of traces on ionograms are demonstrated. Usually, an automatic scaling of the ionograms requires a lot of machine time however implementation of an artificial neural network speeds up computations significantly allowing to process incoming ionograms even in the real time mode. The choice of architecture of an artificial neural network is substantiated. The U-Net architecture was chosen. The method of creating and training the neural network is described. The artificial neural network development process included choosing the number of layers, types of activation functions, optimization method and input layer size. Software developed was written in Python programming language with use of the Keras library. Examples of data used for training of the artificial neural network are shown. The results of testing an artificial neural network are presented. The data obtained with the artificial neural network are compared with the results of manual processing of ionograms. Data for training the artificial neural network were obtained in March, 2017 using the IPS-42 ionosonde installed at the Ukrainian Antarctic Akademik Vernadsky station; data for testing were obtained in 2017 and 2020. The developed artificial neural network has minor flaws but they are easily eliminated by retraining the network on a more representative dataset (obtained in various years and seasons). The general results of testing indicate good prospects in further developing this artificial neural network and software for working with it.

Research on Highway Slope Monitoring Data Prediction Based on Long Short-term Memory Network

This paper first analyzes the basic structure and operation principle of Long Short-Term Memory Network (LSTM), and combines the characteristics of road slope monitoring data to determine the data form of the input layer and the type of activation function. Then, combined with the slope displacement monitoring data, a slope monitoring data prediction model based on LSTM is constructed, and the main structural parameters of the LSTM are optimized to predict the slope monitoring data. Finally, the data prediction results are analyzed, and the reliability of the prediction results is verified by analyzing the influence of the number of monitoring data samples on the prediction results.

Automation of the process of selecting hyperparameters for artificial neural networks for processing retrospective text information

Neural network technologies are successfully used in solving problems from various areas of the economy - industry, agriculture, medicine. The problems of substantiating the choice of architecture and hyperparameters of artificial neural networks (ins) aimed at solving various classes of applied problems are caused by the need to improve the quality and speed of deep ins training. Various methods of optimizing ins hyperparameters are known, for example, using genetic algorithms, but this requires writing additional software. To optimize the process of selecting hyperparameters, Google research has developed the KerasTuner Toolkit, which is a user-friendly platform for automated search for optimal hyperparameter combinations. In the described Kerastuner Toolkit, you can use random search, Bayesian optimization, or Hyperband methods. In numerical experiments, 14 hyperparameters varied: the number of blocks of convolutional layers and their forming filters, the type of activation functions, the parameters of the «dropout» regulatory layers, and others. The studied tools demonstrated high optimization efficiency while simultaneously varying more than a dozen parameters of the convolutional network, while the calculation time on the Colaboratory platform for the studied INM architectures was several hours, even with the use of GPU graphics accelerators. For ins focused on processing and recognizing text information in natural language (NLP), the recognition quality has been improved to 83-92%.

A derived least square fast learning network model

The extreme learning machine (ELM) requires a large number of hidden layer nodes in the training process. Thus, random parameters will exponentially increase and affect network stability. Moreover, the single activation function affects the generalization capability of the network. This paper proposes a derived least square fast learning network (DLSFLN) to solve the aforementioned problems. DLSFLN uses the inheritance of some functions to obtain various activation functions through continuous differentiation of functions. The types of activation functions were increased and the mapping capability of hidden layer neurons was enhanced when the random parameter dimension was maintained. DLSFLN randomly generates the input weights and hidden layer thresholds and uses the least square method to determine the connection weights between the output and the input layers and that between the output and the input nodes. The regression and classification experiments show that DLSFLN has a faster training speed and better training accuracy, generalization capability, and stability compared with other neural network algorithms, such as fast learning network(FLN).

Generic model implementation of deep neural network activation functions using GWO-optimized SCPWL model on FPGA

The implementation of non-linear Activation Functions (AFs) within the Artificial Neural Network (ANN) on the Field Programmable Gate Array (FPGA) is substantial due to the various applications it performs. Accuracy, speed and complexity are the most crucial factors considered in this implementation. Building non-linear AFs in a reconfigurable ANN requires either sequential operations and/or additional complexity. In this paper, a generic model for three types of non-linear AFs (Logistic sigmoid (LogSig), Tan sigmoid (TanSig) and Radial Basis Function (RBF)) has been designed based on Simplicial Canonical Piecewise Linear (SCPWL) model that is optimized using Grey Wolf Optimizer (GWO(Algorithm. The designed model has been achieved by nine segments of the SCPWL model. The input of the AFs is ranging from (−8 to 8). Matlab has been deployed to design, optimize, simulate and validate this model. The maximum errors were 5.2e−3, 15.4e−3 and 7e−3 for LogSig, TanSig and RBF respectively. And, the Mean Square Error (MSE) were 1.81e−6, 1.22e−5 and 1.42e−5 for LogSig, TanSig and RBF respectively. The Matlab/HDL Coder has been used to generate the VHDL codes. The Xilinx Arty A7 (Xc7a35ticsg324-1L) FPGA kit is used to validate the designed model on Vivado Design Suite software. It has been noticed that it takes 581 Look-Up Tables (LUTs), nine DSP slices and a delay of (35.346 ns) to implement the nine SCPWL segments for any linear and non-linear AF. For validation, a complete ANN has been built with three hidden layers, each layer contain with one of the proposed AF models.

Lagrange exponential stability of quaternion‐valued memristive neural networks with time delays

This paper investigates the Lagrange global exponential stability of the quaternion‐valued memristive neural networks (QVMNNs). Two kinds of activation functions based on different assumptions are considered. Then, based on the Lyapunov function approach, decomposition method, and some inequality skills, two novel sufficient conditions for lagrange stability of QVMNNs are provided corresponding to different types of activation functions. Lastly, simulation examples are provided to demonstrate the correctness of our theoretical results.

A hybrid approach for a cost estimate of residential buildings in Egypt at the early stage

Owners need clear information about the cost at the early stage, while information about drawings and designs are limited in this stage. The purpose of this research is to propose a hybrid model using artificial neural networks and the regression analysis to estimate the cost of residential buildings in Egypt. Real data were collected from 174 real residential projects in Egypt. The Delphi technique was used to reach a consensus on the key factors affecting early stage cost estimation. Artificial neural network models were developed with various numbers of hidden layers and types of activation functions. The sensitivity analysis showed that the most effective factors in the cost estimate at the early stage are the number of floors and the area of the floors. The multiple linear regression, polynomial regression, gamma regression, and Poisson regression were used. The proposed hybrid model was extracted from the ANN model and the regression models using multiple linear regression. The mean absolute percentage error of the hybrid model was 10.64% which is less than the absolute percentage error of the ANN model and the regression models. The results of the hybrid model indicate that the model was successful in estimating the cost of residential projects and would be useful for decision-makers in the construction industry.

Determining optical constants of 2D materials with neural networks from multi-angle reflectometry data

Synthetically generated multi-angle reflectometry data is used to train a neural network based learning system to estimate the refractive index of atomically thin layered materials in the visible part of the electromagnetic spectrum. Unlike previously developed regression based optical characterization methods, the prediction is achieved via classification by using the probabilities of each input element belonging to a label as weighting coefficients in a simple analytical formula. Various types of activation functions and gradient descent optimizers are tested to determine the optimum combination yielding the best performance. For the verification of the proposed method’s accuracy, four different materials are studied. In all cases, the maximum error is calculated to be less than 0.3%. Considering the highly dispersive nature of the studied materials, this result is a substantial improvement in terms of accuracy and efficiency compared to traditional approaches.

Implementation of Quantum Gates using MATLAB

Now days we require low cost and high performance computational based applications. Quantum inspired computational device or circuit performs effective result compare to classical based devices. In the development of quantum-based devices and network needs number of quantum logic gates. Here we studied mathematical description of different types of single and multiple qubits-based quantum logic gates, the reversibility property of quantum gates also proved mathematically. We analyze the cost and effectiveness of each quantum gates has been implemented using neural network with the help of MATLAB. The cost and effectiveness of quantum gates has been analyzed with the comparison of different types of activation function.

A study on the relationship between the rank of input data and the performance of random weight neural network

Random feature mapping (RFM) is the core operation in the random weight neural network (RWNN). Its quality has a significant impact on the performance of a RWNN model. However, there has been no good way to evaluate the quality of RFM. In this paper, we introduce a new concept called dispersion degree of matrix information distribution (DDMID), which can be used to measure the quality of RFM. We used DDMID in our experiments to explain the relationship between the rank of input data and the performance of the RWNN model and got some interesting results. We found that: (1) when the rank of input data reaches a certain threshold, the model’s performance increases with the increase in the rank; (2) the impact of the rank on the model performance is insensitive to the type of activation functions and the number of hidden nodes; (3) if the DDMID of an RFM matrix is very small, it implies that the first $$\mathbf{k}$$ singular values in the singular value matrix of the RFM matrix contain too much information, which usually has a negative impact on the final closed-form solution of the RWNN model. Besides, we verified the improvement effect of intrinsic plasticity (IP) algorithm on RFM by using DDMID. The experimental results showed that DDMID allows researchers evaluate the mapping quality of data features before model training, so as to predict the effect of data preprocessing or network initialization without model training. We believe that our findings could provide useful guidance when constructing and analyzing a RWNN model.