Types Of Activation Functions Research Articles

ПОШУК ОПТИМАЛЬНОЇ ТОПОЛОГІЇ ШТУЧНИХ НЕЙРОННИХ МЕРЕЖ НА БАЗІ БАГАТОВИМІРНИХ ПОЛІНОМІВ ЛЕЖАНДРА.

Earlier in [1], it was concluded that it is necessary to improve the learning algorithms of neural networks operating in systems that generate electricity using renewable energy sources. This article is intended to acquaint the reader with a new type of activation functions of artificial neural networks (ANN), namely - the use of Legendre polynomials, as well as a new method of learning ANN, when this process is not sequential, as usual, but in parallel. The accepted statements made it possible to make sure that the new, designed neural network has better properties (such as training time and less value of learning error) than the standard ones. The relevance of this topic lies in the following provisions: - improving the interaction between the solar station and artificial intelligence systems, through increased productivity; - taking into account the transients in the electrical network by means of intelligent control, through the use of neural networks of the proposed architecture. The developed neural networks have found their application in the work of a photovoltaic station. Their main purpose is to fulfill the forecast in the electrical networks of the amount of generated power. To successfully complete the task, the following goals were set and solved: to analyze and compare standard activation functions and algorithms for ANN training, to show methods and describe the improvement of networks, to demonstrate the application of developed ANN in photovoltaic problems. This article was designed to acquaint with the new method of building neural networks, which is based on seeing the transmission of signals in a non-sequential way, such as parallel, with certain features of the connection with which it was given in the text. The paper also demonstrates the use of the Legendre polynomial using qualitative neural network activation functions that work with solar panels. For confirmation in the article the answers to calculations are given. In future materials it is planned to streamline in more detail the process of modeling and compiling a mathematical calculation for the construction of neural networks.

Supervised learning in the presence of concept drift: a modelling framework

We present a modelling framework for the investigation of supervised learning in non-stationary environments. Specifically, we model two example types of learning systems: prototype-based learning vector quantization (LVQ) for classification and shallow, layered neural networks for regression tasks. We investigate so-called student–teacher scenarios in which the systems are trained from a stream of high-dimensional, labeled data. Properties of the target task are considered to be non-stationary due to drift processes while the training is performed. Different types of concept drift are studied, which affect the density of example inputs only, the target rule itself, or both. By applying methods from statistical physics, we develop a modelling framework for the mathematical analysis of the training dynamics in non-stationary environments. Our results show that standard LVQ algorithms are already suitable for the training in non-stationary environments to a certain extent. However, the application of weight decay as an explicit mechanism of forgetting does not improve the performance under the considered drift processes. Furthermore, we investigate gradient-based training of layered neural networks with sigmoidal activation functions and compare with the use of rectified linear units. Our findings show that the sensitivity to concept drift and the effectiveness of weight decay differs significantly between the two types of activation function.

Global dissipativity of fuzzy bidirectional associative memory neural networks with proportional delays

This article aimed to investigate the problem of global dissipativity of Fuzzy Bidirectional Associative Memory Neural Networks (FBAMNNs- for short) with proportional delays. Via Lyapunov Functionals (LFs- for short) and Linear Matrix Inequality (LMI- for short) approach, we obtained new sufficient conditions to guarantee the global dissipativity and global exponential dissipativity of the proposed model. In addition, two different types of activation functions are considered, including general bounded and Lipschitz-type activation functions. Moreover, the globally attractive and globally exponentially attractive sets are presented. Lastly, two numerical examples are given to illustrate the effectiveness of the developed results.

Global Lagrange stability analysis of retarded SICNNs

Abstract The stability in the Lagrange sense for cellular neural networks (CNNs) has proven to be one of the effective tools to study multi-stable dynamics of the neural networks. In this article, rather than studying the existence and Lyapunov stability of an equilibrium point we investigate multi-stable dynamics of shunting inhibitory cellular neural networks (SICNNs) with time-varying delays and coefficients. This is the first paper that addresses the Lagrange stability for SICNNs. By constructing proper Lyapunov functions and using inequality techniques, we analyze three different types of activation functions, namely, bounded, sigmoid and Lipschitz-like type activation functions. New delay-dependent sufficient criteria are derived to ensure the global Lagrange stability for SICNNs. Furthermore, globally exponentially attractive sets are given for the different activation functions. Finally, an illustrating example with numerical simulations is given to support the theoretical results.

An overview of activation functions used in Neuronal Networks: Application for plant disease diagnosis

Deep learning techniques, particularly Convolutional Neural Networks (CNNs), have led to significant progress in image processing. Many applications in automatic identification of plant diseases have been developed. This work adopts a new approach that focuses on studying a relevant parameter that make a significant impact on the performance of CNNs, namely, the variants of activation function, particularly the most famous used functions and their influence on the model’s performance and accuracy. We will also present the different types of activation functions, which are also called transfer functions. Then, and through a case study application to the plant disease detection, we will have the opportunity to compare the results of these different functions with a graphical presentation using evaluation metrics, such as accuracy functions and loss functions as Binary Cross-Entropy. The training of the models was carried out using a free accessible database of 20,639 photographs, taken both in the laboratory and in real conditions from the crop fields. The data includes three plant species in fifteen distinct classes of combinations [plant, disease], including some healthy plants.

Performance of HAF in Attention-BiLSTM for predicting the quality of automobile ancillary suppliers

Deep learning plays an important role in Machine learning and Artificial Neural Networks. Deep learning resembles the working of human brain in data processing for detecting objects and making decisions. Training and predicting data could happen through various deep learning algorithms available. One such algorithm is Bidirectional LSTM which could be used for classification and prediction of data in machine learning. Attention layer is a cognitive process in neural networks that gives attention and weights towards certain important features. The activation functions are mathematical equations used in neural networks to learn a complex problem which determines the output. Activation functions decide which feature to be activated and define the output. There are 7 types of Activation functions, they are tanh, sigmoid, relu,Leaky ReLU, Parametric ReLU and softmax. The proposed novel work uses Modified tanH with ReLU activation functions to analyze the performance of Auto Ancillary suppliers. Automobile Ancillary spare parts are one of the core industries in Indian Economy. Suppliers of Auto Ancillary Manufacturers provide the raw materials to the product Manufacturers which should be good in quality with reasonable cost and delivered on time.

A comparison of activation functions in multilayer neural network for predicting the production and consumption of electricity power

Predicting electricity power is an important task, which helps power utilities in improving their systems’ performance in terms of effectiveness, productivity, management and control. Several researches had introduced this task using three main models: engineering, statistical and artificial intelligence. Based on the experiments, which used artificial intelligence models, multilayer neural networks model has proven its success in predicting many evaluation datasets. However, the performance of this model depends mainly on the type of activation function. Therefore, this paper introduces an experimental study for investigating the performance of the multilayer neural networks model with respect to different activation functions and different depths of hidden layers. The experiments in this paper cover the comparison among eleven activation functions using four benchmark electricity datasets. The activation functions under examination are sigmoid, hyperbolic tangent, SoftSign, SoftPlus, ReLU, Leak ReLU, Gaussian, ELU, SELU, Swish and Adjust-Swish. Experimental results show that ReLU and Leak ReLU activation functions outperform their counterparts in all datasets.

A generalized varying-parameter recurrent neural network for super solution of quadratic programming problem

To obtain superior convergent speed, a generalized varying-parameter recurrent neural network (GVP-RNN) is established and analyzed for solving quadratic programming (QP) problems. Different from the classical fix-parameter neural network, such as Zhang neural network (ZNN) and finite-time convergent differential neural network (FT-CDNN), GVP-RNN dynamics can achieve exponential convergent results and possess high precision. By Lagrange theorem, QP problems can be transformed to a time-varying matrix equation, which is prepared for the GVP-RNN model. Theoretical analysis has shown the GVP-RNN model takes a super-exponential convergence comparing the corresponding simulated results of ZNN and FT-CDNN. Simulation comparisons are presented to evaluate the theoretical performance of the GVP-RNN with three types of activation functions. Furthermore, the proposed GVP-RNN model is analyzed and applied to control kinematic trajectory of redundant manipulators.

Pattern Recognition of Development Stage of Creepage Discharge of Oil–Paper Insulation under AC–DC Combined Voltage Based on OS-ELM

The recognition of the creepage discharge development process of oil–paper insulation under AC–DC combined voltage is the basis for fault monitoring and diagnosis of converter transformers; however, only a few related studies are available. In this study, the AC–DC combined voltage with a ratio of 1:1 was used to develop a recognition method for the creepage discharge development process of an oil–paper insulation under a cylinder–plate electrode structure. First, the pulse current method was used to collect the discharge signals in the creepage discharge development process. Then, 24 characteristic parameters were extracted from four types of creepage discharge characteristic spectra after dimensionality reduction. Finally, based on the online sequential extreme learning machine (OS-ELM) algorithm, these characteristic parameters were used to recognize the development stage of the creepage discharge of the oil–paper insulation. The results showed that when the size of the sample training set used in the OS-ELM algorithm is close to the number of hidden layer neurons, a high recognition accuracy can be obtained, and the type of activation function has little influence on the recognition accuracy. Four stages of the creepage discharge development process were recognized using the OS-ELM algorithm; the trend was the same as that of the characteristic parameters of the entire creepage discharge development process. The recognition accuracy was 91.4%. The algorithm has a high computing speed and high accuracy and can train data in batches. Therefore, it can be widely used in the field of online monitoring and evaluation of electrical equipment status.

Neural networks with finite-time convergence for solving time-varying linear complementarity problem

Time-varying linear complementarity problem (TLCP) has received a great deal of attention due to its broad variety of scientific and engineering applications. Several efficient Zhang neural networks are introduced for solving TLCP in this paper. Theoretical analysis shows that the related error function of the model proposed in this paper eventually tends to zero. The state convergence time periods of those Zhang neural networks with three types of activation functions are proved to be finite and can be quantitatively estimated by using some given parameters. Further, it is shown that the proposed neural network is of noise-tolerance, which means the neural network is more appropriate for a wider application. Moreover, in order to implement neural network numerically, a related discrete-time version is also studied. Finally, numerical simulations confirm the analysis of the proposed models concretely.

Performance of HAF in Attention-BiLSTM for predicting the quality of automobile ancillary suppliers

Deep learning plays an important role in Machine learning and Artificial Neural Networks. Deep learning resembles the working of human brain in data processing for detecting objects and making decisions. Training and predicting data could happen through various deep learning algorithms available. One such algorithm is Bidirectional LSTM which could be used for classification and prediction of data in machine learning. Attention layer is a cognitive process in neural networks that gives attention and weights towards certain important features. The activation functions are mathematical equations used in neural networks to learn a complex problem which determines the output. Activation functions decide which feature to be activated and define the output. There are 7 types of Activation functions, they are tanh, sigmoid, relu,Leaky ReLU, Parametric ReLU and softmax. The proposed novel work uses Modified tanH with ReLU activation functions to analyze the performance of Auto Ancillary suppliers. Automobile Ancillary spare parts are one of the core industries in Indian Economy. Suppliers of Auto Ancillary Manufacturers provide the raw materials to the product Manufacturers which should be good in quality with reasonable cost and delivered on time.

Improving the Performance of Deep Neural Networks Using Two Proposed Activation Functions

In artificial neural networks, activation functions play a significant role in the learning process. Choosing the proper activation function is a major factor in achieving a successful learning performance. Many activation functions are sufficient universal approximators, but their performance is lacking. Thus, many efforts have been directed toward activation functions to improve the learning performance of artificial neural networks. However, the learning process involves many challenges, such as saturation, dying, and exploding/vanishing the gradient problems. The contribution of this work resides in several axes. First, we introduce two novel activation functions: absolute linear units and inverse polynomial linear units. Both activation functions are augmented by an adjustable parameter that controls the slope of the gradient. Second, we present a comprehensive study and a taxonomy of various types of activation functions. Third, we conduct a broad range of experiments on several deep neural architecture models with consideration of network type and depth. Fourth, we evaluate the proposed activation functions' performance in image and text classification tasks. For this purpose, several public benchmark datasets are utilized to evaluate and compare the performance of the proposed functions with that of a group of common activation functions. Finally, we deeply analyze the impact of several common activation functions on deep network architectures. Results reveal that the proposed functions outperform most of the popular activation functions in several benchmarks. The statistical study of the overall experiments on both classification categories indicates that the proposed activation functions are robust and superior among all the competitive activation functions in terms of average accuracy.

Diagnosis of Lung Cancer Disease Based on Back-Propagation Artificial Neural Network Algorithm

Early stage detection of lung cancer is important for successful controlling of the diseases, also to offer additional chance to the patients in order to survive. So , algorithms that are related with computer vision and Image processing are extremely important for early medical diagnosis of lung cancer. In current work () computed tomography scan images were collected from several patients Classification was done using Back Propagation Artificial Neural Network ( ).It is considered as a powerful artificially intelligent technique with training rule for optimization to update the weights of the overall connections in order to determine the abnormal image. Several pre-processing operations and morphologic techniques were introduced to improve the condition of the image and make it suitable for detection cancer.Histogram and () Gray Level Co-occurrence Matrix were applied toget best features extraction analysis from lung image.Three types of activation functions(trainlm ,trainbr ,traingd) were used which gives a significant accuracy for detecting cancer in scan lung image related to the suggested algorithm. Best results were obtained with accuracy rate 95.9 % in trainlm activation function.. Graphic User Interface ( ) was displaying to show the final diagnosis for lung.

L_p_ Approximation by ReLU Neural Networks

We know that we can use the neural networks for the approximation of functions for many types of activation functions. Here, we treat only neural networks with simple and particular activation function called rectified linear units (ReLU). The main aim of this paper is to introduce a type of constructive universal approximation theorem and estimate the error of the universal approximation. We will obtain optimal approximation if we have a basis independent of the target function. We prove a type of Debao Chen's theorem for approximation.

An artificial neural network for analysis of ionograms obtained by ionosonde at the Ukrainian Antarctic Akademik Vernadsky station

The article presents the developed artificial neural network for F2 ionosphere layer traces scaling on ionograms obtained using the IPS-42 ionosonde installed at the Ukrainian Antarctic Akademik Vernadsky station. The parameters of the IPS-42 ionosonde and the features of the data obtained with it, in particular the format of the output files, are presented. The advantages of using an artificial neural network for identification of traces on ionograms are demonstrated. Usually, an automatic scaling of the ionograms requires a lot of machine time however implementation of an artificial neural network speeds up computations significantly allowing to process incoming ionograms even in the real time mode. The choice of architecture of an artificial neural network is substantiated. The U-Net architecture was chosen. The method of creating and training the neural network is described. The artificial neural network development process included choosing the number of layers, types of activation functions, optimization method and input layer size. Software developed was written in Python programming language with use of the Keras library. Examples of data used for training of the artificial neural network are shown. The results of testing an artificial neural network are presented. The data obtained with the artificial neural network are compared with the results of manual processing of ionograms. Data for training the artificial neural network were obtained in March, 2017 using the IPS-42 ionosonde installed at the Ukrainian Antarctic Akademik Vernadsky station; data for testing were obtained in 2017 and 2020. The developed artificial neural network has minor flaws but they are easily eliminated by retraining the network on a more representative dataset (obtained in various years and seasons). The general results of testing indicate good prospects in further developing this artificial neural network and software for working with it.

Research on Highway Slope Monitoring Data Prediction Based on Long Short-term Memory Network

This paper first analyzes the basic structure and operation principle of Long Short-Term Memory Network (LSTM), and combines the characteristics of road slope monitoring data to determine the data form of the input layer and the type of activation function. Then, combined with the slope displacement monitoring data, a slope monitoring data prediction model based on LSTM is constructed, and the main structural parameters of the LSTM are optimized to predict the slope monitoring data. Finally, the data prediction results are analyzed, and the reliability of the prediction results is verified by analyzing the influence of the number of monitoring data samples on the prediction results.

Automation of the process of selecting hyperparameters for artificial neural networks for processing retrospective text information

Neural network technologies are successfully used in solving problems from various areas of the economy - industry, agriculture, medicine. The problems of substantiating the choice of architecture and hyperparameters of artificial neural networks (ins) aimed at solving various classes of applied problems are caused by the need to improve the quality and speed of deep ins training. Various methods of optimizing ins hyperparameters are known, for example, using genetic algorithms, but this requires writing additional software. To optimize the process of selecting hyperparameters, Google research has developed the KerasTuner Toolkit, which is a user-friendly platform for automated search for optimal hyperparameter combinations. In the described Kerastuner Toolkit, you can use random search, Bayesian optimization, or Hyperband methods. In numerical experiments, 14 hyperparameters varied: the number of blocks of convolutional layers and their forming filters, the type of activation functions, the parameters of the «dropout» regulatory layers, and others. The studied tools demonstrated high optimization efficiency while simultaneously varying more than a dozen parameters of the convolutional network, while the calculation time on the Colaboratory platform for the studied INM architectures was several hours, even with the use of GPU graphics accelerators. For ins focused on processing and recognizing text information in natural language (NLP), the recognition quality has been improved to 83-92%.

A derived least square fast learning network model

The extreme learning machine (ELM) requires a large number of hidden layer nodes in the training process. Thus, random parameters will exponentially increase and affect network stability. Moreover, the single activation function affects the generalization capability of the network. This paper proposes a derived least square fast learning network (DLSFLN) to solve the aforementioned problems. DLSFLN uses the inheritance of some functions to obtain various activation functions through continuous differentiation of functions. The types of activation functions were increased and the mapping capability of hidden layer neurons was enhanced when the random parameter dimension was maintained. DLSFLN randomly generates the input weights and hidden layer thresholds and uses the least square method to determine the connection weights between the output and the input layers and that between the output and the input nodes. The regression and classification experiments show that DLSFLN has a faster training speed and better training accuracy, generalization capability, and stability compared with other neural network algorithms, such as fast learning network(FLN).

Generic model implementation of deep neural network activation functions using GWO-optimized SCPWL model on FPGA

The implementation of non-linear Activation Functions (AFs) within the Artificial Neural Network (ANN) on the Field Programmable Gate Array (FPGA) is substantial due to the various applications it performs. Accuracy, speed and complexity are the most crucial factors considered in this implementation. Building non-linear AFs in a reconfigurable ANN requires either sequential operations and/or additional complexity. In this paper, a generic model for three types of non-linear AFs (Logistic sigmoid (LogSig), Tan sigmoid (TanSig) and Radial Basis Function (RBF)) has been designed based on Simplicial Canonical Piecewise Linear (SCPWL) model that is optimized using Grey Wolf Optimizer (GWO(Algorithm. The designed model has been achieved by nine segments of the SCPWL model. The input of the AFs is ranging from (−8 to 8). Matlab has been deployed to design, optimize, simulate and validate this model. The maximum errors were 5.2e−3, 15.4e−3 and 7e−3 for LogSig, TanSig and RBF respectively. And, the Mean Square Error (MSE) were 1.81e−6, 1.22e−5 and 1.42e−5 for LogSig, TanSig and RBF respectively. The Matlab/HDL Coder has been used to generate the VHDL codes. The Xilinx Arty A7 (Xc7a35ticsg324-1L) FPGA kit is used to validate the designed model on Vivado Design Suite software. It has been noticed that it takes 581 Look-Up Tables (LUTs), nine DSP slices and a delay of (35.346 ns) to implement the nine SCPWL segments for any linear and non-linear AF. For validation, a complete ANN has been built with three hidden layers, each layer contain with one of the proposed AF models.

Lagrange exponential stability of quaternion‐valued memristive neural networks with time delays

This paper investigates the Lagrange global exponential stability of the quaternion‐valued memristive neural networks (QVMNNs). Two kinds of activation functions based on different assumptions are considered. Then, based on the Lyapunov function approach, decomposition method, and some inequality skills, two novel sufficient conditions for lagrange stability of QVMNNs are provided corresponding to different types of activation functions. Lastly, simulation examples are provided to demonstrate the correctness of our theoretical results.