Types Of Activation Functions Research Articles

An Intelligent Algorithm for Semantic Feature Analysis and Translation of English Texts

Accurate and rapid translation is conducive to the cultural communication of different languages. This paper briefly introduces the long short-term memory (LSTM) algorithm. To enhance the performance of the LSTM algorithm, semantic features were introduced, and semantic similarity was used to screen the translations that are more in line with the semantics of the source text. Then, simulation experiments were conducted. The experiments first examined the effects of the quantity of hidden layer nodes and the type of activation function in LSTM on the translation performance. Then, the LSTM algorithm was compared with the recurrent neural network (RNN) and traditional LSTM algorithms.The proposed translation algorithm showed the best performance when there were 512 hidden layer nodes and the activation function was sigmoid, it performed better than the other two translation algorithms, and the obtained result was consistent with the semantic meaning of the source text and smooth.

Lightweight Food Recognition via Aggregation Block and Feature Encoding

Food image recognition has recently been given considerable attention in the multimedia field in light of its possible implications on health. The characteristics of the dispersed distribution of ingredients in food images put forward higher requirements on the long-range information extraction ability of neural networks, leading to more complex and deeper models. Nevertheless, the lightweight version of food image recognition is essential for improved implementation on end devices and sustained server-side expansion. To address this issue, we present Aggregation Feature Net (AFNet), a lightweight network that is capable of effectively capturing both global and local features from food images. In AFNet, we develop a novel convolution based on a residual model by encoding global features through row-wise and column-wise information integration. Merging aggregation block with classic local convolution yields a framework that works as the backbone of the network. Based on the efficient use of parameters by the aggregation block, we constructed a lightweight food image recognition network with fewer layers and a smaller scale, assisted by a new type of activation function. Experimental results on four popular food recognition datasets demonstrate that our approach achieves state-of-the-art performance with higher accuracy and fewer FLOPs and parameters. For example, in comparison to the current state-of-the-art model of MobileViTv2, AFNet achieved 88.4% accuracy of the top-1 level on the ETHZ Food-101 dataset, with similar parameters and FLOPs but 1.4% more accuracy. The source code will be provided in supplementary materials.

Evolutionary Optimization of Deep Learning Models: Enhance Performance through Hyperparameter and Structural Tuning with Evolutionary Algorithms

Evolutionary optimization is a strong way to improve the performance of deep learning models by using evolutionary algorithms (EAs) to fine-tune both structure elements and hyperparameters. This method is based on biological evolution and uses processes like crossing, mutation, and selection to make model setups better over time. When the model and hyperparameter space get more complicated, it can be hard to use traditional methods for adjusting hyperparameters, like grid search or random search, because they need a lot of computing power and don't work as well. Evolutionary algorithms, on the other hand, can quickly move through these high-dimensional spaces and find the best or almost best designs that make the model work better. This research looks into how evolutionary algorithms can be used to improve deep learning models. It focuses on two main areas: setting hyperparameters and making changes to the models' structures. Hyperparameters, like learning rate, batch size, and the number of layers, have a big effect on how the model learns and how accurate it is in the end. The evolutionary method encodes these hyperparameters into chromosomes and then iteratively evolves a population of models, picking the best ones based on a fitness function that usually shows how accurate or lost the models are. Random changes are made by mutations, and traits from high-performing models are combined in crosses. This creates variety and stops things from convergent too soon. Changing the model's design, such as the amount of neurons in each layer, the type of activation functions, and the patterns of connections, is called structural tweaking. These building blocks can be changed on the fly by evolutionary algorithms, which can find new, efficient designs that human creators might miss. This feature is especially helpful when making complicated networks like convolutional neural networks (CNNs) or recurrent neural networks (RNNs), where the best structure isn't always clear. Results from real life show that evolutionary optimization not only makes models more accurate, but it also makes them more stable and able to generalize. This method gives a scalable and adaptable strategy for optimizing deep learning models. This makes it a very useful tool for students and professionals who want to get the best performance in a wide range of situations.

OUTPUT POWER ESTIMATION OF A PHOTOVOLTAIC PANEL BY EXTREME LEARNING MACHINE

In this study, the output power of a photovoltaic (PV) panel under different operating conditions was estimated with the help of an extreme learning algorithm (ELM). For this purpose, a PV panel with a power of 180W was installed, and the open circuit voltage, short circuit current, panel temperature, and solar radiation of this panel were measured and recorded at regular intervals. A total of 75 measurement data were obtained. The maximum power of the panel was calculated using the open circuit voltage and short circuit current information. While panel temperature and solar radiation were given as inputs to the regression model of the PV panel based on ELM, the output of the regression model was taken as the maximum power of the PV panel. To improve the prediction accuracy of ELM, the number of input neurons of ELM and the type of activation function used in the hidden layer were determined by trial and error method. The generated PV data set is separated into training and testing sets. The performance of the method was examined with the 5-fold cross-validation method. For this purpose, the dataset was divided into 5 equal parts. One of these parts was used for testing the ELM and the remaining four sets were used for training the ELM, and this was done by changing the test set each time. Thus, the network was trained and tested 5 times with different sets, and the test result of the network was obtained by averaging the sum of the performances of all test functions. Regression results obtained from ELM are given for different numbers of hidden layer neurons and different types of activation functions in the hidden layer. The best prediction result of ELM was obtained for the case where the hidden layer activation function was tangent sigmoid and the number of hidden layer neurons was 20. The R-values were found to be 1 when the number of hidden layer neurons was 20 and tangent and radial basis activation functions were used. From the results obtained, it has been seen that ELM predicts the output power of the PV panel with very high accuracy. It is concluded that ELM is a useful tool for estimating the PV panel output power.

Neural network optimizer of proportional-integral-differential controller parameters

Wide application of proportional-integral-differential (PID)-regulator in industry requires constant improvement of methods of its parameters adjustment. The paper deals with the issues of optimization of PID-regulator parameters with the use of neural network technology methods. A methodology for choosing the architecture (structure) of neural network optimizer is proposed, which consists in determining the number of layers, the number of neurons in each layer, as well as the form and type of activation function. Algorithms of neural network training based on the application of the method of minimizing the mismatch between the regulated value and the target value are developed. The method of back propagation of gradients is proposed to select the optimal training rate of neurons of the neural network. The neural network optimizer, which is a superstructure of the linear PID controller, allows increasing the regulation accuracy from 0.23 to 0.09, thus reducing the power consumption from 65% to 53%. The results of the conducted experiments allow us to conclude that the created neural superstructure may well become a prototype of an automatic voltage regulator (AVR)-type industrial controller for tuning the parameters of the PID controller.

Evolutionary Perspectives on Neural Network Generations: A Critical Examination of Models and Design Strategies

Abstract: In the last few years, Neural Networks have become more common in different areas due to their ability to learn intricate patterns and provide precise predictions. Nonetheless, creating an efficient neural network model is a difficult task that demands careful thought of multiple factors, such as architecture, optimization method, and regularization technique. This paper aims to comprehensively overview the state-of-the-art artificial neural network (ANN) generation and highlight key challenges and opportunities in machine learning applications. It provides a critical analysis of current neural network model design methodologies, focusing on the strengths and weaknesses of different approaches. Also, it explores the use of different deep neural networks (DNN) in image recognition, natural language processing, and time series analysis. In addition, the text explores the advantages of selecting optimal values for various components of an Artificial Neural Network (ANN). These components include the number of input/output layers, the number of hidden layers, the type of activation function used, the number of epochs, and the model type selection. Setting these components to their ideal values can help enhance the model's overall performance and generalization. Furthermore, it identifies some common pitfalls and limitations of existing design methodologies, such as overfitting, lack of interpretability, and computational complexity. Finally, it proposes some directions for future research, such as developing more efficient and interpretable neural network architectures, improving the scalability of training algorithms, and exploring the potential of new paradigms, such as Spiking Neural Networks, quantum neural networks, and neuromorphic computing.

Floating-Point Quantization Analysis of Multi-Layer Perceptron Artificial Neural Networks

The impact of quantization in Multi-Layer Perceptron (MLP) Artificial Neural Networks (ANNs) is presented in this paper. In this architecture, the constant increase in size and the demand to decrease bit precision are two factors that contribute to the significant enlargement of quantization errors. We introduce an analytical tool that models the propagation of Quantization Noise Power (QNP) in floating-point MLP ANNs. Contrary to the state-of-the-art approach, which compares the exact and quantized data experimentally, the proposed algorithm can predict the QNP theoretically when the effect of operation quantization and Coefficient Quantization Error (CQE) are considered. This supports decisions in determining the required precision during the hardware design. The algorithm is flexible in handling MLP ANNs of user-defined parameters, such as size and type of activation function. Additionally, a simulation environment is built that can perform each operation on an adjustable bit precision. The accuracy of the QNP calculation is verified with two publicly available benchmarked datasets, using the default precision simulation environment as a reference.

Conditional random <i>k</i> satisfiability modeling for <i>k</i> = 1, 2 (CRAN2SAT) with non-monotonic Smish activation function in discrete Hopfield neural network

<abstract> <p>The current development of logic satisfiability in discrete Hopfield neural networks (DHNN)has been segregated into systematic logic and non-systematic logic. Most of the research tends to improve non-systematic logical rules to various extents, such as introducing the ratio of a negative literal and a flexible hybrid logical structure that combines systematic and non-systematic structures. However, the existing non-systematic logical rule exhibited a drawback concerning the impact of negative literal within the logical structure. Therefore, this paper presented a novel class of non-systematic logic called conditional random <italic>k</italic> satisfiability for <italic>k</italic> = 1, 2 while intentionally disregarding both positive literals in second-order clauses. The proposed logic was embedded into the discrete Hopfield neural network with the ultimate goal of minimizing the cost function. Moreover, a novel non-monotonic Smish activation function has been introduced with the aim of enhancing the quality of the final neuronal state. The performance of the proposed logic with new activation function was compared with other state of the art logical rules in conjunction with five different types of activation functions. Based on the findings, the proposed logic has obtained a lower learning error, with the highest total neuron variation <italic>TV</italic> = 857 and lowest average of Jaccard index, <italic>JSI</italic> = 0.5802. On top of that, the Smish activation function highlights its capability in the DHNN based on the result ratio of improvement <italic>Zm</italic> and <italic>TV</italic>. The ratio of improvement for Smish is consistently the highest throughout all the types of activation function, showing that Smish outperforms other types of activation functions in terms of <italic>Zm</italic> and <italic>TV.</italic> This new development of logical rule with the non-monotonic Smish activation function presents an alternative strategy to the logic mining technique. This finding will be of particular interest especially to the research areas of artificial neural network, logic satisfiability in DHNN and activation function.</p> </abstract>

Approximation and interpolation with neural network

In this paper we show that multilayer feedforward networks with one single hidden layer.and certain types of activation functions can approximate univariant continuous functions defined on a compact set. arbitrarily well. In particular, our results contain some usual activation functions such as sigmoidal functions, RELU functions and threshold functions. Besides, since interpolation problems are highly related to approximation problem, we demonstrate that a wide range of functions have the ability to interpolate and generalize our results to functions which are not polynomial on R. Compared to existing results by numerous work, our methods are more intuitive and less technical. Lastly, the paper discusses the possibility of combining interpolation property and approximating property together, and demonstrates that given any Riemann integrable functions on a compact set in R, with several points on its graph, the finite combination of monotone sigmoidal functions can pass through these points and approximate the given function arbitrarily well with respect to L^1 (dx) (in the sense of Riemann integral) when the number of points getting large.

A deep learning framework for solving forward and inverse problems of power-law fluids

We for the first time leverage deep learning approaches to solve forward and inverse problems of two-dimensional laminar flows for power-law fluids. We propose a deep-learning framework, called Power-Law-Fluid-Net (PL-Net). We develop a surrogate model to solve the forward problems of the power-law fluids, and solve the inverse problems utilizing only a small set of measurement data under the assumption that boundary conditions (BCs) can be partially known. In the design of the methods, we incorporate the hard boundary condition constraints to accelerate the iteration of stochastic gradient descent methods for minimizing loss functions. For the forward problems, by incorporating the constitutive parameters into the input variables of neural networks, the PL-Net serves as a surrogate model for simulating the pressure-driven flows inside pipes having cross sections of varying shapes. We investigate the influences of the BC type, activation function type, and number of collocation points on the accuracy of numerical solutions. For the inverse problems, the PL-Net infers the physical quantities or constitutive parameters from a small number of measurements of flow field variables. The BCs of the inverse problems can even be partially known. We demonstrate the effects of BC type, number of sensors, and noise level on accuracy of inferred quantities. Computational examples indicate the high accuracy of the PL-Net in tackling both the forward and inverse problems of the power-law fluids.

Reachability Analysis of Neural Network Control Systems

Neural network controllers (NNCs) have shown great promise in autonomous and cyber-physical systems. Despite the various verification approaches for neural networks, the safety analysis of NNCs remains an open problem. Existing verification approaches for neural network control systems (NNCSs) either can only work on a limited type of activation functions, or result in non-trivial over-approximation errors with time evolving. This paper proposes a verification framework for NNCS based on Lipschitzian optimisation, called DeepNNC. We first prove the Lipschitz continuity of closed-loop NNCSs by unrolling and eliminating the loops. We then reveal the working principles of applying Lipschitzian optimisation on NNCS verification and illustrate it by verifying an adaptive cruise control model. Compared to state-of-the-art verification approaches, DeepNNC shows superior performance in terms of efficiency and accuracy over a wide range of NNCs. We also provide a case study to demonstrate the capability of DeepNNC to handle a real-world, practical, and complex system. Our tool DeepNNC is available at https://github.com/TrustAI/DeepNNC.

Approximation of classifiers by deep perceptron networks

We employ properties of high-dimensional geometry to obtain some insights into capabilities of deep perceptron networks to classify large data sets. We derive conditions on network depths, types of activation functions, and numbers of parameters that imply that approximation errors behave almost deterministically. We illustrate general results by concrete cases of popular activation functions: Heaviside, ramp sigmoid, rectified linear, and rectified power. Our probabilistic bounds on approximation errors are derived using concentration of measure type inequalities (method of bounded differences) and concepts from statistical learning theory.

Deep Learning Optimisation of Static Malware Detection with Grid Search and Covering Arrays

This paper investigates the impact of several hyperparameters on static malware detection using deep learning, including the number of epochs, batch size, number of layers and neurons, optimisation method, dropout rate, type of activation function, and learning rate. We employed the cAgen tool and grid search optimisation from the scikit-learn Python library to identify the best hyperparameters for our Keras deep learning model. Our experiments reveal that cAgen is more efficient than grid search in finding the optimal parameters, and we find that the selection of hyperparameter values has a significant impact on the model’s accuracy. Specifically, our approach leads to significant improvements in the neural network model’s accuracy for static malware detection on the Ember dataset (from 81.2% to 95.7%) and the Kaggle dataset (from 94% to 98.6%). These results demonstrate the effectiveness of our proposed approach, and have important implications for the field of static malware detection.

Extraction and recognition of music melody features using a deep neural network

The music melody can be used to distinguish the genre style of music and can also be used for retrieving music works. This paper used a deep learning algorithm, the convolutional neural network (CNN), to extract the features of musical melodies and recognize genres. Three-tuple samples were used as training samples in the training process. Orthogonal experiments were conducted on the number of music segments and the type of activation function in the algorithm in the simulation experiments. The CNN algorithm was compared with support vector machine (SVM) and traditional CNN algorithms. The results showed that there were obvious differences in the pitch and melody curves of different genres of music; the recognition performance was best when the number of music segments was six and the activation function was relu; the CNN algorithm trained by three-tuple samples had better recognition accuracy and spent less recognition time.

Sliding-Mode Synchronization Control of Complex-Valued Inertial Neural Networks With Leakage Delay and Time-Varying Delays

This work explores the synchronization problem of two nonidentical complex-valued inertial neural networks (CVINNs) considering time-varying delays, leakage delay, and external disturbances. The entire analysis does not use reduced-order conversion, nor does it involve the separation of real and imaginary parts, but directly focuses on the original system. First, an integral sliding-mode surface suitable for the system is proposed. Second, the efficient sliding-mode control laws are designed, under which the state trajectories of the closed-loop dynamic error systems can be driven onto the predefined sliding-mode surface in finite time. Then, not requiring the time-varying delays to be differentiable, by constructing innovative Lyapunov–Krasovskii functionals, the synchronization criteria are obtained in the forms of the linear matrix inequality techniques. Eventually, for the systems with different types of activation functions, the corresponding numerical verification and comparison are carried out.

DNN surrogates for turbulence closure in CFD-based shape optimization

A DNN-based surrogate for turbulence (and transition) closure of the Reynolds-Averaged Navier–Stokes (RANS) equations is presented. The DNN configuration, namely the hyperparameters (number of layers, number of neurons, type of activation functions) and the DNN input data result from a Metamodel-Assisted Evolutionary Algorithm (MAEA)-based optimization. The trained DNN replaces the numerical solution of the turbulence and/or transition model PDEs during the solution of the RANS equations, by estimating the necessary turbulent viscosity field in each pseudo-time step iteration. The gain from using such a computational tool becomes pronounced in cases in which many calls to the CFD tools are necessary; a typical example is CFD-based (shape) optimization. Thus, the new model (abbreviated to RANS-DNN) is used as the evaluation software in MAEA-based shape optimization problems. Three aerodynamic cases covering a wide gamut of applications from 2D to 3D, internal to external and compressible to incompressible flows are selected to demonstrate the capabilities of the RANS-DNN model. Using the RANS-DNN instead of the standard RANS solver a significant reduction in the optimization turnaround time is achieved in cases dealing with an isolated airfoil, a turbomachinery cascade and a car.

A Method for Calculating the Derivative of Activation Functions Based on Piecewise Linear Approximation

Nonlinear functions are widely used as activation functions in artificial neural networks, which have a great impact on the fitting ability of artificial neural networks. Due to the complexity of the activation function, the computation of the activation function and its derivative requires a lot of computing resources and time during training. In order to improve the computational efficiency of the derivatives of the activation function in the back-propagation of artificial neural networks, this paper proposes a method based on piecewise linear approximation method to calculate the derivative of the activation function. This method is hardware-friendly and universal, it can efficiently compute various nonlinear activation functions in the field of neural network hardware accelerators. In this paper, we use least squares to improve a piecewise linear approximation calculation method that can control the absolute error and get less number of segments or smaller average error, which means fewer hardware resources are required. We use this method to perform a segmented linear approximation to the original or derivative function of the activation function. Both types of activation functions are substituted into a multilayer perceptron for binary classification experiments to verify the effectiveness of the proposed method. Experimental results show that the same or even slightly higher classification accuracy can be achieved by using this method, and the computation time of the back-propagation is reduced by 4–6% compared to the direct calculation of the derivative directly from the function expression using the operator encapsulated in PyTorch. This shows that the proposed method provides an efficient solution of nonlinear activation functions for hardware acceleration of neural networks.

Global attractive set of neural networks with neutral item

This paper investigates the global attractive set of neural networks with neutral item. To better deal with the neutral terms, different types of activation functions are considered. Based on matrix measures, inequality techniques, and Lyapunov theory, three new types of Lyapunov functions are designed to find the global attractive set of the system. We give out a simulation example to verify the validity of theory results. The result is very inclusive, whether the system has equilibrium or not. As long as the system is stable, we can find its global attractive set.

A Study on a Probabilistic Method for Designing Artificial Neural Networks for the Formation of Intelligent Technology Assemblies with High Variability

Currently, ensemble approaches based, among other things, on the use of non-network models are powerful tools for solving data analysis problems in various practical applications. An important problem in the formation of ensembles of models is ensuring the synergy of solutions by using the properties of a variety of basic individual solutions; therefore, the problem of developing an approach that ensures the maintenance of diversity in a preliminary pool of models for an ensemble is relevant for development and research. This article is devoted to the study of the possibility of using a method for the probabilistic formation of neural network structures developed by the authors. In order to form ensembles of neural networks, the influence of parameters of neural network structure generation on the quality of solving regression problems is considered. To improve the quality of the overall ensemble solution, using a flexible adjustment of the probabilistic procedure for choosing the type of activation function when filling in the layers of a neural network is proposed. In order to determine the effectiveness of this approach, a number of numerical studies on the effectiveness of using neural network ensembles on a set of generated test tasks and real datasets were conducted. The procedure of forming a common solution in ensembles of neural networks based on the application of an evolutionary method of genetic programming is also considered. This article presents the results of a numerical study that demonstrate a higher efficiency of the approach with a modified structure formation procedure compared to a basic approach of selecting the best individual neural networks from a preformed pool. These numerical studies were carried out on a set of test problems and several problems with real datasets that, in particular, describe the process of ore-thermal melting.

Effect of layer-specific synaptic retention characteristics on the accuracy of deep neural networks

By modeling the change of weight/bias over time due to the retention behavior of charge trap device (CTD), we study the influence of synaptic retention characteristics on the inference accuracy of the deep neural network (DNN) considering the activation function and neural network type. After training the neural network composed of 3 fully-connected (FC) layers for the MNIST test and the VGG16 neural network for the CIFAR 10 test, the performance of neural networks is researched by changing weight/bias over time. The performance degradation of a neural network as weights/bias change over time depends on the type of activation function. Sensitivities to weight/bias loss differ depending on the depth of neural networks or the type of network layers.