Selection Of Activation Function Research Articles

Tipping prediction of a class of large-scale radial-ring neural networks

Understanding the emergence and evolution of collective dynamics in large-scale neural networks remains a complex challenge. This paper seeks to address this gap by applying dynamical systems theory, with a particular focus on tipping mechanisms. First, we introduce a novel (n+mn)-scale radial-ring neural network and employ Coates’ flow graph topological approach to derive the characteristic equation of the linearized network. Second, through deriving stability conditions and predicting the tipping point using an algebraic approach based on the integral element concept, we identify critical factors such as the synaptic transmission delay, the self-feedback coefficient, and the network topology. Finally, we validate the methodology’s effectiveness in predicting the tipping point. The findings reveal that increased synaptic transmission delay can induce and amplify periodic oscillations. Additionally, the self-feedback coefficient and the network topology influence the onset of tipping points. Moreover, the selection of activation function impacts both the number of equilibrium solutions and the convergence speed of the neural network. Lastly, we demonstrate that the proposed large-scale radial-ring neural network exhibits stronger robustness compared to lower-scale networks with a single topology. The results provide a comprehensive depiction of the dynamics observed in large-scale neural networks under the influence of various factor combinations.

An Accelerated Dual-Integral Structure Zeroing Neural Network Resistant to Linear Noise for Dynamic Complex Matrix Inversion

The problem of inverting dynamic complex matrices remains a central and intricate challenge that has garnered significant attention in scientific and mathematical research. The zeroing neural network (ZNN) has been a notable approach, utilizing time derivatives for real-time solutions in noiseless settings. However, real-world disturbances pose a significant challenge to a ZNN’s convergence. We design an accelerated dual-integral structure zeroing neural network (ADISZNN), which can enhance convergence and restrict linear noise, particularly in complex domains. Based on the Lyapunov principle, theoretical analysis proves the convergence and robustness of ADISZNN. We have selectively integrated the SBPAF activation function, and through theoretical dissection and comparative experimental validation we have affirmed the efficacy and accuracy of our activation function selection strategy. After conducting numerous experiments, we discovered oscillations and improved the model accordingly, resulting in the ADISZNN-Stable model. This advanced model surpasses current models in both linear noisy and noise-free environments, delivering a more rapid and stable convergence, marking a significant leap forward in the field.

Geographic heterogeneity of activation functions in urban real-time flood forecasting: Based on seasonal trend decomposition using Loess-Temporal Convolutional Network-Gated Recurrent Unit model

Urban real-time flood forecasting is crucial for flood prevention and sustainable development, but it poses challenges due to data inputs and activation functions selection in data-driven models without sufficient focus of geographic heterogeneity. In this study, a novel Seasonal Trend Decomposition using Loess (STL)-Temporal Convolutional Network (TCN)-Gated Recurrent Unit (GRU) model is proposed to improve urban real-time flood forecasting accuracy, twenty-one different activation functions are considered for geographic heterogeneity. Experiments are conducted at six urban drainage system locations in Odense, Denmark. The results show that: (1) STL effectively prepares data for forecasting using TCN and GRU models, leading improved performance compared to single models. STL-TCN-GRU deep learning model demonstrates strong applicability in urban forecasting, achieving an overall accuracy of 0.0079, 0.0140, 0.0482 and 0.9581 in Mean Absolute Error (MAE), Root Mean Square Error (RMSE), Mean Absolute Percentage Error (MAPE) and Nash-Sutcliffe efficiency coefficient (NSE), respectively. (2) Softsign emerges as the best activation function for forecasting in lower drainage system locations, with average accuracy of 0.0094, 0.0018, 0.0049 and 0.9938 in MAE, RMSE, MAPE and NSE, respectively. Furthermore, Softsign proves to be the best activation function for forecasting in middle drainage system locations, with average accuracy of 0.0047, 0.0081, 0.0309 and 0.9547 in MAE, RMSE, MAPE and NSE, respectively. Swish is the best activation function for forecasting in upper drainage system locations, with average accuracy of 0.0052, 0.0080, 0.0627 and 0.9060 in MAE, RMSE, MAPE and NSE, respectively. This study provides valuable insights for urban real-time flood forecasting modeling with high accuracy and evidence for activation function selection in data-driven models like STL-TCN-GRU for geographic heterogeneity.

Research on Pedestrian Re-identification Techniques in Dynamic Scenes Using Convolutional Neural Networks

Abstract Pedestrian re-recognition is the process of retrieving pedestrians with the same identity information as a given pedestrian from a cross-domain view candidate image dataset or a non-overlapping surveillance video sequence using computer vision techniques. The goal of this paper is to use convolutional neural network techniques to re-recognize pedestrians in dynamic scenes. Through the use of convolutional calculations, activation function selection, and other techniques, this paper provides basic technical support for the research of pedestrian re-recognition technology. A Siamese network is obtained by applying convolutional neural networks to pedestrian recognition as the main discriminative model for subsequent research. In order to effectively solve the problem of occlusion, illumination, and other possible interference with the recognition effect in dynamic scenes, this paper adopts the image enhancement method of random erasure and introduces the attention mechanism to improve the robustness of the model to the occlusion of pedestrian images. Through the examination of the model on the dataset of the average accuracy mean (MAP) and other indicators and the actual application in the construction site and other scenes, it is proved that the pedestrian re-recognition model used in this paper has a more significant recognition performance compared with other models, and can still maintain more than 80% of the accuracy rate under the application of dynamic and complex scenes.

A New Method for Calculating Prediction Parameters of Surface Deformation in the Mining Area

The accurate calculation of mining-induced surface deformation has important guiding significance for efficient and safe production in mining areas. The probability integral method (PIM) is a main prediction method in China, and the selection of its parameters is directly related to the prediction accuracy of surface deformation in mining areas. To overcome shortcomings of PIM and other methods, this paper proposed a prediction model of the parameters of PIM combining a multiple regression model and an extreme learning machine. In this paper, the Huainan mining area was selected as the research object, the influence factors of PIM parameters were analyzed and the accuracy of the model was verified. The influence of the number of hidden layer nodes, the selection of activation function and the proportion of training set and test set in the model were analyzed. The conclusions suggest that the PIM parameters calculated in this paper could be used to predict mining subsidence and obtain surface movement and deformation data. The research results provide an effective method for the selection of surface deformation prediction parameters of new working faces or faces lacking measured data.

The effect of activation functions on accuracy, convergence speed, and misclassification confidence in CNN text classification: a comprehensive exploration

Convolutional neural networks (CNNs) have become a useful tool for a wide range of applications such as text classification. However, CNNs are not always sufficiently accurate to be useful in certain applications. The selection of activation functions within CNN architecture can affect the efficacy of the CNN. However, there is limited research regarding which activation functions are best for CNN text classification. This study tested sixteen activation functions across three text classification datasets and six CNN structures, to determine the effects of activation function on accuracy, iterations to convergence, and Positive Confidence Difference (PCD). PCD is a novel metric introduced to compare how activation functions affected a network’s classification confidence. Tables were presented to compare the performance of the activation functions across the different CNN architectures and datasets. Top performing activation functions across the different tests included the symmetrical multi-state activation function, sigmoid, penalised hyperbolic tangent, and generalised swish. An activation function’s PCD was the most consistent evaluation metric during activation function assessment, implying a close relationship between activation functions and network confidence that has yet to be explored.

Soybean Leaf Disease Detection

Abstract: With the continuous integration of computer technology into agricultural production, it also reduces personnel costs while improving agricultural production efficiency and quality. Crop disease control is an important part of agricultural production, and the use of computer vision technology to quickly and accurately identify crop diseases is an important means of ensuring a good harvest of agricultural products and promoting agricultural modernization. In this paper, a recognition method based on deep learning is proposed based on soybean brown spot. The method is divided into image pre- treatment and disease identification. Based on traditional threshold segmentation, the pre-processing process first uses the HSI colour space to filter the information of the normal area of the leaf, adopts OTSU to set the threshold to segment the original image under the Lab colour space, and then merges the segmented images. The final spot segmentation image is obtained. Compared with the renderings of several other commonly used methods of segmentation, this method can better separate the lesions from the leaves. In terms of disease identification, in order to adapt to the working conditions of large samples of farmland operations, a convolutional neural network (CNN) of continuous convolutional layers was constructed with the help of Caffe to extract more advanced features of the image. In the selection of activation functions, this paper selects the Max out unit with stronger fitting ability, and in order to reduce the parameters in the network and prevent the network from overfitting, the sparse Max out unit is used, which effectively improves the performance of the Max out convolutional neural network. The experimental results show that the algorithm is superior to the algorithm based on ordinary convolutional neural network in identifying large sample crop diseases.

A Modified Particle Swarm Optimization Algorithm for Optimizing Artificial Neural Network in Classification Tasks

Artificial neural networks (ANNs) have achieved great success in performing machine learning tasks, including classification, regression, prediction, image processing, image recognition, etc., due to their outstanding training, learning, and organizing of data. Conventionally, a gradient-based algorithm known as backpropagation (BP) is frequently used to train the parameters’ value of ANN. However, this method has inherent drawbacks of slow convergence speed, sensitivity to initial solutions, and high tendency to be trapped into local optima. This paper proposes a modified particle swarm optimization (PSO) variant with two-level learning phases to train ANN for image classification. A multi-swarm approach and a social learning scheme are designed into the primary learning phase to enhance the population diversity and the solution quality, respectively. Two modified search operators with different search characteristics are incorporated into the secondary learning phase to improve the algorithm’s robustness in handling various optimization problems. Finally, the proposed algorithm is formulated as a training algorithm of ANN to optimize its neuron weights, biases, and selection of activation function based on the given classification dataset. The ANN model trained by the proposed algorithm is reported to outperform those trained by existing PSO variants in terms of classification accuracy when solving the majority of selected datasets, suggesting its potential applications in challenging real-world problems, such as intelligent condition monitoring of complex industrial systems.

An improved deep extreme learning machine to predict the remaining useful life of lithium-ion battery

To aim at the problem of inaccurate prediction of the remaining useful life of the lithium-ion battery, an improved grey wolf optimizer optimizes the deep extreme learning machine (CGWO-DELM) data-driven forecasting method is proposed. This method uses the grey wolf optimization algorithm based on an adaptive normal cloud model to optimize the bias of the deep extreme learning machine, the weight of the input layer, the selection of activation function, and the number of hidden layer nodes. In this article, indirect health factors that can characterize the degradation of battery performance are extracted from the discharge process, and the correlation between them and capacity is analyzed using the Pearson coefficient and Kendel coefficient. Then, the CGWO-DELM prediction model is constructed to predict the capacitance of the lithium-ion battery. The remaining useful life of lithium-ion batteries is indirectly predicted with a 1.44 A·h failure threshold. The prediction results are compared with deep extreme learning machines, long-term memory, other prediction methods, and the current public prediction methods. The results show that the CGWO-DELM prediction method can more accurately predict the remaining useful life of lithium-ion batteries.

Research on Soybean Disease Identification Method Based on Deep Learning

With the continuous integration of computer technology into agricultural production, it also reduces personnel costs while improving agricultural production efficiency and quality. Crop disease control is an important part of agricultural production, and the use of computer vision technology to quickly and accurately identify crop diseases is an important means of ensuring a good harvest of agricultural products and promoting agricultural modernization. In this paper, a recognition method based on deep learning is proposed based on soybean brown spot. The method is divided into image pretreatment and disease identification. Based on traditional threshold segmentation, the preprocessing process first uses the HSI color space to filter the information of the normal area of the leaf, adopts OTSU to set the threshold to segment the original image under the Lab color space, and then merges the segmented images. The final spot segmentation image is obtained. Compared with the renderings of several other commonly used methods of segmentation, this method can better separate the lesions from the leaves. In terms of disease identification, in order to adapt to the working conditions of large samples of farmland operations, a convolutional neural network (CNN) of continuous convolutional layers was constructed with the help of Caffe to extract more advanced features of the image. In the selection of activation functions, this paper selects the Maxout unit with stronger fitting ability, and in order to reduce the parameters in the network and prevent the network from overfitting, the sparse Maxout unit is used, which effectively improves the performance of the Maxout convolutional neural network. The experimental results show that the algorithm is superior to the algorithm based on ordinary convolutional neural network in identifying large sample crop diseases.

Deep learning method for 3-DOF motion prediction of unmanned surface vehicles based on real sea maneuverability test

Motion prediction in a real sea area is a relatively difficult problem due to the lack of reliable measurement of the environment and an accurate model of ship motion under external disturbance. A deep learning method is applied in this study for the modeling of 3-DOF motion prediction under a real sea environment with the influence of currents and waves. The network with angle feature and the network with trigonometric feature are validated for the identification of the influence of the environment regarding winds, waves, and currents, in which the selection of activation function is further discussed. The developed method is validated by learning the data generated through the turning test of JARI-USV in a real sea area in Rizhao, Shandong. Analysis on the turning radius and the drift distance of the integrated track shows maximum relative errors of 5.44% and 9.09%, and the derived drift angle has a maximum absolute error of 4.84°. The prediction results verify the ability of the designed network to capture the environment influence on ship motion, as well as the turning feature of the ship itself from the maneuvering test data in a real sea trial.

A Deep Learning Modeling Framework to Capture Mixing Patterns in Reactive-Transport Systems

Prediction and control of chemical mixing are vital for many scientific areas such as subsurface reactive transport, climate modeling, combustion, epidemiology, and pharmacology. Due to the complex nature of mixing in heterogeneous and anisotropic media, the mathematical models related to this phenomenon are not analytically tractable. Numerical simulations often provide a viable route to predict chemical mixing accurately. However, contemporary modeling approaches for mixing cannot utilize available spatial-temporal data to improve the accuracy of the future prediction and can be compute-intensive, especially when the spatial domain is large and for long-term temporal predictions. To address this knowledge gap, we will present in this paper a deep-learning (DL) modeling framework applied to predict the progress of chemical mixing under fast bimolecular reactions. This framework uses convolutional neural networks (CNN) for capturing spatial patterns and long short-term memory (LSTM) networks for forecasting temporal variations in mixing. By careful design of the framework -- placement of non-negative constraint on the weights of the CNN and the selection of activation function, the framework ensures non-negativity of the chemical species at all spatial points and for all times. Our DL-based framework is fast, accurate, and requires minimal data for training.

Neural Networks with À La Carte Selection of Activation Functions

Activation functions (AFs), which are pivotal to the success (or failure) of a neural network, have received increased attention in recent years, with researchers seeking to design novel AFs that improve some aspect of network performance. In this paper we take another direction, wherein we combine a slew of known AFs into successful architectures, proposing three methods to do so beneficially: 1) generate AF architectures at random, 2) use Optuna, an automatic hyper-parameter optimization software framework, with a Tree-structured Parzen Estimator (TPE) sampler, and 3) use Optuna with a Covariance Matrix Adaptation Evolution Strategy (CMA-ES) sampler. We show that all methods often produce significantly better results for 25 classification problems when compared with a standard network composed of ReLU hidden units and a softmax output unit. Optuna with the TPE sampler emerged as the best AF architecture-producing method.

Investigating Neural Activation Effects on Deep Belief Echo-State Networks for Prediction Toward Smart Ocean Environment Monitoring

Ocean sensor data prediction has become a promising means for smart ocean monitoring. In alternative solutions, deep neural networks (DNNs) are considered as a good choice. The determination of activation functions in DNNs has a significant effect on training speed and nonlinear approximation. In this paper, the effect of activation functions on a deep computing model called deep belief echo-state network (DBEN) is studied in the scenario of ocean time series prediction. Here, different forms, including hyperbolic tangent, rectified linear unit, exponential linear unit, swish, softplus and their variants, are considered. The purpose is to investigate, from the perspectives of accuracy and training efficiency, whether certain activation function in DBEN is completely universal for the different tasks of ocean sensor data processing or not. On a great deal of real-world ocean time series of different characteristics, the results show that the selection of activation functions in DBEN is task-related. Specially, these newly introduced activation functions are more beneficial to the accurate predictions for conventional and chemical data sets compared with sigmoid benchmark. The statistical analysis further verifies this finding.

A Activation Function Selection of CNN for Inductive Motor Static Fault Diagnosis

A Activation Function Selection of CNN for Inductive Motor Static Fault Diagnosis

RETRACTED: Identification of microbial pollution sources in high-humidity buildings based on deep learning and image particle recognition

The presence of microorganisms in the air poses challenges to the hospital's control, commercial Indoor Air Quality (IAQ), and residential building projects. This work analyzes microbial control agencies for power generation, transmission, and air and proposes engineering control methods for ventilation commonly used to control microbial contamination of indoor air. Mathematical computational models of building ventilation systems are used to study microbial contamination control. Alone, the encompassing first-individual picture can be semantically connected with comparative circumstances before, individual chronicle space, and social design representation. All in all, first-individual recordings permit you to follow regular interests and connections to gatherings of people. It learns to use FPGAs to capture the egocentrism and visual implications of future orbital motor groups. Based on Deep Neural Network (DNN), deep learning architectures have successfully image recognition tasks of microbial particles. These architectures use a cascade of convolutional layers and activation functions. The number of layers for training, the number of neurons in each layer, the selection of activation functions, and the setting of optimization algorithms are significant. DNNs have been used to build cognitive feature extraction implementations and have been reported to achieve good results.

Associated Learning: Decomposing End-to-End Backpropagation Based on Autoencoders and Target Propagation.

Backpropagation (BP) is the cornerstone of today's deep learning algorithms, but it is inefficient partially because of backward locking, which means updating the weights of one layer locks the weight updates in the other layers. Consequently, it is challenging to apply parallel computing or a pipeline structure to update the weights in different layers simultaneously. In this letter, we introduce a novel learning structure, associated learning (AL), that modularizes the network into smaller components, each of which has a local objective. Because the objectives are mutually independent, AL can learn the parameters in different layers independently and simultaneously, so it is feasible to apply a pipeline structure to improve the training throughput. Specifically, this pipeline structure improves the complexity of the training time from , which is the time complexity when using BP and stochastic gradient descent (SGD) for training, to , where is the number of training instances and is the number of hidden layers. Surprisingly, even though most of the parameters in AL do not directly interact with the target variable, training deep models by this method yields accuracies comparable to those from models trained using typical BP methods, in which all parameters are used to predict the target variable. Consequently, because of the scalability and the predictive power demonstrated in the experiments, AL deserves further study to determine the better hyperparameter settings, such as activation function selection, learning rate scheduling, and weight initialization, to accumulate experience, as we have done over the years with the typical BP method. In addition, perhaps our design can also inspire new network designs for deep learning. Our implementation is available at https://github.com/SamYWK/Associated_Learning.

Evolutionary Extreme Learning Machine with novel activation function for credit scoring

The term credit scoring is extensively used in credit industries for decision making and measuring the risk associated with an applicant. It uses applicants’ historical data for credit risk evaluation by applying machine learning or statistical techniques. Credit risk evaluation has become progressively important field in financial risk management for credit industries. In this study, Extreme Learning Machine (ELM) is utilized as a classification tool for credit risk evaluation model. ELM requires more number of hidden neurons and random determination of the input weights and hidden biases. Moreover, ELM performance depends on activation function, weights and biases assigned to hidden neurons. An appropriate approach for selection of activation function, weights and biases may improve the performance of ELM. Hence, we have proposed a novel activation function and an evolutionary approach to get optimized weights and biases by utilizing Bat optimization algorithm. Further, the simulations are performed on four bench-marked credit scoring datasets with various activation functions. Simulation results demonstrate that proposed Evolutionary ELM (EELM) is more suitable for credit risk evaluation.

A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks.

The rapid development of deep neural networks (DNNs) in recent years can be attributed to the various techniques that address gradient explosion and vanishing. In order to understand the principle behind these techniques and develop new methods, plenty of metrics have been proposed to identify networks that are free of gradient explosion and vanishing. However, due to the diversity of network components and complex serial-parallel hybrid connections in modern DNNs, the evaluation of existing metrics usually requires strong assumptions, complex statistical analysis, or has limited application fields, which constraints their spread in the community. In this paper, inspired by the Gradient Norm Equality and dynamical isometry, we first propose a novel metric called Block Dynamical Isometry, which measures the change of gradient norm in individual blocks. Because our Block Dynamical Isometry is norm-based, its evaluation needs weaker assumptions compared with the original dynamical isometry. To mitigate challenging derivation, we propose a highly modularized statistical framework based on free probability. Our framework includes several key theorems to handle complex serial-parallel hybrid connections and a library to cover the diversity of network components. Besides, several sufficient conditions for prerequisites are provided. Powered by our metric and framework, we analyze extensive initialization, normalization, and network structures. We find that our Block Dynamical Isometry is a universal philosophy behind them. Then, we improve some existing methods based on our analysis, including an activation function selection strategy for initialization techniques, a new configuration for weight normalization, a depth-aware way to derive coefficients in SeLU, and initialization/weight normalization in DenseNet. Moreover, we propose a novel normalization technique named second moment normalization, which has 30 percent fewer computation overhead than batch normalization without accuracy loss and has better performance under micro batch size. Last but not least, our conclusions and methods are evidenced by extensive experiments on multiple models over CIFAR-10 and ImageNet.

An Optimal NARX Neural Network Identification Model for a Magnetorheological Damper With Force-Distortion Behavior

This paper presents an optimal NARX neural network identification model for a magnetorheological (MR) damper with the force-distortion behavior. An intensive experimental study is conducted for designing the NARX network architecture to enhance modeling accuracy and availability, and the activation function selection, weights and biases of the selected network are optimized by differential evolution algorithm (DEA). Different experimental training and validation samples are used for network training. The prediction capability of the optimal NARX model is verified by new measured test data. The test and comparative results show that the optimal NARX network model can satisfactorily emulate the dynamic behavior of MR damper and effectively capture its distortion behavior occurred with the increased current. The developed inverse NARX network model can effectively estimate the required current and track desired damping force. Moreover, the effects of different noise disturbance on the NARX network model performance are analyzed, and the model error varies slightly with a small noise disturbance. The accuracy of the results supports that the use of this modelling technique for identifying irregular nonlinear models of MR damper and similar devices.