Types Of Activation Functions Research Articles

Observer-based sliding mode synchronization control of complex-valued neural networks with inertial term and mixed time-varying delays

In this paper, the synchronization problem of complex-valued inertial neural networks is studied via sliding mode control (SMC). Both mixed time-varying delays and unknown control disturbances are considered. In the absence of the equivalent transformations of real- and complex-valued systems, the systems are analyzed as an entirety form in complex domain. A disturbance observer is designed to estimate the unknown control disturbance. By constructing suitable integral switching surface function and innovative Lyapunov–Krasovskii functionals, a delay-dependent synchronization criterion is proposed on the basis of linear matrix inequality technique. An efficient SMC law based on the disturbance observer is designed, and the accessibility analysis of the predefined switching surface is provided. Eventually, numerical verification based on two types of activation functions, as well as the superiority and practicability analysis are provided.

ANN Activation Function Comparative Study for Sinusoidal Data

An artificial neural network (ANN) has neurons that are interconnected. Inside the neuron, there is an activation function that determines the value issued by the neuron. There are several types of activation functions. The choice of activation function will determine the result of an ANN. This paper describes the performance comparison of several activation functions used in an ANN in processing sinusoidal signals. There are three types of activation functions being compared, those are Sigmoid, Tansig, and ReLU. The sinusoidal dataset has come from simulation data of the PMSM FOC control process. The results indicated that the Tansig activation function is the best choice for sinusoid data.

Hyperparameter Sensitivity Analysis of Deep Learning-Based Pipe Burst Detection Model for Multiregional Water Supply Networks

In a deep learning model, the effect of the model may vary depending on the setting of the hyperparameters. Despite the importance of such hyperparameter determination, most previous studies related to burst detection models of the water supply pipe network used hyperparameters applied in other fields as-is or made a trial-and-error setting based on experience, which is a limitation. In this paper, a study was conducted on the deep learning hyperparameter determination of a deep neural network (DNN)-based real-time detection model of pipe burst accidents. The pipe burst model predicted water pressure by using operation data in units of 1 min, and the data period applied for the model training was less than 1 month (1, 2, and 3 weeks) in order to consider frequent changes in the system. A sensitivity analysis was first performed on the type of activation function and the period of the learning data, which may have different effects depending on the characteristics of the target problem. The number of hidden layers related to the network structure and the number of neurons in each hidden layer were set as hyperparameters for additional sensitivity analysis. The sensitivity analysis results were derived and compared using four quantified prediction error indicators. In addition, the model running time was analyzed to evaluate the practical applicability of the development model. From the results, it was confirmed that excellent effects could be expected if using a rectifier function as the activation function, 144 nodes in the hidden layer, which is eight times the number of nodes in the input layer, and four hidden layers. Additionally, by analyzing the appropriate period of training data required for model pressure prediction through prediction error and driving time, it was confirmed that it was most appropriate to use the data of two weeks. By applying the hyperparameter values determined through detailed sensitivity analysis and by applying the data of one week including actual burst accidents to the built-up pressure prediction model, the accident detection and predictive performance of the model were verified. The rational determination of the period of input factors for the optimal hyperparameter setting and model building, as in this study, is very necessary and very important as it can serve to ensure the continuity of the operation effects of the deep learning model.

Predicting dry matter intake in Canadian Holstein dairy cattle using milk mid-infrared reflectance spectroscopy and other commonly available predictors via artificial neural networks.

Dry matter intake (DMI) is a fundamental component of the animal's feed efficiency, but measuring DMI of individual cows is expensive. Mid-infrared reflectance spectroscopy (MIRS) on milk samples could be an inexpensive alternative to predict DMI. The objectives of this study were (1) to assess if milk MIRS data could improve DMI predictions of Canadian Holstein cows using artificial neural networks (ANN); (2) to investigate the ability of different ANN architectures to predict unobserved DMI; and (3) to validate the robustness of developed prediction models. A total of 7,398 milk samples from 509 dairy cows distributed over Canada, Denmark, and the United States were analyzed. Data from Denmark and the United States were used to increase the training data size and variability to improve the generalization of the prediction models over the lactation. For each milk spectra record, the corresponding weekly average DMI (kg/d), test-day milk yield (MY, kg/d), fat yield (FY, g/d), and protein yield (PY, g/d), metabolic body weight (MBW), age at calving, year of calving, season of calving, days in milk, lactation number, country, and herd were available. The weekly average DMI was predicted with various ANN architectures using 7 predictor sets, which were created by different combinations MY, FY, PY, MBW, and MIRS data. All predictor sets also included age of calving and days in milk. In addition, the classification effects of season of calving, country, and lactation number were included in all models. The explored ANN architectures consisted of 3 training algorithms (Bayesian regularization, Levenberg-Marquardt, and scaled conjugate gradient), 2 types of activation functions (hyperbolic tangent and linear), and from 1 to 10 neurons in hidden layers). In addition, partial least squares regression was also applied to predict the DMI. Models were compared using cross-validation based on leaving out 10% of records (validation A) and leaving out 10% of cows (validation B). Superior fitting statistics of models comprising MIRS information compared with the models fitting milk, fat and protein yields suggest that other unknown milk components may help explain variation in weekly average DMI. For instance, using MY, FY, PY, and MBW as predictor variables produced a predictive accuracy (r) ranging from 0.510 to 0.652 across ANN models and validation sets. Using MIRS together with MY, FY, PY, and MBW as predictors resulted in improved fitting (r = 0.679-0.777). Including MIRS data improved the weekly average DMI prediction of Canadian Holstein cows, but it seems that MIRS predicts DMI mostly through its association with milk production traits and its utility to estimate a measure of feed efficiency that accounts for the level of production, such as residual feed intake, might be limited and needs further investigation. The better predictive ability of nonlinear ANN compared with linear ANN and partial least squares regression indicated possible nonlinear relationships between weekly average DMI and the predictor variables. In general, ANN using Bayesian regularization and scaled conjugate gradient training algorithms yielded slightly better weekly average DMI predictions compared with ANN using the Levenberg-Marquardt training algorithm.

Eksperimen Pengujian Optimizer dan Fungsi Aktivasi Pada Code Clone Detection dengan Pemanfaatan Deep Neural Network (DNN)

The problem of similarity (similarity) of program code can be one solution to the plagiarism detection approach. Plagiarism raises a form of action and consequences of plagiarism itself if the source used is not open source. Plagiarism is an act of deception of the work of others without the knowledge of the original author, which violates a Copyright and Moral Rights. With the increasing amount of data and data complexity, deep learning provides solutions for predictive analytics, with increased processing capabilities and optimal processor utilization. Deep learning shows success and improves the classification model in this field. On the other hand, clone detection code with massive, varied and high-speed data volumes requires feature extraction. With the potential of deep learning to extract better features, deep learning techniques are suitable for code clone detection. For this reason, it is necessary to develop a clone detection code that can process data from a programming language by utilizing deep learning. Based on the results of experiments conducted on 100 PHP program code data files, experimented with several types of activation function and optimizer methods. The average value of the resulting accuracy is good. With a variety of activation functions that we use such as Relu, Linear, Sigmoid, Softmax, Tanh, Elu, Selu, Softplus, Softsign, hard, and sigmoid, as well as variations of the optimizer used are Adagrad, RMSProp, SGD, Adadelta, Adam, Adamax and Nadam , the best attribute selection is in the Selu function and the RMSProp optimizer. The number of epochs used is 1000, the number of neurons per layer is 500 and the best number of hidden layers is 10, the average accuracy is 0.900

APPLICATION OF THE EXTREME LEARNING MACHINE (ELM) METHOD IN PREDICTING THE COMBINED STOCK PRICE INDEX (IHSG) IN INDONESIA

The stock market is one area that continues to attract the attention of investors and financial researchers. This research explores the application of the Extreme Learning Machine (ELM) method to predict the Composite Stock Price Index (IHSG) in Indonesia. ELM is known for its fast learning capabilities with minimal prerequisite network architecture. In this research, three types of activation functions, namely Sigmoid, ReLU, and Tanh, are applied to ELM to compare their performance in predicting IHSG. Monthly IHSG data is used for model training and testing. Data preprocessing steps, such as dividing the data into Training and Test sets, are applied before feeding it into the model. Model performance was evaluated using Root Mean Square Error (RMSE) and compared for each activation function. The research results show that each activation function has a different impact on the IHSG prediction performance. In this research, the ReLU activation function showed the best performance in predicting IHSG compared to other activation functions, with a Root Mean Square Error (RMSE) of 1 x 10-16. These results show that the model's predictive performance in estimating actual values is very good.

Operation-mode recognition of surface microdischarge based on visible image and deep learning

Discharging images contain useful information regarding the operation mode of surface microdischarge (SMD). To solve the shortcomings of low efficiency, high cost, and long operation time of existing SMD operation-mode recognition methods, a convolutional neural network (CNN) based on deep learning is introduced herein. The visible image library of SMD at different applied voltages, dielectric sheets with different dielectric constants, and dielectric sheets with different thicknesses and exposure times are constructed using a digital camera. The typical structure of a CNN is discussed, and the hyperparameters, including the number of network layers, convolution kernel size, number of neurons in the fully connected layer, and activation function type that affect the recognition accuracy of the CNN are investigated. The optimal structure of the CNN for the SMD operation-mode recognition is obtained via training. The recognition accuracy of the CNN is compared with those of three traditional machine learning methods: support vector machine (SVM), decision tree (DT), and random forest (RF). Test results show that the recognition accuracy based on the CNN is 99.745%, which is better than those of the SVM, DT, and RF. Finally, an SMD operation-mode online recognition method based on the CNN is proposed.

A hybrid extreme learning machine model with harris hawks optimisation algorithm: an optimised model for product demand forecasting applications

Accurate and real-time product demand forecasting is the need of the hour in the world of supply chain management. Predicting future product demand from historical sales data is a highly non-linear problem, subject to various external and environmental factors. In this work, we propose an optimised forecasting model - an extreme learning machine (ELM) model coupled with the Harris Hawks optimisation (HHO) algorithm to forecast product demand in an e-commerce company. ELM is preferred over traditional neural networks mainly due to its fast computational speed, which allows efficient demand forecasting in real-time. Our ELM-HHO model performed significantly better than ARIMA models that are commonly used in industries to forecast product demand. The performance of the proposed ELM-HHO model was also compared with traditional ELM, ELM auto-tuned using Bayesian Optimisation (ELM-BO), Gated Recurrent Unit (GRU) based recurrent neural network and Long Short Term Memory (LSTM) recurrent neural network models. Different performance metrics, i.e., Root Mean Squared Error (RMSE), Mean Absolute Percentage Error (MAPE) and Mean Percentage Error (MPE) were used for the comparison of the selected models. Horizon forecasting at 3 days and 7 days ahead was also performed using the proposed approach. The results revealed that the proposed approach is superior to traditional product demand forecasting models in terms of prediction accuracy and it can be applied in real-time to predict future product demand based on the previous week’s sales data. In particular, considering RMSE of forecasting, the proposed ELM-HHO model performed 62.73% better than the statistical ARIMA(7,1,0) model, 40.73% better than the neural network based GRU model, 34.05% better than the neural network based LSTM model, 27.16% better than the traditional non-optimised ELM model with 100 hidden nodes and 11.63% better than the ELM-BO model in forecasting product demand for future 3 months. The novelty of the proposed approach lies in the way the fast computational speed of ELMs has been combined with the accuracy gained by tuning hyperparameters using HHO. An increased number of hyperparameters has been optimised in our methodology compared to available models. The majority of approaches to improve the accuracy of ELM so far have only focused on tuning the weights and the biases of the hidden layer. In our hybrid model, we tune the number of hidden nodes, the number of input time lags and even the type of activation function used in the hidden layer in addition to tuning the weights and the biases. This has resulted in a significant increase in accuracy over previous methods. Our work presents an original way of performing product demand forecasting in real-time in industry with highly accurate results which are much better than pre-existing demand forecasting models.

Flatness Prediction of Cold Rolled Strip Based on Deep Neural Network with Improved Activation Function.

With the improvement of industrial requirements for the quality of cold rolled strips, flatness has become one of the most important indicators for measuring the quality of cold rolled strips. In this paper, the strip production data of a 1250 mm tandem cold mill in a steel plant is modeled by an improved deep neural network (the improved DNN) to improve the accuracy of strip shape prediction. Firstly, the type of activation function is analyzed, and the monotonicity of the activation function is deemed independent of the convexity of the loss function in the deep network. Regardless of whether the activation function is monotonic, the loss function is not strictly convex. Secondly, the non-convex optimization of the loss functionextended from the deep linear network to the deep nonlinear network, is discussed, and the critical point of the deep nonlinear network is identified as the global minimum point. Finally, an improved Swish activation function based on batch normalization is proposed, and its performance is evaluated on the MNIST dataset. The experimental results show that the loss of an improved Swish function is lower than that of other activation functions. The prediction accuracy of a deep neural network (DNN) with an improved Swish function is 0.38% more than that of a deep neural network (DNN) with a regular Swish function. For the DNN with the improved Swish function, the mean square error of the prediction for the flatness of cold rolled strip is reduced to 65% of the regular DNN. The accuracy of the improved DNN is up to and higher than the industrial requirements. The shape prediction of the improved DNN will assist and guide the industrial production process, reducing the scrap yield and industrial cost.

Application of Artificial Neural Networks for Mangrove Mapping Using Multi-Temporal and Multi-Source Remote Sensing Imagery

Mangroves, as unique coastal wetlands with numerous benefits, are endangered mainly due to the coupled effects of anthropogenic activities and climate change. Therefore, acquiring reliable and up-to-date information about these ecosystems is vital for their conservation and sustainable blue carbon development. In this regard, the joint use of remote sensing data and machine learning algorithms can assist in producing accurate mangrove ecosystem maps. This study investigated the potential of artificial neural networks (ANNs) with different topologies and specifications for mangrove classification in Iran. To this end, multi-temporal synthetic aperture radar (SAR) and multi-spectral remote sensing data from Sentinel-1 and Sentinel-2 were processed in the Google Earth Engine (GEE) cloud computing platform. Afterward, the ANN topologies and specifications considering the number of layers and neurons, learning algorithm, type of activation function, and learning rate were examined for mangrove ecosystem mapping. The results indicated that an ANN model with four hidden layers, 36 neurons in each layer, adaptive moment estimation (Adam) learning algorithm, rectified linear unit (Relu) activation function, and the learning rate of 0.001 produced the most accurate mangrove ecosystem map (F-score = 0.97). Further analysis revealed that although ANN models were subjected to accuracy decline when a limited number of training samples were used, they still resulted in satisfactory results. Additionally, it was observed that ANN models had a high resistance when training samples included wrong labels, and only the ANN model with the Adam learning algorithm produced an accurate mangrove ecosystem map when no data standardization was performed. Moreover, further investigations showed the higher potential of multi-temporal and multi-source remote sensing data compared to single-source and mono-temporal (e.g., single season) for accurate mangrove ecosystem mapping. Overall, the high potential of the proposed method, along with utilizing open-access satellite images and big-geo data processing platforms (i.e., GEE, Google Colab, and scikit-learn), made the proposed approach efficient and applicable over other study areas for all interested users.

Low Complexity MPC-DSVPWM for Current Control of PMSM Using Neural Network Approach

Multilayer neural network-based model predictive control (MLNN-MPC) has received a lot of attention in different power electronic applications. However, the computational burden often imposes limitations in low-order DSPs especially if a large number of voltage vectors (VVs) are used. The execution time of MLNN-MPC in low-order DSPs is affected heavily by the number of input, output, neurons in the hidden layer, and the type of activation function. Furthermore, MLNN contains many parameters that needed to be optimized, such as initial weights, number of iterations, and number of neurons. Therefore, in this study, a creative single-layer neural network-based model predictive control with discrete space vector PWM (SLNN-MPC-DSVPWM) is proposed to overcome these limitations. The main advantages of the proposed method include easy implementation on low-order DSPs, better performance compared with MLNN-MPC, allowing the use of a large number of VVs, and no initialization of lookup tables for all VVs. The proposed SLNN is trained using the Levenberg-Marquardt algorithm and results in an execution time of only 8 μs compared with the complexity of the conventional MPC-DSVPWM and recent MLNN-MPC methods. The SLNN-MPC-DSVPWM is validated by both simulation and experimental results for permanent magnet synchronous motors.

SPECIFICS OF THE LEARNING ERROR DEPENDENCE OF MULTILAYERED NEURAL NETWORKS FROM THE ACTIVATION FUNCTION DURING THE PROCESS OF PRINTED DIGITS IDENTIFICATION

In this paper, we investigated the learning process during an identification of printed digits from the type of an activation function. The study of the activation function type and the number of iterations in the learning process of the neural system was carried out using the Fourier spectra analysis of the learning error function and branching diagrams. For this purpose, a program for the multilayer neural network was developed in the Python software, which involves setting the number of hidden layers as well as the number of neurons inside them and changes in the learning rate. The learning rate was considered as a constant, and its optimal value, where the best learning rate is observed, was determined. To analyze the learning rate effect on the educational process, we used a logistic function describing the frequency doubling process. It is shown that the learning error function is characterized by bifurcation processes leading to a chaotic state when η >0.8. The optimal learning rate value that determines the emergence of the doubling process of the local minima number is determined. It was found that the sigmoidal activation function (as compared to the activation functions ReLU and hyperbolic tangent) best satisfies the learning process of the three-layer neural network for recognizing digits, given an array of 4x7 zeros and ones. Compared to other activation functions, there is an insignificant change in the learning error during the transition from one digit to another. It is shown that an increase in the number of hidden layers does not lead to a sharp increase during the learning error. An increase in the learning iterations number is accompanied by the appearance of periodic dependences of the logistic function value of the learning rate, the period of which is a variable of the number of iterations and the learning rate. Using Fourier spectra of the error function from the learning rate value, it can be argued that an increase in the number of iterations leads to an increase in the number of harmonics, which eventually leads to the appearance of a chaotic state of the neural network. Keywords : Multilayered neural network, activation function, optimal learning rate, digital identification.

System for detecting car models based on machine learning

In this work a neural network is created that recognises the car model and its make. It also classifies the video of a car based on its model. Nearly 16,000 photos of cars from over 30 distinct automobile models and manufacturers were used to train the network. The Convolutional Neural Network(CNN) model is employed here. Relu is the type of activation function employed. The resnet34 CNN model was utilised here. The image classification model resnet34 is a 34-layer convolutional neural network. The layer before the fully connected layer is replaced with the layer which has only classifications of roughly 30 different car make and their models. The model is then trained on the dataset once more and an accuracy of 80% is attained at that point of the time.

Classification of EEG signals for facial expression and motor execution with deep learning

Recently, algorithms of machine learning are widely used with the field of electroencephalography (EEG) brain-computer interfaces (BCI). The preprocessing stage for the EEG signals is performed by applying the principle component analysis (PCA) algorithm to extract the important features and reducing the data redundancy. A model for classifying EEG, time series, signals for facial expression and some motor execution processes had been designed. A neural network of three hidden layers with deep learning classifier had been used in this work. Data of four different subjects were collected by using a 14 channels Emotiv EPOC+ device. EEG dataset samples including ten action classes for the facial expression and some motor execution movements are recorded. A classification results with accuracy range (91.25-95.75%) for the collected samples were obtained with respect to: number of samples for each class, total number of EEG dataset samples and type of activation function within the hidden and the output layer neurons. A time series EEG signal was taken as signal values not as image or histogram, analysed and classified with deep learning to obtain the satisfied results of accuracy.

Predicting rice diseases across diverse agro-meteorological conditions using an artificial intelligence approach.

With the aid of a plant disease forecasting model, the emergence of plant diseases in a given region can be predicted ahead of time. This makes it easier to take proactive steps to reduce losses before they occur. The proposed model attempts to find an association between agrometeorological parameters and the occurrence of the four types of rice diseases. Rice is the staple food of people in Maharashtra. The four major diseases that occur on rice crops are focused on this paper (namely Rice Blast, False Smut, Bacterial Blight and Brown Spot) as these diseases spread rapidly and lead to economic loss. This research paper demonstrates the usage of artificial neural network (ANN) to detect, classify and predict the occurrence of rice diseases based on diverse agro-meteorological conditions. The results were carried out on two cases of dataset split that is 70–30% and 80–20%. The various types of activation function (AF) such as sigmoid, tanH, ReLU and softmax are implemented and compared based on various evaluation metrics such as overall Accuracy, Precision, Recall and F1 score. It can be concluded that the softmax AF applied to 70–30% split of dataset gives the highest accuracy of 92.15% in rice disease prediction.

Research on the emotional tendency of web texts based on long short-term memory network

AbstractThrough the analysis of emotional tendency in online public opinion, governments and enterprises can stabilize people’s emotion more effectively and maintain social stability. The problem studied in this paper is how to analyze the emotional tendency of online public opinion efficiently, and finally, this paper chooses deep learning algorithm to perform fast analysis of emotional tendency of online public opinion. This paper briefly introduced the structure of the basic model used for emotional tendency analysis of online public opinion and the convolutional neural network (CNN) model used for text emotion classification. Then, the CNN model was improved by long short-term memory (LSTM). A simulation experiment was carried out on MATLAB for the improved text emotion classification model to verify the influence of activation function type on the improved model and the performance difference between the improved model and support vector machine (SVM) and traditional CNN models. The results showed that the improved classification model that adopted the sigmoid activation function had higher accuracy and was less affected by language than the relu and tanh activation functions; the improved classification model had the highest accuracy, recall rate, andF-value in classifying emotional tendency of web texts, followed by the traditional CNN model and the SVM model.

Deep AutoEncoders in Recommender Systems: An Application about Internet of Things Service Recommendation

The Internet of Things (IoT) is a system that includes small interconnected devices sharing information through the Internet. Future expects an increasing number of services in IoT devices. Therefore, recommending IoT services will be a vital task for the future of IoT and the convenience of the users. This study aims to find the best methodology to provide IoT service recommendation. With this aim, this study proposes deep autoencoders methodology to recommend services and applications to users based on the devices they own. Deep autoencoders utilize neural networks to predict user service preference matrix. The data used in this study is constructed from a real-world Social IoT dataset. The results showed that deep autoencoders outperformed the state-of-the-art recommendation methods. According to the results, Deep autoencoders improved performance indicators varying between 13.5% and 69.5% compared to other methods. Findings also indicate that the performance of the deep autoencoders can be enhanced by using ELU (exponential linear units), a specific type of activation function. This study contributes to the IoT service recommendation literature by proposing a superior approach when compared to the traditional recommendation techniques.

A TOPSIS-ELM framework for stock index price movement prediction

In recent years Extreme Learning Machine (ELM) has gained the interest of various researchers due to its superior generalization and approximation capability. The network architecture and type of activation functions are the two important factors that influence the performance of an ELM. Hence in this study, a Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) oriented multi-criteria decision making (MCDM) framework is suggested for analyzing various ELM models developed with distinct activation functions with respect to sixteen evaluation criteria. Evaluating the performance of the ELM with respect to multiple criteria instead of single criterion can help in designing a more robust network. The proposed framework is used as a binary classification system for pursuing the problem of stock index price movement prediction. The model is empirically evaluated by using historical data of three stock indices such as BSE SENSEX, S&P 500 and NIFTY 50. The empirical study has disclosed promising results by evaluating ELM with different activation functions as well as multiple criteria.

Power Forecasting from Solar Panels Using Artificial Neural Network in UTHM Parit Raja

This paper presents a step-by-step procedure for the simulation of photovoltaic modules with numerical values, using MALTAB/Simulink software. The proposed model is developed based on the mathematical model of PV module, which based on PV solar cell employing one-diode equivalent circuit. The output current and power characteristics curves highly depend on some climatic factors such as radiation and temperature, are obtained by simulation of the selected module. The collected data are used in developing Artificial Neural Network (ANN) model. Multilayer Perceptron (MLP) and Radial Basis Function (RBF) are the techniques used to forecast the outputs of the PV. Various types of activation function will be applied such as Linear, Logistic Sigmoid, Hyperbolic Tangent Sigmoid and Gaussian. The simulation results show that the Logistic Sigmoid is the best technique which produce minimal root mean square error for the system.

Fixed-Time Synchronization of Second-Order MNNs in Quaternion Field

In this paper, we propose a novel class of networks named as quaternion-valued inertial memristor-based neural networks (QVIMNNs) by introducing inertial term and memristor into traditional quaternion-valued neural networks (QVNNs). The problem of fixed-time synchronization of the QVIMNNs is investigated based on the variable transformation and Lyapunov functional method. It is shown that two types of activation functions are considered and a novel criteria guaranteeing fixed-time synchronization for each cases are then achieved by designing different types of controllers. This paper attempts to pave a new way to investigate neural networks classes with numerical simulations support to demonstrate the correctness of the obtained results.