Types Of Activation Functions Research Articles

A hybrid approach for a cost estimate of residential buildings in Egypt at the early stage

Owners need clear information about the cost at the early stage, while information about drawings and designs are limited in this stage. The purpose of this research is to propose a hybrid model using artificial neural networks and the regression analysis to estimate the cost of residential buildings in Egypt. Real data were collected from 174 real residential projects in Egypt. The Delphi technique was used to reach a consensus on the key factors affecting early stage cost estimation. Artificial neural network models were developed with various numbers of hidden layers and types of activation functions. The sensitivity analysis showed that the most effective factors in the cost estimate at the early stage are the number of floors and the area of the floors. The multiple linear regression, polynomial regression, gamma regression, and Poisson regression were used. The proposed hybrid model was extracted from the ANN model and the regression models using multiple linear regression. The mean absolute percentage error of the hybrid model was 10.64% which is less than the absolute percentage error of the ANN model and the regression models. The results of the hybrid model indicate that the model was successful in estimating the cost of residential projects and would be useful for decision-makers in the construction industry.

Determining optical constants of 2D materials with neural networks from multi-angle reflectometry data

Synthetically generated multi-angle reflectometry data is used to train a neural network based learning system to estimate the refractive index of atomically thin layered materials in the visible part of the electromagnetic spectrum. Unlike previously developed regression based optical characterization methods, the prediction is achieved via classification by using the probabilities of each input element belonging to a label as weighting coefficients in a simple analytical formula. Various types of activation functions and gradient descent optimizers are tested to determine the optimum combination yielding the best performance. For the verification of the proposed method’s accuracy, four different materials are studied. In all cases, the maximum error is calculated to be less than 0.3%. Considering the highly dispersive nature of the studied materials, this result is a substantial improvement in terms of accuracy and efficiency compared to traditional approaches.

Implementation of Quantum Gates using MATLAB

Now days we require low cost and high performance computational based applications. Quantum inspired computational device or circuit performs effective result compare to classical based devices. In the development of quantum-based devices and network needs number of quantum logic gates. Here we studied mathematical description of different types of single and multiple qubits-based quantum logic gates, the reversibility property of quantum gates also proved mathematically. We analyze the cost and effectiveness of each quantum gates has been implemented using neural network with the help of MATLAB. The cost and effectiveness of quantum gates has been analyzed with the comparison of different types of activation function.

A study on the relationship between the rank of input data and the performance of random weight neural network

Random feature mapping (RFM) is the core operation in the random weight neural network (RWNN). Its quality has a significant impact on the performance of a RWNN model. However, there has been no good way to evaluate the quality of RFM. In this paper, we introduce a new concept called dispersion degree of matrix information distribution (DDMID), which can be used to measure the quality of RFM. We used DDMID in our experiments to explain the relationship between the rank of input data and the performance of the RWNN model and got some interesting results. We found that: (1) when the rank of input data reaches a certain threshold, the model’s performance increases with the increase in the rank; (2) the impact of the rank on the model performance is insensitive to the type of activation functions and the number of hidden nodes; (3) if the DDMID of an RFM matrix is very small, it implies that the first $$\mathbf{k}$$ singular values in the singular value matrix of the RFM matrix contain too much information, which usually has a negative impact on the final closed-form solution of the RWNN model. Besides, we verified the improvement effect of intrinsic plasticity (IP) algorithm on RFM by using DDMID. The experimental results showed that DDMID allows researchers evaluate the mapping quality of data features before model training, so as to predict the effect of data preprocessing or network initialization without model training. We believe that our findings could provide useful guidance when constructing and analyzing a RWNN model.

New complex-valued activation functions

We present a new type of activation functions for a complex-valued neuralnetwork (CVNN). A proposed activation function is constructed such that itfixes a given ellipse. We obtain an application to a complex-valued Hopfieldneural network (CVHNN) using a special form of the introduced complexfunctions as an activation function. Considering the interesting geometricproperties of the plane curve ellipse such as focusing property, weemphasize that these properties may have possible applications in variousneural networks.

Global Lagrange Stability of Inertial Neutral Type Neural Networks with Mixed Time-Varying Delays

This paper deals with the Lagrange stability of inertial neutral type neural networks with mixed time-varying delays. Two different types of activation functions are considered, including bounded and general unbounded activation functions. Under a proper variable transformation, the original inertial system is converted to a first order differential network. Based on Lyapunov method and applying inequality techniques and analytical method, some sufficient criteria are derived to ensure the global Lagrange exponential stability of the addressed neural networks. Moreover, the global exponential attractive sets are established. These results here generalize and improve the earlier publications on inertial neural networks. Finally, some numerical examples with simulations are given to demonstrate the effectiveness of our theoretical results.

Implementation of linear regression and neural network models for local climate change assessment and forecasting based on time series data

Climate change studies are based on the data processing of the following time series for number of Ukrainian cities: daily precipitation and air temperature from the site of the European Climate Assessment & Dataset (ECAD number of outputs is 1; number of neurons in the hidden layer is 5; activation function type is sigmoid with the given slope 1) provided the consistent forecast with the horizon of 120 months. Low values of the maximum and average relative errors of the neural network model were achieved on the training set (4.45·10-2 and 2.99·10-4, respectively) and on the test set (3.60·10-2 and 5.69·10-3, respectively). Similarly, for the time series of monthly CO2 emissions for the Kyiv city after December 1, 2013, the predictive neural network model [240 × 5 × 1] provided consistent forecast with the horizon of 60 months. In general, the time series of monthly CO2 emission values are characterized by much smaller values of the consistent forecast horizon in comparison with the time series of the average monthly temperature, at least when using predictive neural network models.

Deep Neural Network-Based In-Vehicle People Localization Using Ultra-Wideband Radar

In this paper, we present an in-vehicle people localization technique using a deep neural network (DNN) model that is trained by the experimental data. First, an impulse radio ultra-wide band (IR-UWB) radar is installed inside the vehicle, and received signals are acquired by changing the arrangement of people sitting. Then, on the acquired data, we apply the DNN to train a classifier, which can predict whether a person is sitting or not in each seat. To design a network suitable for our system, we evaluate the performance by changing the type of activation function, the number of layers, and the number of nodes in each hidden layer of the DNN. In addition, we compare the performance of the proposed method with conventional machine learning algorithms such as support vector machine (SVM) and decision tree-based methods. From our measured signals, the proposed DNN-based method can classify all possible cases according to the location and number of people with an accuracy of 99%. Moreover, the advantage of our proposed method is that there is no need to extract features from a given radar signal.

ReachNN

Applying neural networks as controllers in dynamical systems has shown great promises. However, it is critical yet challenging to verify the safety of such control systems with neural-network controllers in the loop. Previous methods for verifying neural network controlled systems are limited to a few specific activation functions. In this work, we propose a new reachability analysis approach based on Bernstein polynomials that can verify neural-network controlled systems with a more general form of activation functions, i.e., as long as they ensure that the neural networks are Lipschitz continuous. Specifically, we consider abstracting feedforward neural networks with Bernstein polynomials for a small subset of inputs. To quantify the error introduced by abstraction, we provide both theoretical error bound estimation based on the theory of Bernstein polynomials and more practical sampling based error bound estimation, following a tight Lipschitz constant estimation approach based on forward reachability analysis. Compared with previous methods, our approach addresses a much broader set of neural networks, including heterogeneous neural networks that contain multiple types of activation functions. Experiment results on a variety of benchmarks show the effectiveness of our approach.

Multilayer Perceptron New Method for Selecting the Architecture Based on the Choice of Different Activation Functions

Multilayer perceptron has a large amount of classifications and regression applications in many fields: pattern recognition, voice, and classification problems. But the architecture choice in particular, the activation function type used for each neuron has a great impact on the convergence and performance. In the present article, the authors introduce a new approach to optimize the selection of network architecture, weights, and activation functions. To solve the obtained model the authors use a genetic algorithm and train the network with a back-propagation method. The numerical results show the effectiveness of the approach shown in this article, and the advantages of the new model compared to the existing previous model in the literature.

A comparative analysis of artificial neural network architectures for building energy consumption forecasting

Smart grids have recently attracted increasing attention because of their reliability, flexibility, sustainability, and efficiency. A typical smart grid consists of diverse components such as smart meters, energy management systems, energy storage systems, and renewable energy resources. In particular, to make an effective energy management strategy for the energy management system, accurate load forecasting is necessary. Recently, artificial neural network–based load forecasting models with good performance have been proposed. For accurate load forecasting, it is critical to determine effective hyperparameters of neural networks, which is a complex and time-consuming task. Among these parameters, the type of activation function and the number of hidden layers are critical in the performance of neural networks. In this study, we construct diverse artificial neural network–based building electric energy consumption forecasting models using different combinations of the two hyperparameters and compare their performance. Experimental results indicate that neural networks with scaled exponential linear units and five hidden layers exhibit better performance, on average than other forecasting models.

Deep Learning for Emotion Recognition in Affective Virtual Reality and Music Applications

This paper presents a deep learning approach to emotion recognition as applied to virtual reality and music predictive analytics. Firstly, it investigates the deep parameter tuning of the multi-hidden layer neural networks, which are also commonly referred to simply as deep networks that are used to conduct emotion detection in virtual reality (VR)- electroencephalography (EEG) predictive analytics. Deep networks have been studied extensively over the last decade and have shown to be among the most accurate methods for predictive analytics in image recognition and speech processing domains. However, most predictive analytics deep network studies focus on the shallow parameter tuning when attempting to boost prediction accuracies, which includes deep network tuning parameters such as number of hidden layers, number of hidden nodes per hidden layer and the types of activation functions used in the hidden nodes. Much less effort has been put into investigating the tuning of deep parameters such as input dropout ratios, L1 (lasso) regularization and L2 (ridge regularization) parameters of the deep networks. As such, the goal of this study is to perform a parameter tuning investigation on these deep parameters of the deep networks for predicting emotions in a virtual reality environment using electroencephalography (EEG) signal obtained when the user is exposed to immersive content. The results show that deep tuning of deep networks in VR-EEG can improve the accuracies of predicting emotions. The best emotion prediction accuracy was improved to over 96% after deep tuning was conducted on the deep network parameters of input dropout ratio, L1 and L2 regularization parameters. Secondly, it investigates a similar possible approach when applied to 4-quadrant music emotion recognition. Recent studies have been characterizing music based on music genres and various classification techniques have been used to achieve the best accuracy rate. Several researches on deep learning have shown outstanding results in relation to dimensional music emotion recognition. Yet, there is no concrete and concise description to express music. In regards to this research gap, a research using more detailed metadata on twodimensional emotion annotations based on the Russell’s model is conducted. Rather than applying music genres or lyrics into machine learning algorithm to MER, higher representation of music information, acoustic features are used. In conjunction with the four classes classification problem, an available dataset named AMG1608 is feed into a training model built from deep neural network. The dataset is first preprocessed to get full access of variables before any machine learning is done. The classification rate is then collected by running the scripts in R environment. The preliminary result showed a classification rate of 46.0%.

Analysis of application a methods of machine learning based on artificial neural networks in the tasks of detecting cybersecurity threats

The article analyzes the application of methods of machine learning based on artificial neural networks in applied problems of detection and classification of cyber threats. The topic of gender is associated with a significant increase in the implementation of the information technology and cybersecurity technologies of the machine learning. The interdependence between the concepts of “artificial intelligence”, “machine learning”, “deep learning” is revealed. In the article, according to the results of the information sources analysis, the main methods of ML, which have been used in the field of cybersecurity, as follows: Bayes’ networks, artificial neural networks, support vector machine, fuzzy logic, and others are highlighted. A brief analysis of methods for detecting cybersecurity threats using information security and cybersecurity, as follows: statistical, signature, heuristic and abnormal detection methods, has been carried out. The general characteristic is given and the advantages and problems that solve the ML methods for the detection of abnormal events are outlined. The paper considers the main types of artificial neural networks that are used in the tasks of detecting cyber threats. In the article, the basis for considering the general application of machine learning methods is taken by artificial neural networks based on multilayered perceptron with a backpropagation. The general structure of artificial neural networks is selected and the basic mathematical expressions of its functioning are presented, the basic types of activation functions of artificial neurons are considered, the general mathematical expression of the calculation cost function for unsupervised machine learning is presented. More substantially consider the issues of the input data choice for systems machine learning (artificial neural networks). It is proposed to use informative data of attack compromise indicators as input to machine learning systems (artificial neural networks). The main data that can be used by the monitoring subsystem of information security and cyber defense can be used to perform detection, classification and forecasting incidents of cybernetic security. The main stages of the process of data processing and detection of cybersecurity incidents using the (artificial neural networks) are identified. The main systems of information protection and cybersecurity in which machine learning systems are implemented are described. According to the results of the article, the main implementation problems of the machine learning methods in the information security systems are highlighted, the main directions of further scientific research are outlined. This work can be used to highlight the subject area during the development and implementation of machine learning technologies in information security and cybersecurity systems

Performance Analysis of Various Activation Functions in Artificial Neural Networks

The development of Artificial Neural Networks (ANNs) has achieved a lot of fruitful results so far, and we know that activation function is one of the principal factors which will affect the performance of the networks. In this work, the role of many different types of activation functions, as well as their respective advantages and disadvantages and applicable fields are discussed, so people can choose the appropriate activation functions to get the superior performance of ANNs.

Pressure prediction of a spark ignition single cylinder engine using optimized extreme learning machine models

In this study, the cyclic of a spark ignition engine using octane fuel is modeled using extreme learning machine, an emergent technology related to single-hidden layer feedforward neural networks (SLFNs). The experimental engine case study was operated with five different engine speeds from 1000 to 3000 rpm, and crankshaft angle from −360° to 360° without exhaust gas recirculation. The mean effective pressure was used to indicate the cyclic variability for the mean of 100 consecutive cycles. In this study the extreme learning machine (ELM), the regularized extreme learning machine and the outlier robust extreme learning machine were applied to predict the conditions of a combustion parameter used to reflect pressure information for entire cycle in a single-cylinder compression ignition naturally aspirated engine. Prediction by ELM models is normally faster than mathematical models employed to solve a set of differential equations by iterative numerical methods. The essence of ELM is that the hidden layer of SLFNs need not be tuned. Nevertheless, the selection of an appropriate ELM topology is crucial in terms of simplicity, velocity and accuracy. The suitable determination of the number of hidden layer nodes (neurons), type of activation function, and sparse connection structure of weights and biases were obtained using a modified biogeography-based optimization approach (BBO), a population-based metaheuristic algorithm inspired on the mathematical model of organism distribution in biological systems. The experimental dataset were used to train ELM models, and the reliability of these models was assessed and compared for two case studies based on performance criteria related to accuracy, sparsity and complexity using a cross-validation procedure. After training, experimental results show that the pressure can be modeled with reasonable accuracy. The results analysis indicated that the proposed optimized ELM and its variants optimized by BBO approaches have potential for prediction the mean effective pressure showed reasonable consistency with the experimental results.

Global dissipativity of a class of quaternion-valued BAM neural networks with time delay

This paper is concerned with the problem of the global dissipativity of a class of quaternion-valued BAM neural networks with time delay. Two different types of activation functions are considered, including general bounded and Lipschitz-type activation functions. Based on the plural decomposition method of quaternion, the quaternion-valued BAM neural network is separated into two complex-valued systems. Using Lyapunov second method and inequality techniques, some sufficient conditions in complex-valued linear matrix inequality form are derived to ensure the global dissipativity of the discussed network. Moreover, the framework of the globally attractive sets are given out as well. Here, the existence and the uniqueness of the equilibrium point needs not to be considered. Finally, two examples are presented and analyzed to illustrate the effectiveness of our theoretical results.

Selected technical issues of deep neural networks for image classification purposes

In recent years, deep learning and especially Deep Neural Networks (DNN) have obtained amazing performance on a variety of problems, in particular in classification or pattern recognition. Among many kinds of DNNs, the Convolutional Neural Networks (CNN) are most commonly used. However, due to their complexity, there are many problems related but not limited to optimizing network parameters, avoiding overfitting and ensuring good generalization abilities. Therefore, a number of methods have been proposed by the researchers to deal with these problems. In this paper, we present the results of applying different, recently developed methods to improve deep neural network training and operating. We decided to focus on the most popular CNN structures, namely on VGG based neural networks: VGG16, VGG11 and proposed by us VGG8. The tests were conducted on a real and very important problem of skin cancer detection. A publicly available dataset of skin lesions was used as a benchmark. We analyzed the influence of applying: dropout, batch normalization, model ensembling, and transfer learning. Moreover, the influence of the type of activation function was checked. In order to increase the objectivity of the results, each of the tested models was trained 6 times and their results were averaged. In addition, in order to mitigate the impact of the selection of learning, test and validation sets, k-fold validation was applied.

Various Threshold Switching Devices for Integrate and Fire Neuron Applications

AbstractThis study demonstrates an integrate and fire (I&F) neuron using threshold switching (TS) devices to implement spike‐based neuromorphic system. An I&F neuron can be realized using the hysteric voltage switch characteristics of a TS device. To investigate the effects of various TS devices on neuron behavior, neurons are compared using three different types of TS device: NbO2‐based insulator‐to‐metal transition (IMT) device, B–Te‐based ovonic threshold switching device, and Ag/HfO2‐based atomic‐switching TS device. The results show that the off‐state resistance and switching time of the TS devices determine the leaky/nonleaky characteristics and types of activation function of neuron, respectively. In addition, it is confirmed that the threshold voltage and on‐state resistance of the TS device determine the total power consumption of neuron. Furthermore, the feasibility of the TS devices in a spiking neural network is verified by simulation program with integrated circuit emphasis (SPICE) simulation. This work indicates applicability of TS‐based I&F neuron in neuromorphic hardware application.

New Activation Functions in CNN and Its Applications

In this paper, the nonlinear activation functions based on fluid dynamics are presented. We propose two types of activation functions by applying the so-called parametric softsign to the negative region. We apply the activati... | Find, read and cite all the research you need on Tech Science Press

Regularization Activation Function for Extreme Learning Machine

Extreme Learning Machine (ELM) algorithm based on single hidden layer feedforward neural networks has shown as the best time series prediction technique. Furthermore, the algorithm has a good generalization performance with extremely fast learning speed. However, ELM facing overfitting problem that can affect the model quality due to the implementation using empirical risk minimization scheme. Therefore, this study aims to improve ELM by introducing an Activation Functions Regularization in ELM called RAF-ELM. The experiment has been conducted in two phases. First, investigating the modified RAF-ELM performance using four type of activation functions which is Sigmoid, Sine, Tribas and Hardlim. In this study, input weight and bias for hidden layers are randomly selected, whereas the best neurons number of hidden layer is determined from 5 to 100. This experiment used UCI benchmark datasets. The number of neurons (99) using Sigmoid activation function shown the best performance. The proposed methods has improved the accuracy performance and learning speed up to 0.016205 MAE and processing time 0.007 seconds respectively compared with conventional ELM and has improved up to 0.0354 MSE for accuracy performance compare with state of the art algorithm. The second experiment is to validate the proposed RAF-ELM using 15 regression benchmark dataset. RAF-ELM has been compared with four neural network techniques namely conventional ELM, Back Propagation, Radial Basis Function and Elman. The results show that RAF-ELM technique obtain the best performance compared to other techniques in term of accuracy for various time series data that come from various domain.