Tanh Functions Research Articles

Sustainable graphitic carbon derived from oil palm frond biomass for supercapacitor application: Effect of redox additive and artificial neural network‑based modeling approach

One-step pyrolyzed graphitic carbon (GC) derived from oil palm frond biomass was synthesized at different durations (1, 3, and 5 h) without utilizing of activating agent. The optimum GC-5 h exhibited a honeycomb-like structure (1.9 nm), high carbon content (84 %), graphitic peak at 2θ ∼ 24.2°, and wide pore size (2.17 nm) suitable to accommodate solvated electrolyte ions. Symmetric supercapacitor (SSC) cells with three redox additives (hydroquinone (HQ), ammonium monovanadate (AM), and potassium ferrocyanide (PF)) in H2SO4 electrolyte are tested. The GC-5 h SSC shows a CS of 595F/g in 0.01 M HQ/H2SO4 electrolyte at a current density of 3 A/g. The cell exhibits an energy density (ED) of 22 Wh kg−1 and a power density (PD) of 2,400 W kg−1. It demonstrates a capacitance retention of 93 % after 10,000 cycles. To develop the intricate interactions between the electrode structure, active operating circumstances, and electrochemical performance of the SSC, an artificial neural network (ANN) approach was applied herein. The model uses the Levenberg-Marquart training method, incorporating the Tanh and Purelin activation functions. The data demonstrate that the constructed ANN model can predict the SSC with values that nearly match the experimental data with an MSE of 6.8122 × 10−5 and R of 0.9989.

Comparative Analysis of Recurrent Neural Network Models Performance in Predicting Bitcoin Prices

The recurring neural network is a deep learning algorithm that is commonly used to develop prediction systems. There are many variants of RNN such as RNN itself, long-short-term memory (LSTM), and gated recurring unit, so it is frequently debatable which algorithm from the RNN family has the most optimal efficiency and computation time. When developing a prediction system, sequential or time series data is required so that an accurate prediction can be made. Sequential or time series data involve data arranged in a time sequence, such as weather data, financial data, carbon emission data, and traffic data recorded over time. This research will be carried out by predicting the three RNN models against historical Bitcoin value data. The research method used is Experimental Design by comparing the performance between the three models on bitcoin value time series data, testing is done by involving hyperparameters such as Tanh, Sigmoid, and ReLU activation functions, batch size, and epochs. The aim of this research is to find out which RNN model can produce the most optimal performance and find out what performance measures can be used to evaluate and compare the performance between the three models. The results of the study show that LSTM is the most effective model with RMSE 0.012441 and MSE 0.000155 but inefficient because it takes 3 minutes 24 seconds to run the computation; in the meantime, the Tanh activation function gives the most optimal prediction than Sigmoid and RelU and therefore should be the main candidate to be used with RNN models when predicting Bitcoin prices.

Performance of Fourier-based activation function in physics-informed neural networks for patient-specific cardiovascular flows

Background and ObjectivesPhysics-informed neural networks (PINNs) can be used to inversely model complex physical systems by encoding the governing partial differential equations and training data into the neural network. However, neural networks are known to be biased towards learning less complex functions, called spectral bias. This has important implications in modeling cardiovascular flows, where spatial frequencies can vary substantially across anatomies and pathologies (e.g., aneurysms or stenoses). Recent evidence suggests that Fourier-based activation functions have desirable properties, and can potentially reduce spectral bias; however, the performance and adequacy of such Fourier activation functions have not yet been evaluated in patient-specific cardiovascular flow applications. MethodsThe performance of sine activation function was evaluated against tanh and swish activation functions in a 1D advection-diffusion problem, an eccentric 2D stenosis model (Re=5000), and a patient-specific 3D aortic model (Re=823) under pulsatile flow conditions. CFD simulations were performed at high spatio-temporal resolution and data points were extracted for training the neural network. The number of training data points were normalized by L/D. The performance of the PINNs framework was evaluated with increasing number of training data points and across all three activation functions. ResultsOur results demonstrate that sine activation function presents desirable characteristics, such as monotonic reduction in errors, relatively faster convergence, and accurate eigen spectra at higher modes, compared to tanh and swish activation functions. Interestingly, for all activation functions, the domain-averaged errors tended to asymptote at ≈15−20% despite substantial increase in training point density. For 2D eccentric stenosis, errors asymptoted at a sensor point density of 40L/D. For 3D patient-specific aorta, this asymptote was achieved at 180L/D for all three activation functions with an error of ≈15% although sine activation function demonstrated relatively faster convergence. ConclusionsWe have demonstrated that Fourier-based activation functions have higher performance in terms of accuracy and convergence properties for cardiovascular flow applications; however, inherent challenges of neural networks (e.g., spectral bias) can limit the accuracy to ≈15% under physiological, 3D patient-specific blood flow conditions.

Hardware-Optimized Regression Tree-Based Sigmoid and Tanh Functions for Machine Learning Applications

Hardware-Optimized Regression Tree-Based Sigmoid and Tanh Functions for Machine Learning Applications

Exact solutions, symmetry groups and conservation laws for some (2+1)-dimensional nonlinear physical models

In this paper, the Bernoulli sub-equation function method is used to construct new exact travelling wave solutions for two important physical models: (2+1)-dimensional hyperbolic nonlinear Schrödinger (HNLS) equation and (2+1)-dimensional Heisenberg ferromagnetic spin chain (HFSC) equation. These solutions provide valuable insights into the behavior of these models, described in terms of exponential and hyperbolic tangent (tanh) functions. The study also involves an exploration of the infinitesimal generators and symmetry groups through the Lie symmetry method. In addition, by using multiplier approach, the conservation laws are established for these models. Graphical simulation of some solutions in the form of two-dimensional and three-dimensional are plotted to understanding of the underlying physical phenomena and mathematical properties of the (2+1)-dimensional HNLS and HFSC equations. The solutions and graphing are performed using Maple software.

Research on Multiple Evaluation of English Teaching in Higher Vocational School-Enterprise Cooperation in the Era of Multimedia Informatization

Abstract With the progress of the information age, the evaluation of students in higher vocational colleges and universities is not single, and more and more colleges and universities pay more attention to the overall development of students and use diversified evaluation of students’ abilities. This paper constructs a multivariate evaluation model for English teaching in higher vocational school-enterprise cooperation based on deep learning networks and forest stochastic algorithms. By analyzing the application of convolutional neural networks in teaching quality evaluation, Sigmoid and Tanh functions are used to improve the efficiency of English teaching quality evaluation. The bagging method in integrated learning involves averaging the model prediction results of each subset to arrive at the final evaluation results. The multivariate evaluation model is utilized in English teaching evaluation to examine the impact of English teaching under multivariate evaluation. The results show that the evaluation accuracy of the pluralistic evaluation model for English teaching ideas, teaching objectives, teaching contents and teaching effects are 0.981, 0.893, 0.904, and 0.924, respectively, and the students’ liking degree of English is increased from 0.459 to 0.793 under the pluralistic evaluation, which is conducive to the improvement of the teaching effect of English in higher vocational colleges and universities, and provides a new reference for the teaching of English. Perspective.

A Performance-Driven Economic Analysis of a LSTM Neural Network Used for Predicting Building Energy Consumption

Abstract The energy sector occupies a central role in European policy regarding the sustainability and climate change ambitions. This economic sector gained even more importance in the context of a growing energy demand and the threat to energy security and stability in Eastern Europe. The administrative changes from the past years and the shift towards renewable energy sources created the momentum for a general transformation of the energy sector. The research in this field is much needed in order to find the best solutions to predict the energy demand and how to efficiently satisfy it. The purpose of this paper is to assess the economic impact of using artificial neural networks on a dataset that captures information from one building about the energy consumption. The network follows a Long – Short Term Memory architecture and it is characterized by a Root Mean Squared Propagation function for optimization. The analysis consists in the comparison of the performance of three activation functions: Rectified Linear Unit, Sigmoid and Tanh. The results complement the existing research in the field by focusing on the prediction of energy consumption at building level. The case study indicates that the Rectified Linear Unit and Tanh functions are more appropriate than Sigmoid to be used in an LSTM network applied on energy consumption data. The former performs better in terms of accuracy, measured by the Mean Absolute Value, and has similar computational costs to Tanh, with a slightly larger value for training time.

Skin Type Detection with Deep Learning: A Comparative Analysis

There are many factors that can change and affect appearance, including age and environment. Knowing the skin type helps to choose the products best suited to the needs of the skin and therefore the right skin care. Recently, the increasing demand for cosmetics and the scarcity of well-equipped cosmetologists have encouraged cosmetology centers to meet the need by using artificial intelligence applications. Deep learning applications can give high accuracy results in determining the skin type. Recent research shows that learning performs better on nonlinear data than machine learning methods. The aim of this study is to find the best classification model for skin type prediction in skin analysis data with deep learning. For this purpose, 4 different optimization algorithms as Sgd, Adagrad, Adam and Adamax; Tanh and ReLU activation functions and combinations of different neuron numbers using, 16 different models were created.In experimental studies, the performance of the models varies according to the parameters, and it has been observed that the most successful deep neural network model is the model consisting of 64 neurons, Sgd optimization function and ReLU activation function combination with a success rate of 93.75. The accuracy result obtained has a higher classification success compared to other methods, and shows that deep neural networks can make an accurate skin type classification.

All-Optical Recurrent Neural Network With Reconfigurable Activation Function

Optical Neural Networks (ONNs) can be promising alternatives for conventional electrical neural networks as they offer ultra-fast data processing with low energy consumption. However, lack of suitable nonlinearity is standing in their way of achieving this goal. While this problem can be circumvented in feed-forward neural networks, the performance of the recurrent neural networks (RNNs) depends heavily on their nonlinearity. In this paper, we first propose and numerically demonstrate a novel reconfigurable optical activation function, named ROA, based on adding or subtracting the outputs of two saturable absorbers (SAs). RAO can provide both bounded and unbounded outputs by facilitating an electrically programmable adder/subtractor design. Second, with the help of RAO, which can be altered to resemble Tanh or Sigmoid activation functions, we take a step further and numerically demonstrate OptoRNN, a new design for an all-optical RNN in free-space optics. Moreover, by benefiting from mathematical modeling of the multiplication noise, as well as an altered loss function, we enable vector-matrix multiplication (VMM) six times more parallel than conventional optical VMM method for the linear part of the OptoRNN. Finally, through comprehensive simulation studies, we demonstrate that utilizing the OptoRNN, we can achieve 96.3% train accuracy for sequential-MNIST dataset.

Peramalan Data Cuaca Ekstrim Indonesia Menggunakan Model ARIMA dan Recurrent Neural Network

Extreme weather modeling is a challenge for modeling experts in Indonesia and the world. Extreme weather prediction is a complex problem because the chances of it happening are very small, so the developed models often have a low level of accuracy. The purpose of this research is to combine the classic model, Autoregressive Integrated Moving Average (ARIMA), recurrent neural network (RNN) model using Adam and SGD estimation (RNN-Adam and RNN-SGD) with the reLU, tanh, sigmoid and gaussian activation functions. In addition, the ARIMA-RNN mix model was also demonstrated in this study. These models are applied to monthly period extreme weather data obtained from the Meteorology, Climatology and Geophysics Agency (BMKG) of West Sulawesi Province which are converted into training data and test data. The RMSE value is used to see the level of prediction accuracy in both training data and test data. Based on the research results, the best model obtained for modeling Indonesia’s extreme weather is the ARIMA-RNN-Adam mix model with the reLU activation function based on the RMSE value on the training and test data. At n = 50, the smallest RMSE and MSE values of the third model are the ARIMA-RNN-Adam model which is 0.23212 using the reLU activation function, then the ARIMA-RNN-SGD model which is 0.25432 with the same activation function, while the ARIMA value is 0.3270. At n=100 it can be seen that the smallest RMSE and MSE values of the three models are the ARIMA-RNN-Adam model which is equal to 0.25149 using the reLU activation function, then the ARIMA-RNN-SGD model which is equal to 0.25256 with the same activation function, while the ARIMA value is 0.2644.

Synthesis, structure, radiation attenuation efficacy as well as prediction of density using artificial intelligence techniques of lead borate lithium zinc strontium glasses

Glass systems (6-glasses) of chemical formula (60-x)B2O3-10Li2O-20ZnO-10PbO-xSrO: 0 (Sr-0) ≤ x ≤ 25 (Sr-25) mol% have been manufactured. Density was measured experimentally and predicted via artificial intelligence (AI) and machine learning (ML) techniques. As well as radiation attenuation efficacy has been evaluated. Experimentally, density was varied from 3.2824 ± 0.0001 to 3.8717 ± 0.0001 g/cm3. Sample densities were predicated via the artificial neural network (ANN), Tanh and sigmoid activation functions showed the best density prediction with a maximum R2 of highest 0.8900. The best density prediction was also obtained by the random forest regression (RFR) model with R2 = 0.920 which is observed to be the highest compared to the polynomial. The sample encoded as Sr-25 had the greatest linear attenuation coefficient (LAC) values out of all the samples that were examined. The transmission factor (TF%) is simultaneously dependent on the samples' density and the energy of the incoming photons for example the sample of a density of 3.65 g/cm3 (Sr-15) and thickness of x = 1.0 cm. For instance, the TF values are 2.5%, 51.7%, 79.5%, 88.7%, 89.3%, 89.3%, and 88.9% for the energies 0.08, 0.3, 1.0, 5.0, 8.0, 10.0, and 15.0 MeV, respectively. In terms of the half value layer (HVL), the sample encoded as Sr-0 has the greatest value while the sample Sr-25 possessed the lowest HVL value among the studied glass mixtures.

L-SecNet: Towards Secure and Lightweight Deep Neural Network Inference

The advances in machine learning technology has promoted its great potential for deep neural network (DNN) inference powered applications of Internet of Things (IoT), such as facial verification cameras and speech recognition assistants. The current deployment of these applications also raises serious privacy concerns, especially when sensitive individual information is accessed easily by various IoT devices. Fortunately, the cryptography-based solutions are able to execute secure inference without infringing the user’s raw data and the model owner’s proprietary model. However, existing works suffer from impractically high latency and low accuracy, stemming primarily from the evaluations of the non-linear layers in DNN. In this paper, we propose L-SecNet, a lightweight secure neural network inference system that provides efficient inference services without sacrificing accuracy and privacy. Specifically, to reduce latency caused by comparison operations in non-linear layers, we subtly combine additive secret sharing and multiplicative secret sharing to design a lightweight secure comparison protocol. Further, we approximate the commonly used and time-consuming activation functions (including Sigmoid and Tanh functions) with the non-linear sin function instead of the linear polynomial approximation functions in the existing works. In order to maintain low running latency while meeting the requirement of high accuracy, a secure and lightweight protocol for a sin function is proposed. Our theoretical analysis and empirical experiments evaluate the security and efficiency of the L-SecNet system. Compared with the state-of-the-art works, L-SecNet saves up to about 80 times bandwidth and about 53 times runtime.

Pure-quartic solitons in presence of weak nonlocality

In this paper, we study the characteristics of pure-quartic optical solitons in a nonlinear medium having a weakly nonlocal response. We propose an nonlinear Schrödinger equation incorporating pure fourth-order diffraction, Kerr nonlinearity and weak nonlocality to describe the transmission of solitons in the system. We obtain analytic pure-quartic soliton solutions to this model in a variety of shapes and forms, unlike the case of the model without taking into account the weak nonlocal contribution. Besides, we find that the obtained localized structures have both bright and dark type waveforms in the transmission system. Interestingly, the functional forms of these solitons are expressed in terms of sech and tanh functions which do not possess oscillating tails, markedly different from the conventional pure-quartic solitons in the absence of weak nonlocality. These structures exist for both positive and negative quartic diffraction coefficient whereas pure-quartic solitons in the absence of nonlocality occur only for negative fourth-order dispersion. The numerical examples are presented for illustrating the soliton evolution dynamics in the optical waveguide.

Surrogate modeling of heat transfers of nanofluids in absorbent tubes with fins based on deep convolutional neural network

In this paper, we propose and investigate a deep convolutional neural network-based surrogate model for fast prediction of heat transfer of nanofluid in absorbent tubes with fins. Inspired by Unet, the proposed model consists of a contracting path and an expanding path, and skip connections are replaced by residual blocks to enhance prediction performance. The model can predict temperature field of tubes’ specified cross-sections in any quantity. The nanofluid-filled absorbent tubes with two types of fins (rectangular and semiepllise) are investigated, where the shapes of the fins are variable. The results show that the proposed model can predict the temperature field with very high accuracy and extremely fast speed: the average prediction error is less than 0.15% for all studied cases, and the prediction speed is more than 4 orders faster than numerical simulation with OpenFOAM. In addition, we analyze the main factors affecting the network model, including the learning rate, data size, network's layer, activation function and input presentation. It is found that the influence of activation function is significant, where ReLU6 is able to reduce the max prediction error to 2.7%, while the Tanh and Sigmoid functions have two and four times larger errors, respectively. The results of our current work show great application potential of the deep neural network-based surrogate model for rapid 3D geometry optimization of the nanofluid-filled fined absorbent tube of heat exchangers and solar collectors.

Leisure Sports Behavior Recognition Algorithm Based on Deep Residual Network

Aiming at the related problems existing in the field of leisure sports computing, in order to study the behavior recognition of leisure sports by deep residual network, based on the deep residual neural network theory, the behavior recognition algorithm and the corresponding robust model are used to analyze the leisure sports related samples, and the correlation model is used to predict and analyze the leisure sports related content. The results show that the change curves of Sig and Tanh functions can be divided into slow increasing stage, linear increasing stage, and stable stage. The y value corresponding to ReLU curve shows a linear change trend with the increase of x value. The Leaky function’s corresponding curve can be divided into two stages. The function coincides with the ReLU function in the first quadrant and remains linear in the third quadrant. The activation function curves corresponding to layers 56 and 20 have a relatively large variation range, and both of them show an overall trend of gradual decline. On the whole, the curve value corresponding to layer 56 is higher than that corresponding to layer 20, indicating that the method of layer 20 is relatively good and the corresponding training error is relatively low. It can be seen from the robustness recognition rate of various methods under different training samples that Fl has the highest overall data recognition rate while Sc has relatively poor stability. However, the recognition rate of IDCC and DCC shows a relatively flat trend, indicating that these two methods have certain advantages in describing the robust recognition rate. The research results can provide theoretical support for the application of deep residual neural networks in other fields.

Resource efficient activation functions for neural network accelerators

Implementations of machine learning models in resource-limited embedded systems are becoming highly desired. This has led to a need for resource-efficient building blocks for computing the mathematical operations required for neural network training and inferencing. Efficient activation functions for low-end hardware devices with limited hardware capabilities are important. In this work, we present a method for generating symmetric and asymmetric activation functions for deep and convolution neural networks. Furthermore, we propose a solution that simultaneously computes a symmetric activation function with an integrated scaling functionality for Long Short Term Memory (LSTM) models. This effectively eliminates two of the three element-wise multipliers in an LSTM cell. Also, this built-in scaling requires no additional computation time because it is integrated within the computation of the symmetric non-linear mapping. This approach replaces the need to compute several Tanh activation functions and element-wise multipliers separately. A resource-efficient approximate multiplier is also proposed to eliminate the third element-wise multiplier and potentially replace all the resource-hungry multipliers. The digital implementation of the proposed method is highly amenable to parallelization and is extremely resource-efficient. We record an area-saving on field-programmable gate arrays with different precision. Our proposal’s formulaic equivalents are also computationally fast on CPU-based engines. On an embedded ARM processor, our method achieves a speedup of at least 4.37× for the proposed functions. We show that LSTMs with our method can achieve up to 3.5× resource footprint saving when compared to the hard activation implementation. We demonstrate that our method achieves competitive results with negligible loss of performance.

Modelling and Computational Experiment to Obtain Optimized Neural Network for Battery Thermal Management Data

The focus of this work is to computationally obtain an optimized neural network (NN) model to predict battery average Nusselt number (Nuavg) data using four activations functions. The battery Nuavg is highly nonlinear as reported in the literature, which depends mainly on flow velocity, coolant type, heat generation, thermal conductivity, battery length to width ratio, and space between the parallel battery packs. Nuavg is modeled at first using only one hidden layer in the network (NN1). The neurons in NN1 are experimented from 1 to 10 with activation functions: Sigmoidal, Gaussian, Tanh, and Linear functions to get the optimized NN1. Similarly, deep NN (NND) was also analyzed with neurons and activations functions to find an optimized number of hidden layers to predict the Nuavg. RSME (root mean square error) and R-Squared (R2) is accessed to conclude the optimized NN model. From this computational experiment, it is found that NN1 and NND both accurately predict the battery data. Six neurons in the hidden layer for NN1 give the best predictions. Sigmoidal and Gaussian functions have provided the best results for the NN1 model. In NND, the optimized model is obtained at different hidden layers and neurons for each activation function. The Sigmoidal and Gaussian functions outperformed the Tanh and Linear functions in an NN1 model. The linear function, on the other hand, was unable to forecast the battery data adequately. The Gaussian and Linear functions outperformed the other two NN-operated functions in the NND model. Overall, the deep NN (NND) model predicted better than the single-layered NN (NN1) model for each activation function.

Broad Autoencoder Features Learning for Classification Problem

Activation functions such as Tanh and Sigmoid functions are widely used in Deep Neural Networks (DNNs) and pattern classification problems. To take advantages of different activation functions, the Broad Autoencoder Features (BAF) is proposed in this work. The BAF consists of four parallel-connected Stacked Autoencoders (SAEs) and each of them uses a different activation function, including Sigmoid, Tanh, ReLU, and Softplus. The final learned features can merge such features by various nonlinear mappings from original input features with such a broad setting. This helps to excavate more information from the original input features. Experimental results show that the BAF yields better-learned features and classification performances.

Broad Autoencoder Features Learning for Classification Problem

Activation functions such as Tanh and Sigmoid functions are widely used in Deep Neural Networks (DNNs) and pattern classification problems. To take advantages of different activation functions, the Broad Autoencoder Features (BAF) is proposed in this work. The BAF consists of four parallel-connected Stacked Autoencoders (SAEs) and each of them uses a different activation function, including Sigmoid, Tanh, ReLU, and Softplus. The final learned features can merge such features by various nonlinear mappings from original input features with such a broad setting. This helps to excavate more information from the original input features. Experimental results show that the BAF yields better-learned features and classification performances.

Logish: A new nonlinear nonmonotonic activation function for convolutional neural network

Activation function is an important component of the convolutional neural network. Recently, nonlinear nonmonotonic activation functions such as Swish and Mish have illustrated good performance in deep learning structures. In this paper, we propose a new nonlinear nonmonotonic activation function called Logish, which can be represented by f(x)=x·ln[1+sigmoid(x)]. Firstly, we take the logarithmic operation to reduce the numerical range of sigmoid(x)+1, then we employ variable x to make the negative output have a strong regularization effect. Furthermore, we evaluate the image classification performance of Logish and its variant f(x)=αx·ln[1+sigmoid(βx)] in simple and complex networks with top–1 accuracy. Experimental results demonstrate that Logish’s variant (α=1,β=10) can achieve 94.8% top-1 accuracy with ResNet–50 network on CIFAR 10 dataset, and can reach 99.24% top-1 accuracy with DenseNet on MNIST dataset and 88.52% top-1 accuracy with SE-Inception-v4 network on SVHN dataset respectively. It is higher than the Sigmoid, Tanh, ReLU, Swish and Mish activation functions in the corresponding dataset. It also verifies the performance and effectiveness of Logish.