Tanh Activation Function Research Articles

Radiation shielding capacity and machine learning density prediction of boro-bismuth cadmium zinc glasses

Bismuth borate cadmium zinc glasses were prepared using melt quenching technique for shielding applications. The glasses had the composition (80-x)B2O3-xBi2O3-10CdO-10ZnO, where 0<x<20mol%. The densities of the glasses were predicted using machine learning tools and were found to increase with increasing Bi2O3 content. This increase in density was correlated with changes in the infrared spectrum of the glasses. The density of the glass changed from 4.410 g/cm³ to 6.9830 g/cm³ when bismuth oxide (Bi2O3) was added to the glass network. This is because Bi2O3 changes the structure of the glass. The presence of octahedral BiO6 units in the glass is confirmed by the peak at 514 cm-1 in the infrared spectrum. This peak shifts to a lower wavenumber as the amount of Bi2O3 in the glass increases. The shift of the infrared peak from 688 cm-1 to 596 cm-1 indicates that Bi2O3 is actively participating in the glass network, which results in a decrease in the number of BO3 units. The shift of the infrared peak from 1060 cm-1 to 1137 cm-1 is attributed to B-O stretching vibrations in BO4- units, and also confirms that the number of BO4 units is increasing. The density of the glass was predicted using machine learning algorithms that are trained on the chemical composition and experimental density of glasses. Polynomial regression and artificial neural networks (ANNs) were used to predict the density of glass from its chemical composition. Polynomial regression performed best with a 10th-degree polynomial, with an R2 value of 0.8820. ANNs performed best with a tanh activation function, with an R2 value of 0.8923. Random forest regression performed better than other machine learning models at predicting the density of glass, with an R2 value of 0.920. The predicted values were very close to the experimental values. The mass (MAC) and linear (LAC) attenuation coefficients of radiation shielding followed the following trend: (MAC, LAC)0 < (MAC, LAC)5 < (MAC, LAC)10 < (MAC, LAC)15 < (MAC, LAC)20. The half value layer followed the trend: HVL followed as: (HVL)0 > (HVL)5 > (HVL)10 > (HVL)15 > (HVL)20. The Radiation Protection Effectiveness (RPE) reinforces the superiority of BBCZ-20 glasses in shielding against gamma radiation, making them promising candidates for radiation protection applications.

Polarization differential interference contrast microscopy with physics-inspired plug-and-play denoiser for single-shot high-performance quantitative phase imaging.

We present single-shot high-performance quantitative phase imaging with a physics-inspired plug-and-play denoiser for polarization differential interference contrast (PDIC) microscopy. The quantitative phase is recovered by the alternating direction method of multipliers (ADMM), balancing total variance regularization and a pre-trained dense residual U-net (DRUNet) denoiser. The custom DRUNet uses the Tanh activation function to guarantee the symmetry requirement for phase retrieval. In addition, we introduce an adaptive strategy accelerating convergence and explicitly incorporating measurement noise. After validating this deep denoiser-enhanced PDIC microscopy on simulated data and phantom experiments, we demonstrated high-performance phase imaging of histological tissue sections. The phase retrieval by the denoiser-enhanced PDIC microscopy achieves significantly higher quality and accuracy than the solution based on Fourier transforms or the iterative solution with total variance regularization alone.

An improved convolutional neural network approach for damage localization in composite materials based on modal analysis

Damage detection of composite materials using modal parameters has limitations in terms of sensitivity to small or localized damage and limited accuracy in damage localization. To address this issue, an enhanced channel attention residual network (ECARNet) damage detection model for composite laminates is proposed. First, finite element analysis is used to obtain training samples, which are processed as two-dimensional data to take full advantage of the convolutional neural network. Then, the residual module uses a multilayer perceptron instead of the traditional convolutional layers to learn the correlation between channels to enhance the generalization ability of the model, and uses the tanh activation function to retain negative information. Finally, a channel focus mechanism is introduced to enable the network to learn key features adaptively. Experimental results on two datasets with different levels of damage demonstrate the superior detection performance of ECARNet, achieving average detection accuracies of 98.13% and 97.94% respectively. A comparison with other methods verifies the effectiveness and reliability of the proposed approach. Furthermore, the effectiveness of the new method is validated on real-world test data.

Physics-informed neural networks for approximating dynamic (hyperbolic) PDEs of second order in time: Error analysis and algorithms

We consider the approximation of a class of dynamic partial differential equations (PDEs) of second order in time by the physics-informed neural network (PINN) approach, and provide an error analysis of PINN for the wave equation, the nonlinear Klein-Gordon equation and the linear elastodynamic equation. Our analyses show that, with feed-forward neural networks having two hidden layers and the tanh activation function, the PINN approximation errors for the solution field, its time derivative and its gradient field can be effectively bounded by the training loss and the number of training data points (quadrature points). Our analyses further suggest new forms for the training loss function, which contain certain residuals that are crucial to the error estimate but would be absent from the canonical PINN loss formulation. Adopting these new forms for the loss function leads to a variant PINN algorithm. We present ample numerical experiments with the new PINN algorithm for the wave equation, the Sine-Gordon equation and the linear elastodynamic equation, which show that the method can capture the solution well.

Urban Vegetation Extraction from High-Resolution Remote Sensing Imagery on SD-UNet and Vegetation Spectral Features

Urban vegetation plays a crucial role in the urban ecological system. Efficient and accurate extraction of urban vegetation information has been a pressing task. Although the development of deep learning brings great advantages for vegetation extraction, there are still problems, such as ultra-fine vegetation omissions, heavy computational burden, and unstable model performance. Therefore, a Separable Dense U-Net (SD-UNet) was proposed by introducing dense connections, separable convolutions, batch normalization layers, and Tanh activation function into U-Net. Furthermore, the Fake sample set (NIR-RG), NDVI sample set (NDVI-RG), and True sample set (RGB) were established to train SD-UNet. The obtained models were validated and applied to four scenes (high-density buildings area, cloud and misty conditions area, park, and suburb) and two administrative divisions. The experimental results show that the Fake sample set can effectively improve the model’s vegetation extraction accuracy. The SD-UNet achieves the highest accuracy compared to other methods (U-Net, SegNet, NDVI, RF) on the Fake sample set, whose ACC, IOU, and Recall reached 0.9581, 0.8977, and 0.9577, respectively. It can be concluded that the SD-UNet trained on the Fake sample set not only is beneficial for vegetation extraction but also has better generalization ability and transferability.

Optimizing cotton‐picking robotic manipulator and inverse kinematics modeling using evolutionary algorithm‐assisted artificial neural network

AbstractThis study presents a particle swarm optimization (PSO) algorithm‐assisted neural‐network‐based inverse kinematics solution for a 4‐DoF (degree‐of‐freedom) cotton harvesting robot. A novel setup was developed to measure the three‐dimensional locations of in‐field cotton bolls. Dimensional optimization of the manipulator was conducted using the PSO algorithm to minimize torque requirements at joints. With the optimized links’ lengths, the targeted end‐effector positions were achieved effectively (coefficient of determination (R2) &gt; 99.88). The genetic algorithm optimized the neural network architecture to include three hidden layers with [64 64 32] neurons, identifying the Tanh activation function as the optimal configuration. A custom loss function was used during the training of artificial neural network (ANN). Using angles predicted by the trained ANN, the end‐effector reached targeted positions with positioning errors below 13.0 mm. A hybrid model consisting of an ANN and PSO algorithm was developed to further reduce the error. This trained hybrid model resulted in a positioning error below 1.0 mm with inference time of 6.07 s during simulation phase. As compared to the ANN and PSO algorithm, hybrid model reduced the positioning error and inference time (&gt;40.0%), respectively. For hybrid model, the mean percentage errors of 0.25%, 0.39%, and 0.84% were observed along the x‐, y‐, and z‐axis. A positioning error below 9.0 mm occurred during evaluation of the hybrid model with the fabricated manipulator. Hence, the developed hybrid model precisely determines the joint angles, allowing the end‐effector of the cotton harvesting robot to reach at targeted pose with minimum error.

Sparse polynomial optimisation for neural network verification

The prevalence of neural networks in applications is expanding at an increasing rate. It is becoming clear that providing robust guarantees on systems that use neural networks is very important, especially in safety-critical applications. A trained neural network’s sensitivity to adversarial attacks is one of its greatest shortcomings. To provide robust guarantees, one popular method that has seen success is to describe and bound the activation functions in the neural network using equality and inequality constraints. However, there are numerous ways to form these bounds, providing a trade-off between conservativeness and complexity. We approach the problem from a different perspective, using sparse polynomial optimisation theory and Positivstellensatz, a key result in real algebraic geometry. The former exploits the natural cascading structure of the neural network using ideas from chordal sparsity while the latter tests the emptiness of a semi-algebraic set using algebra, to provide tight bounds. We show that bounds can be tightened significantly, whilst the computational time remains reasonable. We compare the solve times of different solvers and show how accuracy can be improved at the expense of increased computation time. In particular, we show that with the sparse polynomial framework the solve time and accuracy can be improved over other methods for neural network verification, for networks with ReLU, sigmoid and tanh activation functions.

Novel ANOVA-Statistic-Reduced Deep Fully Connected Neural Network for the Damage Grade Prediction of Post-Earthquake Buildings.

Earthquakes are cataclysmic events that can harm structures and human existence. The estimation of seismic damage to buildings remains a challenging task due to several environmental uncertainties. The damage grade categorization of a building takes a significant amount of time and work. The early analysis of the damage rate of concrete building structures is essential for addressing the need to repair and avoid accidents. With this motivation, an ANOVA-Statistic-Reduced Deep Fully Connected Neural Network (ASR-DFCNN) model is proposed that can grade damages accurately by considering significant damage features. A dataset containing 26 attributes from 762,106 damaged buildings was used for the model building. This work focused on analyzing the importance of feature selection and enhancing the accuracy of damage grade categorization. Initially, a dataset without primary feature selection was utilized for damage grade categorization using various machine learning (ML) classifiers, and the performance was recorded. Secondly, ANOVA was applied to the original dataset to eliminate the insignificant attributes for determining the damage grade. The selected features were subjected to 10-component principal component analysis (PCA) to scrutinize the top-ten-ranked significant features that contributed to grading the building damage. The 10-component ANOVA PCA-reduced (ASR) dataset was applied to the classifiers for damage grade prediction. The results showed that the Bagging classifier with the reduced dataset produced the greatest accuracy of 83% among all the classifiers considering an 80:20 ratio of data for the training and testing phases. To enhance the performance of prediction, a deep fully connected convolutional neural network (DFCNN) was implemented with a reduced dataset (ASR). The proposed ASR-DFCNN model was designed with the sequential keras model with four dense layers, with the first three dense layers fitted with the ReLU activation function and the final dense layer fitted with a tanh activation function with a dropout of 0.2. The ASR-DFCNN model was compiled with a NADAM optimizer with the weight decay of L2 regularization. The damage grade categorization performance of the ASR-DFCNN model was compared with that of other ML classifiers using precision, recall, F-Scores, and accuracy values. From the results, it is evident that the ASR-DFCNN model performance was better, with 98% accuracy.

Data Mining for Heart Disease Prediction Based on Echocardiogram and Electrocardiogram Data

Traditional methods of detecting cardiac illness are often problematic in the medical field. The doctor must next study and interpret the findings of the patient's medical record received from the electrocardiogram and echocardiogram. These tasks often take a long time and require patience. The use of computational technology in medicine, especially the study of cardiac disease, is not new. Scientists are continuously striving for the most reliable method of diagnosing a patient's cardiac illness, particularly when an integrated system is constructed. The study attempted to propose an alternative for identifying cardiac illness using a supervised learning technique, namely the multi-layer perceptron (MLP). The study started with the collection of patient medical record data, which yielded up to 534 data points, followed by pre-processing and transformation to provide up to 324 data points suitable to be employed by learning algorithms. The last step is to create a heart disease classification model with distinct activation functions using MLP. The degree of classification accuracy, k-fold cross-validation, and bootstrap are all used to test the model. According to the findings of the study, MLP with the Tanh activation function is a more accurate prediction model than logistics and Relu. The classification accuracy level (CA) for MLP with Tanh and k-fold cross-validation is 0.788 in a data-sharing situation, while it is 0.672 with Bootstrap. MLP using the Tanh activation function is the best model based on the CA level and the AUC value, with values of 0.832 (k-fold cross-validation) and 0.857 (bootstrap).

Cost effective Tanh activation function circuits based on fast piecewise linear logic

In this paper, an approximate 16-bit non-linear Tanh function circuit design is proposed for activation function in neural networks and digital signal processing, which is based on piecewise linear calculation. By sacrificing acceptable accuracy in computation, the proposed logic circuit structure can furtherly improve the overall performance of the design, with marked reduction in delay, area and power consumption. Linear calculation segments with simple shift-and-add logic architectures are adopted for easy hardware implementation. Additionally, tailored bias tuning and approximate accumulator design make a fine balance between complexity and error distribution. The mean absolute error (MAE) of the entire design is 0.0049. Compared with other state-of-art works in 90 nm CMOS process, the area of the circuit is 582μm2, and the energy is 0.0996 mW/GHz at 0.71 GHz. Moreover, the area-delay product is reduced by over 79.7% and the energy-delay product is reduced by over 42.4%. In the evaluation of its suitability in large applications, the accurate Tanh circuit and the proposed Tanh circuit are adopted in two neural networks to classify the Fashion-MNIST data and emotional EEG data. When the model converges, the accuracies of the test set only differs with a maximum of 0.2% and 0.3%.

Application of Artificial Neural Network Method using Hyperparameter Tuning for Predicting of Euro Exchange Rupiah

The Covid-19 pandemic has significantly impacted the economic decline in many countries, such as Italy, the United States and the European Union. Indonesia, also affected by Covid-19, was not spared from economic turmoil, especially in the foreign exchange market, where the rupiah exchange rate against the Euro experienced significant fluctuations in early 2020, hampering international trade and investment activities. Therefore, an appropriate method is needed to predict changes in the rupiah exchange rate against the Euro to minimize the obstacles. This study uses the ANN model to predict the Rupiah (Rp) exchange rate against the Euro (€). The best model is obtained through the hyper-tuning process. The optimal parameter values obtained are the input layer with 10 nodes, 2 hidden layers with 19 nodes and 13 nodes, the output layer, dropout of 0.2, 32 batch sizes, 100 epochs, and the Tanh activation function in the distribution scheme of 90% training data and 10 % testing data. Based on the MAPE value of 0.0042% and 0.0041% obtained, the prediction results on the selling and buying rates of the Rupiah against the Euro, it can be concluded that the model has good predictive ability with an accuracy value of 99.996%.Keywords: Currency Exchange Rates, Data Mining, Machine Learning, Artificial Neural Networks

LSTM Hyperparameters optimization with Hparam parameters for Bitcoin Price Prediction

Machine learning and deep learning algorithms produce very different results with different examples of their hyperparameters. Algorithm parameters require optimization because they aren't specific for all problems. In this paper Long Short-Term Memory (LSTM), eight different hyperparameters (go-backward, epoch, batch size, dropout, activation function, optimizer, learning rate and, number of layers) were used to examine to daily and hourly Bitcoin datasets. The effects of each parameter on the daily dataset on the results were evaluated and explained These parameters were examined with hparam properties of Tensorboard. As a result, it was seen that examining all combinations of parameters with hparam produced the best test Mean Square Error (MSE) values with hourly dataset 0.000043633 and daily dataset 0.00073843. Both datasets produced better results with the tanh activation function. Finally, when the results are interpreted, the daily dataset produces better results with a small learning rate and small dropout values, whereas the hourly dataset produces better results with a large learning rate and large dropout values.

BRAIN TUMOR DETECTION FROM MRI IMAGES USING DISCRETE COSINE TRANSFORM FEATURES AND EXTREME LEARNING MACHINE

A brain tumor is an abnormal growth of brain tissue and characterized by excessive cell proliferation in certain parts of the brain. One of the current, reliable technologies that can be used to identify brain tumors is Magnetic Resonance Imaging (MRI) scans. The scanned MRI images are then conventionally monitored and examined by a specialist for the presence of tumors. As the number of people suffering from brain tumors is significantly increasing and their corresponding mortality rate has reached 18,600 by 2021, research on designing more effective and efficient tools to assist medical specialists in identifying brain tumors is considered of great importance. In a previous study, a machine learning-based model demonstrated its ability to detect brain tumors with a classification accuracy of 92%. Several hyperparameters were computationally tested using public MRI datasets to obtain the most reliable detection/binary classification accuracy on MRI brain images. Sophisticated model accuracy was achieved by testing various neuronal units and ELM activation functions, followed by inserting a feature map extracted from the Discrete Cosine Transform (DCT). The model obtained the highest testing accuracy of 95% with several 20 ELM neuron units with a tanh activation function.

Affect and stress detection based on feature fusion of LSTM and 1DCNN

The impact of emotions on health, especially stress, is receiving increasing attention. It is important to provide a non-invasive affect detection system that can be continuously monitored for a long period of time. Multi-sensor fusion strategies can better improve the performance of affect detection models, but there are also problems such as insufficient feature extraction and poor spatiotemporal feature fusion. Therefore, this study proposes a feature-level fusion method based on long short-term memory and one-dimensional convolutional neural network to extract temporal and spatial features of electrocardiogram, electromyogram, electrical activity, temperature, accelerator and response data, respectively, and then fuse them in a summation fashion for affect and stress detection. In particular, we added the tanh activation function before feature fusion, which can improve the model’s performance. We used the wearable affect and stress detection dataset to train the model, which includes three different emotion states (neutral, stress, and amusement) for three-class emotion classification with accuracy and F1-scores of 87.82% and 86.68%, respectively. Due to the importance of stress, we also studied binary classification for stress detection, where neutral and amusement were combined as non-stress, with accuracy and F1-scores of 94.9% and 94.98%, respectively. The performance of the proposed model outperforms other control models and can effectively improve the performance of affect and stress detection, and promote medical care, health care and elderly care.

Combination Of SqueezeNet And Multilayer Backpropagation Algorithm In Hanacaraka Script Recognition

Javanese script is one of Indonesia's cultural heritages that are increasingly rarely used today. The difficulty of recognizing the shapes of letters, let alone writing them, is the main obstacle in using the Hanacaraka script. This research offers an alternative to Hanacaraka script recognition using a combination of image feature extraction and machine learning, where we utilize a pre-trained SquzeeNet model and Multilayer Backpropagation algorithm. Of the 18 models built using ReLu, Sigmoid, and Tanh activation functions, we found that the Tanh activation function, using the combination of 50-50-100 neuron configuration and 25 epochs, was the most optimal function used to classify the training data with accuracy, precision, and recall values of 93.8%. Meanwhile, the Tanh activation function, using the 50-100-50 neuron configuration and 50 epochs, is the most optimal function to classify the testing data, with accuracy, precision, and recall values of 89.1%, 89.5%, and 89.5%. All built models show a training and testing performance ratio below 10%. From this result, we conclude that all models have good reliability in the training and testing classification process.

The effect of amplitude modification in S-shaped activation functions on neural network regression

Time series forecasting using multilayer feed-forward neural networks (FNN) is potential to give high accuracy. Several factors influence the accuracy. One of them is the choice of activation functions (AFs). There are several AFs commonly used in FNN with their specific characteristics, such as bounded type AFs. They include sigmoid, softsign, arctan, and tanh. This paper investigates the effect of the amplitude in the bounded AFs on the FNNs’ accuracy. The theoretical investigations use simplified FNN models: linear equation and linear combination. The results show that the higher amplitudes give higher accuracy than typical amplitudes in softsign, arctan, and tanh AFs. However, in sigmoid AF, the amplitude changes do not influence the accuracy. These theoretical results are supported by experiments using the FNN model for time series prediction of 10 foreign exchanges from different continents compared to the US dollar. Based on the experiments, the optimum amplitude of the AFs should be high, that is greater or equal to 100 times of the maximum input values to the FNN, and the accuracy gains up to 3–10 times.

INVESTIGATION OF POLYPS IN ENDOSCOPY IMAGES BY USING DEEP LEARNING ALGORITHM

Recent advances in machine learning, particularly with regard to deep learning, help to recognize and classify objects in medical images. In this study, endoscopy images were examined and deep learning method was used to classify healthy and polyp cells. For the proposed system, a database was created from the archives of General Surgery Department Endoscopy Unit in Kutahya Evliya Celebi Training and Research Hospital. The database contains 93 polyps and 216 normal images from 54 archive records. For image multiplexing, a total of 1236 images were obtained by rotating each image 90 degrees around its axis. While 2/3 of the randomly selected data from this obtained data was used for training the model, the rest of the data was reserved for testing. K-fold Cross Validation method was used to reduce the variability of performance results. In this study, 48 different models were created by using different activation and optimization functions to find the best classification model in deep learning. According to the experimental results, it was observed that accuracy of the models depends on the selected parameters; the best model with the accuracy rate of 91% was obtained with 64 neurons in the hidden layer, ReLU activation function and RmsProp optimization method whereas the worst model with the accuracy rate of 76% was obtained with 32 neurons in the hidden layer, Tanh activation and PmsProp optimization functions. Accordingly, classification performance of polyp images can be optimized by utilizing different activation and optimization methods during the design of deep learning models.

A closer-look at lithium strontium boro-fluoride glasses doped with CeO2 and Yb2O3 ions: Synthesis, radiation shielding properties, and prediction of density using artificial intelligence techniques

Eight glasses with chemical compositions (50-x)B2O3–28SrF2–22Li2O-xCeO2-xYb2O3 (x = 0 (CY0), 0.025 (CY1), 0.05 (CY2), 0.075 (CY3), 0.10 (CY4), 0.20 (CY5), 0.50 (CY6), and 1 (CY7) mol%) were synthesized using the melt quenching process. Density of the prepared glass was measured experimentally by Archimedes' principle. It was varied from 2.4582 g cm−3 to 4.6587 g cm−3. The density prediction based on the chemical composition of the glass and experimental density as inputs for various machine learning algorithms has been investigated. The Polynomial regression successfully fit the glass data and best density prediction obtained at 10-degree polynomial with R2 values 0.8726. The Artificial neural network also predicted the glass data using different activation functions and best density prediction obtained for tanh activation function with R2 = 0.8740 which a best regression values compare to polynomial fit. The Random forest regression (RFR) predicted the best density prediction compare to other Artificial intelligence models and predicted values are very close the experimental values with R2 = 0.985 which is best fit for the glass data.

Diagonal Recurrent Neural Network-Based Hysteresis Modeling.

The Preisach model and the neural networks are two of the most popular strategies to model hysteresis. In this article, we first mathematically prove that the rate-independent Preisach model is actually a diagonal recurrent neural network (dRNN) with the binary step activation function. For the first time, the hysteresis nature and conditions of the classical dRNN with the tanh activation function are mathematically discovered and investigated, instead of using the common black-box approach and its variants. It is shown that the dRNN neuron is a versatile rate-dependent hysteresis system under specific conditions. The dRNN composed of those neurons can be used for modeling the rate-dependent hysteresis and it can approximate the Preisach model with arbitrary precision with specific parameters for rate-independent hysteresis modeling. Experiments show that the classical dRNN models both kinds of hysteresis more accurately and efficiently than the Preisach model.

Aspect-based sentiment analysis for hotel reviews using an improved model of long short-term memory

Advances in information technology have given rise to online hotel reservation options. The user review feature is an important factor during the online booking of hotels. Generally, most online hotel booking service providers provide review and rating features for assessing hotels. However, not all service providers provide rating features or recap reviews for every aspect of the hotel services offered. Therefore, we propose a method to summarise reviews based on multiple aspects, including food, room, service, and location. This method uses long short-term memory (LSTM), together with hidden layers and automation of the optimal number of hidden neurons. The F1-measure value of 75.28% for the best model was based on the fact that (i) the size of the first hidden layer is 1,200 neurons with the tanh activation function, and (ii) the size of the second hidden layer is 600 neurons with the ReLU activation function. The proposed model outperforms the baseline model (also known as standard LSTM) by 10.16%. It is anticipated that the model developed through this study can be accessed by users of online hotel booking services to acquire a review recap on more specific aspects of services offered by hotels