Tanh Activation Function Research Articles

Intelligent and Analog CMOS ASIC Development of Angular Rate Error Compensation for MEMS Gyroscope

Background & Objective: MEMS-based gyroscopes are used in angular rate detection where precision is an important parameter; however, gyroscope output is limited by angular rate error. For minimizing these types of non-idealities, conventional external hardware-based analog or digital circuits have limitations for using in compact applications. CMOS analog ASIC for angular rate error compensation is necessary as both MEMS-CMOS technologies are supplementary and compatible. Method: In this paper, the output of MEMS gyroscope is taken as input for the compensation circuit which results in compensated angular rate. ANN is used in intelligent compensation circuit for error reduction in which offline data is trained and minimum optimum error of MSE of 1.72e-4 is achieved. ANN uses tanh sigmoidal activation function and back propagation trained MLP model with three neurons in the hidden layer. The equivalent ANN is implemented by CMOS ASIC where each neuron is implemented using Gilbert multiplier cell, differential analog adder, and differential amplifier as tanh sigmoidal circuit using OrCAD-PSpice 10.5 with 0.35 μ m technology. These blocks consist of differential configuration which has the capability of common mode interference rejection as noise becomes comparable at lower values of input analog signal. The entire ASIC consumes 77.8 mW of power which is far less and compact in size as compared to available external hardware interface circuits. Result and Conclusion: MEMS gyroscope with proposed analog ASIC becomes smart sensor with ANN based intelligent interface circuit. The proposed compensation cum interface circuit gives the average angular rate error of 1.91% in the range of minimum 0% to maximum 27% leading to improved accuracy.

Speed Up the Training of Neural Machine Translation

Neural machine translation (NMT) has achieved notable achievements in recent years. Although existing models provide reasonable translation performance, they cost too much training time. Especially, when the corpus is enormous, their computational cost will be extremely high. In this paper, we propose a novel NMT model based on the conventional bidirectional recurrent neural network (bi-RNN). In this model, we apply a tanh activation function, which can learn the future and history context information more sufficiently, to speed up the training process. Experimental results on tasks of German–English and English–French translation demonstrate that the proposed model can save much training time compared with the state-of-the-art models and provide better translation performances.

Dual Rectified Linear Units (DReLUs): A replacement for tanh activation functions in Quasi-Recurrent Neural Networks

In this paper, we introduce a novel type of Rectified Linear Unit (ReLU), called a Dual Rectified Linear Unit (DReLU). A DReLU, which comes with an unbounded positive and negative image, can be used as a drop-in replacement for a tanh activation function in the recurrent step of Quasi-Recurrent Neural Networks (QRNNs) [1]. Similar to ReLUs, DReLUs are less prone to the vanishing gradient problem, they are noise robust, and they induce sparse activations.We independently reproduce the QRNN experiments of Bradbury et al. [1] and compare our DReLU-based QRNNs with the original tanh-based QRNNs and Long Short-Term Memory networks (LSTMs) on sentiment classification and word-level language modeling. Additionally, we evaluate on character-level language modeling, showing that we are able to stack up to eight QRNN layers with DReLUs, thus making it possible to improve the current state-of-the-art in character-level language modeling over shallow architectures based on LSTMs.

An Automatic Coffee Plant Diseases Identification Using Hybrid Approaches of Image Processing and Decision Tree

<p>Coffee Leaf Rust (CLR), Coffee Berry Disease (CBD) and Coffee Wilt Disease (CWD) are the three main diseases that attack coffee plants. This paper presents the identification of these types diseases using hybrid approaches of image processing and decision tree. The images are taken from Southern Ethiopia, Jimma and Zegie. In this paper backpropagation artificial neural network (BPNN) and decision tree had been used as techniques; a total of 9100 images were collected. From these, 70% are used for training and the remaining 30% are used for testing. In general, 94.5% accuracy achieved when decision tree and BPNN with tanh activation function are combined.</p>

Deep reinforcement learning: Algorithm, applications, and ultra-low-power implementation

In order to overcome the limitation of traditional reinforcement learning techniques on the restricted dimensionality of state and action spaces, the recent breakthroughs of deep reinforcement learning (DRL) in Alpha Go and playing Atari set a good example in handling large state and action spaces of complicated control problems. The DRL technique is comprised of an offline deep neural network (DNN) construction phase and an online deep Q-learning phase. In the offline phase, DNNs are utilized to derive the correlation between each state–action pair of the system and its value function. In the online phase, a deep Q-learning technique is adopted based on the offline-trained DNN to derive the optimal action and meanwhile update the value estimates and the DNN.This paper is the first to provide a comprehensive study of applications of the DRL framework on cloud computing and residential smart grid systems along with efficient hardware implementations. Based on the introduction of the general DRL framework, we develop two applications, one for the cloud computing resource allocation problem and one for the residential smart grid user-end task scheduling problem. The former could achieve up to 54.1% energy saving compared with baselines through automatically and dynamically distributing resources to servers. The latter achieves up to 22.77% total energy cost reduction compared with the baseline algorithm.The DRL framework is mainly utilized for the complicated control problems and requires light-weight and low-power implementations in edge and portable systems. In order to achieve this goal, we develop the ultra-low-power implementation of the DRL framework using the stochastic computing technique, which has the potential of significantly enhancing the computation speed and reducing hardware footprint and therefore the power/energy consumption. The overall implementation is based on the effective stochastic computing-based implementations of approximate parallel counter-based inner product blocks and tanh activation functions. The stochastic computing-based implementation achieves only 57941.61 μm2 area and 6.30 mW power with 412.47 ns delay.

Predicting the Future with Artificial Neural Network

Accurate prediction of future values of time series data is crucial for strategic decision making such as inventory management, budget planning, customer relationship management, marketing promotion, and efficient allocation of resources. However, time series prediction can be very challenging especially when there are elements of uncertainty including natural disaster, change in government policies and weather condition. In this research, four different multilayer perceptron (MLP) artificial neural networks have been discussed and compared with Autoregressive Integrated Moving Average (ARIMA) for this task. The models are evaluated using two statistical performance evaluation measures, Root Mean Squared Error (RMSE) and coefficient of determination (R2). The experimental result shows that a 4-layer MLP architecture using the tanh activation function in each of the hidden layer and a linear function in the output layer has the lowest prediction error and the highest coefficient of determination among the configured multilayer perceptron neural networks. In addition, comparative analysis of performance result reveals that the multilayer perceptron neural network MLP has a lower prediction error than the ARIMA model.

Predicting Thermal Behavior of Secondary Organic Aerosols.

Volume concentrations of secondary organic aerosol (SOA) are measured in 139 steady-state, single precursor hydrocarbon oxidation experiments after passing through a temperature controlled inlet. The response to change in temperature is well predicted through a feedforward Artificial Neural Network. The most parsimonious model, as indicated by Akaike's Information Criterion, Corrected (AIC,C), utilizes 11 input variables, a single hidden layer of 4 tanh activation function nodes, and a single linear output function. This model predicts thermal behavior of single precursor aerosols to less than ±5%, which is within the measurement uncertainty, while limiting the problem of overfitting. Prediction of thermal behavior of SOA can be achieved by a concise number of descriptors of the precursor hydrocarbon including the number of internal and external double bonds, number of methyl- and ethyl- functional groups, molecular weight, and number of ring structures, in addition to the volume of SOA formed, and an indicator of which of four oxidant precursors was used to initiate reactions (NOx photo-oxidation, photolysis of H2O2, ozonolysis, or thermal decomposition of N2O5). Additional input variables, such as chamber volumetric residence time, relative humidity, initial concentration of oxides of nitrogen, reacted hydrocarbon concentration, and further descriptors of the precursor hydrocarbon, including carbon number, number of oxygen atoms, and number of aromatic ring structures, lead to over fit models, and are unnecessary for an efficient, accurate predictive model of thermal behavior of SOA. This work indicates that predictive statistical modeling methods may be complementary to descriptive techniques for use in parametrization of air quality models.

Short term wind speed prediction using multiple kernel pseudo inverse neural network

An accurate short-term wind speed prediction algorithm based on the efficient kernel ridge pseudo inverse neural network (KRPINN) variants is proposed in this paper. The use of nonlinear kernel functions in pseudo inverse neural networks eliminates the trial and error approach of choosing the number of hidden layer neurons and their activation functions. The robustness of the proposed method has been validated in comparison with other models such as pseudo inverse radial basis function (PIRBF) and Legendre tanh activation function based neural network, i.e., PILNNT, whose input weights to the hidden layer weights are optimized using an adaptive firefly algorithm, i.e., FFA. However, since the individual kernel functions based KRPINN may not be able to produce accurate forecasts under chaotically varying wind speed conditions, a linear combination of individual kernel functions is used to build the multi kernel ridge pseudo inverse neural network (MK-RPINN) for providing improved forecasting accuracy, generalization, and stability of the wind speed prediction model. Several case studies have been presented to validate the accuracy of the short-term wind speed prediction models using the real world wind speed data from a wind farm in the Wyoming State of USA over time horizons varying from 10 minutes to 5 hours.

Speech Processing for Text Independent Amharic Language Dialect Recognition

<span style="color: #666666; font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 11.2px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; display: inline !important; float: none;">Dialect is a difference of verbal communication spoken by people from a particular society or geographic area so the paper focuses on Amharic language dialect recognition. In this paper, the authors have used backpropagation artificial neural network, VQ(vector quantization), (Gaussian Mixture Models) and a combination of GMM and backpropagation artificial neural network for classifying dialects of Amharic language speakers. In this research, a total of 100 speakers for each group of dialects are considered each having about 10 seconds duration is collected. The feature vectors of Mel frequency cepstral coefficients (MFCC) had been used to recognize the dialects of speakers. In this research paper the recognition model that uses a tanh activation function have a better result instead of using the Logistic Sigmoid activation function in backpropagation artificial neural network. After conducting the above experiments 95.7% accuracy achieved when GMM and backpropagation artificial neural network with tanh activation function are combined.</span>

Prediction of ultrasonic osmotic dehydration properties of courgette by ANN

In this research, ultrasound assisted osmotic dehydration of courgette rings using sorbitol/sucrose solution under different temperature (5, 25 and 50 °C for 2 h) was investigated. Sucrose (35%, w/v) and sorbitol solutions (5, 10 and 15%, w/v) were used for osmotic dehydration processes. The reliability of using an artificial neural network (ANN) approach for predicting the osmotic dehydration properties of courgette was investigated. Immersion time, type of treatment, osmotic solution temperature and concentration were selected as input variables and solid gain and water loss were chosen as the outputs of the network. Results showed that all processing factors had a significant effect on the solid gain and water loss (P<0.01). Increasing osmotic solution concentration and temperature lead to increases in water loss and solid gain for both samples of ultrasonicated and non-ultrasonicated treatments. The results of ANN indicated that, tanh activation function with 46 neurons in first and second hidden laye...

Modeling the Effect of Crude Oil Impacted Sand on the Properties of Concrete Using Artificial Neural Networks

A network of the feedforward-type artificial neural networks (ANNs) was used to predict the compressive strength of concrete made from crude oil contaminated soil samples at 3, 7, 14, 28, 56, 84, and 168 days at different degrees of contamination of 2.5%, 5%, 10%, 15%, 20% and 25%. A total of 49 samples were used in the training, testing, and prediction phase of the modeling in the ratio 32 : 11 : 7. The TANH activation function was used and the maximum number of iterations was limited to 20,000 the model used a momentum of 0.6 and a learning rate of 0.031056. Twenty (20) different architectures were considered and the most suitable one was the 2-2-1. Statistical analysis of the output of the network was carried out and the correlation coefficient of the training and testing data is 0.9955712 and 0.980097. The result of the network has shown that the use of neural networks is effective in the prediction of the compressive strength of concrete made from crude oil impacted sand.

AN FUZZY NEURAL APPROACH FOR MEDICAL IMAGE RETRIEVAL

Image retrieval based on a query image is necessary for effective and efficient use the information that is stored in medical image databases. Medical Image Retrieval is difficult as not only the localization and directionality of human visual system is to be considered but also the pathological condition. Image identification and segmentation for feature extraction pose a challenge to image retrieval process. Challenges posed include large number of images to be processed for the image retrieval and identifying the region of interest automatically to optimize the search. In this study, we propose a novel image segmentation algorithm Fuzzy Edge Detection and Segmentation (FEDS). The proposed FEDS algorithm is tested on medical images and for classification of images, a bell fuzzy multilayer perceptron is proposed. The proposed neural network Bell Fuzzy Multi Layer Perceptron (BF-MLP) Neural network is constructed by introducing a fuzzy logic in hidden layer with the sugeno model and bell function. The proposed neural network consists of two layers with the first layer being a tanh activation function and the second layer containing the bell fuzzy activation function. The proposed FEDS method was implemented using Matlab and Modelsim. A total of 44 images were considered with three class labels. The edge obtained for which segmentation is done using the proposed segmentation algorithm. The proposed BF-MLP neural network algorithm was implemented using Visual Studio and the classification accuracy compared with MLP Neural Network with sigmoid activation function. In this study, a fuzzy segmentation algorithm and a fuzzy classification algorithm is proposed to improve the medical image retrieval accuracy. The proposed segmentation algorithm, Fuzzy Edge Detection and Segmentation (FEDS), was implemented using Matlab and features were extracted using Fast Hartley Transform (FHT). The extracted features were used to train the proposed neural network, Bell Fuzzy Multi Layer Perceptron Neural Network (BF-MLP). 44 images with 3 class labels were used to test the algorithm and classification accuracy of 93.2% was obtained.

Neural Network Approaches for Prediction of Pistachio Drying Kinetics

Abstract Thin-layer drying characteristics of whole pistachio were investigated by using a hot air convective dryer at a constant airflow velocity of 2 m.s-1 and air temperature in the range of 40-70°C. The experimental drying data were fitted to the eight well-known drying models i.e. the Newton, Page, Modified Page, Henderson and Pabis, Logarithmic, Diffusion, Thomson and New models. A predictive model using artificial neural network was proposed in order to obtain on-line predictions of moisture kinetics during drying of pistachio nut. For monitoring the drying process of pistachio, different activation function of neural networks such as tangent hyperbolic (tanh) and logarithmic sigmoid (logsig) were utilized. Drying time and air drying temperature were considered as network input and moisture ratio was as network output. The result indicated that tanh activation function gave better results than logsig activation function for monitoring the moisture ratio. Generally, perceptron neural network with logsig activation function as a goodness activation function was able to predict moisture ratio with 7 neuron in first and second hidden layer with R2 value equal 0.994. Investigation of validation data demonstrated that the predicted and experimental dying data were in good agreement. Comparing the R2 (coefficient of determination) and MAE using the developed ANN model it was concluded that the neural network could be used for on-line state estimation of drying characteristics and control of drying processes.

Utility of coactive neuro-fuzzy inference system for pan evaporation modeling in comparison with multilayer perceptron

Estimation of pan evaporation (E pan) using black-box models has received a great deal of attention in developing countries where measurements of E pan are spatially and temporally limited. Multilayer perceptron (MLP) and coactive neuro-fuzzy inference system (CANFIS) models were used to predict daily E pan for a semi-arid region of Iran. Six MLP and CANFIS models comprising various combinations of daily meteorological parameters were developed. The performances of the models were tested using correlation coefficient (r), root mean square error (RMSE), mean absolute error (MAE) and percentage error of estimate (PE). It was found that the MLP6 model with the Momentum learning algorithm and the Tanh activation function, which requires all input parameters, presented the most accurate E pan predictions (r = 0.97, RMSE = 0.81 mm day−1, MAE = 0.63 mm day−1 and PE = 0.58 %). The results also showed that the most accurate E pan predictions with a CANFIS model can be achieved with the Takagi–Sugeno–Kang (TSK) fuzzy model and the Gaussian membership function. Overall performances revealed that the MLP method was better suited than CANFIS method for modeling the E pan process.

Neural network tomography: Network replication from output surface geometry

Multilayer perceptron networks whose outputs consist of affine combinations of hidden units using the tanh activation function are universal function approximators and are used for regression, typically by reducing the MSE with backpropagation. We present a neural network weight learning algorithm that directly positions the hidden units within input space by numerically analyzing the curvature of the output surface. Our results show that under some sampling requirements, this method can reliably recover the parameters of a neural network used to generate a data set.

\title{Connectionist model of action learning and naming}

\title{Connectionist model of action learning and naming}

Short term flood forecasting using General Recurrent neural network modeling a comparative study

artificial neural networks (ANNs) have been applied to various hydrologic problems recently. This research demonstrates dynamic neural approach by applying general recurrent neural network to rainfall-runoff modeling for the upper area of Wardha River in India. The model is developed by processing online data over time using dynamic modeling. Methodologies and techniques by applying different learning rule, activation function and input layer structure are presented in this paper and a comparison for the short term runoff prediction results between them is also conducted. The prediction results of the general recurrent neural network with Momentum learning rule and Tanh activation function with Axon as input layer structure indicates a satisfactory performance in the three hours ahead of time prediction. The conclusions also indicate that general recurrent neural network with Momentum learning rule and Tanh activation function with Axon as input layer structure is more versatile than other combinations for general recurrent neural network and can be considered as an alternate and practical tool for predicting short term flood flow. General Terms:- Prediction, Radial basis function neural network, Time lagged recurrent neural network.

Correction to “Lower Bounds on VC-Dimension of Smoothly Parameterized Function Classes”

The earlier article gives lower bounds on the VC-dimension of various smoothly parameterized function classes. The results were proved by showing a relationship between the uniqueness of decision boundaries and the VC-dimension of smoothly parameterized function classes. The proof is incorrect; there is no such relationship under the conditions stated in the article. For the case of neural networks with tanh activation functions, we give an alternative proof of a lower bound for the VC-dimension proportional to the number of parameters, which holds even when the magnitude of the parameters is restricted to be arbitrarily small.

On the Geometry of Feedforward Neural Network Error Surfaces

Many feedforward neural network architectures have the property that their overall input-output function is unchanged by certain weight permutations and sign flips. In this paper, the geometric structure of these equioutput weight space transformations is explored for the case of multilayer perceptron networks with tanh activation functions (similar results hold for many other types of neural networks). It is shown that these transformations form an algebraic group isomorphic to a direct product of Weyl groups. Results concerning the root spaces of the Lie algebras associated with these Weyl groups are then used to derive sets of simple equations for minimal sufficient search sets in weight space. These sets, which take the geometric forms of a wedge and a cone, occupy only a minute fraction of the volume of weight space. A separate analysis shows that large numbers of copies of a network performance function optimum weight vector are created by the action of the equioutput transformation group and that these copies all lie on the same sphere. Some implications of these results for learning are discussed.