Two-layer Neural Network Model Research Articles

HieNN-DWE: A hierarchical neural network with dynamic word embeddings for document level sentiment classification

The recent rapid developments of neural networks have stimulated significant performance progress on document level sentiment classification. The existing researches focus on extracting long-distance features from documents, which fail to exploit the deep semantic information of documents. Besides, few of them have taken the representations of polysemous words into consideration, such that the semantic information of documents are not precisely exploited. To tackle these issues, we propose a novel hierarchical neural network model based on dynamic word embeddings (HieNN-DWE) for document level sentiment classification. Specifically, the dynamic word embeddings are produced by ELMo (Embedding from Language Models), which can deeply exploit the semantic information for polysemous words. The embeddings are then fed into HieNN-DWE which is a two-layer hierarchical neural network model. Note that the first layer utilizes bidirectional gated recurrent unit (BiGRU) as well as attention mechanism to encode sentences. The second layer uses both BiGRU and convolutional neural network (CNN) to capture global and local features from sentence representations. Thus, the final document representations are constructed by concatenating the outputs from BiGRU and CNN. Moreover, experimental results have shown that HieNN-DWE outperforms existing competitive methods on four public datasets for document level sentiment classification.

Vehicle Deceleration Prediction Based on Deep Neural Network at Braking Conditions

The smart regenerative braking system in electric vehicles implements automatic control of the regeneration torque of motor to improve driver’s comfort and energy efficiency. To apply this system, the accurate prediction of the vehicle deceleration states is the preliminary step to reflect the driver’s behaviors. In this paper, we proposed a vehicle deceleration prediction model via deep neural network, which consists of a sequential recurrent neural network model with long-short term memory cell and a two-layer conventional neural network model. This model accommodates the physical constraint to designate the vehicle stop location in front of the traffic signals. The model is trained by vehicle experiment data with three drivers through the hyper-parameter optimization method. Using this model, the deceleration characteristics are characterized by two explicit parameters such that deceleration point, maximum point according to the initial slope and the shape of the braking profile. Using these two parameters as clustering variables through a K-means clustering method, the deceleration types are classified. These deceleration types to the input to the prediction model results in higher prediction accuracy of the vehicle states. The driving style of the three drivers at braking situations is analyzed according to the deceleration types as well.

Determination of Heavy Metal Soil Contaminants Based on Photoacoustic Spectroscopy

Heavy metal (HM) contamination in soil is a threat to human health and environmental safety. This paper presents an infrared photoacoustic spectroscopic non-destructive testing for HM evaluation by a robust customized photoacoustic spectrometric system. Cadmium (Cd) was selected as the target contamination which was blended in soil samples. A two-layer feed-forward artificial neural network (ANN) model was developed for HM concentration estimation. The results show that the standard normal variate and continuum removal methods are the best preprocessing algorithms regarding the maximal correction coefficient (R2 > 0.90) and the minimal root mean square error criteria. The correction coefficient analysis shows a better prediction performance (R2 > 0.95) for a HM concentration estimation with the two preprocessing methods. In conclusion, the prediction accuracy can be significantly improved by implementing specific preprocessing and training methods. The ANN model-based infrared photoacoustic spectroscopic method has a future potential for the featureless HM contamination estimation in soil.

A comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics

A fairly comprehensive analysis is presented for the gradient descent dynamics for training two-layer neural network models in the situation when the parameters in both layers are updated. General initialization schemes as well as general regimes for the network width and training data size are considered. In the over-parametrized regime, it is shown that gradient descent dynamics can achieve zero training loss exponentially fast regardless of the quality of the labels. In addition, it is proved that throughout the training process the functions represented by the neural network model are uniformly close to that of a kernel method. For general values of the network width and training data size, sharp estimates of the generalization error is established for target functions in the appropriate reproducing kernel Hilbert space.

Dynamic Behavior of the Solutions for a Two-layer Neural Network Model with Delays

Dynamic Behavior of the Solutions for a Two-layer Neural Network Model with Delays

Modelling and Optimizing Pyrene Removal from the Soil by Phytoremediation using Response Surface Methodology, Artificial Neural Networks, and Genetic Algorithm

This study aimed to model and optimize pyrene removal from the soil contaminated by sorghum bicolor plant using Response Surface Methodology (RSM) and Artificial Neural Network (ANN) with Genetic Algorithm (GA) approach. Here, the effects of indole acetic acid (IAA) and pseudomonas aeruginosa bacteria on increasing pyrene removal efficiency by phytoremediation process was studied. The experimental design was done using the Box-Behnken Design (BBD) technique. In the RSM model, the non-linear second-order model was in good agreement with the laboratory results. A two-layer Feed-Forward Back-Propagation Neural Network (FFBPNN) model was designed. Various training algorithms were evaluated and the Levenberg Marquardt (LM) algorithm was selected as the best one. Existence of eight neurons in the hidden layer leads to the highest R and lowest MSE and MAE. The results of the GA determined the optimum performance conditions. The results showed that using indole acetic acid and pseudomonas bacteria increased the efficiency of the sorghum plant in removing pyrene from the soil. The comparison obviously indicated that the prediction capability of the ANN model was much better than that of the RSM model.

Eski Dilde Kullanılan Sözcükler Arasındaki Anlamsal Yakınlıkların Doğal Dil İşleme Yöntemleriyle Tespiti

Leveraging machine learning techniques in NLP domain has been a very hot research field due to the advancements in artificial intelligence area. Despite the popularity of this field, there is no known study on application of ML techniques on old Turkish language. This study aims to fill in this gap where 32000 pages of text has been downloaded from the websites of Ministry of Culture and a two-layer neural network model has been built on top of them to discover the semantic similarities between Turkish words in old Turkish language. The algorithm has been run with different parameters such as window size, dimension size, sampling size etc. and the produced vector spaces are uploaded into public servers for the purposes of enabling a RESTful API based query interface. Also a web UI has been created to provide a querying mechanism for regular users. The services that are developed can be used for two different purposes. One of them is to integrate these services into existing old Turkish language dictionary websites that are made available by third party providers as well as other institutions such as Ministry of Culture and Turkish Language Institution. Secondly, the developed services are intended to be used for mitigating the translation errors made during the translation of old Turkish texts into modern Turkish language in the areas of history and Turkish literature. Also enabling these services for public use will encourage other researchers to pursue this academic work and compare their results with the experimental results presented in this paper to make further improvements in this field.

Neural Network Modelling to Characterize Steel Continuous Casting Process Parameters and Prediction of Casting Defects

The current work outlines application of a data-driven multilayer perceptron-based artificial neural network (ANN) model to characterize the influence of melt compositions, tundish temperature, tundish superheat, casting speed and mould oscillation frequency on the important processing parameters such as mould powder consumption rate, oscillation mark depth and metallurgical length during continuous casting process. A two-layer feedforward back-propagation neural network model has been developed for predicting the probability of occurrence of defect in the cast product. The network training architecture has been optimized using a gradient-based algorithm, namely the back-propagation algorithm. The neural network predictions are found to be in good agreement with regard to oscillation mark depth, mould powder consumption rate, metallurgical length and probability of occurrence of defect using data obtained from an operating Indian steel plant (Rashtriya Ispat Nigam Limited, Visakhapatnam). The ANN model prediction has been validated successfully with multiple linear regression analysis carried out on each data set.

Locations in the Neocortex: A Theory of Sensorimotor Object Recognition Using Cortical Grid Cells.

The neocortex is capable of anticipating the sensory results of movement but the neural mechanisms are poorly understood. In the entorhinal cortex, grid cells represent the location of an animal in its environment, and this location is updated through movement and path integration. In this paper, we propose that sensory neocortex incorporates movement using grid cell-like neurons that represent the location of sensors on an object. We describe a two-layer neural network model that uses cortical grid cells and path integration to robustly learn and recognize objects through movement and predict sensory stimuli after movement. A layer of cells consisting of several grid cell-like modules represents a location in the reference frame of a specific object. Another layer of cells which processes sensory input receives this location input as context and uses it to encode the sensory input in the object’s reference frame. Sensory input causes the network to invoke previously learned locations that are consistent with the input, and motor input causes the network to update those locations. Simulations show that the model can learn hundreds of objects even when object features alone are insufficient for disambiguation. We discuss the relationship of the model to cortical circuitry and suggest that the reciprocal connections between layers 4 and 6 fit the requirements of the model. We propose that the subgranular layers of cortical columns employ grid cell-like mechanisms to represent object specific locations that are updated through movement.

Predicting the helpfulness score of online reviews using convolutional neural network

The smart cities aim to provide an infrastructure to their citizens that reduces both their time and effort. An example of such an available infrastructure is electronic shopping. Electronic shopping has become the hotbeds of many customers as it is easier to judge the quality of the product based on the review information. The purpose of this study is to predict the best helpful online product review, out of the several thousand reviews available for the product using review representation learning. The prediction is done using a two-layered convolutional neural network model. The review texts are embedded into low-dimensional vectors using a pre-trained model. To learn the best features of the review text, three filters are used to learn tri-gram, four-gram, and five-gram features of the text. The proposed approach is found to be better than existing machine learning based models which used hand-crafted features. The very low value of mean squared error confirms the prediction accuracy of the proposed method. The proposed method can be easily applied to any kind of review as the features are calculated only from the review text and not from other domain knowledge. The proposed model helps in predicting the helpfulness score of new reviews as soon as it gets posted on the product review page.

Power Grid Enterprise Intelligent Risk Identification Model Considering Multi-Attribute and Low Correlation Data

Accurate risk calculation is beneficial to improve the accuracy of risk identification, which plays a crucial role in auditing in smart grid enterprises. With the development of smart grid, electric utilities have accumulated vast amount of data, big data which has low correlation with risk can cumulative effect on risk and the indicators that affect risk have multiple attributes. In view of above problem, in this paper, a two-layer neural network model is proposed, which inputs the multi-attribute of indicators and the full-scale indicators of different degrees of influence into the model. The problems that the accumulation of multi-attribute and multiple low-related factors of the indicators have an effect on risk calculation are solved in the model. This paper also builds a big data audit platform to carry out the intelligent data collection and risk calculate The results of the examples show that the model has higher accuracy of risk calculation.

Stability and Hopf bifurcation analysis of a simplified six-neuron tridiagonal two-layer neural network model with delays

Firstly, a general tridiagonal two-layer neural network model with 2n-neuron is proposed, where every layer has time delay. A new method of Hopf bifurcation analysis is introduced by matrix decomposition in this paper. Through factoring the tridiagonal matrix, the characteristic equation of the neural network model is simplified. Secondly, by studying the eigenvalue equations of the related linear system for the special six-neuron (three neurons per layer) two-layer neural network model, the sufficient conditions for experiencing the Hopf bifurcation are obtained. The conditions obtained by the new method proposed in this paper are simpler and more practical than those obtained by the traditional Hurwitz discriminant method. Next, based on the normal form method and the center manifold theorem, the explicit formulae about the stability of the bifurcating periodic solution and the direction of the Hopf bifurcation are established. Finally, the main results obtained in this paper are illustrated by three numerical simulation examples.

A Privacy-Preserving Identification Mechanism for Mobile Sensing Systems

In the applications for mobile sensing, the trustworthy of sensed data should be put on the first place. The identification of participants can ensure data trustworthy but will reveal the privacy of the participants to a great extent. In this paper, we propose a privacy-preserving identification mechanism for mobile sensing systems to select sensed data dynamically to protect participant’s sensitive information. It solves the contradiction between “privacy protection” and “identification”. It divides data privacy sensitivity of the data sensed from the task that participants attended, allowing participants to define their own privacy sensitivity, then selects sensed data dynamically and uses differential privacy to process the data with high privacy sensitivity. It can not only protect participants’ privacy, but also identify participants’ IDs. In order to achieve identification, a two-layer neural network model is used to train and learn the participant’s style of action and generate an identity feature database. The experimental results show that the proposed mechanism can provide a trustworthy platform for mobile sensing systems.

Blackbox modeling of central heating and cooling plant equipment performance

The current article presents an analysis conducted upon the sensor data gathered from the distribution control system of a central heating and cooling plant in Ottawa, Canada. After observing that the performance of four boilers and five chillers of this plant vary substantially in time under steady-state conditions, data-driven models were developed to explain this variability from the archived sensor data. By employing a forward stepwise regression and a repeated random sub-sampling cross-validation approach, two-layer feed-forward artificial neural network models with 7 to 15 hidden-nodes were selected for each boiler and chiller. The selected boiler models could explain 84% to 95% of the variability in a boiler's efficiency, and the selected chiller models could explain 65% to 94% of the variability in a chiller's coefficient of performance. Among studied nine variables, the most informative ones to predict a boiler's efficiency were identified as follows: flue gas O2 concentration, pressure, part-load ratio, forced draft fan state, and return water flow rate. Unlike boilers, all four studied variables were found useful in predicting a chiller's coefficient of performance. These four variables were the return water flow rate, part-load ratio, outdoor temperature, and return water temperature. A residual analysis was conducted to verify the appropriateness of the selected models to the datasets. In addition, potential use cases for the selected models were discussed with illustrative examples.

Prominent lateral spread of imaged evoked activity beyond cortical columns in barrel cortex provides foundation for coding whisker identity.

The posterior medial barrel subfield (PMBSF) of a rat primary somatosensory cortex exquisitely demonstrates topography and columnar organization, defining features of sensory cortices in the mammalian brain. Optical imaging and neuronal recordings in rat PMBSF demonstrate how evoked cortical activity following single whisker stimulation also rapidly spreads laterally into surrounding cortices, disregarding columnar and modality boundaries. The current study quantifies the spatial prominence of such lateral activity spreads by demonstrating that functional connectivity between laterally spaced cortical locations is actually stronger than between vertically spaced cortical locations. Further, the total amount of evoked activity within and beyond single column boundaries was quantified based on intrinsic signal optical imaging, single units and local field potentials recordings, revealing that the vast majority of whisker evoked activity in PMBSF occurs beyond columnar boundaries. Finally, a simple two-layer artificial neural network model of PMBSF demonstrates the capacity of extracolumnar evoked activity spread to provide a foundation for accurate whisker stimulus classification that is robust to random scaling of inputs and local noise. Indeed, classification performance improved when more of the lateral spread was included in the model, providing further evidence for the relevance of the lateral spread.

Artificial Neural Network Modelling for the Removal of Fe (III) from Aqueous Solutions Using Chitosan Magnetite Nano Composites (CMNs)

A two-layer Artificial Neural Network (ANN) model was engendered to soothsay the deliberation efficacy of Fe (II) particles from fluid arrangement utilizing chitosan magnetite nano composites (CMNs). The sorbet stock arrangement was yare by dissolving a pre-computed amount of FeCl3 in twofold refined water to give last fixation 100 mgl−1. The stock arrangement was debilitated to get standard arrangements with fixation in the scope of 5-30 mgl−1 and their last pH was transmuted in accordance with 4.5. Fifty millilitres of FeCl3 arrangement of fancied focus was put in a 125 ml Erlenmeyer flagon containing 0.02 g of CMN sorbent. A period of 3 hours was discovered adequate to accomplish the balance. The ANN model was intended to suspect sorption efficacy of CMNs for target metal particle by amalgamating back spread (BP) with guideline segment examination. A sigmoid axon was utilized as exchange capacity for information and yield layer. The Levenberg-Marquardt calculation (LMA) was connected, giving a base estimation of mean squared mistake (MSE) for preparing and cross approbation at the 6th place of decimal.

Expectation propagation for neural networks with sparsity-promoting priors

We propose a novel approach for nonlinear regression using a two-layer neural network (NN) model structure with sparsity-favoring hierarchical priors on the network weights. We present an expectati...

Two-layer tree-connected feed-forward neural network model for neural cryptography

Neural synchronization by means of mutual learning provides an avenue to design public key exchange protocols, bringing about what is known as neural cryptography. Two identically structured neural networks learn from each other and reach full synchronization eventually. The full synchronization enables two networks to have the same weight, which can be used as a secret key for many subsequent cryptographic purposes. It is striking to observe that after the first decade of neural cryptography, the tree parity machine (TPM) network with hidden unit $K=3$ appears to be the sole network that is suitable for a neural protocol. No convincingly secure neural protocol is well designed by using other network structures despite considerable research efforts. With the goal of overcoming the limitations of a suitable network structure, in this paper we develop a two-layer tree-connected feed-forward neural network (TTFNN) model for a neural protocol. The TTFNN model captures the notion that two partners are capable of exchanging a vector with multiple bits in each time step. An in-depth study of the dynamic process of TTFNN-based protocols is then undertaken, based upon which a feasible condition is theoretically obtained to seek applicable protocols. Afterward, according to two analytically derived heuristic rules, a complete methodology for designing feasible TTFNN-based protocols is elaborated. A variety of feasible neural protocols are constructed, which exhibit the effectiveness and benefits of the proposed model. With another look from the perspective of application, TTFNN-based instances, which can outperform the conventional TPM-based protocol with respect to synchronization speed, are also experimentally confirmed.

Prediction of Turnover Number of Cellulose 1,4-Beta-Cellobiosidase

The turnover number is an important parameter to distinguish whether an enzyme is practically workable. Therefore the prediction of turnover number of enzyme will reduce the workload to conduct time-consuming and costly experiments to determine the turnover number. However, no studies have been so far conducted to predict them with respect to cellulose 1,4-beta-cellobiosidase, which is an enzyme used in industries, especially in bio-fuel industry. It is important to develop methods to predict the turnover numbers of cellulose 1,4-beta-cellobiosidases from both wild-type and mutations. In this study, we used neural network models with different amino acid properties, pH levels, temperatures and substrates as inputs to predict the turnover number. The results show that 11 out of 25 amino acid properties analyzed can work as predictor and the amino acid distribution probability is the best one because it can reach smaller mean squared errors during convergence and higher correlation coefficient in two-layer neural network models. This study demonstrates the probability that the neural network model can approximately predict the turnover number of cellulose 1,4-betacellobiosidase.

Depression-Biased Reverse Plasticity Rule Is Required for Stable Learning at Top-Down Connections

Top-down synapses are ubiquitous throughout neocortex and play a central role in cognition, yet little is known about their development and specificity. During sensory experience, lower neocortical areas are activated before higher ones, causing top-down synapses to experience a preponderance of post-synaptic activity preceding pre-synaptic activity. This timing pattern is the opposite of that experienced by bottom-up synapses, which suggests that different versions of spike-timing dependent synaptic plasticity (STDP) rules may be required at top-down synapses. We consider a two-layer neural network model and investigate which STDP rules can lead to a distribution of top-down synaptic weights that is stable, diverse and avoids strong loops. We introduce a temporally reversed rule (rSTDP) where top-down synapses are potentiated if post-synaptic activity precedes pre-synaptic activity. Combining analytical work and integrate-and-fire simulations, we show that only depression-biased rSTDP (and not classical STDP) produces stable and diverse top-down weights. The conclusions did not change upon addition of homeostatic mechanisms, multiplicative STDP rules or weak external input to the top neurons. Our prediction for rSTDP at top-down synapses, which are distally located, is supported by recent neurophysiological evidence showing the existence of temporally reversed STDP in synapses that are distal to the post-synaptic cell body.