Two-layer Recurrent Neural Network Research Articles

Deep Time-Aware Attention Neural Network for Sequential Recommendation

For click-through rate (CTR) prediction tasks, a good prediction performance can be obtained by full explorations of both user behavior and item behavior. Since user’s interests have a great influence on user’s behaviors, it is very important to learn users’ intrinsic interests according to their behaviors. User interests are not only diverse but also in dynamic change. However, the dynamics of user interests’ change are not fully taken into account by the majority of current CTR models. The latest sequential recommendation algorithm ignores the subjectivity of users when it uses a two-layer recurrent neural network to model the item behavior from the perspective of the evolution of items. In this work, we propose a recurrent neural network model called DTIAN (Deep Time-Aware Interest Attention Network) to address these issues. By leveraging the user behaviors and the corresponding temporal information, DTIAN captures user interests and intent changes to the target item. Therefore, the users’ recent interests are enhanced compared to early interests with the attention mechanism. In addition, each module of the proposed model can be plugged into other mainstream models to improve the performance of current models. The experimental results show that the proposed DTIAN can outperform the current popular CTR prediction models slightly and significantly reduce the training time, which makes it possible to implement lightweight models.

A Mixed Deep Recurrent Neural Network for MEMS Gyroscope Noise Suppressing

Currently, positioning, navigation, and timing information is becoming more and more vital for both civil and military applications. Integration of the global navigation satellite system and /inertial navigation system is the most popular solution for various carriers or vehicle positioning. As is well-known, the global navigation satellite system positioning accuracy will degrade in signal challenging environments. Under this condition, the integration system will fade to a standalone inertial navigation system outputting navigation solutions. However, without outer aiding, positioning errors of the inertial navigation system diverge quickly due to the noise contained in the raw data of the inertial measurement unit. In particular, the micromechanics system inertial measurement unit experiences more complex errors due to the manufacturing technology. To improve the navigation accuracy of inertial navigation systems, one effective approach is to model the raw signal noise and suppress it. Commonly, an inertial measurement unit is composed of three gyroscopes and three accelerometers, among them, the gyroscopes play an important role in the accuracy of the inertial navigation system’s navigation solutions. Motivated by this problem, in this paper, an advanced deep recurrent neural network was employed and evaluated in noise modeling of a micromechanics system gyroscope. Specifically, a deep long short term memory recurrent neural network and a deep gated recurrent unit–recurrent neural network were combined together to construct a two-layer recurrent neural network for noise modeling. In this method, the gyroscope data were treated as a time series, and a real dataset from a micromechanics system inertial measurement unit was employed in the experiments. The results showed that, compared to the two-layer long short term memory, the three-axis attitude errors of the mixed long short term memory–gated recurrent unit decreased by 7.8%, 20.0%, and 5.1%. When compared with the two-layer gated recurrent unit, the proposed method showed 15.9%, 14.3%, and 10.5% improvement. These results supported a positive conclusion on the performance of designed method, specifically, the mixed deep recurrent neural networks outperformed than the two-layer gated recurrent unit and the two-layer long short term memory recurrent neural networks.

Bidirectional constraint satisfaction in rational strategic decision making

We describe the properties of a constraint satisfaction network that is able to reason and decide rationally in strategic games. We use the structure of Bidirectional Associative Memory (BAM), a minimal two-layer recurrent neural network, and assume that network layers represent self and other strategies, whereas connection weights encode best responses. We apply BAM to finite-strategy two-player games, and show that network activation in the long run is restricted to the set of rationalizable strategies. The network is not guaranteed to reach a stable activation state, but any pure strategy profile that constitutes a stable state in the network must be a pure strategy Nash equilibrium. We illustrate the properties of the network using the traveler’s dilemma, the rock–paper–scissors game, and coordination games. The network’s behavior also depends on starting activation states, and we show how biases in these starting states can resolve equilibrium selection problems. Strategic decision making is a key part of complex social behavior, and our results illustrate how bidirectional constraint satisfaction networks can perform rational computations in this domain.

Looking deeper and transferring attention for image captioning

Image captioning is a challenging task which requires not only to extract semantic information but also to generate descriptions with correct sentences. Most of the previous researches employ one-layer or two-layer Recurrent Neural Network (RNN) as the language model to predict sentence words. The language model may easily deal with the word information for a noun or an object, however, it may not be able to learn a verb or an adjective. To address this issue, a deep attention based language model is proposed to learn more abstract word information and three stacked approaches are designed to process attention. The proposed model makes full use of the Long Short Term Memory (LSTM) network and employs the transferred current attention to enhance extra spatial information. The experimental results on the benchmark MSCOCO and Flickr30K datasets have verified the effectiveness of the proposed model.

Tube-Based Robust Model Predictive Control of Nonlinear Systems via Collective Neurodynamic Optimization

This paper presents a collective neurodynamic approach to robust model predictive control (MPC) of discrete-time nonlinear systems affected by bounded uncertainties. The proposed control law is a combination of an MPC within an invariant tube for a nominal system and an ancillary state feedback control. The nominal system is first transformed to a linear parameter-varying (LPV) system, and then its MPC signal is computed by solving a convex optimization problem sequentially in real time using a two-layer recurrent neural network (RNN). The ancillary state feedback control is obtained by means of gain scheduling via robust pole assignment using two RNNs. While the nominal MPC generates an optimal state trajectory in the absence of uncertainties, the ancillary state feedback control confines the actual states within an invariant tube in the presence of uncertainties. Simulation results on stabilization control of three mechatronic systems are provided to substantiate the effectiveness and characteristics of the neurodynamics-based robust MPC approach.

A two-layer recurrent neural network for nonsmooth convex optimization problems.

In this paper, a two-layer recurrent neural network is proposed to solve the nonsmooth convex optimization problem subject to convex inequality and linear equality constraints. Compared with existing neural network models, the proposed neural network has a low model complexity and avoids penalty parameters. It is proved that from any initial point, the state of the proposed neural network reaches the equality feasible region in finite time and stays there thereafter. Moreover, the state is unique if the initial point lies in the equality feasible region. The equilibrium point set of the proposed neural network is proved to be equivalent to the Karush-Kuhn-Tucker optimality set of the original optimization problem. It is further proved that the equilibrium point of the proposed neural network is stable in the sense of Lyapunov. Moreover, from any initial point, the state is proved to be convergent to an equilibrium point of the proposed neural network. Finally, as applications, the proposed neural network is used to solve nonlinear convex programming with linear constraints and L1 -norm minimization problems.

Robust model predictive control of nonlinear systems with unmodeled dynamics and bounded uncertainties based on neural networks.

This paper presents a neural network approach to robust model predictive control (MPC) for constrained discrete-time nonlinear systems with unmodeled dynamics affected by bounded uncertainties. The exact nonlinear model of underlying process is not precisely known, but a partially known nominal model is available. This partially known nonlinear model is first decomposed to an affine term plus an unknown high-order term via Jacobian linearization. The linearization residue combined with unmodeled dynamics is then modeled using an extreme learning machine via supervised learning. The minimax methodology is exploited to deal with bounded uncertainties. The minimax optimization problem is reformulated as a convex minimization problem and is iteratively solved by a two-layer recurrent neural network. The proposed neurodynamic approach to nonlinear MPC improves the computational efficiency and sheds a light for real-time implementability of MPC technology. Simulation results are provided to substantiate the effectiveness and characteristics of the proposed approach.

Model Predictive Control for Tracking of Underactuated Vessels Based on Recurrent Neural Networks

In this paper, a model predictive control (MPC) scheme is presented for tracking of underactuated vessels with only two available controls: namely, surge force and yaw moment. When no external disturbance is explicitly considered, the proposed MPC approach iteratively solves a formulated quadratic programming (QP) problem using a single-layer recurrent neural network called the general projection network over a finite receding horizon. When additive disturbances are taken into account, a reformulated minimax optimization problem is iteratively solved by using a two-layer recurrent neural network. The applied neural networks are both stable in the sense of Lyapunov and globally convergent to the exact optimal solutions of reformulated convex programming problems. Simulation results are provided to demonstrate the effectiveness and characteristics of the proposed neurodynamics-based MPC approaches to vessel tracking control.

A two-layer recurrent Neural Network based approach for overlay multicast

Overlay multicast has become one of the most promising multicast solutions for IP network, and Neutral Network(NN) has been a good candidate for searching optimal solutions to the constrained shortest routing path in virtue of its powerful capacity for parallel computation. Though traditional Hopfield NN can tackle the optimization problem, it is incapable of dealing with large scale networks due to the large number of neurons. In this paper, a neural network for overlay multicast tree computation is presented to reliably implement routing algorithm in real time. The neural network is constructed as a two-layer recurrent architecture, which is comprised of Independent Variable Neurons (IDVN) and Dependent Variable Neurons (DVN), according to the independence of the decision variables associated with the edges in directed graph. Compared with the heuristic routing algorithms, it is characterized as shorter computational time, fewer neurons, and better precision.

Neural network-based modeling and parameter identification of switched reluctance motors

Phase windings of switched reluctance machines are modeled by a nonlinear inductance and a resistance that can be estimated from standstill test data. During online operation, the model structures and parameters of SRMs may differ from the standstill ones because of saturation and losses, especially at high current. To model this effect, a damper winding is added into the model structure. This paper proposes an application of artificial neural network to identify the nonlinear model of SRMs from operating data. A two-layer recurrent neural network has been adopted here to estimate the damper currents from phase voltage, phase current, rotor position, and rotor speed. Then, the damper parameters can be identified using maximum likelihood estimation techniques. Finally, the new model and parameters are validated from operating data.

A two-layer recurrent neural network for redundancy resolution of manipulators through joint torque minimization

A two-layer recurrent neural network for redundancy resolution of manipulators with local joint torque minimization is presented. The proposed recurrent neural network is composed of two bidirectionally connected layers of neuron arrays. While the command signals of desired acceleration of the end-effector are fed into the input layer, the output layer generates the joint acceleration vector of the manipulator with joint torque being minimized. The redundancy resolution obtained by the proposed recurrent neural network approach is compared with that obtained by previous literature. The proposed recurrent neural network is shown to be capable of generating redundancy resolution of manipulators with global stable minimized joint torque in real time.

Truck Backer-Upper Control using Dynamic Neural Network

A dynamic neural processor (DNP) is used in this paper to solve the problem of backing a trailer truck to the loading dock. It is demonstrated through computer simulations that the DNP could provide the proper steering for backing the truck to the loading zone very quickly compared to a two-layer recurrent neural network.