Discrete-time Recurrent Neural Networks Research Articles

Supervised Feature Selection via Collaborative Neurodynamic Optimization.

As a crucial part of machine learning and pattern recognition, feature selection aims at selecting a subset of the most informative features from the set of all available features. In this article, supervised feature selection is at first formulated as a mixed-integer optimization problem with an objective function of weighted feature redundancy and relevancy subject to a cardinality constraint on the number of selected features. It is equivalently reformulated as a bound-constrained mixed-integer optimization problem by augmenting the objective function with a penalty function for realizing the cardinality constraint. With additional bilinear and linear equality constraints for realizing the integrality constraints, it is further reformulated as a bound-constrained biconvex optimization problem with two more penalty terms. Two collaborative neurodynamic optimization (CNO) approaches are proposed for solving the formulated and reformulated feature selection problems. One of the proposed CNO approaches uses a population of discrete-time recurrent neural networks (RNNs), and the other use a pair of continuous-time projection networks operating concurrently on two timescales. Experimental results on 13 benchmark datasets are elaborated to substantiate the superiority of the CNO approaches to several mainstream methods in terms of average classification accuracy with three commonly used classifiers.

Input-to-state Practical Stability of Event-triggered Estimators for Discrete-time Recurrent Neural Networks With Unknown Time-delay

Input-to-state Practical Stability of Event-triggered Estimators for Discrete-time Recurrent Neural Networks With Unknown Time-delay

Real-Time Tracking Control and Efficiency Analyses for Stewart Platform Based on Discrete-Time Recurrent Neural Network

Real-Time Tracking Control and Efficiency Analyses for Stewart Platform Based on Discrete-Time Recurrent Neural Network

Theoretical analysis of co-existing periodic orbits in sparse binary neural networks

A sparse binary neural network is a discrete-time recurrent neural network characterized by local binary connections and signum-type neuron models. The dynamics is described by an autonomous difference equation of binary state variables. Depending on the parameters, the network generates various periodic orbits of binary vectors. Multiple periodic orbits can co-exist and the network exhibits one of them depending on initial condition. Real/potential applications of the periodic orbits include control signals of switching circuits and basic signals for time-series approximation. The network is well suited for theoretical analysis and simple hardware implementation. This paper considers shift-type periodic orbits in the networks. As the main theoretical results, we clarify the number, period, and stability of the periodic orbits for key parameters. As a first step to the applications in the future, we present an FPGA-based hardware prototype and confirm typical periodic orbits experimentally.

Tracking Control of Cable-Driven Planar Robot Based on Discrete-Time Recurrent Neural Network With Immediate Discretization Method

In recent years, the cable-driven planar robot has made fruitful achievements in many fields, but the related researches are scarce yet in the industrial engineering field. In this paper, as a powerful tool for solving discrete time-varying problems, discrete-time recurrent neural network (DTRNN) is extended to drive the cable-driven planar robot for discrete real-time tracking control, which is derived by a new immediate discretization method, and thus is termed as ID-DTRNN model. Specifically, firstly, we present the physical structure and mathematical model of cable-driven planar robot. Then, new ID-DTRNN model is proposed and applied for driving such cable-driven planar robot, which bases on the a different way of construction of traditional DTRNN model. Through numerical experiments, the feasibility, validity and physical reliability of the ID-DTRNN model for discrete real-time tracking control of cable-driven planar robot are fully verified. In addition, in the real world, physical experiments of the cable-driven planar robot are presented, which successfully promote the development of physical application of ID-DTRNN model, and fill the gap of such model in the industrial engineering field.

Novel Discrete-Time Recurrent Neural Network for Robot Manipulator: A Direct Discretization Technical Route.

Controlling and processing of time-variant problem is universal in the fields of engineering and science, and the discrete-time recurrent neural network (RNN) model has been proven as an effective method for handling a variety of discrete time-variant problems. However, such model usually originates from the discretization research of continuous time-variant problem, and there is little research on the direct discretization method. To address the aforementioned problem, this article introduces a novel discrete-time RNN model for solving the discrete time-variant problem in a pioneering manner. Specifically, a discrete time-variant nonlinear system, which originates from the mathematical modeling of serial robot manipulator, is presented as a target problem. For solving the problem, first, the technique of second-order Taylor expansion is used to deal with the discrete time-variant nonlinear system, and the novel discrete-time RNN model is proposed subsequently. Second, the theoretical analyses are investigated and developed, which shows the convergence and precision of the proposed discrete-time RNN model. Furthermore, three distinct numerical experiments verify the excellent performance of the proposed discrete-time RNN model. In addition, a robot manipulator example further verifies the effectiveness and practicability of the proposed novel discrete-time RNN model.

Frequency domain approach to the critical step size of discrete-time recurrent neural networks

Frequency domain approach to the critical step size of discrete-time recurrent neural networks

Recurrent neural circuits for perceptual grouping

Neurons in the visual cortex are sensitive to context: Responses to stimuli presented within their classical receptive fields (CRFs) are modulated by stimuli in their surrounding extra-classical receptive fields (eCRFs). However, the circuits underlying these contextual effects are not well understood, and little is known about how these circuits drive perception during everyday vision. We tackle these questions by approximating circuit-level eCRF models with a differentiable discrete-time recurrent neural network that is trainable with gradient-descent. After optimizing model synaptic connectivity and dynamics for object contour detection in natural images, the neural-circuit model rivals human observers on the task with far better sample efficiency than state-of-the-art computer vision approaches. Notably, the model also exhibits CRF and eCRF phenomena typically associated with primate vision. The model’s ability to accurately detect object contours also critically depends on these effects, and these contextual effects are not found in ablated versions of the model. Finally, we derive testable predictions about the neural mechanisms responsible for contextual integration and illustrate their importance for accurate and efficient perceptual grouping.

New Criterias of Synchronization for Discrete-Time Recurrent Neural Networks with Time-Varying Delay via Event-Triggered Control

This paper mainly researches the synchronization issue of discrete-time recurrent neural networks (DTRNNs) with time-varying delay based on event-triggered control (ETC). ETC can effectively decrease the quantity of controller updates performed and the utilization of communication resources. By using Lyapunov–Krasovskii functional (LKF), Schur complement lemma, discrete time free weight matrix method, linear matrix inequalities (LMIs) and other analytical methods, the stability conditions of the error system are deduced. Accordingly, a class of event-triggered state feedback controllers is designed. Finally, through two numerical examples with simulations, the effectiveness of the controller is verified.

Stronger separation of analog neuron hierarchy by deterministic context-free languages

The computational power of discrete-time recurrent neural networks (NNs) with the saturated-linear activation function depends on the descriptive complexity of their weight parameters encoding the NN program. In order to study the power of increasing analogicity in NNs between integer (finite automata) and arbitrary rational weights (Turing machines), we have established the analog neuron hierarchy 0ANNs ⊂ 1ANNs ⊂ 2ANNs ⊆ 3ANNs where αANN is a binary-state NN that is extended with α⩾0 extra analog-state neurons with rational weights. In our previous work, we have compared it to the traditional Chomsky hierarchy and separated its first two levels. The separation 1ANNs ⫋ 2ANNs has been witnessed by the non-regular deterministic context-free language (DCFL) L#={0n1n|n⩾1} which cannot be recognized by any 1ANN even with real weights, while any DCFL is accepted by a 2ANN with rational weights. In this paper, we strengthen this separation by showing that any non-regular DCFL (DFCL’) cannot be recognized by 1ANNs with real weights, which means DCFL’s ⊂ (2ANNs \\ 1ANNs), implying 1ANNs ∩ DCFLs = 0ANNs. For this purpose, we show that any 1ANN that would recognize a DFCL’ can be augmented to a larger 1ANN that would recognize L#, which does not exists.

UKF-Based Neural Training for Nonlinear Systems Identification and Control Improvement

Numerous academic works have addressed the identification and control problem for complex dynamic systems. In recent decades, the use of control algorithms based on neural networks (NNs) has been highlighted, which have shown satisfactory results in the trajectory tracking control for a class of discrete-time nonlinear systems. The present work proposes an efficient learning law for discrete-time recurrent high order neural networks (RHONNs), using a training algorithm based on an unscented Kalman filter (UKF). This learning law is applied through a decentralized neural block control using UKF (DNBC-UKF), for trajectory tracking control of a class of discrete-time nonlinear systems. The proposed controller is experimentally evaluated by means of real-time tests in a two degrees of freedom (DOF) vertical direct-drive robotic manipulator against a decentralized neural block control using EKF (DNBC-EKF). The Lyapunov stability theory is used to show that the identification errors of RHONNs are semi-globally uniformly ultimately bounded (SGUUB), the RHONN weights remains bounded, and the closed-loop tracking errors go to zero.

Learning Impulsive Pinning Control of Complex Networks

In this paper, we present an impulsive pinning control algorithm for discrete-time complex networks with different node dynamics, using a linear algebra approach and a neural network as an identifier, to synthesize a learning control law. The model of the complex network used in the analysis has unknown node self-dynamics, linear connections between nodes, where the impulsive dynamics add feedback control input only to the pinned nodes. The proposed controller consists of the linearization for the node dynamics and a reorder of the resulting quadratic Lyapunov function using the Rayleigh quotient. The learning part of the control is done with a discrete-time recurrent high order neural network used for identification of the pinned nodes, which is trained using an extended Kalman filter algorithm. A numerical simulation is included in order to illustrate the behavior of the system under the developed controller. For this simulation, a 20-node complex network with 5 different node dynamics is used. The node dynamics consists of discretized versions of well-known continuous chaotic attractors.

Design and Implementation of a Real Time Control System for a 2DOF Robot Based on Recurrent High Order Neural Network Using a Hardware in the Loop Architecture

In this paper, a real-time implementation of a sliding-mode control (SMC) in a hardware-in-loop architecture is presented for a robot with two degrees of freedom (2DOF). It is based on a discrete-time recurrent neural identification method, as well as the high performance obtained from the advantages of this architecture. The identification process uses a discrete-time recurrent high-order neural network (RHONN) trained with a modified extended Kalman filter (EKF) algorithm. This is a method for calculating the covariance matrices in the EKF algorithm, using a dynamic model with the associated and measurement noises, and it increases the performance of the proposed methodology. On the other hand, the decentralized discrete-time SMC technique is used to minimize the motion error. A Virtex 7 field programmable gate array (FPGA) is configured based on a hardware-in-loop real-time implementation to validate the proposed controller. A series of several experiments demonstrates the robustness of the algorithm, as well as its immunity to noise and the inherent robustness to external perturbation, as are typically found in the input reference signals of a 2DOF manipulator robot.

Novel Discrete-Time Recurrent Neural Networks Handling Discrete-Form Time-Variant Multi-Augmented Sylvester Matrix Problems and Manipulator Application.

In this article, the discrete-form time-variant multi-augmented Sylvester matrix problems, including discrete-form time-variant multi-augmented Sylvester matrix equation (MASME) and discrete-form time-variant multi-augmented Sylvester matrix inequality (MASMI), are formulated first. In order to solve the above-mentioned problems, in continuous time-variant environment, aided with the Kronecker product and vectorization techniques, the multi-augmented Sylvester matrix problems are transformed into simple linear matrix problems, which can be solved by using the proposed discrete-time recurrent neural network (RNN) models. Second, the theoretical analyses and comparisons on the computational performance of the recently developed discretization formulas are presented. Based on these theoretical results, a five-instant discretization formula with superior property is leveraged to establish the corresponding discrete-time RNN (DTRNN) models for solving the discrete-form time-variant MASME and discrete-form time-variant MASMI, respectively. Note that these DTRNN models are zero stable, consistent, and convergent with satisfied precision. Furthermore, illustrative numerical experiments are given to substantiate the excellent performance of the proposed DTRNN models for solving discrete-form time-variant multi-augmented Sylvester matrix problems. In addition, an application of robot manipulator further extends the theoretical research and physical realizability of RNN methods.

Discrete-Time H2 Neural Control Using Reinforcement Learning.

In this article, we discuss H2 control for unknown nonlinear systems in discrete time. A discrete-time recurrent neural network is used to model the nonlinear system, and then, the H2 tracking control is applied based on the neural model. Since this neural H2 control is very sensitive to the neural modeling error, we use reinforcement learning and another neural approximator to improve tracking accuracy and robustness of the controller. The stabilities of the neural identifier and the H2 tracking control are proven. The convergence of the approach is also given. The proposed method is validated with the control of the pan and tilt robot and the surge tank.

Nonfragile H∞ State Estimation for Recurrent Neural Networks With Time-Varying Delays: On Proportional-Integral Observer Design.

In this article, a novel proportional-integral observer (PIO) design approach is proposed for the nonfragile H∞ state estimation problem for a class of discrete-time recurrent neural networks with time-varying delays. The developed PIO is equipped with more design freedom leading to better steady-state accuracy compared with the conventional Luenberger observer. The phenomena of randomly occurring gain variations, which are characterized by the Bernoulli distributed random variables with certain probabilities, are taken into consideration in the implementation of the addressed PIO. Attention is focused on the design of a nonfragile PIO such that the error dynamics of the state estimation is exponentially stable in a mean-square sense, and the prescribed H∞ performance index is also achieved. Sufficient conditions for the existence of the desired PIO are established by virtue of the Lyapunov-Krasovskii functional approach and the matrix inequality technique. Finally, a simulation example is provided to demonstrate the effectiveness of the proposed PIO design scheme.

Discrete-Time Recurrent Neural Network for Solving Bound-Constrained Time-Varying Underdetermined Linear System

A typical recurrent neural network (RNN) has been developed for the online solution of a time-varying underdetermined linear system (TVULS) with bound constraint. This article provides a complete investigation by proposing a new discrete-time RNN (DTRNN) model to solve the bound-constrained TVULS. In particular, the continuous-time RNN (CTRNN) model in the existing literature for solving the bound-constrained TVULS is presented. The new DTRNN model is established and investigated by utilizing the Taylor difference formula to discretize the CTRNN model for determining the solution of the TVULS with bound constraint. Theoretical analysis and numerical results are provided to validate the effectiveness and superiority of the proposed DTRNN model. The model applicability is substantiated through the simulations on a PUMA560 robotic manipulator using the proposed DTRNN model.

Analog neuron hierarchy

In order to refine the analysis of the computational power of discrete-time recurrent neural networks (NNs) between the binary-state NNs which are equivalent to finite automata (level 3 in the Chomsky hierarchy), and the analog-state NNs with rational weights which are Turing-complete (Chomsky level 0), we study an intermediate model αANN of a binary-state NN that is extended with α≥0 extra analog-state neurons. For rational weights, we establish an analog neuron hierarchy 0ANNs ⊂ 1ANNs ⊂ 2ANNs ⊆ 3ANNs and separate its first two levels. In particular, 0ANNs coincide with the binary-state NNs (Chomsky level 3) being a proper subset of 1ANNs which accept at most context-sensitive languages (Chomsky level 1) including some non-context-free ones (above Chomsky level 2). We prove that the deterministic (context-free) language L#={0n1n∣n≥1} cannot be recognized by any 1ANN even with real weights. In contrast, we show that deterministic pushdown automata accepting deterministic languages can be simulated by 2ANNs with rational weights, which thus constitute a proper superset of 1ANNs. Finally, we prove that the analog neuron hierarchy collapses to 3ANNs by showing that any Turing machine can be simulated by a 3ANN having rational weights, with linear-time overhead.

Network dilution and asymmetry in an efficient brain

ABSTRACT The ultimate goal of neuroscience is to ultimately understand how the brain functions. The advancement of brain imaging shows us how the brain continuously alternates complex activity patterns and experimentally reveals how these patterns are responsible for memory, association, reasoning, and countless other tasks. Two fundamental parameters, dilution (the number of connections per node), and symmetry (the number of bidirectional connections of the same weight) characterise two fundamental features underlying the networks that connect the single neurons in the brain and generate these patterns. Mammalian brains show large variations of dilution, and mostly asymmetric connectivity, unfortunately the advantages which drove evolution to these state of network dilution and asymmetry are still unknown. Here, we studied the effects of symmetry and dilution on a discrete-time recurrent neural network with McCulloch–Pitts neurons. We use an exhaustive approach, in which we probe all possible inputs for several randomly connected neuron networks with different degrees of dilution and symmetry. We find an optimum value for the synaptic dilution and symmetry, which turns out to be in striking quantitative agreement with what previous researchers have found in the brain cortex, neocortex and hippocampus. The diluted asymmetric brain shows high memory capacity and pattern recognition speed, but most of all it is the less energy-consumptive with respect to fully connected and symmetric network topologies.

An Autonomous Path Controller in a System on Chip for Shrimp Robot

This paper presents a path planning and trajectory tracking system for a BlueBotics Shrimp III®, which is an articulate mobile robot for rough terrain navigation. The system includes a decentralized neural inverse optimal controller, an inverse kinematic model, and a path-planning algorithm. The motor control is obtained based on a discrete-time recurrent high order neural network trained with an extended Kalman filter, and an inverse optimal controller designed without solving the Hamilton Jacobi Bellman equation. To operate the whole system in a real-time application, a Xilinx Zynq® System on Chip (SoC) is used. This implementation allows for a good performance and fast calculations in real-time, in a way that the robot can explore and navigate autonomously in unstructured environments. Therefore, this paper presents the design and implementation of a real-time system for robot navigation that integrates, in a Xilinx Zynq® System on Chip, algorithms of neural control, image processing, path planning, and inverse kinematics and trajectory tracking.