Simultaneous Recurrent Network Research Articles

Analysis of Learning Heuristic Estimates for Grid Planning with Cellular Simultaneous Recurrent Networks

Analysis of Learning Heuristic Estimates for Grid Planning with Cellular Simultaneous Recurrent Networks

Novel deep generative simultaneous recurrent model for efficient representation learning

Representation learning plays an important role for building effective deep neural network models. Deep generative probabilistic models have shown to be efficient in the data representation learning task which is usually carried out in an unsupervised fashion. Throughout the past decade, there has been almost exclusive focus on the learning algorithms to improve representation capability of the generative models. However, effective data representation requires improvement in both learning algorithm and architecture of the generative models. Therefore, improvement to the neural architecture is critical for improved data representation capability of deep generative models. Furthermore, the prevailing class of deep generative models such as deep belief network (DBN), deep Boltzman machine (DBM) and deep sigmoid belief network (DSBN) are inherently unidirectional and lack recurrent connections ubiquitous in the biological neuronal structures. Introduction of recurrent connections may offer further improvement in data representation learning performance to the deep generative models. Consequently, for the first time in literature, this work proposes a deep recurrent generative model known as deep simultaneous recurrent belief network (D-SRBN) to efficiently learn representations from unlabeled data. Experimentation on four benchmark datasets: MNIST, Caltech 101 Silhouettes, OCR letters and Omniglot show that the proposed D-SRBN model achieves superior representation learning performance while utilizing less computing resources when compared to the four state-of-the-art generative models such as deep belief network (DBN), DBM, DSBN and VAE (variational auto-encoder).

Sparse Simultaneous Recurrent Deep Learning for Robust Facial Expression Recognition.

Facial expression recognition is a challenging task that involves detection and interpretation of complex and subtle changes in facial muscles. Recent advances in feed-forward deep neural networks (DNNs) have offered improved object recognition performance. Sparse feature learning in feed-forward DNN models offers further improvement in performance when compared to the earlier handcrafted techniques. However, the depth of the feed-forward DNNs and the computational complexity of the models increase proportional to the challenges posed by the facial expression recognition problem. The feed-forward DNN architectures do not exploit another important learning paradigm, known as recurrency, which is ubiquitous in the human visual system. Consequently, this paper proposes a novel biologically relevant sparse-deep simultaneous recurrent network (S-DSRN) for robust facial expression recognition. The feature sparsity is obtained by adopting dropout learning in the proposed DSRN as opposed to usual handcrafting of additional penalty terms for the sparse representation of data. Theoretical analysis of S-DSRN shows that the dropout learning offers desirable properties such as sparsity, and prevents the model from overfitting. Experimental results also suggest that the proposed method yields better performance accuracy, requires reduced number of parameters, and offers reduced computational complexity than that of the previously reported state-of-the-art feed-forward DNNs using two of the most widely used publicly available facial expression data sets. Furthermore, we show that by combining the proposed neural architecture with a state-of-the-art metric learning technique significantly improves the overall recognition performance. Finally, a graphical processing unit (GPU)-based implementation of S-DSRN is obtained for real-time applications.

A COMPARATIVE STUDY OF VARIOUS METHODS OF ANN FOR SOLVING TSP PROBLEM

Abstract This paper represents TSP (Travelling Salesman Problem) by using Artificial Neural Networks.A comparative study of various methods of ANN is shown here for solving TSP problem.The Travelling Salesman Problem is a classical combinational optimization problem, which is a simple to state but very difficult to solve. This problem is to find the shortest possible tour through a set of N vertices so that each vertex is visited exactly once. TSP can be solved by Hopfield Network, Self-organization Map, and Simultaneous Recurrent Network. Hopfield net is a fully connected network, where every vertex is connected with each other forwardly and backwardly. So starting the walk from a vertex we can travel all the other vertex exactly once and return to starting vertex again.

Large-scale pose-invariant face recognition using cellular simultaneous recurrent network

In this work, we propose a novel technique for face recognition with +/-90 degrees pose variations in image sequences using a cellular simultaneous recurrent network (CSRN). We formulate the recognition problem with such large-pose variations as an implicit temporal prediction task for CSRN. We exploit a face extraction algorithm based on the scale-space method and facial structural knowledge as a preprocessing step. Further, to reduce computational cost, we obtain eigenfaces for a set of image sequences for each person and use these reduced pattern vectors as the input to CSRN. CSRN learns how to associate each face class/person in the training phase. A modified distance metric between successive frames of test and training output pattern vectors indicate either a match or mismatch between the two corresponding face classes. We extensively evaluate our CSRN-based face recognition technique using the publicly available VidTIMIT Audio-Video face dataset. Our simulation shows that for this dataset with large-scale pose variations, we can obtain an overall 77% face recognition rate.

Quantum inspired PSO for the optimization of simultaneous recurrent neural networks as MIMO learning systems

Training a single simultaneous recurrent neural network (SRN) to learn all outputs of a multiple-input-multiple-output (MIMO) system is a difficult problem. A new training algorithm developed from combined concepts of swarm intelligence and quantum principles is presented. The training algorithm is called particle swarm optimization with quantum infusion (PSO-QI). To improve the effectiveness of learning, a two-step learning approach is introduced in the training. The objective of the learning in the first step is to find the optimal set of weights in the SRN considering all output errors. In the second step, the objective is to maximize the learning of each output dynamics by fine tuning the respective SRN output weights. To demonstrate the effectiveness of the PSO-QI training algorithm and the two-step learning approach, two examples of an SRN learning MIMO systems are presented. The first example is learning a benchmark MIMO system and the second one is the design of a wide area monitoring system for a multimachine power system. From the results, it is observed that SRNs can effectively learn MIMO systems when trained using the PSO-QI algorithm and the two-step learning approach.

Training Winner-Take-All Simultaneous Recurrent Neural Networks

The winner-take-all (WTA) network is useful in database management, very large scale integration (VLSI) design, and digital processing. The synthesis procedure of WTA on single-layer fully connected architecture with sigmoid transfer function is still not fully explored. We discuss the use of simultaneous recurrent networks (SRNs) trained by Kalman filter algorithms for the task of finding the maximum among N numbers. The simulation demonstrates the effectiveness of our training approach under conditions of a shared-weight SRN architecture. A more general SRN also succeeds in solving a real classification application on car engine data.

同時リカレントネットワークによる不連続な非線形関数の統計的近似学習法

同時リカレントネットワークによる不連続な非線形関数の統計的近似学習法

A STATISTICAL APPROXIMATION LEARNING METHOD FOR SIMULTANEOUS RECURRENT NETWORKS

In this paper, a statistical approximation learning (SAL) method is proposed for a new type of neural networks, simultaneous recurrent networks (SRNs). The SRNs have the capability to approximate non-smooth functions which cannot be approximated by using conventional multi-layer perceptrons (MLPs). However, the most of the learning methods for the SRNs are computationally expensive due to their inherent recursive calculations. To solve this problem, a novel approximation learning method is proposed by using a statistical relation between the time-series of the network outputs and the network configuration parameters. Simulation results show that the proposed method can learn a strongly nonlinear function efficiently.

Hierarchical Neural Network Modeling for Infrared Spectra Interpretation of Modified Starches

The goal of this research was to demonstrate a new approach to multilevel hierarchical modeling using neural networks. In a situation where the data set is small and error-prone, it is necessary to build multiple models of input−output relationships, combining these models into a hierarchical structure. This allows utilization of the best aspects of every model, eliminating the need to choose “the best one”, when the general sample itself is not known exactly. At the first stage of modeling we used feed-forward artificial neural networks for spectra interpretation. Then at the second stage a simultaneous recurrent network was used to correct the predictions made at the first stage. This architecture enables enhancement of the generalization ability according to the similarity of the input patterns. The suggested method was applied to the interpretation of spectra of modified starches. This has definite practical value, since authentication of food is very important for the consumers and the food industry ...

IMPLEMENTING BACK-PROPAGATION- THROUGH-TIME LEARNING ALGORITHM USING CELLULAR NEURAL NETWORKS

In a programmable (multistage) cellular neural network (CNN) structure, the CPU is a CNN universal chip which supports massively parallel computations on patterns and images, including videos. In this paper, we decompose the structure of a class of simultaneous recurrent networks (SRN) into a CNN program and run it on a von Neumann-like stored program CNN structure. To train the SRN, we map the back-propagation-through-time (BTT) learning algorithm into a sequence of CNN subroutines to achieve real-time performance via a CNN universal chip. By computing in parallel, the CNN universal chip can be programmed to implement in real time the BTT learning algorithm, which has a very high time complexity. An estimate of the time complexity of the BTT learning algorithm based on the CNN universal chip is presented. For small-scale problems, our simulation results show that a CNN implementation of the BTT learning algorithm for a two-dimensional SRN is at least 10,000 times faster than that based on state-of-the-art sequential workstations. For the few large-scale problems which we have so far simulated, the CNN implemented BTT learning algorithm maintained virtually the same time complexity with a learning time of a few seconds, while those implemented on state-of-the-art sequential workstations dramatically increased their time complexity, often requiring several days of running time. Several examples are presented to demonstrate how efficiently a CNN universal chip can speed up the learning algorithm for both off-line and on-line applications.