Recurrent Network Research Articles

Eye-LRCN: A Long-Term Recurrent Convolutional Network for Eye Blink Completeness Detection.

Computer vision syndrome causes vision problems and discomfort mainly due to dry eye. Several studies show that dry eye in computer users is caused by a reduction in the blink rate and an increase in the prevalence of incomplete blinks. In this context, this article introduces Eye-LRCN, a new eye blink detection method that also evaluates the completeness of the blink. The method is based on a long-term recurrent convolutional network (LRCN), which combines a convolutional neural network (CNN) for feature extraction with a bidirectional recurrent neural network that performs sequence learning and classifies the blinks. A Siamese architecture is used during CNN training to overcome the high-class imbalance present in blink detection and the limited amount of data available to train blink detection models. The method was evaluated on three different tasks: blink detection, blink completeness detection, and eye state detection. We report superior performance to the state-of-the-art methods in blink detection and blink completeness detection, and remarkable results in eye state detection.

Predictive coding networks for temporal prediction.

One of the key problems the brain faces is inferring the state of the world from a sequence of dynamically changing stimuli, and it is not yet clear how the sensory system achieves this task. A well-established computational framework for describing perceptual processes in the brain is provided by the theory of predictive coding. Although the original proposals of predictive coding have discussed temporal prediction, later work developing this theory mostly focused on static stimuli, and key questions on neural implementation and computational properties of temporal predictive coding networks remain open. Here, we address these questions and present a formulation of the temporal predictive coding model that can be naturally implemented in recurrent networks, in which activity dynamics rely only on local inputs to the neurons, and learning only utilises local Hebbian plasticity. Additionally, we show that temporal predictive coding networks can approximate the performance of the Kalman filter in predicting behaviour of linear systems, and behave as a variant of a Kalman filter which does not track its own subjective posterior variance. Importantly, temporal predictive coding networks can achieve similar accuracy as the Kalman filter without performing complex mathematical operations, but just employing simple computations that can be implemented by biological networks. Moreover, when trained with natural dynamic inputs, we found that temporal predictive coding can produce Gabor-like, motion-sensitive receptive fields resembling those observed in real neurons in visual areas. In addition, we demonstrate how the model can be effectively generalized to nonlinear systems. Overall, models presented in this paper show how biologically plausible circuits can predict future stimuli and may guide research on understanding specific neural circuits in brain areas involved in temporal prediction.

Design and Experimental Validation of Deep Reinforcement Learning-Based Fast Trajectory Planning and Control for Mobile Robot in Unknown Environment.

This article is concerned with the problem of planning optimal maneuver trajectories and guiding the mobile robot toward target positions in uncertain environments for exploration purposes. A hierarchical deep learning-based control framework is proposed which consists of an upper level motion planning layer and a lower level waypoint tracking layer. In the motion planning phase, a recurrent deep neural network (RDNN)-based algorithm is adopted to predict the optimal maneuver profiles for the mobile robot. This approach is built upon a recently proposed idea of using deep neural networks (DNNs) to approximate the optimal motion trajectories, which has been validated that a fast approximation performance can be achieved. To further enhance the network prediction performance, a recurrent network model capable of fully exploiting the inherent relationship between preoptimized system state and control pairs is advocated. In the lower level, a deep reinforcement learning (DRL)-based collision-free control algorithm is established to achieve the waypoint tracking task in an uncertain environment (e.g., the existence of unexpected obstacles). Since this approach allows the control policy to directly learn from human demonstration data, the time required by the training process can be significantly reduced. Moreover, a noisy prioritized experience replay (PER) algorithm is proposed to improve the exploring rate of control policy. The effectiveness of applying the proposed deep learning-based control is validated by executing a number of simulation and experimental case studies. The simulation result shows that the proposed DRL method outperforms the vanilla PER algorithm in terms of training speed. Experimental videos are also uploaded, and the corresponding results confirm that the proposed strategy is able to fulfill the autonomous exploration mission with improved motion planning performance, enhanced collision avoidance ability, and less training time.

Sign Language Word Detection Using LRCN

Sign language is the most effective communication for deaf or hard-of-hearing people. Specialized training is required to understand sign language, and as such, people without disabilities around them cannot communicate effectively. The main objective of this study is to develop a mechanism for streamlining the deep learning model for sign language recognition by utilizing the 30 most prevalent words in our everyday lives. The dataset was designed through 30 ASL (American Sign Language) words consisting of custom-processed video sequences, which consist of 5 subjects and 50 sample videos for each class. The CNN model can be applied to video frames to extract spatial properties. Using CNN’s acquired data, the LSTM model may then predict the action being performed in the video. We present and evaluate the results of two separate datasets—the Pose dataset and the Raw video dataset. The dataset was trained with the Long-term Recurrent Convolutional Network (LRCN) approach. Finally, a test accuracy of 92.66% was reached for the raw dataset, while 93.66% for the pose dataset.

ЗАСТОСУВАННЯ ГЛИБОКИХ ШТУЧНИХ НЕЙРОННИХ МЕРЕЖ ДЛЯ КЛАСИФІКАЦІЇ МУЛЬТИМОДАЛЬНИХ ДАНИХ

Multimodal data analysis is gaining attention in recent research. Pu Liang et al. (2023) provide a comprehensive overview on multimodal machine learning, highlighting its founda-tions, challenges and achievements in recent years. More problem-oriented works propose new methods and applications for multimodal ML, such a Ngiam et al. (2011) propose to use joint audio and video data to improve speech recognition accuracy; Sun, Wand and Li (2018) describe application of multimodal classification for breast cancer prognosis prediction; Mao et al. (2014) propose an architecture of multimodal recurrent network to generate text de-scription of images and so on. However, such works usually focus on the task itself and meth-ods therein, and not on integrating multimodal data processing into other software systems. The goal of this research is to propose a way to conduct multimodal data processing, specifically as a part of a digital twin systems, thus efficiency and near-real-time operation are required. The paper presents an approach to conduct parallel multimodal data classification, adapting to available computing power. The method is modular and scalable and intended for in digital twin application as a part of analysis and modeling tools. Later, the detailed example of such a software module is discussed. It uses multimodal data from open datasets to detect and classify the behavior of pets using deep learning mod-els. Videos are processed using two artificial neural networks: YOLOv3 object detection net-work to process individual frames of the video and a relatively simple convolutional network to classify sounds based on their frequency spectra. Constructed module uses a producer-consumer parallel processing pattern and allows processing 5 frames per second of a video on available hardware, which can be sufficiently improved by using GPU acceleration or more paralleled processing threads.

Messages are Never Propagated Alone: Collaborative Hypergraph Neural Network for Time-Series Forecasting.

This paper delves into the problem of correlated time-series forecasting in practical applications, an area of growing interest in a multitude of fields such as stock price prediction and traffic demand analysis. Current methodologies primarily represent data using conventional graph structures, yet these fail to capture intricate structures with non-pairwise relationships. To address this challenge, we adopt dynamic hypergraphs in this study to better illustrate complex interactions, and introduce a novel hypergraph neural network model named CHNN for correlated time series forecasting. In more detail, CHNN leverages both semantic and topological similarities via an interaction model and hypergraph diffusion process, thereby constructing comprehensive collaborative correlation scores that effectively guide spatial message propagation. In addition, it incorporates short-term temporal information to generate efficient spatio-temporal feature maps. Lastly, a long-term temporal module is proposed to generate future predictions utilizing both temporal attention and a gated recurrent network. Comprehensive experiments conducted on four real-world datasets, i.e., Tiingo, Stocktwits, NYC-Taxi, and Social Network demonstrate that the proposed CHNN markedly outperforms a range of benchmark methods.

Brain Tumor Detection Using Classification Model

This study aims to develop automated methods for detecting and segmenting brain tumors in MRI scans using neural networks and conventional image processing. Accurately identifying tumor location and boundaries is crucial for diagnosis and determining the most effective treatment approach. Manual review by radiologists is time-consuming and subjective. However, recent advances in deep learning now enable more efficient analysis of large MRI datasets.
 A variety of neural network models have been explored for tumor segmentation, including recurrent networks, autoencoders and convolutional networks. RNNs are well-suited to capture contextual information across sequential MRI slices. Autoencoders perform unsupervised feature learning from the images. CNNs have shown promise due to their ability to directly learn visual patterns from pixel values. Unsupervised clustering techniques such as fuzzy c-means and k-means are also investigated to group tumor pixels based on intensity similarity without labeled data.
 Previous research combining these techniques has achieved high segmentation accuracy between 84-97% using additional image processing steps. Automated segmentation provides benefits over manual inspection such as reduced workload and more consistent delineation of tumor margins. It can precisely outline growths to aid radiologists in diagnosis and pathologists in assessing malignancy.
 The detection and segmentation of brain tumors has applications in both radiology and oncology. Radiologists could benefit from faster scan review and automated report generation. Neurosurgeons would gain insights into a tumor's size, location and infiltration to determine the most suitable surgical approach. Oncologists could utilize quantitative tumor features to select the most targeted radiation therapy or chemotherapy protocols. Overall, continued advances in deep and machine learning hold promise to improve diagnosis and management of brain cancer patients.

Universal approximation property of stochastic configuration networks for time series

For the purpose of processing sequential data, such as time series, and addressing the challenge of manually tuning the architecture of traditional recurrent neural networks (RNNs), this paper introduces a novel approach-the Recurrent Stochastic Configuration Network (RSCN). This network is constructed based on the random incremental algorithm of stochastic configuration networks. Leveraging the foundational structure of recurrent neural networks, our learning model commences with a modest-scale recurrent neural network featuring a single hidden layer and a solitary hidden node. Subsequently, the node parameters of the hidden layer undergo incremental augmentation through a random configuration process, with corresponding weights assigned structurally. This iterative expansion continues until the network satisfies predefined termination criteria. Noteworthy is the adaptability of this algorithm to handle time series data, exhibiting superior performance compared to traditional recurrent neural networks with similar architectures. The experimental results presented in this paper underscore the efficacy of the proposed RSCN for sequence data processing, showcasing its advantages over conventional recurrent neural networks in the context of the performed experiments.

Learning fused lasso parameters in portfolio selection via neural networks

In recent years, fused lasso models are becoming popular in several fields, such as computer vision, classification and finance. In portfolio selection, they can be used to penalize active positions and portfolio turnover. Despite efficient algorithms and software for solving non-smooth optimization problems have been developed, the amount of regularization to apply is a critical issue, especially if we have to achieve a financial aim. We propose a data-driven approach for learning the regularization parameters in a fused lasso formulation of the multi-period portfolio selection problem, able to realize a given financial target. We design a neural network architecture based on recurrent networks for learning the functional dependence between the regularization parameters and the input data. In particular, the Long Short-Term Memory networks are considered for their ability to process sequential data, such as the time series of the asset returns. Numerical experiments performed on market data show the effectiveness of our approach.

On the Expressivity of Recurrent Neural Cascades

Recurrent Neural Cascades (RNCs) are the recurrent neural networks with no cyclic dependencies among recurrent neurons. This class of recurrent networks has received a lot of attention in practice. Besides training methods for a fixed architecture such as backpropagation, the cascade architecture naturally allows for constructive learning methods, where recurrent nodes are added incrementally one at a time, often yielding smaller networks. Furthermore, acyclicity amounts to a structural prior that even for the same number of neurons yields a more favourable sample complexity compared to a fully-connected architecture. A central question is whether the advantages of the cascade architecture come at the cost of a reduced expressivity. We provide new insights into this question. We show that the regular languages captured by RNCs with sign and tanh activation with positive recurrent weights are the star-free regular languages. In order to establish our results we developed a novel framework where capabilities of RNCs are assessed by analysing which semigroups and groups a single neuron is able to implement. A notable implication of our framework is that RNCs can achieve the expressivity of all regular languages by introducing neurons that can implement groups.

Context-Aware Iteration Policy Network for Efficient Optical Flow Estimation

Existing recurrent optical flow estimation networks are computationally expensive since they use a fixed large number of iterations to update the flow field for each sample. An efficient network should skip iterations when the flow improvement is limited. In this paper, we develop a Context-Aware Iteration Policy Network for efficient optical flow estimation, which determines the optimal number of iterations per sample. The policy network achieves this by learning contextual information to realize whether flow improvement is bottlenecked or minimal. On the one hand, we use iteration embedding and historical hidden cell, which include previous iterations information, to convey how flow has changed from previous iterations. On the other hand, we use the incremental loss to make the policy network implicitly perceive the magnitude of optical flow improvement in the subsequent iteration. Furthermore, the computational complexity in our dynamic network is controllable, allowing us to satisfy various resource preferences with a single trained model. Our policy network can be easily integrated into state-of-the-art optical flow networks. Extensive experiments show that our method maintains performance while reducing FLOPs by about 40%/20% for the Sintel/KITTI datasets.

Decoupling Degradations with Recurrent Network for Video Restoration in Under-Display Camera

Under-display camera (UDC) systems are the foundation of full-screen display devices in which the lens mounts under the display. The pixel array of light-emitting diodes used for display diffracts and attenuates incident light, causing various degradations as the light intensity changes. Unlike general video restoration which recovers video by treating different degradation factors equally, video restoration for UDC systems is more challenging that concerns removing diverse degradation over time while preserving temporal consistency. In this paper, we introduce a novel video restoration network, called D2RNet, specifically designed for UDC systems. It employs a set of Decoupling Attention Modules (DAM) that effectively separate the various video degradation factors. More specifically, a soft mask generation function is proposed to formulate each frame into flare and haze based on the diffraction arising from incident light of different intensities, followed by the proposed flare and haze removal components that leverage long- and short-term feature learning to handle the respective degradations. Such a design offers an targeted and effective solution to eliminating various types of degradation in UDC systems. We further extend our design into multi-scale to overcome the scale-changing of degradation that often occur in long-range videos. To demonstrate the superiority of D2RNet, we propose a large-scale UDC video benchmark by gathering HDR videos and generating realistically degraded videos using the point spread function measured by a commercial UDC system. Extensive quantitative and qualitative evaluations demonstrate the superiority of D2RNet compared to other state-of-the-art video restoration and UDC image restoration methods.

Can neural networks benefit from objectives that encourage iterative convergent computations? A case study of ResNets and object classification.

Recent work has suggested that feedforward residual neural networks (ResNets) approximate iterative recurrent computations. Iterative computations are useful in many domains, so they might provide good solutions for neural networks to learn. However, principled methods for measuring and manipulating iterative convergence in neural networks remain lacking. Here we address this gap by 1) quantifying the degree to which ResNets learn iterative solutions and 2) introducing a regularization approach that encourages the learning of iterative solutions. Iterative methods are characterized by two properties: iteration and convergence. To quantify these properties, we define three indices of iterative convergence. Consistent with previous work, we show that, even though ResNets can express iterative solutions, they do not learn them when trained conventionally on computer-vision tasks. We then introduce regularizations to encourage iterative convergent computation and test whether this provides a useful inductive bias. To make the networks more iterative, we manipulate the degree of weight sharing across layers using soft gradient coupling. This new method provides a form of recurrence regularization and can interpolate smoothly between an ordinary ResNet and a "recurrent" ResNet (i.e., one that uses identical weights across layers and thus could be physically implemented with a recurrent network computing the successive stages iteratively across time). To make the networks more convergent we impose a Lipschitz constraint on the residual functions using spectral normalization. The three indices of iterative convergence reveal that the gradient coupling and the Lipschitz constraint succeed at making the networks iterative and convergent, respectively. To showcase the practicality of our approach, we study how iterative convergence impacts generalization on standard visual recognition tasks (MNIST, CIFAR-10, CIFAR-100) or challenging recognition tasks with partial occlusions (Digitclutter). We find that iterative convergent computation, in these tasks, does not provide a useful inductive bias for ResNets. Importantly, our approach may be useful for investigating other network architectures and tasks as well and we hope that our study provides a useful starting point for investigating the broader question of whether iterative convergence can help neural networks in their generalization.

An Adaptive Multimodal Data Vessel Trajectory Prediction Model Based on a Satellite Automatic Identification System and Environmental Data

Ship trajectory prediction is essential for ensuring safe route planning and to have advanced warning of the dangers at sea. With the development of deep learning, most of the current research has explored advanced prediction methods based on historical spatio-temporal Automatic Identification System (AIS) data. However, environmental factors such as sea wind and visibility also affect ship navigation in real-world maritime shipping. Therefore, developing reliable models utilizing multimodal data, such as AIS and environmental data, is challenging. In this research, we design an adaptive multimodal vessel trajectory data prediction model (termed AMD) based on satellite AIS and environmental data. The AMD model mainly consists of an AIS-based extraction network, an environmental-based extraction network, and a fusion block. In particular, this work considers multimodal data such as historical spatio-temporal information and environmental factors. Time stamps and distances are correlated with AIS and environmental data, and a multilayer perceptron and gated recurrent unit networks are used to design multimodal feature extraction networks. Finally, the fusion block realizes the fusion output of multimodal features to improve the reliability of the AMD model. Several quantitative and qualitative experiments are conducted using real-world AIS and multimodal environmental datasets. Numerous experimental results prove that prediction performance using multimodal data can ensure satisfactory accuracy and reliability while exhibiting a positive impact on improving maritime transport services.

Deep reinforcement learning approach with hybrid action space for mobile charging in wireless rechargeable sensor networks

Mobile charging is feasible to deal with the energy-constrained problem in wireless rechargeable sensor networks (WRSNs). The mobile chargers (MCs) are usually employed to charge the sensors sequentially according to the charging schemes. Existing studies assume that each sensor should be charged to its maximum energy capacity or to a fixed upper threshold before the next one can be charged. However, they neglect to control the charging time adaptively of each sensor according to the charging demand. Hence, in this paper, we assume that the charging time of each sensor can be controlled, and we study the joint optimization of charging sequence and charging time problem (JCSCT). Correspondingly, we propose a novel deep reinforcement learning with hybrid action space approach for JCSCT (DRLH-JCSCT), which utilizes deep q-network (DQN) to generate the charging sequence, and adopts deep deterministic policy gradient (DDPG) to determine the charging time. An attention-based encoder–decoder model is integrated in the actor network of DDPG, and a modified bi-directional gate recurrent unit network (MBGRU) is utilized as the decoder. We also design a novel reward function to evaluate the quality of the charging actions. Simulations demonstrate the improved charging performance of the proposed approach, with a longer network lifetime and fewer failed sensors compared with the existing mobile charging scheduling approaches.

Co-dependent excitatory and inhibitory plasticity accounts for quick, stable and long-lasting memories in biological networks

The brain’s functionality is developed and maintained through synaptic plasticity. As synapses undergo plasticity, they also affect each other. The nature of such ‘co-dependency’ is difficult to disentangle experimentally, because multiple synapses must be monitored simultaneously. To help understand the experimentally observed phenomena, we introduce a framework that formalizes synaptic co-dependency between different connection types. The resulting model explains how inhibition can gate excitatory plasticity while neighboring excitatory–excitatory interactions determine the strength of long-term potentiation. Furthermore, we show how the interplay between excitatory and inhibitory synapses can account for the quick rise and long-term stability of a variety of synaptic weight profiles, such as orientation tuning and dendritic clustering of co-active synapses. In recurrent neuronal networks, co-dependent plasticity produces rich and stable motor cortex-like dynamics with high input sensitivity. Our results suggest an essential role for the neighborly synaptic interaction during learning, connecting micro-level physiology with network-wide phenomena.

R-GCN: a residual-gated recurrent unit convolution network model for anomaly detection in blockchain transactions

R-GCN: a residual-gated recurrent unit convolution network model for anomaly detection in blockchain transactions

Short‐term power forecasting method for 5G photovoltaic base stations on non‐sunny days based on SDN‐integrated INGO‐BP and RGAN

AbstractIn response to the suboptimal efficiency observed in the network configuration and administration of 5G photovoltaic base stations (PVBSs), as well as the inherent limitations in accurately forecasting photovoltaic power during inclement weather conditions, this research article introduces a concise and effective method for short‐term power prediction of PVBSs when subjected to non‐sunny weather conditions, leveraging the paradigm of software defined networking. This method ingeniously amalgamates the improved Northern Goshawk optimized back propagation neural network (INGO‐BP) and recurrent generative adversarial networks (RGAN). The investigation commences by delineating a comprehensive PVBS system framework underpinned by SDN principles. Subsequently, it employs the Pearson correlation coefficient technique and principal component analysis for the judicious selection of pertinent features from the PVBS photovoltaic dataset, thereby ameliorating the issue of multicollinearity within the predictive model. To culminate the research endeavour, the INGO algorithm is enlisted to optimize the weightings and bias parameters of the BP neural network, while RGAN is harnessed for the acquisition of distinctive distribution patterns in photovoltaic data across diverse meteorological conditions. This augmentation enhances the model's capacity to discern the intricate long‐term mappings between photovoltaic power and meteorological data. The empirical findings substantiate the elevated accuracy of the INGO‐BP prediction model across sunny, cloudy, and rainy weather conditions, with a discernible enhancement in predictive accuracy for PVBS performance under non‐sunny weather conditions facilitated by data augmentation via RGAN.

Comparative Analysis of Linear Models and Artificial Neural Networks for Sugar Price Prediction

Sugar is an important commodity that is used beyond the food industry. It can be produced from sugarcane and sugar beet, depending on the region. Prices worldwide differ due to high volatility, making it difficult to estimate their forecast. Thus, the present work aims to predict the prices of kilograms of sugar from four databases: the European Union, the United States, Brazil, and the world. To achieve this, linear methods from the Box and Jenkins family were employed, together with classic and new approaches of artificial neural networks: the feedforward Multilayer Perceptron and extreme learning machines, and the recurrent proposals Elman Network, Jordan Network, and Echo State Networks considering two reservoir designs. As performance metrics, the MAE and MSE were addressed. The results indicated that the neural models were more accurate than linear ones. In addition, the MLP and the Elman networks stood out as the winners.

A Deep Learning Approach to Merge Rule-Based and Human-Operated Camera Control for Teleoperated Robotic Systems

Controlling a laparoscopic camera during robotic surgery represents a multifaceted challenge, demanding considerable physical and cognitive exertion from operators. While manual control presents the advantage of enabling optimal viewing angles, it is offset by its taxing nature. In contrast, current autonomous camera systems offer predictability in tool tracking but are often rigid, lacking the adaptability of human operators. This research investigates the potential of two distinct network architectures: a dense neural network (DNN) and a recurrent network (RNN), both trained using a diverse dataset comprising autonomous and human-driven camera movements. A comparative assessment of network-controlled, autonomous, and human-operated camera systems is conducted to gauge network efficacies. While the dense neural network exhibits proficiency in basic tool tracking, it grapples with inherent architectural limitations that hinder its ability to master the camera’s zoom functionality. In stark contrast, the recurrent network excels, demonstrating a capacity to sufficiently replicate the behaviors exhibited by a mixture of both autonomous and human-operated methods. In total, 96.8% of the dense network predictions had up to a one-centimeter error when compared to the test datasets, while the recurrent network achieved a 100% sub-millimeter testing error. This paper trains and evaluates neural networks on autonomous and human behavior data for camera control.