Convolutional Recurrent Network Research Articles

Application of Recurrent Graph Convolutional Networks to the Neural State Estimation Problem

Neural State Estimation (NSE) is a novel application of deep learning which is concerned with interpolating the state of a distribution power grid from a limited amount of sensor data and can be represented as a non-linear graph time-series nowcasting problem. Although several authors have proposed their solutions for NSE, there is neither a comparison of approaches nor an industry-standard state of the art model yet. The main purpose of this paper is to compare these solutions to a promising new approach: recurrent graph convolutional neural networks. There are theoretical reasons to assume that this class of models is suited for solving NSE. Our experiments verify that they achieve similar performance while also presenting many unique advantages compared to the previously proposed models.

MHAR: a novel convolutional recurrent model for recognizing motion-based human activity

With sensors built into smartphones and wearable devices, Human Activity Recognition (HAR) makes it possible to identify regular activities. One of the challenges is to access the motion information from activity data and extract the features from this information. A general model trained without motion features may perform poorly while recognizing the contextual patterns in human activities. In this paper, we proposed the mHAR (motion HAR) model for recognizing static, gradual, and dynamic motion activity to address this challenge. We examine the impact of the proposed model against the general non-convolutional and convolutional recurrent networks in terms of accuracy rates and F1 score. As an experimental dataset, the publicly accessible WISDM was employed. Additionally, the findings of the experiments demonstrate that, in contrast to the unbalanced class problem, motion features play a significant role in activity recognition by achieving accuracy rates of 92.29% and an average F1 score of 83.33%.

Speech Enhancement via Mask-Mapping Based Residual Dense Network

Masking-based and spectrum mapping-based methods are the two main algorithms of speech enhancement with deep neural network (DNN). But the mapping-based methods only utilizes the phase of noisy speech, which limits the upper bound of speech enhancement performance. Masking-based methods need to accurately estimate the masking which is still the key problem. Combining the advantages of above two types of methods, this paper proposes the speech enhancement algorithm MM-RDN (masking-mapping residual dense network) based on masking-mapping (MM) and residual dense network (RDN). Using the logarithmic power spectrogram (LPS) of consecutive frames, MM estimates the ideal ratio masking (IRM) matrix of consecutive frames. RDN can make full use of feature maps of all layers. Meanwhile, using the global residual learning to combine the shallow features and deep features, RDN obtains the global dense features from the LPS, thereby improves estimated accuracy of the IRM matrix. Simulations show that the proposed method achieves attractive speech enhancement performance in various acoustic environments. Specifically, in the untrained acoustic test with limited priors, e.g., unmatched signal-to-noise ratio (SNR) and unmatched noise category, MM-RDN can still outperform the existing convolutional recurrent network (CRN) method in the measures of perceptual evaluation of speech quality (PESQ) and other evaluation indexes. It indicates that the proposed algorithm is more generalized in untrained conditions.

Word-level dual channel with multi-head semantic attention interaction for community question answering

<abstract><p>The semantic matching problem detects whether the candidate text is related to a specific input text. Basic text matching adopts the method of statistical vocabulary information without considering semantic relevance. Methods based on Convolutional neural networks (CNN) and Recurrent networks (RNN) provide a more optimized structure that can merge the information in the entire sentence into a single sentence-level representation. However, these representations are often not suitable for sentence interactive learning. We design a multi-dimensional semantic interactive learning model based on the mechanism of multiple written heads in the transformer architecture, which not only considers the correlation and position information between different word levels but also further maps the representation of the sentence to the interactive three-dimensional space, so as to solve the problem and the answer can select the best word-level matching pair, respectively. Experimentally, the algorithm in this paper was tested on Yahoo! and StackEx open-domain datasets. The results show that the performance of the method proposed in this paper is superior to the previous CNN/RNN and BERT-based methods.</p></abstract>

ISTVT: Interpretable Spatial-Temporal Video Transformer for Deepfake Detection

With the rapid development of Deepfake synthesis technology, our information security and personal privacy have been severely threatened in recent years. To achieve a robust Deepfake detection, researchers attempt to exploit the joint spatial-temporal information in the videos, like using recurrent networks and 3D convolutional networks. However, these spatial-temporal models remain room to improve. Another general challenge for spatial-temporal models is that people do not clearly understand what these spatial-temporal models really learn. To address these two challenges, in this paper, we propose an Interpretable Spatial-Temporal Video Transformer (ISTVT), which consists of a novel decomposed spatial-temporal self-attention and a self-subtract mechanism to capture spatial artifacts and temporal inconsistency for robust Deepfake detection. Thanks to this decomposition, we propose to interpret ISTVT by visualizing the discriminative regions for both spatial and temporal dimensions via the relevance (the pixel-wise importance on the input) propagation algorithm. We conduct extensive experiments on large-scale datasets, including FaceForensics++, FaceShifter, DeeperForensics, Celeb-DF, and DFDC datasets. Our strong performance of intra-dataset and cross-dataset Deepfake detection demonstrates the effectiveness and robustness of our method, and our visualization-based interpretability offers people insights into our model.

Variable Length Sequential Iterable Convolutional Recurrent Network for UWB-IR Vehicle Target Recognition

A variable length sequential iterable convolutional recurrent network (VS-ICRN) is proposed in this paper, aiming at improving the vehicle target recognition ability for the Ultra-Wideband Impulse Radar (UWB-IR). Firstly, the array imaging technology is introduced into the UWB-IR, and thus a range-angle imaging method for the array UWB-IR is put forward, to simulate the array UWB-IR vehicles image under different observation conditions. Secondly, in order to make full use of both the deep features in the single image and the deep associated features between the sequence images, a VS-ICRN model is proposed, which includes three sub-modules: the image feature extraction based on the iterable convolution, the variable length sequential image associated feature extraction and the target classification, respectively. Finally, the experiment on the simulation dataset and the MSATAR is carried out to validate the effectiveness of the proposed method. The experimental results on the simulation dataset show that when SNR=-10dB, the proposed method is superior in the recognition rate to the GoogLeNet and AlexNet methods with 16% and 19%, respectively. Meanwhile, the proposed VS-ICRN method only needs 1.38% parameters quantity to achieve a comparable recognition rate as GoogLeNet on MSTAR dataset.

Speech Denoising and Compensation for Hearing Aids Using an FTCRN-Based Metric GAN

Hearing aids aims to improve speech intelligibility for hearing impaired patients to levels comparable to those for normal hearing listeners. However, the interference of environmental noises greatly increase the difficulty of hearing loss compensation. Most related research only focuses on one aspect of noise reduction and hearing loss compensation. In this letter, we propose a metric generative adversarial framework based on a frequency-time convolution recurrent network for joint noise reduction and hearing loss compensation. The audiogram is extended along the frequency axis to form embedded features. A metric discriminator is introduced and the optimization of the generator is guided by an evaluation score related to hearing loss compensation. Additional perceptual-based losses are set to stabilize optimization. Experimental results show that the proposed method can better reduce noise and compensate for hearing loss compared with other algorithms.

Enhancing Road Safety through Accurate Detection of Hazardous Driving Behaviors with Graph Convolutional Recurrent Networks

Car accidents remain a significant public safety issue worldwide, with the majority of them attributed to driver errors stemming from inadequate driving knowledge, non-compliance with regulations, and poor driving habits. To improve road safety, Driving Behavior Detection (DBD) systems have been proposed in several studies to identify safe and unsafe driving behavior. Many of these studies have utilized sensor data obtained from the Controller Area Network (CAN) bus to construct their models. However, the use of publicly available sensors is known to reduce the accuracy of detection models, while incorporating vendor-specific sensors into the dataset increases accuracy. To address the limitations of existing approaches, we present a reliable DBD system based on Graph Convolutional Long Short-Term Memory Networks (GConvLSTM) that enhances the precision and practicality of DBD models using public sensors. Additionally, we incorporate non-public sensors to evaluate the model's effectiveness. Our proposed model achieved a high accuracy of 97.5\% for public sensors and an average accuracy of 98.1\% for non-public sensors, indicating its consistency and accuracy in both settings. To enable local driver behavior analysis, we deployed our DBD system on a Raspberry Pi at the network edge, with drivers able to access daily driving condition reports, sensor data, and prediction results through a monitoring dashboard. Furthermore, the dashboard issues voice warnings to alert drivers of hazardous driving conditions. Our findings demonstrate that the proposed system can effectively detect hazardous and unsafe driving behavior, with potential applications in improving road safety and reducing the number of accidents caused by driver errors.

Human–Robot Collaboration Using Sequential-Recurrent-Convolution-Network-Based Dynamic Face Emotion and Wireless Speech Command Recognitions

The proposed sequential recurrent convolution network (SRCN) includes two parts: one convolution neural network (CNN) and a sequence of long short-term memory (LSTM) models. The CNN is to achieve the feature vector of face emotion or speech command. Then, a sequence of LSTM models with the shared weight reflects a sequence of inputs provided by a (pre-trained) CNN with a sequence of input sub-images or spectrograms corresponding to face emotion and speech command, respectively. Simply put, one SRCN for dynamic face emotion recognition (SRCN-DFER) and another SRCN for wireless speech command recognition (SRCN-WSCR) are developed. The proposed approach not only effectively tackles the recognitions of dynamic mapping of face emotion and speech command with average generalized recognition rate of 98% and 96.7% but also prevents the overfitting problem in a noisy environment. The comparisons among mono and stereo visions, Deep CNN, and ResNet50 confirm the superiority of the proposed SRCN-DFER. The comparisons among SRCN-WSCR with noise-free data, SRCN-WSCR with noisy data, and multiclass support vector machine validate its robustness. Finally, the human-robot collaboration (HRC) using our developed omnidirectional service robot, including human and face detections, trajectory tracking by the previously designed adaptive stratified finite-time saturated control, face emotion and speech command recognitions, and music play, validates the effectiveness, feasibility, and robustness of the proposed method.

A hybrid end-to-end learning approach for breast cancer diagnosis: convolutional recurrent network

In this study, mammography images are classified as normal, benign, and malignant using the Mammographic Image Analysis Society (MIAS) and INbreast datasets. After the preprocessing of each image, the processed images are given as input to two different end-to-end deep networks. The first network contains only a Convolutional Neural Network (CNN), while the second network is a hybrid structure that includes both the CNN and Bidirectional Long Short Term Memories (BiLSTM). The classification accuracy obtained using the first and second hybrid architectures is 97.60% and 98.56% for the MIAS dataset, respectively. In addition, experiments performed for the INbreast dataset at the study's end prove the proposed method's effectiveness. These results are comparable to those obtained in previous popular studies. The proposed study contributes to previous studies in terms of preprocessing steps, deep network design, and high diagnostic accuracy.

Self-optimizing multi-domain auxiliary fusion deep complex convolution recurrent network for speech enhancement

A self-optimizing multi-domain auxiliary fusion deep complex convolution recurrent network for speech enhancement (AMDCCRN) was developed to further improve the feature completeness, correlation expression, and global optimization efficiency of the deep speech enhancement model for speech signals. By constructing a multi-domain auxiliary fusion deep complex convolution recurrent network (MDCCRN), the correlation representation of speech features and features in different spatial domains was enriched. The sparrow search algorithm (PGSSA), which is based on a game theory parallel and global optimization strategy, was introduced to improve parallel search capability and to further augment the efficiency and performance of the model's hyperparameter adaptive search. At the same time, PGSSA was used for adaptive optimization of six key model parameters required for AMDCCRN construction. Finally, a speech enhancement model suitable for multi-domain auxiliary fusion with self-learning ability was constructed. Test results on two common speech corpus data sets, THCHS-30 and WSJ0, showed that the proposed method achieved a better speech enhancement effect than other existing methods and that the validity and generalization of the proposed method had been verified.

Online Learning Engagement Recognition Using Bidirectional Long-Term Recurrent Convolutional Networks

Background: Online learning is currently adopted by educational institutions worldwide to provide students with ongoing education during the COVID-19 pandemic. However, online learning has seen students lose interest and become anxious, which affects learning performance and leads to dropout. Thus, measuring students’ engagement in online learning has become imperative. It is challenging to recognize online learning engagement due to the lack of effective recognition methods and publicly accessible datasets. Methods: This study gathered a large number of online learning videos of students at a normal university. Engagement cues were used to annotate the dataset, which was constructed with three levels of engagement: low engagement, engagement, and high engagement. Then, we introduced a bi-directional long-term recurrent convolutional network (BiLRCN) for online learning engagement recognition in video. Result: An online learning engagement dataset has been constructed. We evaluated six methods using precision and recall, where BiLRCN obtained the best performance. Conclusions: Both category balance and category similarity of the data affect the performance of the results; it is more appropriate to consider learning engagement as a process-based evaluation; learning engagement can provide intervention strategies for teachers from a variety of perspectives and is associated with learning performance. Dataset construction and deep learning methods need to be improved, and learning data management also deserves attention.

Learning spatiotemporal embedding with gated convolutional recurrent networks for translation initiation site prediction

Accurately predicting translation initiation sites (TIS) from genomic sequences is crucial for understanding gene regulation and function. TIS prediction methods’ feature vectors are not discriminative enough to lead to unsatisfactory predictive results. In this work, we devise an efficient gated convolutional recurrent network (GCR-Net) with residual learning to dynamically extract dependency patterns of raw genomic sequences in an efficient fusion strategy and successfully improve the performance of the TIS prediction. GCR-Net mainly includes exponential gated convolutional residual networks (EGCRN) and bidirectional gated recurrent unit (Bi-GRU) networks. Particularly, we devise the novel EGCRN to extract multiple complex patterns of the spatial dimension from genomic sequences, where we design an exponential gated linear unit (EGLU) to reduce the vanishing gradient problem. Moreover, we combine EGLU with shortcut connections to develop the stacked gated mechanism based on convolutions that benefit information propagation across layers. Then, we use Bi-GRU with identity connections to learn long-term dependency patterns of the temporal dimension from genomic sequences. Besides, we evaluate our GCR-Net model on four TIS datasets, and experiments demonstrate that GCR-Net is an efficient deep learning-based TIS prediction tool and obtains superior performance compared to the baseline methods.

Dynamic graph convolutional recurrent imputation network for spatiotemporal traffic missing data

In real-world intelligent transportation systems, the spatiotemporal traffic data collected from sensors often exhibit missing or corrupted data, significantly hindering the development of traffic data research. Missing data imputation is a classic research topic that encompasses a wide range of methods. However, these methods are typically underdeveloped in two aspects: the dynamic spatial dependencies of the road network over time, and the information extraction and utilization of diverse data. In this study, we design a novel deep learning architecture – Dynamic Graph Convolutional Recurrent Imputation Network (DGCRIN) – as a tool to impute missing traffic data. The DGCRIN employs a graph generator and dynamic graph convolutional gated recurrent unit (DGCGRU) to perform fine-grained modeling of the dynamic spatiotemporal dependencies of road network. Additionally, an auxiliary GRU learns the missing pattern information of the data, and a fusion layer with a decay mechanism is introduced to fuse a diverse range of information. This architecture enables the DGCRIN to be highly adaptable to complex scenarios involving missing data. Extensive experiments on two datasets demonstrate the superiority of DGCRIN over multiple baseline models.

Routing hypergraph convolutional recurrent network for network traffic prediction

Routing hypergraph convolutional recurrent network for network traffic prediction

Deep MCANC: A deep learning approach to multi-channel active noise control

Traditional multi-channel active noise control (MCANC) is based on adaptive filtering and usually uses a separate control unit for each channel. This paper introduces a deep learning based approach for multi-channel active noise control (ANC). The proposed approach, called deep MCANC, encodes optimal control parameters corresponding to different noises and environments, and jointly computes the multiple canceling signals to cancel or attenuate the primary noises captured at error microphones. A convolutional recurrent network (CRN) is employed for complex spectral mapping where the summated power of error signals is used as the loss function for CRN training. Deep MCANC is a fixed-parameter ANC approach and large-scale multi-condition training is employed to achieve robustness against a variety of noises. We explore the performance of deep MCANC with different setups and investigate the impact of factors such as the number of loudspeakers and microphones, and the position of a secondary source, on ANC performance. Experimental results show that deep MCANC is effective for wideband noise reduction and generalizes well to untrained noises. Moreover, the proposed approach is robust against variations in reference signals and works well in the presence of nonlinear distortions.

Multimodal EEG Emotion Recognition Based on the Attention Recurrent Graph Convolutional Network

EEG-based emotion recognition has become an important part of human–computer interaction. To solve the problem that single-modal features are not complete enough, in this paper, we propose a multimodal emotion recognition method based on the attention recurrent graph convolutional neural network, which is represented by Mul-AT-RGCN. The method explores the relationship between multiple-modal feature channels of EEG and peripheral physiological signals, converts one-dimensional sequence features into two-dimensional map features for modeling, and then extracts spatiotemporal and frequency–space features from the obtained multimodal features. These two types of features are input into a recurrent graph convolutional network with a convolutional block attention module for deep semantic feature extraction and sentiment classification. To reduce the differences between subjects, a domain adaptation module is also introduced to the cross-subject experimental verification. This proposed method performs feature learning in three dimensions of time, space, and frequency by excavating the complementary relationship of different modal data so that the learned deep emotion-related features are more discriminative. The proposed method was tested on the DEAP, a multimodal dataset, and the average classification accuracies of valence and arousal within subjects reached 93.19% and 91.82%, respectively, which were improved by 5.1% and 4.69%, respectively, compared with the only EEG modality and were also superior to the most-current methods. The cross-subject experiment also obtained better classification accuracies, which verifies the effectiveness of the proposed method in multimodal EEG emotion recognition.

DEF-Net: A Dual-Encoder Fusion Network for Fundus Retinal Vessel Segmentation

The deterioration of numerous eye diseases is highly related to the fundus retinal structures, so the automatic retinal vessel segmentation serves as an essential stage for efficient detection of eye-related lesions in clinical practice. Segmentation methods based on encode-decode structures exhibit great potential in retinal vessel segmentation tasks, but have limited feature representation ability. In addition, they don’t effectively consider the information at multiple scales when performing feature fusion, resulting in low fusion efficiency. In this paper, a newly model, named DEF-Net, is designed to segment retinal vessels automatically, which consists of a dual-encoder unit and a decoder unit. Fused with recurrent network and convolution network, a dual-encoder unit is proposed, which builds a convolutional network branch to extract detailed features and a recurrent network branch to accumulate contextual features, and it could obtain richer features compared to the single convolution network structure. Furthermore, to exploit the useful information at multiple scales, a multi-scale fusion block used for facilitating feature fusion efficiency is designed. Extensive experiments have been undertaken to demonstrate the segmentation performance of our proposed DEF-Net.

Multivariate selection-combination short-term wind speed forecasting system based on convolution-recurrent network and multi-objective chameleon swarm algorithm

Wind energy, as a typical environmentally friendly source of energy for power generation, has the advantages of being renewable and emitting no greenhouse gases. Moreover, in wind power generation, accurate wind speed prediction is vital. However, most existing forecasting models use only univariate time series forecasting models, ignoring the effect of other variables on wind speed and the improvement of the model using optimization algorithms, resulting in lower accuracy and stability. Aiming to fill this gap, we develop a complete multivariate selection-combination short-term wind speed forecasting system, which is composed of two advanced feature-selection methods; six single forecasting models based on convolutional and recurrent neural networks and a multi-objective chameleon swarm optimization algorithm. We prove theoretically that the proposed multi-objective chameleon swarm optimization algorithm has Pareto optimal solutions and performs best in some test functions by comparing it with other multi-objective swarm optimization algorithms. For wind speed prediction in summer, our proposed prediction system achieves a mean absolute percentage error of 1.937%, 2.110% and 2.584% in one-, two- and three-step forecasting, respectively, which is higher than single and other combined models.

Toward Privacy Preserving Federated Learning in Internet of Vehicular Things: Challenges and Future Directions

The Internet of vehicular things (IoVT) is turning into an indubitably evolving area of interest in either industrial or academic domains. The tremendous information exchanging between IoVT devices enable the development of a wide variety of vehicular applications i.e., intelligent transportation systems and autonomous driving system, etc. However, the sensitivity of this information resulted in growing security privacy concerns. Remarkably, federated learning (FL) is a promising paradigm of distributed learning from vehicular data of distinct agents without communicating the raw data among them. FL can appropriately use the computation power of manifold agents to develop efficient and privacy-preserving solutions for IoVT environment. Thus, this study figures out the potential of the FL approach in developing efficient decentralized solutions that consider the security and privacy concerns of the IoVT system. A federated graph convolutional recurrent network (Fed-GCRN) is introduced to learn spatial-temporal information for traffic flows forecasting. The Fed-GCRN introduce an adaptive differential privacy mechanism to realize a better privacy performance tradeoff. Finally, the current challenges related to FL are discussed along with the hopeful future directions that enable the development of more intelligent, secure, and private IoVT applications.