Unidirectional Recurrent Neural Network Research Articles

Bidirectional Hybrid LSTM Based Recurrent Neural Network for Multi-View Stereo.

Recently, deep learning based multi-view stereo (MVS) networks have demonstrated their excellent performance on various benchmarks. In this paper, we present an effective and efficient recurrent neural network (RNN) for accurate and complete dense point cloud reconstruction. Instead of regularizing the cost volume via conventional 3D CNN or unidirectional RNN like previous attempts, we adopt a bidirectional hybrid Long Short-Term Memory (LSTM) based structure for cost volume regularization. The proposed bidirectional recurrent regularization is able to perceive full-space context information comparable to 3D CNNs while saving runtime memory. For post-processing, we introduce a visibility based approach for depth map refinement to obtain more accurate dense point clouds. Extensive experiments on DTU, Tanks and Temples and ETH3D datasets demonstrate that our method outperforms previous state-of-the-art MVS methods and exhibits high memory efficiency at runtime.

Harnessing the Missing Spectral Correlation for Metasurface Inverse Design.

A long-held tenet in computer science asserts that the training of deep learning is analogous to an alchemical furnace, and its "black box" signature brings forth inexplicability. For electromagnetic metasurfaces, the related intelligent applications also get stuck into such a dilemma. Although the past 5 years have witnessed a proliferation of deep learning-based works across complex photonic scenarios, they neglect the already existing but untapped physical laws. Here, the intrinsic correlation between the real and imaginary parts of the spectra are revealed using Kramers-Kronig relations, which is then mimicked by bidirectional information flow in neural network space. Such consideration harnesses the missing spectral connection to extract crucial features effectively. The bidirectional recurrent neural network is benchmarked in metasurface inverse design and compare it with a fully-connected neural network, unidirectional recurrent neural network, and attention-based transformer. Beyond the improved accuracy, the study examines the intermediate information products and physically explains why different network structures yield different performances. The work offers explicable perspectives to utilize physical information in the deep learning field and facilitates many data-intensive research endeavors.

Online Video Super-resolution using Information Replenishing Unidirectional Recurrent Model

Recurrent Neural Networks (RNN) are widespread for Video Super-Resolution (VSR) because of their proven ability to learn spatiotemporal inter-dependencies across the temporal dimension. Despite RNN’s ability to propagate memory across longer sequences of frames, vanishing gradient and error accumulation remain major obstacles to unidirectional RNNs in VSR. Several bi-directional recurrent models are suggested in the literature to alleviate this issue; however, these models are only applicable to offline use cases due to heavy demands for computational resources and the number of frames required per input. This paper proposes a novel unidirectional recurrent model for VSR, namely “Replenished Recurrency with Dual-Duct” (R2D2), that can be used in an online application setting. R2D2 incorporates a recurrent architecture with a sliding-window-based local alignment resulting in a recurrent hybrid architecture. It also uses a dual-duct residual network for concurrent and mutual refinement of local features along with global memory for full utilisation of the information available at each timestamp. With novel modelling and sophisticated optimisation, R2D2 demonstrates competitive performance and efficiency despite the lack of information available at each time-stamp compared to its offline (bi-directional) counterparts. Ablation analysis confirms the additive benefits of the proposed sub-components of R2D2 over baseline RNN models.The PyTorch-based code for the R2D2 model will be released at R2D2 GitRepo.

Multiview Spatial–Spectral Two-Stream Network for Hyperspectral Image Unmixing

Linear spectral unmixing is an important technique in the analysis of mixed pixels in hyperspectral images. In recent years, deep learning-based methods have been garnering increasing attention in hyperspectral unmixing; especially, unsupervised autoencoder (AE) networks that have achieved excellent unmixing performance are a recent trend. While most approaches use spatial information, it is well known that hyperspectral data are characterized by a large number of narrow spectral bands. In order to take full advantage of the hyperspectral bands in unmixing and the spatial information, in this article, we explore multiview spectral and spatial information in an AE-based unmixing framework. We introduce multiview spectral information through spectral partitioning and propose a multiview spatial–spectral two-stream network, called MSSS-Net, which simultaneously learns a spatial stream network and a multiview spectral stream network in an end-to-end fashion for more efficient unmixing. The MSSS-Net is a two-stream deep unmixing network sharing a decoder, where its two AE networks employ recurrent neural networks (RNNs) to collaboratively utilize multiview spectral and spatial information. The spatial stream network branch extracts the spatial features of pixels and its neighbors, while the multiview spectral stream network branch exploits the multiview spectral bands of a pixel. Meanwhile, we design a cascaded bidirectional and unidirectional RNNs’ encoder structure for multiview spatial–spectral information to learn more discriminative deep patch-pixel features. Extensive ablation studies and experiments on both synthetic and real datasets demonstrate the superiority of the MSSS-Net over state-of-the-art unmixing methods.

A Comparative Study of Deep Learning-Based Vulnerability Detection System

Source code static analysis has been widely used to detect vulnerabilities in the development of software products. The vulnerability patterns purely based on human experts are laborious and error prone, which has motivated the use of machine learning for vulnerability detection. In order to relieve human experts of defining vulnerability rules or features, a recent study shows the feasibility of leveraging deep learning to detect vulnerabilities automatically. However, the impact of different factors on the effectiveness of vulnerability detection is unknown. In this paper, we collect two datasets from the programs involving 126 types of vulnerabilities, on which we conduct the first comparative study to quantitatively evaluate the impact of different factors on the effectiveness of vulnerability detection. The experimental results show that accommodating control dependency can increase the overall effectiveness of vulnerability detection F1-measure by 20.3%; the imbalanced data processing methods are not effective for the dataset we create; bidirectional recurrent neural networks (RNNs) are more effective than unidirectional RNNs and convolutional neural network, which in turn are more effective than multi-layer perception; using the last output corresponding to the time step for the bidirectional long short-term memory (BLSTM) can reduce the false negative rate by 2.0% at the price of increasing the false positive rate by 0.5%.

Eye blink completeness detection

Computer users often complain about eye discomfort caused by dry eye syndrome. This is sometimes caused and accompanied by incomplete blinks. There are several algorithms for eye blink detection, but none which would distinguish complete blinks from the incomplete ones. We introduce the first method which detects blink completeness. Blinks differ in speed and duration similar to speech, therefore Recurrent Neural Network (RNN) is used as a classifier due to its suitability for sequence-based features. We show that using unidirectional RNN with time shifting achieves higher performance compared to a bidirectional RNN, which is a suitable choice in this kind of problem where the feature pattern is not yet observed for the initial frames. We report the best results (increase by almost 8%) on the most challenging dataset: Researcher’s night. We formulate a new important problem and state an initial benchmark for further research.

Error detection and accuracy estimation in automatic speech recognition using deep bidirectional recurrent neural networks

Recurrent neural networks (RNNs) have recently been applied as the classifiers for sequential labeling problems. In this paper, deep bidirectional RNNs (DBRNNs) are applied to error detection in automatic speech recognition (ASR), which is a sequential labeling problem. We investigate three types of ASR error detection tasks, i.e. confidence estimation, out-of-vocabulary word detection and error type classification. We also estimate ASR accuracy, i.e. percent correct and word accuracy, from the error type classification results. Experimental results using English and Japanese lecture speech corpora show that the DBRNNs greatly outperform conditional random fields (CRFs) and the other NN structures, i.e. deep feedforward NNs (DNNs) and deep unidirectional RNNs (DURNNs). These performance improvements are because the DBRNNs can take the longer bidirectional context of input feature vectors into account and can model highly nonlinear relationships between the input feature vectors and output labels. In detailed analyses, the DBRNNs show a better generalization ability than the CRFs. These results are thanks to the ability of the DBRNNs to represent (embed) the words in a low-dimensional continuous value vector space. In addition, the superiority of the DBRNNs to the DNNs and DURNNs indicates that the average length of the context of the input feature vectors required for ASR error detection is only a few time steps, however, it will change (lengthen) depending on the situation.