Two-stream Neural Network Research Articles

Modelling a Spatial-Motion Deep Learning Framework to Classify Dynamic Patterns of Videos

Video classification is an essential process for analyzing the pervasive semantic information of video content in computer vision. Traditional hand-crafted features are insufficient when classifying complex video information due to the similarity of visual contents with different illumination conditions. Prior studies of video classifications focused on the relationship between the standalone streams themselves. In this paper, by leveraging the effects of deep learning methodologies, we propose a two-stream neural network concept, named state-exchanging long short-term memory (SE-LSTM). With the model of spatial motion state-exchanging, the SE-LSTM can classify dynamic patterns of videos using appearance and motion features. The SE-LSTM extends the general purpose of LSTM by exchanging the information with previous cell states of both appearance and motion stream. We propose a novel two-stream model Dual-CNNSELSTM utilizing the SE-LSTM concept combined with a Convolutional Neural Network, and use various video datasets to validate the proposed architecture. The experimental results demonstrate that the performance of the proposed two-stream Dual-CNNSELSTM architecture significantly outperforms other datasets, achieving accuracies of 81.62%, 79.87%, and 69.86% with hand gestures, fireworks displays, and HMDB51 datasets, respectively. Furthermore, the overall results signify that the proposed model is most suited to static background dynamic patterns classifications.

Infant Cognitive Scores Prediction with Multi-stream Attention-Based Temporal Path Signature Features.

There is stunning rapid development of human brains in the first year of life. Some studies have revealed the tight connection between cognition skills and cortical morphology in this period. Nonetheless, it is still a great challenge to predict cognitive scores using brain morphological features, given issues like small sample size and missing data in longitudinal studies. In this work, for the first time, we introduce the path signature method to explore hidden analytical and geometric properties of longitudinal cortical morphology features. A novel BrainPSNet is proposed with a differentiable temporal path signature layer to produce informative representations of different time points and various temporal granules. Further, a two-stream neural network is included to combine groups of raw features and path signature features for predicting the cognitive score. More importantly, considering different influences of each brain region on the cognitive function, we design a learning-based attention mask generator to automatically weight regions correspondingly. Experiments are conducted on an in-house longitudinal dataset. By comparing with several recent algorithms, the proposed method achieves the state-of-the-art performance. The relationship between morphological features and cognitive abilities is also analyzed.

Video Saliency Prediction Based on Spatial-Temporal Two-Stream Network

In this paper, we propose a novel two-stream neural network for video saliency prediction. Unlike some traditional methods based on hand-crafted feature extraction and integration, our proposed method automatically learns saliency related spatiotemporal features from human fixations without any pre-processing, post-processing, or manual tuning. Video frames are routed through the spatial stream network to compute static or color saliency maps for each of them. And a new two-stage temporal stream network is proposed, which is composed of a pre-trained 2D-CNN model (SF-Net) to extract saliency related features and a shallow 3D-CNN model (Te-Net) to process these features, for temporal or dynamic saliency maps. It can reduce the requirement of video gaze data, improve training efficiency, and achieve high performance. A fusion network is adopted to combine the outputs of both streams and generate the final saliency maps. Besides, a convolutional Gaussian priors (CGP) layer is proposed to learn the bias phenomenon in viewing behavior to improve the performance of the video saliency prediction. The proposed method is compared with state-of-the-art saliency models on two public video saliency benchmark datasets. The results demonstrate that our model can achieve advanced performance on video saliency prediction.

Design of Outdoor Fire Intelligent Alarm System Based on Image Recognition

Fire is one of the most common serious disasters in human society. It is a kind of burning phenomenon that is out of control in time and space. When a fire occurs, how to detect the fire quickly and remove it in the budding state has become the key content of fire control work. Outdoor fire is very common in our daily life, and once it occurs without effective and timely control, it will cause huge losses. Therefore, it is particularly important to study an intelligent alarm system for outdoor fire. Generally, fire detection technology can be divided into sensor fire detection technology and image fire detection technology. Sensor fire detection technology is low cost and easy to design, but its application field is limited. Under the interference of many factors outside, misjudgement and missed judgement will occur. Image fire detection technology can achieve certain detection function through manual design of features and classifiers, but there are still defects in the application in the actual diversified environment. With the development of neural network technology in recent years, it has made great breakthroughs in the field of image recognition. Its judgment type is obtained through a large number of data training algorithms. Because of its automatic feature extraction and classification characteristics, it can effectively adapt to the external environment. Therefore, this paper proposes an end-to-end two-stream neural network model to detect fires, uses fire video on the network to train the algorithm, and then uses the fire database to test. Compared with the existing fire detection algorithms, it is found that the proposed method has good practicability and versatility, and provides a good reference for the development of fire detection technology.

Convolutional Two-Stream Network Using Multi-Facial Feature Fusion for Driver Fatigue Detection

Road traffic accidents caused by fatigue driving are common causes of human casualties. In this paper, we present a driver fatigue detection algorithm using two-stream network models with multi-facial features. The algorithm consists of four parts: (1) Positioning mouth and eye with multi-task cascaded convolutional neural networks (MTCNNs). (2) Extracting the static features from a partial facial image. (3) Extracting the dynamic features from a partial facial optical flow. (4) Combining both static and dynamic features using a two-stream neural network to make the classification. The main contribution of this paper is the combination of a two-stream network and multi-facial features for driver fatigue detection. Two-stream networks can combine static and dynamic image information, while partial facial images as network inputs can focus on fatigue-related information, which brings better performance. Moreover, we applied gamma correction to enhance image contrast, which can help our method achieve better results, noted by an increased accuracy of 2% in night environments. Finally, an accuracy of 97.06% was achieved on the National Tsing Hua University Driver Drowsiness Detection (NTHU-DDD) dataset.

Spectral and Temporal Feature Learning With Two-Stream Neural Networks for Mental Workload Assessment.

People's mental workload profoundly affects their work efficiency and health. Mental workload assessment can be used to effectively avoid serious accidents caused by excessive mental workload. Both electroencephalogram (EEG) spectral features and its temporal features have proven to be useful in addressing this problem. The fusion of the two types of features can provide rich distinguishing information for improving mental workload assessment. Benefiting from the progress of deep learning, this study proposes the two-stream neural networks (TSNN) for fusing the two types of EEG features. Compared with hand-crafted features, the TSNN can learn and fuse EEG features from the spectral and temporal dimensions automatically without prior knowledge. The TSNN includes a spectral stream and a temporal stream. Each stream consists of a convolutional neural network (CNN) and a temporal convolutional network (TCN) to learn spectral or temporal features from EEG topographic maps. To fuse the learned spectral and temporal information, we concatenate the output of the two streams prior to the fully connected layer. EEG data were collected from 17 subjects who performed n-back tasks with easy, medium, and hard difficulty levels, leading to a three-class mental workload classification. The results show that the TSNN achieves an average accuracy of 91.9%, which is a significant improvement over baseline classifiers based on hand-crafted features. The TSNN also outperforms state-of-the-art deep learning methods developed for EEG classification. The results indicate that the proposed structure is promising for fusing spectral and temporal features for mental workload assessment. In addition, it provides a high-precision approach for potential applications during cognitive activities.

Discriminant Deep Feature Learning based on joint supervision Loss and Multi-layer Feature Fusion for heterogeneous face recognition

Heterogeneous face recognition (HFR) is still a challenging problem in computer vision community due to large appearance difference between near infrared (NIR) and visible light (VIS) modalities. Recently, breakthroughs have been made for traditional face recognition by applying deep learning on a huge amount of labeled VIS face samples. However, the same deep learning approach cannot be simply applied to HFR task due to large domain difference as well as insufficient pairwise images in different modalities during training. In general, the pooling layer of deep network can play the role of feature reduction, but also lead to the loss of useful face information, resulting in a decrease in the performance of HFR problem. It is important to eliminate modal-related information and retain more facial identity information. In this paper, we propose a novel method called Discriminant Deep Feature Learning Based on Joint Supervision Loss and Multi-layer Feature Fusion (DDFLJM) for HFR task. In most of the available CNNs, the softmax loss function is used as the supervision signal to train the deep model. In order to enhance the discriminative power of the deeply learned features, this paper proposes a new loss function called Scatter Loss (SL), which embeds both inter- and intra-class information for effectively training the deep model. To make full use of the various layers of the deep network, a Dimension Reduction Block (DRB) is designed to effectively extract the auxiliary features on multiple mid-level layers. An orthogonality constraint is introduced to the DRB block to reduce spectrum variations of two different modalities. The proposed SL is applied to multiple layers of network for joint supervision training, which enables multiple layers of the network to obtain discriminative identity features. Moreover, a Modified Gate Two-stream Neural Network (MGTNN) is adopted to fuse multiple-layer features. Extensive experiments are carried out on two challenging NIR-VIS HFR datasets CASIA NIR-VIS 2.0 and Oulu-CASIA NIR-VIS, demonstrating the superiority of the proposed method.

Learning to Segment Moving Objects

We study the problem of segmenting moving objects in unconstrained videos. Given a video, the task is to segment all the objects that exhibit independent motion in at least one frame. We formulate this as a learning problem and design our framework with three cues: (i) independent object motion between a pair of frames, which complements object recognition, (ii) object appearance, which helps to correct errors in motion estimation, and (iii) temporal consistency, which imposes additional constraints on the segmentation. The framework is a two-stream neural network with an explicit memory module. The two streams encode appearance and motion cues in a video sequence respectively , while the memory module captures the evolution of objects over time, exploiting the temporal consistency. The motion stream is a convolutional neural network trained on synthetic videos to segment independently moving objects in the optical flow field. The module to build a 'visual memory' in video, i.e., a joint representation of all the video frames, is realized with a convolutional recurrent unit learned from a small number of training video sequences. For every pixel in a frame of a test video, our approach assigns an object or background label based on the learned spatio-temporal features as well as the 'visual memory' specific to the video. We evaluate our method extensively on three benchmarks, DAVIS, Freiburg-Berkeley motion seg-mentation dataset and SegTrack. In addition, we provide an extensive ablation study to investigate both the choice of the training data and the influence of each component in the proposed framework.

Two‐stream neural networks to detect manipulation of JPEG compressed images

With the rapid spread of image editing software, anyone can easily create, distribute, and forge images. Although techniques to detect image forgery have been widely studied, current techniques have significant limitations, such as specific file formats, manipulations, or compression qualities. Although deep learning techniques have been introduced to detect various manipulations, such as blurring, median filtering, and Gaussian noise, these techniques are only suitable to detect forgeries of uncompressed images, and are difficult to apply in practice because most images are compressed for distribution. Therefore, a two-stream neural network approach for image forensics that is robust to compression is proposed. The two-stream neural network is based on constrained convolutional neural network and Markov characteristics to consider compression. Experimental results show that the proposed method overcomes current technique limitations.

GAN-Assisted Two-Stream Neural Network for High-Resolution Remote Sensing Image Classification

Using deep learning to improve the capabilities of high-resolution satellite images has emerged recently as an important topic in automatic classification. Deep networks track hierarchical high-level features to identify objects; however, enhancing the classification accuracy from low-level features is often disregarded. We therefore proposed a two-stream deep-learning neural network strategy, with a main stream utilizing fine spatial-resolution panchromatic images to retain low-level information under a supervised residual network structure. An auxiliary line employed an unsupervised net to extract high-level abstract and discriminative features from multispectral images to supplement the spectral information in the main stream. Various feature extraction types from the neural network were selected and jointed in the novel net, as the combined high- and low-level features could provide a superior solution to image classification. In traditional convolutional neural networks, increased network depth might not influence the network performance perceptibly; however, we introduced a residual neural network to develop the expressive ability of the deeper net, increasing the role of net depth in feature extraction. To enhance feature robustness, we proposed a novel consolidation part in feature extraction. An adversarial net improved the feature extraction capabilities and aided digging the inherent and discriminative features from data, with increased extraction efficacy. Tests on satellite images indicated the high overall accuracy of our novel net, verifying that net depth or number of convolution kernels affected the classification capability. Various comparative tests proved the structural rationality for our two-stream structure.