Two-stream Neural Network Research Articles

Two-stream neural network with different gradient update strategies

Deep neural networks will be affected by various noises in different scenes. Traditional deep neural networks often use gradient descent algorithms to update parameter weights. When the gradient falls to a certain range, it is easy to fall into the local optimal solution. Although the impulse method and other methods can escape from local optimization in some scenarios, they still have some limitations, which will greatly reduce the application effect of the actual scenes. To solve the above problems, a two-stream neural network with different gradient update strategies was proposed. Combined with the gradient ascent algorithm, this method alleviated the disadvantage of deep neural networks falling into local optimality and increased the robustness of neural networks to a certain extent. The experimental results on the CIFAR10 dataset verify that the proposed method can improve the accuracy of various gradient descent optimizers by about 1%, such as SGD, Adagrad, RM-Sprop and Adam. The experimental results on the COCO dataset show that the accuracy of the proposed method is also improved compared with the baseline models PAA and EfficientDet. The method proposed can be widely used in various neural network structures and has good practical significance and application prospects.

Falling Behavior Detection System for Elevator Passengers Based on Deep Learning and Edge Computing

Due to factors such as dense populations and narrow viewing angles, previous deep-learning models for detecting passenger behavior in elevators often lack effectiveness. Traditional cloud-based data transmission methods have issues with high latency, high resource usage, and privacy threats, particularly during periods of high usage. To address these issues, we proposed a falling behavior detection system for elevator passengers based on deep learning and edge computing. A two-stream neural network model improved by 3D ResNet is presented, which utilizes edge computing for elevator passenger fall detection. Our homemade dataset of elevator passenger falling behavior is utilized to train and evaluate the system. The results demonstrate that the system is effective in detecting passengers’ falling behavior in elevators, with an average accuracy of 89.2%. The feasibility of the system in an elevator is also verified, and it has performed well. The application of this system in this field holds significant research value.

RETRACTED ARTICLE: Simulation of video image detection in leisure sports tourism industry based on convolutional neural network

Due to the continuous development of information technology, emerging technologies such as 5G networks, Turing devices and deep learning are widely used in people's lives. At present, the sports and tourism industry has become a pillar industry supporting the national economy. Sports events have attracted many people's attention, and people are willing to devote themselves to sports activities. Participants and visitors are also increasing day by day. The article also conducts research on the rapid detection of human video behavior, obtains more representative features for this demand. This research is of great help to the development of the industry. This allows the network to manipulate the input video behavior at different scales and extract deeper video information. Therefore, this paper proposes another 3D convolutional neural network model that combines a multi-level pyramid network with an attention mechanism. The visual attributes in the video behavior data set are explicitly learned to refine the classification of similar features. This paper also proposes an integrated neural network based on visual attribute enhancement. It also shows good adaptability, and can complete video data processing based on time series. The extraction of time and space functions is realized by the two-stream neural network method, the two networks learn the corresponding pixel ratios, and it is not easy to lose the clues of the action recognition. This paper also proposes a spatiotemporal convolutional neural network video for human body recognition algorithm, which is applied to the leisure tourism industry according to the specific analysis of the above method, and promotes the development of the industry.

TGSNET: A FRACTAL NEURAL NETWORK FOR ACTION RECOGNITION

In the study of action recognition based on optical flow, improving the recognition speed of two-stream neural networks is challenging. In this paper, a new network structure Teacher Guided Student Network (TGSNet) which is based on two-stream and teacher–student architecture is proposed to judge the category of action rapidly in the application. There are two sub-networks with optical flow and RGB frame stream in the network, the optical flow sub-network is assigned as the teacher and the RGB frame stream sub-network as the student. In the training stage, the optical flow sub-network computes the optical flow of the video frame and trains the sub-network then transmits the feature to the RGB frame stream sub-network. The RGB frame stream sub-network uses the RGB frame to mimic the optical flow to train the sub-network. In the test stage, there is only RGB frame stream sub-network existing for action recognition rapidly without computing optical flow. The experimental results show that the TGSNet feeds only by RGB frame stream get a competitive accuracy of 56.7% and a better run-time on HMDB51.

A Two-Stream Neural Network for Pose-Based Hand Gesture Recognition

Pose-based hand gesture recognition has been widely studied in the recent years. Compared with full body action recognition, hand gesture involves joints that are more spatially closely distributed with stronger collaboration. This nature requires a different approach from action recognition to capturing the complex spatial features. Many gesture categories, such as “Grab” and “Pinch,” have very similar motion or temporal patterns posing a challenge on temporal processing. To address these challenges, this article proposes a two-stream neural network with one stream being a self-attention-based graph convolutional network (SAGCN) extracting the short-term temporal information and hierarchical spatial information, and the other being a residual-connection-enhanced bidirectional independently recurrent neural network (IndRNN) for extracting long-term temporal information. The SAGCN has a dynamic self-attention mechanism to adaptively exploit the relationships of all hand joints in addition to the fixed topology and local feature extraction in the GCNs. The proposed method effectively takes advantage of the GCN and IndRNN to capture the temporal-spatial information. The widely used Dynamic Hand Gesture dataset (two evaluation protocols) and First-Person Hand Action dataset are used to validate its effectiveness and our method achieves state-of-the-art performance with 96.31%, 94.05%, and 90.26%, respectively, in terms of recognition accuracy.

Sex Detection of Chicks Based on Audio Technology and Deep Learning Methods.

The sex detection of chicks is an important work in poultry breeding. Separating chicks of different sexes early can effectively improve production efficiency and commercial benefits. In this paper, based on the difference in calls among one-day-old chicks of different sexes, a sex detection method based on chick calls is designed. Deep learning methods were used to classify the calls of chicks and detect their sex. This experiment studies three different varieties of chicks. The short-time zero-crossing rate was used to automatically detect the endpoints of chick calls in audio. Three kinds of audio features were compared: Spectrogram, Cepstrogram and MFCC+Logfbank. The features were used as the input in neural networks, and there were five kinds of neural networks: CNN, GRU, CRNN, TwoStream and ResNet-50. After the cross-comparison experiment of different varieties of chicks, audio features and neural networks, the ResNet-50 neural network trained with the MFCC+Logfbank audio features of three yellow chick calls had the highest test accuracy of 83% when testing Three-yellow chicks' calls. The GRU neural network trained with the Spectrogram audio features of native chick calls had the highest test accuracy of 76.8% when testing Native chicks' calls. The ResNet-50 neural network trained with Spectrogram audio features of flaxen-yellow chick calls had the highest test accuracy of 66.56%when testing flaxen-yellow chick calls. Multiple calls of each chick were detected, and the majority voting method was used to detect the sex of the chicks. The ResNet-50 neural network trained with the Spectrogram of three yellow chick calls had the highest sex detection accuracy of 95% when detecting the three yellow chicks' sex. The GRU neural network trained with the Spectrogram and cepstrogram of native chick calls and the CRNN network trained with the Spectrogram of native chick calls had the highest sex detection accuracy of 90% when detecting the native chicks' sex. The Twostream neural network trained with MFCC+Logfbank of flaxen-yellow chick calls and the ResNet-50 network trained with the Spectrogram of flaxen-yellow chick calls had the highest sex detection accuracy of 80% when detecting the flaxen-yellow chicks' sex. The results of the cross-comparison experiment show that there is a large diversity between the sex differences in chick calls of different breeds. The method is more applicable to chick sex detection in three yellow chicks and less so in native chicks and flaxen-yellow chicks. Additionally, when detecting the sex of chicks of a similar breed to the training chicks, the method obtained better results, while detecting the sex of chicks of other breeds, the detection accuracy was significantly reduced. This paper provides further perspectives on the sex detection method of chicks based on their calls and help and guidance for future research.

SRTNet: a spatial and residual based two-stream neural network for deepfakes detection

SRTNet: a spatial and residual based two-stream neural network for deepfakes detection

Learning dynamic background for weakly supervised moving object detection

Moving Object Detection (MOD) aims at extracting foreground moving objects in videos from static cameras. While low-rank based approaches have achieved impressive success in the MOD task, their performance remains limited on dynamics background scenes. The main reason is that dynamic clutters, e.g., swaying leaves and rippers, are easy to mix up with moving objects in the decomposition model which simply classify the sparse noise as foregrounds. In order to improve the generalization ability of low-rank based moving object detectors, we suggest adding an explicit dynamic clutter component in the decomposition framework with realistic dynamic background modeling. Then the dynamic clutter can be learned through object-free video data due to their self-similarity across time and space. Thus, the moving objects can be naturally separated by a tensor-based decomposition model which formulates the static background by a unidirectional low-rank tensor, learns the dynamic clutter by a two-stream neural network, and constrains moving objects with spatiotemporal continuity. To further provide a more accurate object detection result, an objectness prior is embedded into our model in an attention manner. Extensive experimental results on the challenging datasets of dynamic background clearly demonstrate the superior performance of our model over the state-of-the-art in terms of quantitative metrics and visual quality.

Sound Can Help Us See More Clearly.

In the field of video action classification, existing network frameworks often only use video frames as input. When the object involved in the action does not appear in a prominent position in the video frame, the network cannot accurately classify it. We introduce a new neural network structure that uses sound to assist in processing such tasks. The original sound wave is converted into sound texture as the input of the network. Furthermore, in order to use the rich modal information (images and sound) in the video, we designed and used a two-stream frame. In this work, we assume that sound data can be used to solve motion recognition tasks. To demonstrate this, we designed a neural network based on sound texture to perform video action classification tasks. Then, we fuse this network with a deep neural network that uses continuous video frames to construct a two-stream network, which is called A-IN. Finally, in the kinetics dataset, we use our proposed A-IN to compare with the image-only network. The experimental results show that the recognition accuracy of the two-stream neural network model with uesed sound data features is increased by 7.6% compared with the network using video frames. This proves that the rational use of the rich information in the video can improve the classification effect.

Novel Architecture for Human Re-Identification with a Two-Stream Neural Network and Attention Mechanism

This paper proposes a novel architecture that utilises an attention mechanism in conjunction with multi-stream convolutional neural networks (CNN) to obtain high accuracy in human re-identification (Reid). The proposed architecture consists of four blocks. First, the pre-processing block prepares the input data and feeds it into a spatial-temporal two-stream CNN (STC) with two fusion points that extract the spatial-temporal features. Next, the spatial-temporal attentional LSTM block (STA) automatically fine-tunes the extracted features and assigns weight to the more critical frames in the video sequence by using an attention mechanism. Extensive experiments on four of the most popular datasets support our architecture. Finally, the results are compared with the state of the art, which shows the superiority of this approach.

A Novel Fault Diagnosis Approach Based on Correlation Analysis and Improved Two-Stream Neural Networks

A Novel Fault Diagnosis Approach Based on Correlation Analysis and Improved Two-Stream Neural Networks

Two-stream network with phase map for few-shot classification

As an important research direction of deep learning, few-shot classification mainly solves problems of classification when training samples are scarce. Under this circumstance, how to effectively use the information of existing samples has become the key to solve few-shot classification problems. We propose a method to fully mine existing samples information, which can be implemented in the training and testing stages. Specifically, we introduce a two-stream neural network as an embedding model to extract features of the original image and its phase map, and perform fine-grained adaptive feature fusion to them. The algorithm can learn and extract more fully-informed fusion features under the network. Besides, we also introduce Circle Loss in loss function, which has a non-linear decision boundary. Under the joint supervision of Soft-max Loss and Circle Loss, the distinction between different classes of features in embedding space increases. We conduct experiments on miniImageNet, tieredImageNet, and CIFAR-FS, respectively. The accuracy rate is increased by 0.99%, 0.96%, and 0.91%, which proves the effectiveness of our algorithm.

Artificial Intelligence of Things-assisted two-stream neural network for anomaly detection in surveillance Big Video Data

In the last few years, visual sensors are deployed almost everywhere, generating a massive amount of surveillance video data in smart cities that can be inspected intelligently to recognize anomalous events. In this work, we present an efficient and robust framework to recognize anomalies from surveillance Big Video Data (BVD) using Artificial Intelligence of Things (AIoT). Smart surveillance is an important application of AIoT and we propose a two-stream neural network in this direction. The first stream comprises instant anomaly detection that is functional over resource-constrained IoT devices, whereas second phase is a two-stream deep neural network allowing for detailed anomaly analysis, suited to be deployed as a cloud computing service. Firstly, a self-pruned fine-tuned lightweight convolutional neural network (CNN) classifies the ongoing events as normal or anomalous in an AIoT environment. Upon anomaly detection, the edge device alerts the concerned departments and transmits the anomalous frames to cloud analysis center for their detailed evaluation in the second phase. The cloud analysis center resorts to the proposed two-stream network, modeled from the integration of spatiotemporal and optical flow features through the sequential frames. Fused features flow through a bi-directional long short-term memory (BD-LSTM) layer, which classifies them into their respective anomaly classes, e.g., assault and abuse. We perform extensive experiments over benchmarks built on top of the UCF-Crime and RWF-2000 datasets to test the effectiveness of our framework. We report a 9.88% and 4.01% increase in accuracy when compared to state-of-the-art methods evaluated over the aforementioned datasets.

Violence Detection With Two-Stream Neural Network Based on C3D

In recent years, violence detection has gradually turned into an important research area in computer vision, and have proposed many models with high accuracy. However, the unsatisfactory generalization ability of these methods over different datasets. In this paper, the authors propose a violence detection method based on C3D two-stream network for spatiotemporal features. Firstly, the authors preprocess the video data of RGB stream and optical stream respectively. Secondly, the authors feed the data into two C3D networks to extract features from the RGB flow and the optical flow respectively. Third, the authors fuse the features extracted by the two networks to obtain a final prediction result. To testify the performance of the proposed model, four different datasets (two public datasets and two self-built datasets) are selected in this paper. The experimental results show that our model has good generalization ability compared to state-of-the-art methods, since it not only has good ability on large-scale datasets, but also performs well on small-scale datasets.

Highlight-aware two-stream network for single-image SVBRDF acquisition

This paper addresses the task of estimating spatially-varying reflectance (i.e., SVBRDF) from a single, casually captured image. Central to our method is a highlight-aware (HA) convolution operation and a two-stream neural network equipped with proper training losses. Our HA convolution, as a novel variant of standard (ST) convolution, directly modulates convolution kernels under the guidance of automatically learned masks representing potentially overexposed highlight regions. It helps to reduce the impact of strong specular highlights on diffuse components and at the same time, hallucinates plausible contents in saturated regions. Considering that variation of saturated pixels also contains important cues for inferring surface bumpiness and specular components, we design a two-stream network to extract features from two different branches stacked by HA convolutions and ST convolutions, respectively. These two groups of features are further fused in an attention-based manner to facilitate feature selection of each SVBRDF map. The whole network is trained end to end with a new perceptual adversarial loss which is particularly useful for enhancing the texture details. Such a design also allows the recovered material maps to be disentangled. We demonstrate through quantitative analysis and qualitative visualization that the proposed method is effective to recover clear SVBRDFs from a single casually captured image, and performs favorably against state-of-the-arts. Since we impose very few constraints on the capture process, even a non-expert user can create high-quality SVBRDFs that cater to many graphical applications.

Pedestrian's Intention Recognition, Fusion of Handcrafted Features in a Deep Learning Approach

The safety of vulnerable road users (VRU) is a major concern for both advanced driver assistance systems (ADAS) and autonomous vehicle manufacturers. To guarantee people safety on roads, autonomous vehicles must be able to detect the presence of pedestrians, track them, and predict their intention to cross the road. Most of the earlier work on pedestrian intention recognition focused on using either handcrafted features or an end-to-end deep learning approach. In this project, we investigate the impact of fusing handcrafted features with auto learned features by using a two-stream neural network architecture. Our results show that the combined approach improves the performance. Furthermore, the proposed method achieved very good results on the JAAD dataset. Depending on whether we considered the immediate frames before the crossing or only half a second before that point, we received prediction accuracy of 91%, and 84%, respectively.

Developmental Stage Classification of Embryos Using Two-Stream Neural Network with Linear-Chain Conditional Random Field.

The developmental process of embryos follows a monotonic order. An embryo can progressively cleave from one cell to multiple cells and finally transform to morula and blastocyst. For time-lapse videos of embryos, most existing developmental stage classification methods conduct per-frame predictions using an image frame at each time step. However, classification using only images suffers from overlapping between cells and imbalance between stages. Temporal information can be valuable in addressing this problem by capturing movements between neighboring frames. In this work, we propose a two-stream model for developmental stage classification. Unlike previous methods, our two-stream model accepts both temporal and image information. We develop a linear-chain conditional random field (CRF) on top of neural network features extracted from the temporal and image streams to make use of both modalities. The linear-chain CRF formulation enables tractable training of global sequential models over multiple frames while also making it possible to inject monotonic development order constraints into the learning process explicitly. We demonstrate our algorithm on two time-lapse embryo video datasets: i) mouse and ii) human embryo datasets. Our method achieves 98.1% and 80.6% for mouse and human embryo stage classification, respectively. Our approach will enable more pro-found clinical and biological studies and suggests a new direction for developmental stage classification by utilizing temporal information.

A Nonintrusive Load Identification Model Based on Time-Frequency Features Fusion

Nonintrusive load monitoring (NILM) plays a key role in the real-time electricity consumption monitoring of household appliances. However, it is difficult to realize high precision load identification by using a single waveform feature. Therefore, this article proposes a two-stream convolutional neural network based on current time-frequency feature fusion for nonintrusive load identification. First, a time series image coding method for current time-frequency multi-feature fusion is proposed. The method can extract the time domain and frequency domain features of the current timing signal effectively. Then, we present a two-stream neural network combining the gated recurrent unit (GRU) and a two-dimensional convolutional neural network (2D-CNN) to improve the load identification performance. Finally, the experimental results on the PLAID and IDOUC datasets show that the proposed model outperforms the state-of-the-art methods.

Fast sparse coding networks for anomaly detection in videos

The semi-supervised video anomaly detection assumes that only normal video clips are available for training. Therefore, the intuitive idea is either to learn a dictionary by sparse coding or to train encoding-decoding neural networks by minimizing the reconstruction errors. For the former, the optimization of sparse coefficients is extremely time-consuming. For the latter, this manner cannot guarantee that an abnormal data corresponds to a larger reconstruction error due to the strong generalization of neural networks. To remedy their weaknesses and leverage their strengths, we propose a Fast Sparse Coding Network (FSCN) based on High-level Features. First, we propose a two-stream neural network to extract Spatial-Temporal Fusion Features (STFF) in hidden layers. With the STFF at hand, we use a Fast Sparse Coding Network to build a normal dictionary. By leveraging the predictor to produce approximate sparse coefficients, our FSCN generates sparse coefficients within a forward pass, which is simple and computationally efficient. Compared with traditional sparse coding based methods, FSCN is hundreds of or even thousands of times faster at the test stage. Extensive experiments on benchmark datasets demonstrate that our method reaches the state-of-the-art level.11Code will be released at https://github.com/Roc-Ng/FSCN_AnomalyDetection.

Gait Recognition Method of Underground Coal Mine Personnel Based on Densely Connected Convolution Network and Stacked Convolutional Autoencoder

Biological recognition methods often use biological characteristics such as the human face, iris, fingerprint, and palm print; however, such images often become blurred under the limitation of the complex environment of the underground, which leads to low identification rates of underground coal mine personnel. A gait recognition method via similarity learning named Two-Stream neural network (TS-Net) is proposed based on a densely connected convolution network (DenseNet) and stacked convolutional autoencoder (SCAE). The mainstream network based on DenseNet is mainly used to learn the similarity of dynamic deep features containing spatiotemporal information in the gait pattern. The auxiliary stream network based on SCAE is used to learn the similarity of static invariant features containing physiological information. Moreover, a novel feature fusion method is adopted to achieve the fusion and representation of dynamic and static features. The extracted features are robust to angle, clothing, miner hats, waterproof shoes, and carrying conditions. The method was evaluated on the challenging CASIA-B gait dataset and the collected gait dataset of underground coal mine personnel (UCMP-GAIT). Experimental results show that the method is effective and feasible for the gait recognition of underground coal mine personnel. Besides, compared with other gait recognition methods, the recognition accuracy has been significantly improved.