Two-stream Convolutional Neural Network Research Articles

Bolt-Loosening Detection Using 1D and 2D Input Data Based on Two-Stream Convolutional Neural Networks.

At present, the detection accuracy of bolt-loosening diagnoses is still not high. In order to improve the detection accuracy, this paper proposes a fault diagnosis model based on the TSCNN model, which can simultaneously extract fault features from vibration signals and time-frequency images and can precisely detect the bolt-loosening states. In this paper, the LeNet-5 network is improved by adjusting the size and number of the convolution kernels, introducing the dropout operation, and building a two-dimensional convolutional neural network (2DCNN) model. Combining the advantages of a one-dimensional convolutional neural network (1DCNN) with wide first-layer kernels to suppress high-frequency noise, a two-stream convolutional neural network (TSCNN) is proposed based on 1D and 2D input data. The proposed model uses raw vibration signals and time-frequency images as input and automatically extracts sensitive features and representative information. Finally, the effectiveness and superiority of the proposed approach are verified by practical experiments that are carried out on a machine tool guideway. The experimental results show that the proposed approach can effectively achieve end-to-end bolt-loosening fault diagnoses, with an average recognition accuracy of 99.58%. In addition, the method can easily achieve over 93% accuracy when the SNR is over 0 dB without any denoising preprocessing. The results show that the proposed approach not only achieves high classification accuracy but also has good noise immunity.

Temporal grafter network: Rethinking LSTM for effective video recognition

Long short-term memory (LSTM) networks are widely used to handle temporal or sequential data, and have great potential for video recognition. Existing LSTM-based video recognition methods either insert LSTM modules at the end of 2D convolutional neural networks (CNNs), called global LSTM methods, or build networks solely by stacking multiple LSTM modules. Unfortunately, these LSTM-based video recognition methods are not competitive, compared to state-of-the-art 3D CNNs or two-stream CNNs. In order to fully explore the potential of LSTM, this paper rethinks its role in video recognition network architectures and proposes a novel Temporal Grafter Network (TGN). Specifically, we develop an efficient and effective variant of convolutional LSTM module, which is grafted between different stages of very deep 2D CNNs for temporal modeling and delivery. Our TGN can capture local motion patterns of varying scales inherent in feature maps from high to low resolutions, while attending to the spatial context information and modeling global temporal dependency across the whole video. The proposed TGN can capture and transmit temporal information throughout very deep 2D CNNs, overcoming the downsides of existing LSTM-based methods and being able to make full use of the potential of LSTM for effective video recognition and early action recognition. We perform extensive ablation study to verify the effectiveness of our proposed methods, and experiments on three widely used video benchmarks show that our methods can achieve performance matching or better than the state-of-the-arts.

Spatiotemporal image fusion using multiscale attention-aware two-stream convolutional neural networks

Spatiotemporal fusion (STF) is considered as a promising way to produce remote sensing images at fine scales in both space and time by blending two types of satellite images. The learning-based STF approaches with deep convolutional neural networks can provide the unified framework to address both gradual and abrupt changes. This paper intends to develop an enhanced learning-based STF using multiscale attention-aware two-stream convolutional neural networks (MACNN). With a coarse image at the prediction date and two pairs of coarse and fine images at other dates as inputs, it employs a multiscale module to characterize different sizes of objects and a spatial and channel attention module to emphasize important information in feature learning. Two experiments on real Landsat and MODIS images are conducted to demonstrate the effectiveness of the proposed MACNN and it outperforms four existing STF methods in both visual and quantitative.

Single Image Dehazing Based on Two-Stream Convolutional Neural Network

Objective The haze weather environment leads to the deterioration of the visual effect of the image, and it is difficult to carry out the work of the advanced vision task. Therefore, dehazing the haze image is an important step before the execution of the advanced vision task. Traditional dehazing algorithms achieve image dehazing by improving image brightness and contrast or constructing artificial priors such as color attenuation priors and dark channel priors, but the effect is unstable when dealing with complex scenes. In the method based on convolutional neural network, the image dehazing network of the encoding and decoding structure does not consider the difference before and after the dehazing image, and the image spatial information is lost in the encoding stage. In order to overcome these problems, this paper proposes a novel end-to-end two-stream convolutional neural network for single image dehazing. Method The network model is composed of a spatial information feature stream and a high-level semantic feature stream. The spatial information feature stream retains the detailed information of the dehazing image, and the high-level semantic feature stream extracts the multi-scale structural features of the dehazing image. A spatial information auxiliary module is designed between the feature streams. This module uses the attention mechanism to construct a unified expression of different types of information, and realizes the gradual restoration of the clear image with the semantic information auxiliary spatial information in the dehazing network. A parallel residual twicing module is proposed, which performs dehazing on the difference information of features at different stages to improve the model’s ability to discriminate haze images. Result The peak signal-to-noise ratio and structural similarity are used to quantitatively evaluate the similarity between the dehazing results of each algorithm and the original image. The structure similarity and peak signal-to-noise ratio of the method in this paper reached 0.852 and 17.557dB on the Hazerd dataset, which were higher than all comparison algorithms. On the SOTS dataset, the indicators are 0.955 and 27.348dB, which are sub-optimal results. In experiments with real haze images, this method can also achieve excellent visual restoration effects.

Multi-head attention-based two-stream EfficientNet for action recognition

Recent years have witnessed the popularity of using two-stream convolutional neural networks for action recognition. However, existing two-stream convolutional neural network-based action recognition approaches are incapable of distinguishing some roughly similar actions in videos such as sneezing and yawning. To solve this problem, we propose a Multi-head Attention-based Two-stream EfficientNet (MAT-EffNet) for action recognition, which can take advantage of the efficient feature extraction of EfficientNet. The proposed network consists of two streams (i.e., a spatial stream and a temporal stream), which first extract the spatial and temporal features from consecutive frames by using EfficientNet. Then, a multi-head attention mechanism is utilized on the two streams to capture the key action information from the extracted features. The final prediction is obtained via a late average fusion, which averages the softmax score of spatial and temporal streams. The proposed MAT-EffNet can focus on the key action information at different frames and compute the attention multiple times, in parallel, to distinguish similar actions. We test the proposed network on the UCF101, HMDB51 and Kinetics-400 datasets. Experimental results show that the MAT-EffNet outperforms other state-of-the-art approaches for action recognition.

Adoption of Artificial Intelligence Along with Gesture Interactive Robot in Musical Perception Education Based on Deep Learning Method

The present work aims to improve students’ interest in music teaching and promote modern teaching. A distributed application system of artificial intelligence gesture interactive robot is designed through deep learning technology and applied to music perception education. First, the user’s gesture instruction data is collected through the double channel convolution neural network (DCCNN). It uses the double-size convolution kernel to extract feature information in the image and collect the video frame’s gesture instruction. Secondly, a two-stream convolutional neural network (two-stream CNN) recognizes the collected gesture instruction data. The spatial and temporal information is extracted from RGB color mode (RGB) images and optical flow images and input into the two-stream CNN to fuse the prediction results of each network as the final detection result. Then, the distributed system used by the interactive robot is introduced. This structure can improve the stability of the interactive systems and reduce the requirements for local hardware performance. Finally, experiments are conducted to test the gesture command acquisition and recognition network, and the performance of the gesture interactive robot in practice. The results indicate that combining convolution kernels of [Formula: see text] and [Formula: see text] can increase the recognition accuracy of DCCNN to 98% and effectively collect gesture instruction data. The gesture recognition accuracy of two-stream CNN after training reaches 90%, higher than the mainstream dynamic gesture recognition algorithm trained with the same data set. Finally, the recognition test of gesture instructions is carried out on the gesture interactive robot reported here. The results show that the recognition accuracy of the gesture interactive robots is more than 90%, meeting the routine interaction needs. Therefore, the interactive gesture robot has good reliability and stability and is applicable to music perception teaching. The research reported here has guiding significance for establishing music teaching with multiple perception modes.

Spatial-temporal interaction learning based two-stream network for action recognition

Two-stream convolutional neural networks have been widely applied to action recognition. However, two-stream networks are usually adopted to capture spatial information and temporal information separately, which normally ignore the strong complementarity and correlation between spatial and temporal information in videos. To solve this problem, we propose a Spatial-Temporal Interaction Learning Two-stream network (STILT) for action recognition. Our proposed two-stream (i.e., a spatial stream and a temporal stream) network has a spatial–temporal interaction learning module, which uses an alternating co-attention mechanism between two streams to learn the correlation between spatial features and temporal features. The spatial–temporal interaction learning module allows the two streams to guide each other and then generates optimized spatial attention features and temporal attention features. Thus, the proposed network can establish the interactive connection between two streams, which efficiently exploits the attended spatial and temporal features to improve recognition accuracy. Experiments on three widely used datasets (i.e., UCF101, HMDB51 and Kinetics) show that the proposed network outperforms the state-of-the-art models in action recognition.

Video behavior recognition algorithm based on two-stream heterogeneous convolutional neural network

Abstract In video behavior recognition, making full use of the spatio-temporal information contained in video frame is the critical point to further improve the recognition accuracy. In our study, we propose a video behavior recognition algorithm based on two-stream heterogeneous network, so that we can further extract the spatio-temporal feature information in the video frame, and finally take full advantage of the spatio-temporal features in the video sequence. In view of the characteristics of RGB and optical flow images, this paper uses DenseNet121 and Inception-V4 to structure a two-stream network, to fully extract the spatio-temporal feature information in the video. In this paper, UCF101 dataset is used to evaluate the algorithm, and the recognition accuracy is 91.7%, which verified the reliability of the proposed algorithm.

A Two-Stream CNN With Simultaneous Detection and Segmentation for Robotic Grasping

The manipulating robots receive much attention by offering better services, where object grasping is still challenging especially under background interferences. In this article, a novel two-stream grasping convolutional neural network (CNN) with simultaneous detection and segmentation is proposed. The proposed method is cascaded by an improved simultaneous detection and segmentation network BlitzNet and a two-stream grasping CNN TsGNet. The improved BlitzNet introduces the channel-based attention mechanism, and achieves an improvement of detection accuracy and segmentation accuracy with the combination of the learning of multitask loss weightings and background suppression. Based on the obtained bounding box and the segmentation mask of the target object, the target object is separated from the background, and the corresponding depth map and grayscale map are sent to TsGNet. By adopting depthwise separable convolution and designed global deconvolution network, TsGNet achieves the best grasp detection with only a small amount of network parameters. This best grasp in the pixel coordinate system is converted to a desired 6-D pose for the robot, which drives the manipulator to execute grasping. The proposed method combines a grasping CNN with simultaneous detection and segmentation to achieve the best grasp with a good adaptability to background. With the Cornell grasping dataset, the image-wise accuracy and object-wise accuracy of the proposed TsGNet are 93.13% and 92.99%, respectively. The effectiveness of the proposed method is verified by the experiments.

Spatiotemporal Temperature Fusion Based on a Deep Convolutional Network

High-spatiotemporal-resolution land surface temperature (LST) images are essential in various fields of study. However, due to technical constraints, sensing systems have difficulty in providing LSTs with both high spatial and high temporal resolution. In this study, we propose a multi-scale spatiotemporal temperature-image fusion network (MSTTIFN) to generate high-spatial-resolution LST products. The MSTTIFN builds nonlinear mappings between the input Moderate Resolution Imaging Spectroradiometer (MODIS) LSTs and the out- put Landsat LSTs at the target date with two pairs of references and therefore enhances the resolution of time-series LSTs. We conduct experiments on the actual Landsat and MODIS data in two study areas (Beijing and Shandong) and compare our proposed MSTTIFN with four competing methods: the Spatial and Temporal Adaptive Reflectance Fusion Model, the Flexible Spatiotemporal Data Fusion Model, a two-stream convolutional neural network (StfNet), and a deep learning-based spatiotemporal temperature-fusion network. Results reveal that the MSTTIFN achieves the best and most stable performance.

Fake visual content detection using two-stream convolutional neural networks

Rapid progress in adversarial learning has enabled the generation of realistic-looking fake visual content. To distinguish between fake and real visual content, several detection techniques have been proposed. The performance of most of these techniques however drops off significantly if the test and the training data are sampled from different distributions. This motivates efforts towards improving the generalization of fake detectors. Since current fake content generation techniques do not accurately model the frequency spectrum of the natural images, we observe that the frequency spectrum of the fake visual data contains discriminative characteristics that can be used to detect fake content. We also observe that the information captured in the frequency spectrum is different from that of the spatial domain. Using these insights, we propose to complement frequency and spatial domain features using a two-stream convolutional neural network architecture called TwoStreamNet. We demonstrate the improved generalization of the proposed two-stream network to several unseen generation architectures, datasets, and techniques. The proposed detector has demonstrated significant performance improvement compared to the current state-of-the-art fake content detectors with the fusing of frequency and spatial domain streams also improving the generalization of the detector.

A Novel English Translation Model in Complex Environments Using Two-Stream Convolutional Neural Networks.

Although translation is an essential component of learning English, it does not receive the attention it merits in the modern English classroom. Teachers and students primarily emphasize listening, reading, and writing while neglecting the development of translation skills. The English test in China now reflects the fact that there are now very specific requirements for students' translation skills. As a result, we should emphasize developing students' translation skills when teaching them English. The following experimental data can be obtained following the study and experiment on the English translation simulation model based on the two-stream convolutional neural network: English vocabulary and grammar have passing and excellent rates of 90 and 57 percent, respectively, while reading has passing and excellent rates of 69 and 8 percent, respectively. The ability of students to translate into English has significantly improved after using the English translation simulation model based on the two-stream convolutional neural network.

Spatial-Temporal Information Aggregation and Cross-Modality Interactive Learning for RGB-D-Based Human Action Recognition

The RGB-D-based human action recognition is gaining increasing attention because the different modalities can provide complementary information. However, the recognition performance is still not satisfactory due to the limited ability to learn spatial-temporal feature and insufficient inter-model interaction. In this paper, we propose a novel approach for RGB-D human action recognition by aggregating spatial-temporal information and implementing cross-modality interactive learning. Firstly, a spatial-temporal information aggregation module (STIAM) is proposed to utilizes sample convolutional neural networks (CNNs) to aggregate the spatial-temporal information in entire RGB-D sequence into lightweight representations efficiently. This allows the model to extract richer spatial-temporal features with limited extra memory and computational cost. Secondly, a cross-modality interactive module (CMIM) is proposed to fully fuse the multi-modal complementary information. Moreover, a multi-modal interactive network (MMINet) is constructed for RGB-D-based action recognition by embeding the above two modules into the two-stream CNNs. In order to verify the universality of our approach, two backbones are deployed in the two-stream architecture, successively. Ablation experiments demonstrate that the proposed STIAM can bring significant improvement in recognizing actions. CMIM can further play the advantages of complementary features of multiple modalities. Extensive experiments on NTU RGB+D 60, NTU RGB+D 120 and PKU-MMD datasets proved the effectiveness of the proposed approach. The proposed approach achieves an accuracy of 94.3% and 96.5% for cross-subject and cross-view on NTU RGB+D 60, 91.7% and 92.6% for cross-subject and cross-setup on NTU RGB+D 120, 93.6% and 94.2% for cross-subject and cross-view on PKU-MMD datasets, which are the state-of-the-art performance. Further analysis denotes that our approach has advantages in recognizing subtle actions.

Research on remote sensing image classification method using two-stream convolutional neural network

Research on remote sensing image classification method using two-stream convolutional neural network

Temporal and Spectral Feature Learning With Two-Stream Convolutional Neural Networks for Appliance Recognition in NILM

Non-intrusive load monitoring (NILM) can monitor the operating state and energy consumption of appliances without deploying sub-meters and is promising to be widely used in residential communities. With the rapid increase of electric loads in amount and type, constructing representative load signatures and designing effective classification models are becoming increasingly crucial for NILM. In this paper, temporal and spectral load signatures that preserve sufficient information are constructed from the monitored energy data. The fusion of these two types of load signatures can provide rich distinguishing features for improving the performance of appliance recognition in NILM. Benefiting from the development of deep learning, this study proposes the two-stream convolutional neural networks (TSCNN) to extract the features from the two types of load signatures and perform classification. Furthermore, this study introduces the affinity propagation clustering strategy to mitigate the negative impact of intra-class variety mainly caused by multi-state loads in appliance recognition. The experimental results on public NILM datasets demonstrate that the proposed method outperforms most of the existing methods based on the voltage-current trajectory or recurrence graph in the recognition accuracy of submetered and aggregated measurements.

Human Instance Segmentation Based on Two-Stream Convolutional Neural Network

Human Instance Segmentation Based on Two-Stream Convolutional Neural Network

A novel motion recognition method based on improved two-stream convolutional neural network and sparse feature fusion

Motion recognition is a hot topic in the field of computer vision. It is a challenging task. Motion recognition analysis is closely related to the network input, network structure and feature fusion. Due to the noise in the video, traditional methods cannot better obtain the feature information resulting in the problem of inaccurate motion recognition. Feature selection directly affects the efficiency of recognition, and there are still many problems to be solved in the multi-level feature fusion process. In this paper, we propose a novel motion recognition method based on an improved two-stream convolutional neural network and sparse feature fusion. In the low-rank space, because sparse features can effectively capture the information of motion objects in the video, meanwhile, we supplement the network input data, in view of the lack of information interaction in the network, we fuse the high-level semantic information and low-level detail information to recognize the motions by introducing attention mechanism, which makes the performance of the two-stream convolutional neural network have more advantages. Experimental results on UCF101 and HMDB51 data sets show that the proposed method can effectively improve the performance of motion recognition.

Research on remote sensing image classification method using two-stream convolutional neural network

Research on remote sensing image classification method using two-stream convolutional neural network

The Application of Deep Convolution Neural Network in Volleyball Video Behavior Recognition

The purpose is to solve the problems of solid subjectivity in the traditional manual analysis of volleyball game videos and the traditional Human Behavior Recognition (HBR) algorithm, such as excessive calculation, high hardware conditions, and poor long-stream video modeling ability. Firstly, this paper expounds on the relevant theories. Secondly, a fusion Convolutional Neural Network (CNN)-based HBR model is implemented by combining two-stream CNN (TSCNN), Three-Dimensional (3D) CNN, and Long Short-Term Memory (LSTM) network. Notably, the LSTM has an excellent long-term Dynamic Information Extraction (DIE) ability. Finally, the public dataset is selected to verify the model’s volleyball-game-video-oriented HBR performance. The experimental results corroborate that: (1) The parameters of the proposed fusion-CNN-based HBR model are determined as follows: the number of video segments is three, the average method is used for feature fusion, and then the HBR accuracy is the highest when the fusion ratio of spatial feature map and temporal feature map is 4:6, and the learning rate is 0.0014. (2) The average HBR accuracy of the proposed fusion-CNN-based HBR model on three different datasets is 4%, 2.7%, and 3% higher than that of other popular networks, respectively. The improvement effect of the model is remarkable, and it is suitable for the study of Human Behavior Analysis (HBA) in volleyball match videos. Finally, the proposed HBR model can provide more accurate results for volleyball video-based HBR, which is significant in promoting the rapid development of volleyball sports.

SDCAFNet: A Deep Convolutional Neural Network for Land-Cover Semantic Segmentation With the Fusion of PolSAR and Optical Images

Due to the different imaging mechanisms between optical and polarimetric synthetic aperture radar (PolSAR) images, determining how to effectively use such complementary information has become an interesting and challenging problem. Convolutional neural networks (CNNs) and other deep neural networks have achieved good experimental results in remote sensing land-cover semantic segmentation. However, the CNN convolution structure can extract only the features within the receptive field in the spatial dimension without focusing on the relationship between multiple channels; therefore, it is impossible to realize fusion and complementarity between multiple channels. In this paper, we propose a novel spatial dense channel attention fusion network (SDCAFNet), which takes optical and PolSAR images as different inputs and completes feature fusion and semantic segmentation within a neural network. First, SDCAFNet uses a two-stream siamese CNN network to realize the preliminary feature coding of optical and PolSAR images. Then, a spatial dense channel attention module (SDCAM) is proposed. The channel activation values obtained at different positions are combined in the spatial dense matrix, which can describe the attention in the feature fusion process. Finally, we introduce the fused features into the symmetric skip-connection decoder composed of multiple symmetric decoder blocks to realize end-to-end land-cover semantic segmentation. Experimental results show that SDCAFNet can effectively learn the correlation between optical and PolSAR channels and has a better segmentation accuracy than other methods.