Two-stream Network Research Articles

A simulated two-stream network via multilevel distillation of reviewed features and decoupled logits for video action recognition

A simulated two-stream network via multilevel distillation of reviewed features and decoupled logits for video action recognition

Faster-slow network fused with enhanced fine-grained features for action recognition

Two-stream methods, which separate human actions and backgrounds into temporal and spatial streams visually, have shown promising results in action recognition datasets. However, prior researches emphasize motion modeling but overlook the robust correlation between motion features and spatial information, causing restriction of the model’s ability to recognize behaviors entailing occlusions or rapid changes. Therefore, we introduce Faster-slow, an improved framework for frame-level motion features. It introduces a Behavioural Feature Enhancement (BFE) module based on a novel two-stream network with different temporal resolutions. BFE consists of two components: MM, which incorporates motion-aware attention to capture dependencies between adjacent frames; STC, which enhances spatio-temporal and channel information to generate optimized features. Overall, BFE facilitates the extraction of finer-grained motion information, while ensuring a stable fusion of information across both streams. We evaluate the Faster-slow on the Atomic Visual Actions dataset, and the Faster-AVA dataset constructed in this paper, yielding promising experimental results.

MLFF-Net: a multi-model late feature fusion network for skin disease classification

Early diagnosis is paramount to preventing skin diseases and reducing mortality, given their global prevalence. Visual detection by experts using dermoscopy images has become the gold standard for detecting skin cancer. However, a significant challenge in skin cancer detection and classification lies in the similarity of appearance among skin disease lesions and the complexity of dermoscopic images. In response, we developed multi-model late feature fusion network (MLFF-Net), a multi-model late feature fusion network tailored for skin disease detection. Our approach begins with image pre-processing techniques to enhance image quality. We then employ a two-stream network comprising an enhanced densely linked network (DenseNet-121) and a vision transformer (ViTb16). We leverage shallow and deep feature fusion, late fusion, and an attention module to enhance the model’s feature extraction efficiency. The subsequent feature fusion module constructs multi-receptive fields to capture disease information across various scales and uses generalized mean pooling (GeM) pooling to reduce the spatial dimensions of lesion characteristics. Finally, we implement and test our skin lesion categorization model, demonstrating its effectiveness. Despite the combination, convolutional neural network (CNN) outperforms ViT approaches, with our model enhancing the accuracy of the best model by 6.1%.

BCMFIFuse: A Bilateral Cross-Modal Feature Interaction-Based Network for Infrared and Visible Image Fusion

The main purpose of infrared and visible image fusion is to produce a fusion image that incorporates less redundant information while incorporating more complementary information, thereby facilitating subsequent high-level visual tasks. However, obtaining complementary information from different modalities of images is a challenge. Existing fusion methods often consider only relevance and neglect the complementarity of different modalities’ features, leading to the loss of some cross-modal complementary information. To enhance complementary information, it is believed that more comprehensive cross-modal interactions should be provided. Therefore, a fusion network for infrared and visible fusion is proposed, which is based on bilateral cross-feature interaction, termed BCMFIFuse. To obtain features in images of different modalities, we devise a two-stream network. During the feature extraction, a cross-modal feature correction block (CMFC) is introduced, which calibrates the current modality features by leveraging feature correlations from different modalities in both spatial and channel dimensions. Then, a feature fusion block (FFB) is employed to effectively integrate cross-modal information. The FFB aims to explore and integrate the most discriminative features from the infrared and visible image, enabling long-range contextual interactions to enhance global cross-modal features. In addition, to extract more comprehensive multi-scale features, we develop a hybrid pyramid dilated convolution block (HPDCB). Comprehensive experiments on different datasets reveal that our method performs excellently in qualitative, quantitative, and object detection evaluations.

An unsupervised video anomaly detection method via Optical Flow decomposition and Spatio-Temporal feature learning

The purpose of this paper is to present an unsupervised video anomaly detection method using Optical Flow decomposition and Spatio-Temporal feature learning (OFST). This method employs a combination of optical flow reconstruction and video frame prediction to achieve satisfactory results. The proposed OFST framework is composed of two modules: the Multi-Granularity Memory-augmented Autoencoder with Optical Flow Decomposition (MG-MemAE-OFD) and a Two-Stream Network based on Spatio-Temporal feature learning (TSN-ST). The MG-MemAE-OFD module is composed of three functional blocks: optical flow decomposition, autoencoder, and multi-granularity memory networks. The optical flow decomposition block is used to extract the main motion information of objects in optical flow, and the granularity memory network is utilized to memorize normal patterns and improve the quality of the reconstructions. To predict video frames, we introduce a two-stream network based on spatiotemporal feature learning (TSN-ST), which adopts parallel standard Transformer blocks and a temporal block to learn spatiotemporal features from video frames and optical flows. The OFST combines these two modules so that the prediction error of abnormal samples is further increased due to the larger reconstruction error. In contrast, the normal samples obtain a lower reconstruction error and prediction error. Therefore, the anomaly detection capability of the method is greatly enhanced. Our proposed model was evaluated on public datasets. Specifically, in terms of the area under the curve (AUC), our model achieved an accuracy of 85.74% on the Ped1 dataset, 99.62% on the Ped2 dataset, 93.89% on the Avenue dataset, and 76.0% on the ShanghaiTech Dataset. Our experimental results show an average improvement of 1.2% compared to the current state-of-the-art.

Visible thermal person re-identification via multi-branch modality residual complementary learning

Visible-thermal Person Re-identification (VT Re-ID) is a challenging task in all-weather surveillance system. Existing methods concentrate on extracting the modality-shared features, ignoring the discriminative inter-modality complementary features. To tackle this issue, we propose a multi-branch modality residual complementary learning method which consists of the modality residual complementary learning (MRCL) module and the multi-branch feature learning (MBFL) module. The MRCL module can be easily integrated into existing CNN baselines and drive the network to focus on both intra-modality and inter-modality information. On one hand, we adopt the basic two-stream network to obtain the intra-modality features, on the other hand, we capture the inter-modality complementary features within the residual image obtained by cross-modality correlation saliency erasing operation. To handle the intra-modality variations, we employ the MBFL module to capture local spatial features and local channel features, then integrate them with global features to achieve part-to-part and high-level semantic information matching. Finally, the discriminability and robustness of the ultimate representations are enhanced by multi-branch constraint loss learning. Extensive experiments on RegDB and SYSU-MM01 datasets demonstrate the superiority of our proposed method compared with state-of-the-art methods.

Invisible gas detection: An RGB-thermal cross attention network and a new benchmark

The widespread use of various chemical gases in industrial processes necessitates effective measures to prevent their leakage during transportation and storage, given their high toxicity. Thermal infrared-based computer vision detection techniques provide a straightforward approach to identify gas leakage areas. However, the development of high-quality algorithms has been challenging due to the low texture in thermal images and the lack of open-source datasets. In this paper, we present the RGB-Thermal Cross Attention Network (RT-CAN), which employs an RGB-assisted two-stream network architecture to integrate texture information from RGB images and gas area information from thermal images. Additionally, to facilitate the research of invisible gas detection, we introduce Gas-DB, an extensive open-source gas detection database including about 1.3K well-annotated RGB-thermal images with eight variant collection scenes. Experimental results demonstrate that our method successfully leverages the advantages of both modalities, achieving state-of-the-art (SOTA) performance among RGB-thermal methods, surpassing single-stream SOTA models in terms of accuracy, Intersection of Union (IoU), and F2 metrics by 4.86%, 5.65%, and 4.88%, respectively. The code and data can be found at https://github.com/logic112358/RT-CAN.

Enhancing Human Activity Recognition through Integrated Multimodal Analysis: A Focus on RGB Imaging, Skeletal Tracking, and Pose Estimation.

Human activity recognition (HAR) is pivotal in advancing applications ranging from healthcare monitoring to interactive gaming. Traditional HAR systems, primarily relying on single data sources, face limitations in capturing the full spectrum of human activities. This study introduces a comprehensive approach to HAR by integrating two critical modalities: RGB imaging and advanced pose estimation features. Our methodology leverages the strengths of each modality to overcome the drawbacks of unimodal systems, providing a richer and more accurate representation of activities. We propose a two-stream network that processes skeletal and RGB data in parallel, enhanced by pose estimation techniques for refined feature extraction. The integration of these modalities is facilitated through advanced fusion algorithms, significantly improving recognition accuracy. Extensive experiments conducted on the UTD multimodal human action dataset (UTD MHAD) demonstrate that the proposed approach exceeds the performance of existing state-of-the-art algorithms, yielding improved outcomes. This study not only sets a new benchmark for HAR systems but also highlights the importance of feature engineering in capturing the complexity of human movements and the integration of optimal features. Our findings pave the way for more sophisticated, reliable, and applicable HAR systems in real-world scenarios.

A two-stream-network based steganalysis network: TSNet

The three-stage framework consisting of preprocessing, feature extraction and classification is the most commonly used structure in traditional deep learning based steganalysis networks (SNs). Preprocessing, as the core part of the three-stage framework, typically adopts fixed high-pass filters to generate filter residuals, and exploits a feature enhancement module following high-pass filters to improve signal-to-noise ratio of the stego signal, thus strengthening the expression of steganographic features. However, fixed high-pass filters can only provide filter residuals through the linear relationships, rather than complex dependencies automatically learned, among local pixels, which suggests that fixed high-pass filters lack powerful ability of extracting comprehensive steganographic features from the beginning of SNs. Inspired by the motivation, a two-stream-network based SN called TSNet is proposed to extract comprehensive steganographic features from two different perspectives, namely automatic learning and fixed filtering. The discrete cosine transform attention mechanism (DCTAM)-based fusion module (DCTAM-FM) with considering the steganalysis properties is proposed to fuse two streams and enhance channel representation using DCTAM, which is conductive to improving the detection accuracy. Two streams aim to explore different types of residuals from two perspectives, but inevitably lead to a considerable number of parameters and high computational complexity. To this end, the feature extraction stage contains lightweight depthwise separable convolution blocks and the linearly increasing style of the number of channels. Combining the two-stream network, DCTAM-FM as well as the lightweight feature extraction stage, TSNet obtains deep composite features, which can fully express the comprehensive steganographic features, and thus brings a high detection accuracy. Experimental results from three benchmark datasets demonstrate that TSNet outperforms current state-of-the-art methods in detection accuracy and has an acceptable computational complexity.

Spectral Decomposition and Transformation for Cross-domain Few-shot Learning

Cross-domain few-shot Learning (CDFSL) is proposed to first pre-train deep models on a source domain dataset where sufficient data is available, and then generalize models to target domains to learn from only limited data. However, the gap between the source and target domains greatly hampers the generalization and target-domain few-shot finetuning. To address this problem, we analyze the domain gap from the aspect of frequency-domain analysis. We find the domain gap could be reflected by the compositions of source-domain spectra, and the lack of compositions in the source datasets limits the generalization. Therefore, we aim to expand the coverage of spectra composition in the source datasets to help the source domain cover a larger range of possible target-domain information, to mitigate the domain gap. To achieve this goal, we propose the Spectral Decomposition and Transformation (SDT) method, which first randomly decomposes the spectrogram of the source datasets into orthogonal bases, and then randomly samples different coordinates in the space formed by these bases. We integrate the above process into a data augmentation module, and further design a two-stream network to handle augmented images and original images respectively. Experimental results show that our method achieves state-of-the-art performance in the CDFSL benchmark dataset.

A two-stream sign language recognition network based on keyframe extraction method

Sign language recognition (SLR) tasks are typically performed on a large number of continuous frames of sign language videos, making it challenging to utilize information efficiently and accurately from such a vast amount of data. Firstly, to efficiently utilize the information, it is necessary to select key information from numerous video frames to summarize the video. Therefore, this paper proposes a new method for key frame extraction that efficiently summarizes video information and addresses the issues of key frame clustering and data imbalance present in previous methods. Secondly, for more accurate information utilization, it is essential to learn the correct sign language expression features from a large dataset. Hence, we establish the Full-hand two-stream network to focus on the most crucial hand features in sign language expressions. The Full stream employs attention blocks to extract deep-level information and establishes temporal dependencies using global attention module. The Hand stream utilizes hand attention to focus on hand-specific feature information. Ultimately, our approach achieves state-of-the-art results on the CSL–500 Dataset and competitive results on the LSA64 Dataset. This paper validates the effectiveness of keyframe extraction methods and two-stream networks on the AUTSL dataset.

Integrating instance-level knowledge to see the unseen: A two-stream network for video object segmentation

Existing matching-based video object segmentation (VOS) approaches carry inherent limitations in segmenting pixels that have never appeared in the previous frames (i.e., unseen pixels). In this paper, we introduce a Two-Stream Network (TSN), which addresses this issue by distinguishing between seen and unseen pixels softly and processes them with two streams. Particularly, a pixel division module is devised to generate a routing map, distinguishing between seen and unseen pixels. Guided by the routing map, TSN integrates instance-level knowledge from an instance stream and pixel-level information from a pixel stream explicitly, generating the final segmentation result. The soft partitioning strategy allows for flexibility and adaptability in the fusion process. Additionally, the compact instance stream encodes and leverages instance-level knowledge, resulting in improved segmentation accuracy of the unseen pixels. Extensive experiments demonstrate the effectiveness of our proposed TSN, and we also report state-of-the-art performance on public VOS benchmarks.

A Remaining Useful Life Prediction Method of Mechanical Equipment Based on Particle Swarm Optimization-Convolutional Neural Network-Bidirectional Long Short-Term Memory

In industry, forecast prediction and health management (PHM) is used to improve system reliability and efficiency. In PHM, remaining useful life (RUL) prediction plays a key role in preventing machine failures and reducing operating costs, especially for reliability requirements such as critical components in aviation as well as for costly equipment. With the development of deep learning techniques, many RUL prediction methods employ convolutional neural network (CNN) and long short-term memory (LSTM) networks and demonstrate superior performance. In this paper, a novel two-stream network based on a bidirectional long short-term memory neural network (BiLSTM) is proposed to establish a two-stage residual life prediction model for mechanical devices using CNN as the feature extractor and BiLSTM as the timing processor, and finally, a particle swarm optimization (PSO) algorithm is used to adjust and optimize the network structural parameters for the initial data. Under the condition of lack of professional knowledge, the adaptive extraction of the features of the data accumulated by the enterprise and the effective processing of a large amount of timing data are achieved. Comparing the prediction results with other models through examples, it shows that the model established in this paper significantly improves the accuracy and efficiency of equipment remaining life prediction.

ATST-Net: A method to identify early symptoms in the upper and lower extremities of PD

Bradykinesia, a core symptom of motor disorders in Parkinson's disease (PD), is a major criterion for screening early PD patients in clinical practice. Currently, many studies have proposed automatic assessment schemes for bradykinesia in PD. However, existing schemes suffer from problems such as dependence on professional equipment, single evaluation tasks, difficulty in obtaining samples and low accuracy. This paper proposes a manual feature extraction- and neural network-based method to evaluate bradykinesia, effectively solving the problem of a small sample size. This method can automatically assess finger tapping (FT), hand movement (HM), toe tapping (TT) and bilateral foot sensitivity tasks (LA) through a unified model. Data were obtained from 120 individuals, including 93 patients with Parkinson's disease and 27 age- and sex-matched normal controls (NCs). Manual feature extraction and Attention Time Series Two-stream Networks (ATST-Net) were used for classification. Accuracy rates of 0.844, 0.819, 0.728, and 0.768 were achieved for FT, HM, TT, and LA, respectively. To our knowledge, this study is the first to simultaneously evaluate the upper and lower limbs using a unified model that has significant advantages in both model training and transfer learning.

CTSN: Predicting cloth deformation for skeleton-based characters with a two-stream skinning network

We present a novel learning method using a two-stream network to predict cloth deformation for skeleton-based characters. The characters processed in our approach are not limited to humans, and can be other targets with skeleton-based representations such as fish or pets. We use a novel network architecture which consists of skeleton-based and mesh-based residual networks to learn the coarse features and wrinkle features forming the overall residual from the template cloth mesh. Our network may be used to predict the deformation for loose or tight-fitting clothing. The memory footprint of our network is low, thereby resulting in reduced computational requirements. In practice, a prediction for a single cloth mesh for a skeleton-based character takes about 7 ms on an nVidia GeForce RTX 3090 GPU. Compared to prior methods, our network can generate finer deformation results with details and wrinkles.

AF-FTTSnet: An end-to-end two-stream convolutional neural network for online quality monitoring of robotic welding

Online welding quality monitoring (WQM) is crucial for intelligent welding, and deep learning approaches considering spatiotemporal features for WQM tasks show great potential. However, one of the important challenges for existing approaches is to balance the spatiotemporal representation learning capability and computational efficiency, which makes it challenging to adapt welding processes with complex and drastic molten pool dynamic behavior. This paper proposes a novel approach for WQM using molten pool visual sensing and deep learning considering spatiotemporal features, the proposed deep learning network called attention fusion based frame-temporality two-stream network (AF-FTTSnet). Firstly, a passive vision sensor is used to acquire continuous dynamic molten pool images. Meanwhile, temporal difference images are computed to provide novel features and temporal representations. Then, a two-stream feature extraction module is designed to concurrently extract rich spatiotemporal features from molten pool images and temporal difference images. Finally, an attention fusion module with the ability to automatically identify and weight the most relevant features is designed to achieve optimal fusion of the two-stream features. The shop welding experimental results indicate that the proposed AF-FTTSnet model can effectively and robustly recognize five typical welding states during helium arc welding, with an accuracy of 99.26%. This model has been demonstrated to exhibit significant performance improvements compared to mainstream temporal sequence models. Available: https://github.com/Just199806/TSCNN/tree/master.

Deep Fusion Module for Video Action Recognition

In video action recognition, effective spatiotemporal modeling is crucial. However, traditional two-stream methods face challenges in integrating spatial information from RGB images and temporary information from optical flow without long-range temporal modelling. To address these limitations, we propose the Deep Fusion Module (DFM), which focuses on the deep fusion of spatial and temporal information and consists of two components. First, we propose an Attention Fusion Module (AFM) to effectively fuse the shallow features obtained from a two-stream network, thereby facilitating the integration of spatial and temporal information. Next, we incorporate a SpatioTemporal Module (STM), comprising a ConvGRU and a 1×1 convolution, to model long-range temporal dependency and fuse spatial-temporal features. Experiments on the UCF101 dataset show that our method achieves 96.5% accuracy, outperforming baseline two-stream models by 0.3%.

Mask guided two-stream network for end-to-end few-shot action recognition

For few-shot video action recognition, it is essential to extract and align features from different videos. However, these operations can be complicated and unreliable due to the complexity of the video scene and the limitations of existing alignment algorithms. To enhance the saliency of the action-related features, we introduce segmentation mask frame sequences as prior information and devise a two-stream feature fusion module to fuse the multimodal features. Furthermore, we propose a self-attention-based temporal alignment module to predict the optimal alignment matrix between the features of samples in the query and support sets. This module avoids solving additional optimization problems in computing the alignment matrix, thus reducing the difficulty of the model for end-to-end learning. Our approach achieves competitive performance on four public datasets. We also experimentally validate the effectiveness of the proposed modules.

A regularized constrained two-stream convolution augmented Transformer for aircraft engine remaining useful life prediction

Remaining Useful Life (RUL) prediction is of great significance for maintaining the reliability and safety of industrial equipment. To address the challenges faced by existing methods in simultaneously extracting local and global degradation information from monitoring data. This paper proposes a Two-Stream Convolution Augmented Transformer (TACT) model based on L2 regularization constraint. Specifically, we design the parallel multi-scale Convolution Neural Network (CNN) and Transformer module to combine the local modeling ability of CNN and the global modeling ability of Transformer to improve the overall architecture of RUL prediction model. Moreover, the two-stream network based on the parallel structure also realizes the synchronous extraction of different time steps and sensor features in the sequence. Then, in the process of model training, the prediction reliability constraint is fused, the delay prediction constraint term is introduced, and the L2 regularization loss function is constructed. Finally, extensive experiments on the commercial modular aero-propulsion system simulation (C-MAPSS) show that our model provides competitive performance in terms of Root-Mean-Square Error (RMSE) and Score metrics. Compared to the state-of-the-art method based on Recurrent Neural Network (RNN) or CNN and its variants, Score is reduced by at least 2.71% and RMSE by at least 3.13%. Compared to the Transformer-based improved method, the Score is decreased by at least 4.54% and the RMSE is decreased by at least 2.78%. The effectiveness of the proposed method is demonstrated.

Two-Stream Network for Korean Natural Language Understanding

This study pioneers a dual-stream network architecture tailored for Korean Natural Language Understanding (NLU), focusing on enhancing comprehension by distinct processing of syntactic and semantic aspects. The hypothesis is that this bifurcation can lead to a more nuanced and accurate understanding of the Korean language, which often presents unique syntactic and semantic challenges not fully addressed by generalized models. The validation of this novel architecture employs the Korean Natural Language Inference (koNLI) and Korean Semantic Textual Similarity (koSTS) datasets. By evaluating the model's performance on these datasets, the study aims to determine its efficacy in accurately parsing and interpreting Korean text's syntactic structure and semantic meaning. Preliminary results from this research are promising. They indicate that the dual-stream approach significantly enhances the model's capability to understand and interpret complex Korean sentences. This improvement is crucial in NLU, especially for language-specific applications. The implications of this study are far-reaching. The methodology and findings could pave the way for more sophisticated NLU applications tailored to the Korean language, such as advanced sentiment analysis, nuanced text summarization, and more effective conversational AI systems. Moreover, this research contributes significantly to the broader field of NLU by underscoring the importance and efficacy of developing language-specific models, moving beyond the one-size-fits-all approach of general language models. Thus, this study is a testament to the potential of specialized approaches in language understanding technologies.