Two-stream Network Research Articles

A two-stream network with complementary feature fusion for pest image classification

Pests are diverse and the available datasets often contain an uneven number of examples for different pests (a.k.a., the long-tail distribution). This poses a great challenge to learning-based classification methods, especially deep networks, and often leads to degraded performance, especially for the minority (tail) classes. This paper presents a deep learning integration architecture based on decoupling training and fusion learning, which integrates different models with complementary performance on pest datasets with a long-tailed distribution to improve the overall classification performance of pests. A deep neural network is designed that fuses two complementary deep learning models at the feature level, which consists of a convolution neural network (ConvNeXt) and a Swin Transformer model for decoupling training. Experiments are conducted using three datasets (d0, insect, and IP102), and evaluation on accuracy, recall, and F1-Score is reported. For the large-scale pest dataset with long-tailed distribution IP102, the accuracy achieves 76.1%, which outperforms the state-of-the-art methods. In addition, the accuracy for d0 and insect datasets are 98.5% and 92.3%, respectively.

A Cross-Modality Person Re-Identification Method Based on Joint Middle Modality and Representation Learning

Modality differences and intra-class differences have been hot research problems in the field of cross-modality person re-identification currently. In this paper, we propose a cross-modality person re-identification method based on joint middle modality and representation learning. To reduce the modality differences, a middle modal generator is used to map different modal images to a unified feature space to generate middle modality images. A two-stream network with parameter sharing is used to extract the combined features of the original image and the middle modality image. In addition, a multi-granularity pooling strategy combining global features and local features is used to improve the representation learning capability of the model and further reduce the modality differences. To reduce the intra-class differences, the model is further optimized by combining distribution consistency loss, label smoothing cross-entropy loss, and hetero-center triplet loss to reduce the intra-class distance and accelerate the model convergence. In this paper, we use the publicly available datasets RegDB and SYSU-MM01 for validation. The results show that the proposed approach in this paper reaches 68.11% mAP in All Search mode for the SYSU-MM01 dataset and 86.54% mAP in VtI mode for the RegDB dataset, with a performance improvement of 3.29% and 3.29%, respectively, which demonstrate the effectiveness of the proposed method.

Blood flow characterization in nailfold capillary using optical flow-assisted two-stream network and spatial-temporal image

The blood flow velocity in the nailfold capillary is an important indicator of the status of microcirculation. The conventional manual processing method is both laborious and prone to human artifacts. A feasible way to solve this problem is to use machine learning to assist in image processing and diagnosis. Inspired by the Two-Stream Convolutional Networks, this study proposes an optical flow-assisted two-stream network to segment nailfold blood vessels. Firstly, we use U-Net as the spatial flow network and the dense optical flow as the temporal stream. The results show that the optical flow information can effectively improve the integrity of the segmentation of blood vessels. The overall accuracy is 94.01 %, the Dice score is 0.8099, the IoU score is 0.6806, and the VOE score is 0.3194. Secondly, The flow velocity of the segmented blood vessel is determined by constructing the spatial-temporal (ST) image. The blood flow velocity evaluated is consistent with the typical blood flow speed reported. This study proposes a novel two-stream network for blood vessel segmentation of nailfold capillary images. Combined with ST image and line detection method, it provides an effective workflow for measuring the blood flow velocity of nailfold capillaries.

Hierarchical supervisions with two-stream network for Deepfake detection

Recently, the quality of face generation and manipulation has reached impressive levels, making it difficult even for humans to distinguish real and fake faces. At the same time, methods to distinguish fake faces from reals came out, such as Deepfake detection. However, the task of Deepfake detection remains challenging, especially the low-quality fake images circulating on the Internet and the diversity of face generation methods. In this work, we propose a new Deepfake detection network that could effectively distinguish both high-quality and low-quality faces generated by various generation methods. First, we design a two-stream framework that incorporates a regular spatial stream and a frequency stream to handle the low-quality problem since we find that the frequency domain artifacts of low-quality images will be preserved. Second, we introduce hierarchical supervisions in a coarse-to-fine manner, which consists of a coarse binary classification branch to classify reals and fakes and a five-category classification branch to classify reals and four different types of fakes. Extensive experiments have proved the effectiveness of our framework on several widely used datasets.

KNN-Based Machine Learning Classifier Used on Deep Learned Spatial Motion Features for Human Action Recognition.

Human action recognition is an essential process in surveillance video analysis, which is used to understand the behavior of people to ensure safety. Most of the existing methods for HAR use computationally heavy networks such as 3D CNN and two-stream networks. To alleviate the challenges in the implementation and training of 3D deep learning networks, which have more parameters, a customized lightweight directed acyclic graph-based residual 2D CNN with fewer parameters was designed from scratch and named HARNet. A novel pipeline for the construction of spatial motion data from raw video input is presented for the latent representation learning of human actions. The constructed input is fed to the network for simultaneous operation over spatial and motion information in a single stream, and the latent representation learned at the fully connected layer is extracted and fed to the conventional machine learning classifiers for action recognition. The proposed work was empirically verified, and the experimental results were compared with those for existing methods. The results show that the proposed method outperforms state-of-the-art (SOTA) methods with a percentage improvement of 2.75% on UCF101, 10.94% on HMDB51, and 0.18% on the KTH dataset.

Counting piglet suckling events using deep learning-based action density estimation

Analysis of piglet suckling behaviour is important for the evaluation of piglet nutrient ingestion, health, welfare, and affinity with the sow. In this study, an action density estimation network (ADEN) was proposed for counting the events of piglet suckling followed by automated analysis of suckling behaviour. ADEN is a two-stream network primarily composed of 1) a network stream that processes video images with a higher frame rate (faster stream) and 2) a network stream that processes video images with a lower frame rate (slower stream). Each stream consists of a ResNet-50 with five convolutional stages. A multi-stage attention connection (MSAC), composed of four Spatial-Temporal-Channel (STC) multi-attention structures, is proposed to bridge Faster Stream and Slower Stream and capture discriminative features. The output attention features from each faster stream stage are laterally fused into the corresponding slower stream stage in a concatenating manner. Following this, the features from the last convolutional stages in the Slow stream and Fast stream are fused using concatenation and are then decoded by using three convolutional layers. The last convolutional layer outputs a heatmap on the action density of piglet suckling behaviour. Finally, the number of suckling events is predicted by integrating all the pixel values in the heatmap. Experimental and comparative tests were conducted to validate the effectiveness of the proposed ADEN with a training dataset and a test dataset from 14 pig pens. The 507 video clips (126,750 images for 7 h) from the 1-9th pens were selected as training datasets. The 143 video clips (35,750 images for 2 h) from the 10-13th pens were selected as short-term test datasets. One untrimmed video (162,000 images for 9 h) from the 14th pen was used to ultimately evaluate the action density estimation performance of the ADEN. ADEN was compared with seven approaches and its superiority was demonstrated with an r = 0.938, an RMSE = 1.080, and a MAE = 0.967 in short video clips and r = 0.982, MAE = 0.161, and RMSE = 0.563 in the untrimmed long video. The ADEN proved it feasible to predict the number of suckling events by using action density estimation.

Enhanced features in image manipulation detection

Image editing technology leads to tampering traces hidden in images hard to be identified, which results in great challenges for image manipulation detection. Therefore, it is necessary to enhance the characteristic response of tampering traces. In this paper, a two-stream network is exploited to extract tampering features from both semantic level and noise level through end-to-end training. At the input end of the network, a multi-scale high-frequency constrained convolution kernel module is proposed to extract multi-channel and multi-scale noise fingerprint information, which enhances the tampering traces. Moreover, we leverage multi-task learning to realize the classification of tampering techniques, the detection and segmentation of tampering regions. The tampered segmentation mask attention is introduced as a novel constraint to be fed back to the detection branch, which further enhances the tampering characteristics and improves the detection performance. Experiments on COVER, CASIA and NIST16 datasets prove that the proposed framework outperforms existing methods quantitatively and qualitatively. The proposed method effectively enhances the characteristics of tampered regions. Moreover, the proposed method has high robustness on JPEG recompression and image resize.

SFANet: A Spectrum-Aware Feature Augmentation Network for Visible-Infrared Person Reidentification.

Visible-Infrared person reidentification (VI-ReID) is a challenging matching problem due to large modality variations between visible and infrared images. Existing approaches usually bridge the modality gap with only feature-level constraints, ignoring pixel-level variations. Some methods employ a generative adversarial network (GAN) to generate style-consistent images, but it destroys the structure information and incurs a considerable level of noise. In this article, we explicitly consider these challenges and formulate a novel spectrum-aware feature augmentation network named SFANet for cross-modality matching problem. Specifically, we put forward to employ grayscale-spectrum images to fully replace RGB images for feature learning. Learning with the grayscale-spectrum images, our model can apparently reduce modality discrepancy and detect inner structure relations across the different modalities, making it robust to color variations. At feature level, we improve the conventional two-stream network by balancing the number of specific and sharable convolutional blocks, which preserve the spatial structure information of features. Additionally, a bidirectional tri-constrained top-push ranking loss (BTTR) is embedded in the proposed network to improve the discriminability, which efficiently further boosts the matching accuracy. Meanwhile, we further introduce an effective dual-linear with batch normalization identification (ID) embedding method to model the identity-specific information and assist BTTR loss in magnitude stabilizing. On SYSU-MM01 and RegDB datasets, we conducted extensively experiments to demonstrate that our proposed framework contributes indispensably and achieves a very competitive VI-ReID performance.

Fast cancer metastasis location based on dual magnification hard example mining network in whole-slide images

Breast cancer has become the most common form of cancer among women. In recent years, deep learning has shown great potential in aiding the diagnosis of pathological images, particularly through the use of convolutional neural networks for locating lymph node metastasis under gigapixel whole slide images (WSIs). However, the massive size of these images at the highest magnification introduces redundant computation during the inference process. Additionally, the diversity of biological textures and structures within WSIs can cause confusion for classifiers, particularly in identifying hard examples. As a result, the trade-off between accuracy and efficiency remains a critical issue for whole-slide image metastasis localization. In this paper, we propose a novel two-stream network that takes a pair of low- and high-magnification image patches as input for identifying hard examples during the training phase. Specifically, our framework focuses on samples where the outputs of the two magnification networks are dissimilar. We adopt a dual magnification hard mining loss to re-weight the ambiguous samples. To more efficiently locate tumor metastasis cells in whole slide images, the two stream networks are decomposed into a cascaded network during the inference phase. The low magnification WSIs scanned by the low-mag network generate a coarse probability map, and the suspicious areas in the map are refined by the high-mag network. Finally, we evaluate our fast location dual magnification hard example mining network on the Camelyon16 breast cancer whole-slide image dataset. Experiments demonstrate that our proposed method achieves a 0.871 FROC score with a faster inference time, and our high magnification network also achieves a 0.88 FROC score.

A cross-modal deep metric learning model for disease diagnosis based on chest x-ray images.

The emergence of unknown diseases is often with few or no samples available. Zero-shot learning and few-shot learning have promising applications in medical image analysis. In this paper, we propose a Cross-Modal Deep Metric Learning Generalized Zero-Shot Learning (CM-DML-GZSL) model. The proposed network consists of a visual feature extractor, a fixed semantic feature extractor, and a deep regression module. The network belongs to a two-stream network for multiple modalities. In a multi-label setting, each sample contains a small number of positive labels and a large number of negative labels on average. This positive-negative imbalance dominates the optimization procedure and may prevent the establishment of an effective correspondence between visual features and semantic vectors during training, resulting in a low degree of accuracy. A novel weighted focused Euclidean distance metric loss is introduced in this regard. This loss not only can dynamically increase the weight of hard samples and decrease the weight of simple samples, but it can also promote the connection between samples and semantic vectors corresponding to their positive labels, which helps mitigate bias in predicting unseen classes in the generalized zero-shot learning setting. The weighted focused Euclidean distance metric loss function can dynamically adjust sample weights, enabling zero-shot multi-label learning for chest X-ray diagnosis, as experimental results on large publicly available datasets demonstrate.

THAT-Net: Two-layer hidden state aggregation based two-stream network for traffic accident prediction

Traffic accident prediction based on dashboard cameras is of great significance for ensuring the guaranteed-safety of self-driving systems and reducing the occurrence of accidents. Due to complexity of traffic scenarios and wide range of object's motion, early prediction of accidents remains a challenge. However, the existing methods process all traffic objects and all video frames indiscriminately, thus leading to poor prediction performance. To address this issue, we propose a novel traffic accident prediction framework, namely two-layer hidden state aggregation based two-stream network (THAT-Net). The proposed method first fuses the spatial and temporal flow to capture complementary spatio-temporal information, which filters out irrelevant objects in traffic scenes to reduce the influence of the object's motion state. Furthermore, a two-layer hidden state aggregation structure is designed to reintegrate the hidden state weights of gated recurrent units. It captures contextual information through frame-level and segment-level aggregation and decreases the influence of irrelevant frames to reduce the complexity of traffic scenes. Experiments based on two real-world datasets show the state-of-the-art performance of THAT-Net. Our proposed method achieves the highest accuracy by predicting accidents 0.48 sec to 2.8 sec earlier compared to the baseline methods in more challenging situations. Our code is available at: https://github.com/redeyezt/THAT-Net.

Multi-View Human Action Recognition Using Skeleton Based-FineKNN with Extraneous Frame Scrapping Technique.

Human action recognition (HAR) is one of the most active research topics in the field of computer vision. Even though this area is well-researched, HAR algorithms such as 3D Convolution Neural Networks (CNN), Two-stream Networks, and CNN-LSTM (Long Short-Term Memory) suffer from highly complex models. These algorithms involve a huge number of weights adjustments during the training phase, and as a consequence, require high-end configuration machines for real-time HAR applications. Therefore, this paper presents an extraneous frame scrapping technique that employs 2D skeleton features with a Fine-KNN classifier-based HAR system to overcome the dimensionality problems.To illustrate the efficacy of our proposed method, two contemporary datasets i.e., Multi-Camera Action Dataset (MCAD) and INRIA Xmas Motion Acquisition Sequences (IXMAS) dataset was used in experiment. We used the OpenPose technique to extract the 2D information, The proposed method was compared with CNN-LSTM, and other State of the art methods. Results obtained confirm the potential of our technique. The proposed OpenPose-FineKNN with Extraneous Frame Scrapping Technique achieved an accuracy of 89.75% on MCAD dataset and 90.97% on IXMAS dataset better than existing technique.

Exploring complementarity of global and local information for effective lip reading

Lip reading aims at recognizing texts from a talking face without audio information. Recently, some works have focused on how to effectively extract the spatial information and temporal information. We introduce an innovative two-stream network to make full use of the complementarity of global spatial information and local spatial information. The global spatial information is directly generated by the global stream. Furthermore, we design a patches selection module in the local stream to conveniently select the critical local information using attention mechanism. Then, the fusion features of the two streams and the global features are fed into the temporal module to further explore the temporal clues. To guide the selection of the local information from the fused features and to make the global stream and local stream learn from each other, we design a global information guide loss and a mutual learning loss, respectively. Finally, extensive experiments on both LRW and CAS-VSR-W1K datasets demonstrate the superiority of our two-stream work.

Deep interactive image segmentation based on region and Boundary-click guidance

In interactive image segmentation, the target object of interest can be extracted based on the guidance of user interactions. One of the main goals in this task is to reduce the user interaction burden and ensure satisfactory segmentation with as few interactions as possible. Thanks to the development of deep learning technology, neural network-based interactive approaches have significantly improved the segmentation performance through powerful feature representation. Only limited point (click) interaction is required for the user to complete the segmentation. This paper mainly follows the deep learning-based interactive segmentation methods and explores more efficient interaction strategies and effective segmentation models. We further simplify user interaction to two clicks, where the first click is utilized to select the target region and the other aims to determine the target boundary. Based on the region and boundary clicks, an interactive two-stream network structure is naturally derived to learn the region and boundary features of interest. Furthermore, we also construct an information diffusion module to better propagate the region and boundary-click labels, which helps to enhance the similarity within the region and the discrimination between boundaries. Vast experiments on the popular GrabCut, Berkeley, DAVIS, MS COCO and SBD datasets verified the effectiveness of the proposed method.

A Two-Stream Deep Imaging Method for Multifrequency Capacitively Coupled Electrical Resistance Tomography

In this work, a novel deep imaging method is proposed for multifrequency capacitively coupled electrical resistance tomography (MFCCERT). A two-stream network consisting of a low-frequency stream and a high-frequency stream is developed according to the frequency characteristics of the interested impedance. Meanwhile, a cross-stream information intersection approach, which combines hyper-dense connection and gated channel transformation (GCT), is proposed to fuse the complementary information in the multifrequency impedance measurements. The multifrequency measurements of CCERT in the frequency range of 0.5–5 MHz are divided into the low-frequency band and the high-frequency band, which are taken as the inputs of the two streams of the network, respectively. With the proposed cross-stream information intersection approach, the useful features of the impedance in the same frequency band and the features of the impedance from the two frequency bands are fused. Experiments were carried out with the 12-electrode CCERT sensor to obtain the multifrequency impedance measurements. Both simulation and experimental data were used to test the developed two-stream network. Imaging results indicate that the proposed deep imaging method is effective. Compared with the single-stream Unet, the developed network has better information fusion capability and image reconstruction performance.

A Progressive Difference Method for Capturing Visual Tempos on Action Recognition

Visual tempos show the dynamics of action instances, characterizing the diversity of the actions, such as walking slowly and running quickly. To facilitate action recognition, it is essential to capture visual tempos. To this end, previous methods sample raw videos at multiple frame rates or integrate multi-scale temporal features. These methods inevitably introduce two-stream networks or feature-level pyramid structures, leading to expensive computation. In this work, we propose a progressive difference method to capture visual tempos for efficient action recognition, by computing coarse-to-fine motion information within a small neighborhood around temporal frames. Specifically, the uniform sampling method is first applied to each video, and then first-order temporal differences around each frame are calculated to describe local motions. On the basis of differences, further computing the variations of differences, namely second-order differences, can gradually capture fine-grained spatiotemporal features and characterize the areas where the motion cues are more prominent. On one hand, multi-order motion differences can be combined with raw input to describe the diversity of the actions. On the other hand, the variations of first-order differences information can be used to activate first-order salient motion regions, thereby facilitating the discrimination of finer-grained actions. Our method can be combined with existing backbones in a plug-and-play manner. Extensive experiments are conducted on several video benchmarks, including Kinetics400, HMDB51, UCF101, UAV-Human, Something-Something V1 and V2. We also give detailed analysis and qualitative experiments to demonstrate the effectiveness of our method.

Adversarial pan-sharpening attacks for object detection in remote sensing

Pan-sharpening, as one of the most commonly used techniques in remote sensing systems, aims to fuse texture-rich PAN images and multi-spectral MS images to obtain texture-rich MS images. With the development of deep learning, CNN based Pan-sharpening methods have received more and more attention in recent years. Since Pan-sharpening technique can integrate the complementary information of Pan and MS images, researchers usually apply object detectors on these pan-sharpened images to achieve reliable detection results. However, recent studies have shown that Deep Learning-based object detection methods are vulnerable to adversarial examples, i.e., adding imperceptible noise to clean images can fool well-trained deep neural networks. It is interesting to combine the pan-sharpening technique with adversarial examples to attack object detectors in remote sensing. In this paper, we propose a framework to generate adversarial pan-sharpened images. Specifically, we propose a two-stream network to generate the pan-sharpened images, and then utilize the shape loss and label loss to perform the attack task. To guarantee the quality of pan-sharpened images, a perceptual loss is utilized to balance spectral preservation and attacking performance. Experimental results demonstrate that the proposed method can generate effective adversarial pan-sharpened images that maintain a high success rate for white-box attacks and achieve transferability for black-box attacks.

Target Classification in Unattended Ground Sensors With a Two-Stream Convolutional Network

Field motion target classification is the classification of ground motion targets such as personnel, wheeled, and tracked vehicles. The common classification methods are to use a single signal such as acoustic or seismic signal as input, which extracts less in-depth information than multimodal. In order to improve the accuracy of field target classification and recognition, an acoustic–seismic multimodal fusion network model with two-stream networks, i.e., WaveNet based on the raw audio and LMNet based on logmel spectrogram, is proposed. This article first introduces asymmetric convolution to extract time–frequency information separately and designs a temporal attention module to enable the network to make full use of the relevant temporal information in different channels. Then, the proposed network is used to extract and fuse the depth features of the four-channel acoustic signal and the single-channel seismic signal to obtain the final target classification results. Meanwhile, a data enhancement scheme is explored in order to avoid the possible overfitting caused by the limited training data. Notably, the model achieves 86.07%, 92.56%, and 98.08% classification accuracy on the seismic, acoustic, and acoustic–seismic datasets, respectively, and the classification accuracy of the multimodal model is significantly higher than that of the unimodal model. The ablation study shows that the classification accuracy of the framework is improved by 0.62–6.83% compared to the pure nets. Compared with existing deep-learning models, the best result of multimodal model demonstrates a relative improvement of 2.05%.

Enhancing the Generalization for Text Classification through Fusion of Backward Features.

Generalization has always been a keyword in deep learning. Pretrained models and domain adaptation technology have received widespread attention in solving the problem of generalization. They are all focused on finding features in data to improve the generalization ability and to prevent overfitting. Although they have achieved good results in various tasks, those models are unstable when classifying a sentence whose label is positive but still contains negative phrases. In this article, we analyzed the attention heat map of the benchmarks and found that previous models pay more attention to the phrase rather than to the semantic information of the whole sentence. Moreover, we proposed a method to scatter the attention away from opposite sentiment words to avoid a one-sided judgment. We designed a two-stream network and stacked the gradient reversal layer and feature projection layer within the auxiliary network. The gradient reversal layer can reverse the gradient of features in the training stage so that the parameters are optimized following the reversed gradient in the backpropagation stage. We utilized an auxiliary network to extract the backward features and then fed them into the main network to merge them with normal features extracted by the main network. We applied this method to the three baselines of TextCNN, BERT, and RoBERTa using sentiment analysis and sarcasm detection datasets. The results show that our method can improve the sentiment analysis datasets by 0.5% and the sarcasm detection datasets by 2.1%.

Multiscale Feature Fusion for Skin Lesion Classification.

Skin cancer has a high mortality rate, and early detection can greatly reduce patient mortality. Convolutional neural network (CNN) has been widely applied in the field of computer-aided diagnosis. To improve the ability of convolutional neural networks to accurately classify skin lesions, we propose a multiscale feature fusion model for skin lesion classification. We use a two-stream network, which are a densely connected network (DenseNet-121) and improved visual geometry group network (VGG-16). In the feature fusion module, we construct multireceptive fields to obtain multiscale pathological information and use generalized mean pooling (GeM pooling) to reduce the spatial dimensionality of lesion features. Finally, we built and tested a system with the developed skin lesion classification model. The experiments were performed on the dataset ISIC2018, which can achieve a good classification performance with a test accuracy of 91.24% and macroaverages of 95%.