Dual-stream Network Research Articles

Integrating category-related key regions with a dual-stream network for remote sensing scene classification

Remote sensing image scene classification has made great progress with deep learning. Due to complex backgrounds and the large number of objects with inhomogeneous sizes, remote sensing image scene classification still remains challenging. In fact, only a few parts of regions are expected to be representative of the scene. In this paper, we propose a dual-stream framework for remote sensing classification with the help of category-related regions. Swin Transformer-based global feature extractor is used to produce coarse classification results. A category-related key region localization module is developed to extract the representative region from the whole image. The ResNet50 is employed as a local feature extractor to mine critical features from prominent regions. Finally, the classification results is obtained with the shared weighted classify head on global-local features. Extensive experiments show that our method provides a promising approach for key region locations and performs best on three public remote sensing datasets.

S3-Net: A Self-Supervised Dual-Stream Network for Radiology Report Generation.

Intelligent medicine is eager to automatically generate radiology reports to ease the tedious work of radiologists. Previous researches mainly focused on the text generation with encoder-decoder structure, while CNN networks for visual features ignored the long-range dependencies correlated with textual information. Besides, few studies exploit cross-modal mappings to promote radiology report generation. To alleviate the above problems, we propose a novel end-to-end radiology report generation model dubbed Self-Supervised dual-Stream Network (S3-Net). Specifically, a Dual-Stream Visual Feature Extractor (DSVFE) composed of ResNet and SwinTransformer is proposed to capture more abundant and effective visual features, where the former focuses on local response and the latter explores long-range dependencies. Then, we introduced the Fusion Alignment Module (FAM) to fuse the dual-stream visual features and facilitate alignment between visual features and text features. Furthermore, the Self-Supervised Learning with Mask(SSLM) is introduced to further enhance the visual feature representation ability. Experimental results on two mainstream radiology reporting datasets (IU X-ray and MIMIC-CXR) show that our proposed approach outperforms previous models in terms of language generation metrics.

Two-Stream Network for Korean Natural Language Understanding

This study pioneers a dual-stream network architecture tailored for Korean Natural Language Understanding (NLU), focusing on enhancing comprehension by distinct processing of syntactic and semantic aspects. The hypothesis is that this bifurcation can lead to a more nuanced and accurate understanding of the Korean language, which often presents unique syntactic and semantic challenges not fully addressed by generalized models. The validation of this novel architecture employs the Korean Natural Language Inference (koNLI) and Korean Semantic Textual Similarity (koSTS) datasets. By evaluating the model's performance on these datasets, the study aims to determine its efficacy in accurately parsing and interpreting Korean text's syntactic structure and semantic meaning. Preliminary results from this research are promising. They indicate that the dual-stream approach significantly enhances the model's capability to understand and interpret complex Korean sentences. This improvement is crucial in NLU, especially for language-specific applications. The implications of this study are far-reaching. The methodology and findings could pave the way for more sophisticated NLU applications tailored to the Korean language, such as advanced sentiment analysis, nuanced text summarization, and more effective conversational AI systems. Moreover, this research contributes significantly to the broader field of NLU by underscoring the importance and efficacy of developing language-specific models, moving beyond the one-size-fits-all approach of general language models. Thus, this study is a testament to the potential of specialized approaches in language understanding technologies.

Air Traffic Controller Fatigue Detection by Applying a Dual-Stream Convolutional Neural Network to the Fusion of Radiotelephony and Facial Data

The role of air traffic controllers is to direct and manage highly dynamic flights. Their work requires both efficiency and accuracy. Previous studies have shown that fatigue in air traffic controllers can impair their work ability and even threaten flight safety, which makes it necessary to carry out research into how to optimally detect fatigue in controllers. Compared with single-modality fatigue detection methods, multi-modal detection methods can fully utilize the complementarity between diverse types of information. Considering the negative impacts of contact-based fatigue detection methods on the work performed by air traffic controllers, this paper proposes a novel AF dual-stream convolutional neural network (CNN) architecture that simultaneously extracts controller radio telephony fatigue features and facial fatigue features and performs two-class feature-fusion discrimination. This study designed two independent convolutional processes for facial images and radio telephony data and performed feature-level fusion of the extracted radio telephony and facial image features in the fully connected layer, with the fused features transmitted to the classifier for fatigue state discrimination. The experimental results show that the detection accuracy of radio telephony features under a single modality was 62.88%, the detection accuracy of facial images was 96.0%, and the detection accuracy of the proposed AF dual-stream CNN network architecture reached 98.03% and also converged faster. In summary, a dual-stream network architecture based on facial data and radio telephony data is proposed for fatigue detection that is faster and more accurate than the other methods assessed in this study.

Deep video inpainting detection and localization based on ConvNeXt dual-stream network

Currently, deep learning-based video inpainting algorithms can fill in a specified video region with visually plausible content, usually leaving imperceptible traces. Since deep video inpainting methods can be used to maliciously manipulate video content, there is an urgent need for an effective method to detect and localize deep video inpainting. In this paper, we propose a dual-stream video inpainting detection network, which includes a ConvNeXt dual-stream encoder and a multi-scale feature cross-fusion decoder. To further explore the spatial and temporal traces left by deep inpainting, we extract motion residuals and enhance them using 3D convolution and SRM filtering. Furthermore, we extract filtered residuals using LoG and Laplacian filtering. These residuals are then entered into ConvNeXt, thereby learning discriminative inpainting features. To enhance detection accuracy, we design a top-down pyramid decoder that aims at deep fusion of multi-dimensional multi-scale features to fully exploit the information of different dimensions and levels in detail. We created two datasets containing state-of-the-art video inpainting algorithms and conducted various experiments to evaluate our approach. The experimental results demonstrate that our approach outperforms existing methods and attains a competitive performance despite encountering unseen inpainting algorithms.

Dual-stream feature fusion network for person re-identification

Person re-identification (Re-ID) has made significant progress in recent years. However, it still faces numerous challenges in real scenarios. Although researchers have proposed various solutions, the issue of similar clothing colors remains an obstacle in improving the performance of person re-identification. To solve this issue, we propose a dual-stream feature fusion network (DSFF-Net) to extract discriminative features from pedestrian images in two color spaces. Specifically, a dual-stream network is designed to extract RGB global features, grayscale global features, and local features of pedestrian images to increase the richness of pedestrian representations. A channel attention module is designed to direct the network to focus on the salient features of pedestrians. An embedding mixed pooling is designed, which integrates the outputs of global average pooling (GAP) and global max pooling (GMP) to obtain more discriminative global features. Besides, it can also remove redundant information and increase the discrimination of pedestrian representations. A fine-grained local feature embedding fusion operation is designed to obtain more discriminative local features by embedding and fusing fine-grained local features of RGB and grayscale pedestrian images. Since the final pedestrian representation fuses both global features and fine-grained discriminative features in RGB and grayscale spaces, DSFF-Net increases the discriminative capability and richness of pedestrian representations. Moreover, we conduct extensive experiments on three datasets, Market-1501 DukeMTMC-Reid, and CUHK03, and our method achieves the Rank-1/mAP of 95.9%/89.1%, 89.0%/79.2%, and 81.2%/78.7%, respectively. Experimental results show that the performance of DSFF-Net is better than those of most of the state-of-the-art person Re-ID methods.

Video process detection for space electrostatic suspension material experiment in China’s Space Station

The purpose of video process detection is to identify key processes of interest and associated semantic categories, which is an important computer vision task for localizing key processes in space science experiments. To detect experimental processes in space electrostatic suspension material experiment on China’s Space Station, we propose a Slow-and-Fast Dual-Stream Network. It extracts multi-scale temporal semantic information by designing a Slow-and-Fast Temporal Stream structure and effectively detects the experimental processes with various temporal scales. In addition, the Two Path Temporal Pyramid Network (TP2N) enhances the bi-direction flow among multi-scale features on the temporal dimension to alleviate boundary sensitivity. The validation experiments on our established dataset demonstrate the effectiveness of our proposed method with 62.43% mAP, outperforming the related state-of-the-art temporal action localization methods with nearly 7% mAP gain. This paper is the first application of deep learning in space electrostatic suspension material experiments on China’s Space Station and further provides important technical support for the identification of key processes in space science experiments.

TSRNet: A Dual-stream Network for Refining 3D Tooth Segmentation.

The field of 3D tooth segmentation has made considerable advances thanks to deep learning, but challenges remain with coarse segmentation boundaries and prediction errors. In this paper, we introduce a novel learnable method to refine coarse results obtained from existing 3D tooth segmentation algorithms. The refinement framework features a dual-stream network called TSRNet (Tooth Segmentation Refinement Network) to rectify defective boundary and distance maps extracted from the coarse segmentation. The boundary map provides explicit boundary information, while the distance map provides gradient information in the form of the shortest geodesic distance between the vertex and the segmentation boundary. Following well-designed rules, the two refined maps are utilized to move the coarse tooth boundaries toward their correct positions through an iterative refinement process. The two-stage refinement method is validated on both 3D tooth and segmentation benchmark datasets. Extensive experiments demonstrate that our method significantly improves upon the coarse results from baseline methods and achieves state-of-the-art performance. (Code will be publicly available at https://github.com/bibi547/TSRNet.git.).

Lightweight Dual Stream Network With Knowledge Distillation for RGB-D Scene Parsing

Lightweight Dual Stream Network With Knowledge Distillation for RGB-D Scene Parsing

Edge-Guided Dual-Stream Network for Plastic Greenhouse Extraction from Remote Sensing Image

Edge-Guided Dual-Stream Network for Plastic Greenhouse Extraction from Remote Sensing Image

A learnable support selection scheme for boosting few-shot segmentation

Upon reevaluating recent studies of Few-Shot Segmentation (FSS), a key observation is that the random selection of support images is not always the optimal. In this situation, the support images cannot provide the useful guidance for the segmentation task. Therefore, we argue that a similarity-based support selection scheme, which selects support images according to the similarity between the query and candidate support images, is able to boost the performance of an FSS network. To this end, we propose a Siamese Support Selection Network (SSSN) which can be end-to-end trained along with an FSS network. We also leverage the joint utilization of a Convolutional Neural Network (CNN) and a Transformer network on top of a new feature fusion method to further improve the performance. To our knowledge, none of the similarity-based support selection scheme and the dual-stream network have been utilized for the FSS task before. Experimental results show that our FSS approach outperforms its counterparts on three data sets. In particular, the SSSN is able to boost the performance of an FSS network. We believe that these promising results should be due to the ability of the SSSN to select the top similar support images, which are useful for the FSS task.11Code is available at https://indtlab.github.io/projects/SSSN.

WaterFormer: A coupled transformer and CNN network for waterbody detection in optical remotely-sensed imagery

As one of the most significant components of the ecosystem, waterbody needs to be highly monitored at different spatial and temporal scales. Nevertheless, waterbody variations in shape, size, and reflectivity, complicated and varied types of land covers, and environmental scene diversity, present colossal challenges in achieving accurate waterbody detection (WD). In this paper, we propose a novel network coupled with the Transformer and convolutional neural network (CNN), termed WaterFormer, to automatically, efficiently, and accurately delineate waterbodies from optical high-resolution remotely sensed (HR-RS) images. This network mainly includes a dual-stream CNN, a cross-level Vision Transformer, a light-weight attention module, and a sub-pixel up-sampling module. First, the dual-stream network abstracts waterbody features at multi-views and different levels. Then, to exploit the long-range dependencies between low-level spatial information and high-order semantic features, the cross-level Vision Transformer is embedded into the dual-stream, aiming at improving WD accuracy. Afterwards, the light-weight attention module is adopted to provide semantically strong feature abstractions by enhancing discrimination neurons, and the sub-pixel up-sampling module is employed to further generate high-resolution and high-quality class-specific representations. Quantitative and qualitative evaluations demonstrated that the WaterFormer provided a promising means for detecting waterbody areas in satellite images under complex scene conditions. Moreover, comparative analyses with the state-of-the-art (SOTA) alternatives, e.g., MSFENet, MSAFNet, and BiSeNet, also verified the generalization and superiority of the WaterFormer in WD tasks. The assessment results exhibited that the WaterFormer gained an average accuracy of 97.24%, average precision of 94.59%, average recall of 91.95%, average F1-score of 93.24%, and average Kappa index of 0.9133, respectively. Additionally, we presented an open-access HR satellite imagery waterbody dataset, a mesoscale dataset with high-quality and high-precision waterbody annotation to facilitate future research in this field. The dataset has been released at https://github.com/NJdeuK/WD_Dataset.

CDS-Net: Cooperative dual-stream network for image manipulation detection

To accurately locate manipulated regions, many existing approaches employ a dual-stream framework to extract a wide range of manipulation clues, including local noise, edge artifacts, and global inconsistency. However, these approaches treat each stream in isolation and fail to consider the complementary and mutual guidance ability between the streams. Moreover, we notice the use of vanilla vision transformers in previous approaches can result in disruptions of object semantics, causing incomplete predictions. To address these challenges, we introduce the cooperative dual-stream network (CDS-Net) comprising an RGB Stream and a Noise Stream. In the Noise Stream, we propose a K-means Transformer (KT) that encourages both inter-patch and intra-patch information transmission to mitigate the semantic fragmentation phenomenon caused by patch partitioning. Additionally, we introduce a novel Feature Interaction Block (FIB) that explicitly encourages cross-stream collaboration at each encoding stage. Comprehensive experiments on publicly available datasets demonstrate the effectiveness and robustness of CDS-Net.

Multichannel InSAR elevation reconstruction method based on dual-stream network

This paper presents a multichannel InSAR elevation reconstruction method based on deep learning, where a dual-stream network consisting of an elevation reconstruction stream and a boundary detection stream, named as ERSBDS, is built to reconstruct elevation maps for observed terrains, from multiple interferograms. First, the elevation reconstruction stream adopts a modified DeepLabV3+ architecture, in which the Xception network is replaced by the lightweight network called MobileNetV3 in the encoder for not only reducing the network parameters but also maintaining the performance of the network, and then a spatial attention module is added to the encoding and decoding path to enhance the network's attention to the spatial information of feature maps. Second, the boundary detection stream is mainly composed of residual blocks, which can detect the boundary information of observed terrains and merge it into the elevation reconstruction stream to improve the accuracy of elevation reconstruction for observed scenes. Finally, a suitable data set is constructed to enable the trained network to accurately reconstruct elevation maps for observed scenes. The experiments for multichannel InSAR elevation reconstruction for observed scenes demonstrate the effectiveness of the proposed method, and show the advantages of this method in the accuracy and efficiency of elevation reconstruction, compared with some of the most commonly used methods.

DSCA: A dual-stream network with cross-attention on whole-slide image pyramids for cancer prognosis

DSCA: A dual-stream network with cross-attention on whole-slide image pyramids for cancer prognosis

TS-DSANN: Texture and shape focused dual-stream attention neural network for benign-malignant diagnosis of thyroid nodules in ultrasound images.

Recently, accurate diagnosis of thyroid nodules has played a critical role in precision medicine and healthcare system management. Due to complicated changes in ultrasound features of texture, and similar visual appearance of benign-malignant nodules, the identification of cancerous thyroid lesions from a given ultrasound image still faces challenges for even experienced radiologists. Learning-based computer-aided diagnosis (CAD) systems have accordingly attracted more and more attention recently. However, little research is committed to developing a deep learning-based CAD system that has greater conformity with radiologists' diagnostic decision-making. In this study, we devise a texture and shape focused dual-stream attention neural network, dubbed TS-DSANN. Specifically, in the texture focused stream, we utilize the ImageNet pre-trained ResNet34 to guide the network to recognize texture-related nodule attributes. Meanwhile, in the shape focused stream, in addition to using ResNet34 backbone, jointly learning from scratch with the contour obtained by contour detection module to enhance the extraction of shape features. Afterward, we employ a concatenation operation to aggregate the abovementioned two stream networks for capturing richer and more representative features. Finally, we further utilize an online class activation mapping mechanism to assist the dual-stream network in generating a localization heatmap to obtain more visualization attention to the nodule from the whole image, and supervise classifier's attention in decision-making. Experimental results conducted on the two-center thyroid nodule ultrasound datasets verify that our proposed method has improved the classification performance, superior to the state-of-the-art methods.

Frequency and content dual stream network for image dehazing

Image dehazing can improve image clarity and visual effect, which plays a pivotal role in many computer vision tasks. Existing dehazing methods are mostly based on a single feature stream and tend to ignore the low-frequency characteristics of haze. In this paper, we propose a dual stream network for image dehazing. To enhance the edge information and texture detail of the image, we construct a frequency stream based on attention octave convolution. We decompose the features into high and low-frequency branches in the frequency stream to obtain different structural information. By adding a residual channel attention block, the attention octave convolution can extract frequency features more efficiently and effectively. Due to the lower resolution of low-frequency features in the frequency stream, the frequency stream features alone are insufficient for recovering the overall content of the image. Therefore, a content stream was added to compensate for the information lost in the frequency stream. By fusing the outputs of two feature streams, the network achieves an enhanced dehazing performance. The results show that our method is superior to other state-of-the-art algorithms in quantitative evaluation and visual impact.

Using a dual-stream attention neural network to characterize mild cognitive impairment based on retinal images

Mild cognitive impairment (MCI) is a critical transitional stage between normal cognition and dementia, for which early detection is crucial for timely intervention. Retinal imaging has been shown as a promising potential biomarker for MCI. This study aimed to develop a dual-stream attention neural network to classify individuals with MCI based on multi-modal retinal images. Our approach incorporated a cross-modality fusion technique, a variable scale dense residual model, and a multi-classifier mechanism within the dual-stream network. The model utilized a residual module to extract image features and employed a multi-level feature aggregation method to capture complex context information. Self-attention and cross-attention modules were utilized at each convolutional layer to fuse features from optical coherence tomography (OCT) and fundus modalities, resulting in multiple output losses. The neural network was applied to classify individuals with MCI, Alzheimer's disease, and control participants with normal cognition. Through fine-tuning the pre-trained model, we classified community-dwelling participants into two groups based on cognitive impairment test scores. To identify retinal imaging biomarkers associated with accurate prediction, we used the Gradient-weighted Class Activation Mapping technique. The proposed method achieved high precision rates of 84.96% and 80.90% in classifying MCI and positive test scores for cognitive impairment, respectively. Notably, changes in the optic nerve head on fundus photographs or OCT images among patients with MCI were not used to discriminate patients from the control group. These findings demonstrate the potential of our approach in identifying individuals with MCI and emphasize the significance of retinal imaging for early detection of cognitive impairment.

Dual-stream EfficientNet with adversarial sample augmentation for COVID-19 computer aided diagnosis

Though a series of computer aided measures have been taken for the rapid and definite diagnosis of 2019 coronavirus disease (COVID-19), they generally fail to achieve high enough accuracy, including the recently popular deep learning-based methods. The main reasons are that: (a) they generally focus on improving the model structures while ignoring important information contained in the medical image itself; (b) the existing small-scale datasets have difficulty in meeting the training requirements of deep learning. In this paper, a dual-stream network based on the EfficientNet is proposed for the COVID-19 diagnosis based on CT scans. The dual-stream network takes into account the important information in both spatial and frequency domains of CT scans. Besides, Adversarial Propagation (AdvProp) technology is used to address the insufficient training data usually faced by the deep learning-based computer aided diagnosis and also the overfitting issue. Feature Pyramid Network (FPN) is utilized to fuse the dual-stream features. Experimental results on the public dataset COVIDx CT-2A demonstrate that the proposed method outperforms the existing 12 deep learning-based methods for COVID-19 diagnosis, achieving an accuracy of 0.9870 for multi-class classification, and 0.9958 for binary classification. The source code is available at https://github.com/imagecbj/covid-efficientnet.

Breast TransFG Plus: Transformer-based fine-grained classification model for breast cancer grading in Hematoxylin-Eosin stained pathological images

Background and ObjectiveBreast cancer is one of the most common malignancies in women, and the pathological grading of breast cancer is very important for the prognosis of breast cancer. But classification of breast Hematoxylin-Eosin (HE) stained pathological images by deep learning for breast cancer grading is difficult due to morphological similarities between different grades. Therefore, it is essential to have an efficient and accurate method of breast cancer grading. MethodIn this paper, a transformer-based fine-grained classification model named Breast TransFG Plus is proposed for breast cancer grading. Targeting the widespread distribution of cells in breast HE stained pathological images, this paper proposes part selection module plus, which is a kind of key information extraction method based on matrix addition, and double head classification structure, which is a kind of dual stream network structure. And balanced sampling based on data augmentation is used in the training set. ResultsThe proposed method has been evaluated on a public dataset and the classification accuracy, precision and recall on the test set are 99.39%, 99.18% and 99.59%, the number of parameters is 93 M. And it is proved that the proposed model is superior to previous studies by comparing with them. ConclusionsBreast TransFG Plus can achieve efficient and accurate breast cancer grading, and has the potential to meet clinical computer-aided diagnosis needs.