Dual Branch Research Articles

Joint image restoration for object detection in snowy weather

AbstractAlthough existing object detectors achieve encouraging performance of object detection and localisation under real ideal conditions, the detection performance in adverse weather conditions (snowy) is very poor and not enough to cope with the detection task in adverse weather conditions. Existing methods do not deal well with the effect of snow on the identity of object features or usually ignore or even discard potential information that can help improve the detection performance. To this end, the authors propose a novel and improved end‐to‐end object detection network joint image restoration. Specifically, in order to address the problem of identity degradation of object detection due to snow, an ingenious restoration‐detection dual branch network structure combined with a Multi‐Integrated Attention module is proposed, which can well mitigate the effect of snow on the identity of object features, thus improving the detection performance of the detector. In order to make more effective use of the features that are beneficial to the detection task, a Self‐Adaptive Feature Fusion module is introduced, which can help the network better learn the potential features that are beneficial to the detection and eliminate the effect of heavy or large local snow in the object area on detection by a special feature fusion, thus improving the network's detection capability in snowy. In addition, the authors construct a large‐scale, multi‐size snowy dataset called Synthetic and Real Snowy Dataset (SRSD), and it is a good and necessary complement and improvement to the existing snowy‐related tasks. Extensive experiments on a public snowy dataset (Snowy‐weather Datasets) and SRSD indicate that our method outperforms the existing state‐of‐the‐art object detectors.

VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection

The recent contrastive language-image pre-training (CLIP) model has shown great success in a wide range of image-level tasks, revealing remarkable ability for learning powerful visual representations with rich semantics. An open and worthwhile problem is efficiently adapting such a strong model to the video domain and designing a robust video anomaly detector. In this work, we propose VadCLIP, a new paradigm for weakly supervised video anomaly detection (WSVAD) by leveraging the frozen CLIP model directly without any pre-training and fine-tuning process. Unlike current works that directly feed extracted features into the weakly supervised classifier for frame-level binary classification, VadCLIP makes full use of fine-grained associations between vision and language on the strength of CLIP and involves dual branch. One branch simply utilizes visual features for coarse-grained binary classification, while the other fully leverages the fine-grained language-image alignment. With the benefit of dual branch, VadCLIP achieves both coarse-grained and fine-grained video anomaly detection by transferring pre-trained knowledge from CLIP to WSVAD task. We conduct extensive experiments on two commonly-used benchmarks, demonstrating that VadCLIP achieves the best performance on both coarse-grained and fine-grained WSVAD, surpassing the state-of-the-art methods by a large margin. Specifically, VadCLIP achieves 84.51% AP and 88.02% AUC on XD-Violence and UCF-Crime, respectively. Code and features are released at https://github.com/nwpu-zxr/VadCLIP.

High-Performance Defect Detection Methods for Real-Time Monitoring of Ceramic Additive Manufacturing Process Based on Small-Scale Datasets

Vat photopolymerization is renowned for its high flexibility, efficiency, and precision in ceramic additive manufacturing. However, due to the impact of random defects during the recoating process, ensuring the yield of finished products is challenging. At present, the industry mainly relies on manual visual inspection to detect defects; this is an inefficient method. To address this limitation, this paper presents a method for ceramic vat photopolymerization defect detection based on a deep learning framework. The framework innovatively adopts a dual-branch object detection approach, where one branch utilizes a fully convolution network to extract the features from fused images and the other branch employs a differential Siamese network to extract the differential information between two consecutive layer images. Through the design of the dual branches, the decoupling of image feature layers and image spatial attention weights is achieved, thereby alleviating the impact of a few abnormal points on training results and playing a crucial role in stabilizing the training process, which is suitable for training on small-scale datasets. Comparative experiments are implemented and the results show that using a Resnet50 backbone for feature extraction and a HED network for the differential Siamese network module yields the best detection performance, with an obtained F1 score of 0.89. Additionally, as a single-stage defect object detector, the model achieves a detection frame rate of 54.01 frames per second, which meets the real-time detection requirements. By monitoring the recoating process in real-time, the manufacturing fluency of industrial equipment can be effectively enhanced, contributing to the improvement of the yield of ceramic additive manufacturing products.

Defect Detection Algorithm for Battery Cell Casings Based on Dual-Coordinate Attention and Small Object Loss Feedback

To address the issue of low accuracy in detecting defects of battery cell casings with low space ratio and small object characteristics, the low space ratio feature and small object feature are studied, and an object detection algorithm based on dual-coordinate attention and small object loss feedback is proposed. Firstly, the EfficientNet-B1 backbone network is employed for feature extraction. Secondly, a dual-coordinate attention module is introduced to preserve more positional information through dual branches and embed the positional information into channel attention for precise localization of the low space ratio features. Finally, a small object loss feedback module is incorporated after the bidirectional feature pyramid network (BiFPN) for feature fusion, balancing the contribution of small object loss to the overall loss. Experimental comparisons on a battery cell casing dataset demonstrate that the proposed algorithm outperforms the EfficientDet-D1 object detection algorithm, with an average precision improvement of 4.23%. Specifically, for scratches with low space ratio features, the improvement is 13.21%; for wrinkles with low space ratio features, the improvement is 9.35%; and for holes with small object features, the improvement is 3.81%. Moreover, the detection time of 47.6 ms meets the requirements of practical production.

WGDPool: A broad scope extraction for weighted graph data

Graph pooling is a commonly used operation in graph neural networks to reduce the size of graph representation. To extract key information, pooling and representation need to be coupled. We propose a graph pooling method for weighted graphs called WGDPool (Weighted Graph Dual Pooling). Unlike traditional graph representation learning methods, the weight information of edges is also fed into convolutional graph neural networks (ConvGNN) to obtain graph representations. Dual branch convolutional graph neural networks is designed to learn the nodes’ and edges’ embeddings independently, and they are fused into a comprehensive representation of graph data. Pooling, as a tool of feature extraction and scale reduction of graph representation, adopts a differentiable version of k-means clustering and a multi-item parameterized loss function. Cut loss, orthogonality loss, clustering loss, and reconstruction loss are simultaneously considered. By parameterization, WGDPool is competent for diverse graph tasks. WGDPool outperformed other graph pooling methods in such common supervised and unsupervised tasks as biological or chemical classification, bibliography clustering and integrated circuit partition, demonstrating the effectiveness of our proposed pooling method.

Coupled image and kernel prior learning for high-generalized super-resolution

Deep convolution neural networks (DCNNs) have demonstrated great success on single image super resolution, where most existing methods aim to construct a deep model in an fully-supervised way using large amount of synthetic image pairs under ideal degradation assumption. The super-resolution performance would be significantly degraded for the low-resolution images captured under uncontrolled imaging conditions with complicated degradation procedures. To handle the above limitations, this work proposes a high-generalized image super-resolution framework by synergistically learning the latent image and the degradation process to achieve the specific priors for an under-studying observation in an unsupervised manner. Specifically, we incorporate dual branches of networks to configure our framework, where one is structured with both convolution and transformer blocks to learn local-to-global prior to generate high-quality image while the other aims to learn the kernel prior with a simple convolution-based architecture. The kernel prior subnet is pre-trained to enhance the stability of the joint optimization for the overall unsupervised learning procedure. With the estimated degradation kernel and target image, we produce the approximated low-resolution version with a convolution-based degradation block to formulate the loss function for the specific learning. Moreover, to pursue visually plausible image generation, we further incorporate a perceptual loss by leveraging a pre-trained discriminator in Gan-based SR model. Extensive experiments on several benchmark datasets have demonstrated the effectiveness of our proposed high-generalized super-resolution model, manifesting superiority over both supervised and unsupervised state-of-the-art SR models.

DBSTGNN-Att: Dual Branch Spatio-Temporal Graph Neural Network with an Attention Mechanism for Cellular Network Traffic Prediction

As network technology continues to develop, the popularity of various intelligent terminals has accelerated, leading to a rapid growth in the scale of wireless network traffic. This growth has resulted in significant pressure on resource consumption and network security maintenance. The objective of this paper is to enhance the prediction accuracy of cellular network traffic in order to provide reliable support for the subsequent base station sleep control or the identification of malicious traffic. To achieve this target, a cellular network traffic prediction method based on multi-modal data feature fusion is proposed. Firstly, an attributed K-nearest node (KNN) graph is constructed based on the similarity of data features, and the fused high-dimensional features are incorporated into the graph to provide more information for the model. Subsequently, a dual branch spatio-temporal graph neural network with an attention mechanism (DBSTGNN-Att) is designed for cellular network traffic prediction. Extensive experiments conducted on real-world datasets demonstrate that the proposed method outperforms baseline models, such as temporal graph convolutional networks (T-GCNs) and spatial–temporal self-attention graph convolutional networks (STA-GCNs) with lower mean absolute error (MAE) values of 6.94% and 2.11%, respectively. Additionally, the ablation experimental results show that the MAE of multi-modal feature fusion using the attributed KNN graph is 8.54% lower compared to that of the traditional undirected graphs.

Dual branch deep learning network for detection and stage grading of diabetic retinopathy

Diabetic retinopathy is a severe complication of diabetes that can lead to permanent blindness if not treated promptly. Early and accurate diagnosis of the disease is essential for successful treatment. This paper introduces a deep learning method for the detection and stage grading of diabetic retinopathy, using a single fundus retinal image. Our model utilizes transfer learning, employing two state-of-the-art pre-trained models as feature extractors and fine-tuning them on a new dataset. The proposed model is trained on a large multi-center dataset, including the APTOS 2019 dataset, obtained from publicly available sources. It achieves remarkable performance in diabetic retinopathy detection and stage classification on the APTOS 2019, outperforming the established literature. For binary classification, the proposed approach achieves an accuracy of 98.50%, a sensitivity of 99.46%, and a specificity of 97.51%. In stage grading, it achieves a quadratic weighted kappa of 93.00%, an accuracy of 89.60%, a sensitivity of 89.60%, and a specificity of 97.72%. The proposed approach serves as a reliable screening and stage grading tool for diabetic retinopathy, offering significant potential to enhance clinical decision-making and patient care.

Online Learning State Evaluation Method Based on Face Detection and Head Pose Estimation.

In this paper, we propose a learning state evaluation method based on face detection and head pose estimation. This method is suitable for mobile devices with weak computing power, so it is necessary to control the parameter quantity of the face detection and head pose estimation network. Firstly, we propose a ghost and attention module (GA) base face detection network (GA-Face). GA-Face reduces the number of parameters and computation in the feature extraction network through the ghost module, and focuses the network on important features through a parameter-free attention mechanism. We also propose a lightweight dual-branch (DB) head pose estimation network: DB-Net. Finally, we propose a student learning state evaluation algorithm. This algorithm can evaluate the learning status of students based on the distance between their faces and the screen, as well as their head posture. We validate the effectiveness of the proposed GA-Face and DB-Net on several standard face detection datasets and standard head pose estimation datasets. Finally, we validate, through practical cases, that the proposed online learning state assessment method can effectively assess the level of student attention and concentration, and, due to its low computational complexity, will not interfere with the student's learning process.

A Dual-Branch Speech Enhancement Model with Harmonic Repair

Recent speech enhancement studies have mostly focused on completely separating noise from human voices. Due to the lack of specific structures for harmonic fitting in previous studies and the limitations of the traditional convolutional receptive field, there is an inevitable decline in the auditory quality of the enhanced speech, leading to a decrease in the performance of subsequent tasks such as speech recognition and speaker identification. To address these problems, this paper proposes a Harmonic Repair Large Frame enhancement model, called HRLF-Net, that uses a harmonic repair network for denoising, followed by a real-imaginary dual branch structure for restoration. This approach fully utilizes the harmonic overtones to match the original harmonic distribution of speech. In the subsequent branch process, it restores the speech to specifically optimize its auditory quality to the human ear. Experiments show that under HRLF-Net, the intelligibility and quality of speech are significantly improved, and harmonic information is effectively restored.

Face Forgery Detection with Long-Range Noise Features and Multilevel Frequency-Aware Clues

The widespread dissemination of high-fidelity fake faces created by face forgery techniques has caused serious trust concerns and ethical issues in modern society. Consequently, face forgery detection has emerged as a prominent topic of research to prevent technology abuse. Although, most existing face forgery detectors demonstrate success when evaluating high-quality faces under intra-dataset scenarios, they often overfit manipulation-specific artifacts and lack robustness to postprocessing operations. In this work, we design an innovative dual-branch collaboration framework that leverages the strengths of the transformer and CNN to thoroughly dig into the multimodal forgery artifacts from both a global and local perspective. Specifically, a novel adaptive noise trace enhancement module (ANTEM) is proposed to remove high-level face content while amplifying more generalized forgery artifacts in the noise domain. Then, the transformer-based branch can track long-range noise features. Meanwhile, considering that subtle forgery artifacts could be described in the frequency domain even in a compression scenario, a multilevel frequency-aware module (MFAM) is developed and further applied to the CNN-based branch to extract complementary frequency-aware clues. Besides, we incorporate a collaboration strategy involving cross-entropy loss and single center loss to enhance the learning of more generalized representations by optimizing the fusion features of the dual branch. Extensive experiments on various benchmark datasets substantiate the superior generalization and robustness of our framework when compared to the competing approaches.

Super‐resolution reconstruction based on generative adversarial networks with dual branch half instance normalization

AbstractThis paper proposes a super‐resolution reconstruction model, SRPGANto improve the visual quality of images based on generative adversarial networks (GANs) by improving the network structures of the generator and the discriminator. In the generator, a dual branch residual block is designed instead of the residual block, including a branch with an attention mechanism and a branch without an attention mechanism, to extract more differentiated features. Normalization methods are explored to avoid unstable training and bath normalization artifacts and use a half instance normalization layer that is more suitable for underlying visual problems compared with traditional batch normalization. In the discriminator, PatchGAN is applied instead of typical GAN to improve the generation of local texture by discriminating each patch rather than the global image. The experimental results on the public datasets demonstrate that the proposed SRPGAN can achieve excellent quantitative evaluation while improving the visual quality of reconstructed images.

DBNet-SI: Dual branch network of shift window attention and inception structure for skin lesion segmentation

The U-shaped convolutional neural network (CNN) has attained remarkable achievements in the segmentation of skin lesion. However, given the inherent locality of convolution, this architecture cannot capture long-range pixel dependencies and multiscale global contextual information effectively. Moreover, repeated convolutions and downsampling operations can readily result in the omission of intricate local fine-grained details. In this paper, we proposed a U-shaped network (DBNet-SI) equipped with a dual-branch module that combines shift window attention and inception structures. First, we proposed a dual-branch module that combines shift window attention and inception structures (MSI) to better capture multiscale global contextual information and long-range pixel dependencies. Specifically, we have devised a cross-branch bidirectional interaction module within the MSI module to enable information complementarity between the two branches in the channel and spatial dimensions. Therefore, MSI is capable of extracting distinguishing and comprehensive features to accurately identify the skin lesion boundaries. Second, we have devised a progressive feature enhancement and information compensation module (PFEIC), which progressively compensates for fine-grained features through reconstructed skip connections and integrated global context attention modules. The results of the experiment show the superior segmentation performance of DBNet-SI compared with other deep learning models for skin lesion segmentation in the ISIC2017 and ISIC2018 datasets. Ablation studies demonstrate that our model can effectively extract rich multiscale global contextual information and compensate for the loss of local details.

Research on the short-term wind power prediction with dual branch multi-source fusion strategy

The smooth operation of the power system imposes high requirements on wind power forecasting. Given the forecasting challenges brought by the coexistence of strong and weak disturbances in wind power data, this paper proposes a hierarchical anti-disturbance mechanism to raise the accuracy of single-point wind power prediction. In the first-level disturbance processing, we prioritize the treatment of strong disturbances in wind power data through a dual-branch multi-source fusion prediction structure. The wind power subsequences with distinct time–frequency characteristics are divided into two sets, processed by the stationary set prediction branch and the drastic set prediction branch, respectively. For the drastic set, we incorporate microenvironmental multi-source data surrounding the wind turbine as input and propose a parallel-structured attention fusion module CA (Convolution attention fusion block) to provide effective data support for the prediction module, then extract rich fusion features through deep temporal LSTM algorithm and ultimately achieve fine-grained forecasting. For weak disturbances that have not been adequately addressed in the previous step, we conduct a secondary treatment through error correction strategy. By employing the hierarchical processing approach, this paper accomplishes a comprehensive handling of disturbances with different intensities. Experimental results on four data sets show that the proposed model DEWFM (Dual-branch and error correction wind power forecast) demonstrates remarkable performance advantages in comparing with the baseline model and other advanced models, the average values of MAE reaching 1.679, revealing the effectiveness of the proposed method in enhancing the precision of short-term wind power prediction.

Multi-Scale Spatial–Spectral Residual Attention Network for Hyperspectral Image Classification

Hyperspectral images (HSIs) encapsulate a vast amount of information due to their expansive size and high number of channel dimensions. However, they are insufficiently utilized for ineffective feature extraction, particularly for regions with few samples and predominant edges. To fully leverage the spatial–spectral features of HSIs, a dual-branch multi-scale spatial–spectral residual attention network (MSRAN) that integrates multi-scale feature extraction with residual attention mechanisms is proposed. MSRAN independently extracts spatial and spectral features through dual branches, minimizing the interference between these features and enhancing the focus on feature extraction in different dimensions. Specifically, in the spectral feature extraction branch, diverse-scale 3D convolution kernels capture extended spectral sequence characteristics and neighborhood spectral features. The convolution fusion emphasizes the weight of the central pixel to be classified, followed by the use of spectral residual attention mechanisms to extract enhanced central-pixel spectral features. In the spatial feature extraction branch, multi-level receptive fields are utilized to extract various fine-grained spatial contours, edges, and local detailed features, which are further processed through spatial residual attention to effectively extract spatial composite features. Finally, the convolution fusion module adaptively integrates the center-enhanced spectral features with multi-level fine-grained spatial features for classification. Extensive comparative experiments and ablation studies demonstrate that MSRAN achieves highly competitive results on two classic datasets from Pavia University and Salinas as well as on a novel dataset of WHU-Hi-LongKou.

It takes two: Dual Branch Augmentation Module for domain generalization

Although great success has been achieved in various computer vision tasks, deep neural networks (DNNs) suffer dramatic performance degradation when evaluated on out-of-distribution data. Domain generalization (DG) is proposed to handle this problem by learning domain-agnostic information from multiple source domains to generalize well on unseen target domains. Several methods resort to Fourier transform due to its simplicity and efficiency. They argue that amplitude spectra imply domain-specific information, which should be suppressed, while phase counterparts imply domain-agnostic information, which should be preserved. However, these methods only suppress the domain-specific information in source domains and neglect the relationship with target domains, leading to the persistence of the domain gap. Besides, these methods preserve domain-agnostic information by keeping phase components unchanged, causing them to be underutilized. In this paper, we propose Dual Branch Augmentation Module (DBAM) by leveraging Fourier transform and taking advantage of both amplitude and phase spectra. For the amplitude branch, we propose Inner-domain Amplitude Distribution Rectification (IADR) and Cross-domain Amplitude Dirichlet Mixup (CADM) to stabilize the training process and explore more feature space. In addition, we propose Test-time Amplitude Prototype Calibration (TAPC) to construct the connection between source and target domains during evaluation to further mitigate the domain gap. For the phase branch, we propose Random Symmetric Phase Perturbation (RSPP) to enhance the robustness for recognizing domain-agnostic information. With the combined contributions of the two branches, DBAM significantly surpasses other state-of-the-art (SOTA) methods. Extensive experiments on four benchmarks and further analysis demonstrate the effectiveness of DBAM.

Scale-Aware Super-Resolution Network With Dual Affinity Learning for Lesion Segmentation From Medical Images.

Convolutional neural networks (CNNs) have shown remarkable progress in medical image segmentation. However, the lesion segmentation remains a challenge to state-of-the-art CNN-based algorithms due to the variance in scales and shapes. On the one hand, tiny lesions are hard to delineate precisely from the medical images which are often of low resolutions. On the other hand, segmenting large-size lesions requires large receptive fields, which exacerbates the first challenge. In this article, we present a scale-aware super-resolution (SR) network to adaptively segment lesions of various sizes from low-resolution (LR) medical images. Our proposed network contains dual branches to simultaneously conduct lesion mask SR (LMSR) and lesion image SR (LISR). Meanwhile, we introduce scale-aware dilated convolution (SDC) blocks into the multitask decoders to adaptively adjust the receptive fields of the convolutional kernels according to the lesion sizes. To guide the segmentation branch to learn from richer high-resolution (HR) features, we propose a feature affinity (FA) module and a scale affinity (SA) module to enhance the multitask learning of the dual branches. On multiple challenging lesion segmentation datasets, our proposed network achieved consistent improvements compared with other state-of-the-art methods. Code will be available at: https://github.com/poiuohke/SASR_Net.

Simultaneous segmentation and classification of colon cancer polyp images using a dual branch multi-task learning network.

Accurate classification and segmentation of polyps are two important tasks in the diagnosis and treatment of colorectal cancers. Existing models perform segmentation and classification separately and do not fully make use of the correlation between the two tasks. Furthermore, polyps exhibit random regions and varying shapes and sizes, and they often share similar boundaries and backgrounds. However, existing models fail to consider these factors and thus are not robust because of their inherent limitations. To address these issues, we developed a multi-task network that performs both segmentation and classification simultaneously and can cope with the aforementioned factors effectively. Our proposed network possesses a dual-branch structure, comprising a transformer branch and a convolutional neural network (CNN) branch. This approach enhances local details within the global representation, improving both local feature awareness and global contextual understanding, thus contributing to the improved preservation of polyp-related information. Additionally, we have designed a feature interaction module (FIM) aimed at bridging the semantic gap between the two branches and facilitating the integration of diverse semantic information from both branches. This integration enables the full capture of global context information and local details related to polyps. To prevent the loss of edge detail information crucial for polyp identification, we have introduced a reverse attention boundary enhancement (RABE) module to gradually enhance edge structures and detailed information within polyp regions. Finally, we conducted extensive experiments on five publicly available datasets to evaluate the performance of our method in both polyp segmentation and classification tasks. The experimental results confirm that our proposed method outperforms other state-of-the-art methods.

IDDNet: a deep interactive dual-domain convolutional neural network with auxiliary modality for fast MRI reconstruction

Reconstructing a complete image accurately from an undersampled <i>k</i>-space matrix is a viable approach for magnetic resonance imaging (MRI) acceleration. In recent years, numerous deep learning (DL)-based methods have been employed to improve MRI reconstruction. Among these methods, the cross-domain method has been proven to be effective. However, existing cross-domain reconstruction algorithms sequentially link the image domain and <i>k</i>-space networks, disregarding the interplay between different domains, consequently leading to a deficiency in reconstruction accuracy. In this work, we propose a deep interactive dual-domain network (IDDNet) with an auxiliary modality for accelerating MRI reconstruction to effectively extract pertinent information from multiple MR domains and modalities. The IDDNet first extracts shallow features from low-resolution target modalities in the image domain to obtain visual representation information. In the following feature processing, a parallel interactive architecture with dual branches is designed to extract deep features from relevant information of dual domains simultaneously to avoid redundant priority priors in sequential links. Furthermore, the model uses additional information from the auxiliary modality to refine the structure and improve the reconstruction accuracy. Numerous experiments at different sampling masks and acceleration rates on the MICCAI BraTS 2019 brain and fastMRI knee datasets show that IDDNet achieves excellent accelerated MRI reconstruction performance.

Dual Branch Multi-Level Semantic Learning for Few-Shot Segmentation.

Few-shot semantic segmentation aims to segment novel-class objects in a query image with only a few annotated examples in support images. Although progress has been made recently by combining prototype-based metric learning, existing methods still face two main challenges. First, various intra-class objects between the support and query images or semantically similar inter-class objects can seriously harm the segmentation performance due to their poor feature representations. Second, the latent novel classes are treated as the background in most methods, leading to a learning bias, whereby these novel classes are difficult to correctly segment as foreground. To solve these problems, we propose a dual-branch learning method. The class-specific branch encourages representations of objects to be more distinguishable by increasing the inter-class distance while decreasing the intra-class distance. In parallel, the class-agnostic branch focuses on minimizing the foreground class feature distribution and maximizing the features between the foreground and background, thus increasing the generalizability to novel classes in the test stage. Furthermore, to obtain more representative features, pixel-level and prototype-level semantic learning are both involved in the two branches. The method is evaluated on PASCAL- 5i 1 -shot, PASCAL- 5i 5 -shot, COCO- 20i 1 -shot, and COCO- 20i 5 -shot, and extensive experiments show that our approach is effective for few-shot semantic segmentation despite its simplicity.