Multi-branch Network Research Articles

A Neural Network Based on Supervised Multi-View Contrastive Learning and Two-Stage Feature Fusion for Face Anti-Spoofing

As one of the most crucial parts of face detection, the accuracy and the generalization of face anti-spoofing are particularly important. Therefore, it is necessary to propose a multi-branch network to improve the accuracy and generalization of the detection of unknown spoofing attacks. These branches consist of several frequency map encoders and one depth map encoder. These encoders are trained together. It leverages multiple frequency features and generates depth map features. High-frequency edge texture is beneficial for capturing moiré patterns, while low-frequency features are sensitive to color distortion. Depth maps are more discriminative than RGB images at the visual level and serve as useful auxiliary information. Supervised Multi-view Contrastive Learning enhances multi-view feature learning. Moreover, a two-stage feature fusion method effectively integrates multi-branch features. Experiments on four public datasets, namely CASIA-FASD, Replay–Attack, MSU-MFSD, and OULU-NPU, demonstrate model effectiveness. The average Half Total Error Rate (HTER) of our model is 4% (25% to 21%) lower than the Adversarial Domain Adaptation method in inter-set evaluations.

FA-HRNet: A New Fusion Attention Approach for Vegetation Semantic Segmentation and Analysis

Semantic segmentation of vegetation in aerial remote sensing images is a critical aspect of vegetation mapping. Accurate vegetation segmentation effectively informs real-world production and construction activities. However, the presence of species heterogeneity, seasonal variations, and feature disparities within remote sensing images poses significant challenges for vision tasks. Traditional machine learning-based methods often struggle to capture deep-level features for the segmentation. This work proposes a novel deep learning network named FA-HRNet that leverages the fusion of attention mechanism and a multi-branch network structure for vegetation detection and segmentation. Quantitative analysis from multiple datasets reveals that our method outperforms existing approaches, with improvements in MIoU and PA by 2.17% and 4.85%, respectively, compared with the baseline network. Our approach exhibits significant advantages over the other methods regarding cross-region and cross-scale capabilities, providing a reliable vegetation coverage ratio for ecological analysis.

Auxiliary Feature Fusion and Noise Suppression for HOI Detection

In recent years, one-stage HOI (Human–Object Interaction) detection methods tend to divide the original task into multiple sub-tasks by using a multi-branch network structure. However, there is no sufficient attention to information communication between these branches. The inference approach in the cascaded structure is singular, while fully parallel methods will disrupt the associations between different pieces of information. Besides, noise interference may occur during the fusion of different features and thus affect the detection performance. To address these issues, this article proposes a one-stage three-branch parallel HOI detection method, which treats HOI as three separate sub-tasks (human detection, object detection, and interaction detection) and leverages three distinct reasoning relationships to generate richer relational information. Firstly , an auxiliary feature fusion (AFF) module is introduced, which integrates features originally extracted independently to form fused features enriched with supplementary information. This approach strengthens communication between branches in the network while handling the three sub-tasks concurrently, thereby facilitating the exchange of more contextual information. Secondly , to mitigate noise interference generated during the fusion process, a fusion noise suppression (FNS) module is introduced, which effectively suppresses noise and enhances the model’s performance in interaction detection tasks. Finally , experiments are conducted on two major benchmark datasets, and experimental results show that our HOI detection method is superior to previous methods. Also, ablation studies confirm the effectiveness of all the components in our proposed method.

Graph masked self-distillation learning for prediction of mutation impact on protein–protein interactions

Assessing mutation impact on the binding affinity change (ΔΔG) of protein–protein interactions (PPIs) plays a crucial role in unraveling structural-functional intricacies of proteins and developing innovative protein designs. In this study, we present a deep learning framework, PIANO, for improved prediction of ΔΔG in PPIs. The PIANO framework leverages a graph masked self-distillation scheme for protein structural geometric representation pre-training, which effectively captures the structural context representations surrounding mutation sites, and makes predictions using a multi-branch network consisting of multiple encoders for amino acids, atoms, and protein sequences. Extensive experiments demonstrated its superior prediction performance and the capability of pre-trained encoder in capturing meaningful representations. Compared to previous methods, PIANO can be widely applied on both holo complex structures and apo monomer structures. Moreover, we illustrated the practical applicability of PIANO in highlighting pathogenic mutations and crucial proteins, and distinguishing de novo mutations in disease cases and controls in PPI systems. Overall, PIANO offers a powerful deep learning tool, which may provide valuable insights into the study of drug design, therapeutic intervention, and protein engineering.

Cross-Channel Specific-Mutual Feature Transfer Learning for Motor Imagery EEG Signals Decoding.

In recent years, with the rapid development of deep learning, various deep learning frameworks have been widely used in brain-computer interface (BCI) research for decoding motor imagery (MI) electroencephalogram (EEG) signals to understand brain activity accurately. The electrodes, however, record the mixed activities of neurons. If different features are directly embedded in the same feature space, the specific and mutual features of different neuron regions are not considered, which will reduce the expression ability of the feature itself. We propose a cross-channel specific-mutual feature transfer learning (CCSM-FT) network model to solve this problem. The multibranch network extracts the specific and mutual features of brain's multiregion signals. Effective training tricks are used to maximize the distinction between the two kinds of features. Suitable training tricks can also improve the effectiveness of the algorithm compared with novel models. Finally, we transfer two kinds of features to explore the potential of mutual and specific features to enhance the expressive power of the feature and use the auxiliary set to improve identification performance. The experimental results show that the network has a better classification effect in the BCI Competition IV-2a and the HGD datasets.

Rapid identification and early warning of axial compressor stall based on multiscale CNN-SVM-FC model

Early prewarning of compressor stall and surge is crucial to avoid aircraft engine instability, yet it is challenging due to the complex and unstable flow field characterized by multiple modes and multiscale features. To enhance the multi-scale feature representation capability of Convolutional Neural Network-Support Vector Machine (CNN-SVM) algorithm, a novel classifier modelling method combined multiscale windows with CNN-SVM is introduced for stall prewarning in this paper, named Multiscale CNN-SVM-FC. Multiscale detection windows are utilized to adaptively identify various pressure features during the stall process. Additionally, to reduce the false alarm rate, a fuzzy control algorithm is integrated with the temporal accumulation of prediction results from the multi-branch network for joint analysis. A series of test data from a five-stage axial compressor at different operating speeds is used to verify this method. The results indicate that the proposed Multiscale CNN-SVM-FC method enhances the accuracy of classification and reduces the false alarm rate compared to the standard CNN-SVM model, achieving over 99% accuracy in identifying unstable states under various speeds. Compared to three traditional stall prewarning methods, the Multiscale CNN-SVM-FC model provides an average warning signal 164 milliseconds ahead of stall, and reduces the uncertainty associated with threshold selection, which typically relies on engineering experience.

Adaptive Deep Neural Network for Click-Through Rate estimation

Over the past few years, deep learning networks have been applied in the estimation of the Click-Through Rate (CTR). The main task of the CTR model lies in predicting a user’s positive/negative response to an item. Most of existing CTR models are constituted of two parts: a deep neural network (DNN) and a wide model (e.g., deep cross network). The wide models are generally designed to learn the feature interaction and contain much more parameters than the DNN. As a consequence, it is quite difficult for existing CTR models to work on the resource-limited devices. Blindly removing the wide model will significantly encumber the predictive performance of the algorithm. In this paper, we adopt the knowledge distillation to train a single multi-branch network and assemble all branches as a teacher network. Each branch network is a simple DNN which not only matches the ground-truth label distribution but also aligns the prediction distribution of the teacher network. One of branch networks, termed as Adaptive Deep Neural Network (ADNN), is trained independently and further combined with a wide model to learn feature interactions. Our method does not require pre-training any high-capacity teacher models, which endows our method with higher efficiency compared with existing ones. The experimental results tested on Criteo and Avazu datasets show that the hybrid model outperforms state-of-the-art methods, and the light model ADNN also has a considerable performance accuracy over certain modern complex models, demonstrating the superiority of the methodology.

Sig-RFPNet: multi-branch network and multi-classifier method for offline signature recognition

Sig-RFPNet: multi-branch network and multi-classifier method for offline signature recognition

Unsupervised multi-branch network with high-frequency enhancement for image dehazing

Recently, CycleGAN-based methods have been widely applied to the unsupervised image dehazing and achieved significant results. However, most existing CycleGAN-based methods ignore that the input of the generator contains two different distributions of data which can often lead to confusion in the learning process of the generator, consequently limiting the final dehazing performance. Moreover, reconstructing clear images through model architecture design and loss functions is an indirect constraint, making it difficult to compensate for the missing high-frequency information, such as textures and structures in the extracted features from hazy images. To address these issues, in this paper, we propose an Unsupervised Multi-Branch with High-Frequency Enhancement Network (UME-Net) which contain an Multi-Branch Dehazing Network (MBDN) and a High-Frequency Components Enhancement Module (HFEM). Specifically, MBDN constructs a single unsupervised dehazing network with Shared Encoding Module (SEM) and Multi-Branch Decoding Module (MDM). SEM enhance the consistency of feature representation and MDM effectively addresses the confusion during the generator learning process in CycleGAN-based methods. Furthermore, based on a key observation that hazy images and their corresponding clear images exhibit only subtle differences in high-frequency information, the HFEM is designed to compensates for the missing high-frequency information in the network which further enhances the reconstruction capability of the UME-Net for restore edge and texture information in obscured by dense haze. Experimental results on challenging benchmark datasets demonstrate the superiority of our UME-Net over SOTA unsupervised image dehazing methods. The source code is available at https://www.github.com/thislzm/UME-Net.

Local context attention learning for fine-grained scene graph generation

Fine-grained scene graph generation aims to parse the objects and their fine-grained relationships within scenes. Despite the significant progress in recent years, their performance is still limited by two major issues: (1) ambiguous perception under a global view; (2) the lack of reliable, fine-grained annotations. We argue that understanding the local context is important in addressing the two issues. However, previous works often overlook it, which limits their effectiveness in fine-grained scene graph generation. To tackle this challenge, we introduce a Local-context Attention Learning method that concentrates on local context and can generate high-reliability, fine-grained annotations. It comprises two components: (1) The Fine-grained Location Attention Network (FLAN), a multi-branch network that encompasses global and local branches, can attend to local informative context and perceive granularity levels in different regions, thereby adaptively enhancing the learning of fine-grained locations. (2) The Fine-grained Location Label Transfer (FLLT) method identifies coarse-grained labels inconsistent with the local context and determines which labels should be transferred through the global confidence thresholding strategy, finally transferring them to reliable local context-consistent fine-grained ones. Experiments conducted on the Visual Genome, OpenImage, and GQA-200 datasets show that the proposed methods achieve significant improvements on the fine-grained scene graph generation task. By addressing the challenge mentioned above, our method also achieves state-of-the-art performances on the three datasets.

Leadwise clustering multi-branch network for multi-label ECG classification

The 12-lead electrocardiogram (ECG) is widely used for diagnosing cardiovascular diseases in clinical practice. Recently, deep learning methods have become increasingly effective for automatically classifying ECG signals. However, most current research simply combines the 12-lead ECG signals into a matrix without fully considering the intrinsic relationships between the leads and the heart's structure. To better utilize medical domain knowledge, we propose a multi-branch network for multi-label ECG classification and introduce an intuitive and effective lead grouping strategy. Correspondingly, we design multi-branch networks where each branch employs a multi-scale convolutional network structure to extract more comprehensive features, with each branch corresponding to a lead combination. To better integrate features from different leads, we propose a feature weighting fusion module. We evaluate our method on the PTB-XL dataset for classifying 4 arrhythmia types and normal rhythm, and on the China Physiological Signal Challenge 2018 (CPSC2018) database for classifying 8 arrhythmia types and normal rhythm. Experimental results on multiple multi-label datasets demonstrate that our proposed multi-branch network outperforms state-of-the-art networks in multi-label classification tasks

Equilibrating the impact of fluid scattering attenuation on underwater optical imaging via adaptive parameter learning.

Fluid scattering attenuation is a pervasive and intricate physical phenomenon in underwater environment, which challenges optical imaging for various visual tasks. This paper presents a Learnable Physical Imaging Model (LPIM) for equilibrating the impact of the fluid scattering attenuation on underwater optical imaging. Whereas the imaging medium has the liquidity and diversity, an adaptive physical imaging framework is proposed to learn the nonlinear degradation parameter for various complex underwater environment. This adaptive imaging model is jointed by adversarial competition and directional learning strategy, which makes our model learn more excellent properties from the unpaired referring image, and guides the restoration network to directionally adjust the imaging model parameters. To describe the real scenarios better, a multi-branch network, i.e., R-Net, D-Net and BN-net, is developed to learn and capture the color, light intensity and attenuation information. Experimental results on several datasets demonstrate that our approach outperforms the state-of-the-arts in both color balance and visual effect. The LPIM provides an innovative framework designed to counteract image degradation resulting from fluid scattering and attenuation.

HR-YOLO: A Multi-Branch Network Model for Helmet Detection Combined with High-Resolution Network and YOLOv5

Automatic detection of safety helmet wearing is significant in ensuring safe production. However, the accuracy of safety helmet detection can be challenged by various factors, such as complex environments, poor lighting conditions and small-sized targets. This paper presents a novel and efficient deep learning framework named High-Resolution You Only Look Once (HR-YOLO) for safety helmet wearing detection. The proposed framework synthesizes safety helmet wearing information from the features of helmet objects and human pose. HR-YOLO can use features from two branches to make the bounding box of suppression predictions more accurate for small targets. Then, to further improve the iterative efficiency and accuracy of the model, we design an optimized residual network structure by using Optimized Powered Stochastic Gradient Descent (OP-SGD). Moreover, a Laplace-Aware Attention Model (LAAM) is designed to make the YOLOv5 decoder pay more attention to the feature information from human pose and suppress interference from irrelevant features, which enhances network representation. Finally, non-maximum suppression voting (PA-NMS voting) is proposed to improve detection accuracy for occluded targets, using pose information to constrain the confidence of bounding boxes and select optimal bounding boxes through a modified voting process. Experimental results demonstrate that the presented safety helmet detection network outperforms other approaches and has practical value in application scenarios. Compared with the other algorithms, the proposed algorithm improves the precision, recall and mAP by 7.27%, 5.46% and 7.3%, on average, respectively.

Bearings remaining useful life prediction across equipment-operating conditions based on multisource-multitarget domain adaptation

Transfer learning has been extensively used to build bearings remaining useful life prediction models under multi-operating conditions. However, most models only consider single-target domain, the generalization of the model under across equipment-operating conditions cannot reach the development of modern industry. Therefore, this study proposes a multisource-multitarget domain adaption transfer learning network (MSMTDATLN) hoping to solve across equipment-operating conditions prediction. Constructing a one-total and multi-branch network in MSMTDATLN to extract domain-invariant and domain-specific features from multisource-multitarget domains, adaptively eliminating the differences between domain-specific features, enhancing adaptive regression capabilities by aligning output decision boundaries. Improving model generalization in multitarget domains by the above methods, while enabling end-to-end training and inference. Experimental results on across equipment-operating conditions datasets validate the proposed method reduces RMSE index by 7% compared with other advanced algorithms while providing better model generalization, which can provide theoretical foundation and technical support for the bearing prediction tasks across equipment-operating conditions.

Multi-Branch Network for Color Image Denoising Using Dilated Convolution and Attention Mechanisms.

Image denoising is regarded as an ill-posed problem in computer vision tasks that removes additive noise from imaging sensors. Recently, several convolution neural network-based image-denoising methods have achieved remarkable advances. However, it is difficult for a simple denoising network to recover aesthetically pleasing images owing to the complexity of image content. Therefore, this study proposes a multi-branch network to improve the performance of the denoising method. First, the proposed network is designed based on a conventional autoencoder to learn multi-level contextual features from input images. Subsequently, we integrate two modules into the network, including the Pyramid Context Module (PCM) and the Residual Bottleneck Attention Module (RBAM), to extract salient information for the training process. More specifically, PCM is applied at the beginning of the network to enlarge the receptive field and successfully address the loss of global information using dilated convolution. Meanwhile, RBAM is inserted into the middle of the encoder and decoder to eliminate degraded features and reduce undesired artifacts. Finally, extensive experimental results prove the superiority of the proposed method over state-of-the-art deep-learning methods in terms of objective and subjective performances.

Fault diagnosis of axial piston pump based on multi-source subdomain adaptation and sensor data fusion

The axial piston pump is the core component in hydraulic systems. Its condition monitoring and fault diagnosis are crucial to ensure the safe and reliable operation of hydraulic systems. However, most of the existing fault diagnosis methods for axial piston pumps use the same working condition data. In actual operation, axial piston pump often experience varying loads, and the collected data is typically correlated but has different distributions. Therefore, a transfer learning method of multi-source subdomain adaptation and sensor fusion (MSASF) is proposed for fault diagnosis of axial piston pump. The proposed MSASF has three modules, a shared feature extraction module, a domain-specific feature extraction module and an output decision module. Firstly, the adaptive weighted fusion of multi-sensor data features is realized by the shared feature extraction module and the common features of multi-source heterogeneous data are extracted. Secondly, in the domain-specific feature extraction module, multi-branch network is used to extract features of each pair of source and target domains and the local maximum mean difference is utilized to align the sub-domain distribution of each pair of source and target domains. Finally, in the output decision module, the distribution distance between each pair of source domain and target domain is calculated using maximum mean discrepancy to obtain its weighted score. Combined with the classification output of each source domain, the final diagnosis decision is made. A dataset was constructed for the axial piston pump design fault experiment, and four sets of transfer tasks were designed to compare with those of seven classic methods. The experimental results showed that the proposed MSASF method exhibits a superior domain adaptation effect and fault diagnosis performance.

Multi-source domain adaptive network based on local kernelized higher-order moment matching for rotating machinery fault diagnosis

Unsupervised domain adaptation has been extensively researched in rotating-machinery cross-domain fault diagnosis. A multi-source domain adaptive network based on local kernelized higher-order moment matching is constructed in this research for rotating-machinery fault diagnosis. Firstly, a multi-branch network is designed to map each source-target pair to a domain-specific shared space and to extract domain-invariant features using domain adversarial thought. Then, a local kernelized higher-order moment matching algorithm is proposed to perform fine-grained matching in shared category subspace. Finally, a feature fusion strategy based on the local domain distribution deviation is applied to synthesize the output features of multiple classifiers to obtain diagnostic results. The experimental validation of two-branch and three-branch networks on two public datasets is carried out and average diagnostic accuracies both exceed 99%. The results demonstrate the effectiveness and superiority of the approach.

A Selective Multi-Branch Network for Edge-Oriented Object Localization and Classification

This study introduces a novel selective multi-branch network architecture designed to speed up object localization and classification on low-performance edge devices. The concept builds upon the You Only Look at Interested Cells (YOLIC) method, which was proposed by us earlier. In this approach, we categorize cells of interest (CoIs) into distinct regions of interest (RoIs) based on their locations and urgency. We then employ some expert branch networks for detailed object detection in each of the RoIs. To steer these branches effectively, a selective attention unit is added into the detection process. This unit can locate RoIs that are likely to contain objects under concern and trigger corresponding expert branch networks. The inference can be more efficient because only part of the feature map is used to make decisions. Through extensive experiments on various datasets, the proposed network demonstrates its ability to reduce the inference time while still maintaining competitive performance levels compared to the current detection algorithms.

High-Resolution Multi-Scale Feature Fusion Network for Running Posture Estimation

Running posture estimation is a specialized task in human pose estimation that has received relatively little research attention due to the lack of appropriate datasets. To address this issue, this paper presents the construction of a new benchmark dataset called “Running Human”, which was specifically designed for running sports. This dataset contains over 1000 images along with comprehensive annotations for 1288 instances of running humans, including bounding boxes and keypoint annotations on the human body. Additionally, a Receptive Field Spatial Pooling (RFSP) module was developed to tackle the challenge of joint occlusion, which is common in running sports images. This module was incorporated into the High-Resolution Network (HRNet) model, resulting in a novel network model named the Running Human Posture Network (RHPNet). By expanding the receptive field and effectively utilizing multi-scale features extracted from the multi-branch network, the RHPNet model significantly enhances the accuracy of running posture estimation. On the Running Human dataset, the proposed method achieved state-of-the-art performance. Furthermore, experiments were conducted on two benchmark datasets. Compared to the state-of-the-art ViTPose-L method, when applied to the COCO dataset, RHPNet demonstrated comparable prediction accuracy while utilizing only one tenth of the parameters and one eighth of the floating-point operations (FLOPs). On the MPII dataset, RHPNet achieves a PCKh@0.5 score of 92.0, which is only 0.5 points lower than the state-of-the-art method, PCT. These experimental results provide strong validation for the effectiveness and excellent generalization ability of the proposed method.

Universal 12-lead ECG representation for signal denoising and cardiovascular disease detection by fusing generative and contrastive learning

With the wide use of electrocardiogram (ECG) technology, more and more ECGs have been collected and stored. However, ECG labeling is costly and laborious, the utilization of unlabeled ECG is still a critical challenge. Self-supervised learning (SSL) is a way to deal with this problem, which can learn representations from unlabeled ECG and use them for downstream tasks. In this study, a novel SSL model that fuses generative learning (denoising autoencoder, DAE) and contrastive learning (CL) is proposed to learn robust representations from unlabeled ECG for downstream denoising and classification. The 12-lead ECGs are separated into one-dimensional single-lead ECGs and ECG-specific noises are added to the original ECGs as the data augmentation method. To improve the model's denoising and feature extraction abilities, the reconstruction loss and contrastive loss are combined during the pretraining phase. In the downstream classification task, a multi-branch network is used to enhance the correlation between different ECG leads. As a result, it improves the denoising performance over the standard DAE by an average of 5.20%. In the classification tasks for the Chapman, PTB-XL, and CPSC2018 databases, compared to existing work, the accuracies in linear probe are increased by at least 2.49%, 1.09%, and 2.61%, respectively. In addition, an SoC (system-on-a-chip) based heterogeneous deployment scheme is designed. It is 13.69–14.26 times faster in inference than pure software deployment and enables real-time detection of ECG signals. The proposed method provides a good way to utilize unlabeled ECG, and the designed deployment scheme has great potential for medical applications.