Global Branch Research Articles

Local-Global Feature Adaptive Fusion Network for Building Crack Detection.

Cracks represent one of the most common types of damage in building structures and it is crucial to detect cracks in a timely manner to maintain the safety of the buildings. In general, tiny cracks require focusing on local detail information while complex long cracks and cracks similar to the background require more global features for detection. Therefore, it is necessary for crack detection to effectively integrate local and global information. Focusing on this, a local-global feature adaptive fusion network (LGFAF-Net) is proposed. Specifically, we introduce the VMamba encoder as the global feature extraction branch to capture global long-range dependencies. To enhance the ability of the network to acquire detailed information, the residual network is added as another local feature extraction branch, forming a dual-encoding network to enhance the performance of crack detection. In addition, a multi-feature adaptive fusion (MFAF) module is proposed to integrate local and global features from different branches and facilitate representative feature learning. Furthermore, we propose a building exterior wall crack dataset (BEWC) captured by unmanned aerial vehicles (UAVs) to evaluate the performance of the proposed method used to identify wall cracks. Other widely used public crack datasets are also utilized to verify the generalization of the method. Extensive experiments performed on three crack datasets demonstrate the effectiveness and superiority of the proposed method.

GLDBF: Global and local dual-branch fusion network for no-reference point cloud quality assessment

No-reference Point Cloud Quality Assessment (NR-PCQA) is a challenge in the field of media quality assessment, such as inability to accurately capture quality-related features due to the unique scattered structure of points and less considering global features and local features jointly in the existing no-reference PCQA metrics. To address these challenges, we propose a Global and Local Dual-Branch Fusion (GLDBF) network for no-reference point cloud quality assessment. Firstly, sparse convolution is used to extract the global quality feature of distorted Point Clouds (PCs). Secondly, graph weighted PointNet++ is proposed to extract the multi-level local features of point cloud, and the offset attention mechanism is further used to enhance local effective features. Transformer-based fusion module is also proposed to fuse multi-level local features. Finally, we joint the global and local dual branch fusion modules via multilayer perceptron to predict the quality score of distorted PCs. Experimental results show that the proposed algorithm can achieves state-of-the-art performance compared with existing methods in assessing the quality of distorted PCs.

FDT‐Net: Frequency‐Aware Dual‐Branch Transformer‐Based Optic Cup and Optic Disk Segmentation With Parallel Contour Information Mining and Uncertainty‐Guided Refinement

ABSTRACTAccurate segmentation of the optic cup and disc in fundus images is crucial for the prevention and diagnosis of glaucoma. However, challenges arise due to factors such as blood vessels, and mainstream networks often demonstrate limited capacity in extracting contour information. In this paper, we propose a segmentation framework named FDT‐Net, which is based on a frequency‐aware dual‐branch Transformer (FDBT) architecture with parallel contour information mining and uncertainty‐guided refinement. Specifically, we design a FDBT that operates in the frequency domain. This module leverages the inherent contextual awareness of Transformers and utilizes Discrete Cosine Transform (DCT) transformation to mitigate the impact of certain interference factors on segmentation. The FDBT comprises global and local branches that independently extract global and local information, thereby enhancing segmentation results. Moreover, to further mine additional contour information, this study develops the parallel contour information mining (PCIM) module to operate in parallel. These modules effectively capture more details from the edges of the optic cup and disc while avoiding mutual interference, thus optimizing segmentation performance in contour regions. Furthermore, we propose an uncertainty‐guided refinement (UGR) module, which generates and quantifies uncertainty mass and leverages it to enhance model performance based on subjective logic theory. The experimental results on two publicly available datasets demonstrate the superior performance and competitive advantages of our proposed FDT‐Net. The code for this project is available at https://github.com/Rookie49144/FDT‐Net.

Infrared image super-resolution reconstruction based on visible light image guidance and recursive fusion

Due to the limitations of hardware equipment, infrared images generally have problems such as low resolution, blurred details and poor visual quality during acquisition. Using visible light images to guide the super-resolution reconstruction of infrared images is an effective way to improve the resolution of infrared images. However, the imaging principles of visible light images and infrared images are different, resulting in differences in detail information between the two images, so problems such as blur and ghosting may occur during reconstruction. This paper proposes an infrared image super-resolution network based on visible light image guidance and recursive fusion. In this network, a flow Fourier residual module is designed, and modules of different depths are used to extract information of different frequencies in visible light images and infrared images, so that each module focuses on the appropriate frequency information. At the same time, a hybrid attention module is used to obtain detail information in multimodal images from channel and spatial perspectives, and fuse them in a complementary way, which helps to eliminate the generation of artifacts. On this basis, a global recursive fusion branch is designed to consider the correlation between multi-layer features, adaptively fuse multi-layer features, and generate clearer high-resolution infrared images. Experimental results show that compared with the comparative methods, this method performs better in objective evaluation indicators; in terms of subjective visual comparison, the images reconstructed by this method have clearer textures and fewer artifacts, and better object distinction in complex environments.

Normalized solutions of quasilinear Schrödinger equations with a general nonlinearity

We are concerned with solutions of the following quasilinear Schrödinger equations − div ( φ 2 ( u ) ∇ u ) + φ ( u ) φ ′ ( u ) | ∇ u | 2 + λ u = f ( u ) , x ∈ R N with prescribed mass ∫ R N u 2 d x = c , where N ⩾ 3, c > 0, λ ∈ R appears as the Lagrange multiplier and φ ∈ C 1 ( R , R + ). The nonlinearity f ∈ C ( R , R ) is allowed to be mass-subcritical, mass-critical and mass-supercritical at origin and infinity. Via a dual approach, the fixed point index and a global branch approach, we establish the existence of normalized solutions to the problem above. The results extend previous results by L. Jeanjean, J. J. Zhang and X.X. Zhong to the quasilinear case.

Dual-Stream Feature Collaboration Perception Network for Salient Object Detection in Remote Sensing Images

As the core technology of artificial intelligence, salient object detection (SOD) is an important approach to improve the analysis efficiency of remote sensing images by intelligently identifying key areas in images. However, existing methods that rely on a single strategy, convolution or Transformer, exhibit certain limitations in complex remote sensing scenarios. Therefore, we developed a Dual-Stream Feature Collaboration Perception Network (DCPNet) to enable the collaborative work and feature complementation of Transformer and CNN. First, we adopted a dual-branch feature extractor with strong local bias and long-range dependence characteristics to perform multi-scale feature extraction from remote sensing images. Then, we presented a Multi-path Complementary-aware Interaction Module (MCIM) to refine and fuse the feature representations of salient targets from the global and local branches, achieving fine-grained fusion and interactive alignment of dual-branch features. Finally, we proposed a Feature Weighting Balance Module (FWBM) to balance global and local features, preventing the model from overemphasizing global information at the expense of local details or from inadequately mining global cues due to excessive focus on local information. Extensive experiments on the EORSSD and ORSSD datasets demonstrated that DCPNet outperformed the current 19 state-of-the-art methods.

Bifurcation structure of steady states for a cooperative model with population flux by attractive transition

AbstractThis paper studies the steady states to a diffusive Lotka–Volterra cooperative model with population flux by attractive transition. The first result gives many bifurcation points on the branch of the positive constant solution under the weak cooperative condition. The second result shows every steady state approaches a solution of the scalar field equation as the coefficients of the flux tend to infinity. Indeed, the numerical simulation using pde2path exhibits the global bifurcation branch of the cooperative model with large population flux is near that of the scalar field equation.

Research on Person Re-Identification through Local and Global Attention Mechanisms and Combination Poolings.

This research proposes constructing a network used for person re-identification called MGNACP (Multiple Granularity Network with Attention Mechanisms and Combination Poolings). Based on the MGN (Multiple Granularity Network) that combines global and local features and the characteristics of the MGN branch, the MGNA (Multiple Granularity Network with Attentions) is designed by adding a channel attention mechanism to each global and local branch of the MGN. The MGNA, with attention mechanisms, learns the most identifiable information about global and local features to improve the person re-identification accuracy. Based on the constructed MGNA, a single pooling used in each branch is replaced by combination pooling to form MGNACP. The combination pooling parameters are the proportions of max pooling and average pooling in combination pooling. Through experiments, suitable combination pooling parameters are found, the advantages of max pooling and average pooling are preserved and enhanced, and the disadvantages of both types of pooling are overcome, so that poolings can achieve optimal results in MGNACP and improve the person re-identification accuracy. In experiments on the Market-1501 dataset, MGNACP achieved competitive experimental results; the values of mAP and top-1 are 88.82% and 95.46%. The experimental results demonstrate that MGNACP is a competitive person re-identification network, and that the attention mechanisms and combination poolings can significantly improve the person re-identification accuracy.

An Efficient Multi-Branch Attention Network for Person Re-Identification

Due to the absence of tailored designs that address challenges such as variations in scale, disparities in illumination, and instances of occlusion, the implementation of current person re-identification techniques remains challenging in practical applications. An Efficient Multi-Branch Attention Network over OSNet (EMANet) is proposed. The structure is composed of three parts, the global branch, relational branch, and global contrastive pooling branch, and corresponding features are obtained from different branches. With the attention mechanism, which focuses on important features, DAS attention evaluates the significance of learned features, awarding higher ratings to those that are deemed crucial and lower ratings to those that are considered distracting. This approach leads to an enhancement in identification accuracy by emphasizing important features while discounting the influence of distracting ones. Identity loss and adaptive sparse pairwise loss are used to efficiently facilitate the information interaction. In experiments on the Market-1501 mainstream dataset, EMANet exhibited high identification accuracies of 96.1% and 89.8% for Rank-1 and mAP, respectively. The results indicate the superiority and effectiveness of the proposed model.

A deep learning-based method for the detection and segmentation of breast masses in ultrasound images

Objective. Automated detection and segmentation of breast masses in ultrasound images are critical for breast cancer diagnosis, but remain challenging due to limited image quality and complex breast tissues. This study aims to develop a deep learning-based method that enables accurate breast mass detection and segmentation in ultrasound images. Approach. A novel convolutional neural network-based framework that combines the You Only Look Once (YOLO) v5 network and the Global-Local (GOLO) strategy was developed. First, YOLOv5 was applied to locate the mass regions of interest (ROIs). Second, a Global Local-Connected Multi-Scale Selection (GOLO-CMSS) network was developed to segment the masses. The GOLO-CMSS operated on both the entire images globally and mass ROIs locally, and then integrated the two branches for a final segmentation output. Particularly, in global branch, CMSS applied Multi-Scale Selection (MSS) modules to automatically adjust the receptive fields, and Multi-Input (MLI) modules to enable fusion of shallow and deep features at different resolutions. The USTC dataset containing 28 477 breast ultrasound images was collected for training and test. The proposed method was also tested on three public datasets, UDIAT, BUSI and TUH. The segmentation performance of GOLO-CMSS was compared with other networks and three experienced radiologists. Main results. YOLOv5 outperformed other detection models with average precisions of 99.41%, 95.15%, 93.69% and 96.42% on the USTC, UDIAT, BUSI and TUH datasets, respectively. The proposed GOLO-CMSS showed superior segmentation performance over other state-of-the-art networks, with Dice similarity coefficients (DSCs) of 93.19%, 88.56%, 87.58% and 90.37% on the USTC, UDIAT, BUSI and TUH datasets, respectively. The mean DSC between GOLO-CMSS and each radiologist was significantly better than that between radiologists (p < 0.001). Significance. Our proposed method can accurately detect and segment breast masses with a decent performance comparable to radiologists, highlighting its great potential for clinical implementation in breast ultrasound examination.

DSNet: dual-stream network for fine-grained ship classification in optical remote sensing images

ABSTRACT The fine-grained ship classification (FGSC) task in remote sensing aims to distinguish subordinate ship categories. Due to the high similarity among different subcategories and the significant variations in pose, scale, rotation, and color within the same subcategory in the fine-grained ship datasets, the FGSC task is extremely challenging. To alleviate these issues, a novel approach, named Dual-Stream Network (DSNet), is proposed in this paper for FGSC. DSNet consists of two sub-branches: the Local Residual Branch (LRB) and the Global Representation Branch (GRB). Specifically, the LRB effectively extracts the most discriminative local fine-grained features of the target using Full-scale Skip Connections (FSC) and Stripe Pooling Add (SPA) modules. The GRB captures global contextual information through Swin-Transformer (Swin-T), establishing remote dependencies of the target and extracting global coarse-grained features. Furthermore, a Lightweight Fusion (LF) block is designed in the feature fusion module to combine the feature information from the two branches. This ensures that the discriminative global and local information complement each other without overwhelming one another, thereby effectively improving the performance of the model. Extensive experiments and analyses demonstrate that our proposed method achieves state-of-the-art performance on two mainstream FGSC datasets in remote sensing.

Local and Global Context-Enhanced Lightweight CenterNet for PCB Surface Defect Detection.

Printed circuit board (PCB) surface defect detection is an essential part of the PCB manufacturing process. Currently, advanced CCD or CMOS sensors can capture high-resolution PCB images. However, the existing computer vision approaches for PCB surface defect detection require high computing effort, leading to insufficient efficiency. To this end, this article proposes a local and global context-enhanced lightweight CenterNet (LGCL-CenterNet) to detect PCB surface defects in real time. Specifically, we propose a two-branch lightweight vision transformer module with local and global attention, named LGT, as a complement to extract high-dimension features and leverage context-aware local enhancement after the backbone network. In the local branch, we utilize coordinate attention to aggregate more powerful features of PCB defects with different shapes. In the global branch, Bi-Level Routing Attention with pooling is used to capture long-distance pixel interactions with limited computational cost. Furthermore, a Path Aggregation Network (PANet) feature fusion structure is incorporated to mitigate the loss of shallow features caused by the increase in model depth. Then, we design a lightweight prediction head by using depthwise separable convolutions, which further compresses the computational complexity and parameters while maintaining the detection capability of the model. In the experiment, the LGCL-CenterNet increased the mAP@0.5 by 2% and 1.4%, respectively, in comparison to CenterNet-ResNet18 and YOLOv8s. Meanwhile, our approach requires fewer model parameters (0.542M) than existing techniques. The results show that the proposed method improves both detection accuracy and inference speed and indicate that the LGCL-CenterNet has better real-time performance and robustness.

Meta-collaborative comparison for effective cross-domain few-shot learning

Recent advancements in cross-domain few-shot learning (CD-FSL) primarily focus on learning to compare global representations between query and support images for classification. However, due to the notorious cross-domain semantic gap, the ideal global representations can be totally different across domains, thereby solely learning to compare global representations is not sufficient to achieve effective generalization in challenging cases. To mitigate this problem, we present a Meta-collaborative Comparison Network (MeCo-Net) for CD-FSL, which imitates humans to recognize unfamiliar objects through collaborative comparison on both global and local representations. Following this idea, paralleling with a conventional global comparison branch, we additionally feed random crops of both query and support images into a feature encoder to separately extract their local representations. Subsequently, we associate these local representations across images through bipartite graph matching for local comparison. Thanks to the complementary global and local comparisons, we can obtain a more generalizable classifier for CD-FSL by meta-integrating them for final prediction. Experimental results on eight benchmarks demonstrate that the proposed model generalizes to multiple target domains with state-of-the-art performance without the need for fine-tuning.

Relational-branchformer: Novel framework for audio-visual speech recognition

This study embraced the state-of-the-art Branchformer series architecture within the realm of automatic speech recognition, supplanting the widely utilized Conformer architecture. This substitution offers an innovative remedy tailored to audio-visual speech recognition tasks. Building upon the Branchformer architecture, enhancements were made, culminating in the proposal of the Relational-Branchformer (R-Branchformer). The convolutional attention relation module was innovatively incorporated to augment the connectivity between the local and global branches by meticulously considering their interrelations and interplays. Consequently, this module facilitates the mutual embedding of local and global contextual information, ultimately leading to a substantial enhancement in model performance. Our model was grounded in the utilization of the connectionist temporal classification (CTC) loss, wherein intermediate CTC losses were incorporated between blocks. Moreover, through the reference and enhancement of the gated interlayer collaboration module, which superseded the inter CTC module, the conditional independence assumption intrinsic to the CTC model was effectively relaxed. As a consequence, this augmentation markedly bolstered the overall performance of our model. Furthermore, the audio-visual output enhancement module was proposed, which adeptly assimilates information from both audio and visual modalities to enrich the representation of audio-visual information. Consequently, the R-Branchformer model achieved remarkable word error rates of 1.7% and 1.5% on the LRS2 and LRS3 test sets, respectively, exemplifying its state-of-the-art performance in audio-visual speech recognition tasks.

Two-stage deep learning classification for diabetic retinopathy using gradient weighted class activation mapping

The fundus images of patients with Diabetic Retinopathy (DR) often display numerous lesions scattered across the retina. Current methods typically utilize the entire image for network learning, which has limitations since DR abnormalities are usually localized. Training Convolutional Neural Networks (CNNs) on global images can be challenging due to excessive noise. Therefore, it's crucial to enhance the visibility of important regions and focus the recognition system on them to improve accuracy. This study investigates the task of classifying the severity of diabetic retinopathy in eye fundus images by employing appropriate preprocessing techniques to enhance image quality. We propose a novel two-branch attention-guided convolutional neural network (AG-CNN) with initial image preprocessing to address these issues. The AG-CNN initially establishes overall attention to the entire image with the global branch and then incorporates a local branch to compensate for any lost discriminative cues. We conduct extensive experiments using the APTOS 2019 DR dataset. Our baseline model, DenseNet-121, achieves average accuracy/AUC values of 0.9746/0.995, respectively. Upon integrating the local branch, the AG-CNN improves the average accuracy/AUC to 0.9848/0.998, representing a significant advancement in state-of-the-art performance within the field.

Local-aware global attention network for person re-identification based on body and hand images

Learning representative, robust and discriminative information from images is essential for effective person re-identification (Re-Id). In this paper, we propose a compound approach for end-to-end discriminative deep feature learning for person Re-Id based on both body and hand images. We carefully design the Local-Aware Global Attention Network (LAGA-Net), a multi-branch deep network architecture consisting of one branch for spatial attention, one branch for channel attention, one branch for global feature representations and another branch for local feature representations. The attention branches focus on the relevant features of the image while suppressing the irrelevant backgrounds. The global and local branches intends to capture global context and fine-grained information, respectively. A set of ablation study shows that each component contributes to the increased performance of the LAGA-Net. Extensive evaluations on four popular body-based person Re-Id benchmarks and two publicly available hand datasets demonstrate that our proposed method consistently outperforms existing state-of-the-art methods.

LBRT: Local-Information-Refined Transformer for Image Copy-Move Forgery Detection.

The current deep learning methods for copy-move forgery detection (CMFD) are mostly based on deep convolutional neural networks, which frequently discard a large amount of detail information throughout convolutional feature extraction and have poor long-range information extraction capabilities. The Transformer structure is adept at modeling global context information, but the patch-wise self-attention calculation still neglects the extraction of details in local regions that have been tampered with. A local-information-refined dual-branch network, LBRT (Local Branch Refinement Transformer), is designed in this study. It performs Transformer encoding on the global patches segmented from the image and local patches re-segmented from the global patches using a global modeling branch and a local refinement branch, respectively. The self-attention features from both branches are precisely fused, and the fused feature map is then up-sampled and decoded. Therefore, LBRT considers both global semantic information modeling and local detail information refinement. The experimental results show that LBRT outperforms several state-of-the-art CMFD methods on the USCISI dataset, CASIA CMFD dataset, and DEFACTO CMFD dataset.

Local context attention learning for fine-grained scene graph generation

Fine-grained scene graph generation aims to parse the objects and their fine-grained relationships within scenes. Despite the significant progress in recent years, their performance is still limited by two major issues: (1) ambiguous perception under a global view; (2) the lack of reliable, fine-grained annotations. We argue that understanding the local context is important in addressing the two issues. However, previous works often overlook it, which limits their effectiveness in fine-grained scene graph generation. To tackle this challenge, we introduce a Local-context Attention Learning method that concentrates on local context and can generate high-reliability, fine-grained annotations. It comprises two components: (1) The Fine-grained Location Attention Network (FLAN), a multi-branch network that encompasses global and local branches, can attend to local informative context and perceive granularity levels in different regions, thereby adaptively enhancing the learning of fine-grained locations. (2) The Fine-grained Location Label Transfer (FLLT) method identifies coarse-grained labels inconsistent with the local context and determines which labels should be transferred through the global confidence thresholding strategy, finally transferring them to reliable local context-consistent fine-grained ones. Experiments conducted on the Visual Genome, OpenImage, and GQA-200 datasets show that the proposed methods achieve significant improvements on the fine-grained scene graph generation task. By addressing the challenge mentioned above, our method also achieves state-of-the-art performances on the three datasets.

Dual-stream multi-dependency graph neural network enables precise cancer survival analysis

Histopathology image-based survival prediction aims to provide a precise assessment of cancer prognosis and can inform personalized treatment decision-making in order to improve patient outcomes. However, existing methods cannot automatically model the complex correlations between numerous morphologically diverse patches in each whole slide image (WSI), thereby preventing them from achieving a more profound understanding and inference of the patient status. To address this, here we propose a novel deep learning framework, termed dual-stream multi-dependency graph neural network (DM-GNN), to enable precise cancer patient survival analysis. Specifically, DM-GNN is structured with the feature updating and global analysis branches to better model each WSI as two graphs based on morphological affinity and global co-activating dependencies. As these two dependencies depict each WSI from distinct but complementary perspectives, the two designed branches of DM-GNN can jointly achieve the multi-view modeling of complex correlations between the patches. Moreover, DM-GNN is also capable of boosting the utilization of dependency information during graph construction by introducing the affinity-guided attention recalibration module as the readout function. This novel module offers increased robustness against feature perturbation, thereby ensuring more reliable and stable predictions. Extensive benchmarking experiments on five TCGA datasets demonstrate that DM-GNN outperforms other state-of-the-art methods and offers interpretable prediction insights based on the morphological depiction of high-attention patches. Overall, DM-GNN represents a powerful and auxiliary tool for personalized cancer prognosis from histopathology images and has great potential to assist clinicians in making personalized treatment decisions and improving patient outcomes.

Omnidirectional image quality assessment with local–global vision transformers

With the rising popularity of omnidirectional images (ODIs) in virtual reality applications, the need for specialized image quality assessment (IQA) methods becomes increasingly critical. Traditional IQA approaches, designed for rectilinear images, often fail to evaluate ODIs accurately due to their 360-degree scene representation. Addressing this, we introduce the Local–Global Transformer for 360-degree Image Quality Assessment (LGT360IQ). This novel framework features dual branches tailored to mimic top-down and bottom-up visual attention mechanisms, adapted for the spherical characteristics of ODIs. The local branch processes tangent viewports from salient regions within the equirectangular projection image, extracting detailed features for granular quality assessment. In parallel, the global branch utilizes a task-dependent token sampling strategy for holistic image feature processing and quality score prediction. This integrated approach combines local and global information, offering an effective IQA method for ODIs. Our extensive evaluation across three benchmark ODI datasets, CVIQ, OIQA, and ODI, demonstrates LGT360IQ superior performance and establishes its role in advancing the field of IQA for omnidirectional images.