Global Feature Extractor Research Articles

DecoupleCLIP: A Novel Cross-Modality Decouple Model for Painting Captioning

Image captioning aims to describe the content in an image, which plays a critical role in image understanding. Existing methods tend to generate the text for more distinct natural images. These models can not be well for paintings containing more abstract meaning due to the limitation of objective parsing without related knowledge. To alleviate, we propose a novel cross-modality decouple model to generate the objective and subjective parsing separately. Concretely, we propose to encode both subjective semantic and implied knowledge contained in the paintings. The key point of our framework is decoupled CLIP-based branches (DecoupleCLIP). For the objective caption branch, we utilize the CLIP model as the global feature extractor and construct a feature fusion module for global clues. Based on the objective caption branch structure, we add a multimodal fusion module called the artistic conception branch. In this way, the objective captions can constrain artistic conception content. We conduct extensive experiments to demonstrate our DecoupleCLIP’s superior ability over our new dataset. Our model achieves nearly 2% improvement over other comparison models on CIDEr.

Multi-view brain functional connectivity and hierarchical fusion for EEG-based emotion recognition

Emotional states evolve gradually from their onset to full manifestation, reflected in changes in brain functional connectivity. Existing research often focuses only on the final emotional state, missing the rich, dynamic information of emotional evolution. This paper proposes a novel network framework leveraging global and local brain functional connectivity through a multi-view approach. We introduce a global and local connectivity modeling method using electroencephalogram signals to simulate brain connectivity changes across time and specific moments. A residual-based local detail feature extractor combined with a coordinate attention module captures long-range dependencies and maintains positional accuracy. A global feature extractor is also designed for the effective extraction of emotional information. Finally, a hierarchical multi-scale cross-attention feature fusion module integrates features across different levels to enhance emotion recognition performance and robustness. Extensive experiments on three publicly available datasets demonstrate the framework’s superiority and generalization capability.

GLCV-NET: An automatic diagnosis system for advanced liver fibrosis using global–local cross view in B-mode ultrasound images

Background and objective: Advanced liver fibrosis is a critical stage in the evaluation of chronic liver disease (CLD), holding clinical significance in the development of treatment strategies and estimating the disease progression. Methods:This paper proposes an innovative Global–Local Cross-View Network (GLCV-Net) for the automatic diagnosis of advanced liver fibrosis from ultrasound (US) B-mode images. The proposed method consists of three main components: 1. A Segmentation-enhanced Global Hybrid Feature Extractor for segmenting the liver parenchyma and extracting global features; 2. A Heatmap-weighted Local Feature Extractor for selecting candidate regions and automatically identifying suspicious areas to construct local features; 3. A Scale-adaptive Fusion Module to balance the contributions of global and local scales in evaluating advanced liver fibrosis. Results:The predictive performance of the model was validated on an internal dataset of 1003 chronic liver disease (CLD) patients and an external dataset of 46 CLD patients, both subjected to liver fibrosis staging through pathological assessment. On the internal dataset, GLCV-Net achieved 86.9% accuracy, 85.0% recall, 85.4% precision, and 85.2% F1-score. Further validation on the external dataset confirmed its robustness, with scores of 86.1% in accuracy, 83.1% in recall, 80.8% in precision, and 81.9% in F1-score. Conclusion:These results underscore the GLCV-Net’s potential as a promising approach for non-invasively and accurately diagnosing advanced liver fibrosis in CLD patients, breaking through the limitations of traditional methods by integrating global and local information of liver fibrosis, significantly enhancing diagnostic accuracy.

A personalized federated meta-learning method for intelligent and privacy-preserving fault diagnosis

Intelligent fault diagnosis methods have made significant progress during the past decade, often following a centralized training paradigm that gathers data from multiple industrial users, which results in high acquisition costs and raise data privacy concerns. Federated Learning (FL) provides a promising solution to intrinsically protect data privacy by jointly training a shared global model without sharing individual data, but the varying data distributions among clients pose challenges for diagnostic performance. To address these problems, a novel FL framework called federated meta-learning based on fine-grained classifier reconstruction (FedFGCR) is presented in this paper. First, an adaptive interpolation strategy based on meta-learning is designed to compute the optimal combination of global and local feature extractors as the initialization of the local training. Additionally, we introduce a new regularization term that allows the local model to leverage global data, enhancing model generalization by incorporating global semantic feature information. Furthermore, a fine-grained classifier reconstruction method is proposed to improve the personalized diagnosis ability for local samples. Comparative experiments with mainstream FL-based fault diagnosis methods have been conducted, demonstrating FedFGCR’s superiority with accuracy gains ranging from 0.96% to 5.41% on the CWRU dataset and from 5.01% to 9.95% on the KAIST dataset.

Multi-source subdomain negative transfer suppression and multiple pseudo-labels guidance alignment: A method for fault diagnosis under cross-working conditions

Extensive researches have been conducted on transfer learning based fault diagnosis. However, negative information transfer may arise due to significant differences in the subdomain distribution across multiple source domains (MSDs). Most existing methods focus solely on the impact of subdomains from a single source domain (SSD) on the target domain (TD). Therefore, this paper proposed a novel multi-stage alignment multi-source subdomain adaptation (MAMSA) method. The global feature extractor is designed to extract domain-invariant features. Three domain-specific feature extractors capture high-level fault features from different domains with a customized adaptation strategy, which combines adversarial learning and distribution alignment based on multiple pseudo-label-guided local maximum mean discrepancy (MP-LMMD) to learn subdomain-invariant features. MP-LMMD utilizes pseudo-labels generated from all classifiers in the TD to guide the alignment of subdomains, suppressing negative transfer from the source domains (SDs). The experimental results indicate that the MAMSA method has excellent capabilities to suppress negative transfer, and the diagnostic performance can be greatly promoted with MAMSA under cross-working conditions.

GDM-depth: Leveraging global dependency modelling for self-supervised indoor depth estimation

Self-supervised depth estimation algorithms eschew depth ground truth and employ the convolutional U-Net with a fixed receptive field which confines its focus primarily to nearby spatial distances. These factors obscure adequate supervision during image reconstruction, consequently hindering accurate depth estimation, particularly in complex indoor scenes. The pure transformer framework can perform global modelling to provide more semantic information. However, the cost is significant. To tackle these challenges, we introduce GDM-Depth, which utilizes global dependency modelling to offer more precise depth guidance from the network itself. Initially, we propose integrating learnable tree filters with unary terms, leveraging the structural properties of spanning trees to facilitate efficient long-range interactions. Subsequently, instead of replacing the convolutional framework entirely, we employ the transformer to design a scale-aware global feature extractor, establishing global relationships among local features at various scales, achieving both efficiency and cost-effectiveness. Furthermore, inter-class disparities between depth global and local features are observed. To address this issue, we introduce the global feature injector to further enhance the representation. GDM-Depth's effectiveness is demonstrated on the NYUv2, ScanNet, and InteriorNet depth datasets, achieving impressive test set performances of 87.2%, 83.1%, and 76.1% in key indicators δ<0.125, respectively.

Multi-view depth estimation based on multi-feature aggregation for 3D reconstruction

Robust 3D reconstruction depends critically on the accuracy of depth maps, and multi-view depth estimate is greatly aided by Multi-View Stereo (MVS). Nevertheless, challenges still exist, particularly in accurately estimating depth in regions with low texture and occlusion edges. We introduce MFA-MVSNet, a novel Multi-Feature Aggregation Multi-View Stereo Network that aggregates diverse features in order to address these issues. The Local Feature Extractor (LFE) is incorporated into MFA-MVSNet to adaptively capture local features across various receptive fields. It also incorporates a Multi-feature Fusion Module (MFM) to combine local features with global features extracted by the Global Feature Extractor (GFE). We utilize neural edge information in a CNN-based Depth Refinement Module (DRM) to iteratively filter the depth map to address depth errors in low-textured regions and occlusion edges. To further improve network robustness and training efficiency, we also introduce a Region Perception Loss (RPL) to lessen the impact of both easily matched and mismatched areas. On both the DTU and BlendedMVS datasets, experimental results demonstrate that MFA-MVSNet outperforms recent advanced methods in terms of depth map quality. Additionally, we incorporate MFA-MVSNet into a multi-view 3D reconstruction pipeline, utilizing its depth maps to enhance downstream tasks such as point cloud reconstruction and neural radiance fields (NeRF), leading to improved 3D reconstruction performance.

MTJNet: Multi-task joint learning network for advancing medicinal plant and leaf classification

Identifying and classifying medicinal plants is crucial for pharmaceutical research, traditional medicine, and biodiversity conservation. However, manual classification is time-consuming, error-prone, and requires expert knowledge. This problem statement highlights the need for an automated, efficient, and accurate method by which to distinguish between different species and ensure the quality and safety of medicinal plant leaves. The state-of-the-art methods employ only leaf or plant images for the classification; however, these methods work in particular scenarios (only leaf or plant). In this study, we first integrate the dataset of medicinal plants with corresponding leaves to develop a generalizable model and then apply pre-processing techniques (using vein morphometric features and Sobel edge detection) to highlight the significant characteristics in the images. Then, we propose a novel multi-task joint learning network (MTJNet) for classifying medicinal plant leaves. The proposed MTJNet includes local and global feature extractors that retrieve prominent features for robust classification. Further, we blend the features extracted from the local and global networks to enhance the contextual features. Last, we assign the embedded features to dense layers for additional feature extraction and classification. To evaluate the proposed MTJNet, we employ an Indian medicinal leaf dataset and achieve a precision of 99.60%, a recall of 99.62%, an accuracy of 99.71%, and an F1 score of 99.58%. The experimental results show that the MTJNet statistically defeats prevalent models, thereby proving its effectiveness for medical and industrial applications.

A Real-Time License Plate Detection and Recognition Model in Unconstrained Scenarios.

Accurate and fast recognition of vehicle license plates from natural scene images is a crucial and challenging task. Existing methods can recognize license plates in simple scenarios, but their performance degrades significantly in complex environments. A novel license plate detection and recognition model YOLOv5-PDLPR is proposed, which employs YOLOv5 target detection algorithm in the license plate detection part and uses the PDLPR algorithm proposed in this paper in the license plate recognition part. The PDLPR algorithm is mainly designed as follows: (1) A Multi-Head Attention mechanism is used to accurately recognize individual characters. (2) A global feature extractor network is designed to improve the completeness of the network for feature extraction. (3) The latest parallel decoder architecture is adopted to improve the inference efficiency. The experimental results show that the proposed algorithm has better accuracy and speed than the comparison algorithms, can achieve real-time recognition, and has high efficiency and robustness in complex scenes.

FIRCNet: Feature-based image reconstruction with classification learning for person re-identification

As an important task in computer vision, the person re-identification problem has two main approaches, identification and verification, in solving this problem. This paper proposed a network associating these two separate methods in training, including a reconstructive model. This design encompasses three core modules: a global feature extractor, a reconstructive decoder, and a classification network. Thus, the total loss function seamlessly integrates triplet loss, cross-entropy loss, and a reconstruction loss, optimizing the network for feature discriminability, accurate identity classification, and faithful image reconstruction concurrently. The global feature plays a pivotal role in both the training and testing phases, aiding in metric learning and ensuring a distinctive representation of identities. To train the decoder and the backbone network, which collectively forms the extractor, simultaneously, the reconstructive model as a part of a wide-range autoencoder can recover the original image from extracted features. Subsequently, the classification module classifies these features, assigning them to distinct person IDs, thereby aiding in precise identity recognition. Experimental results on three major re-id datasets give a demonstration of the notable improvement in the backbone network and the advantages of this approach compared to the state-of-the-art methods.

Integrating category-related key regions with a dual-stream network for remote sensing scene classification

Remote sensing image scene classification has made great progress with deep learning. Due to complex backgrounds and the large number of objects with inhomogeneous sizes, remote sensing image scene classification still remains challenging. In fact, only a few parts of regions are expected to be representative of the scene. In this paper, we propose a dual-stream framework for remote sensing classification with the help of category-related regions. Swin Transformer-based global feature extractor is used to produce coarse classification results. A category-related key region localization module is developed to extract the representative region from the whole image. The ResNet50 is employed as a local feature extractor to mine critical features from prominent regions. Finally, the classification results is obtained with the shared weighted classify head on global-local features. Extensive experiments show that our method provides a promising approach for key region locations and performs best on three public remote sensing datasets.

Multi-node knowledge graph assisted distributed fault detection for large-scale industrial processes based on graph attention network and bidirectional LSTMs

Modern industrial processes are characterized by extensive, multiple operation units, and strong coupled correlation of subsystems. Fault detection of large-scale processes is still a challenging problem, especially for tandem plant-wide processes in multiple fields such as water treatment process. In this paper, a novel distributed graph attention network-bidirectional long short-term memory (D-GATBLSTM) fault detection model is proposed for large-scale industrial processes. Firstly, a multi-node knowledge graph (MNKG) is constructed using a joint data and knowledge driven strategy. Secondly, for large-scale industrial process, a global feature extractor of graph attention networks (GATs) is constructed, on the basis of which, sub-blocks are decomposed based on MNKG. Then, local feature extractors of bidirectional long short-term memory (Bi-LSTM) for each sub-block are constructed, in which correlations among multiple sub-blocks are considered. Finally, a multi-subblock fusion collaborative prediction model is constructed and the comprehensive fault detection results are given by the grid search method. The effectiveness of our D-GATBLSTM is exemplified in a secure water treatment process case, where it outperforms baseline models compared, with 27% improvement in precision, 15% increase in recall, and overall F-score enhancement of 0.22.

A heterogeneous attention fusion mechanism for the cross-environment scene classification of the home service robot

There have been many methods to improve the capacity of scene classification of service robots. However, most of them are proposed from a technical standpoint but without reference to any cognitive principle of the brain, and furthermore, from design to evaluation, the particularity of the robot task is still not fully considered, such as cross-environment generalization, explicit semantic preservation and interpretation. Thus, the scene cognitive behavior of robots is far from humans, and their environmental adaptability is still poor. It is difficult to complete learning place concepts from discrete fragments and then continuously perceiving them with a limited view in unvisited spaces. Inspired by the recent findings from neuroscience, an attention-based global and object attribute fusion mechanism (AGOFM for short) constructed by three parts is proposed to overcome these deficiencies. In the global attribute part, a global feature extractor and a sequence context extractor are used to generate the holistic feature. The involved context integrates limited views to form an overall impression of a scene for guiding attention. In the object attribute part, a novel object vector is proposed. It simultaneously involves the detected object quantity, category and confidence information, which are all related to the vector index and high-level semantics. In the attention generation part, two sorted top-X characteristics deriving from the above two parts are fed into a fully connected (FC) network with batch normalization to generate effective attention. The attention weights are then applied to the batch normalized global and object vectors respectively, and subsequently, the two heterogeneous information are directly fused by another FC network to achieve scene classification. The policies for multi-learner fusion and frame rejection are also provided. Finally, a novel evaluation paradigm is proposed that the model is trained on a discrete prior dataset, and then the inference is tested on a traditional dataset and two robot view datasets. This simulates the cross-environment situation. Under such severe conditions, the results demonstrate that the proposed method outperforms several popular methods.

Uni4Eye++: A General Masked Image Modeling Multi-modal Pre-training Framework for Ophthalmic Image Classification and Segmentation.

A large-scale labeled dataset is a key factor for the success of supervised deep learning in most ophthalmic image analysis scenarios. However, limited annotated data is very common in ophthalmic image analysis, since manual annotation is time-consuming and labor-intensive. Self-supervised learning (SSL) methods bring huge opportunities for better utilizing unlabeled data, as they do not require massive annotations. To utilize as many unlabeled ophthalmic images as possible, it is necessary to break the dimension barrier, simultaneously making use of both 2D and 3D images as well as alleviating the issue of catastrophic forgetting. In this paper, we propose a universal self-supervised Transformer framework named Uni4Eye++ to discover the intrinsic image characteristic and capture domain-specific feature embedding in ophthalmic images. Uni4Eye++ can serve as a global feature extractor, which builds its basis on a Masked Image Modeling task with a Vision Transformer architecture. On the basis of our previous work Uni4Eye, we further employ an image entropy guided masking strategy to reconstruct more-informative patches and a dynamic head generator module to alleviate modality confusion. We evaluate the performance of our pre-trained Uni4Eye++ encoder by fine-tuning it on multiple downstream ophthalmic image classification and segmentation tasks. The superiority of Uni4Eye++ is successfully established through comparisons to other state-of-the-art SSL pre-training methods. Our code is available at Github1.

DaylilyNet: A Multi-Task Learning Method for Daylily Leaf Disease Detection.

Timely detection and management of daylily diseases are crucial to prevent yield reduction. However, detection models often struggle with handling the interference of complex backgrounds, leading to low accuracy, especially in detecting small targets. To address this problem, we propose DaylilyNet, an object detection algorithm that uses multi-task learning to optimize the detection process. By incorporating a semantic segmentation loss function, the model focuses its attention on diseased leaf regions, while a spatial global feature extractor enhances interactions between leaf and background areas. Additionally, a feature alignment module improves localization accuracy by mitigating feature misalignment. To investigate the impact of information loss on model detection performance, we created two datasets. One dataset, referred to as the 'sliding window dataset', was obtained by splitting the original-resolution images using a sliding window. The other dataset, known as the 'non-sliding window dataset', was obtained by downsampling the images. Experimental results in the 'sliding window dataset' and the 'non-sliding window dataset' demonstrate that DaylilyNet outperforms YOLOv5-L in mAP@0.5 by 5.2% and 4.0%, while reducing parameters and time cost. Compared to other models, our model maintains an advantage even in scenarios where there is missing information in the training dataset.

Two-Stream Network One-Class Classification Model for Defect Inspections.

Defect inspection is important to ensure consistent quality and efficiency in industrial manufacturing. Recently, machine vision systems integrating artificial intelligence (AI)-based inspection algorithms have exhibited promising performance in various applications, but practically, they often suffer from data imbalance. This paper proposes a defect inspection method using a one-class classification (OCC) model to deal with imbalanced datasets. A two-stream network architecture consisting of global and local feature extractor networks is presented, which can alleviate the representation collapse problem of OCC. By combining an object-oriented invariant feature vector with a training-data-oriented local feature vector, the proposed two-stream network model prevents the decision boundary from collapsing to the training dataset and obtains an appropriate decision boundary. The performance of the proposed model is demonstrated in the practical application of automotive-airbag bracket-welding defect inspection. The effects of the classification layer and two-stream network architecture on the overall inspection accuracy were clarified by using image samples collected in a controlled laboratory environment and from a production site. The results are compared with those of a previous classification model, demonstrating that the proposed model can improve the accuracy, precision, and F1 score by up to 8.19%, 10.74%, and 4.02%, respectively.

Robust Feature-Guided Generative Adversarial Network for Aerial Image Semantic Segmentation against Backdoor Attacks

Profiting from the powerful feature extraction and representation capabilities of deep learning (DL), aerial image semantic segmentation based on deep neural networks (DNNs) has achieved remarkable success in recent years. Nevertheless, the security and robustness of DNNs deserve attention when dealing with safety-critical earth observation tasks. As a typical attack pattern in adversarial machine learning (AML), backdoor attacks intend to embed hidden triggers in DNNs by poisoning training data. The attacked DNNs behave normally on benign samples, but when the hidden trigger is activated, its prediction is modified to a specified target label. In this article, we systematically assess the threat of backdoor attacks to aerial image semantic segmentation tasks. To defend against backdoor attacks and maintain better semantic segmentation accuracy, we construct a novel robust generative adversarial network (RFGAN). Motivated by the sensitivity of human visual systems to global and edge information in images, RFGAN designs the robust global feature extractor (RobGF) and the robust edge feature extractor (RobEF) that force DNNs to learn global and edge features. Then, RFGAN uses robust global and edge features as guidance to obtain benign samples by the constructed generator, and the discriminator to obtain semantic segmentation results. Our method is the first attempt to address the backdoor threat to aerial image semantic segmentation by constructing the robust DNNs model architecture. Extensive experiments on real-world scenes aerial image benchmark datasets demonstrate that the constructed RFGAN can effectively defend against backdoor attacks and achieve better semantic segmentation results compared with the existing state-of-the-art methods.

GIFDTI: Prediction of drug-target interactions based on global molecular and intermolecular interaction representation learning.

Drug discovery and drug repurposing often rely on the successful prediction of drug-target interactions (DTIs). Recent advances have shown great promise in applying deep learning to drug-target interaction prediction. One challenge in building deep learning-based models is to adequately represent drugs and proteins that encompass the fundamental local chemical environments and long-distance information among amino acids of proteins (or atoms of drugs). Another challenge is to efficiently model the intermolecular interactions between drugs and proteins, which plays vital roles in the DTIs. To this end, we propose a novel model, GIFDTI, which consists of three key components: the sequence feature extractor (CNNFormer), the global molecular feature extractor (GF), and the intermolecular interaction modeling module (IIF). Specifically, CNNFormer incorporates CNN and Transformer to capture the local patterns and encode the long-distance relationship among tokens (atoms or amino acids) in a sequence. Then, GF and IIF extract the global molecular features and the intermolecular interaction features, respectively. We evaluate GIFDTI on six realistic evaluation strategies and the results show it improves DTI prediction performance compared to state-of-the-art methods. Moreover, case studies confirm that our model can be a useful tool to accurately yield low-cost DTIs. The codes of GIFDTI are available at https://github.com/zhaoqichang/GIFDTI.

Global Perception-Based Robust Parking Space Detection Using a Low-Cost Camera

In the city, parking space detection can help car owners save time in finding parking spaces, and it can also provide help for autonomous parking. However, existing parking space detection methods based on embedded geomagnetic sensors are usually costly and complex to deploy in a large area. To reduce costs while obtaining sufficient performance, it is desirable to detecting parking spaces with cameras. We adopt a low-cost visual method to solve the above problems, using a convolutional neural network to achieve the parking space classification. We propose a general module called Global Perceptual Feature Extractor (GPFE) based on transformer to achieve global attention, which can be easily combined with other classification networks to improve the accuracy and robustness of the entire system, and alleviate the impact of illumination changing and other issues. We verify the proposed method with other methods on three public parking space detection datasets (CNREXT, PKLot and ACPDS). The results show that the accuracy of the network combined with the GPFE module has been generally improved, especially on scenes with low and high illumination, which verifies the robustness and performance of the module.

Stereo RGB and Deeper LIDAR-Based Network for 3D Object Detection in Autonomous Driving

3D object detection has become an emerging task in autonomous driving scenarios. Most of previous works process 3D point clouds using either projection-based or voxel-based models. However, both approaches contain some drawbacks. The voxel-based methods lack semantic information, while the projection-based methods suffer from numerous spatial information loss when projected to different views. In this paper, we propose the Stereo RGB and Deeper LIDAR (SRDL) framework which can utilize semantic and spatial information simultaneously such that the performance of network for 3D object detection can be improved naturally. Specifically, the network generates candidate boxes from stereo pairs and combines different region-wise features using a deep fusion scheme. The stereo strategy offers more information for prediction compared with prior works. Then, several local and global feature extractors are stacked in the segmentation module to capture richer deep semantic geometric features from point clouds. After aligning the interior points with fused features, the proposed network refines the prediction in a more accurate manner and encodes the whole box in a novel compact method. The decent experimental results on the challenging KITTI detection benchmark demonstrate the effectiveness of utilizing both stereo images and point clouds for 3D object detection.