Discriminative Local Features Research Articles

Transformer-based descriptors with fine-grained region supervisions for visual place recognition

Visual place recognition is a fundamental component in autonomous systems and robotics, which is easily limited in the real world with different viewpoints and changes in appearance. Existing approaches to tackle this problem mainly rely on dominant CNN-based architectures, which are difficult to model global correlation. Most recently, there has emerged little work that focuses on the effectiveness of the Transformer in modeling long-range dependencies, but this strategy ignores local interactions thus fails to localize the really important regions. To address the above issue, this paper proposes an effective Transformer-based architecture that takes full advantages of the strengths of Transformer related to global context modeling and local specific region capturing. We first design a dual-level Transformer descriptor encoder to successively perform self-attention within local windows and global extent of the CNN feature map to obtain multi-scale spatial context, which combines local interaction and global information. Specifically, multi-layer classification tokens from the Transformer encoder are integrated to form the global image representation. Moreover, a Transformer-guided geometric verification module is introduced to leverage the strengths of the hierarchical Transformer’s inherent self-attention mechanism for fusing multi-level attention, which is employed to filter the output token to obtain key patches and associated attention weights for achieving spatial matching. Finally, we propose a descriptor refinement strategy that employs fine-grained region-level supervisions to further enhance the capability of the network to learn local discriminative features, which effectively alleviates the confusion caused from weak image-level labels. Extensive experiments on benchmark datasets show that our approach outperforms state-of-the-art methods with promising performance.

Registration‐enhanced multiple instance learning for cervical cancer whole slide image classification

AbstractHistopathological diagnosis is the golden standard for cancer diagnosis and grading. The past decade has witnessed great successes of vision‐based deep learning in computer‐aided cancer diagnosis. However, fully‐supervised methods require collecting large amounts of annotated data, which is a major challenge. In this research, we propose RMIL, a novel registration‐enhanced multiple instance learning pipeline, for cervical cancer biopsy grading of whole slide images as well as patch‐level regions. It requires only slide‐level annotations and a small amount of patch‐level annotations for training. To enhance it, an efficient registration method is implemented to generate more labeled patches with discriminative local features as data augmentations. In addition, attention mechanisms and self‐supervised learning are introduced to enhance feature extraction. Experiments showed that its classification accuracy is increased by 11.11%, compared to the vanilla multiple instance learning methods. The method has great potential for transferring to other scenarios.

LET-Net: locally enhanced transformer network for medical image segmentation

AbstractMedical image segmentation has attracted increasing attention due to its practical clinical requirements. However, the prevalence of small targets still poses great challenges for accurate segmentation. In this paper, we propose a novel locally enhanced transformer network (LET-Net) that combines the strengths of transformer and convolution to address this issue. LET-Net utilizes a pyramid vision transformer as its encoder and is further equipped with two novel modules to learn more powerful feature representation. Specifically, we design a feature-aligned local enhancement module, which encourages discriminative local feature learning on the condition of adjacent-level feature alignment. Moreover, to effectively recover high-resolution spatial information, we apply a newly designed progressive local-induced decoder. This decoder contains three cascaded local reconstruction and refinement modules that dynamically guide the upsampling of high-level features by their adaptive reconstruction kernels and further enhance feature representation through a split-attention mechanism. Additionally, to address the severe pixel imbalance for small targets, we design a mutual information loss that maximizes task-relevant information while eliminating task-irrelevant noises. Experimental results demonstrate that our LET-Net provides more effective support for small target segmentation and achieves state-of-the-art performance in polyp and breast lesion segmentation tasks.

Fine-Grained Image Retrieval via Object Localization

In this paper, a network consisting of an object localization module and a discriminative feature extraction module is designed for fine-grained image retrieval (FGIR). In order to reduce the interference of complex backgrounds, the object localization module is introduced into the network before feature extraction. By selecting the convolutional feature descriptors, the main object is separated from the background, and thus, most of the interference is filtered out. Further, in order to improve the overall performance of the network, a discriminative filter bank is introduced into the network as the local feature detector. Hence, the local discriminative features can be extracted directly from the original feature map. The experimental results based on the CUB-200-2011 and Cars-196 datasets demonstrate that the proposed method can improve the performance of FGIR.

Facial expression recognition network with slow convolution and zero-parameter attention mechanism

Facial expression recognition (FER) has been successfully applied to the fields of automatic driving, classroom education, and many other industries. However, a lot of existing problems induced by different lighting, posture, and skin color make many difficulties in extracting effective features, resulting in low accuracy of FER. to address these issues, a novel slow evolution and zero parameter attention mechanism (SC-ZAM) is proposed to reliably extract discriminative feature from facial images for FER. In the proposed SC-ZAM, Slow convolution module is used to reduce the speed of image dimensionality reduction, extract considerably reliable facial features, and enhances the representation of these features to extract more effective global information of facial expressions. Meanwhile, zero-parameter attention modules are applied to enhance the focus on the local discriminative feature areas. The improved attention module focuses on highly distinguishable facial features and restrains the weight of non-facial areas to locate more detailed local facial areas in the global information extracted from the network. Experimental results demonstrate that our SC-ZAM method is robust against different FER tasks on our infant dataset and several public datasets, and is superior to other state-of-the-art methods.

Bi-Level Implicit Semantic Data Augmentation for Vehicle Re-Identification

Vehicle re-identification (Re-ID) aims at finding the target vehicle identity from multi-camera surveillance videos, which plays an important role in the intelligent transportation system (ITS). It suffers from the subtle discrepancy among vehicles from the same vehicle model and large variation across different viewpoints of the same vehicle. To enhance the robustness of Re-ID models, many methods exploit additional detection or segmentation models to extract discriminative local features. Some others employ data-driven methods to enrich the diversity of the training data, such as the data augmentation and 3D-based data generation, so that the Re-ID model can obtain stronger robustness against intra-class variations. However, these methods either rely on extra annotations or greatly increase the computational cost. In this paper, we propose the Bi-level Implicit semantic Data Augmentation (BIDA) framework to solve this problem from two aspects. (1) We implicitly augment the images semantically in the feature space according to the identity-level and superclass-level intra-class variations, which can generate more diverse semantic augmentations beyond the intra-identity variations. (2) We introduce the similarity ranking constraints on the augmented training set by extending the sample-wise triplet loss to the distribution-wise one, which can effectively reduce meaningless semantic transformations and improve the discrimination of the feature. We conduct extensive experiments on VeRi-776, VehicleID and Cityflow benchmarks to reveal the effectiveness of our method. And we achieve new state-of-the-art performance on VeRi-776.

Feature relocation network for fine-grained image classification

In fine-grained image classification, there are only very subtle differences between classes. It is challenging to learn local discriminative features and remove distractive features in fine-grained image classification. Existing fine-grained image classification methods learn discriminative feature mainly via manual part annotation or attention mechanisms. However, due to the large intraclass variance and interclass similarity, the discriminative information and distractive information still are not distinguished effectively. To address this problem, we propose a feature relocation network (FRe-Net) which takes advantage of the different natures of features learned from different stages of the network. Our network consists of a distractive feature learning module and a relocated high-level feature learning module. In the distractive feature learning module, we propose to exploit the difference between low-level features and high-level features to design a distractive loss Ldistractive, which guides the attention to locate distractive regions more accurately. In the relocated high-level feature learning module, we enhance the representing capacity of the middle-level feature via the attention module and subtract the distractive feature learned from the distractive feature learning module in order to learn more local discriminative features. In end-to-end model training, the distractive feature learning module and the relocated high-level feature learning module are beneficial to each other via joint optimization. We conducted comprehensive experiments on three benchmark datasets widely used in fine-grained image classification. The experimental results show that FRe-Net achieves state-of-the-art performance, which validates the effectiveness of FRe-Net.

HDA[formula omitted]L: Hierarchical Domain-Augmented Adaptive Learning for sketch-based 3D shape retrieval

The sketch-based 3D shape retrieval has been an active but challenging task for several decades. In this paper, we deeply analyze the challenges and propose a novel Hierarchical Domain-Augmented Adaptive Learning (HDA2L) for sketch-based 3D shape retrieval. The first notable challenge is the vast cross-modality discrepancies between sketches and 3D shapes. To this issue, the existing methods restrict the consistency of the final features by establishing a shared cross-domain loss but ignore the feature extraction process, resulting in a limited effect. We argue that the mutual information of the samples from the same class but different domains can provide an important cue to enhance common features captured in the feature extraction process. Thus, we design an Inter-Domain Augmented Network (Inter-DAN) by employing inter-domain feature correlation learning to capture cross-domain mutual information to learn augmented common global features for both sketches and 3D shapes. Another challenge is that the input sketch is various: it may be particularly abstract and contains only the overall outline of the target model, or it may be too sketchy and only contains some salient local regions of the target model. Though existing methods have demonstrated their capability to capture overall features, they always ignore the learning of local discriminative features and fail to adapt to the various changes of input sketches. To address this issue, we design an Intra-Domain Augmented Network (Intra-DAN) for sketches and 3D shapes, respectively, which learns augmented local discriminative features by adopting cascading cross-layer bilinear pooling operations. In addition, we design a Source-Agnostic Adversarial Network (SAAN) to accomplish the adaptive hierarchical domain features fusion, which forces the network to adaptively focus on more discriminative information from global features and local features and further adapt the diversity of the input sketches. The experiments on three benchmark datasets demonstrate that our method obtains superior retrieval performance than the state-of-the-art sketch-based 3D shape retrieval approaches.

GiT: Graph Interactive Transformer for Vehicle Re-identification.

Transformers are more and more popular in computer vision, which treat an image as a sequence of patches and learn robust global features from the sequence. However, pure transformers are not entirely suitable for vehicle re-identification because vehicle re-identification requires both robust global features and discriminative local features. For that, a graph interactive transformer (GiT) is proposed in this paper. In the macro view, a list of GiT blocks are stacked to build a vehicle re-identification model, in where graphs are to extract discriminative local features within patches and transformers are to extract robust global features among patches. In the micro view, graphs and transformers are in an interactive status, bringing effective cooperation between local and global features. Specifically, one current graph is embedded after the former level's graph and transformer, while the current transform is embedded after the current graph and the former level's transformer. In addition to the interaction between graphs and transforms, the graph is a newly-designed local correction graph, which learns discriminative local features within a patch by exploring nodes' relationships. Extensive experiments on three large-scale vehicle re-identification datasets demonstrate that our GiT method is superior to state-of-the-art vehicle re-identification approaches.

Feature Aggregation via Attention Mechanism for Visible-Thermal Person Re-Identification

Visible-thermal person re-identification (VT-ReID) is an image retrieval task that aims at matching the target pedestrian across the visible and thermal modalities. However, intra-class variations and cross-modality discrepancy degrade the performance of VT-ReID. Recent methods focus on extracting discriminative local features of each modality to alleviate the intra-class variations and cross-modality discrepancy, but these methods ignore semantic relations between the local features of two modalities, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , the spatial relations and channel relations. In this paper, we proposed a feature aggregation module (FAM) to enhance the correlation between local features including spatial dependencies and channel dependencies. Furthermore, FAM implements cross-modality feature aggregation on the enhanced features to reduce the cross-modality discrepancy. Moreover, we also proposed near neighbor cross-modality loss (NNCLoss) to mine feature consistency between modalities by constructing a cross-modality near neighbor set, which facilitates feature alignment between two modalities. Extensive experiments on two datasets demonstrate the superior performance of our approach over the existing state-of-the-arts.

Multi-Granularity Dilated Transformer for Lung Nodule Classification via Local Focus Scheme

Intelligent lung nodules classification is a meaningful and challenging research topic for early precaution of lung cancers, which aims to diagnose the malignancy of candidate nodules from the pulmonary computed tomography images. Nowadays, deep learning methods have made significant achievements in the medical field and promoted developments of lung nodules classification. Nevertheless, mainstream CNNs-based networks typically excel in learning coarse-grained local feature representations via stacked local-aware and weight-shared convolutions, and cannot practically model the long-range context interaction and the spatial dependencies. To tackle the above difficulties, we innovatively propose an effective Multi-Granularity Dilated Transformer to learn the long-range context relations, and explore fine-grained local details via the proposed Local Focus Scheme. Specifically, we delicately design a novel Deformable Dilated Transformer to incorporate diverse contextual information with self-attention for learning long-range global spatial dependencies. Moreover, numerous investigations indicate that local details are extremely crucial to classify indistinguishable lung nodules. Thus, we propose the Local Focus Scheme to focus on the more discriminative local features by modeling channel-wise grouped topology. Consequently, the Multi-Granularity Dilated Transformer is constructed by leveraging the Local Focus Scheme to guide the Deformable Dilated Transformer for learning fine-grained local cues. Experimental results on the mainstream benchmark LIDC-IDRI demonstrate the superiority of our model compared with the state-of-the-art methods.

Hybrid routing transformer for zero-shot learning

Zero-shot learning (ZSL) aims to learn models that can recognize unseen image semantics based on the training of data with seen semantics. Recent studies either leverage the global image features or mine discriminative local patch features to associate the extracted visual features to the semantic attributes. However, due to the lack of the necessary top-down guidance and semantic alignment for ensuring the model attend to the real attribute-correlation regions, these methods still encounter a significant semantic gap between the visual modality and the attribute modality, which makes their prediction on unseen semantics unreliable. To solve this problem, this paper establishes a novel transformer encoder-decoder model, called hybrid routing transformer (HRT). In HRT encoder, we embed an active attention, which is constructed by both the bottom-up and the top-down dynamic routing pathways to generate the attribute-aligned visual feature. While in HRT decoder, we use static routing to calculate the correlation among the attribute-aligned visual features, the corresponding attribute semantics, and the class attribute vectors to generate the final class label predictions. This design makes the presented transformer model a hybrid of 1) top-down and bottom-up attention pathways and 2) dynamic and static routing pathways. Comprehensive experiments on three widely-used benchmark datasets, namely CUB, SUN, and AWA2, are conducted. The obtained experimental results demonstrate the effectiveness of the proposed method. Our code is released in https://github.com/KORIYN/HRT.

Heterogeneous feature-aware Transformer-CNN coupling network for person re-identification.

Person re-identification plays an important role in the construction of the smart city. A reliable person re-identification system relieves users from the inefficient work of identifying the specific individual from enormous numbers of photos or videos captured by different surveillance devices. The most existing methods either focus on local discriminative features without global contextual information or scatter global features while ignoring the local features, resulting in ineffective attention to irregular pedestrian zones. In this article, a novel Transformer-CNN Coupling Network (TCCNet) is proposed to capture the fluctuant body region features in a heterogeneous feature-aware manner. We employ two bridging modules, the Low-level Feature Coupling Module (LFCM) and the High-level Feature Coupling Module (HFCM), to improve the complementary characteristics of the hybrid network. It is significantly helpful to enhance the capacity to distinguish between foreground and background features, thereby reducing the unfavorable impact of cluttered backgrounds on person re-identification. Furthermore, the duplicate loss for the two branches is employed to incorporate semantic information from distant preferences of the two branches into the resulting person representation. The experiments on two large-scale person re-identification benchmarks demonstrate that the proposed TCCNet achieves competitive results compared with several state-of-the-art approaches. The mean Average Precision (mAP) and Rank-1 identification rate on the MSMT17 dataset achieve 66.9% and 84.5%, respectively.

Deep graph clustering with multi-level subspace fusion

Attributed graph clustering combines both node attributes and graph structure information of data samples and has demonstrated satisfactory performance in various applications. However, how to choose the proper neighborhood for attributed graph clustering remains to be a challenge. A larger neighborhood may cause over-smoothed representations with less discrimination for clustering while the short-range ignore distant nodes and fails to capture the global information. In this paper, we propose a novel deep attributed graph clustering network with a multi-level subspace fusion module to address this issue. The first contribution of our work is to insert multiple self-expressive modules between low-level and high-level layers to promote more favorable features for clustering. The constraint of shared self-expressive matrix facilitates to preserve intrinsic structure without pre-defined neighborhoods as the previous methods do. Moreover, we introduce a novel loss function that leverages traditional reconstruction and the proposed structure fusion loss to effectively preserve multi-level clustering structures with both global and local discriminative features. Extensive experiments on public benchmark datasets validate the effectiveness of our proposed model compared with the state-of-the-art attribute graph clustering competitors by considerable margins.

Person Re-Identification Method Based on the Construction of Graph Convolutional Network with Attribute Feature

To fully pay attention to identity-sensitive feature information and utilize the correlations of inter-attributes and attributes-body parts, this paper proposes a person re-identification (re-ID) method based on the construction of graph convolutional network (GCN) with crucial attribute feature and body parts. First, it establishes the multiscale context-aware network (MSCAN) using dilated convolution with different expansion ratios, which can learn multiscale context information and obtain diversified global features. Subsequently, the human parsing model is utilized to extract the body part features. According to the attribute importance degree, the paper constructs low-dimensional GCN integrating the vital attributes and body parts of person descriptions to obtain discriminative local features. Finally, based on attribute prediction, it reduces the range of the images to be matched with discriminating possible objects from query images, thereby simplifying retrieval process. The experimental results demonstrate that the novel designed method can effectively improve person re-ID performance and achieve competitive evaluation results on typical public testing datasets.

Single-Object-Based Region Growth: Key Area Localization Model for Remote Sensing Image Scene Classification

Remote sensing image scene classification is a challenging task due to the large differences within the same classes and a large number of similar scenes among different classes. To tackle this problem, this paper proposes a single-object-based region growth algorithm to effectively localize the most key area in the whole image, so as to generate more discriminative local fine-grained features for the image scene. Concurrently, a local-global two-branch network is designed to utilize the features of the images from multiple perspectives, respectively. Specially, the global branch extracts global features (such as contour, texture) from the whole image, and local branch extracts more local features from the local key area. Finally, the global and local classification scores are integrated to make the final decision. Experiments are performed on three publicly available data sets, and the results show that this method can achieve higher accuracy compared to most existing state-of-the-art methods.

Multi-Granularity Mutual Learning Network for Object Re-Identification

Object re-identification (re-ID), which is key and fundamental technology for intelligent transportation systems, is a challenging task including person re-ID and vehicle re-ID. It aims to retrieve a given target object from the gallery images captured by different cameras. In this task, it is necessary to extract fine-grained and discriminative features to deal with complex inter-class and intra-class variations caused by the changes of camera viewpoints and object poses. Existing methods focus on learning discriminative local features to improve the re-ID performance. Some state-of-the-art methods use key point detection model to locate local features, which also increases the additional computational cost as side effect. Another type of method focuses on how to learn features of different granularity from rigid stripes of different scales. However, there is little attention paid to how to effectively coalesce multi-granularity features without additional calculation cost. To tackle this issue, this paper proposes the Multi-granularity Mutual Learning Network (MMNet) and makes two contributions. 1) We introduce the multi-granularity jigsaw puzzle module into object re-ID to impel the network to learn local discriminative features from multiple visual granularities by breaking spatial correlation in original images. 2) We propose a parameter-free multi-scale feature reconstruction module to facilitate mutual learning of features at multiple grain levels, thereby both global features and local features have strong representation capabilities. Extensive experiments demonstrate the effectiveness of our proposed modules and the superiority of our method over various state-of-the-art methods on both person and vehicle re-ID benchmarks.

A geometry-aware attention network for semantic segmentation of MLS point clouds

Semantic segmentation of mobile laser scanning (MLS) point clouds can provide meaningful 3 D semantic information of urban facilities for various applications. However, it still remains a challenge to extract accurate 3 D semantic information from MLS point cloud data due to its irregular 3 D geometric structure in a large-scale outdoor scene. To this end, this study develops a geometry-aware attention point network (GAANet) with geometric properties of the point cloud as a reference. Specifically, the proposed method first builds a graph-like region for each input point to establish the geometric correlation toward its neighbors for robustly descripting local geometry-aware features. Thereafter, the method introduces a novel multi-head attention mechanism to efficiently learn local discriminative features on the constructed graphs and a feature combination operation to capture both local and global geometric dependencies inside fused point features for significantly facilitating the segmentation of small or incomplete 3 D objects at point-level. Finally, an adaptive loss function is appended to handle class imbalance for the overall performance improvement. The validation experiments on two challenging benchmarks demonstrate the effectiveness and powerful generation ability of the proposed method, which achieves state-of-the-art performance with mean IoU of 65.09% and 95.20% in the Toronto-3D and Oakland 3-D MLS dataset, respectively.

Progressive Erasing Network with consistency loss for fine-grained visual classification

Fine-grained Visual Categorization (FGVC) in computer vision aims to recognize images belonging to multiple subordinate categories of a super-category. The difficulty of FGVC lies in the close resemblance among inter-classes and large variations among intra-classes. Most existing networks only focus on a few discriminative regions, while ignoring many subtle complementary features. So we propose a Progressive Erasing Network (PEN). In PEN, a Multi-Grid Erasure mechanism augments data samples and assists in capturing the local discriminative features, where the overall structure of the image is destroyed indirectly through pixel-wise erasure. Cross-layer feature aggregation by extracting salient class features is of great significance in FGVC. However, the capability of cross-layer feature representation based on a simple aggregation strategy is still inefficient. To this end, the proposed Consistency loss explores the cross-layer semantic affinity, which guides the Cross-Layer Incentive (CLI) block to mine more efficient feature representations of different granularity. We also integrate Cross Entropy and Complementary Entropy to take the distribution of negative classes into account for better classification performance. Our method uses end-to-end training with only classification labels. Experimental results show that our model outperforms the state-of-the-art on three fine-grained benchmarks.

Automated classification of cervical lymph-node-level from ultrasound using Depthwise Separable Convolutional Swin Transformer

There are few studies on cervical ultrasound lymph-node-level classification which is very important for qualitative diagnosis and surgical treatment of diseases. Currently, ultrasound examination relies on the subjective experience of physicians to judge the level of the cervical lymph nodes, which is easily misclassified. Unlike other automated diagnostic tasks, lymph-node-level classification needs to focus on global structural information. Besides, there is a large range of sternocleidomastoid muscles in levels II, III and IV, which leads to small inter-class differences in these levels, so it also needs to focus on key local areas to extract strong distinguishable features. In this paper, we propose the Depthwise Separable Convolutional Swin Transformer, introducing the deepwise separable convolution branch into the self-attention mechanism to capture discriminative local features. Meanwhile, to address the problem of data imbalance, a new loss function is proposed to improve the performance of the classification network. In addition, for the ultrasound data collected by different devices, low contrast and blurring problems of ultrasound imaging, a unified pre-processing algorithm is designed. The model was validated on 1146 cases of cervical ultrasound lymph node collected from the Sixth People’s Hospital of Shanghai. The average accuracy precision, sensitivity, specificity, and F1 value of the model for the valid dataset after five-fold cross-validation were 80.65%, 80.68%, 78.73%, 95.99% and 79.42%, respectively. It has been verified by visualization methods that the Region of Interest (ROI) of the model is similar or consistent with the observed region of the experts.