Supervised Object Localization Research Articles

PCSformer: Pair-wise Cross-scale Sub-prototypes mining with CNN-transformers for weakly supervised semantic segmentation

Generating initial seeds is an important step in weakly supervised semantic segmentation (WSSS). Our approach concentrates on generating and refining initial seeds. The convolutional neural networks (CNNs)–based initial seeds focus only on the most discriminative regions and lack global information about the target. The Vision Transformer (ViT)–based approach can capture long-range feature dependencies due to the unique advantage of the self-attention mechanism. Still, we find that it suffers from distractor object leakage and background leakage problems. Based on these observations, we propose PCSformer, which improves the model’s ability to extract features through a Pair-wise Cross-scale (PC) strategy and solves the problem of distractor object leakage by further extracting potential target features through Sub-Prototypes (SP) mining. In addition, the proposed Conflict Self-Elimination (CSE) module further alleviates the background leakage problem. We validate our approach on the widely adopted Pascal VOC 2012 and MS COCO 2014, and extensive experiments demonstrate our superior performance. Furthermore, our method proves to be competitive for WSSS in medical images and challenging scenarios involving deformable and cluttered scenes. Additionally, we extend the PCSformer to weakly supervised object localization tasks, further highlighting its scalability and versatility. The code is available at https://github.com/ChunmengLiu1/PCSformer.

Clustering-inspired channel selection method for weakly supervised object localization

Weakly Supervised Object Localization (WSOL) aims to utilize the features learned by a classifier on the image-level labels to locate target objects. However, these existing channel selection methods for WSOL still cannot effectively select the important channels and remove the unimportant ones. To address this issue, we propose a Clustering-inspired Channel Selection method based on Class Activation Maps (CCS-CAM). Compared with the traditional methods, the advantage of CCS-CAM is that it is very simple yet effective for channel selection due to the K-means clustering based on Class Activation Maps. It can effectively ensure both object localization and classification accuracy. The effectiveness of the proposed CCS-CAM method has been demonstrated using multiple public datasets, with GT-Know Loc reaching 87.9% and 63.71% on the CUB200-2011 and ImageNet-1k respectively, which is superior to the other state-of-the-art methods.

Generalized Weakly Supervised Object Localization.

With the goal of learning to localize specific object semantics using the low-cost image-level annotation, weakly supervised object localization (WSOL) has been receiving increasing attention in recent years. Although existing literatures have studied a number of major issues in this field, one important yet challenging scenario, where the test object semantics may appear in the training phase (seen categories) or never been observed before (unseen categories), is still beyond the exploration of the existing works. We define this scenario as the generalized WSOL (GWSOL) and make a pioneering effort to study it in this article. By leveraging attribute vectors to associate seen and unseen categories, we involve threefold modeling components, i.e., the class-sensitive modeling, semantic-agnostic modeling, and content-aware modeling, into a unified end-to-end learning framework. Such design enables our model to recognize and localize unconstrained object semantics, learn compact and discriminative features that could represent the potential unseen categories, and customize content-aware attribute weights to avoid localizing on misleading attribute elements. To advance this research direction, we contribute the bounding-box manual annotations to the widely used AwA2 dataset and benchmark the GWSOL methods. Comprehensive experiments demonstrate the effectiveness of our proposed learning framework and each of the considered modeling components.

Feature disparity learning for weakly supervised object localization

Weakly supervised object localization (WSOL) aims to localize objects with only image-level labels. As a common WSOL method, adversarial erasing always masks the most discriminative region in the feature space to compel the network to localize more regions of the object. However, with the discriminative region vanishing, the localizer is confused when distinguishing the regions of object from the background. In this paper, we propose a new feature disparity learning (FDL), which encourages the network to learn more distinctive features from the object region with similarity measurement after feature enhancement. Specifically, we first introduce a Spatial Vector Cross Attention (SVCA) module. This module enhances responses in less discriminative region of erased feature maps by reintegrating the spatial distribution of features through the capture of interdependencies among spatial vectors on each channel. Furthermore, we propose a feature complementarity loss to measure the similarity between unerased features and erased features, guiding the network to learn feature disparities caused by adversarial erasing for improved localization and classification. Several experimental studies demonstrate a significant increase in localization performance over the existing state-of-the-art erasing methods on the CUB 200–2011 and ILSVRC 2016 datasets.

Anti-Adversarially Manipulated Attributions for Weakly Supervised Semantic Segmentation and Object Localization.

Obtaining accurate pixel-level localization from class labels is a crucial process in weakly supervised semantic segmentation and object localization. Attribution maps from a trained classifier are widely used to provide pixel-level localization, but their focus tends to be restricted to a small discriminative region of the target object. An AdvCAM is an attribution map of an image that is manipulated to increase the classification score produced by a classifier before the final softmax or sigmoid layer. This manipulation is realized in an anti-adversarial manner, so that the original image is perturbed along pixel gradients in directions opposite to those used in an adversarial attack. This process enhances non-discriminative yet class-relevant features, which make an insufficient contribution to previous attribution maps, so that the resulting AdvCAM identifies more regions of the target object. In addition, we introduce a new regularization procedure that inhibits the incorrect attribution of regions unrelated to the target object and the excessive concentration of attributions on a small region of the target object. Our method achieves a new state-of-the-art performance in weakly and semi-supervised semantic segmentation, on both the PASCAL VOC 2012 and MS COCO 2014 datasets. In weakly supervised object localization, it achieves a new state-of-the-art performance on the CUB-200-2011 and ImageNet-1K datasets.

Weakly Supervised Object Localization with Background Suppression Erasing for Art Authentication and Copyright Protection

Weakly Supervised Object Localization with Background Suppression Erasing for Art Authentication and Copyright Protection

Multi-Layer Decoupling Attention Network for Weakly Supervised Object Localization

Multi-Layer Decoupling Attention Network for Weakly Supervised Object Localization

Localizing From Classification: Self-Directed Weakly Supervised Object Localization for Remote Sensing Images.

In recent years, object localization and detection methods in remote sensing images (RSIs) have received increasing attention due to their broad applications. However, most previous fully supervised methods require a large number of time-consuming and labor-intensive instance-level annotations. Compared with those fully supervised methods, weakly supervised object localization (WSOL) aims to recognize object instances using only image-level labels, which greatly saves the labeling costs of RSIs. In this article, we propose a self-directed weakly supervised strategy (SD-WSS) to perform WSOL in RSIs. To specify, we fully exploit and enhance the spatial feature extraction capability of the RSIs' classification model to accurately localize the objects of interest. To alleviate the serious discriminative region problem exhibited by previous WSOL methods, the spatial location information implicit in the classification model is carefully extracted by GradCAM ++ to guide the learning procedure. Furthermore, to eliminate the interference from complex backgrounds of RSIs, we design a novel self-directed loss to make the model optimize itself and explicitly tell it where to look. Finally, we review and annotate the existing remote sensing scene classification dataset and create two new WSOL benchmarks in RSIs, named C45V2 and PN2. We conduct extensive experiments to evaluate the proposed method and six mainstream WSOL methods with three backbones on C45V2 and PN2. The results demonstrate that our proposed method achieves better performance when compared with state-of-the-arts.

Exploring Intrinsic Discrimination and Consistency for Weakly Supervised Object Localization.

Weakly supervised object localization (WSOL) is a challenging and promising task that aims to localize objects solely based on the supervision of image category labels. In the absence of annotated bounding boxes, WSOL methods must employ the intrinsic properties of the image classification task pipeline to generate object localizations. In this work, we propose a WSOL method for exploring the Intrinsic Discrimination and Consistency in the image classification task pipeline, and call it as IDC. First, we develop a Triplet Metrics Based Foreground Modeling (TMFM) framework to directly predict object foreground regions using intrinsic discrimination. Unlike Class Activation Map (CAM) based methods that also rely on intrinsic discrimination, our TMFM framework alleviates the problem of only focusing on the most discriminative parts by optimizing foreground and background regions synergistically. Second, we design a Dual Geometric Transformation Consistency Constraints (DGTC2) training strategy to introduce additional supervision and regularization constraints for WSOL by leveraging intrinsic geometric transformation consistency. The proposed pixel-wise and object-wise consistency constraint losses cost-effectively provide spontaneous supervision for WSOL. Extensive experiments show that our IDC method achieves significant and consistent performance gains compared to existing state-of-the-art WSOL approaches. Code is available at: https://github.com/vignywang/IDC.

TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization.

Weakly supervised object localization (WSOL), which trains object localization models using solely image category annotations, remains a challenging problem. Existing approaches based on convolutional neural networks (CNNs) tend to miss full object extent while activating discriminative object parts. Based on our analysis, this is caused by CNN's intrinsic characteristics, which experiences difficulty to capture object semantics at long distances. In this article, we introduce the vision transformer to WSOL, with the aim to capture long-range semantic dependency of features by leveraging transformer's cascaded self-attention mechanism. We propose the token semantic coupled attention map (TS-CAM) method, which first decomposes class-aware semantics and then couples the semantics with attention maps for semantic-aware activation. To capture object semantics at long distances and avoid partial activation, TS-CAM performs spatial embedding by partitioning an image to a set of patch tokens. To incorporate object category information to patch tokens, TS-CAM reallocates category-related semantics to each patch token. The patch tokens are finally coupled with attention maps which are semantic-agnostic to perform semantic-aware object localization. By introducing semantic tokens to produce semantic-aware attention maps, we further explore the capability of TS-CAM for multicategory object localization. Experiments show that TS-CAM outperforms its CNN-CAM counterpart by 11.6% and 28.9% on ILSVRC and CUB-200-2011 datasets, respectively, improving the state-of-the-art with large margins. TS-CAM also demonstrates superiority for multicategory object localization on the Pascal VOC dataset. The code is available at github.com/yuanyao366/ts-cam-extension.

Learning Local Semantic Region Activations for Weakly Supervised Object Localization

Learning Local Semantic Region Activations for Weakly Supervised Object Localization

Weakly supervised object localization via knowledge distillation based on foreground–background contrast

Weakly supervised object localization (WSOL) is a challenging task that aims to localize objects in images using only image-level labels. Despite the widespread use of WSOL methods based on class activation mapping (CAM), such methods do not consider that the network may overly focus on local regions of the most interesting objects during localization, thus neglecting the overall information. To address this issue, we introduces an additional attention branch for convolutional neural networks (CNNs) that utilizes the attention mechanism of multi-layer perceptron (MLP) to enhance the network’s learning of global information and supervise the feature learning of CNNs online through knowledge distillation, thereby improving the localization accuracy of WSOL. Specifically, we designs a new loss function using the generated features to combine with contrastive learning, effectively dividing the foreground and background of the image to provide more accurate pseudo-labels for subsequent classification and localization tasks. In the experiments, we tested our method on the CUB-200-2011 dataset and compared it with existing methods. The experimental results show that our method achieves good performance in WSOL tasks.

Background-Aware Classification Activation Map for Weakly Supervised Object Localization.

Weakly supervised object localization (WSOL) relaxes the requirement of dense annotations for object localization by using image-level annotation to supervise the learning process. However, most WSOL methods only focus on forcing the object classifier to produce high activation score on object parts without considering the influence of background locations, causing excessive background activations and ill-pose background score searching. Based on this point, our work proposes a novel mechanism called the background-aware classification activation map (B-CAM) to add background awareness for WSOL training. Besides aggregating an object image-level feature for supervision, our B-CAM produces an additional background image-level feature to represent the pure-background sample. This additional feature can provide background cues for the object classifier to suppress the background activations on object localization maps. Moreover, our B-CAM also trained a background classifier with image-level annotation to produce adaptive background scores when determining the binary localization mask. Experiments indicate the effectiveness of the proposed B-CAM on four different types of WSOL benchmarks, including CUB-200, ILSVRC, OpenImages, and VOC2012 datasets.

Module of Axis-based Nexus Attention for weakly supervised object localization

Weakly supervised object localization tasks remain challenging to identify and segment an entire object rather than only discriminative parts of the object. To tackle this problem, corruption-based approaches have been devised, which involve the training of non-discriminative regions by corrupting (e.g., erasing) the input images or intermediate feature maps. However, this approach requires an additional hyperparameter, the corrupting threshold, to determine the degree of corruption and can unfavorably disrupt training. It also tends to localize object regions coarsely. In this paper, we propose a novel approach, Module of Axis-based Nexus Attention (MoANA), which helps to adaptively activate less discriminative regions along with the class-discriminative regions without an additional hyperparameter, and elaborately localizes an entire object. Specifically, MoANA consists of three mechanisms (1) triple-view attentions representation, (2) attentions expansion, and (3) features calibration mechanism. Unlike other attention-based methods that train a coarse attention map with the same values across elements in feature maps, MoANA trains fine-grained values in an attention map by assigning different attention values to each element. We validated MoANA by comparing it with various methods. We also analyzed the effect of each component in MoANA and visualized attention maps to provide insights into the calibration.

Integration of Multi-scale CAM and Attention for Weakly Supervised Defects Localization on Surface Defective Apple

Weakly supervised object localization (WSOL) is a task of localizing an object in an image using only image-level labels. Previous studies have followed the conventional class activation mapping (CAM) pipeline. However, we reveal the current CAM approach suffers from problems which cause original CAM could not capture the complete defects features. This work utilizes a convolutional neural network (CNN) pretrained on image-level labels to generate class activation maps in a multi-scale manner to highlight discriminative regions. Additionally, a vision transformer (ViT) pretrained was treated to produce multi-head attention maps as an auxiliary detector. By integrating the CNN-based CAMs and attention maps, our approach localizes defective regions without requiring bounding box or pixel-level supervision during training. We evaluate our approach on a dataset of apple images with only image-level labels of defect categories. Experiments demonstrate our proposed method aligns with several Object Detection models performance, hold a promise for improving localization.

Background Activation Suppression for Weakly Supervised Object Localization and Semantic Segmentation

Background Activation Suppression for Weakly Supervised Object Localization and Semantic Segmentation

DiPS: Discriminative pseudo-label sampling with self-supervised transformers for weakly supervised object localization

Self-supervised vision transformers (SSTs) have shown great potential to yield rich localization maps that highlight different objects in an image. However, these maps remain class-agnostic since the model is unsupervised. They often tend to decompose the image into multiple maps containing different objects while being unable to distinguish the object of interest from background noise objects. In this paper, Discriminative Pseudo-label Sampling (DiPS) is introduced to leverage these class-agnostic maps for weakly-supervised object localization (WSOL), where only image-class labels are available. Given multiple attention maps, DiPS relies on a pre-trained classifier to identify the most discriminative regions of each attention map. This ensures that the selected ROIs cover the correct image object while discarding the background ones, and, as such, provides a rich pool of diverse and discriminative proposals to cover different parts of the object. Subsequently, these proposals are used as pseudo-labels to train our new transformer-based WSOL model designed to perform classification and localization tasks. Unlike standard WSOL methods, DiPS optimizes performance in both tasks by using a transformer encoder and a dedicated output head for each task, each trained using dedicated loss functions. To avoid overfitting a single proposal and promote better object coverage, a single proposal is randomly selected among the top ones for a training image at each training step. Experimental results11Our code is available: https://github.com/shakeebmurtaza/dips on the challenging CUB, ILSVRC, OpenImages, and TelDrone datasets indicate that our architecture, in combination with our transformer-based proposals, can yield better localization performance than state-of-the-art methods.

Coarse2Fine: Local Consistency Aware Re-prediction for Weakly Supervised Object Localization

Weakly supervised object localization aims to localize objects of interest by using only image-level labels. Existing methods generally segment activation map by threshold to obtain mask and generate bounding box. However, the activation map is locally inconsistent, i.e., similar neighboring pixels of the same object are not equally activated, which leads to the blurred boundary issue: the localization result is sensitive to the threshold, and the mask obtained directly from the activation map loses the fine contours of the object, making it difficult to obtain a tight bounding box. In this paper, we introduce the Local Consistency Aware Re-prediction (LCAR) framework, which aims to recover the complete fine object mask from locally inconsistent activation map and hence obtain a tight bounding box. To this end, we propose the self-guided re-prediction module (SGRM), which employs a novel superpixel aggregation network to replace the post-processing of threshold segmentation. In order to derive more reliable pseudo label from the activation map to supervise the SGRM, we further design an affinity refinement module (ARM) that utilizes the original image feature to better align the activation map with the image appearance, and design a self-distillation CAM (SD-CAM) to alleviate the locator dependence on saliency. Experiments demonstrate that our LCAR outperforms the state-of-the-art on both the CUB-200-2011 and ILSVRC datasets, achieving 95.89% and 70.72% of GT-Know localization accuracy, respectively.

Positive-weighting feature enhancement for weakly supervised object localization

Weakly Supervised Object Localization (WSOL) techniques identify the object location by only using image-level labels, without any bonding box annotations. A restriction of these techniques is that traditionally a well-trained model only focuses on the most discriminative parts of an image, instead of the whole object. To overcome this problem, most of the WSOL methods dropped some or all crucial discriminative parts to force the model to learn from the secondary discriminative region as well. However, only a small number of methods are dedicated to improving the utilization of the weights within the deep neural network model. In this paper, we propose a plug-in module, Positive-weighting Feature Enhancement (PFE), to correctly utilize existing information embedded in positive weights. The proposed method is made up of two major components: 1) a complementary Noisy Feature Elimination loss (NFE) to minimize noisy features, and 2) a Dynamic Concealment Mask (DCM) to prevent secondary discriminative features from being eliminated while minimizing noisy features. Based on the experiments, our solution outperforms previous method without modifying the architecture on two single-object benchmark datasets, CUB-200-2011 and ILSVRC 2012, and also on one multi-objects dataset, VOC 2007.

Entropy regularization for weakly supervised object localization

The goal of weakly-supervised object localization (WSOL) is to train a localization model without the location information of the object(s). Recently, most existing WSOL methods capture the object with an attention map extracted from a classification network. However, it has been observed that we need to sacrifice classification performances to achieve the best WSOL score. We conjecture that this is because the objective of classification training, minimizing entropy between one-hot ground truth and predicted class probability, is not entirely consistent with that of localization. In this paper, we investigate how the entropy of predicted class probability affects localization performances, where we conclude that there is a sweet spot for localization with respect to entropy. Hence, we propose a new training strategy that adopts entropy regularization for finding the optimal point effectively. Specifically, we add an additional term to the loss function, which minimizes the entropy between a uniform distribution and the predicted class probability vector. The proposed method is easy to implement since we do not need to modify the architecture or data pipeline. In addition, our method is efficient in that almost zero additional resources are required. Most importantly, our method improves WSOL scores significantly, which has been shown through extensive experiments.