Weakly Supervised Object Localization Research Articles

Improving weakly-supervised object localization using adversarial erasing and pseudo label

Weakly-supervised learning approaches have gained significant attention due to their ability to reduce the effort required for human annotations in training neural networks. This paper investigates a framework for weakly-supervised object localization, which aims to train a neural network capable of predicting both the object class and its location using only images and their image-level class labels. The proposed framework consists of a shared feature extractor, a classifier, and a localizer. The localizer predicts pixel-level class probabilities, while the classifier predicts the object class at the image level. Since image-level class labels are insufficient for training the localizer, weakly-supervised object localization methods often encounter challenges in accurately localizing the entire object region. To address this issue, the proposed method incorporates adversarial erasing and pseudo labels to improve localization accuracy. Specifically, novel losses are designed to utilize adversarially erased foreground features and adversarially erased feature maps, reducing dependence on the most discriminative region. Additionally, the proposed method employs pseudo labels to suppress activation values in the background while increasing them in the foreground. The proposed method is applied to two backbone networks (MobileNetV1 and InceptionV3) and is evaluated on three publicly available datasets (ILSVRC-2012, CUB-200-2011, and PASCAL VOC 2012). The experimental results demonstrate that the proposed method outperforms previous state-of-the-art methods across all evaluated metrics.

Clustering-inspired channel selection method for weakly supervised object localization

Weakly Supervised Object Localization (WSOL) aims to utilize the features learned by a classifier on the image-level labels to locate target objects. However, these existing channel selection methods for WSOL still cannot effectively select the important channels and remove the unimportant ones. To address this issue, we propose a Clustering-inspired Channel Selection method based on Class Activation Maps (CCS-CAM). Compared with the traditional methods, the advantage of CCS-CAM is that it is very simple yet effective for channel selection due to the K-means clustering based on Class Activation Maps. It can effectively ensure both object localization and classification accuracy. The effectiveness of the proposed CCS-CAM method has been demonstrated using multiple public datasets, with GT-Know Loc reaching 87.9% and 63.71% on the CUB200-2011 and ImageNet-1k respectively, which is superior to the other state-of-the-art methods.

Generalized Weakly Supervised Object Localization.

With the goal of learning to localize specific object semantics using the low-cost image-level annotation, weakly supervised object localization (WSOL) has been receiving increasing attention in recent years. Although existing literatures have studied a number of major issues in this field, one important yet challenging scenario, where the test object semantics may appear in the training phase (seen categories) or never been observed before (unseen categories), is still beyond the exploration of the existing works. We define this scenario as the generalized WSOL (GWSOL) and make a pioneering effort to study it in this article. By leveraging attribute vectors to associate seen and unseen categories, we involve threefold modeling components, i.e., the class-sensitive modeling, semantic-agnostic modeling, and content-aware modeling, into a unified end-to-end learning framework. Such design enables our model to recognize and localize unconstrained object semantics, learn compact and discriminative features that could represent the potential unseen categories, and customize content-aware attribute weights to avoid localizing on misleading attribute elements. To advance this research direction, we contribute the bounding-box manual annotations to the widely used AwA2 dataset and benchmark the GWSOL methods. Comprehensive experiments demonstrate the effectiveness of our proposed learning framework and each of the considered modeling components.

Feature disparity learning for weakly supervised object localization

Weakly supervised object localization (WSOL) aims to localize objects with only image-level labels. As a common WSOL method, adversarial erasing always masks the most discriminative region in the feature space to compel the network to localize more regions of the object. However, with the discriminative region vanishing, the localizer is confused when distinguishing the regions of object from the background. In this paper, we propose a new feature disparity learning (FDL), which encourages the network to learn more distinctive features from the object region with similarity measurement after feature enhancement. Specifically, we first introduce a Spatial Vector Cross Attention (SVCA) module. This module enhances responses in less discriminative region of erased feature maps by reintegrating the spatial distribution of features through the capture of interdependencies among spatial vectors on each channel. Furthermore, we propose a feature complementarity loss to measure the similarity between unerased features and erased features, guiding the network to learn feature disparities caused by adversarial erasing for improved localization and classification. Several experimental studies demonstrate a significant increase in localization performance over the existing state-of-the-art erasing methods on the CUB 200–2011 and ILSVRC 2016 datasets.

Exploring Intrinsic Discrimination and Consistency for Weakly Supervised Object Localization.

Weakly supervised object localization (WSOL) is a challenging and promising task that aims to localize objects solely based on the supervision of image category labels. In the absence of annotated bounding boxes, WSOL methods must employ the intrinsic properties of the image classification task pipeline to generate object localizations. In this work, we propose a WSOL method for exploring the Intrinsic Discrimination and Consistency in the image classification task pipeline, and call it as IDC. First, we develop a Triplet Metrics Based Foreground Modeling (TMFM) framework to directly predict object foreground regions using intrinsic discrimination. Unlike Class Activation Map (CAM) based methods that also rely on intrinsic discrimination, our TMFM framework alleviates the problem of only focusing on the most discriminative parts by optimizing foreground and background regions synergistically. Second, we design a Dual Geometric Transformation Consistency Constraints (DGTC2) training strategy to introduce additional supervision and regularization constraints for WSOL by leveraging intrinsic geometric transformation consistency. The proposed pixel-wise and object-wise consistency constraint losses cost-effectively provide spontaneous supervision for WSOL. Extensive experiments show that our IDC method achieves significant and consistent performance gains compared to existing state-of-the-art WSOL approaches. Code is available at: https://github.com/vignywang/IDC.

Localizing From Classification: Self-Directed Weakly Supervised Object Localization for Remote Sensing Images.

In recent years, object localization and detection methods in remote sensing images (RSIs) have received increasing attention due to their broad applications. However, most previous fully supervised methods require a large number of time-consuming and labor-intensive instance-level annotations. Compared with those fully supervised methods, weakly supervised object localization (WSOL) aims to recognize object instances using only image-level labels, which greatly saves the labeling costs of RSIs. In this article, we propose a self-directed weakly supervised strategy (SD-WSS) to perform WSOL in RSIs. To specify, we fully exploit and enhance the spatial feature extraction capability of the RSIs' classification model to accurately localize the objects of interest. To alleviate the serious discriminative region problem exhibited by previous WSOL methods, the spatial location information implicit in the classification model is carefully extracted by GradCAM ++ to guide the learning procedure. Furthermore, to eliminate the interference from complex backgrounds of RSIs, we design a novel self-directed loss to make the model optimize itself and explicitly tell it where to look. Finally, we review and annotate the existing remote sensing scene classification dataset and create two new WSOL benchmarks in RSIs, named C45V2 and PN2. We conduct extensive experiments to evaluate the proposed method and six mainstream WSOL methods with three backbones on C45V2 and PN2. The results demonstrate that our proposed method achieves better performance when compared with state-of-the-arts.

TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization.

Weakly supervised object localization (WSOL), which trains object localization models using solely image category annotations, remains a challenging problem. Existing approaches based on convolutional neural networks (CNNs) tend to miss full object extent while activating discriminative object parts. Based on our analysis, this is caused by CNN's intrinsic characteristics, which experiences difficulty to capture object semantics at long distances. In this article, we introduce the vision transformer to WSOL, with the aim to capture long-range semantic dependency of features by leveraging transformer's cascaded self-attention mechanism. We propose the token semantic coupled attention map (TS-CAM) method, which first decomposes class-aware semantics and then couples the semantics with attention maps for semantic-aware activation. To capture object semantics at long distances and avoid partial activation, TS-CAM performs spatial embedding by partitioning an image to a set of patch tokens. To incorporate object category information to patch tokens, TS-CAM reallocates category-related semantics to each patch token. The patch tokens are finally coupled with attention maps which are semantic-agnostic to perform semantic-aware object localization. By introducing semantic tokens to produce semantic-aware attention maps, we further explore the capability of TS-CAM for multicategory object localization. Experiments show that TS-CAM outperforms its CNN-CAM counterpart by 11.6% and 28.9% on ILSVRC and CUB-200-2011 datasets, respectively, improving the state-of-the-art with large margins. TS-CAM also demonstrates superiority for multicategory object localization on the Pascal VOC dataset. The code is available at github.com/yuanyao366/ts-cam-extension.

Knowledge-guided Causal Intervention for Weakly-supervised Object Localization

Knowledge-guided Causal Intervention for Weakly-supervised Object Localization

Weakly supervised object localization via knowledge distillation based on foreground–background contrast

Weakly supervised object localization (WSOL) is a challenging task that aims to localize objects in images using only image-level labels. Despite the widespread use of WSOL methods based on class activation mapping (CAM), such methods do not consider that the network may overly focus on local regions of the most interesting objects during localization, thus neglecting the overall information. To address this issue, we introduces an additional attention branch for convolutional neural networks (CNNs) that utilizes the attention mechanism of multi-layer perceptron (MLP) to enhance the network’s learning of global information and supervise the feature learning of CNNs online through knowledge distillation, thereby improving the localization accuracy of WSOL. Specifically, we designs a new loss function using the generated features to combine with contrastive learning, effectively dividing the foreground and background of the image to provide more accurate pseudo-labels for subsequent classification and localization tasks. In the experiments, we tested our method on the CUB-200-2011 dataset and compared it with existing methods. The experimental results show that our method achieves good performance in WSOL tasks.

Background-Aware Classification Activation Map for Weakly Supervised Object Localization.

Weakly supervised object localization (WSOL) relaxes the requirement of dense annotations for object localization by using image-level annotation to supervise the learning process. However, most WSOL methods only focus on forcing the object classifier to produce high activation score on object parts without considering the influence of background locations, causing excessive background activations and ill-pose background score searching. Based on this point, our work proposes a novel mechanism called the background-aware classification activation map (B-CAM) to add background awareness for WSOL training. Besides aggregating an object image-level feature for supervision, our B-CAM produces an additional background image-level feature to represent the pure-background sample. This additional feature can provide background cues for the object classifier to suppress the background activations on object localization maps. Moreover, our B-CAM also trained a background classifier with image-level annotation to produce adaptive background scores when determining the binary localization mask. Experiments indicate the effectiveness of the proposed B-CAM on four different types of WSOL benchmarks, including CUB-200, ILSVRC, OpenImages, and VOC2012 datasets.

Integration of Multi-scale CAM and Attention for Weakly Supervised Defects Localization on Surface Defective Apple

Weakly supervised object localization (WSOL) is a task of localizing an object in an image using only image-level labels. Previous studies have followed the conventional class activation mapping (CAM) pipeline. However, we reveal the current CAM approach suffers from problems which cause original CAM could not capture the complete defects features. This work utilizes a convolutional neural network (CNN) pretrained on image-level labels to generate class activation maps in a multi-scale manner to highlight discriminative regions. Additionally, a vision transformer (ViT) pretrained was treated to produce multi-head attention maps as an auxiliary detector. By integrating the CNN-based CAMs and attention maps, our approach localizes defective regions without requiring bounding box or pixel-level supervision during training. We evaluate our approach on a dataset of apple images with only image-level labels of defect categories. Experiments demonstrate our proposed method aligns with several Object Detection models performance, hold a promise for improving localization.

DiPS: Discriminative pseudo-label sampling with self-supervised transformers for weakly supervised object localization

Self-supervised vision transformers (SSTs) have shown great potential to yield rich localization maps that highlight different objects in an image. However, these maps remain class-agnostic since the model is unsupervised. They often tend to decompose the image into multiple maps containing different objects while being unable to distinguish the object of interest from background noise objects. In this paper, Discriminative Pseudo-label Sampling (DiPS) is introduced to leverage these class-agnostic maps for weakly-supervised object localization (WSOL), where only image-class labels are available. Given multiple attention maps, DiPS relies on a pre-trained classifier to identify the most discriminative regions of each attention map. This ensures that the selected ROIs cover the correct image object while discarding the background ones, and, as such, provides a rich pool of diverse and discriminative proposals to cover different parts of the object. Subsequently, these proposals are used as pseudo-labels to train our new transformer-based WSOL model designed to perform classification and localization tasks. Unlike standard WSOL methods, DiPS optimizes performance in both tasks by using a transformer encoder and a dedicated output head for each task, each trained using dedicated loss functions. To avoid overfitting a single proposal and promote better object coverage, a single proposal is randomly selected among the top ones for a training image at each training step. Experimental results11Our code is available: https://github.com/shakeebmurtaza/dips on the challenging CUB, ILSVRC, OpenImages, and TelDrone datasets indicate that our architecture, in combination with our transformer-based proposals, can yield better localization performance than state-of-the-art methods.

Positive-weighting feature enhancement for weakly supervised object localization

Weakly Supervised Object Localization (WSOL) techniques identify the object location by only using image-level labels, without any bonding box annotations. A restriction of these techniques is that traditionally a well-trained model only focuses on the most discriminative parts of an image, instead of the whole object. To overcome this problem, most of the WSOL methods dropped some or all crucial discriminative parts to force the model to learn from the secondary discriminative region as well. However, only a small number of methods are dedicated to improving the utilization of the weights within the deep neural network model. In this paper, we propose a plug-in module, Positive-weighting Feature Enhancement (PFE), to correctly utilize existing information embedded in positive weights. The proposed method is made up of two major components: 1) a complementary Noisy Feature Elimination loss (NFE) to minimize noisy features, and 2) a Dynamic Concealment Mask (DCM) to prevent secondary discriminative features from being eliminated while minimizing noisy features. Based on the experiments, our solution outperforms previous method without modifying the architecture on two single-object benchmark datasets, CUB-200-2011 and ILSVRC 2012, and also on one multi-objects dataset, VOC 2007.

Entropy regularization for weakly supervised object localization

The goal of weakly-supervised object localization (WSOL) is to train a localization model without the location information of the object(s). Recently, most existing WSOL methods capture the object with an attention map extracted from a classification network. However, it has been observed that we need to sacrifice classification performances to achieve the best WSOL score. We conjecture that this is because the objective of classification training, minimizing entropy between one-hot ground truth and predicted class probability, is not entirely consistent with that of localization. In this paper, we investigate how the entropy of predicted class probability affects localization performances, where we conclude that there is a sweet spot for localization with respect to entropy. Hence, we propose a new training strategy that adopts entropy regularization for finding the optimal point effectively. Specifically, we add an additional term to the loss function, which minimizes the entropy between a uniform distribution and the predicted class probability vector. The proposed method is easy to implement since we do not need to modify the architecture or data pipeline. In addition, our method is efficient in that almost zero additional resources are required. Most importantly, our method improves WSOL scores significantly, which has been shown through extensive experiments.

Deep Weakly-Supervised Learning Methods for Classification and Localization in Histology Images: A Survey

Using state-of-the-art deep learning (DL) models to diagnose cancer from histology data presents several challenges related to the nature and availability of labeled histology images, including image size, stain variations, and label ambiguity. In addition, cancer grading and the localization of regions of interest (ROIs) in such images normally rely on both image- and pixel-level labels, with the latter requiring a costly annotation process. Deep weakly-supervised object localization (WSOL) methods provide different strategies for low-cost training of DL models. Given only image-class annotations, these methods can be trained to simultaneously classify an image, and yield class activation maps (CAMs) for ROI localization. This paper provides a review of deep WSOL methods to identify and locate diseases in histology images, without the need for pixel-level annotations. We propose a taxonomy in which these methods are divided into bottom-up and top-down methods according to the information flow in models. Although the latter have seen only limited progress, recent bottom-up methods are currently driving a lot of progress with the use of deep WSOL methods. Early works focused on designing different spatial pooling functions. However, those methods quickly peaked in term of localization accuracy and revealed a major limitation, namely, – the under-activation of CAMs, which leads to high false negative localization. Subsequent works aimed to alleviate this shortcoming and recover the complete object from the background, using different techniques such as perturbation, self-attention, shallow features, pseudo-annotation, and task decoupling.<br>In the present paper, representative deep WSOL methods from our taxonomy are also evaluated and compared in terms of classification and localization accuracy using two challenging public histology datasets – one for colon cancer (GlaS), and a second, for breast cancer (CAMELYON16). Overall, the results indicate poor localization performance, particularly for generic methods that were initially designed to process natural images. Methods designed to address the challenges posed by histology data often use priors such as ROI size, or additional pixel-wise supervision estimated from a pre-trained classifier, allowing them to achieve better results. However, all the methods suffer from high false positive/negative localization. Classification performance is mainly affected by the model selection process, which uses either the classification or the localization metric. Finally, four key challenges are identified in the application of deep WSOL methods in histology, namely, – under-/over-activation of CAMs, sensitivity to thresholding, and model selection – and research avenues are provided to mitigate them. Our code is publicly available at <a href='https://github.com/jeromerony/survey_wsl_histology'>https://github.com/jeromerony/survey_wsl_histology</a>

Self-taught cross-domain few-shot learning with weakly supervised object localization and task-decomposition

The shift between the source and target domains is the main challenge in cross-domain few-shot learning (CD-FSL) tasks. However, the target domain is unknown during training in the source domain, which results in a lack of directed guidance for target tasks. We observe that, owing to similar backgrounds in the target domains, self-labelled samples can be applied as prior tasks to transfer knowledge onto target tasks. Accordingly, we propose a task-expansion-decomposition framework for CD-FSL called the self-taught (ST) approach, which alleviates the problem of non-target guidance by constructing task-oriented metric spaces. Specifically, weakly supervised object localization (WSOL) and self-supervised technologies are employed to enrich task-oriented samples by exchanging and rotating discriminative regions, which generates a more abundant task set. Then, these tasks are decomposed into several tasks to complete few-shot recognition and rotation classification tasks. Transferring the source knowledge onto the target tasks and focusing on discriminative regions is beneficial to the task completion. We conducted extensive experiments under a cross-domain setting, including eight target domains: CUB, Cars, Places, Plantae, CropDieases, EuroSAT, ISIC, and ChestX. The experimental results demonstrate that the proposed ST approach is applicable to various metric-based models and provides promising improvements to CD-FSL.

Evaluation for Weakly Supervised Object Localization: Protocol, Metrics, and Datasets.

Weakly-supervised object localization (WSOL) has gained popularity over the last years for its promise to train localization models with only image-level labels. Since the seminal WSOL work of class activation mapping (CAM), the field has focused on how to expand the attention regions to cover objects more broadly and localize them better. However, these strategies rely on full localization supervision for validating hyperparameters and model selection, which is in principle prohibited under the WSOL setup. In this paper, we argue that WSOL task is ill-posed with only image-level labels, and propose a new evaluation protocol where full supervision is limited to only a small held-out set not overlapping with the test set. We observe that, under our protocol, the five most recent WSOL methods have not made a major improvement over the CAM baseline. Moreover, we report that existing WSOL methods have not reached the few-shot learning baseline, where the full-supervision at validation time is used for model training instead. Based on our findings, we discuss some future directions for WSOL. Source code and dataset are available at https://github.com/clovaai/wsolevaluation https://github.com/clovaai/wsolevaluation.

HiCT: Hierarchical Comprehend of Transformer for Weakly Supervised Object Localization

The weakly supervised object localization (WSOL) has always been a very challenging research subject in the field of computer vision, which aims to predict the localization of objects in an image using only an image-level class labeling approach. The traditional convolutional neural network (CNN) based-approaches utilize the local class activation and discrimination for classification guidance, and the biggest drawback of CNN is that it cannot capture the remote feature dependencies between pixels. Recently, the transformer architecture has been deployed in the WSOL, but the transformer cannot well capture local features. To address the above problems, we propose HiCT (Hierarchical comprehend of transformer), a simple and effective visual converter variation method. Moreover, we also propose a discriminative-based attention layer (DAL), which aims to mine the local feature information by utilizing the global token attention graph mechanism. To further improve the coverage of object localization, we introduce the spatial aware digging module (SADM). In addition, a set of complementarity loss calculators to patch hierarchy (CPH) is proposed to improve the sample class aggregation capability of our model. Finally, we conducted experiments on two commonly used datasets of CUB-200-2011 and ILSVRC, so as to verify the effectiveness of our method.

SPA2Net: Structure-Preserved Attention Activated Network for Weakly Supervised Object Localization.

By exploring the localizable representations in deep CNN, weakly supervised object localization (WSOL) methods could determine the position of the object in each image just trained by the classification task. However, the partial activation problem caused by the discriminant function makes the network unable to locate objects accurately. To alleviate this problem, we propose Structure-Preserved Attention Activated Network (SPA2Net), a simple and effective one-stage WSOL framework to explore the ability of structure preservation of deep features. Different from traditional WSOL approaches, we decouple the object localization task from the classification branch to reduce their mutual influence by involving a localization branch which is online refined by a self-supervised structural-preserved localization mask. Specifically, we employ the high-order self-correlation as structural prior to enhance the perception of spatial interaction within convolutional features. By succinctly combining the structural prior with spatial attention, activations by SPA2Net will spread from part to the whole object during training. To avoid the structure-missing issue caused by the classification network, we furthermore utilize the restricted activation loss (RAL) to distinguish the difference between foreground and background in the channel dimension. In conjunction with the self-supervised localization branch, SPA2Net can directly predict the class-irrelevant localization map while prompting the network to pay more attention to the target region for accurate localization. Extensive experiments on two publicly available benchmarks, including CUB-200-2011 and ILSVRC, show that our SPA2Net achieves substantial and consistent performance gains compared with baseline approaches. The code and models are available at https://github.com/MsterDC/SPA2Net.

Learning Consistency From High-Confidence Pseudo-Labels for Weakly Supervised Object Localization

Weakly supervised object localization (WSOL) tasks aim to classify and locate a single object under the supervision of only image-level labels. Pseudo-supervised learning methods have been shown to be effective for WSOL, which divided WSOL tasks into two decoupled subtasks: classification and localization. The decoupled framework has been proven to be effective in improving the performance of the localization subtask, but the predicted localizations are not robust enough due to the noise of pseudo-labels. Based on the assumption that the localization model should have similar predictions on different versions of the same image, we propose an additional refinement stage to learn more consistent localization. Specifically, in the refinement stage, we propose a simple and effective method for evaluating the confidence of pseudo-labels based on classification discrimination, and by learning consistency from high-confidence pseudo-labels, we further refine the localization model to get better localization performance. Besides, in the initialization stage, we propose a mask-based pseudo-label generator to initialize the localization model. We conduct experiments on two benchmark datasets: CUB-200-2011 and ImageNet-1k. Experimental results show that our two-stage approach achieves 94.01% <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">GT-Konwn</i> localization accuracy on the CUB-200-2011 testing dataset, and 65.23% <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">GT-Konwn</i> localization accuracy on the ImageNet-1k validation dataset. Moreover, when directly applied to the pseudo-supervised localization model, our refinement stage could achieve 94.05% and 67.13% <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">GT-Konwn</i> localization accuracy on CUB-200-2011 and ImageNet-1k datasets, respectively, which outperforms the corresponding pseudo-supervised localization model with 3.34% and 2.34% accuracy.