Supervised Object Localization Research Articles

Diverse Complementary Part Mining for Weakly Supervised Object Localization.

Weakly Supervised Object Localization (WSOL) aims to localize objects with only image-level labels, which has better scalability and practicability than fully supervised methods in the actual deployment. However, a common limitation for available techniques based on classification networks is that they only highlight the most discriminative part of the object, not the entire object. To alleviate this problem, we propose a novel end-to-end part discovery model (PDM) to learn multiple discriminative object parts in a unified network for accurate object localization and classification. The proposed PDM enjoys several merits. First, to the best of our knowledge, it is the first work to directly model diverse and robust object parts by exploiting part diversity, compactness, and importance jointly for WSOL. Second, three effective mechanisms including diversity, compactness, and importance learning mechanisms are designed to learn robust object parts. Therefore, our model can exploit complementary spatial information and local details from the learned object parts, which help to produce precise bounding boxes and discriminate different object categories. Extensive experiments on two standard benchmarks demonstrate that our PDM performs favorably against state-of-the-art WSOL approaches.

Group-Wise Learning for Weakly Supervised Semantic Segmentation.

Acquiring sufficient ground-truth supervision to train deep visual models has been a bottleneck over the years due to the data-hungry nature of deep learning. This is exacerbated in some structured prediction tasks, such as semantic segmentation, which require pixel-level annotations. This work addresses weakly supervised semantic segmentation (WSSS), with the goal of bridging the gap between image-level annotations and pixel-level segmentation. To achieve this, we propose, for the first time, a novel group-wise learning framework for WSSS. The framework explicitly encodes semantic dependencies in a group of images to discover rich semantic context for estimating more reliable pseudo ground-truths, which are subsequently employed to train more effective segmentation models. In particular, we solve the group-wise learning within a graph neural network (GNN), wherein input images are represented as graph nodes, and the underlying relations between a pair of images are characterized by graph edges. We then formulate semantic mining as an iterative reasoning process which propagates the common semantics shared by a group of images to enrich node representations. Moreover, in order to prevent the model from paying excessive attention to common semantics, we further propose a graph dropout layer to encourage the graph model to capture more accurate and complete object responses. With the above efforts, our model lays the foundation for more sophisticated and flexible group-wise semantic mining. We conduct comprehensive experiments on the popular PASCAL VOC 2012 and COCO benchmarks, and our model yields state-of-the-art performance. In addition, our model shows promising performance in weakly supervised object localization (WSOL) on the CUB-200-2011 dataset, demonstrating strong generalizability. Our code is available at: https://github.com/Lixy1997/Group-WSSS.

Adversarial Transformers for Weakly Supervised Object Localization.

Weakly supervised object localization (WSOL) aims at localizing objects with only image-level labels, which has better scalability and practicability than fully supervised methods. However, without pixel-level supervision, existing methods tend to generate rough localization maps, which hinders localization performance. To alleviate this problem, we propose an adversarial transformer network (ATNet), which aims to obtain a well-learned localization model with pixel-level pseudo labels. The proposed ATNet enjoys several merits. First, we design an object transformer ( G ) that can generate localization maps and pseudo labels effectively and dynamically, and a part transformer ( D ) to accurately discriminate detailed local differences between localization maps and pseudo labels. Second, we propose to train G and D via an adversarial process, where G can generate more accurate localization maps approaching pseudo labels to fool D . To the best of our knowledge, this is the first work to explore transformers with adversarial training to obtain a well-learned localization model for WSOL. Extensive experiments with four backbones on two standard benchmarks demonstrate that our ATNet achieves favorable performance against state-of-the-art WSOL methods. Besides, our adversarial training can provide higher robustness against adversarial attacks.

Uncertainty-Aware Forward Correction for Weakly Supervised Solar Panel Mapping From High-Resolution Aerial Images

Solar panel mapping from high resolution aerial images is becoming increasingly crucial to grid planning and operation, where weakly supervised approach has been explored. To cope with the noisy nature of pseudo labels (PLs) generated by weakly supervised object localization, we propose an effective uncertainty-aware forward correction (UA-FC) method to learn clean predictions from the noisy PLs. The proposed method consists of two steps: heteroscedastic uncertainty estimation and forward correction procedure. The purpose of the first step is to produce uncertainty as an indicator for the instance-dependent noise. The second step includes a target mapping network to produce clean predictions and a transition function to model the relationship between clean predictions and noisy PLs. As estimating every probability of one class flipped into another is difficult and time-consuming, we introduce heteroscedastic uncertainty as a measurement and propose an uncertainty-based transformation function to map clean predictions into noisy ones. By minimizing the errors between the noisy predictions and noisy labels, the target mapping network is able to offer clean predictions close to the actual objects. Extensive experiments on an aerial data set reveal that the proposed method outperforms other state-of-the-art methods by a large margin, especially in recovering the boundary of the objects.

Task-Aware Weakly Supervised Object Localization With Transformer.

Weakly supervised object localization (WSOL) aims to predict both object locations and categories with only image-level class labels. However, most existing methods rely on class-specific image regions for localization, resulting in incomplete object localization. To alleviate this problem, we propose a novel end-to-end task-aware framework with a transformer encoder-decoder architecture (TAFormer) to learn class-agnostic foreground maps, including a representation encoder, a localization decoder, and a classification decoder. The proposed TAFormer enjoys several merits. First, the designed three modules can effectively perform class-agnostic localization and classification in a task-aware manner, achieving remarkable performance for both tasks. Second, an optimal transport algorithm is proposed to provide pixel-level pseudo labels to online refine foreground maps. To the best of our knowledge, this is the first work by exploring a task-aware framework with a transformer architecture and an optimal transport algorithm to achieve accurate object localization for WSOL. Extensive experiments with four backbones on two standard benchmarks demonstrate that our TAFormer achieves favorable performance against state-of-the-art methods. Furthermore, we show that the proposed TAFormer provides higher robustness against adversarial attacks and noisy labels.

Evaluating Weakly Supervised Object Localization Methods Right? A Study on Heatmap-Based Xai and Neural Backed Decision Tree

Evaluating Weakly Supervised Object Localization Methods Right? A Study on Heatmap-Based Xai and Neural Backed Decision Tree

Weakly Supervised Learning for Object Localization Based on an Attention Mechanism

Recently, deep learning has been successfully applied to object detection and localization tasks in images. When setting up deep learning frameworks for supervised training with large datasets, strongly labeling the objects facilitates good performance; however, the complexity of the image scene and large size of the dataset make this a laborious task. Hence, it is of paramount importance that the expensive work associated with the tasks involving strong labeling, such as bounding box annotation, is reduced. In this paper, we propose a method to perform object localization tasks without bounding box annotation in the training process by means of employing a two-path activation-map-based classifier framework. In particular, we develop an activation-map-based framework to judicially control the attention map in the perception branch by adding a two-feature extractor so that better attention weights can be distributed to induce improved performance. The experimental results indicate that our method surpasses the performance of the existing deep learning models based on weakly supervised object localization. The experimental results show that the proposed method achieves the best performance, with 75.21% Top-1 classification accuracy and 55.15% Top-1 localization accuracy on the CUB-200-2011 dataset.

Hierarchical complementary learning for weakly supervised object localization

Weakly supervised object localization (WSOL) is a challenging problem that aims to localize objects without ground-truth bounding boxes. A common approach is to train the model that generates a class activation map (CAM) to localize the discriminative features of the object. Unfortunately, the limitation of this method is that they detect just a part of the object and not the whole object. To solve this problem, previous works have removed some parts of the image (Zhang et al., 2018; Zhang et al., 2018; Singh and Lee, 2017; Choe and Shim, 2019) to force the model to detect the full object extent. However, these methods require one or many hyper-parameters to erase the appropriate pixels on the image, which could involve a loss of information. In this paper, we propose a Hierarchical Complementary Learning Network method (HCLNet) that helps the CNN to perform better on classification and localization. HCLNet uses a complementary CAM to generate multiple maps that detect different parts of the object. Unlike previous works, this method does not need any extra hyper-parameters, as well as does not introduce a big loss of information. In order to fuse these different maps, two different fusion strategies known as the addition strategy and the l1-norm strategy have been used. These strategies allow to detect the whole object while excluding the background. Extensive experiments show that HCLNet obtains better performance than state-of-the-art methods.

Adaptive attention augmentor for weakly supervised object localization

Weakly Supervised Object Localization (WSOL) is a technique for obtaining the object location from attention maps of the classification network, without using bounding box annotations. Existing WSOL approaches lack the modeling of the correlations between different regions of the target object. Hence they can only locate some discriminative attentions, which are small and sparse. Besides, they introduce too many background attentions when mining more object parts. In this paper, we propose a novel Adaptive Attention Augmentor (A3) to adaptively augment the target object attentions on class attention maps. It can supplement object attentions by discovering the semantic correspondence between different regions and dynamically suppress background attentions through the proposed Focal Dice loss. Extensive experiments demonstrate the effectiveness of our approach. On the ILSVRC dataset, A3 achieves a new state-of-the-art localization performance. On the fine-grained datasets including CUB-200–2011 and Cars-196, it also achieves very competitive results.

Simple feature pyramid network for weakly supervised object localization using multi-scale information

Simple feature pyramid network for weakly supervised object localization using multi-scale information

Lung Lesion Localization of COVID-19 From Chest CT Image: A Novel Weakly Supervised Learning Method.

Chest computed tomography (CT) image data is necessary for early diagnosis, treatment, and prognosis of Coronavirus Disease 2019 (COVID-19). Artificial intelligence has been tried to help clinicians in improving the diagnostic accuracy and working efficiency of CT. Whereas, existing supervised approaches on CT image of COVID-19 pneumonia require voxel-based annotations for training, which take a lot of time and effort. This paper proposed a weakly-supervised method for COVID-19 lesion localization based on generative adversarial network (GAN) with image-level labels only. We first introduced a GAN-based framework to generate normal-looking CT slices from CT slices with COVID-19 lesions. We then developed a novel feature match strategy to improve the reality of generated images by guiding the generator to capture the complex texture of chest CT images. Finally, the localization map of lesions can be easily obtained by subtracting the output image from its corresponding input image. By adding a classifier branch to the GAN-based framework to classify localization maps, we can further develop a diagnosis system with improved classification accuracy. Three CT datasets from hospitals of Sao Paulo, Italian Society of Medical and Interventional Radiology, and China Medical University about COVID-19 were collected in this article for evaluation. Our weakly supervised learning method obtained AUC of 0.883, dice coefficient of 0.575, accuracy of 0.884, sensitivity of 0.647, specificity of 0.929, and F1-score of 0.640, which exceeded other widely used weakly supervised object localization methods by a significant margin. We also compared the proposed method with fully supervised learning methods in COVID-19 lesion segmentation task, the proposed weakly supervised method still leads to a competitive result with dice coefficient of 0.575. Furthermore, we also analyzed the association between illness severity and visual score, we found that the common severity cohort had the largest sample size as well as the highest visual score which suggests our method can help rapid diagnosis of COVID-19 patients, especially in massive common severity cohort. In conclusion, we proposed this novel method can serve as an accurate and efficient tool to alleviate the bottleneck of expert annotation cost and advance the progress of computer-aided COVID-19 diagnosis.

Region-based dropout with attention prior for weakly supervised object localization

Weakly supervised object localization (WSOL) methods utilize the internal feature responses of a classifier trained only on image-level labels. Classifiers tend to focus on the most discriminative part of the target object, instead of considering its full extent. Adversarial erasing (AE) techniques have been proposed to ameliorate this problem. These techniques erase the most discriminative part during training, thereby encouraging the classifiers to learn the less discriminative parts of the object. Despite the success of AE-based methods, we have observed that the hyperparameters fail to generalize across model architectures and datasets. Therefore, new sets of hyperparameters must be determined for each architecture and dataset. The selection of hyperparameters frequently requires strong supervision (e.g., pixel-level annotations or human inspection). Because WSOL is premised on the assumption that such strong supervision is absent, the applicability of AE-based methods is limited. In this paper, we propose the region-based dropout with attention prior (RDAP) algorithm, which features hyperparameter transferability. We combined AE with regional dropout algorithms that provide greater stability against the selection of hyperparameters. We empirically confirmed that the RDAP method achieved state-of-the-art localization accuracy on four architectures, namely VGG-GAP, InceptionV3, ResNet-50 SE, and PreResNet-18, and two datasets, namely CUB-200-2011 and ImageNet-1k, with a single set of hyperparameters.

Contrastive and consistent feature learning for weakly supervised object localization and semantic segmentation

Weakly supervised learning attempts to construct predictive models by learning with weak supervision. In this paper, we concentrate on weakly supervised object localization and semantic segmentation tasks. Existing methods are limited to focusing on narrow discriminative parts or overextending the activations to less discriminative regions even on backgrounds. To mitigate these problems, we regard the background as an important cue that guides the feature activation to cover the entire object to the right extent, and propose two novel objective functions: 1) contrastive attention loss and 2) foreground consistency loss. Contrastive attention loss draws the foreground feature and its dropped version close together and pushes the dropped foreground feature away from the background feature. Foreground consistency loss favors agreement between layers and provides early layers with a sense of objectness. Using both losses leads to balanced improvements over localization and segmentation accuracy by boosting activations on less discriminative regions but restraining the activation in the target object extent. For better optimizing the above losses, we use the non-local attention blocks to replace channel-pooled attention leading to enhanced attention maps considering the spatial similarity. Finally, our method achieves state-of-the-art localization performance on CUB-200-2011, ImageNet, and OpenImages benchmarks regarding top-1 localization accuracy, MaxBoxAccV2, and PxAP. We also demonstrate the effectiveness of our method in improving segmentation performance measured by mIoU on the PASCAL VOC dataset.

Weakly Supervised Object Localization and Detection: A Survey.

As an emerging and challenging problem in the computer vision community, weakly supervised object localization and detection plays an important role for developing new generation computer vision systems and has received significant attention in the past decade. As methods have been proposed, a comprehensive survey of these topics is of great importance. In this work, we review (1) classic models, (2) approaches with feature representations from off-the-shelf deep networks, (3) approaches solely based on deep learning, and (4) publicly available datasets and standard evaluation metrics that are widely used in this field. We also discuss the key challenges in this field, development history of this field, advantages/disadvantages of the methods in each category, the relationships between methods in different categories, applications of the weakly supervised object localization and detection methods, and potential future directions to further promote the development of this research field.

Multi-Scale Low-Discriminative Feature Reactivation for Weakly Supervised Object Localization.

For weakly supervised object localization (WSOL), how to avoid the network focusing only on some small discriminative parts is a main challenge needed to solve. The widely-used Class Activation Mapping (CAM) based paradigm usually employs Adversarial Learning (AL) strategy to search more object parts by constantly hiding discovered object features, but the adversarial process is difficult to control. In this paper, we propose a novel CAM-based framework with Multi-scale Low-Discriminative Feature Reactivation (mLDFR) for WSOL. The mLDFR framework reactivates the low-discriminative object parts via bottom-up continuous feature maps recalibration and multi-scale object category mapping. Compared with the AL-based methods, our method fully improves the localization power of the network without damaging the classification power and can perform multi-instance localization, which are hard to achieve under the AL-based framework. Moreover, the mLDFR framework is flexible, and can be built on the top of various classical CNN backbones. Experimental results demonstrate the superiority of our method. With VGG16 as backbone, we achieve 46.96% Cls-Loc top1 err and 66.12% CorLoc on ILSVRC2014, 38.07% Cls-Loc top1 err and 75.04% CorLoc on CUB200-2011, surpassing the state-of-the-arts by a large margin.

Entropy guided adversarial model for weakly supervised object localization

Weakly Supervised Object Localization is challenging due to the lack of bounding box annotations. Previous works tend to generate a Class Activation Map (CAM) to localize the object. However, the CAM highlights only the most discirminative part of the object and does not highlight the whole object. To address this problem, we propose an Entropy Guided Adversarial model (EGA model) to perform better localization of objects. EGA model uses adversarial learning method to create adversarial examples, i.e., images where a perturbation is added. Treating adversarial examples as data augmentation regularize our model as well as detect more discriminative visual pattern on the CAM. We further apply the Shannon entropy on the generated CAM to guide the model during training. Minimizing the entropy loss forces the model to generate a high-confident CAM. The high-confident CAM detects the whole object while excludes the background. Extensive experiments show that EGA model improves classification and localization performances on state-of-the-art benchmarks. Ablation experiments also show that both the adversarial learning and the entropy loss contribute to the algorithm performance.

Hierarchical saliency mapping for weakly supervised object localization based on class activation mapping

Weakly supervised object localization is a basic research in the field of computer vision. In this paper, a hierarchical saliency mapping network for object localization is proposed and designed to avoid missing detailed information of potential object. Based on the classical convolution network, we remove the fully connected part and add multiple information extraction branches. The network extracts information from convolution layers of different scales to generate Hierarchical Saliency Map. Hierarchical Saliency Maps that include Hierarchical-Class Activation Map and Hierarchical-Spatial Pyramid Saliency Map fuse deep-level features and low-level features to locate object. The datasets used for testing are Caltech-UCSD Birds 200, Caltech101 and ImageNet. Compared with Class Activation Map and Spatial Pyramid Saliency Map, the localization accuracy has been improved. This method can be used for fine-grained classification, object tracking and other fields.

GridMix: Strong regularization through local context mapping

Recently developed regularization techniques improve the networks generalization by only considering the global context. Therefore, the network tends to focus on a few most discriminative subregions of an image for prediction accuracy, leading the network being sensitive to unseen or noisy data. To address this disadvantage, we introduce the concept of local context mapping by predicting patch-level labels and combine it with a method of local data augmentation by grid-based mixing, called GridMix. Through our analysis of intermediate representations, we show that our GridMix can effectively regularize the network model. Finally, our evaluation results indicate that GridMix outperforms state-of-the-art techniques in classification and adversarial robustness, and it achieves a comparable performance in weakly supervised object localization.

A Two-Branch Network for Weakly Supervised Object Localization

Weakly supervised object localization (WSOL) has attracted intense interest in computer vision for instance level annotations. As a hot research topic, a number of existing works concentrated on utilizing convolutional neural network (CNN)-based methods, which are powerful in extracting and representing features. The main challenge in CNN-based WSOL methods is to obtain features covering the entire target objects, not only the most discriminative object parts. To overcome this challenge and to improve the detection performance of feature extracting related WSOL methods, a CNN-based two-branch model was presented in this paper to locate objects using supervised learning. Our method contained two branches, including a detection branch and a self-attention branch. During the training process, the two branches interacted with each other by regarding the segmentation mask from the other branch as the pseudo ground truth labels of itself. Our model was able to focus on capturing the information of all the object parts due to the self-attention mechanism. Additionally, we embedded multi-scale detection into our two-branch method to output two-scale features. We evaluated our two-branch network on the CUB-200-2011 and VOC2007 datasets. The pointing localization, intersection over union (IoU) localization, and correct localization precision (CorLoc) results demonstrated competitive performance with other state-of-the-art methods in WSOL.

PsyNet: Self-Supervised Approach to Object Localization Using Point Symmetric Transformation

Existing co-localization techniques significantly lose performance over weakly or fully supervised methods in accuracy and inference time. In this paper, we overcome common drawbacks of co-localization techniques by utilizing self-supervised learning approach. The major technical contributions of the proposed method are two-fold. 1) We devise a new geometric transformation, namely point symmetric transformation and utilize its parameters as an artificial label for self-supervised learning. This new transformation can also play the role of region-drop based regularization. 2) We suggest a heat map extraction method for computing the heat map from the network trained by self-supervision, namely class-agnostic activation mapping. It is done by computing the spatial attention map. Based on extensive evaluations, we observe that the proposed method records new state-of-the-art performance in three fine-grained datasets for unsupervised object localization. Moreover, we show that the idea of the proposed method can be adopted in a modified manner to solve the weakly supervised object localization task. As a result, we outperform the current state-of-the-art technique in weakly supervised object localization by a significant gap.