Supervised Object Localization Research Articles

Weakly Supervised Underwater Object Real-time Detection Based on High-resolution Attention Class Activation Mapping and Category Hierarchy

Recently, deep learning-based underwater object detection technology has achieved remarkable success. However, the accuracy and completeness of dataset instance annotation are crucial for its success. The quality of underwater images is low, severe objects clustering, and occlusion, acquiring object's annotations demands substantial time and labor costs, while mis annotation and missed annotation can also degrade model performance and limit their application in practical scenarios. To address this issue, this paper presents a novel weakly supervised underwater object real-time detection method, which is divided into two subtasks: weakly supervised object localization and real-time object detection. In the weakly supervised object localization task, we design a novel category hierarchy structure network that integrates the high-resolution attention-class activation mapping algorithm to obtain high-quality object class activation maps, weaken background interference, and obtain more complete object regions. The parameterized spatial loss module is devised to enable the model to escape from local optimal solutions, thus accurately and efficiently obtaining object pseudo-detection annotation boxes. For the real-time object detection task, the single-stage detector YOLOv7 is selected as the basic detection model, and an object perception loss function is designed based on the class activation map to jointly supervise the training process. A method for filtering noisy pseudo-supervision information is proposed to enhance the pseudo-supervision information involved in training. Ablation experiments and multi-method comparison experiments were conducted on the URPC and RUOD datasets, and the results verify the effectiveness of the proposed strategy, and our model exhibits significant advantages in detection performance and detection efficiency compared to current mainstream and advanced models.

Semantic-Constraint Matching for transformer-based weakly supervised object localization

Weakly supervised object localization (WSOL) strives to localize objects with only image-level supervision. WSOL often faces challenges such as incomplete localization due to classifier bias and over-localization in real scenes where objects and backgrounds are strongly associated or structurally similar. While the latest Transformer-based methods effectively enhance localization performance by leveraging long-range feature dependencies, they may inadvertently amplify divergent background activation and remain susceptible to classification bias. To this end, we proposed a novel Semantic-Constraint Matching (SeCM) plug-in module tailored for transformer-based approaches. In detail, a local patch shuffle strategy is first introduced to disentangle partial contextual linkages, thereby creating image pairs. Then a semantic matching module extracts co-object knowledge from the primal-shuffled image pairs, drives the network to identify the association of foreground with semantic label to suppress divergent activation. Moreover, to alleviate incomplete localization and prevent excessive suppression of activation, we propose leveraging multi-modal class-specific textual representations to guide object localization by complementing intra-class priori diverse knowledge. Extensive experimental results conducted on CUB-200-2011 and ILSVRC datasets show that our method can achieve the new state-of-the-art performance.

TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization.

Weakly supervised object localization (WSOL), which trains object localization models using solely image category annotations, remains a challenging problem. Existing approaches based on convolutional neural networks (CNNs) tend to miss full object extent while activating discriminative object parts. Based on our analysis, this is caused by CNN's intrinsic characteristics, which experiences difficulty to capture object semantics at long distances. In this article, we introduce the vision transformer to WSOL, with the aim to capture long-range semantic dependency of features by leveraging transformer's cascaded self-attention mechanism. We propose the token semantic coupled attention map (TS-CAM) method, which first decomposes class-aware semantics and then couples the semantics with attention maps for semantic-aware activation. To capture object semantics at long distances and avoid partial activation, TS-CAM performs spatial embedding by partitioning an image to a set of patch tokens. To incorporate object category information to patch tokens, TS-CAM reallocates category-related semantics to each patch token. The patch tokens are finally coupled with attention maps which are semantic-agnostic to perform semantic-aware object localization. By introducing semantic tokens to produce semantic-aware attention maps, we further explore the capability of TS-CAM for multicategory object localization. Experiments show that TS-CAM outperforms its CNN-CAM counterpart by 11.6% and 28.9% on ILSVRC and CUB-200-2011 datasets, respectively, improving the state-of-the-art with large margins. TS-CAM also demonstrates superiority for multicategory object localization on the Pascal VOC dataset. The code is available at github.com/yuanyao366/ts-cam-extension.

PCSformer: Pair-wise Cross-scale Sub-prototypes mining with CNN-transformers for weakly supervised semantic segmentation

Generating initial seeds is an important step in weakly supervised semantic segmentation (WSSS). Our approach concentrates on generating and refining initial seeds. The convolutional neural networks (CNNs)–based initial seeds focus only on the most discriminative regions and lack global information about the target. The Vision Transformer (ViT)–based approach can capture long-range feature dependencies due to the unique advantage of the self-attention mechanism. Still, we find that it suffers from distractor object leakage and background leakage problems. Based on these observations, we propose PCSformer, which improves the model’s ability to extract features through a Pair-wise Cross-scale (PC) strategy and solves the problem of distractor object leakage by further extracting potential target features through Sub-Prototypes (SP) mining. In addition, the proposed Conflict Self-Elimination (CSE) module further alleviates the background leakage problem. We validate our approach on the widely adopted Pascal VOC 2012 and MS COCO 2014, and extensive experiments demonstrate our superior performance. Furthermore, our method proves to be competitive for WSSS in medical images and challenging scenarios involving deformable and cluttered scenes. Additionally, we extend the PCSformer to weakly supervised object localization tasks, further highlighting its scalability and versatility. The code is available at https://github.com/ChunmengLiu1/PCSformer.

Clustering-inspired channel selection method for weakly supervised object localization

Weakly Supervised Object Localization (WSOL) aims to utilize the features learned by a classifier on the image-level labels to locate target objects. However, these existing channel selection methods for WSOL still cannot effectively select the important channels and remove the unimportant ones. To address this issue, we propose a Clustering-inspired Channel Selection method based on Class Activation Maps (CCS-CAM). Compared with the traditional methods, the advantage of CCS-CAM is that it is very simple yet effective for channel selection due to the K-means clustering based on Class Activation Maps. It can effectively ensure both object localization and classification accuracy. The effectiveness of the proposed CCS-CAM method has been demonstrated using multiple public datasets, with GT-Know Loc reaching 87.9% and 63.71% on the CUB200-2011 and ImageNet-1k respectively, which is superior to the other state-of-the-art methods.

Generalized Weakly Supervised Object Localization.

With the goal of learning to localize specific object semantics using the low-cost image-level annotation, weakly supervised object localization (WSOL) has been receiving increasing attention in recent years. Although existing literatures have studied a number of major issues in this field, one important yet challenging scenario, where the test object semantics may appear in the training phase (seen categories) or never been observed before (unseen categories), is still beyond the exploration of the existing works. We define this scenario as the generalized WSOL (GWSOL) and make a pioneering effort to study it in this article. By leveraging attribute vectors to associate seen and unseen categories, we involve threefold modeling components, i.e., the class-sensitive modeling, semantic-agnostic modeling, and content-aware modeling, into a unified end-to-end learning framework. Such design enables our model to recognize and localize unconstrained object semantics, learn compact and discriminative features that could represent the potential unseen categories, and customize content-aware attribute weights to avoid localizing on misleading attribute elements. To advance this research direction, we contribute the bounding-box manual annotations to the widely used AwA2 dataset and benchmark the GWSOL methods. Comprehensive experiments demonstrate the effectiveness of our proposed learning framework and each of the considered modeling components.

Feature disparity learning for weakly supervised object localization

Weakly supervised object localization (WSOL) aims to localize objects with only image-level labels. As a common WSOL method, adversarial erasing always masks the most discriminative region in the feature space to compel the network to localize more regions of the object. However, with the discriminative region vanishing, the localizer is confused when distinguishing the regions of object from the background. In this paper, we propose a new feature disparity learning (FDL), which encourages the network to learn more distinctive features from the object region with similarity measurement after feature enhancement. Specifically, we first introduce a Spatial Vector Cross Attention (SVCA) module. This module enhances responses in less discriminative region of erased feature maps by reintegrating the spatial distribution of features through the capture of interdependencies among spatial vectors on each channel. Furthermore, we propose a feature complementarity loss to measure the similarity between unerased features and erased features, guiding the network to learn feature disparities caused by adversarial erasing for improved localization and classification. Several experimental studies demonstrate a significant increase in localization performance over the existing state-of-the-art erasing methods on the CUB 200–2011 and ILSVRC 2016 datasets.

Anti-Adversarially Manipulated Attributions for Weakly Supervised Semantic Segmentation and Object Localization.

Obtaining accurate pixel-level localization from class labels is a crucial process in weakly supervised semantic segmentation and object localization. Attribution maps from a trained classifier are widely used to provide pixel-level localization, but their focus tends to be restricted to a small discriminative region of the target object. An AdvCAM is an attribution map of an image that is manipulated to increase the classification score produced by a classifier before the final softmax or sigmoid layer. This manipulation is realized in an anti-adversarial manner, so that the original image is perturbed along pixel gradients in directions opposite to those used in an adversarial attack. This process enhances non-discriminative yet class-relevant features, which make an insufficient contribution to previous attribution maps, so that the resulting AdvCAM identifies more regions of the target object. In addition, we introduce a new regularization procedure that inhibits the incorrect attribution of regions unrelated to the target object and the excessive concentration of attributions on a small region of the target object. Our method achieves a new state-of-the-art performance in weakly and semi-supervised semantic segmentation, on both the PASCAL VOC 2012 and MS COCO 2014 datasets. In weakly supervised object localization, it achieves a new state-of-the-art performance on the CUB-200-2011 and ImageNet-1K datasets.

Weakly Supervised Object Localization with Background Suppression Erasing for Art Authentication and Copyright Protection

Weakly Supervised Object Localization with Background Suppression Erasing for Art Authentication and Copyright Protection

Multi-Layer Decoupling Attention Network for Weakly Supervised Object Localization

Multi-Layer Decoupling Attention Network for Weakly Supervised Object Localization

Re-perceive Global Vision of Transformer for Remote Sensing Images Weakly Supervised Object Localization

Re-perceive Global Vision of Transformer for Remote Sensing Images Weakly Supervised Object Localization

Localizing From Classification: Self-Directed Weakly Supervised Object Localization for Remote Sensing Images.

In recent years, object localization and detection methods in remote sensing images (RSIs) have received increasing attention due to their broad applications. However, most previous fully supervised methods require a large number of time-consuming and labor-intensive instance-level annotations. Compared with those fully supervised methods, weakly supervised object localization (WSOL) aims to recognize object instances using only image-level labels, which greatly saves the labeling costs of RSIs. In this article, we propose a self-directed weakly supervised strategy (SD-WSS) to perform WSOL in RSIs. To specify, we fully exploit and enhance the spatial feature extraction capability of the RSIs' classification model to accurately localize the objects of interest. To alleviate the serious discriminative region problem exhibited by previous WSOL methods, the spatial location information implicit in the classification model is carefully extracted by GradCAM ++ to guide the learning procedure. Furthermore, to eliminate the interference from complex backgrounds of RSIs, we design a novel self-directed loss to make the model optimize itself and explicitly tell it where to look. Finally, we review and annotate the existing remote sensing scene classification dataset and create two new WSOL benchmarks in RSIs, named C45V2 and PN2. We conduct extensive experiments to evaluate the proposed method and six mainstream WSOL methods with three backbones on C45V2 and PN2. The results demonstrate that our proposed method achieves better performance when compared with state-of-the-arts.

Exploring Intrinsic Discrimination and Consistency for Weakly Supervised Object Localization.

Weakly supervised object localization (WSOL) is a challenging and promising task that aims to localize objects solely based on the supervision of image category labels. In the absence of annotated bounding boxes, WSOL methods must employ the intrinsic properties of the image classification task pipeline to generate object localizations. In this work, we propose a WSOL method for exploring the Intrinsic Discrimination and Consistency in the image classification task pipeline, and call it as IDC. First, we develop a Triplet Metrics Based Foreground Modeling (TMFM) framework to directly predict object foreground regions using intrinsic discrimination. Unlike Class Activation Map (CAM) based methods that also rely on intrinsic discrimination, our TMFM framework alleviates the problem of only focusing on the most discriminative parts by optimizing foreground and background regions synergistically. Second, we design a Dual Geometric Transformation Consistency Constraints (DGTC2) training strategy to introduce additional supervision and regularization constraints for WSOL by leveraging intrinsic geometric transformation consistency. The proposed pixel-wise and object-wise consistency constraint losses cost-effectively provide spontaneous supervision for WSOL. Extensive experiments show that our IDC method achieves significant and consistent performance gains compared to existing state-of-the-art WSOL approaches. Code is available at: https://github.com/vignywang/IDC.

Learning Local Semantic Region Activations for Weakly Supervised Object Localization

Learning Local Semantic Region Activations for Weakly Supervised Object Localization

Boosting Weakly Supervised Object Localization and Segmentation With Domain Adaption.

Weakly supervised object localization (WSOL), adopting only image-level annotations to learn the pixel-level localization model, can release human resources in the annotation process. Most one-stage WSOL methods learn the localization model with multi-instance learning, making them only activate discriminative object parts rather than the whole object. In our work, we attribute this problem to the domain shift between the training and test process of WSOL and provide a novel perspective that views WSOL as a domain adaption (DA) task. Under this perspective, a DA-WSOL pipeline is elaborated to better assist WSOL with DA approaches by considering the specificities for the adaption of WSOL. Our DA-WSOL pipeline can discern the source-related and the Universum samples from other target samples based on a proposed target sampling strategy and then utilize them to solve the sample unbalancing and label unmatching between the source and target domain of WSOL. Experiments show that our pipeline outperforms SOTA methods on three WSOL benchmarks and can improve the performance of downstream weakly supervised semantic segmentation tasks. Codes are available at https://github.com/zh460045050/dawsol.

Adaptive Zone Learning for Weakly Supervised Object Localization.

Weakly supervised object localization (WSOL) stands as a pivotal endeavor within the realm of computer vision, entailing the location of objects utilizing merely image-level labels. Contemporary approaches in WSOL have leveraged FPMs, yielding commendable outcomes. However, these existing FPM-based techniques are predominantly confined to rudimentary strategies of either augmenting the foreground or diminishing the background presence. We argue for the exploration and exploitation of the intricate interplay between the object's foreground and its background to achieve efficient object localization. In this manuscript, we introduce an innovative framework, termed adaptive zone learning (AZL), which operates on a coarse-to-fine basis to refine FPMs through a triad of adaptive zone mechanisms. First, an adversarial learning mechanism (ALM) is employed, orchestrating an interplay between the foreground and background regions. This mechanism accentuates coarse-grained object regions in a mutually adversarial manner. Subsequently, an oriented learning mechanism (OLM) is unveiled, which harnesses local insights from both foreground and background in a fine-grained manner. This mechanism is instrumental in delineating object regions with greater granularity, thereby generating better FPMs. Furthermore, we propose a reinforced learning mechanism (RLM) as the compensatory mechanism for adversarial design, by which the undesirable foreground maps are refined again. Extensive experiments on CUB-200-2011 and ILSVRC datasets demonstrate that AZL achieves significant and consistent performance improvements over other state-of-the-art WSOL methods.

Weakly supervised object localization via knowledge distillation based on foreground–background contrast

Weakly supervised object localization (WSOL) is a challenging task that aims to localize objects in images using only image-level labels. Despite the widespread use of WSOL methods based on class activation mapping (CAM), such methods do not consider that the network may overly focus on local regions of the most interesting objects during localization, thus neglecting the overall information. To address this issue, we introduces an additional attention branch for convolutional neural networks (CNNs) that utilizes the attention mechanism of multi-layer perceptron (MLP) to enhance the network’s learning of global information and supervise the feature learning of CNNs online through knowledge distillation, thereby improving the localization accuracy of WSOL. Specifically, we designs a new loss function using the generated features to combine with contrastive learning, effectively dividing the foreground and background of the image to provide more accurate pseudo-labels for subsequent classification and localization tasks. In the experiments, we tested our method on the CUB-200-2011 dataset and compared it with existing methods. The experimental results show that our method achieves good performance in WSOL tasks.

Background-Aware Classification Activation Map for Weakly Supervised Object Localization.

Weakly supervised object localization (WSOL) relaxes the requirement of dense annotations for object localization by using image-level annotation to supervise the learning process. However, most WSOL methods only focus on forcing the object classifier to produce high activation score on object parts without considering the influence of background locations, causing excessive background activations and ill-pose background score searching. Based on this point, our work proposes a novel mechanism called the background-aware classification activation map (B-CAM) to add background awareness for WSOL training. Besides aggregating an object image-level feature for supervision, our B-CAM produces an additional background image-level feature to represent the pure-background sample. This additional feature can provide background cues for the object classifier to suppress the background activations on object localization maps. Moreover, our B-CAM also trained a background classifier with image-level annotation to produce adaptive background scores when determining the binary localization mask. Experiments indicate the effectiveness of the proposed B-CAM on four different types of WSOL benchmarks, including CUB-200, ILSVRC, OpenImages, and VOC2012 datasets.

Module of Axis-based Nexus Attention for weakly supervised object localization

Weakly supervised object localization tasks remain challenging to identify and segment an entire object rather than only discriminative parts of the object. To tackle this problem, corruption-based approaches have been devised, which involve the training of non-discriminative regions by corrupting (e.g., erasing) the input images or intermediate feature maps. However, this approach requires an additional hyperparameter, the corrupting threshold, to determine the degree of corruption and can unfavorably disrupt training. It also tends to localize object regions coarsely. In this paper, we propose a novel approach, Module of Axis-based Nexus Attention (MoANA), which helps to adaptively activate less discriminative regions along with the class-discriminative regions without an additional hyperparameter, and elaborately localizes an entire object. Specifically, MoANA consists of three mechanisms (1) triple-view attentions representation, (2) attentions expansion, and (3) features calibration mechanism. Unlike other attention-based methods that train a coarse attention map with the same values across elements in feature maps, MoANA trains fine-grained values in an attention map by assigning different attention values to each element. We validated MoANA by comparing it with various methods. We also analyzed the effect of each component in MoANA and visualized attention maps to provide insights into the calibration.

Integration of Multi-scale CAM and Attention for Weakly Supervised Defects Localization on Surface Defective Apple

Weakly supervised object localization (WSOL) is a task of localizing an object in an image using only image-level labels. Previous studies have followed the conventional class activation mapping (CAM) pipeline. However, we reveal the current CAM approach suffers from problems which cause original CAM could not capture the complete defects features. This work utilizes a convolutional neural network (CNN) pretrained on image-level labels to generate class activation maps in a multi-scale manner to highlight discriminative regions. Additionally, a vision transformer (ViT) pretrained was treated to produce multi-head attention maps as an auxiliary detector. By integrating the CNN-based CAMs and attention maps, our approach localizes defective regions without requiring bounding box or pixel-level supervision during training. We evaluate our approach on a dataset of apple images with only image-level labels of defect categories. Experiments demonstrate our proposed method aligns with several Object Detection models performance, hold a promise for improving localization.