Unseen Object Classes Research Articles

Physics-Informed Masked Autoencoder for active sparse imaging

Imaging technology based on detecting individual photons has seen tremendous progress in recent years, with broad applications in autonomous driving, biomedical imaging, astronomical observation, and more. Comparing with conventional methods, however, it takes much longer time and relies on sparse and noisy photon-counting data to form an image. Here we introduce Physics-Informed Masked Autoencoder (PI-MAE) as a fast and efficient approach for data acquisition and image reconstruction through hardware implementation of the MAE (Masked Autoencoder). We examine its performance on a single-photon LiDAR system when trained on digitally masked MNIST data. Our results show that, with 1.8×10-6\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$1.8\ imes 10^{-6}$$\\end{document} or less detected photons per pulse and down to 9 detected photons per pixel, it achieves high-quality image reconstruction on unseen object classes with 90% physical masking. Our results highlight PI-MAE as a viable hardware accelerator for significantly improving the performance of single-photon imaging systems in photon-starving applications.

Vision-aided grasp classification: design and evaluation of compact CNN for prosthetic hands

Powered prosthetic hands capable of executing various grasp patterns are highly sought-after solutions for upper limb amputees. A crucial requirement for such prosthetic hands is the accurate identification of the intended grasp pattern and subsequent activation of the prosthetic digits accordingly. Vision-based grasp classification techniques offer improved coordination between amputees and prosthetic hands without physical contact. Deep learning methods, particularly Convolutional Neural Networks (CNNs), are utilized to process visual information for classification. The key challenge lies in developing a model that can effectively generalize across various object shapes and accurately classify grasp classes. To address this, a compact CNN model named GraspCNet is proposed, specifically designed for grasp classification in prosthetic hands. The use of separable convolutions reduces the computational burden, making it potentially suitable for real-time applications on embedded systems. The GraspCNet model is designed to learn and generalize from object shapes, allowing it to effectively classify unseen objects beyond those included in the training dataset. The proposed model was trained and tested using various standard object data sets. A cross-validation strategy has been adopted to perform better in seen and unseen object class scenarios. The average accuracy achieved was 82.22% and 75.48% in the case of seen, and unseen object classes respectively. In computer-based real-time experiments, the GraspCNet model achieved an accuracy of 69%. A comparative analysis with state-of-the-art techniques revealed that the proposed GraspCNet model outperformed most benchmark techniques and demonstrated comparable performance with the DcnnGrasp method. The compact nature of the GraspCNet model suggests its potential for integration with other sensing modalities in prosthetic hands.

Attention-Assisted Feature Comparison and Feature Enhancement for Class-Agnostic Counting.

In this study, we address the class-agnostic counting (CAC) challenge, aiming to count instances in a query image, using just a few exemplars. Recent research has shifted towards few-shot counting (FSC), which involves counting previously unseen object classes. We present ACECount, an FSC framework that combines attention mechanisms and convolutional neural networks (CNNs). ACECount identifies query image-exemplar similarities, using cross-attention mechanisms, enhances feature representations with a feature attention module, and employs a multi-scale regression head, to handle scale variations in CAC. ACECount's experiments on theFSC-147 dataset exhibited the expected performance. ACECount achieved a reduction of 0.3 in the mean absolute error (MAE) on the validation set and a reduction of 0.26 on the test set of FSC-147, compared to previous methods. Notably, ACECount also demonstrated convincing performance in class-specific counting (CSC) tasks. Evaluation on crowd and vehicle counting datasets revealed that ACECount surpasses FSC algorithms like GMN, FamNet, SAFECount, LOCA, and SPDCN, in terms of performance. These results highlight the robust dataset generalization capabilities of our proposed algorithm.

Implicit and explicit attention mechanisms for zero-shot learning

Zero-Shot Learning (ZSL) aims to recognise unseen object classes which are not observed during the training phase. Most of the existing methods on ZSL focus on learning a compatibility function between the image representation and class semantic information. Few others concentrate on learning image representation by combining local and global features. However, the existing approaches still fail to address the bias issue towards the seen classes. This paper proposes implicit and explicit attention mechanisms to address the existing bias problem in generalised ZSL models. We formulate the implicit attention mechanism with a self-supervised image angle rotation task, which focuses on specific image features aiding in solving the task. On the other hand, the explicit attention mechanism is composed via the consideration of a multi-headed self-attention mechanism in the Vision Transformer model, which learns to attend important image locations and map global image features to semantic space during the training stage. We have conducted comprehensive experiments on three popular benchmarks: AWA2, CUB and SUN, where the effectiveness of our proposed attention mechanisms is shown in both discriminative and generative settings. Our extensive experiments show that our method has achieved state-of-the-art performance obtaining the highest harmonic mean on all three datasets, which is very encouraging to consider the ViT-based attention mechanisms for ZSL tasks in the future.

Zero-Shot Object Detection: Joint Recognition and Localization of Novel Concepts

Zero shot learning (ZSL) identifies unseen objects for which no training images are available. Conventional ZSL approaches are restricted to a recognition setting where each test image is categorized into one of several unseen object classes. We posit that this setting is ill-suited for real-world applications where unseen objects appear only as a part of a complete scene, warranting both ‘recognition’ and ‘localization’ of the unseen category. To address this limitation, we introduce a new ‘Zero-Shot Detection’ (ZSD) problem setting, which aims at simultaneously recognizing and locating object instances belonging to novel categories, without any training samples. We introduce an integrated solution to the ZSD problem that jointly models the complex interplay between visual and semantic domain information. Ours is an end-to-end trainable deep network for ZSD that effectively overcomes the noise in the unsupervised semantic descriptions. To this end, we utilize the concept of meta-classes to design an original loss function that achieves synergy between max-margin class separation and semantic domain clustering. In order to set a benchmark for ZSD, we propose an experimental protocol for the large-scale ILSVRC dataset that adheres to practical challenges, e.g., rare classes are more likely to be the unseen ones. Furthermore, we present a baseline approach extended from conventional recognition to the ZSD setting. Our extensive experiments show a significant boost in performance (in terms of mAP and Recall) on the imperative yet difficult ZSD problem on ImageNet detection, MSCOCO and FashionZSD datasets.

Hierarchical Open-Set Object Detection in Unseen Data

In this paper, we propose an open-set object detection framework based on a dynamic hierarchical structure with incremental learning capabilities for unseen object classes. We were motivated by the observation that deep features extracted from visual objects show a strong hierarchical clustering property. The hierarchical feature model (HFM) was used to learn a new object class by using collaborative sampling (CS), and open-set-aware active semi-supervised learning (ASSL) algorithms. We divided object proposals into superclasses by using the agglomerative clustering algorithm. Data samples in each superclass node were classified into multiple augmented class nodes instead of directly associating with regular object classes. One or more augmented class nodes are related to a regular object class, and each augmented class has only one superclass. Object proposals from inexperienced data distribution are assigned to an augmented class node. Dynamic HFM nodes in the decision path are assembled to constitute an ensemble prediction, and the new augmented object is associated with a new regular object class. Our experimental results showed that the proposed method uses standard benchmark datasets such as PASCAL VOC, MS COCO, ILSVRC DET, and local datasets to perform better than state-of-the-art techniques.

Transductive Zero-Shot Learning With a Self-Training Dictionary Approach.

As an important and challenging problem in computer vision, zero-shot learning (ZSL) aims at automatically recognizing the instances from unseen object classes without training data. To address this problem, ZSL is usually carried out in the following two aspects: 1) capturing the domain distribution connections between seen classes data and unseen classes data and 2) modeling the semantic interactions between the image feature space and the label embedding space. Motivated by these observations, we propose a bidirectional mapping-based semantic relationship modeling scheme that seeks for cross-modal knowledge transfer by simultaneously projecting the image features and label embeddings into a common latent space. Namely, we have a bidirectional connection relationship that takes place from the image feature space to the latent space as well as from the label embedding space to the latent space. To deal with the domain shift problem, we further present a transductive learning approach that formulates the class prediction problem in an iterative refining process, where the object classification capacity is progressively reinforced through bootstrapping-based model updating over highly reliable instances. Experimental results on four benchmark datasets (animal with attribute, Caltech-UCSD Bird2011, aPascal-aYahoo, and SUN) demonstrate the effectiveness of the proposed approach against the state-of-the-art approaches.

Interactive Medical Image Segmentation Using Deep Learning With Image-Specific Fine Tuning.

Convolutional neural networks (CNNs) have achieved state-of-the-art performance for automatic medical image segmentation. However, they have not demonstrated sufficiently accurate and robust results for clinical use. In addition, they are limited by the lack of image-specific adaptation and the lack of generalizability to previously unseen object classes (a.k.a. zero-shot learning). To address these problems, we propose a novel deep learning-based interactive segmentation framework by incorporating CNNs into a bounding box and scribble-based segmentation pipeline. We propose image-specific fine tuning to make a CNN model adaptive to a specific test image, which can be either unsupervised (without additional user interactions) or supervised (with additional scribbles). We also propose a weighted loss function considering network and interaction-based uncertainty for the fine tuning. We applied this framework to two applications: 2-D segmentation of multiple organs from fetal magnetic resonance (MR) slices, where only two types of these organs were annotated for training and 3-D segmentation of brain tumor core (excluding edema) and whole brain tumor (including edema) from different MR sequences, where only the tumor core in one MR sequence was annotated for training. Experimental results show that: 1) our model is more robust to segment previously unseen objects than state-of-the-art CNNs; 2) image-specific fine tuning with the proposed weighted loss function significantly improves segmentation accuracy; and 3) our method leads to accurate results with fewer user interactions and less user time than traditional interactive segmentation methods.