Zero-shot Learning Algorithms Research Articles

Estimating Wingbeat Frequency of Hummingbirds using a No-labeling Learning Computer Vision Approach.

Wingbeat frequency estimation is an important aspect for the study of avian flight, energetics, and behavioral patterns, among others. Hummingbirds, in particular, are ideal subjects to test a method for this estimation due to their fast wing motions and unique aerodynamics, which results from their ecological diversification, adaptation to high-altitude environments, and sexually selected displays. Traditionally, wingbeat frequency measurements have been done via "manual" image/sound processing. In this study, we present an automated method to detect, track, classify, and monitor hummingbirds in high-speed video footage, accurately estimating their wingbeat frequency using computer vision techniques and signal analysis. Our approach utilizes a zero-shot learning algorithm that eliminates the need for labeling during training. Results demonstrate that our method can produce automated wingbeat frequency estimations with minimal supervision, closely matching those performed by trained human observers. This comparison indicates that our method can, in some scenarios, achieve low or zero error compared to a human, making it a valuable tool for flight analysis. Automating video analysis can assist wingbeat frequency estimation by reducing processing time and, thus, lowering barriers to analyze biological data on fields such as aerodynamics, foraging behavior, and signaling.

Zero-Shot Learning Algorithms for Object Recognition in Medical and Navigation Applications

One category-invariant methodology for object identification is zero-shot learning (ZSL), which uses semantic embeddings to categorize unseen categories. The ZSL method is indispensable in fields where data is scarce, including medical diagnosis and navigation. This paper proposes an Improved ZSL (I-ZSL) framework to increase the object recognition accuracy and generalization for the medical and navigation applications. The proposed framework is a hybrid architecture that uses Variational Autoencoders (VAEs) for robust feature generation and Transformer-based embeddings for semantic alignment. A domain-adaptive classifier, trained through contrastive learning, bridges the gap between the seen and unseen classes. The classifier has identified the framework with minimal training data in medical diagnostics for rare disease diagnosis. The proposed I-ZSL framework achieved a 20% improvement in F1-score over state-of-the-art models. In navigation, it demonstrated 25% better performance in novel landmark recognition under dynamic environmental conditions. These results show the framework's efficiency in addressing domain-specific challenges. This work presents ZSL with great potential to further object recognition in applications with significant impacts.

Zero-Shot Learning for Accurate Project Duration Prediction in Crowdsourcing Software Development

Crowdsourcing Software Development (CSD) platforms, i.e., TopCoder, function as intermediaries connecting clients with developers. Despite employing systematic methodologies, these platforms frequently encounter high task abandonment rates, with approximately 19% of projects failing to meet satisfactory outcomes. Although existing research has focused on task scheduling, developer recommendations, and reward mechanisms, there has been insufficient attention to the support of platform moderators, or copilots, who are essential to project success. A critical responsibility of copilots is estimating project duration; however, manual predictions often lead to inconsistencies and delays. This paper introduces an innovative machine learning approach designed to automate the prediction of project duration on CSD platforms. Utilizing historical data from TopCoder, the proposed method extracts pertinent project attributes and preprocesses textual data through Natural Language Processing (NLP). Bidirectional Encoder Representations from Transformers (BERT) are employed to convert textual information into vectors, which are then analyzed using various machine learning algorithms. Zero-shot learning algorithms exhibit superior performance, with an average accuracy of 92.76%, precision of 92.76%, recall of 99.33%, and an f-measure of 95.93%. The implementation of the proposed automated duration prediction model is crucial for enhancing the success rate of crowdsourcing projects, optimizing resource allocation, managing budgets effectively, and improving stakeholder satisfaction.

Generalized zero-shot domain adaptation via coupled conditional variational autoencoders

Domain adaptation aims to exploit useful information from the source domain where annotated training data are easier to obtain to address a learning problem in the target domain where only limited or even no annotated data are available. In classification problems, domain adaptation has been studied under the assumption all classes are available in the target domain regardless of the annotations. However, a common situation where only a subset of classes in the target domain are available has not attracted much attention. In this paper, we formulate this particular domain adaptation problem within a generalized zero-shot learning framework by treating the labelled source-domain samples as semantic representations for zero-shot learning. For this novel problem, neither conventional domain adaptation approaches nor zero-shot learning algorithms directly apply. To solve this problem, we present a novel Coupled Conditional Variational Autoencoder (CCVAE) which can generate synthetic target-domain image features for unseen classes from real images in the source domain. Extensive experiments have been conducted on three domain adaptation datasets including a bespoke X-ray security checkpoint dataset to simulate a real-world application in aviation security. The results demonstrate the effectiveness of our proposed approach both against established benchmarks and in terms of real-world applicability.

Unknown network attack detection method based on reinforcement zero-shot learning

With the increasing growth of zero-day attacks, traditional machine learning-based network intrusion detection systems (NIDS) are difficult to cope with a large amount of unknown network attacks without labeled data. To this end, this paper proposes a new unknown network attack detection method, which combines zero-shot learning algorithm with reinforcement learning algorithm. First, the feature vector in traffic data and the semantic vector in threat intelligence are encoded in the hidden space by variational autoencoder, so that the two modalities are matched in the hidden space, and then the hidden features of known class and unknown class are used to input into the training classifier to get the predicted of the labeling results, and then use asynchronous advantage actor-critic based algorithm to learn and correct the attack results that are misclassified by the variational autoencoder to improve the accuracy of detection. Experiments prove that the proposed method in this paper achieves more than 95% accuracy on four unknown network attack classes that are difficult to identify, namely MAILBOMB, U2R, POD, and SNAPATTACK, which illustrates the effectiveness and feasibility of the method.

Attributes learning network for generalized zero-shot learning

In the absence of unseen training data, zero-shot learning algorithms utilize the semantic knowledge shared by the seen and unseen classes to establish the connection between the visual space and the semantic space, so as to realize the recognition of the unseen classes. However, in real applications, the original semantic representation cannot well characterize both the class-specificity structure and discriminative information in dimension space, which leads to unseen classes being easily misclassified into seen classes. To tackle this problem, we propose a Salient Attributes Learning Network (SALN) to generate discriminative and expressive semantic representation under the supervision of the visual features. Meanwhile, ℓ1,2-norm constraint is employed to make the learned semantic representation well characterize the class-specificity structure and discriminative information in dimension space. Then feature alignment network projects the learned semantic representation into visual space and a relation network is adopted for classification. The performance of the proposed approach has made progress on the five benchmark datasets in generalized zero-shot learning task, and in-depth experiments indicate the effectiveness and excellence of our method.

Zero-shot image classification using coupled dictionary embedding

Zero-shot learning (ZSL) is a framework to classify images that belong to unseen visual classes using their semantic descriptions about the unseen classes. We develop a new ZSL algorithm based on coupled dictionary learning. The core idea is to enforce the visual features and the semantic attributes of an image to share the same sparse representation in an intermediate embedding space, modeled as the shared input space of two sparsifying dictionaries. In the ZSL training stage, we use images from a number of seen classes for which we have access to both the visual and the semantic attributes to train two coupled dictionaries that can represent both the visual and the semantic feature vectors of an image using a single sparse vector. In the ZSL testing stage and in the absence of labeled data, images from unseen classes are mapped into the attribute space by finding the joint-sparse representations using solely the visual dictionary via solving a LASSO problem. The image is then classified in the attribute space given semantic descriptions of unseen classes. We also provide attribute-aware and transductive formulations to tackle the “domain-shift” and the “hubness” challenges for ZSL, respectively. Experiments on four primary datasets using VGG19 and GoogleNet visual features, are provided. Our performances using VGG19 features are 91.0%, 48.4%, and 89.3% on the SUN, the CUB, and the AwA1 datasets, respectively. Our performances on the SUN, the CUB, and the AwA2 datasets are 57.0%,49.7%, and 71.7%, respectively, when GoogleNet features are used. Comparison with existing methods demonstrates that our method is effective and compares favorably against the state-of-the-art. In particular, our algorithm leads to decent performance on the all four datasets.22Early partial results of this paper is presented at 2018 AAAI (Kolouri, Rostami, Owechko, & Kim, 2018).

Dual discriminative auto-encoder network for zero shot image recognition

Zero Shot learning (ZSL) aims to use the information of seen classes to recognize unseen classes, which is achieved by transferring knowledge of the seen classes from the semantic embeddings. Since the domains of the seen and unseen classes do not overlap, most ZSL algorithms often suffer from domain shift problem. In this paper, we propose a Dual Discriminative Auto-encoder Network (DDANet), in which visual features and semantic attributes are self-encoded by using the high dimensional latent space instead of the feature space or the low dimensional semantic space. In the embedded latent space, the features are projected to both preserve their original semantic meanings and have discriminative characteristics, which are realized by applying dual semantic auto-encoder and discriminative feature embedding strategy. Moreover, the cross modal reconstruction is applied to obtain interactive information. Extensive experiments are conducted on four popular datasets and the results demonstrate the superiority of this method.

A novel dataset-specific feature extractor for zero-shot learning

Most of the existing Zero-Shot Learning(ZSL) algorithms adopt pre-trained neural networks as their feature extractors. Since these pre-trained models are not specially designed for ZSL tasks, it is difficult to guarantee the stability and generalization ability of the ZSL algorithms due to the feature mismatch. To alleviate this problem, we propose a novel dataset-specific feature extractor for ZSL according to an attribute-based label tree. Specifically, an attribute-based label tree is firstly built via K-means clustering and then the information extracted from the label tree is used to fine-tune the parameters of the pre-trained models in order to make the extracted features more suitable for the current ZSL task. The experimental results on three typical ZSL datasets show that our approach can effectively improve the predictive accuracy of the existing ZSL algorithms and significantly accelerate their convergence rate. Additionally we explain the experimental phenomena from the perspective of feature visualization, which experimentally show that the features extracted by our method are much more separable than those of the original pre-trained models.

Generalized zero-shot learning for action recognition with web-scale video data

Action recognition in surveillance video makes our life safer by detecting the criminal events or predicting violent emergencies. However, efficient action recognition is not free of difficulty. First, there are so many action classes in daily life that we cannot pre-define all possible action classes beforehand. Moreover, it is very hard to collect real-word videos for certain particular actions such as steal and street fight due to legal restrictions and privacy protection. These challenges make existing data-driven recognition methods insufficient to attain desired performance. Zero-shot learning is potential to be applied to solve these issues since it can perform classification without positive example. Nevertheless, current zero-shot learning algorithms have been studied under the unreasonable setting where seen classes are absent during the testing phase. Motivated by this, we study the task of action recognition in surveillance video under a more realistic generalized zero-shot setting, where testing data contains both seen and unseen classes. To our best knowledge, this is one of the first works to study video action recognition under the generalized zero-shot setting. We firstly perform extensive empirical studies on several existing zero-shot leaning approaches under this new setting on a web-scale video data. Our experimental results demonstrate that, under the generalize setting, typical zero-shot learning methods are no longer effective for the dataset we applied. Then, we propose to deploy generalized zero-shot learning which transfers the knowledge of Web video to detect the anomalous actions in surveillance videos. To verify the effectiveness of methods, we further construct a new surveillance video dataset consisting of nine action classes related to the public safety situation.

Fine-Grained Object Recognition and Zero-Shot Learning in Remote Sensing Imagery

Fine-grained object recognition that aims to identify the type of an object among a large number of subcategories is an emerging application with the increasing resolution that exposes new details in image data. Traditional fully supervised algorithms fail to handle this problem where there is low between-class variance and high within-class variance for the classes of interest with small sample sizes. We study an even more extreme scenario named zero-shot learning (ZSL) in which no training example exists for some of the classes. ZSL aims to build a recognition model for new unseen categories by relating them to seen classes that were previously learned. We establish this relation by learning a compatibility function between image features extracted via a convolutional neural network and auxiliary information that describes the semantics of the classes of interest by using training samples from the seen classes. Then, we show how knowledge transfer can be performed for the unseen classes by maximizing this function during inference. We introduce a new data set that contains 40 different types of street trees in 1-ft spatial resolution aerial data, and evaluate the performance of this model with manually annotated attributes, a natural language model, and a scientific taxonomy as auxiliary information. The experiments show that the proposed model achieves 14.3% recognition accuracy for the classes with no training examples, which is significantly better than a random guess accuracy of 6.3% for 16 test classes, and three other ZSL algorithms.

Recognizing unknown objects with attributes relationship model

Generally, training images are essential for a computer vision model to classify specific object class accurately. Unfortunately, there exist countless number of different object classes in real world, and it is almost impossible for a computer vision model to obtain a complete training images for each of the different object class. To overcome this problem, zero-shot learning algorithm was emerged to learn unknown object classes from a set of known object classes information. Among these methods, attributes and image hierarchy are the widely used methods. In this paper, we combine both the strength of attributes and image hierarchy by proposing Attributes Relationship Model (ARM) to perform zero-shot learning. We tested the efficiency of the proposed algorithm on Animals with Attributes (AwA) dataset and manage to achieve state-of-the-art accuracy (50.61%) compare to other recent methods.