Toward Open Set Recognition

State-of-the-art deep neural network recognition systems are designed for a static and closed world. It is usually assumed that the distribution at test time will be the same as the distribution during training. As a result, classifiers are forced to categorise observations into one out of a set of predefined semantic classes. Robotic problems are dynamic and open world; a robot will likely observe objects that are from outside of the training set distribution. Classifier outputs in robotic applications can lead to real-world robotic action and as such, a practical recognition system should not silently fail by confidently misclassifying novel observations. We show how a deep metric learning classification system can be applied to such open set recognition problems, allowing the classifier to label novel observations as unknown. Further to detecting novel examples, we propose an open set active learning approach that allows a robot to efficiently query a user about unknown observations. Our approach enables a robot to improve its understanding of the true distribution of data in the environment, from a small number of label queries. Experimental results show that our approach significantly outperforms comparable methods in both the open set recognition and active learning problems.

Synthetic unknown class learning for learning unknowns

In open-set recognition (OSR), a promising strategy is exploiting pseudo-unknown data outside given K known classes as an additional K+1-th class to explicitly model potential open space. However, treating unknown classes without distinction is unequal for them relative to known classes due to the category-agnostic and scale-agnostic of the unknowns. This inevitably not only disrupts the inherent distributions of unknown classes but also incurs both class-wise and instance-wise imbalances between known and unknown classes. Ideally, the OSR problem should model the whole class space as K+∞, but enumerating all unknowns is impractical. Since the core of OSR is to effectively model the boundaries of known classes, this means just focusing on the unknowns nearing the boundaries of targeted known classes seems sufficient. Thus, as a compromise, we convert the open classes from infinite to K, with a novel concept Target-Aware Universum (TAU) and propose a simple yet effective framework Dual Contrastive Learning with Target-Aware Universum (DCTAU). In details, guided by the targeted known classes, TAU automatically expands the unknown classes from the previous 1 to K, effectively alleviating the distribution disruption and the imbalance issues mentioned above. Then, a novel Dual Contrastive (DC) loss is designed, where all instances irrespective of known or TAU are considered as positives to contrast with their respective negatives. Experimental results indicate DCTAU sets a new state-of-the-art.

Recently, deep learning-driven studies have been introduced for bioacoustic signal classification. Most of them, however, have the limitation that the input of the classifier needs to match with a trained label which is known as closed set recognition (CSR). To this end, the classifier trained by CSR would not cover a real stream task since the input of the classifier has so many variations. To combat real-world tasks, open set recognition (OSR) has been developed. In OSR, randomly collected inputs are fed to the classifier and the classifier predicts target classes and Unknown class. However, this OSR has been spotlighted in the studies of computer vision and speech domains while the domain of bioacoustic signal is less developed. Especially, to our best knowledge, OSR for animal sound classification has not been studied. This paper proposes a novel method for open set bioacoustic signal classification based on Class Anchored Clustering (CAC) loss with closed set unknown bioacoustic signals. To use the closed set unknown signals for training, a total of n +1 classes are used by adding one additional Unknown class to n target classes, and n +1 cross-entropy loss is added to the CAC loss. To evaluate the proposed method, we build an animal sound dataset that includes 101 species of sounds and compare its performance with baseline methods. In the experiments, our proposed method shows higher performance than other baseline methods in the area under the receiver operating curve for detecting target class and unknown class, the classification accuracy of open set signals, and classification accuracy for target classes. As a result, the closed set class samples are well classified while the open set unknown class can be also recognized with high accuracy at the same time.

Learning adversarial semantic embeddings for zero-shot recognition in open worlds

Though a variety of face recognition techniques have been proposed in the literature, only a few of them considered open set recognition problems, which involves the rejection of unregistered subjects in addition to identifying persons registered in the database. Transductive confidence machine (TCM) is a novel strategy for classification associated with valid confidence, with recognition reliability as the ground for rejection. Many popular classification algorithms, such as k-nearest neighbor (kNN), can be plugged into the TCM framework and applied to open-set face recognition. As kernel associative memory model (KAM) has been proposed earlier as an efficient tool for close-set face recognition, this paper extends the KAM model into TCM by proposing a novel nonconformity measurement and corresponding TCM-kAM algorithm. Performance comparisons with published TCM-KNN open-set face recognition methods were conducted with ORL and AR faces, with verified advantages.

A fingerprint spoof detector is a pattern classifier that is used to distinguish a live finger from a fake (spoof) one in the context of an automated fingerprint recognition system. Most spoof detectors are learning-based and rely on a set of training images. Consequently, the performance of any such spoof detector significantly degrades when encountering spoofs fabricated using novel materials not found in the training set. In real-world applications, the problem of fingerprint spoof detection must be treated as an open set recognition problem where incomplete knowledge of the fabrication materials used to generate spoofs is present at training time, and novel materials may be encountered during system deployment. To mitigate the security risk posed by novel spoofs, this paper introduces: 1) the use of the Weibull-calibrated SVM (W-SVM), which is relatively robust for open set recognition, as a novel-material detector and a spoof detector and 2) a scheme for the automatic adaptation of the W-SVM-based spoof detector to new spoof materials that leverages interoperability across classifiers. Experiments conducted on new partitions of the LivDet 2011 database designed for open set evaluation suggest: 1) a 97% increase in the error rate of the existing spoof detectors when tested using new spoof materials and 2) up to 44% improvement in spoof detection performance across spoof materials when the proposed adaptive approach is used.

Most pattern recognition systems are closed set recognition systems in which any input sample is to be classified as belonging to one of the given classes. This paper, however, addresses the open set recognition problem in which a test sample may either come from one of the labeled or come from an unknown class. The number of unknown classes could potentially be unlimited. A compact binary feature (CBF) generated by an ensemble binary classifier (EBC) is proposed to solve the open set recognition problem. This method can be regarded as a type of ECOC (Error-Correcting Output Codes) combined with modern CNN (Convolutional Neural Network) techniques and adapted for open set recognition. By randomly partitioning the known classes into two groups and training a binary classifier with CNN to separate them apart, and by repeating such a procedure for many times, rich information is extracted from the training set in the form of an EBC which can associate any test sample with a CBF that can be matched according to Hamming distance which is very efficient to compute. According to the experiments on the Dunhuang ancient Chinese character dataset, EBC can boost the recognition performance significantly compared with a single feedforward CNN. Apart from that, CBF is very efficient for storage and saves lots of time in feature matching at the cost of more computation in the training phase.

Historical Chinese character recognition has been suffering from the problem of samples labeling, not only the problem of lacking sufficient labeled training samples, but also of sample classes. So the scenario for Historical Chinese character recognition is "open set" recognition, where incomplete labeling of sample classes is present at training time, and unknown classes can be submitted to the system during testing. This paper proposes a method for open set Historical Chinese Character Recognition. For open set recognition, the features available in the training data cannot effectively characterize different kinds of unknown classes. We assume that the features which characterize unknown classes can be derived or learned from other similar data sets. We utilize an auxiliary data set combined with the open set training data set to learn good features to represent historical Chinese characters. The auxiliary data set is translated using Generative Adversarial Networks (GAN) to make sure that the translated data set is as close to the historical Chinese character dataset as possible. Then we construct a neural network for features extraction. The neural network is trained using an alternative training method with the translated auxiliary dataset and incomplete labeled historical Chinese character data set. Last, features are extracted from certain layer of the trained neural network. Unknown samples are detected using statistical modelling of the Euclidean metric between samples. Experimental results show that the proposed method is effective.

This paper provides a generic deep learning method to solve open set recognition problems. In open set recognition, only samples of a limited number of known classes are given for training. During inference, an open set recognizer must not only correctly classify samples from known classes, but also reject samples from unknown classes. Due to these specific requirements, conventional deep learning models that assume a closed set environment cannot be used. Therefore, special open set approaches were taken, including variants of support vector machines and generation-based state-of-the-art methods which model unknown classes by generated samples. In contrast, our proposed method models unknown classes by atypical subsets of training samples. The subsets are obtained through intra-class splitting (ICS). Based on a recently proposed two-stage algorithm using ICS, we propose a one-stage method based on alternating between ICS and the training of a deep neural network. Finally, several experiments were conducted to compare our proposed method with conventional and other state-of-the-art methods. The proposed method based on dynamic ICS showed a comparable or better performance than all considered existing methods regarding balanced accuracy.

Deep networks have produced significant gains for various visual recognition problems, leading to high impact academic and commercial applications. Recent work in deep networks highlighted that it is easy to generate images that humans would never classify as a particular object class, yet networks classify such images high confidence as that given class - deep network are easily fooled with images humans do not consider meaningful. The closed set nature of deep networks forces them to choose from one of the known classes leading to such artifacts. Recognition in the real world is open set, i.e. the recognition system should reject unknown/unseen classes at test time. We present a methodology to adapt deep networks for open set recognition, by introducing a new model layer, OpenMax, which estimates the probability of an input being from an unknown class. A key element of estimating the unknown probability is adapting Meta-Recognition concepts to the activation patterns in the penultimate layer of the network. Open-Max allows rejection of "fooling" and unrelated open set images presented to the system, OpenMax greatly reduces the number of obvious errors made by a deep network. We prove that the OpenMax concept provides bounded open space risk, thereby formally providing an open set recognition solution. We evaluate the resulting open set deep networks using pre-trained networks from the Caffe Model-zoo on ImageNet 2012 validation data, and thousands of fooling and open set images. The proposed OpenMax model significantly outperforms open set recognition accuracy of basic deep networks as well as deep networks with thresholding of SoftMax probabilities.

Open set source camera attribution and device linking

Open set recognition problems exist in many domains. For example in security, new malware classes emerge regularly; therefore malware classification systems need to identify instances from unknown classes in addition to discriminating between known classes. In this paper we present a neural network based representation for addressing the open set recognition problem. In this representation instances from the same class are close to each other while instances from different classes are further apart, resulting in statistically significant improvement when compared to other approaches on three datasets from two different domains.

Open Set Recognition (OSR) poses significant challenges in distinguishing known from unknown classes. In OSR, the overconfidence problem has become a persistent obstacle, where visual recognition models often misclassify unknown objects as known objects with high confidence. This issue stems from the fact that visual recognition models often lack the integration of common-sense knowledge, a feature that is naturally present in language-based models but lacking in visual recognition systems. In this paper, we propose a novel approach to enhance OSR performance by distilling common-sense knowledge into visual prompts. Utilizing text prompts that embody common-sense knowledge about known classes, the proposed visual prompt is learned by extracting semantic common-sense features and aligning them with image features from visual recognition models. The unique aspect of this work is the training of individual visual prompts for each class to encapsulate this common-sense knowledge. Our methodology is model-agnostic, capable of enhancing OSR across various visual recognition models, and computationally light as it focuses solely on training the visual prompts. This research introduces a method for addressing OSR, aiming at a more systematic integration of visual recognition systems with common-sense knowledge. The obtained results indicate an enhancement in recognition accuracy, suggesting the applicability of this approach in practical settings.

Application of deep neural networks in learning underlying dermoscopic patterns and classifying skin-lesion pathology is crucial. It can help in early diagnosis which can lead to timely therapeutic intervention and efficacy. To establish the clinical applicability of such techniques it is important to delineate each pathology with superior accuracy. However, with innumerable types of skin conditions and supervised closed class classification methods trained on limited classes, applicability into clinical workflow could be unattainable. To mitigate this issue our work considers this as an open-set recognition problem. The technique is divided into two stages, closed-set classification of labelled data and open-set recognition for unknown classes which employs an autoencoder for conditional reconstruction of the input image. We compare our technique to a traditional baseline method and demonstrate on ISIC and Derm7pt data, higher accuracy and sensitivity for known as well as unknown classes. In summary, our open-set recognition method for dermoscopic images illustrates high clinical applicability.