Untrained Classes Research Articles

Zero-Shot Image Classification with Rectified Embedding Vectors Using a Caption Generator

Although image recognition technologies are developing rapidly with deep learning, conventional recognition models trained by supervised learning with class labels do not work well when test inputs from untrained classes are given. For example, a recognizer trained to classify Asian bird species cannot recognize the species of kiwi, because the class label “kiwi” and its image samples have not been seen during training. To overcome this limitation, zero-shot classification has been studied recently, and the joint-embedding-based approach has been suggested as one of the promised solutions. In this approach, image features and text descriptions belonging to the same class are trained to be closely located in a common joint-embedding space. Once we obtain the embedding function that can represent the semantic relationship of image–text pairs in training data, test images and text descriptions (prototypes) of unseen classes can also be mapped to the joint-embedding space for classification. The main challenge with this approach is mapping inputs of two different modalities into a common space, and previous works suffer from the inconsistency between the distribution of two feature sets on joint-embedding space extracted from the heterogeneous inputs. To treat this problem, we propose a novel method of employing additional textual information to rectify the visual representation of input images. Since the conceptual information of test classes is generally given as texts, we expect that the additional descriptions from a caption generator can adjust the visual feature for better matching with the representation of the test classes. We also propose to use the generated textual descriptions to augment training samples for learning joint-embedding space. In the experiments on two benchmark datasets, the proposed method shows significant performance improvements of 1.4% on the CUB dataset and 5.5% on the flower dataset, in comparison to existing models.

Adaptive adjustment with semantic embedding for zero-shot object detection

Traditional zero-shot object-detection algorithms detect images of untrained classes in the model with the help of semantic embedding. However, these approaches may perform poorly due to the limitations of fixed semantic embedding. Given that fixed semantic attributes lead to a lack of generalization capabilities in the model, a semantic enhancement mechanism is proposed to update the semantic embedding, which is used to serve the needs of the visual space. Specifically, considering that the original semantic space is not enough to construct a visual-semantic mapping relationship, an augmented semantic embedding (ASE) approach is designed to supplement semantic attribute information. Then, a semantic channel attention mechanism is used to adjust the ASE. The adjustment strategy retains adequate attribute information, which is highly relevant to visual features. Finally, to alleviate the domain shift problem, a clustering association strategy is introduced to establish an inferred relationship, which ensures that the predictor is generalized to the unseen domain during training. The superiority of the proposed method is demonstrated by the MS-COCO and PASCAL VOC datasets.

Prototype augmented network with metric-mixed under limited samples for mechanical intelligent fault recognition

Common deep learning methods can effectively extract fault characteristics and make recognition of mechanical faults. Nevertheless, the results are undesirable when sufficient labeled training data are unavailable. Although metric learning methods can conduct fault recognition with limited samples, they cannot get satisfactory results due to the fixed relation metric function of manually selected. To address this problem we proposed a prototype augmented network with metric-mixed under limited samples (PANM). It combines fixed metric function with the network automatically learning metric, which consists of an embedded module, relation module and metric module, respectively Firstly, the class prototypes are augmented via multilayer residual network in the embedded module. Then, in the relation module, the prototypes are concatenated with training features to automatically learn the corresponding relationship for getting relation scores. Next, the metric module is used to measure the fixed spatial distance and normalize as the weights of each score. Finally, fault identification is taken according to the weighted score. The experiments in two datasets indicate that PANM cannot just identify different fault categories; it may identify new untrained classes without modifying the network parameters as well, which demonstrates practical solving ability and good generalization ability of fault recognition under limited samples.

Node co-activations as a means of error detection—Towards fault-tolerant neural networks

Machine learning has proved an efficient tool, but the systems need tools to mitigate risks during runtime. One approach is fault tolerance: detecting and handling errors before they cause harm. This paper investigates whether rare co-activations – pairs of usually segregated nodes activating together – are indicative of problems in neural networks (NN). These could be used to detect concept drift and flagging untrustworthy predictions. We trained four NNs. For each, we studied how often each pair of nodes activates together. In a separate test set, we counted how many rare co-activations occurred with each input, and grouped the inputs based on whether its classification was correct, incorrect, or whether its class was absent during training. Rare co-activations are much more common in inputs from a class that was absent during training. Incorrectly classified inputs averaged a larger number of rare co-activations than correctly classified inputs, but the difference was smaller. As rare co-activations are more common in unprecedented inputs, they show potential for detecting concept drift. There is also some potential in detecting single inputs from untrained classes. The small difference between correctly and incorrectly predicted inputs is less promising and needs further research.

Rollback Ensemble With Multiple Local Minima in Fine-Tuning Deep Learning Networks.

Image retrieval is a challenging problem that requires learning generalized features enough to identify untrained classes, even with very few classwise training samples. In this article, to obtain generalized features further in learning retrieval data sets, we propose a novel fine-tuning method of pretrained deep networks. In the retrieval task, we discovered a phenomenon in which the loss reduction in fine-tuning deep networks is stagnated, even while weights are largely updated. To escape from the stagnated state, we propose a new fine-tuning strategy to roll back some of the weights to the pretrained values. The rollback scheme is observed to drive the learning path to a gentle basin that provides more generalized features than a sharp basin. In addition, we propose a multihead ensemble structure to create synergy among multiple local minima obtained by our rollback scheme. Experimental results show that the proposed learning method significantly improves generalization performance, achieving state-of-the-art performance on the Inshop and SOP data sets.

An active one-shot learning approach to recognizing land usage from class-wise sparse satellite imagery in smart urban sensing

Urban land usage recognition (ULUR) in smart urban sensing recognizes the physical attributes and socioeconomic functions of urban land resources using pervasive satellite imagery data. Reliable ULUR results are essential for urban management and planning applications. A key limitation of current ULUR solutions is that they cannot identify land usage classes that are not included in the training data (i.e., untrained classes). To address this limitation, this study investigates a new class-wise sparse ULUR problem in which the goal is to learn an accurate ULUR model that can effectively identify the land usage classes of images of the untrained classes. Two critical challenges exist in resolving this problem: (i) How can the ULUR model learn the unknown class-specific visual characteristics for untrained classes without ground truth labels? (ii) How can we effectively generalize the ULUR model from trained classes to untrained ones given the excessive and fine-grained object details in the images? This paper proposes SparseLand, an active one-shot learning approach, to address these challenges using a novel duo-branch deep recognition framework. The results of a real-world ULUR application demonstrated that SparseLand clearly outperforms state-of-the-art baselines with the highest ULUR accuracy under different application scenarios.

Exploring Fuzzy Local Spatial Information Algorithms for Remote Sensing Image Classification

Fuzzy c-means (FCM) and possibilistic c-means (PCM) are two commonly used fuzzy clustering algorithms for extracting land use land cover (LULC) information from satellite images. However, these algorithms use only spectral or grey-level information of pixels for clustering and ignore their spatial correlation. Different variants of the FCM algorithm have emerged recently that utilize local spatial information in addition to spectral information for clustering. Such algorithms are seen to generate clustering outputs that are more enhanced than the classical spectral-based FCM algorithm. Nonetheless, the scope of integrating spatial contextual information with the conventional PCM algorithm, which has several advantages over the FCM algorithm for supervised classification, has not been explored much. This study proposed integrating local spatial information with the PCM algorithm using simpler but proven approaches from available FCM-based local spatial information algorithms. The three new PCM-based local spatial information algorithms: Possibilistic c-means with spatial constraints (PCM-S), possibilistic local information c-means (PLICM), and adaptive possibilistic local information c-means (ADPLICM) algorithms, were developed corresponding to the available fuzzy c-means with spatial constraints (FCM-S), fuzzy local information c-means (FLICM), and adaptive fuzzy local information c-means (ADFLICM) algorithms. Experiments were conducted to analyze and compare the FCM and PCM classifier variants for supervised LULC classifications in soft (fuzzy) mode. The quantitative assessment of the soft classification results from fuzzy error matrix (FERM) and root mean square error (RMSE) suggested that the new PCM-based local spatial information classifiers produced higher accuracies than the PCM, FCM, or its local spatial variants, in the presence of untrained classes and noise. The promising results from PCM-based local spatial information classifiers suggest that the PCM algorithm, which is known to be naturally robust to noise, when integrated with local spatial information, has the potential to result in more efficient classifiers capable of better handling ambiguities caused by spectral confusions in landscapes.

Examination of Kernel Based Noise Classifier with Cross Resolution Dataset and with Untrained Class

Determining the effect of untrained classes in the kernel based noise classifier is the prime object of the paper. It further includes, studying the effect of studied classifier over different datasets. Distinct nine Kernel functions has been associated with conventional supervised Noise Classifier. Landsat8 and Formosat2 along with Resourcesat-1 data have been opted for the performance evaluation. Decrease in classification accuracy has been found, in presence of untrained classes. A subtle consistency has been in classification accuracy in case of cross resolution data sets, thus, showing the robustness of the algorithm.

Examination of Kernel Based Noise Classifier with Cross Resolution Dataset and with Untrained Class

Determining the effect of untrained classes in the kernel based noise classifier is the prime object of the paper. It further includes, studying the effect of studied classifier over different datasets. Distinct nine Kernel functions has been associated with conventional supervised Noise Classifier. Landsat8 and Formosat2 along with Resourcesat-1 data have been opted for the performance evaluation. Decrease in classification accuracy has been found, in presence of untrained classes. A subtle consistency has been in classification accuracy in case of cross resolution data sets, thus, showing the robustness of the algorithm.

Visualizing histopathologic deep learning classification and anomaly detection using nonlinear feature space dimensionality reduction

BackgroundThere is growing interest in utilizing artificial intelligence, and particularly deep learning, for computer vision in histopathology. While accumulating studies highlight expert-level performance of convolutional neural networks (CNNs) on focused classification tasks, most studies rely on probability distribution scores with empirically defined cutoff values based on post-hoc analysis. More generalizable tools that allow humans to visualize histology-based deep learning inferences and decision making are scarce.ResultsHere, we leverage t-distributed Stochastic Neighbor Embedding (t-SNE) to reduce dimensionality and depict how CNNs organize histomorphologic information. Unique to our workflow, we develop a quantitative and transparent approach to visualizing classification decisions prior to softmax compression. By discretizing the relationships between classes on the t-SNE plot, we show we can super-impose randomly sampled regions of test images and use their distribution to render statistically-driven classifications. Therefore, in addition to providing intuitive outputs for human review, this visual approach can carry out automated and objective multi-class classifications similar to more traditional and less-transparent categorical probability distribution scores. Importantly, this novel classification approach is driven by a priori statistically defined cutoffs. It therefore serves as a generalizable classification and anomaly detection tool less reliant on post-hoc tuning.ConclusionRoutine incorporation of this convenient approach for quantitative visualization and error reduction in histopathology aims to accelerate early adoption of CNNs into generalized real-world applications where unanticipated and previously untrained classes are often encountered.

Support Vector Data Description Model to Map Specific Land Cover with Optimal Parameters Determined from a Window-Based Validation Set.

This paper developed an approach, the window-based validation set for support vector data description (WVS-SVDD), to determine optimal parameters for support vector data description (SVDD) model to map specific land cover by integrating training and window-based validation sets. Compared to the conventional approach where the validation set included target and outlier pixels selected visually and randomly, the validation set derived from WVS-SVDD constructed a tightened hypersphere because of the compact constraint by the outlier pixels which were located neighboring to the target class in the spectral feature space. The overall accuracies for wheat and bare land achieved were as high as 89.25% and 83.65%, respectively. However, target class was underestimated because the validation set covers only a small fraction of the heterogeneous spectra of the target class. The different window sizes were then tested to acquire more wheat pixels for validation set. The results showed that classification accuracy increased with the increasing window size and the overall accuracies were higher than 88% at all window size scales. Moreover, WVS-SVDD showed much less sensitivity to the untrained classes than the multi-class support vector machine (SVM) method. Therefore, the developed method showed its merits using the optimal parameters, tradeoff coefficient (C) and kernel width (s), in mapping homogeneous specific land cover.

Boosting-Based EMG Patterns Classification Scheme for Robustness Enhancement

The high conventional accuracy of pattern recognition-based surface myoelectric classification in laboratory experiments does not necessarily result in high accessibility to practical protheses. An obvious reason is the effect of signals of untrained classes caused by the relatively small training dataset. In order to make the classifier robust to untrained classes, a classification scheme is developed based on boosting and random forest classifiers in this paper. Meanwhile, a threshold, the post probability of the prediction, is introduced as a balance (i.e., adjust) between the accurate classification and the rejection of the samples belonging to some untrained classes. The experiments are conducted to compare with other two schemes using linear discriminant analysis and support vector machines. Surface electromyogram signals, labeled with seven isometric movements, are collected from six healthy subjects' forearm. It is shown that the proposed scheme can reach up to about 92% accuracy in recognizing trained classes and 20% for untrained classes. Through adjusting the threshold, the accuracy of rejecting untrained classes reaches up to around 80%, with small decrease in recognizing trained classes (down to 80%). In the analysis of experiments' results, we also find that the proposed scheme has better error distribution among the classes.

Principal components null space analysis for image and video classification

We present a new classification algorithm, principal component null space analysis (PCNSA), which is designed for classification problems like object recognition where different classes have unequal and nonwhite noise covariance matrices. PCNSA first obtains a principal components subspace (PCA space) for the entire data. In this PCA space, it finds for each class "i," an Mi-dimensional subspace along which the class' intraclass variance is the smallest. We call this subspace an approximate null space (ANS) since the lowest variance is usually "much smaller" than the highest. A query is classified into class "i" if its distance from the class' mean in the class' ANS is a minimum. We derive upper bounds on classification error probability of PCNSA and use these expressions to compare classification performance of PCNSA with that of subspace linear discriminant analysis (SLDA). We propose a practical modification of PCNSA called progressive-PCNSA that also detects "new" (untrained classes). Finally, we provide an experimental comparison of PCNSA and progressive PCNSA with SLDA and PCA and also with other classification algorithms-linear SVMs, kernel PCA, kernel discriminant analysis, and kernel SLDA, for object recognition and face recognition under large pose/expression variation. We also show applications of PCNSA to two classification problems in video--an action retrieval problem and abnormal activity detection.

Supervised image classification by MLP and RBF neural networks with and without an exhaustively defined set of classes

The absence of assumptions about the dataset to be classified is one of the major attractions of neural networks for supervised image classification applications. Classification by a neural network does, however, make assumptions about the classes. One key assumption typically made is that the set of classes has been defined exhaustively. If this assumption is unsatisfied, cases of an untrained class will be present and commissioned into the set of trained classes to the detriment of classification accuracy. This was observed in land cover classifications derived with multi-layer perceptron (MLP) and radial basis function (RBF) neural networks in which the presence of an untrained class resulted in a ∼12.5% decrease in the accuracy of crop classifications derived from airborne thematic mapper data. However, since the RBF network partitions feature space locally rather than globally as with the MLP, it was possible to reduce the commission of atypical cases into the set of trained classes through the setting of post-classification thresholds on the RBF network's outputs. As a result it was possible to identify and exclude some cases of untrained classes from a classification with a RBF network which resulted in an increase in classification accuracy.

Hard and soft classifications by a neural network with a non-exhaustively defined set of classes

Many assumptions are typically made in the course of a supervised digital image classification. The focus of this paper is the commonly made assumption of an exhaustively defined set of classes. This assumption is often unsatisfied, with the imagery containing regions of classes that were not included in training the classification. The failure to satisfy the assumed condition was investigated with reference to hard and soft land cover classifications by a feedforward neural network. The accuracy of these classifications was decreased if a non-exhaustively defined set of classes was used. The exclusion of a class from the training stage resulted in a decrease in the accuracy of a hard classification of agricultural crops of up to 21.2%. Moreover, there were marked differences between the 'real' and 'apparent' accuracies of classifications of up to 15.4%. With soft classifications of urban land cover, the presence of an untrained class also degraded classification accuracy. In the soft classification output, the correlation between the actual and estimated proportional cover of a class declined from r =0.97 to r =0.76 when another class was excluded from the training stage of the classification. Possible means to reduce the negative impacts of untrained classes are considered briefly. Post-classification thresholding of the neural network's output unit activation levels may form the basis of a method to identify and remove cases of an untrained class from a hard classification. Alternatively, supporting information on the typicality of class membership may be used to identify cases representing an area containing an untrained class in both hard and soft classifications and this is illustrated with reference to a soft classification.

Estimation of sub-pixel land cover composition in the presence of untrained classes

Remotely sensed data are an attractive source of land cover data over a wide range of spatial and temporal scales. The realisation of the full potential of remote sensing as a source of land cover data is, however, restricted by numerous factors. One commonly encountered problem is the presence of mixed pixels, which cannot be appropriately accommodated in conventional image classification techniques used in thematic mapping from remotely sensed data. This problem has generally been resolved through the adoption of a soft or fuzzy classification from which the fractional coverage of classes in the image pixels may be mapped. In this type of approach, the strength of membership, a pixel displays to a class, is used as a surrogate for the fractional coverage of that class. The accuracy of the resulting land cover representation is, therefore, dependent on the relationships between class membership strength and associated class fractional coverage. Since class membership can only be measured in relation to the classes defined in the training stage of the classification, untrained classes may influence the accuracy of the class composition estimation. For example, a pixel representing an area of an untrained class can only display membership to the trained classes. The effect of an untrained class on the accuracy of sub-pixel class composition estimation will depend on how the class membership strength is calculated. Here, the effect of untrained classes on sub-pixel land cover composition estimation using algorithms that produce relative and absolute measures of class membership was assessed. The algorithms investigated were the widely used fuzzy c-means (FCM) and its possibilistic counterpart, the possibilistic c-means (PCM), algorithms which derive relative and absolute measures of class membership strength, respectively. Both algorithms were able to provide accurate estimates of sub-pixel land cover composition. When all classes had been defined in training a classification, the FCM generally provided the most accurate class composition estimates. The presence of an untrained class, however, could substantially degrade the accuracy of the sub-pixel land cover composition estimates derived from the FCM but had no effect on those from the PCM. Since untrained classes are commonly encountered it may be more appropriate to use approaches such as the PCM in addition to, or instead of, the FCM to enhance the extraction of land cover information from remotely sensed data.