Label Embedding Research Articles

Few-shot activity recognition with cross-modal memory network

Deep learning based action recognition methods require large amount of labelled training data. However, labelling large-scale video data is time consuming and tedious. In this paper, we consider a more challenging few-shot action recognition problem where the training samples are few and rare. To solve this problem, memory network has been designed to use an external memory to remember the experience learned in training and then apply it to few-shot prediction during testing. However, existing memory-based methods just update the visual information with fixed label embeddings in the memory, which cannot adapt well to novel activities during testing. To alleviate the issue, we propose a novel end-to-end cross-modal memory network for few-shot activity recognition. Specifically, the proposed memory architecture stores the dynamic visual and textual semantics for some high-level attributes related to human activities. And the learned memory can provide effective multi-modal information for new activity recognition in the testing stage. Extensive experimental results on two video datasets, including HMDB51 and UCF101, indicate that our method could achieve significant improvements over other previous methods.

A Dictionary Learning Algorithm Based on Dictionary Reconstruction and Its Application in Face Recognition

In recent years, dictionary learning has received more and more attention in the study of face recognition. However, most dictionary learning algorithms directly use the original training samples to learn the dictionary, ignoring noise existing in the training samples. For example, there are differences between different images of the same subject due to changes in illumination, expression, etc. To address the above problems, this paper proposes the dictionary relearning algorithm (DRLA) based on locality constraint and label embedding, which can effectively reduce the influence of noise on the dictionary learning algorithm. In our proposed dictionary learning algorithm, first, the initial dictionary and coding coefficient matrix are directly obtained from the training samples, and then the original training samples are reconstructed by the product of the initial dictionary and coding coefficient matrix. Finally, the dictionary learning algorithm is reapplied to obtain a new dictionary and coding coefficient matrix, and the newly obtained dictionary and coding coefficient matrix are used for subsequent image classification. The dictionary reconstruction method can partially eliminate noise in the original training samples. Therefore, the proposed algorithm can obtain more robust classification results. The experimental results demonstrate that the proposed algorithm performs better in recognition accuracy than some state-of-the-art algorithms.

Compound dictionary learning based classification method with a novel virtual sample generation Technology for Face Recognition

Face recognition has earned its high reputation for many years since its considerable advances. However, it is still faced with the challenge of the small sample size problem. With the inspiration of the axis-symmetrical property of human faces, we propose a novel virtual sample synthesis strategy to address the issue of limited training samples. It is noteworthy that the novel algorithm can produce symmetry-based virtual face images, in which the pixels in symmetric parts of the face image were exchanged. And it is mathematically very tractable and quite easy to implement. In addition, considering the fact that dictionary learning (DL) based classification methods have excellent learning ability, we incorporate the virtual samples to learn virtual dictionary so as to enhance the accuracy of face recognition. Differing from conventional learning algorithms, the proposed method provides new insights into two crucial parts. Firstly, it proposes an originally creative idea and algorithm to automatically generate symmetry-based virtual samples and obtain virtual dictionary. Secondly, the original dictionary and virtual dictionary are integrated to construct the compound dictionary learning based classification in the way of adaptive weighted fusion. In this paper, we take the axis-symmetrical nature of faces into consideration and design a framework to generate compound dictionary, where more satisfactory classification accuracy can be achieved than the original dictionary learning methods, referred as, the locality-constrained and label embedding dictionary learning (LCLE-DL). Moreover, the experimental results demonstrate the superior performance of the proposed method in comparison with state-of-the-art dictionary learning methods.

Incorporating Label Embedding and Feature Augmentation for Multi-Dimensional Classification

Feature augmentation, which manipulates the feature space by integrating the label information, is one of the most popular strategies for solving Multi-Dimensional Classification (MDC) problems. However, the vanilla feature augmentation approaches fail to consider the intra-class exclusiveness, and may achieve degenerated performance. To fill this gap, a novel neural network based model is proposed which seamlessly integrates the Label Embedding and Feature Augmentation (LEFA) techniques to learn label correlations. Specifically, based on attentional factorization machine, a cross correlation aware network is introduced to learn a low-dimensional label representation that simultaneously depicts the inter-class correlations and the intra-class exclusiveness. Then the learned latent label vector can be used to augment the original feature space. Extensive experiments on seven real-world datasets demonstrate the superiority of LEFA over state-of-the-art MDC approaches.

Cross-Modality Attention with Semantic Graph Embedding for Multi-Label Classification

Multi-label image and video classification are fundamental yet challenging tasks in computer vision. The main challenges lie in capturing spatial or temporal dependencies between labels and discovering the locations of discriminative features for each class. In order to overcome these challenges, we propose to use cross-modality attention with semantic graph embedding for multi-label classification. Based on the constructed label graph, we propose an adjacency-based similarity graph embedding method to learn semantic label embeddings, which explicitly exploit label relationships. Then our novel cross-modality attention maps are generated with the guidance of learned label embeddings. Experiments on two multi-label image classification datasets (MS-COCO and NUS-WIDE) show our method outperforms other existing state-of-the-arts. In addition, we validate our method on a large multi-label video classification dataset (YouTube-8M Segments) and the evaluation results demonstrate the generalization capability of our method.

Hierarchical Contextualized Representation for Named Entity Recognition

Named entity recognition (NER) models are typically based on the architecture of Bi-directional LSTM (BiLSTM). The constraints of sequential nature and the modeling of single input prevent the full utilization of global information from larger scope, not only in the entire sentence, but also in the entire document (dataset). In this paper, we address these two deficiencies and propose a model augmented with hierarchical contextualized representation: sentence-level representation and document-level representation. In sentence-level, we take different contributions of words in a single sentence into consideration to enhance the sentence representation learned from an independent BiLSTM via label embedding attention mechanism. In document-level, the key-value memory network is adopted to record the document-aware information for each unique word which is sensitive to similarity of context information. Our two-level hierarchical contextualized representations are fused with each input token embedding and corresponding hidden state of BiLSTM, respectively. The experimental results on three benchmark NER datasets (CoNLL-2003 and Ontonotes 5.0 English datasets, CoNLL-2002 Spanish dataset) show that we establish new state-of-the-art results.

Multi-Label Classification with Label Graph Superimposing

Images or videos always contain multiple objects or actions. Multi-label recognition has been witnessed to achieve pretty performance attribute to the rapid development of deep learning technologies. Recently, graph convolution network (GCN) is leveraged to boost the performance of multi-label recognition. However, what is the best way for label correlation modeling and how feature learning can be improved with label system awareness are still unclear. In this paper, we propose a label graph superimposing framework to improve the conventional GCN+CNN framework developed for multi-label recognition in the following two aspects. Firstly, we model the label correlations by superimposing label graph built from statistical co-occurrence information into the graph constructed from knowledge priors of labels, and then multi-layer graph convolutions are applied on the final superimposed graph for label embedding abstraction. Secondly, we propose to leverage embedding of the whole label system for better representation learning. In detail, lateral connections between GCN and CNN are added at shallow, middle and deep layers to inject information of label system into backbone CNN for label-awareness in the feature learning process. Extensive experiments are carried out on MS-COCO and Charades datasets, showing that our proposed solution can greatly improve the recognition performance and achieves new state-of-the-art recognition performance.

Prior Knowledge Driven Label Embedding for Slot Filling in Natural Language Understanding

Traditional slot filling in natural language understanding (NLU) predicts a one-hot vector for each word. This form of label representation lacks semantic correlation modelling, which leads to severe data sparsity problem, especially when adapting an NLU model to a new domain. To address this issue, a novel label embedding based slot filling framework is proposed in this paper. Here, distributed label embedding is constructed for each slot using prior knowledge. Three encoding methods are investigated to incorporate different kinds of prior knowledge about slots: atomic concepts, slot descriptions, and slot exemplars. The proposed label embeddings tend to share text patterns and reuses data with different slot labels. This makes it useful for adaptive NLU with limited data. Also, since label embedding is independent of NLU model, it is compatible with almost all deep learning based slot filling models. The proposed approaches are evaluated on three datasets. Experiments on single domain and domain adaptation tasks show that label embedding achieves significant performance improvement over traditional one-hot label representation as well as advanced zero-shot approaches.

An innovative multi-label learning based algorithm for city data computing

Investigating correlation between example features and example labels is essential to the solving of classification problems. However, identification and calculation of the correlation between features and labels can be rather difficult in case involving high-dimensional multi-label data. Both feature embedding and label embedding have been developed to tackle this challenge, and a shared subspace for both labels and features is usually learned by applying existing embedding methods to simultaneously reduce the dimension of features and labels. By contrast, this paper suggests learning separate subspaces for features and labels by maximizing the independence between the components in each subspace, as well as maximizing the correlation between these two subspaces. The learned independent label components indicate the fundamental combinations of labels in multi-label datasets, which thus helps to reveal the correlation between labels. Furthermore, the learned independent feature components lead to a compact representation of example features. The connections between the proposed algorithm and existing embedding methods are discussed in detail. Experimental results on real-world multi-label datasets demonstrate that it is necessary for us to explore independent components from multi-label data, and further prove the effectiveness of the proposed algorithm.

A Fusion Model-Based Label Embedding and Self-Interaction Attention for Text Classification

Text classification is a pivotal task in NLP (Natural Language Processing), which has received widespread attention recently. Most of the existing methods leverage the power of deep learning to improve the performance of models. However, these models ignore the interaction information between all the sentences in a text when generating the current text representation, which results in a partial semantics loss. Labels play a central role in text classification. And the attention learned from text-label in the joint space of labels and words is not leveraged, leaving enough room for further improvement. In this paper, we propose a text classification method based on Self-Interaction attention mechanism and label embedding. Firstly, our method introduce BERT (Bidirectional Encoder Representation from Transformers) to extract text features. Then Self-Interaction attention mechanism is employed to obtain text representations containing more comprehensive semantics. Moreover, we focus on the embedding of labels and words in the joint space to achieve the dual-label embedding, which further leverages the attention learned from text-label. Finally, the texts are classified by the classifier according to the weighted labels representations. The experimental results show that our method outperforms other state-of-the-art methods in terms of classification accuracy.

Label Embedding with Partial Heterogeneous Contexts

Label embedding plays an important role in many real-world applications. To enhance the label relatedness captured by the embeddings, multiple contexts can be adopted. However, these contexts are heterogeneous and often partially observed in practical tasks, imposing significant challenges to capture the overall relatedness among labels. In this paper, we propose a general Partial Heterogeneous Context Label Embedding (PHCLE) framework to address these challenges. Categorizing heterogeneous contexts into two groups, relational context and descriptive context, we design tailor-made matrix factorization formula to effectively exploit the label relatedness in each context. With a shared embedding principle across heterogeneous contexts, the label relatedness is selectively aligned in a shared space. Due to our elegant formulation, PHCLE overcomes the partial context problem and can nicely incorporate more contexts, which both cannot be tackled with existing multi-context label embedding methods. An effective alternative optimization algorithm is further derived to solve the sparse matrix factorization problem. Experimental results demonstrate that the label embeddings obtained with PHCLE achieve superb performance in image classification task and exhibit good interpretability in the downstream label similarity analysis and image understanding task.

Distributional Semantics Meets Multi-Label Learning

We present a label embedding based approach to large-scale multi-label learning, drawing inspiration from ideas rooted in distributional semantics, specifically the Skip Gram Negative Sampling (SGNS) approach, widely used to learn word embeddings. Besides leading to a highly scalable model for multi-label learning, our approach highlights interesting connections between label embedding methods commonly used for multi-label learning and paragraph embedding methods commonly used for learning representations of text data. The framework easily extends to incorporating auxiliary information such as label-label correlations; this is crucial especially when many training instances are only partially annotated. To facilitate end-to-end learning, we develop a joint learning algorithm that can learn the embeddings as well as a regression model that predicts these embeddings for the new input to be annotated, via efficient gradient based methods. We demonstrate the effectiveness of our approach through an extensive set of experiments on a variety of benchmark datasets, and show that the proposed models perform favorably as compared to state-of-the-art methods for large-scale multi-label learning.

Two-Stage Label Embedding via Neural Factorization Machine for Multi-Label Classification

Label embedding has been widely used as a method to exploit label dependency with dimension reduction in multilabel classification tasks. However, existing embedding methods intend to extract label correlations directly, and thus they might be easily trapped by complex label hierarchies. To tackle this issue, we propose a novel Two-Stage Label Embedding (TSLE) paradigm that involves Neural Factorization Machine (NFM) to jointly project features and labels into a latent space. In encoding phase, we introduce a Twin Encoding Network (TEN) that digs out pairwise feature and label interactions in the first stage and then efficiently learn higherorder correlations with deep neural networks (DNNs) in the second stage. After the codewords are obtained, a set of hidden layers is applied to recover the output labels in decoding phase. Moreover, we develop a novel learning model by leveraging a max margin encoding loss and a label-correlation aware decoding loss, and we adopt the mini-batch Adam to optimize our learning model. Lastly, we also provide a kernel insight to better understand our proposed TSLE. Extensive experiments on various real-world datasets demonstrate that our proposed model significantly outperforms other state-ofthe-art approaches.

Sparse graphs with smoothness constraints: Application to dimensionality reduction and semi-supervised classification

Sparse representation is a useful tool in machine learning and pattern recognition area. Sparse graphs (graphs constructed using sparse representation of data) proved to be very informative graphs for many learning tasks such as label propagation, embedding, and clustering. It has been shown that constructing an informative graph is one of the most important steps since it significantly affects the final performance of the post graph-based learning algorithm. In this paper, we introduce a new sparse graph construction method that integrates manifold constraints on the unknown sparse codes as a graph regularizer. These constraints seem to be a natural regularizer that was discarded in existing state-of-the art graph construction methods. This regularizer imposes constraints on the graph coefficients in the same way a locality preserving constraint imposes on data projection in non-linear manifold learning. The proposed method is termed Sparse Graph with Laplacian Smoothness (SGLS). We also propose a kernelized version of the SGLS method. A series of experimental results on several public image datasets show that the proposed methods can out-perform many state-of-the-art methods for the tasks of label propagation, nonlinear and linear embedding.

Data scarcity, robustness and extreme multi-label classification

The goal in extreme multi-label classification (XMC) is to learn a classifier which can assign a small subset of relevant labels to an instance from an extremely large set of target labels. The distribution of training instances among labels in XMC exhibits a long tail, implying that a large fraction of labels have a very small number of positive training instances. Detecting tail-labels, which represent diversity of the label space and account for a large fraction (upto 80%) of all the labels, has been a significant research challenge in XMC. In this work, we pose the tail-label detection task in XMC as robust learning in the presence of worst-case perturbations. This viewpoint is motivated by a key observation that there is a significant change in the distribution of the feature composition of instances of these labels from the training set to test set. For shallow classifiers, our robustness perspective to XMC naturally motivates the well-known ell _1-regularized classification. Contrary to the popular belief that Hamming loss is unsuitable for tail-labels detection in XMC, we show that minimizing (convex upper bound on) Hamming loss with appropriate regularization surpasses many state-of-the-art methods. Furthermore, we also highlight the sub-optimality of the co-ordinate descent based solver in the LibLinear package, which, given its ubiquity, is interesting in its own right. We also investigate the spectral properties of label graphs for providing novel insights towards understanding the conditions governing the performance of Hamming loss based one-vs-rest scheme vis-à-vis label embedding methods.

Group preserving label embedding for multi-label classification

Multi-label learning is concerned with the classification of data with multiple class labels. This is in contrast to the traditional classification problem where every data instance has a single label. Due to the exponential size of output space, exploiting intrinsic information in feature and label spaces has been the major thrust of research in recent years and use of parametrization and embedding have been the prime focus. Researchers have studied several aspects of embedding which include label embedding, input embedding, dimensionality reduction and feature selection. These approaches differ from one another in their capability to capture other intrinsic properties such as label correlation, local invariance etc. We assume here that the input data form groups and as a result, the label matrix exhibits a sparsity pattern and hence the labels corresponding to objects in the same group have similar sparsity. In this paper, we study the embedding of labels together with the group information with an objective to build an efficient multi-label classifier. We assume the existence of a low-dimensional space onto which the feature vectors and label vectors can be embedded. In order to achieve this, we address three sub-problems namely; (1) Identification of groups of labels; (2) Embedding of label vectors to a low rank-space so that the sparsity characteristic of individual groups remains invariant; and (3) Determining a linear mapping that embeds the feature vectors onto the same set of points, as in stage 2, in the low-dimensional space. We compare our method with seven well-known algorithms on twelve benchmark data sets. Our experimental analysis manifests the superiority of our proposed method over state-of-art algorithms for multi-label learning.

Reconstruction Regularized Deep Metric Learning for Multi-Label Image Classification.

In this paper, we present a novel deep metric learning method to tackle the multi-label image classification problem. In order to better learn the correlations among images features, as well as labels, we attempt to explore a latent space, where images and labels are embedded via two unique deep neural networks, respectively. To capture the relationships between image features and labels, we aim to learn a two-way deep distance metric over the embedding space from two different views, i.e., the distance between one image and its labels is not only smaller than those distances between the image and its labels' nearest neighbors but also smaller than the distances between the labels and other images corresponding to the labels' nearest neighbors. Moreover, a reconstruction module for recovering correct labels is incorporated into the whole framework as a regularization term, such that the label embedding space is more representative. Our model can be trained in an end-to-end manner. Experimental results on publicly available image data sets corroborate the efficacy of our method compared with the state of the arts.

Motion Based Inference of Social Circles via Self-Attention and Contextualized Embedding

Extracting knowledge from human mobility data is an important task for many downstream applications such as point-of-interest recommendation, motion trace identification, and personalized trip planning. A specific problem that has recently spurred research interest is the so-called Social Circle Inference from Mobility data (SCIM), aiming at inferring relationships among users based on mobility data and without any explicit structured network information. The existing methods either require partial social ties or fail to model the implicit correlations between user links, thereby suffering from critical inference bias. We present a novel SCIM framework, called SCIM via self-Attention and Contextualized-embedding (SCIMAC) - a methodology capturing multiple aspects of users' check-in behavior and complex motion patterns of different users. Instead of directly applying the recurrent model on training user trajectories, the proposed method introduces a new module for context-aware check-in representation learning by adaptively incorporating the internal states of the recurrent layers, which is more effective than the context-independent check-in embedding used in existing social circle inference approaches. To model the underlying correlations between labels, SCIMAC leverages a more sophisticated label embedding technique to adjust the penalties for correlated users, enabling a better understanding of the user's hierarchy in the label space, and alleviating the inference bias. We empirically demonstrate that our SCIMAC model significantly outperforms several state-of-the-art baselines on real-world datasets.

Polite Dialogue Generation Without Parallel Data

Stylistic dialogue response generation, with valuable applications in personality-based conversational agents, is a challenging task because the response needs to be fluent, contextually-relevant, as well as paralinguistically accurate. Moreover, parallel datasets for regular-to-stylistic pairs are usually unavailable. We present three weakly-supervised models that can generate diverse, polite (or rude) dialogue responses without parallel data. Our late fusion model (Fusion) merges the decoder of an encoder-attention-decoder dialogue model with a language model trained on stand-alone polite utterances. Our label-finetuning (LFT) model prepends to each source sequence a politeness-score scaled label (predicted by our state-of-the-art politeness classifier) during training, and at test time is able to generate polite, neutral, and rude responses by simply scaling the label embedding by the corresponding score. Our reinforcement learning model (Polite-RL) encourages politeness generation by assigning rewards proportional to the politeness classifier score of the sampled response. We also present two retrievalbased, polite dialogue model baselines. Human evaluation validates that while the Fusion and the retrieval-based models achieve politeness with poorer context-relevance, the LFT and Polite-RL models can produce significantly more polite responses without sacrificing dialogue quality.

Synthetic training samples for enhanced locality‐constrained dictionary learning

Dictionary learning serves as a considerable role in image processing and pattern recognition. However, when applied to face classification, it may suffer from the issue of the limited quantity of training samples. Therefore, it becomes a challenge to obtain a robust and discriminative dictionary. Recently, locality-constrained and label embedding dictionary learning (LCLE-DL) takes the locality and label information of atoms into account to achieve an effective performance in image classification. In this study, the authors exploit a new approach which uses synthetic training samples to enhance this dictionary learning algorithm, so they name it STS-DL. Firstly, they strengthen the diversities of training samples by producing virtual samples. Secondly, the LCLE-DL algorithm is used to calculate two deviations on the basis of the original training samples and the authors’ newly synthetic samples, respectively. Finally, they integrate them together to perform the classification task, which produces a more promising performance for image recognition. Abundant experiments have been conducted on several benchmark databases, the experimental results illustrate that the proposed STS-DL shows a higher accuracy than the LCLE-DL method, as well as some state-of-the-art dictionary learning and sparse representation algorithms in image classification.