Latent Label Research Articles

Reinforcement Learning for Distantly Supervised Relation Extraction

Relation extraction is a task of identifying semantic relations between entity pairs from plain text, which can benefit a lot of AI applications such as knowledge base construction and answer questioning. The distant supervision strategy is introduced to automatically create large-scale training data, which inevitably suffers from noisy label problem. Recent works handle the sentence-level denoising by reinforcement learning, which regards the labels from distant supervision as the ground-truth. However, few works focus on the label-level denoising that corrects noisy labels directly. In this paper, we propose a reinforcement learning-based label denoising method for distantly supervised relation extraction. The model consists of two modules: extraction network (ENet) and policy network (PNet). The core of our label denoising is designing a policy in the PNet to obtain latent labels, where we can select the actions of using the distantly supervised labels or the predicted labels from the ENet. More concretely, the task can be modeled as an iterative process. First, the ENet predicts the relation probability, through which the model generates state representation. Second, the PNet learns the latent labels with taken actions and uses them to update the ENet. Then the optimized ENet gives the rewards to the PNet. The joint learning of two modules can obtain a reliable latent label and effectively improve the classification performance. The experimental results show that reinforcement learning is effective for noisy label correction and the proposed method can outperform the state-of-the-art relation extraction systems.

Deep Learning from Noisy Image Labels with Quality Embedding.

There is an emerging trend to leverage noisy image datasets in many visual recognition tasks. However, the label noise among datasets severely degenerates the performance of deep learning approaches. Recently, one mainstream is to introduce the latent label to handle label noise, which has shown promising improvement in the network designs. Nevertheless, the mismatch between latent labels and noisy labels still affects the predictions in such methods. To address this issue, we propose a probabilistic model, which explicitly introduces an extra variable to represent the trustworthiness of noisy labels, termed as the quality variable. Our key idea is to identify the mismatch between the latent and noisy labels by embedding the quality variables into different subspaces, which effectively minimizes the influence of label noise. At the same time, reliable labels are still able to be applied for training. To instantiate the model, we further propose a Contrastive-Additive Noise network (CAN), which consists of two important layers: (1) the contrastive layer that estimates the quality variable in the embedding space to reduce the influence of noisy labels; and (2) the additive layer that aggregates the prior prediction and noisy labels as the posterior to train the classifier. Moreover, to tackle the challenges in optimization, we deduce an SGD algorithm with the reparameterization tricks, which makes our method scalable to big data.We validate the proposed method on a range of noisy image datasets. Comprehensive results have demonstrated that CAN outperforms the state-of-the-art deep learning approaches.

Dynamic Hyper-Graph Inference Framework for Computer-Assisted Diagnosis of Neurodegenerative Diseases.

Hyper-graph techniques have been widely investigated in computer vision and medical imaging applications, showing superior performance for modeling complex subject-wise relationships and sufficient flexibility to deal with missing data from multi-modal neuroimaging data. Existing hyper-graph methods, however, are inadequate for two reasons. First, representations are generated only from the observed imaging data, a process that is completely independent of the subsequent data label inference/ classification step. Thus, hyper-graph results constructed in this way may not be consistent with phenotype data such as clinical labels or scores. More critically, it might generate sub-optimal predictions in relation to clinical labels/scores. Second, current hyper-graph inference methods rely on two sequential steps: 1) building the hyper-graph for each individual modality and then predicted latent labels for new subjects upon each constructed hyper-graph and 2) a voting procedure to incorporate inference results across different hyper-graphs. This approach, however, is limited by failing to consider the complex and complementary relationships of multi-modal imaging data with respect to hyper-graph inference procedure. To address these two issues, we propose a novel dynamic hyper-graph inference method supported by a semi-supervised framework. Our method iteratively estimates and adjusts the hyper-graph structures from multi-modal imaging data until consistency between the learned hyper-graph and the observed clinical labels and scores is achieved. This hyper-graph inference framework also eases the integration process of classification (identifying individuals having neurodegenerative disease) and regression (predicting the clinical scores) within the same framework. The experimental results on identifying mild cognition impairment (MCI) subjects and the fine grained recognition of MCI progression stages show improved performance using our proposed hyper-graph inference method compared with conventional methods.

Multi-Label Learning with Global and Local Label Correlation

It is well-known that exploiting label correlations is important to multi-label learning. Existing approaches either assume that the label correlations are global and shared by all instances; or that the label correlations are local and shared only by a data subset. In fact, in the real-world applications, both cases may occur that some label correlations are globally applicable and some are shared only in a local group of instances. Moreover, it is also a usual case that only partial labels are observed, which makes the exploitation of the label correlations much more difficult. That is, it is hard to estimate the label correlations when many labels are absent. In this paper, we propose a new multi-label approach GLOCAL dealing with both the full-label and the missing-label cases, exploiting global and local label correlations simultaneously, through learning a latent label representation and optimizing label manifolds. The extensive experimental studies validate the effectiveness of our approach on both full-label and missing-label data.

Supervised Dirichlet Process Mixtures of Principal Component Analysis

Abstract We introduce probabilistic principal component analysis (PPCA) into Dirichlet Process Mixtures of Generalized Linear Models (DPGLM) and propose a new model called Supervised Dirichlet Process Mixtures of Principal Component Analysis (SDPM-PCA). In SDPM-PCA, we assume covariates and response variable are generated separately through the latent variable of PPCA, and nonparametrically modeled using the Dirichlet Process Mixture. By jointly learning the latent variable, cluster label and response variable, SDPM-PCA performs locally dimensionality reduction within each mixture component, and learns a supervised model based on the latent variable. In this way, SDPM-PCA improves the performance of both dimensionality reduction and prediction on high-dimensional data with all advantages of DPGLM. We also develop an inference algorithm for SDPM-PCA based on variational inference, which provides faster training speed and deterministic approximation compared with sampling algorithms based on MCMC method. Finally, we instantiate SDPM-PCA in regression problem with a Bayesian linear regression model. We test it on several real-world datasets and compare the prediction performance with DPGLM and other regular regression model. Experiment results show that by setting properly latent dimension number, SDPM-PCA would provide better prediction performance on high-dimensional regression problem and avoid the curse of dimensionality problem in DPGLM.

Label distribution learning and label enhancement

This paper introduces the concepts and algorithms for label distribution learning (LDL) and label enhancement. LDL is a general machine learning paradigm with traditional single-label learning and multi-label learning as its special cases. A label distribution covers a certain number of labels, representing the degree to which each label describes the instance. Thus, LDL has been successfully applied to many real-world problems. Unfortunately, many existing datasets only have simple logical labels rather than label distributions. One way to solve the problem is to transform the logical labels into label distributions by mining the latent label importance from the training examples. Such a process of transforming logical labels into label distributions is defined as label enhancement. This paper provides formal definitions of label distribution learning and label enhancement. Subsequently, six representative LDL algorithms and four typical LE algorithms are briefly introduced and comparatively analyzed.

Discriminative Transformation Learning for Fuzzy Sparse Subspace Clustering

This paper develops a novel iterative framework for subspace clustering (SC) in a learned discriminative feature domain. This framework consists of two modules of fuzzy sparse SC and discriminative transformation learning. In the first module, fuzzy latent labels containing discriminative information and latent representations capturing the subspace structure will be simultaneously evaluated in a feature domain. Then the linear transforming operator with respect to the feature domain will be successively updated in the second module with the advantages of more discrimination, subspace structure preservation, and robustness to outliers. These two modules will be alternatively carried out and both theoretical analysis and empirical evaluations will demonstrate its effectiveness and superiorities. In particular, experimental results on three benchmark databases for SC clearly illustrate that the proposed framework can achieve significant improvements than other state-of-the-art approaches in terms of clustering accuracy.

Structured Latent Label Consistent Dictionary Learning for Salient Machine Faults Representation-Based Robust Classification

This paper investigates the salient machine faults representation-based classification issue by dictionary learning. A novel structured latent label consistent dictionary learning (LLC-DL) model is proposed for joint discriminative salient representation and classification. Our LLC-DL deals with the tasks by solving one objective function that aims to minimize the structured reconstruction error, structured discriminative sparse-code error and classification error simultaneously. Also, LLC-DL decomposes given signals into a sparse reconstruction part over structured latent weighted discriminative dictionary, a salient feature extraction part and an error part fitting noise. Specifically, the dictionary is learnt atom by atom, where each dictionary atom is learnt with a latent vector that reduces the disturbance between interclass atoms. The structured coding coefficients are calculated via minimizing the reconstruction error and discriminative sparse code error simultaneously. The salient representations are learnt by embedding signals onto a projection and a robust linear classifier is then trained over the learned salient features directly so that features can be ensured to be optimal for classification, where robust l 2 , 1-norm imposed on the classifier can make the prediction results more accurate. By including a salient feature extraction term, the classification approach of LLC-DL is very efficient, since there is no need to involve an extra time-consuming sparse reconstruction process with the well-trained dictionary for each test signal. Extensive simulations versify the effectiveness of our algorithm.

Data-Dependent Label Distribution Learning for Age Estimation.

As an important and challenging problem in computer vision, face age estimation is typically cast as a classification or regression problem over a set of face samples with respect to several ordinal age labels, which have intrinsically cross-age correlations across adjacent age dimensions. As a result, such correlations usually lead to the age label ambiguities of the face samples. Namely, each face sample is associated with a latent label distribution that encodes the cross-age correlation information on label ambiguities. Motivated by this observation, we propose a totally data-driven label distribution learning approach to adaptively learn the latent label distributions. The proposed approach is capable of effectively discovering the intrinsic age distribution patterns for cross-age correlation analysis on the basis of the local context structures of face samples. Without any prior assumptions on the forms of label distribution learning, our approach is able to flexibly model the sample-specific context aware label distribution properties by solving a multi-task problem, which jointly optimizes the tasks of age-label distribution learning and age prediction for individuals. Experimental results demonstrate the effectiveness of our approach.

Progressive multi-atlas label fusion by dictionary evolution.

Accurate segmentation of anatomical structures in medical images is important in recent imaging based studies. In the past years, multi-atlas patch-based label fusion methods have achieved a great success in medical image segmentation. In these methods, the appearance of each input image patch is first represented by an atlas patch dictionary (in the image domain), and then the latent label of the input image patch is predicted by applying the estimated representation coefficients to the corresponding anatomical labels of the atlas patches in the atlas label dictionary (in the label domain). However, due to the generally large gap between the patch appearance in the image domain and the patch structure in the label domain, the estimated (patch) representation coefficients from the image domain may not be optimal for the final label fusion, thus reducing the labeling accuracy. To address this issue, we propose a novel label fusion framework to seek for the suitable label fusion weights by progressively constructing a dynamic dictionary in a layer-by-layer manner, where the intermediate dictionaries act as a sequence of guidance to steer the transition of (patch) representation coefficients from the image domain to the label domain. Our proposed multi-layer label fusion framework is flexible enough to be applied to the existing labeling methods for improving their label fusion performance, i.e., by extending their single-layer static dictionary to the multi-layer dynamic dictionary. The experimental results show that our proposed progressive label fusion method achieves more accurate hippocampal segmentation results for the ADNI dataset, compared to the counterpart methods using only the single-layer static dictionary.

Classification and Reconstruction of High-Dimensional Signals From Low-Dimensional Features in the Presence of Side Information

This paper offers a characterization of fundamental limits on the classification and reconstruction of high-dimensional signals from low-dimensional features, in the presence of side information. We consider a scenario where a decoder has access both to linear features of the signal of interest and to linear features of the side information signal; while the side information may be in a compressed form, the objective is recovery or classification of the primary signal, not the side information. The signal of interest and the side information are each assumed to have (distinct) latent discrete labels; conditioned on these two labels, the signal of interest and side information are drawn from a multivariate Gaussian distribution. With joint probabilities on the latent labels, the overall signal-(side information) representation is defined by a Gaussian mixture model. We then provide sharp sufficient and/or necessary conditions for these quantities to approach zero when the covariance matrices of the Gaussians are nearly low-rank. These conditions, which are reminiscent of the well-known Slepian-Wolf and Wyner-Ziv conditions, are a function of the number of linear features extracted from the signal of interest, the number of linear features extracted from the side information signal, and the geometry of these signals and their interplay. Moreover, on assuming that the signal of interest and the side information obey such an approximately low-rank model, we derive expansions of the reconstruction error as a function of the deviation from an exactly low-rank model; such expansions also allow identification of operational regimes where the impact of side information on signal reconstruction is most relevant. Our framework, which offers a principled mechanism to integrate side information in high-dimensional data problems, is also tested in the context of imaging applications.

A transversal approach for patch-based label fusion via matrix completion.

Recently, multi-atlas patch-based label fusion has received an increasing interest in the medical image segmentation field. After warping the anatomical labels from the atlas images to the target image by registration, label fusion is the key step to determine the latent label for each target image point. Two popular types of patch-based label fusion approaches are (1) reconstruction-based approaches that compute the target labels as a weighted average of atlas labels, where the weights are derived by reconstructing the target image patch using the atlas image patches; and (2) classification-based approaches that determine the target label as a mapping of the target image patch, where the mapping function is often learned using the atlas image patches and their corresponding labels. Both approaches have their advantages and limitations. In this paper, we propose a novel patch-based label fusion method to combine the above two types of approaches via matrix completion (and hence, we call it transversal). As we will show, our method overcomes the individual limitations of both reconstruction-based and classification-based approaches. Since the labeling confidences may vary across the target image points, we further propose a sequential labeling framework that first labels the highly confident points and then gradually labels more challenging points in an iterative manner, guided by the label information determined in the previous iterations. We demonstrate the performance of our novel label fusion method in segmenting the hippocampus in the ADNI dataset, subcortical and limbic structures in the LONI dataset, and mid-brain structures in the SATA dataset. We achieve more accurate segmentation results than both reconstruction-based and classification-based approaches. Our label fusion method is also ranked 1st in the online SATA Multi-Atlas Segmentation Challenge.

Latent subclass learning-based unsupervised ensemble feature extraction method for hyperspectral image classification

A novel unsupervised ensemble feature learning method for hyperspectral image classification is proposed in this study. Firstly, we randomly sample multiple discriminative subsets from a hyperspectral image with the novel spatially constrained similarity measurement. Each subset consists of a small amount of representative pixels. Each pixel in the subset was assigned with a latent-subclass/pseudo label. Multiple multinomial logistic regression classifiers are then adopted to build relations between pixels and their latent subclass labels, where each classifier is trained with one subset. Finally, the predicted results of different classifiers for a given pixel are assembled as its ensemble feature. More discriminative features are extracted by the proposed method compared with features extracted by traditional unsupervised methods such as principal component analysis and non-negative matrix factorization. Experimental results on hyperspectral image classification demonstrate the effectiveness of the proposed method.

The geometry of kernelized spectral clustering

Clustering of data sets is a standard problem in many areas of science and engineering. The method of spectral clustering is based on embedding the data set using a kernel function, and using the top eigenvectors of the normalized Laplacian to recover the connected components. We study the performance of spectral clustering in recovering the latent labels of i.i.d. samples from a finite mixture of nonparametric distributions. The difficulty of this label recovery problem depends on the overlap between mixture components and how easily a mixture component is divided into two nonoverlapping components. When the overlap is small compared to the indivisibility of the mixture components, the principal eigenspace of the population-level normalized Laplacian operator is approximately spanned by the square-root kernelized component densities. In the finite sample setting, and under the same assumption, embedded samples from different components are approximately orthogonal with high probability when the sample size is large. As a corollary we control the fraction of samples mislabeled by spectral clustering under finite mixtures with nonparametric components.

Progressive Label Fusion Framework for Multi-atlas Segmentation by Dictionary Evolution.

Accurate segmentation of anatomical structures in medical images is very important in neuroscience studies. Recently, multi-atlas patch-based label fusion methods have achieved many successes, which generally represent each target patch from an atlas patch dictionary in the image domain and then predict the latent label by directly applying the estimated representation coefficients in the label domain. However, due to the large gap between these two domains, the estimated representation coefficients in the image domain may not stay optimal for the label fusion. To overcome this dilemma, we propose a novel label fusion framework to make the weighting coefficients eventually to be optimal for the label fusion by progressively constructing a dynamic dictionary in a layer-by-layer manner, where a sequence of intermediate patch dictionaries gradually encode the transition from the patch representation coefficients in image domain to the optimal weights for label fusion. Our proposed framework is general to augment the label fusion performance of the current state-of-the-art methods. In our experiments, we apply our proposed method to hippocampus segmentation on ADNI dataset and achieve more accurate labeling results, compared to the counterpart methods with single-layer dictionary.

Uncovering Hierarchical and Overlapping Communities with a Local-First Approach

Community discovery in complex networks is the task of organizing a network’s structure by grouping together nodes related to each other. Traditional approaches are based on the assumption that there is a global-level organization in the network. However, in many scenarios, each node is the bearer of complex information and cannot be classified in disjoint clusters. The top-down global view of the partition approach is not designed for this. Here, we represent this complex information as multiple latent labels, and we postulate that edges in the networks are created among nodes carrying similar labels. The latent labels are the communities a node belongs to and we discover them with a simple local-first approach to community discovery. This is achieved by democratically letting each node vote for the communities it sees surrounding it in its limited view of the global system, its ego neighborhood, using a label propagation algorithm, assuming that each node is aware of the label it shares with each of its connections. The local communities are merged hierarchically, unveiling the modular organization of the network at the global level and identifying overlapping groups and groups of groups. We tested this intuition against the state-of-the-art overlapping community discovery and found that our new method advances in the chosen scenarios in the quality of the obtained communities. We perform a test on benchmark and on real-world networks, evaluating the quality of the community coverage by using the extracted communities to predict the metadata attached to the nodes, which we consider external information about the latent labels. We also provide an explanation about why real-world networks contain overlapping communities and how our logic is able to capture them. Finally, we show how our method is deterministic, is incremental, and has a limited time complexity, so that it can be used on real-world scale networks.

Hyperspectral Image Unmixing Using a Multiresolution Sticky HDP

This paper is concerned with joint Bayesian endmember extraction and linear unmixing of hyperspectral images using a spatial prior on the abundance vectors. We propose a generative model for hyperspectral images in which the abundances are sampled from a Dirichlet distribution (DD) mixture model, whose parameters depend on a latent label process. The label process is then used to enforces a spatial prior which encourages adjacent pixels to have the same label. A Gibbs sampling framework is used to generate samples from the posterior distributions of the abundances and the parameters of the DD mixture model. The spatial prior that is used is a tree-structured sticky hierarchical Dirichlet process (SHDP) and, when used to determine the posterior endmember and abundance distributions, results in a new unmixing algorithm called spatially constrained unmixing (SCU). The directed Markov model facilitates the use of scale-recursive estimation algorithms, and is therefore more computationally efficient as compared to standard Markov random field (MRF) models. Furthermore, the proposed SCU algorithm estimates the number of regions in the image in an unsupervised fashion. The effectiveness of the proposed SCU algorithm is illustrated using synthetic and real data.

Semantic Analysis and Organization of Spoken Documents Based on Parameters Derived From Latent Topics

Spoken documents are audio signals and are thus not easily displayed on-screen and not easily scanned and browsed by the user. It is therefore highly desirable to automatically construct summaries, titles, latent topic trees and key term-based topic labels for these spoken documents to aid the user in browsing. We refer to this as semantic analysis and organization. Also, as network content is both copious and dynamic, with topics and domains changing everyday, the approaches here must be primarily unsupervised. We propose a framework for unsupervised semantic analysis and organization of spoken documents and for this purpose propose two measures derived from latent topic analysis: latent topic significance and latent topic entropy. We show that these can be integrated into an application system, with which the user can more easily navigate archives of spoken documents. Probabilistic latent semantic analysis is used as a typical example approach for unsupervised topic analysis in most experiments, although latent Dirichlet allocation is also used in some experiments to show that the proposed measures are equally applicable for different analysis approaches. All of the experiments were performed on Mandarin Chinese broadcast news.

Bayesian inference for an adaptive Ordered Probit model: An application to Brain Computer Interfacing

This paper proposes an algorithm for adaptive, sequential classification in systems with unknown labeling errors, focusing on the biomedical application of Brain Computer Interfacing (BCI). The method is shown to be robust in the presence of label and sensor noise. We focus on the inference and prediction of target labels under a nonlinear and non-Gaussian model. In order to handle missing or erroneous labeling, we model observed labels as a noisy observation of a latent label set with multiple classes (≥ 2). Whilst this paper focuses on the method's application to BCI systems, the algorithm has the potential to be applied to many application domains in which sequential missing labels are to be imputed in the presence of uncertainty. This dynamic classification algorithm combines an Ordered Probit model and an Extended Kalman Filter (EKF). The EKF estimates the parameters of the Ordered Probit model sequentially with time. We test the performance of the classification approach by processing synthetic datasets and real experimental EEG signals with multiple classes (2, 3 and 4 labels) for a Brain Computer Interfacing (BCI) experiment.

A Three Component Latent Class Model for Robust Semiparametric Gene Discovery

We propose a robust model for discovering differentially expressed genes which directly incorporates biological significance, i.e., effect dimension. Using the so-called c-fold rule, we transform the expressions into a nominal observed random variable with three categories: below a fixed lower threshold, above a fixed upper threshold or within the two thresholds. Gene expression data is then transformed into a nominal variable with three levels possibly originated by three different distributions corresponding to under expressed, not differential, and over expressed genes. This leads to a statistical model for a 3-component mixture of trinomial distributions with suitable constraints on the parameter space. In order to obtain the MLE estimates, we show how to implement a constrained EM algorithm with a latent label for the corresponding component of each gene. Different strategies for a statistically significant gene discovery are discussed and compared. We illustrate the method on a little simulation study and a real dataset on multiple sclerosis.