Semi-supervised Representation Learning Research Articles

Attribute-aware knowledge based self-abductive for semi-supervised representation learning

Attribute-aware knowledge based self-abductive for semi-supervised representation learning

Semi-Supervised Representation Learning for Segmentation on Medical Volumes and Sequences.

Benefiting from the massive labeled samples, deep learning-based segmentation methods have achieved great success for two dimensional natural images. However, it is still a challenging task to segment high dimensional medical volumes and sequences, due to the considerable efforts for clinical expertise to make large scale annotations. Self/semi-supervised learning methods have been shown to improve the performance by exploiting unlabeled data. However, they are still lack of mining local semantic discrimination and exploitation of volume/sequence structures. In this work, we propose a semi-supervised representation learning method with two novel modules to enhance the features in the encoder and decoder, respectively. For the encoder, based on the continuity between slices/frames and the common spatial layout of organs across subjects, we propose an asymmetric network with an attention-guided predictor to enable prediction between feature maps of different slices of unlabeled data. For the decoder, based on the semantic consistency between labeled data and unlabeled data, we introduce a novel semantic contrastive learning to regularize the feature maps in the decoder. The two parts are trained jointly with both labeled and unlabeled volumes/sequences in a semi-supervised manner. When evaluated on three benchmark datasets of medical volumes and sequences, our model outperforms existing methods with a large margin of 7.3% DSC on ACDC, 6.5% on Prostate, and 3.2% on CAMUS when only a few labeled data is available. Further, results on the M&M dataset show that the proposed method yields improvement without using any domain adaption techniques for data from unknown domain. Intensive evaluations reveal the effectiveness of representation mining, and superiority on performance of our method. The code is available at https://github.com/CcchenzJ/BootstrapRepresentation.

Semi-supervised contrastive regression for pharmaceutical processes

Artificial intelligence methods of time series are starting to play an increasing role in the pharmaceutical field, and in recent years, there have been significant advances in self-supervised representation learning for time series data. However, there are relatively few semi-supervised learning methods for time series, and there is almost no research on semi-supervised representation learning applicable to time series regression tasks. To address this gap, we propose a novel semi-supervised contrastive regression framework (SCRF), which combines two classical frameworks of representation learning. This framework is well-suited for regression problems of time series data from pharmaceutical processes and has been validated on a dataset collected during erythromycin production processes. Our experiments show that SCRF gets better performances than self-supervised and supervised methods, and it is more robust to missing labels, missing data, and random noise. The effectiveness of our novel contrastive learning framework and segmented augmentation methods is demonstrated through experiments.

Counterfactual Graph Learning for Anomaly Detection on Attributed Networks

Graph anomaly detection is attracting remarkable multidisciplinary research interests ranging from finance, healthcare, and social network analysis. Recent advances on graph neural networks have substantially improved the detection performance via semi-supervised representation learning. However, prior work suggests that deep graph-based methods tend to learn spurious correlations. As a result, they fail to generalize beyond training data distribution. In this paper, we aim to identify structural and contextual anomaly nodes in an attributed graph. Based on our preliminary data analyses, spurious correlations can be eliminated with causal subgraph interventions. Therefore, we propose a new graph-based anomaly detection model that can learn causal relations for anomaly detection while generalizing to new environments. To handle situations with varying environments, we steer the generative model to manufacture synthetic environment features, which are exerted on realistic subgraphs to generate counterfactual subgraphs. Further, these counterfactual subgraphs help a few-shot anomaly detection model learn transferable and causal relations across different environments. The experiments on three real-world attributed graphs show that the proposed approach achieves the best performance compared to the state-of-the-art baselines and learns robust causal representations resistant to noises and spurious correlations.

FairSwiRL: fair semi-supervised classification with representation learning

Semi-supervised learning has shown its potential in many real-world applications where only few labeled examples are available. However, when some fairness constraints need to be satisfied, semi-supervised classification models often struggle as they are required to cope with the lack of sufficient information for predicting the target variable while forgetting its relationships with any sensitive and potentially discriminatory attribute. To address this issue, we propose a fair semi-supervised representation learning architecture that leads to fair and accurate classification results even in very challenging scenarios with few labeled (but biased) instances. We show experimentally that our model can be easily adopted in very general settings, as the learned representations may be employed to train any supervised classifier. Moreover, when applied to several synthetic and real-world datasets, our method is competitive with state-of-the-art fair semi-supervised approaches.

Semi-supervised rotation-invariant representation learning for wafer map pattern analysis

Recently, data-driven approaches have been widely employed to analyze the defect patterns in wafer maps, which are crucial for identifying the root causes of failures in the semiconductor fabrication process. Representation learning embeds wafer maps into compact vector representations of useful features, based on which various downstream tasks can be performed to efficiently analyze the patterns on a large scale. If wafer maps are annotated with their defect class labels, the learned representations of wafer maps will be more informative and discriminative in defect patterns. However, the manual labeling of all wafer maps by domain experts is difficult due to practical constraints. In this study, we present a semi-supervised representation learning method that fully utilizes the information from both unlabeled and labeled wafer maps to learn better representations of wafer maps with a lower labeling cost. Given a partially labeled dataset, rotation-invariant representations of wafer maps are learned using the following three objectives. First, each unlabeled wafer map is close to any wafer map of a certain class and far from those of other classes. Second, each pair of labeled wafer maps are close to each other if they belong to the same class and are far from each other otherwise. Third, the different rotations of each wafer map are close to each other for both the unlabeled and labeled wafer maps. The effectiveness of the proposed method is demonstrated for various downstream tasks related to wafer map pattern analysis: visualization, clustering, retrieval, and classifier training.

A unified network embedding algorithm for multi-type similarity measures

Traditional network embedding aims to learn representations by capturing a predefined vertex-to-vertex similarity measure. However, in practice, there are different types of similarity measures (e.g., connectivity and structural similarity), which are appropriate for different downstream applications. Meanwhile, it is hard to select the “best” similarity measure that can mostly benefit the application, considering the required domain knowledge of both application scenario and network science. It sometimes requires to cooperate these similarity measures with each other for achieving better performance. Therefore, automatically integrate multiple types of similarity measures into a uniform network embedding framework is critical to obtain effective vertex representations for a downstream application. In this paper, we address the above problem in social networks, and propose a semi-supervised representation learning algorithm. The general idea of our approach is to impose social influence, which occurs when one’s opinions, emotions, or behaviors are affected by others in a social network. Particularly, we build the connection between a user’s representation vector and the probability of her being influenced by another user to have a particular label (e.g., fraud, personal interest, etc.). We conduct efficient experiments based on six real-world datasets and find a clear improvement of our approach comparing with several state-of-the-art baselines.

Semisupervised representative learning for measuring epidermal thickness in human subjects in optical coherence tomography by leveraging datasets from rodent models.

.Significance: Morphological changes in the epidermis layer are critical for the diagnosis and assessment of various skin diseases. Due to its noninvasiveness, optical coherence tomography (OCT) is a good candidate for observing microstructural changes in skin. Convolutional neural network (CNN) has been successfully used for automated segmentation of the skin layers of OCT images to provide an objective evaluation of skin disorders. Such method is reliable, provided that a large amount of labeled data is available, which is very time-consuming and tedious. The scarcity of patient data also puts another layer of difficulty to make the model more generalizable.Aim: We developed a semisupervised representation learning method to provide data augmentations.Approach: We used rodent models to train neural networks for accurate segmentation of clinical data.Result: The learning quality is maintained with only one OCT labeled image per volume that is acquired from patients. Data augmentation introduces a semantically meaningful variance, allowing for better generalization. Our experiments demonstrate the proposed method can achieve accurate segmentation and thickness measurement of the epidermis.Conclusion: This is the first report of semisupervised representative learning applied to OCT images from clinical data by making full use of the data acquired from rodent models. The proposed method promises to aid in the clinical assessment and treatment planning of skin diseases.

Semi-Supervised Representation Learning via Triplet Loss Based on Explicit Class Ratio of Unlabeled Data

In this paper, we propose a semi-supervised triplet loss function that realizes semi-supervised representation learning in a novel manner. We extend conventional triplet loss, which uses labeled data to achieve representation learning, so that it can deal with unlabeled data. We estimate, in advance, the degree to which each label applies to each unlabeled data point, and optimize the loss function with unlabeled features according to the resulting ratios. Since the proposed loss function has the effect of adjusting the distribution of all unlabeled data, it complements methods based on consistency regularization, which has been extensively studied in recent years. Combined with a consistency regularization-based method, our method achieves more accurate semi-supervised learning. Experiments show that the proposed loss function achieves a higher accuracy than the conventional fine-tuning method.

Boosting semi-supervised network representation learning with pseudo-multitasking

Boosting semi-supervised network representation learning with pseudo-multitasking

Representation learning using Attention Network and CNN for Heterogeneous networks

Network embedding (NE), also known as network representation learning (NRL), is a method to learn a low-dimensional latent representation of nodes in an information network. The real-world data is usually presented in the form of heterogeneous information network (HIN) with multiple types of nodes and edges. Because of the rich information in HINs, it is necessary for a network embedding method to incorporate this information into the low-dimensional potential representation of the nodes as much as possible. In this paper, we propose a semi-supervised representation learning model using a graph attention network and a convolutional neural network (CNN) for HINs, called RANCH. In the part of the graph attention network, we construct a heterogeneous graph attention network using heterogeneous edges to preserve the features of nodes and the structure of network. In the part of the CNN, we leverage a 1D-CNN sentence classification model from natural language processing (NLP) community by adopting edge-constrained truncated random walks to generate node sequences, which can be treated as a corpus of words and sentences. The latter part further integrates the structural information of the network on the basis of the previous part and strengthens the influence of the node’s label information on the node representation. We have performed experiments of node classification on three real-world datasets, and the result shows that our model performs better than the state-of-the-arts.

Rethinking Graph Regularization for Graph Neural Networks

The graph Laplacian regularization term is usually used in semi-supervised representation learning to provide graph structure information for a model f(X). However, with the recent popularity of graph neural networks (GNNs), directly encoding graph structure A into a model, i.e., f(A, X), has become the more common approach. While we show that graph Laplacian regularization brings little-to-no benefit to existing GNNs, and propose a simple but non-trivial variant of graph Laplacian regularization, called Propagation-regularization (P-reg), to boost the performance of existing GNN models. We provide formal analyses to show that P-reg not only infuses extra information (that is not captured by the traditional graph Laplacian regularization) into GNNs, but also has the capacity equivalent to an infinite-depth graph convolutional network. We demonstrate that P-reg can effectively boost the performance of existing GNN models on both node-level and graph-level tasks across many different datasets.

A Hierarchical Graph Convolution Network for Representation Learning of Gene Expression Data

The curse of dimensionality, which is caused by high-dimensionality and low-sample-size, is a major challenge in gene expression data analysis. However, the real situation is even worse: labelling data is laborious and time-consuming, so only a small part of the limited samples will be labelled. Having such few labelled samples further increases the difficulty of training deep learning models. Interpretability is an important requirement in biomedicine. Many existing deep learning methods are trying to provide interpretability, but rarely apply to gene expression data. Recent semi-supervised graph convolution network methods try to address these problems by smoothing the label information over a graph. However, to the best of our knowledge, these methods only utilize graphs in either the feature space or sample space, which restrict their performance. We propose a transductive semi-supervised representation learning method called a hierarchical graph convolution network (HiGCN) to aggregate the information of gene expression data in both feature and sample spaces. HiGCN first utilizes external knowledge to construct a feature graph and a similarity kernel to construct a sample graph. Then, two spatial-based GCNs are used to aggregate information on these graphs. To validate the model's performance, synthetic and real datasets are provided to lend empirical support. Compared with two recent models and three traditional models, HiGCN learns better representations of gene expression data, and these representations improve the performance of downstream tasks, especially when the model is trained on a few labelled samples. Important features can be extracted from our model to provide reliable interpretability.

TEST: Triplet Ensemble Student-Teacher Model for Unsupervised Person Re-Identification.

The self-ensembling methods have achieved amazing performance for semi-supervised representation learning and domain adaptation. However, the disadvantage of these methods is that the teacher network is tightly coupled with the student network, which limits the descriptive ability of the self-ensembling model. To overcome the coupling effect between the teacher network and the student network, we propose a novel Triplet Ensemble Student-Teacher (TEST) model for unsupervised person re-identification, which consists of one teacher network T and two student networks S1 and S2 . Similar to the traditional self-ensembling model, the student network S1 is applied to update the teacher network T . Furthermore, a closed-loop learning mechanism is built in the TEST model by imposing an ensemble consistent constraint between T and S2 , and performing a heterogeneous co-teaching procedure between S1 and S2 . With the closed-loop learning mechanism, the TEST model can loosen the constraint between the teacher T and the student S1 , and enhance the descriptive ability of S1 . Besides, the knowledge exchange between S1 and S2 can ensure that the two student networks can elegantly deal with the noisy labels and avoid coupling. By training the TEST model with the clustering-generated pseudo labels, we can achieve effective and robust representation learning for unsupervised person re-identification. The evaluations on three widely-used benchmarks show that our approach can achieve significant performance compared with state-of-the-art methods.

Influence modeling of opinion switching by Twitter users in public shaming events

Public shaming is mass criticism of someone for perceived violation of an established social norm. These events are ever increasing in number and scope on online social media platforms. An individual instance of shaming is, as a matter of fact, an expression of opinion regarding the target or victim of the event. Public shaming events are more engaging and often evoke stronger opinions from users than word of mouth spreading of information. This work models polarity of a user’s opinion as a function of several factors, such as his/her recently held opinions, received opinions and the environment, in online public shaming events on Twitter. Specifically, we investigate the opinion change process of Twitter users by highlighting the influence of responsible factors. A semi-supervised representation learning model is used to get opinion vectors from tweets. Next, an opinion prediction model based on the user’s past opinions, received opinions and environment is designed. We show that opinion changes can be attributed to factors in varying proportions depending on the direction of the change. Thus, a strategy to stop a shaming event by inducing opinion changes in the participants needs to take into account the direction of the change. Efficacy of possible counter measures to shaming is analyzed based on our findings.

Standard machine learning approaches outperform deep representation learning on phenotype prediction from transcriptomics data

BackgroundThe ability to confidently predict health outcomes from gene expression would catalyze a revolution in molecular diagnostics. Yet, the goal of developing actionable, robust, and reproducible predictive signatures of phenotypes such as clinical outcome has not been attained in almost any disease area. Here, we report a comprehensive analysis spanning prediction tasks from ulcerative colitis, atopic dermatitis, diabetes, to many cancer subtypes for a total of 24 binary and multiclass prediction problems and 26 survival analysis tasks. We systematically investigate the influence of gene subsets, normalization methods and prediction algorithms. Crucially, we also explore the novel use of deep representation learning methods on large transcriptomics compendia, such as GTEx and TCGA, to boost the performance of state-of-the-art methods. The resources and findings in this work should serve as both an up-to-date reference on attainable performance, and as a benchmarking resource for further research.ResultsApproaches that combine large numbers of genes outperformed single gene methods consistently and with a significant margin, but neither unsupervised nor semi-supervised representation learning techniques yielded consistent improvements in out-of-sample performance across datasets. Our findings suggest that using l2-regularized regression methods applied to centered log-ratio transformed transcript abundances provide the best predictive analyses overall.ConclusionsTranscriptomics-based phenotype prediction benefits from proper normalization techniques and state-of-the-art regularized regression approaches. In our view, breakthrough performance is likely contingent on factors which are independent of normalization and general modeling techniques; these factors might include reduction of systematic errors in sequencing data, incorporation of other data types such as single-cell sequencing and proteomics, and improved use of prior knowledge.

Semi-Supervised Representation Learning for Remote Sensing Image Classification Based on Generative Adversarial Networks

In the existing studies on remote sensing image scene classification, the supervised learning methods which are fine-tuned from pre-trained model require a large amount of labeled training data and parameters, while unsupervised learning methods do not make full use of label information, and the classification performance could be improved. In this paper, we introduced semi-supervised learning into generative adversarial network (GAN), so the discriminator learned more discriminative features from labeled data and unlabeled data. Moreover, the mixup data augmentation method was introduced into our classification model to augment the data and stabilized the training process. We carried out extensive experiments for both UC-Merced and NWPU-RESISC45 datasets with a 5-fold cross-validation protocol using a linear SVM as classifier. We trained the proposed method on UC-Merced dataset and achieve an average overall accuracy of 94.05% under 80% training ratio. When trained on NWPU-RESISC45 dataset, the proposed method reached an average overall accuracy of 83.12% and 92.78% under the training ratios of 20% and 80% respectively, which achieves the state-of-the-art deep learning methods without pre-training.

Semi-supervised representation learning via dual autoencoders for domain adaptation

Domain adaptation aims to exploit the knowledge in source domain to promote the learning tasks in target domain, which plays a critical role in real-world applications. Recently, lots of deep learning approaches based on autoencoders have achieved a significance performance in domain adaptation. However, most existing methods focus on minimizing the distribution divergence by putting the source and target data together to learn global feature representations, while they do not consider the local relationship between instances in the same category from different domains. To address this problem, we propose a novel Semi-Supervised Representation Learning framework via Dual Autoencoders for domain adaptation, named SSRLDA. More specifically, we extract richer feature representations by learning the global and local feature representations simultaneously using two novel autoencoders, which are referred to as marginalized denoising autoencoder with adaptation distribution (MDAad) and multi-class marginalized denoising autoencoder (MMDA) respectively. Meanwhile, we make full use of label information to optimize feature representations. Experimental results show that our proposed approach outperforms several state-of-the-art baseline methods.

A COMPARATIVE ANALYSIS OF UNSUPERVISED AND SEMI-SUPERVISED REPRESENTATION LEARNING FOR REMOTE SENSING IMAGE CATEGORIZATION

Abstract. This work aims at investigating unsupervised and semi-supervised representation learning methods based on generative adversarial networks for remote sensing scene classification. The work introduces a novel approach, which consists in a semi-supervised extension of a prior unsupervised method, known as MARTA-GAN. The proposed approach was compared experimentally with two baselines upon two public datasets, UC-MERCED and NWPU-RESISC45. The experiments assessed the performance of each approach under different amounts of labeled data. The impact of fine-tuning was also investigated. The proposed method delivered in our analysis the best overall accuracy under scarce labeled samples, both in terms of absolute value and in terms of variability across multiple runs.

Semi-Supervised Matrix Completion for Cross-Lingual Text Classification

Cross-lingual text classification is the task of assigning labels to observed documents in a label-scarce target language domain by using a prediction model trained with labeled documents from a label-rich source language domain. Cross-lingual text classification is popularly studied in natural language processing area to reduce the expensive manual annotation effort required in the target language domain. In this work, we propose a novel semi-supervised representation learning approach to address this challenging task by inducing interlingual features via semi-supervised matrix completion. To evaluate the proposed learning technique, we conduct extensive experiments on eighteen cross language sentiment classification tasks with four different languages. The empirical results demonstrate the efficacy of the proposed approach, and show it outperforms a number of related cross-lingual learning methods.