Semi-supervised Learning Tasks Research Articles

Generative Adversarial Training for Supervised and Semi-supervised Learning.

Neural networks have played critical roles in many research fields. The recently proposed adversarial training (AT) can improve the generalization ability of neural networks by adding intentional perturbations in the training process, but sometimes still fail to generate worst-case perturbations, thus resulting in limited improvement. Instead of designing a specific smoothness function and seeking an approximate solution used in existing AT methods, we propose a new training methodology, named Generative AT (GAT) in this article, for supervised and semi-supervised learning. The key idea of GAT is to formulate the learning task as a minimax game, in which the perturbation generator aims to yield the worst-case perturbations that maximize the deviation of output distribution, while the target classifier is to minimize the impact of this perturbation and prediction error. To solve this minimax optimization problem, a new adversarial loss function is constructed based on the cross-entropy measure. As a result, the smoothness and confidence of the model are both greatly improved. Moreover, we develop a trajectory-preserving-based alternating update strategy to enable the stable training of GAT. Numerous experiments conducted on benchmark datasets clearly demonstrate that the proposed GAT significantly outperforms the state-of-the-art AT methods in terms of supervised and semi-supervised learning tasks, especially when the number of labeled examples is rather small in semi-supervised learning.

An Empirical Study of Graph-Based Approaches for Semi-supervised Time Series Classification

Time series data play an important role in many applications and their analysis reveals crucial information for understanding the underlying processes. Among the many time series learning tasks of great importance, we here focus on semi-supervised learning based on a graph representation of the data. Two main aspects are studied in this paper. Namely, suitable distance measures to evaluate the similarities between different time series, and the choice of learning method to make predictions based on a given number of pre-labeled data points. However, the relationship between the two aspects has never been studied systematically in the context of graph-based learning. We describe four different distance measures, including (Soft) DTW and MPDist, a distance measure based on the Matrix Profile, as well as four successful semi-supervised learning methods, including the recently introduced graph Allen–Cahn method and Graph Convolutional Neural Network method. We provide results for the novel combination of these distance measures with both the Allen-Cahn method and the GCN algorithm for binary semi-supervised learning tasks for various time-series data sets. In our findings we compare the chosen graph-based methods using all distance measures and observe that the results vary strongly with respect to the accuracy. We then observe that no clear best combination to employ in all cases is found. Our study provides a reproducible framework for future work in the direction of semi-supervised learning for time series with a focus on graph representations.

The Representation of Large-Scale Graph Based on Semi-Supercised Learning

With the development of artificial intelligence, the technology of graph representation has been widely used in graph structure data mining. The traditional graph representation mostly uses adjacency matrix. The representation of adjacency matrix is faced with two major challenges. One reason is extremely inefficient in the storage and computing of large graphs with the massive growth of data; the other is that the characterized data is difficult to retain the original structural information of graphs. Aiming at the above two points, an efficient graph representation method is proposed in this paper. The first-order similarity and second-order similarity of the graph are used as constraints to train the original graph structure data, and the nodes in the graph are mapped to a low-dimensional space, which greatly reduces the cost of storage space and computing while retaining the graph structure information. Experimental results show that the proposed method is significantly better than the Deepwalk and spectral clustering methods in different proportion of label data. Meanwhile, the proposed method can be applied to the semi-supervised learning task with a small number of labels, which provides convenience for the subsequent data analysis.

E-GCN: graph convolution with estimated labels

G raph C onvolutional N etwork (GCN) has been commonly applied for semi-supervised learning tasks. However, the established GCN frequently only considers the given labels in the topology optimization, which may not deliver the best performance for semi-supervised learning tasks. In this paper, we propose a novel G raph C onvolutional N etwork with E stimated labels (E-GCN) for semi-supervised learning. The core design of E-GCN is to learn a suitable network topology for semi-supervised learning by linking both estimated labels and given labels in a centralized network framework. The major enhancement is that both given labels and estimated labels are utilized for the topology optimization in E-GCN, which assists the graph convolution implementation for unknown labels evaluation. Experimental results demonstrate that E-GCN is significantly better than s tate-o f-t he-a rt (SOTA) baselines without estimated labels.

Higher-Order Graph Convolutional Networks With Multi-Scale Neighborhood Pooling for Semi-Supervised Node Classification

Existing popular methods for semi-supervised node classification with high-order convolution improve the learning ability of graph convolutional networks (GCNs) by capturing the feature information from high-order neighborhoods. However, these methods with high-order convolution usually require many parameters and high computational complexity. To address these limitations, we propose HCNP, a new higher-order GCN for semi-supervised node learning tasks, which can simultaneously aggregate information of various neighborhoods by constructing high-order convolution. In HCNP, we reduce the number of parameters using a weight sharing mechanism and combine the neighborhood information via multi-scale neighborhood pooling. Further, HCNP does not require a large number of hidden units, and it fits a few parameters and exhibits low complexity. We show that HCNP matches GCNs in terms of complexity and parameters. Comprehensive evaluations on publication citation datasets (Citeseer, Pubmed, and Cora) demonstrate that the proposed methods outperform MixHop in most cases while maintaining lower complexity and fewer parameters and achieve state-of-the-art performance in terms of accuracy and parameters compared to other baselines.

Probabilistic Reconstruction of Spatio-Temporal Processes Over Multi-Relational Graphs

Given nodal observations that can be limited due to sampling costs or privacy concerns, several network-science-related applications entail reconstruction of values on all network nodes by leveraging topology information. Such a semi-supervised learning (SSL) task has been tackled mainly for graphs capturing a single class of inter-dependencies (or relations) among nodal variables. Faced with multi-relational graphs (MRGs), which emerge in various real-world networks, the present work introduces a principled framework to extrapolate spatio-temporal nodal processes that could be stationary or nonstationary. Broadening the scope of graph kernel-based approaches to MRGs, stationary graph processes are modeled first using a Gaussian mixture (GM) prior, where the covariance matrix of each Gaussian component describes one of the relations in the MRG. To further cope with nonstationary nodal processes, a first-order topology-dependent Gaussian transition prior is considered per relation, what gives rise to a GM transition density that accounts for all relations. In both cases, adapting the expectation-maximization (EM) algorithm yields two novel graph-adaptive solvers that not only reconstructs nodal features over unobserved nodes, but also quantifies the contribution of each relation. To enrich expressiveness of these novel EM-based approaches, multiple kernels per relation are also explored. Experiments with real data showcase the merits of the proposed methods relative to the existing alternatives.

Semi-Supervised Classification of Graph Convolutional Networks with Laplacian Rank Constraints

Graph convolutional networks (GCNs), as an extension of classic convolutional neural networks (CNNs) in graph processing, have achieved good results in completing semi-supervised learning tasks. Traditional GCNs usually use fixed graph to complete various semi-supervised classification tasks, such as chemical molecules and social networks. Graph is an important basis for the classification of GCNs model, and its quality has a large impact on the performance of the model. For low-quality input graph, the classification results of the GCNs model are often not ideal. In order to improve the classification effect of GCNs model, we propose a graph learning method to generate high-quality topological graph, which is more suitable for GCNs model classification. We use the correlation between the data to generate a data similarity matrix, and apply Laplacian rank constraint to similarity matrix, so that the number of connected components of the topological graph is consistent with the number of categories of the original data. Experimental results on 10 real datasets show that our method is better than the comparison method in classification effect.

Regularizing Deep Networks With Semantic Data Augmentation.

Data augmentation is widely known as a simple yet surprisingly effective technique for regularizing deep networks. Conventional data augmentation schemes, e.g., flipping, translation or rotation, are low-level, data-independent and class-agnostic operations, leading to limited diversity for augmented samples. To this end, we propose a novel semantic data augmentation algorithm to complement traditional approaches. The proposed method is inspired by the intriguing property that deep networks are effective in learning linearized features, i.e., certain directions in the deep feature space correspond to meaningful semantic transformations, e.g., changing the background or view angle of an object. Based on this observation, translating training samples along many such directions in the feature space can effectively augment the dataset for more diversity. To implement this idea, we first introduce a sampling based method to obtain semantically meaningful directions efficiently. Then, an upper bound of the expected cross-entropy (CE) loss on the augmented training set is derived by assuming the number of augmented samples goes to infinity, yielding a highly efficient algorithm. In fact, we show that the proposed implicit semantic data augmentation (ISDA) algorithm amounts to minimizing a novel robust CE loss, which adds minimal extra computational cost to a normal training procedure. In addition to supervised learning, ISDA can be applied to semi-supervised learning tasks under the consistency regularization framework, where ISDA amounts to minimizing the upper bound of the expected KL-divergence between the augmented features and the original features. Although being simple, ISDA consistently improves the generalization performance of popular deep models (e.g., ResNets and DenseNets) on a variety of datasets, i.e., CIFAR-10, CIFAR-100, SVHN, ImageNet, and Cityscapes. Code for reproducing our results is available at https://github.com/blackfeather-wang/ISDA-for-Deep-Networks.

Human activity recognition by manifold regularization based dynamic graph convolutional networks

Deep learning has shown superiority to extract more representative features from multimedia data in recent years. Recently, the most typical graph convolutional networks (GCN) has achieved excellent performance in the semi-supervised framework-based data representation learning tasks. GCN successfully generalizes traditional convolutional neural networks to encode arbitrary graphs by exploiting the graph Laplacian-based sample structure information. However, GCN only fuses the static structure information. It is difficult to guarantee that its structure information is optimal during the training process and applicable for all practical applications. To tackle the above problem, in this paper, we propose a manifold regularized dynamic graph convolutional network (MRDGCN). The proposed MRDGCN automatically updates the structure information by manifold regularization until model fitting. In particular, we build an optimization convolution layer formulation to acquire the optimal structure information. Thus, MRDGCN can automatically learn high-level sample features to improve the performance of data representation learning. To demonstrate the effectiveness of our proposed model, we apply MRDGCN on the semi-supervised classification tasks. The extensive experiment results on human activity datasets and citation network datasets validate the performance of MRDGCN compared with GCN and other semi-supervised learning methods.

FMixCutMatch for semi-supervised deep learning

Mixed sample augmentation (MSA) has witnessed great success in the research area of semi-supervised learning (SSL) and is performed by mixing two training samples as an augmentation strategy to effectively smooth the training space. Following the insights on the efficacy of cut-mix in particular, we propose FMixCut, an MSA that combines Fourier space-based data mixing (FMix) and the proposed Fourier space-based data cutting (FCut) for labeled and unlabeled data augmentation. Specifically, for the SSL task, our approach first generates soft pseudo-labels using the model’s previous predictions. The model is then trained to penalize the outputs of the FMix-generated samples so that they are consistent with their mixed soft pseudo-labels. In addition, we propose to use FCut, a new Cutout-based data augmentation strategy that adopts the two masked sample pairs from FMix for weighted cross-entropy minimization. Furthermore, by implementing two regularization techniques, namely, batch label distribution entropy maximization and sample confidence entropy minimization, we further boost the training efficiency. Finally, we introduce a dynamic labeled–unlabeled data mixing (DDM) strategy to further accelerate the convergence of the model. Combining the above process, we finally call our SSL approach as ”FMixCutMatch”, in short FMCmatch. As a result, the proposed FMCmatch achieves state-of-the-art performance on CIFAR-10/100, SVHN and Mini-Imagenet across a variety of SSL conditions with the CNN-13, WRN-28-2 and ResNet-18 networks. In particular, our method achieves a 4.54% test error on CIFAR-10 with 4K labels under the CNN-13 and a 41.25% Top-1 test error on Mini-Imagenet with 10K labels under the ResNet-18. Our codes for reproducing these results are publicly available at https://github.com/biuyq/FMixCutMatch.

Anisotropic Graph Convolutional Network for Semi-Supervised Learning

Graph convolutional networks learn effective node embeddings that have proven to be useful in achieving high-accuracy prediction results in semi-supervised learning tasks, such as node classification. However, these networks suffer from the issue of over-smoothing and shrinking effect of the graph due in large part to the fact that they diffuse features across the edges of the graph using a linear Laplacian flow. This limitation is especially problematic for the task of node classification, where the goal is to predict the label associated with a graph node. To address this issue, we propose an anisotropic graph convolutional network for semi-supervised node classification by introducing a nonlinear function that captures informative features from nodes, while preventing oversmoothing. The proposed framework is largely motivated by the good performance of anisotropic diffusion in image and geometry processing, and learns nonlinear representations based on local graph structure and node features. The effectiveness of our approach is demonstrated on three citation networks and two image datasets, achieving better or comparable classification accuracy results compared to the standard baseline methods.

Robust kernelized graph-based learning

The studies of hidden complex structures in data have popularized the use of graph-based learning methods in semi-supervised and unsupervised learning tasks. Kernelized graph-based methods are proven to perform better, but these methods suffer from the issue of appropriate kernel selection. Instead of using multiple views, these methods generally use a single view. But multi-view methods need a proper weight assignment technique to each view in proportion to their contribution to the learning task. To solve this, a novel Self-weighted Multi-view Multiple Kernel Learning (SMVMKL) framework is proposed using multiple kernels on multiple views that automatically assigns appropriate weight to each kernel of each view without introducing an additional parameter. But the real-world data that is either noisy or corrupt with outliers which may effect the performance of the proposed SMVMKL method. To deal with this, a Robust Self-weighted Multi-view Multiple Kernel Learning (RSMVMKL) framework using the l2,1-norm has also been proposed that reduces the effect of outliers present in the data set. Both the proposed methods have been evaluated on multiple benchmark data sets and result in a performance comparable with the other state-of-the-art multi-view methods considered in this paper.

SCOs: Semi-Supervised Co-Selection by a Similarity Preserving Approach

In this paper, we focus on co-selection of instances and features in the semi-supervised learning scenario. In this context, co-selection becomes a more challenging problem as data contain labeled and unlabeled examples sampled from the same population. To carry out such semi-supervised co-selection, we propose a unified framework, called sCOs, which efficiently integrates labeled and unlabeled parts into the co-selection process. The framework is based on introducing both a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">sparse regularization term</i> and a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">similarity preserving approach</i> . It evaluates the usefulness of features and instances in order to select the most relevant ones, simultaneously. We propose two efficient algorithms that work for both convex and nonconvex functions. To the best of our knowledge, this paper offers, for the first time ever, a study utilizing nonconvex penalties for the co-selection of semi-supervised learning tasks. Experimental results on some known benchmark datasets are provided for validating sCOs and comparing it with some representative methods in the state-of-the art.

A fast graph-based data classification method with applications to 3D sensory data in the form of point clouds

Data classification, where the goal is to divide data into predefined classes, is a fundamental problem in machine learning with many applications, including the classification of 3D sensory data. In this paper, we present a data classification method which can be applied to both semi-supervised and unsupervised learning tasks. The algorithm is derived by unifying complementary region-based and edge-based approaches; a gradient flow of the optimization energy is performed using modified auction dynamics. In addition to being unconditionally stable and efficient, the method is equipped with several properties allowing it to perform accurately even with small labeled training sets, often with considerably fewer labeled training elements compared to competing methods; this is an important advantage due to the scarcity of labeled training data. Some of the properties are: the embedding of data into a weighted similarity graph, the in-depth construction of the weights using, e.g., geometric information, the use of a combination of region-based and edge-based techniques, the incorporation of class size information and integration of random fluctuations. The effectiveness of the method is demonstrated by experiments on classification of 3D point clouds; the algorithm classifies a point cloud of more than a million points in 1–2 min.

Deep representation clustering-based fault diagnosis method with unsupervised data applied to rotating machinery

Despite the recent advances on intelligent data-driven machinery fault diagnostics, large amounts of high-quality supervised data are mostly required for model training. However, it is usually difficult and expensive to collect sufficient labeled data in real industries, and the difficulty in data preparation significantly hinders the application of the intelligent diagnostic methods. In order to address the data sparsity issue with insufficient labeled data, a deep learning-based fault diagnosis method is proposed in this study, exploring additional unsupervised data which are generally easy for collection. A three-stage training scheme is adopted, i.e. pre-training, representation clustering and enhanced supervised learning. The auto-encoder structure is used for feature extraction, and distance metric learning and k-means clustering method are integrated in the neural network architecture for unsupervised learning. Two rotating machinery datasets are used for validations. The proposed method not only achieves promising diagnostic performance on the semi-supervised learning tasks with few labeled data, but also is well suited for pure unsupervised learning problems. The experimental results suggest the proposed method offers a promising approach on exploiting unsupervised data for fault diagnostics.

Combining deep generative and discriminative models for Bayesian semi-supervised learning

Generative models can be used for a wide range of tasks, and have the appealing ability to learn from both labelled and unlabelled data. In contrast, discriminative models cannot learn from unlabelled data, but tend to outperform their generative counterparts in supervised tasks. We develop a framework to jointly train deep generative and discriminative models, enjoying the benefits of both. The framework allows models to learn from labelled and unlabelled data, as well as naturally account for uncertainty in predictive distributions, providing the first Bayesian approach to semi-supervised learning with deep generative models. We demonstrate that our blended discriminative and generative models outperform purely generative models in both predictive performance and uncertainty calibration in a number of semi-supervised learning tasks.

Mutual Improvement Between Temporal Ensembling and Virtual Adversarial Training

The research of semi-supervised learning (SSL) is of great significance because it is very expensive to collect a large quantity of data with labels in some fields. Two recent deep learning-based SSL algorithms, temporal ensembling and virtual adversarial training (VAT), have achieved state-of-the-art accuracy in some classical SSL tasks, while both of them have shortcomings. Because of simply adding random noise to training data, temporal ensembling is not fully utilized. In addition, VAT has considerable time costs because there are two inferences in each epoch for unlabeled samples. In this paper, we propose the use of virtual adversarial perturbations (VAP) in temporal ensembling rather than random noises to improve performance. Moreover, we also find that reusing VAP can accelerate the training process of VAT without losing obvious accuracy. The two methods are validated on MNIST, FashionMNIST and SVHN.

Extreme semi-supervised learning for multiclass classification

Semi-Supervised Support Vector Machines (S3VMs) provide a powerful framework for Semi-Supervised Learning (SSL) tasks which leverage widely available unlabeled data to improve performance. However, there exist three issues in S3VMs: (i) S3VMs require concurrently training c one-against-all (OAA) classifiers (c is the number of classes) for multiclass classification, which is prohibitive for large c; (ii) S3VMs require huge computational time and large storage (because of the large kernel matrix) in large-scale training and testing; (iii) S3VMs require the balance constraint in the unlabeled data, which not only needs prior knowledge from the unlabeled data (the prior knowledge is unavailable in some applications), but also makes their nonconvex optimization problem more intractable. To address these issues, a novel method called Extreme Semi-Supervised Learning (ESSL) is proposed in this paper. First, the framework of Extreme Learning Machine (ELM) is adopted to handle both binary and multiclass classification problems in a unified model. Second, the hidden layer is encoded by an extremely small approximate empirical kernel map (AEKM) to greatly reduce the computational cost and the memory usage for training and testing. Third, the balance constraint (prior knowledge) in the unlabeled data is removed through the elaborative design of weighting function (which emphasizes the importance of labeled data and the minority pattern in the labeled data).By these three ways, ESSL can be solved effectively and efficiently based on alternating optimization (AO). More specifically, ESSL can be analytically and simply solved by generalized pseudoinverse and oneHotMap function (without any optimization solver and the OAA strategy) in the AO procedure, and consequently, better performance and much faster training speed are always achieved in ESSL. Our empirical study shows that ESSL significantly outperforms existing efficient SSL methods (e.g., meanS3VM and SS-ELM) in terms of accuracy, efficiency and memory, especially for large-scale multiclass problems. As an example, on the 20Newsgroups dataset, ESSL respectively runs 45 and 120 times faster than meanS3VM for training and testing with the improvement in accuracy of 3%, while the memory usage is reduced to 1/14. It is noteworthy that even though all the model parameters are with default values, ESSL already produces very excellent performance without fine-tuning parameters.

Data induced masking representation learning for face data analysis

Representation learning models, such as the sparse representation and low-rank representation, have shown pleasing efficacy in exploring the intrinsic data structures for patternrecognition tasks. However, conventional methods ignore the local geometric and similarity information among samples, and the performance is restricted. To address this issue, this paper proposes a novel Data Induced Masking Representation (DIMR) learning model by imposing explicit regularization and low-rank constraint. Specifically, DIMR is formulated for shrinking the representations of inter-class and non-neighbor samples. An extra representation regularization term is deployed with a data induced mask matrix, which can incorporate label and locality priors to guide the learning of affinity representation matrix. The affinity graph derived from DIMR is with low-rank, locality preservation (sparsity) and label guiding, such that it can better characterize the adjacent relationship between samples. Extensive experiments on benchmark face datasets demonstrate the superiority of DIMR for both semi-supervised classification and semi-supervised subspace learning tasks.

Semi-supervised online structure learning for composite event recognition

Online structure learning approaches, such as those stemming from statistical relational learning, enable the discovery of complex relations in noisy data streams. However, these methods assume the existence of fully-labelled training data, which is unrealistic for most real-world applications. We present a novel approach for completing the supervision of a semi-supervised structure learning task. We incorporate graph-cut minimisation, a technique that derives labels for unlabelled data, based on their distance to their labelled counterparts. In order to adapt graph-cut minimisation to first order logic, we employ a suitable structural distance for measuring the distance between sets of logical atoms. The labelling process is achieved online (single-pass) by means of a caching mechanism and the Hoeffding bound, a statistical tool to approximate globally-optimal decisions from locally-optimal ones. We evaluate our approach on the task of composite event recognition by using a benchmark dataset for human activity recognition, as well as a real dataset for maritime monitoring. The evaluation suggests that our approach can effectively complete the missing labels and eventually, improve the accuracy of the underlying structure learning system.