Graph-based Semi-supervised Learning Research Articles

Ensemble Kalman inversion: a derivative-free technique for machine learning tasks

The standard probabilistic perspective on machine learning gives rise to empirical risk-minimization tasks that are frequently solved by stochastic gradient descent (SGD) and variants thereof. We present a formulation of these tasks as classical inverse or filtering problems and, furthermore, we propose an efficient, gradient-free algorithm for finding a solution to these problems using ensemble Kalman inversion (EKI). The method is inherently parallelizable and is applicable to problems with non-differentiable loss functions, for which back-propagation is not possible. Applications of our approach include offline and online supervised learning with deep neural networks, as well as graph-based semi-supervised learning. The essence of the EKI procedure is an ensemble based approximate gradient descent in which derivatives are replaced by differences from within the ensemble. We suggest several modifications to the basic method, derived from empirically successful heuristics developed in the context of SGD. Numerical results demonstrate wide applicability and robustness of the proposed algorithm.

An enhanced graph-based semi-supervised learning algorithm to detect fake users on Twitter

Over the the past decade, social networking services (SNS) have proliferated on the web. The nature of such sites makes identity deception easy, providing a fast means for creating and managing identities, and then connecting with and deceiving others. Fake users are those accounts specifically created for purposes such as stalking or abuse of another user, for slander, or for marketing. The current system for detecting deception depends on behavioral, non-behavioral and user-generated content (UGC) information gathered from users. Although these methods have high detection accuracy, they cannot be implemented in databases with massive volumes of data. To address this issue, this paper proposes an enhanced graph-based semi-supervised learning algorithm (EGSLA) to detect fake users from a large volume of Twitter data. The proposed method encompasses four modules: data collection, feature extraction, classification and decision making. Data collected from Twitter using Scrapy is utilized for the evaluation. The performance of the proposed algorithm is tested with existing game theory, k-nearest neighbor (KNN), support vector machine (SVM) and decision tree techniques. The results show that the proposed EGSLA algorithm achieves 90.3% accuracy in spotting fake users.

Matrix Completion for Graph-Based Deep Semi-Supervised Learning

Convolutional Neural Networks (CNNs) have provided promising achievements for image classification problems. However, training a CNN model relies on a large number of labeled data. Considering the vast amount of unlabeled data available on the web, it is important to make use of these data in conjunction with a small set of labeled data to train a deep learning model. In this paper, we introduce a new iterative Graph-based Semi-Supervised Learning (GSSL) method to train a CNN-based classifier using a large amount of unlabeled data and a small amount of labeled data. In this method, we first construct a similarity graph in which the nodes represent the CNN features corresponding to data points (labeled and unlabeled) while the edges tend to connect the data points with the same class label. In this graph, the missing label of unsupervised nodes is predicted by using a matrix completion method based on rank minimization criterion. In the next step, we use the constructed graph to calculate triplet regularization loss which is added to the supervised loss obtained by initially labeled data to update the CNN network parameters.

Integration of machine learning and mechanistic models accurately predicts variation in cell density of glioblastoma using multiparametric MRI

Glioblastoma (GBM) is a heterogeneous and lethal brain cancer. These tumors are followed using magnetic resonance imaging (MRI), which is unable to precisely identify tumor cell invasion, impairing effective surgery and radiation planning. We present a novel hybrid model, based on multiparametric intensities, which combines machine learning (ML) with a mechanistic model of tumor growth to provide spatially resolved tumor cell density predictions. The ML component is an imaging data-driven graph-based semi-supervised learning model and we use the Proliferation-Invasion (PI) mechanistic tumor growth model. We thus refer to the hybrid model as the ML-PI model. The hybrid model was trained using 82 image-localized biopsies from 18 primary GBM patients with pre-operative MRI using a leave-one-patient-out cross validation framework. A Relief algorithm was developed to quantify relative contributions from the data sources. The ML-PI model statistically significantly outperformed (p < 0.001) both individual models, ML and PI, achieving a mean absolute predicted error (MAPE) of 0.106 ± 0.125 versus 0.199 ± 0.186 (ML) and 0.227 ± 0.215 (PI), respectively. Associated Pearson correlation coefficients for ML-PI, ML, and PI were 0.838, 0.518, and 0.437, respectively. The Relief algorithm showed the PI model had the greatest contribution to the result, emphasizing the importance of the hybrid model in achieving the high accuracy.

Residential Household Non-Intrusive Load Monitoring via Graph-Based Multi-Label Semi-Supervised Learning

Nonintrusive load monitoring refers to inferring what appliances are operating in a household at a given time solely from fluctuations on the main power feeder. It is one approach to demand-side management strategies in the smart grid, and most current research employs machine learning to make those inferences. However, the learning algorithms used usually require knowledge of the set of appliances active at each sample instant in addition to the main feeder fluctuations. This data is not ordinarily available in field usage. As a compromise, we examine semi-supervised learning algorithms, which only need a small sample of observed power signals annotated with active appliances (e.g., from an initial “registration” period). As multiple independent appliances may operate concurrently, we furthermore employ multilabel classification in our solution. Three new graph-based semi-supervised multilabel load monitoring algorithms are proposed and evaluated on five public datasets. We find that the best algorithm can outperform state-of-the-art results on these datasets.

Drug repurposing with network reinforcement

BackgroundDrug repurposing has been motivated to ameliorate low probability of success in drug discovery. For the recent decade, many in silico attempts have received primary attention as a first step to alleviate the high cost and longevity. Such study has taken benefits of abundance, variety, and easy accessibility of pharmaceutical and biomedical data. Utilizing the research friendly environment, in this study, we propose a network-based machine learning algorithm for drug repurposing. Particularly, we show a framework on how to construct a drug network, and how to strengthen the network by employing multiple/heterogeneous types of data.ResultsThe proposed method consists of three steps. First, we construct a drug network from drug-target protein information. Then, the drug network is reinforced by utilizing drug-drug interaction knowledge on bioactivity and/or medication from literature databases. Through the enhancement, the number of connected nodes and the number of edges between them become more abundant and informative, which can lead to a higher probability of success of in silico drug repurposing. The enhanced network recommends candidate drugs for repurposing through drug scoring. The scoring process utilizes graph-based semi-supervised learning to determine the priority of recommendations.ConclusionsThe drug network is reinforced in terms of the coverage and connections of drugs: the drug coverage increases from 4738 to 5442, and the drug-drug associations as well from 808,752 to 982,361. Along with the network enhancement, drug recommendation becomes more reliable: AUC of 0.89 was achieved lifted from 0.79. For typical cases, 11 recommended drugs were shown for vascular dementia: amantadine, conotoxin GV, tenocyclidine, cycloeucine, etc.

Self-reinforced diffusion for graph-based semi-supervised learning

Graph construction is the essential component of graph-based semi-supervised learning. Gaussian kernel weighted graph is widely used in this field. The problem of using such kind of graphs is that they are susceptible to data noise. They cannot approximate the geodesic distance on the underlying manifold appropriately, especially for high-dimensional data. Diffusion process has shown its effectiveness of learning pair-wise affinities which is equivalent to a graph, due to its capability of revealing the geometry structure of the manifold. However, data density distribution and label information are ignored in this process, limiting its application for semi-supervised problems. To address these issues, we propose a variant of the diffusion process, named Self-Reinforced Diffusion, which can make use of the label information. As for data density distribution, we introduce an intuitive affinity term, called self-affinity, which can well approximate density distributions and can be directly diffused on the graph. Extensive experiments on noisy synthetic data and various real-world data have demonstrated the effectiveness of the proposed method on semi-supervised learning.

Fast Semisupervised Learning With Bipartite Graph for Large-Scale Data.

As the captured information in our real word is very scare and labeling sample is time cost and expensive, semisupervised learning (SSL) has an important application in computer vision and machine learning. Among SSL approaches, a graph-based SSL (GSSL) model has recently attracted much attention for high accuracy. However, for most traditional GSSL methods, the large-scale data bring higher computational complexity, which acquires a better computing platform. In order to dispose of these issues, we propose a novel approach, bipartite GSSL normalized (BGSSL-normalized) method, in this paper. This method consists of three parts. First, the bipartite graph between the original data and the anchor points is constructed, which is parameter-insensitive, scale-invariant, naturally sparse, and simple operation. Then, the label of the original data and anchors can be inferred through the graph. Besides, we extend our algorithm to handle out-of-sample for large-scale data by the inferred label of anchors, which not only retains good classification result but also saves a large amount of time. The computational complexity of BGSSL-normalized can be reduced to O(ndm+nm2) , which is a significant improvement compared with traditional GSSL methods that need O(n2d+n3) , where n , d , and m are the number of samples, features, and anchors, respectively. The experimental results on several publicly available data sets demonstrate that our approaches can achieve better classification accuracy with less time costs.

Joint graph based embedding and feature weighting for image classification

Recently, several inductive and flexible nonlinear data projection methods for graph-based semi-supervised learning were proposed. These state-of-the art techniques have a good performance. However, they have not taken into account the relevance of the original features in their model estimation. In this paper, we propose a joint graph-based embedding and feature weighting for getting a flexible and inductive nonlinear data representation on manifolds. The proposed criterion explicitly estimates the feature weights together with the projected data and the linear transformation such that data smoothness and large margins are achieved in the projection space. Moreover, the paper introduces a kernel variant of the model in order to get an inductive nonlinear embedding that is close to a real nonlinear subspace for a good approximation of the embedded data. The proposed frameworks can be seamlessly used in semi-supervised and supervised settings. The resulting optimization problems can be solved efficiently. The proposed embedding methods are evaluated on six public scene and face datasets. Experiments on image classification, in a semi-supervised setting, show that our proposed methods can have a performance that is better than that of many state-of-the-art methods including linear and nonlinear methods.

Fault Diagnosis in Photovoltaic Arrays Using GBSSL Method and Proposing a Fault Correction System

Nonlinear characteristics of solar cells and changes in environmental conditions, such as temperature, and in particular, the intensity of daytime irradiation, make it difficult to identify faults by the conventional means of protection. Therefore, a variety of machine learning techniques are proposed for fault detection in photovoltaic (PV) arrays. In this regard, classifying and identifying the location of a fault event is essential. In addition to fault recognition, selecting the method of fault correction is another issue to be addressed. However, there are scarce investigations in this field. In this paper, a comprehensive method for identifying, classifying, locating, and correcting faults is introduced. The proposed method is assessed with the expansion of the diagnostic space of the graph-based semisupervised learning algorithm and an increased number of class labels. After identifying the type and location of a fault, the system temporarily isolates the fault to function without interruption until it is fully corrected. The problem of overlapping cell data in normal and fault-prone modes is resolved by applying different methods of normalization. The results show that all faults including unlearned and learned in a wide range of environmental conditions, where possible PV arrays are experienced, are properly identified and corrected. Moreover, our studies demonstrate that the proposed system mitigates the output voltage variations over a fault-prone mode.

Large data and zero noise limits of graph-based semi-supervised learning algorithms

Abstract Scalings in which the graph Laplacian approaches a differential operator in the large graph limit are used to develop understanding of a number of algorithms for semi-supervised learning; in particular, the probit algorithm, level set and kriging methods. Both optimization and Bayesian approaches are considered, based around a regularizing quadratic form found from an affine transformation of the Laplacian, raised to a possibly fractional, exponent. Conditions on the parameters defining this quadratic form are identified under which well-defined limiting continuum analogues of the optimization and Bayesian semi-supervised learning problems may be found, thereby shedding light on the design of algorithms in the large graph setting. The large graph limits of the optimization formulations are tackled through Γ-convergence, using the recently introduced T L p metric. The small labeling noise limits of the Bayesian formulations are also identified, and contrasted with pre-existing harmonic function approaches to the problem.

A Graph-Based Semisupervised Deep Learning Model for PolSAR Image Classification

Aiming at improving the classification accuracy with limited numbers of labeled pixels in polarimetric synthetic aperture radar (PolSAR) image classification task, this paper presents a graph-based semisupervised deep learning model for PolSAR image classification. It models the PolSAR image as an undirected graph, where the nodes correspond to the labeled and unlabeled pixels, and the weighted edges represent similarities between the pixels. Upon the graph, we design an energy function incorporating a semisupervision term, a convolutional neural network (CNN) term, and a pairwise smoothness term. The employed CNN extracts abstract and data-driven polarimetric features and outputs class label predictions to the graph model. The semisupervision term enforces the category label constraints on the human-labeled pixels. The pairwise smoothness term encourages class label smoothness and the alignment of class label boundaries with the image edges. Starting from an initialized class label map generated based on $K$ -Wishart distribution hypothesis or superpixel segmentation of PauliRGB images, we iteratively and alternately optimize the defined energy function until it converges. We conducted experiments on two real benchmark PolSAR images, and extensive experiments demonstrated that our approach achieved the state-of-the-art results for PolSAR image classification.

A Sampling Theory Perspective of Graph-Based Semi-Supervised Learning

Graph-based methods have been quite successful in solving unsupervised and semi-supervised learning problems, as they provide a means to capture the underlying geometry of the dataset. It is often desirable for the constructed graph to satisfy two properties: first, data points that are similar in the feature space should be strongly connected on the graph, and second, the class label information should vary smoothly with respect to the graph, where smoothness is measured using the spectral properties of the graph Laplacian matrix. Recent works have justified some of these smoothness conditions by showing that they are strongly linked to the semi-supervised smoothness assumption and its variants. In this work, we reinforce this connection by viewing the problem from a graph sampling theoretic perspective, where class indicator functions are treated as bandlimited graph signals (in the eigenvector basis of the graph Laplacian) and label prediction as a bandlimited reconstruction problem. Our approach involves analyzing the bandwidth of class indicator signals generated from statistical data models with separable and nonseparable classes. These models are quite general and mimic the nature of most real-world datasets. Our results show that in the asymptotic limit, the bandwidth of any class indicator is also closely related to the geometry of the dataset. This allows one to theoretically justify the assumption of bandlimitedness of class indicator signals, thereby providing a sampling theoretic interpretation of graph-based semi-supervised classification.

Joint sparse graph and flexible embedding for graph-based semi-supervised learning

This letter introduces a framework for graph-based semi-supervised learning by estimating a flexible non-linear projection and its linear regression model. Unlike existing works, the proposed framework jointly estimates the graph structure, the non-linear projection, and the linear regression model. By adopting this joint estimation an overall optimality can be reached. A series of experiments are conducted on five image datasets in order to compare the proposed method with some state-of-art semi-supervised methods. This evaluation demonstrates the effectiveness of the proposed embedding method. These experiments show the superiority of the proposed framework over the joint estimation of the graph and soft labels.

Multi Similarity Metric Fusion in Graph-Based Semi-Supervised Learning

In semi-supervised label propagation (LP), the data manifold is approximated by a graph, which is considered as a similarity metric. Graph estimation is a crucial task, as it affects the further processes applied on the graph (e.g., LP, classification). As our knowledge of data is limited, a single approximation cannot easily find the appropriate graph, so in line with this, multiple graphs are constructed. Recently, multi-metric fusion techniques have been used to construct more accurate graphs which better represent the data manifold and, hence, improve the performance of LP. However, most of these algorithms disregard use of the information of label space in the LP process. In this article, we propose a new multi-metric graph-fusion method, based on the Flexible Manifold Embedding algorithm. Our proposed method represents a unified framework that merges two phases: graph fusion and LP. Based on one available view, different simple graphs were efficiently generated and used as input to our proposed fusion approach. Moreover, our method incorporated the label space information as a new form of graph, namely the Correlation Graph, with other similarity graphs. Furthermore, it updated the correlation graph to find a better representation of the data manifold. Our experimental results on four face datasets in face recognition demonstrated the superiority of the proposed method compared to other state-of-the-art algorithms.

Semi-Supervised Classification With Graph Structure Similarity and Extended Label Propagation

Graph-based semi-supervised learning (GSSL) has attracted great attention over the past decade. However, there are still several open problems: (1) how to construct a graph that effectively represents the underlying structure of data and (2) how to incorporate label information of the labeled samples into a procedure of label propagation. Our solution mainly focuses on two aspects: (1) we propose a new graph construction technique by fusing local and global structural similarity (FLGSS). Based on an initial graph structure such as K-nearest neighbors (KNN), we utilize different types of link prediction algorithms to extract local and global graph structure information. These two types of structure information are fused into a graph structure that enhances the ability to represent the data correlation. (2) By incorporating the label correlation with feature similarity of samples, we propose an extended label propagation algorithm (ELP). Through experiments on three different types of datasets, it is shown that our method outperforms other widely used graph construction methods. The extended label inference algorithm achieves better classification results than some state-of-the-art methods. The proposed FLGSS method starts from KNN graph and two link prediction algorithms are performed to construct the graph. With the time complexity analysis, we theoretically deduce that the time complexity of FLGSS is not beyond that of KNN. Meanwhile, the time complexity of ELP remains the same as that of the traditional LP algorithm.

A semi-supervised approach of graph-based with local and global consistency

An approach of graph-based semi-supervised learning is proposed that consider the local and global consistency of data. Like most graph-based semi-supervised learning, the algorithm mainly focused on two key issues: the graph construction and the manifold regularisation framework. In the graph construction, these labelled and unlabelled data are represented as vertices encoding edges weights with the similarity of instances, which means that not only the local geometry information but also the class information are utilised. In manifold regularisation framework, the cost function contains two terms of smoothness constraint and fitting constraint, it is sufficiently smooth with respect to the intrinsic structure revealed by known labelled and unlabelled instances. Specifically, we design the algorithm that uses the normalised Laplacian eigenvectors, which ensure the cost function can converge to closed form expression and then, we provide the convergence proof. Experimental results on various datasets and entity relationship classification show that the proposed algorithm mostly outperforms the popular classification algorithm.

Semi-supervised dimensionality reduction based on local estimation error

The construction of a graph is extremely important in graph-based semi-supervised learning. However, it is unstable by virtue of sensitivity to the selection of neighbourhood parameter and inaccuracy of the edge weights. Inspired by the good performance of the local learning method, this paper proposes a semi-supervised dimensionality reduction based on local estimation error (LEESSDR) algorithm by utilising local learning projections (LLP) to semi-supervised dimensionality reduction. The algorithm sets the edge weights through minimising the local estimation error and can effectively preserve the global geometric structure as well as the local one of the data. Since LLP does not require its input space to be locally linear, even if it is nonlinear, LLP maps it to the feature space by using kernel functions and then obtains its local estimation error in the feature space. The effectiveness of the proposed method is verified on two popular face databases with promising classification accuracy and favourable robustness.

Scene Recognition via Semi-Supervised Multi-Feature Regression

With the development of visual sensor equipment (e.g., personal smart phones, vehicle cameras, surveillance videos and camcorders), scene recognition technology has attracted much attention due to its latent applications in visual surveillance, intelligent traffic and aerial remote sensing. Although some progress has been made in the field of scene recognition in recent years, the complexity of scene images and the inadequate numbers of labeled data pose challenges in this area. Hence, to effectively fuse the multiple features of each image and employ the information of both labeled and unlabeled images for scene recognition, we proposed a semi-supervised multi-feature regression (SSMFR) model in this paper. The SSMFR model possesses three advantages. First, the model propagates the labels of labeled data to unlabeled data by utilizing graph-based semi-supervised learning techniques so that both the information regarding unlabeled data and labeled data can be exploited to gain better performance. Second, SSMFR employs multiple graphs to characterize the structures of multiple feature spaces and adaptively assigns the weight to different graphs. Therefore, SSMFR can efficiently preserve the manifold structure of samples in each feature space and adequately exploit the complementary information of multiple features. Moreover, SSMFR adopts a $l_{2,1}$ -norm constraint to learn a sparse and robust classifier for scene recognition. To solve the SSMFR model, we proposed a simple and efficient iterative update optimization scheme. Finally, we also proved the convergence of SSMFR by theoretical analysis and experiments. Experiments were conducted on several benchmark scene datasets, and the experimental results demonstrated that the proposed SSMFR model can obtain better performance for scene recognition than some other state-of-the-art algorithms.

The game theoretic p-Laplacian and semi-supervised learning with few labels

We study the game theoretic p-Laplacian for semi-supervised learning on graphs, and show that it is well-posed in the limit of finite labeled data and infinite unlabeled data. In particular, we show that the continuum limit of graph-based semi-supervised learning with the game theoretic p-Laplacian is a weighted version of the continuous p-Laplace equation. We also prove that solutions to the graph p-Laplace equation are approximately Hölder continuous with high probability. Our proof uses the viscosity solution machinery and the maximum principle on a graph.