Semi-supervised Classification Problem Research Articles

Transductive Semi-Supervised Metric Network for Reject Inference in Credit Scoring

Credit scoring is an essential technique for credit risk management in the financial industry. However, most credit scoring models face the challenge of reject inference, which refers to the lack of post-loan performance data for rejected applicants, leading to sample selection bias and inaccurate credit assessment. Traditional credit scoring methods tackle this issue by assuming that the missing labels for rejected samples are missing at random (MAR) and by measuring sample similarity directly in the original feature space. Nevertheless, these strategies are not suitable for real-world business scenarios. Inspired by metric learning and transductive learning, we propose a novel credit scoring model called transductive semi-supervised metric network (TSSMN), which formalizes reject inference as a semi-supervised binary classification problem with the prior assumption of missing not at random (MNAR). TSSMN consists of two interconnected modules: the embedding metric network (EMN) that maps samples from the original feature space to the metric space for similarity measurement, and the transductive propagation network (TPN) that performs label propagation based on sample similarity. We evaluate TSSMN on a real-world credit dataset and compare it with traditional credit scoring methods. The results indicate that TSSMN can overcome sample selection bias and more accurately classify credit applicants. Therefore, TSSMN has the potential to enhance credit risk assessment in real-world business scenarios.

NodeMixup: Tackling Under-Reaching for Graph Neural Networks

Graph Neural Networks (GNNs) have become mainstream methods for solving the semi-supervised node classification problem. However, due to the uneven location distribution of labeled nodes in the graph, labeled nodes are only accessible to a small portion of unlabeled nodes, leading to the under-reaching issue. In this study, we firstly reveal under-reaching by conducting an empirical investigation on various well-known graphs. Then, we demonstrate that under-reaching results in unsatisfactory distribution alignment between labeled and unlabeled nodes through systematic experimental analysis, significantly degrading GNNs' performance. To tackle under-reaching for GNNs, we propose an architecture-agnostic method dubbed NodeMixup. The fundamental idea is to (1) increase the reachability of labeled nodes by labeled-unlabeled pairs mixup, (2) leverage graph structures via fusing the neighbor connections of intra-class node pairs to improve performance gains of mixup, and (3) use neighbor label distribution similarity incorporating node degrees to determine sampling weights for node mixup. Extensive experiments demonstrate the efficacy of NodeMixup in assisting GNNs in handling under-reaching. The source code is available at https://github.com/WeigangLu/NodeMixup.

Multimodal graph learning based on 3D Haar semi-tight framelet for student engagement prediction

With the increasing availability of multimodal educational data, there is a growing need to effectively integrate and exploit multiple data sources to enhance student engagement prediction accuracy. In this work, we propose a framework that combines multimodal data, including visual, textual and acoustic modalities that reflect the students’ personalities, their demographic information, their learning behavior and attention, with graph learning techniques. Specifically, 3D Haar semi-tight framelet transforms are developed to capture the inter-modal relationships and model the complex interactions within the multimodal data. Subsequently, we introduce a novel module for adaptive graph structure learning based on the spectrum of multimodal data, which takes into consideration the distinct contributions of low-pass and high-pass framelet coefficients by adaptively weighing their impact. By addressing a standard semi-supervised node classification problem, we successfully achieve the objective of student engagement prediction. The experiment evaluations on a real-world educational dataset demonstrate the effectiveness of the proposed approach, achieving superior performance compared to state-of-the-art methods. Our experimental studies demonstrate the importance of multimodal graph learning in accurately predicting student engagement and its potential to enhance educational outcomes.

DAHGN: Degree-Aware Heterogeneous Graph Neural Network

In recent years, Graph Neural Networks (GNNs), an emerging technology for learning from graph-structured data, have attracted much attention. Despite the widespread applications of GNNs across diverse domains, recent studies have shown that insufficient information aggregation for low-degree nodes (i.e., tail nodes) leads to inferior performance of GNNs. While numerous recent studies have aimed to address the issue of degree bias in homogeneous graphs, there are limited studies on analyzing such unfairness for heterogeneous information networks. In this article, we propose a novel semi-supervised contrastive learning framework named Degree-Aware Heterogeneous Graph Neural Network (DAHGN), which achieves promising performance by alleviating the degree bias. In brief, we first construct two contrastive graph views, one from a heterogeneous graph that focuses on capturing complementary information to enhance the node representations and one from an extracted homogeneous subgraph that focuses on applying different strategies to low- and high- degree nodes to eliminate the degree bias. We then leverage the label learned from the heterogeneous view to reconstruct the positive node pairs in the contrastive learning framework. Finally, we combine the task loss (semi-supervised) and the contrastive loss to optimize the model performance. We conduct extensive experiments on benchmark datasets to validate the effectiveness of our model, and the results demonstrate that the DAHGN outperforms a wide range of state-of-the-art methods in the semi-supervised node classification problem.

Towards Semi-Supervised Universal Graph Classification

Graph neural networks have pushed state-of-the-arts in graph classifications recently. Typically, these methods are studied within the context of supervised end-to-end training, which necessities copious task-specific labels. However, in real-world circumstances, labeled data could be limited, and there could be a massive corpus of unlabeled data, even from unknown classes as a complementary. Towards this end, we study the problem of semi-supervised universal graph classification, which not only identifies graph samples which do not belong to known classes, but also classifies the remaining samples into their respective classes. This problem is challenging due to a severe lack of labels and potential class shifts. In this paper, we propose a novel graph neural network framework named UGNN, which makes the best of unlabeled data from the subgraph perspective. To tackle class shifts, we estimate the certainty of unlabeled graphs using multiple subgraphs, which facilities the discovery of unlabeled data from unknown categories. Moreover, we construct semantic prototypes in the embedding space for both known and unknown categories and utilize posterior prototype assignments inferred from the Sinkhorn-Knopp algorithm to learn from abundant unlabeled graphs across different subgraph views. Extensive experiments on six datasets verify the effectiveness of UGNN in different settings.

Knowledge distillation-driven semi-supervised multi-view classification

Semi-supervised multi-view classification is a critical research topic that leverages the discrepancy between different views and limited annotated samples for pattern recognition in computer vision. However, it encounters a significant challenge: obtaining comprehensive discriminative representations with a scarcity of labeled samples. Although existing methods aim to learn discriminative features by fusing multi-view information, a significant challenge persists due to the difficulty of transferring complementary information and fusing multiple views with limited supervised information. In response to this challenge, this work introduces an innovative algorithm that integrates Self-Knowledge Distillation (Self-KD) to facilitate semi-supervised multi-view classification. Initially, we employ a view-specific feature extractor for each view to learn discriminative representations. Subsequently, we introduce a self-distillation module to drive information interaction across multiple views, enabling mutual learning and refinement of multi-view unified and specific representations. Moreover, we introduce a class-aware contrastive module to alleviate confirmation bias stemming from noise in the generated pseudo-labels during knowledge distillation. To the best of our knowledge, this is the first attempt to extend Self-KD to address semi-supervised multi-view classification problems. Extensive experimental results validate the efficiency of this approach in semi-supervised multi-view classification compared to existing state-of-the-art methods.

Commonsense knowledge powered heterogeneous graph attention networks for semi-supervised short text classification

In real-world scenarios, considerable human power and expert knowledge are required to label data. Therefore, solving short text classification problems in a semi-supervised manner is a good method. Existing graph-based semi-supervised short text classification models can achieve good classification performance. However, these models do not introduce some prior knowledge related to short texts, such as commonsense knowledge and concepts, from the knowledge bases to alleviate the data sparsity of short texts, which limits the improvement of classification performance. To overcome the above limitations, this paper addresses the problem of semi-supervised short text classification by proposing a novel model called the Commonsense Knowledge-Powered Heterogeneous Graph Attention Network (CSK-HGAT). This model mines the short text-related commonsense knowledge from the definitions corresponding to extracted short text keywords and obtains the short text conceptual information from the existing knowledge bases by conceptualizing the identified short text entities. Next, the embeddings are generated separately for short texts, commonsense knowledge, and concepts. Then, a heterogeneous information network (HIN) containing diverse knowledge is constructed by using the semantic relations among the short texts, commonsense knowledge and the concepts. Finally, this HIN is embedded into the heterogeneous graph attention network model (containing node-level and type-level attentions) to fully utilize the mined prior knowledge (i.e., commonsense knowledge and concepts) to achieve effective semi-supervised short text classification. Experimental results on four benchmark datasets show that the proposed CSK-HGAT model outperforms state-of-the-art semi-supervised short text classification models in terms of classification accuracy.

Tri-Training Algorithm for Adaptive Nearest Neighbor Density Editing and Cross Entropy Evaluation.

Tri-training expands the training set by adding pseudo-labels to unlabeled data, which effectively improves the generalization ability of the classifier, but it is easy to mislabel unlabeled data into training noise, which damages the learning efficiency of the classifier, and the explicit decision mechanism tends to make the training noise degrade the accuracy of the classification model in the prediction stage. This study proposes the Tri-training algorithm for adaptive nearest neighbor density editing and cross-entropy evaluation (TTADEC), which is used to reduce the training noise formed during the classifier iteration and to solve the problem of inaccurate prediction by explicit decision mechanism. First, the TTADEC algorithm uses the nearest neighbor editing to label high-confidence samples. Then, combined with the relative nearest neighbor to define the local density of samples to screen the pre-training samples, and then dynamically expand the training set by adaptive technique. Finally, the decision process uses cross-entropy to evaluate the completed base classifier of training and assign appropriate weights to it to construct a decision function. The effectiveness of the TTADEC algorithm is verified on the UCI dataset, and the experimental results show that compared with the standard Tri-training algorithm and its improvement algorithm, the TTADEC algorithm has better classification performance and can effectively deal with the semi-supervised classification problem where the training set is insufficient.

Laplacian Lp norm least squares twin support vector machine

Semi-supervised learning has become a hot learning framework, where large amounts of unlabeled data and small amounts of labeled data are available during the training process. The recently proposed Laplacian least squares twin support vector machine (Lap-LSTSVM) is an excellent tool to solve the semi-supervised classification problem. Motivated by the success of Lap-LSTSVM, in this paper, we propose a novel Laplacian Lp norm least squares twin support vector machine (Lap-LpLSTSVM). There are several advantages of our proposed method: (1) The performance of our proposed Lap-LpLSTSVM can be improved by the adjustability of the value of p. (2) The introduced Lp norm graph regularization term can efficiently exploit the geometric information embedded in the data. (3) An efficient iterative strategy is employed to solve the optimization problem. Besides, to demonstrate that our proposed method can make use of unlabeled data effectively, least squares twin support vector machine (LSTSVM) which only uses the same labeled data is used to compare with our proposed method. The experimental results on both synthetic and real-world datasets show that our proposed method outperforms other state-of-the-art methods and can also deal with noisy datasets.

An Improved Algorithm of Drift Compensation for Olfactory Sensors

This research mainly studies the semi-supervised learning algorithm of different domain data in machine olfaction, also known as sensor drift compensation algorithm. Usually for this kind of problem, it is difficult to obtain better recognition results by directly using the semi-supervised learning algorithm. For this reason, we propose a domain transformation semi-supervised weighted kernel extreme learning machine (DTSWKELM) algorithm, which converts the data through the domain and uses SWKELM algorithmic classification to transform the semi-supervised classification problem of different domain data into a semi-supervised classification problem of the same domain data.

Learning From Negative Links.

Recently, graph convolutional networks (GCNs) and their variants have achieved remarkable successes for the graph-based semisupervised node classification problem. With a GCN, node features are locally smoothed based on the information aggregated from their neighborhoods defined by the graph topology. In most of the existing methods, the graph typologies only contain positive links which are deemed as descriptions for the feature similarity of connected nodes. In this article, we develop a novel GCN-based learning framework that improves the node representation inference capability by including negative links in a graph. Negative links in our method define the inverse correlations for the nodes connected by them and are adaptively generated through a neural-network-based generation model. To make the generated negative links beneficial for the classification performance, this negative link generation model is jointly optimized with the GCN used for class inference through our designed training algorithm. Experiment results show that the proposed learning framework achieves better or matched performance compared to the current state-of-the-art methods on several standard benchmark datasets.

Study-GNN: A Novel Pipeline for Student Performance Prediction Based on Multi-Topology Graph Neural Networks

Student performance prediction has attracted increasing attention in the field of educational data mining, or more broadly, intelligent education or “AI + education”. Accurate performance prediction plays a significant role in solving the problem of a student dropping out, promoting personalized learning and improving teaching efficiency, etc. Traditional student performance prediction methods usually ignore the potential (underlying) relationship among students. In this paper, we use graph structure to reflect the students’ relationships and propose a novel pipeline for student performance prediction based on newly-developed multi-topology graph neural networks (termed MTGNN). In particular, we propose various ways for graph construction based on similarity learning using different distance metrics. Based on the multiple graphs of different topologies, we design an MTGNN module, as a key module in the pipeline, to deal with the semi-supervised node classification problem where each node represents a student (and the node label is the student’s performance, e.g., Pass/Fail/Withdrawal). An attention-based method is developed to produce the unified graph representation in MTGNN. The effectiveness of the proposed pipeline is verified in a case study, where a real-world educational dataset and several existing approaches are used for performance comparison. The experiment results show that, compared with some traditional machine learning methods and the vanilla graph convolutional network with only a single graph topology, our proposed pipeline works effectively and favorably in student performance prediction.

Hypergraph Convolution on Nodes-Hyperedges Network for Semi-Supervised Node Classification

Hypergraphs have shown great power in representing high-order relations among entities, and lots of hypergraph-based deep learning methods have been proposed to learn informative data representations for the node classification problem. However, most of these deep learning approaches do not take full consideration of either the hyperedge information or the original relationships among nodes and hyperedges. In this article, we present a simple yet effective semi-supervised node classification method named Hypergraph Convolution on Nodes-Hyperedges network, which performs filtering on both nodes and hyperedges as well as recovers the original hypergraph with the least information loss. Instead of only reducing the cross-entropy loss over the labeled samples as most previous approaches do, we additionally consider the hypergraph reconstruction loss as prior information to improve prediction accuracy. As a result, by taking both the cross-entropy loss on the labeled samples and the hypergraph reconstruction loss into consideration, we are able to achieve discriminative latent data representations for training a classifier. We perform extensive experiments on the semi-supervised node classification problem and compare the proposed method with state-of-the-art algorithms. The promising results demonstrate the effectiveness of the proposed method.

Semi-supervised classification via full-graph attention neural networks

Graph neural networks (GNNs) leverage graph convolutions or their approximations to extract features of nodes from graph-structured data. Nevertheless, these methods only combine information from nodes’ neighborhoods, without taking into account the impact of other nodes outside neighborhoods. To address the shortcomings, we present full-graph attention neural networks (FGANNs), novel neural network architectures that consider the impact of all nodes when performing self-attention, leveraging masked attention to enable (implicitly) specifying different weights to different nodes in a neighborhood. Under such circumstances, we address several important challenges of spectral-based graph neural networks simultaneously, and make FGANN readily available to semi-supervised classification problems. Extensive experiments on citation networks offer evidence that the proposed approach outperforms state-of-the-art methods.

Seeded random walk for multi-view semi-supervised classification

Recently, multi-view learning has captured widespread attention in the machine learning area, yet it is still crucial and challenging to exploit beneficial patterns from multi-view data. Specifically, very limited work has been devoted to multi-view semi-supervised learning, where only a small number of labeled data points are available for model training. Therefore, a simple yet efficient seeded random walk scheme is proposed in this paper to address the multi-view semi-supervised classification problem, where known labeled data points serve as random seeds to be walked with certain probability. In this scheme, the semi-supervised classification indicator is obtained based primarily on an arrival probability and a reward matrix, which are computed by leveraging an initial distribution from some random seeds. Besides, theoretical analyses are then provided to indicate a connection of the proposed model with the existing manifold ranking method. Finally, comprehensive experiments on eight publicly available data sets demonstrate the superiority of the proposed model against compared state-of-the-art semi-supervised methods and fully supervised classifiers. Furthermore, experimental results also suggest that the proposed method comes with positive robustness and promising generalization capability in terms of data classification.

Accelerated manifold embedding for multi-view semi-supervised classification

Multi-view semi-supervised learning has gained much attention since a great number of unlabeled multi-view data are easy to obtain while few labeled data are available. Accordingly, how to utilize the relationship between labeled and unlabeled data is of significance in multi-view semi-supervised learning. In this paper, we propose an auto-weighted manifold embedding model to address multi-view semi-supervised classification problems, where only a small percentage of labeled data points are used for model training. In the proposed model, data points close to each other in the feature space will be assigned to similar classes in the label space through manifold embedding. Accordingly, the class information of labeled data will be employed in the prediction process for unlabeled data. Moreover, an optimal weight for each view of multi-view data is learned automatically, which enables an encouraging fusion quality for multi-view manifold embedding. To further speed up the calculation process and reduce the computational complexity of the proposed model, an effective accelerated auto-weighted manifold embedding scheme is developed. Besides, theoretical analyses are then provided to indicate a tight approximation bound for the primary manifold embedding method, while the accelerated method is designed by an order decrease of magnitude of computational complexity. Finally, comprehensive experiments on eight publicly available data sets demonstrate the superiority of the proposed models over compared state-of-the-art semi-supervised methods and fully supervised classifiers. Furthermore, the experimental results are suggestive of positive robustness and promising generalization capacity of the proposed methods.

Covariance and correlation measures on a graph in a generalized bag-of-paths formalism

Abstract This work derives closed-form expressions computing the expectation of co-presence and of number of co-occurrences of nodes on paths sampled from a network according to general path weights (a bag of paths). The underlying idea is that two nodes are considered as similar when they often appear together on (preferably short) paths of the network. The different expressions are obtained for both regular and hitting paths and serve as a basis for computing new covariance and correlation measures between nodes, which are valid positive semi-definite kernels on a graph. Experiments on semi-supervised classification problems show that the introduced similarity measures provide competitive results compared to other state-of-the-art distance and similarity measures between nodes.

A novel semi-supervised ensemble algorithm using a performance-based selection metric to non-stationary data streams

In this article, we consider the semi-supervised data stream classification problems. Most of the semi-supervised learning algorithms suffer from a proper selection metric to select from the newly-labeled data points through the training procedure. These approaches mainly employ the probability estimation of the underlying base learners to their predictions as a selection metric, which is not optimal in many cases. Handling different kinds of concept drifts is another issue in data streams. Considering these issues, we propose a novel Semi-Supervised Ensemble algorithm using a Performance-Based Selection metric to data streams, named SSE-PBS. The proposed selection metric is based on a pseudo-accuracy and energy regularization factor. We show that SSE-PBS improves classification performance and handles different kinds of concept drifts. The proposed algorithm can also employ any kind of incremental base learners. In the experiments, we report the results of two base learners on synthetic and real-world datasets. The experiments show that SSE-PBS significantly improves the classification performance of the used underlying base learners. Furthermore, we compare the results to the state-of-the-art supervised and semi-supervised approaches in data streams. The results further show that SSE-PBS outperforms the other methods when there is a small portion of labeled instances.

Semi-supervised regression using diffusion on graphs

In real-world machine learning applications, unlabeled training data are readily available, but labeled data are expensive and hard to obtain. Therefore, semi-supervised learning algorithms have gathered much attention. Previous studies in this area mainly focused on a semi-supervised classification problem, whereas semi-supervised regression has received less attention. In this paper, we proposed a novel semi-supervised regression algorithm using heat diffusion with a boundary-condition that guarantees a closed-form solution. Experiments from artificial and real datasets from business, biomedical, physical, and social domain show that the boundary-based heat diffusion method can effectively outperform the top state of the art methods.

Hypergraph-Structured Autoencoder for Unsupervised and Semisupervised Classification of Hyperspectral Image

Deep neural networks have gained increasing interest in hyperspectral image (HSI) processing. However, prior arts often neglect the high-order correlation among data points, failing to capture intraclass variations. In this letter, we present a unified neural network framework, termed as hypergraph-structured autoencoder (HyperAE), to leverage the high-order relationship among data and learn robust deep representation for downstream tasks. Technically, the proposed method adopts a deep autoencoder regularized by hypergraph structure as the backbone network, which is jointly trained with a task-specific branch, resulting in a multitask architecture. We separately combine the subspace clustering model and the softmax classifier into the HyperAE to deal with HSI unsupervised and semisupervised classification problems. Benefiting from the hypergraph, HyperAE endows traditional networks with the capacity of preserving the high-order structured information. We evaluate the proposed methods on three benchmarking HSI data sets, demonstrating that the proposed HyperAE dramatically outperforms many existing methods with significant margins in both unsupervised and semisupervised HSI classification problems.