Semi-supervised Learning Method Research Articles

ConvNeXt-MHC: improving MHC-peptide affinity prediction by structure-derived degenerate coding and the ConvNeXt model.

Peptide binding to major histocompatibility complex (MHC) proteins plays a critical role in T-cell recognition and the specificity of the immune response. Experimental validation such peptides is extremely resource-intensive. As a result, accurate computational prediction of binding peptides is highly important, particularly in the context of cancer immunotherapy applications, such as the identification of neoantigens. In recent years, there is a significant need to continually improve the existing prediction methods to meet the demands of this field. We developed ConvNeXt-MHC, a method for predicting MHC-I-peptide binding affinity. It introduces a degenerate encoding approach to enhance well-established panspecific methods and integrates transfer learning and semi-supervised learning methods into the cutting-edge deep learning framework ConvNeXt. Comprehensive benchmark results demonstrate that ConvNeXt-MHC outperforms state-of-the-art methods in terms of accuracy. We expect that ConvNeXt-MHC will help us foster new discoveries in the field of immunoinformatics in the distant future. We constructed a user-friendly website at http://www.combio-lezhang.online/predict/, where users can access our data and application.

Effects of non-landslide sampling strategies on machine learning models in landslide susceptibility mapping

This study aims to explore the effects of different non-landslide sampling strategies on machine learning models in landslide susceptibility mapping. Non-landslide samples are inherently uncertain, and the selection of non-landslide samples may suffer from issues such as noisy or insufficient regional representations, which can affect the accuracy of the results. In this study, a positive-unlabeled (PU) bagging semi-supervised learning method was introduced for non-landslide sample selection. In addition, buffer control sampling (BCS) and K-means (KM) clustering were applied for comparative analysis. Based on landslide data from Qiaojia County, Yunnan Province, China, collected in 2014, three machine learning models, namely, random forest, support vector machine, and CatBoost, were used for landslide susceptibility mapping. The results show that the quality of samples selected using different non-landslide sampling strategies varies significantly. Overall, the quality of non-landslide samples selected using the PU bagging method is superior, and this method performs best when combined with CatBoost for predicting (AUC = 0.897) landslides in very high and high susceptibility zones (82.14%). Additionally, the KM results indicated overfitting, displaying high accuracy for validation but poor statistical outcomes for zoning. The BCS results were the worst.

FedDUS: Lung tumor segmentation on CT images through federated semi-supervised with dynamic update strategy

Background and ObjectiveLung tumor annotation is a key upstream task for further diagnosis and prognosis. Although deep learning techniques have promoted automation of lung tumor segmentation, there remain challenges impeding its application in clinical practice, such as a lack of prior annotation for model training and data-sharing among centers. MethodsIn this paper, we use data from six centers to design a novel federated semi-supervised learning (FSSL) framework with dynamic model aggregation and improve segmentation performance for lung tumors. To be specific, we propose a dynamically updated algorithm to deal with model parameter aggregation in FSSL, which takes advantage of both the quality and quantity of client data. Moreover, to increase the accessibility of data in the federated learning (FL) network, we explore the FAIR data principle while the previous federated methods never involve. ResultThe experimental results show that the segmentation performance of our model in six centers is 0.9348, 0.8436, 0.8328, 0.7776, 0.8870 and 0.8460 respectively, which is superior to traditional deep learning methods and recent federated semi-supervised learning methods. ConclusionThe experimental results demonstrate that our method is superior to the existing FSSL methods. In addition, our proposed dynamic update strategy effectively utilizes the quality and quantity information of client data and shows efficiency in lung tumor segmentation. The source code is released on (https://github.com/GDPHMediaLab/FedDUS).

RoPDA: Robust Prompt-Based Data Augmentation for Low-Resource Named Entity Recognition

Data augmentation has been widely used in low-resource NER tasks to tackle the problem of data sparsity. However, previous data augmentation methods have the disadvantages of disrupted syntactic structures, token-label mismatch, and requirement for external knowledge or manual effort. To address these issues, we propose Robust Prompt-based Data Augmentation (RoPDA) for low-resource NER. Based on pre-trained language models (PLMs) with continuous prompt, RoPDA performs entity augmentation and context augmentation through five fundamental augmentation operations to generate label-flipping and label-preserving examples. To optimize the utilization of the augmented samples, we present two techniques: self-consistency filtering and mixup. The former effectively eliminates low-quality samples with a bidirectional mask, while the latter prevents performance degradation arising from the direct utilization of labelflipping samples. Extensive experiments on three popular benchmarks from different domains demonstrate that RoPDA significantly improves upon strong baselines, and also outperforms state-of-the-art semi-supervised learning methods when unlabeled data is included.

Semi-supervised Class-Agnostic Motion Prediction with Pseudo Label Regeneration and BEVMix

Class-agnostic motion prediction methods aim to comprehend motion within open-world scenarios, holding significance for autonomous driving systems. However, training a high-performance model in a fully-supervised manner always requires substantial amounts of manually annotated data, which can be both expensive and time-consuming to obtain. To address this challenge, our study explores the potential of semi-supervised learning (SSL) for class-agnostic motion prediction. Our SSL framework adopts a consistency-based self-training paradigm, enabling the model to learn from unlabeled data by generating pseudo labels through test-time inference. To improve the quality of pseudo labels, we propose a novel motion selection and re-generation module. This module effectively selects reliable pseudo labels and re-generates unreliable ones. Furthermore, we propose two data augmentation strategies: temporal sampling and BEVMix. These strategies facilitate consistency regularization in SSL. Experiments conducted on nuScenes demonstrate that our SSL method can surpass the self-supervised approach by a large margin by utilizing only a tiny fraction of labeled data. Furthermore, our method exhibits comparable performance to weakly and some fully supervised methods. These results highlight the ability of our method to strike a favorable balance between annotation costs and performance. Code will be available at https://github.com/kwwcv/SSMP.

Knowledge Guided Semi-supervised Learning for Quality Assessment of User Generated Videos

Perceptual quality assessment of user generated content (UGC) videos is challenging due to the requirement of large scale human annotated videos for training. In this work, we address this challenge by first designing a self-supervised Spatio-Temporal Visual Quality Representation Learning (ST-VQRL) framework to generate robust quality aware features for videos. Then, we propose a dual-model based Semi Supervised Learning (SSL) method specifically designed for the Video Quality Assessment (SSL-VQA) task, through a novel knowledge transfer of quality predictions between the two models. Our SSL-VQA method uses the ST-VQRL backbone to produce robust performances across various VQA datasets including cross-database settings, despite being learned with limited human annotated videos. Our model improves the state-of-the-art performance when trained only with limited data by around 10%, and by around 15% when unlabelled data is also used in SSL. Source codes and checkpoints are available at https://github.com/Shankhanil006/SSL-VQA.

From Toxic to Trustworthy: Using Self-Distillation and Semi-supervised Methods to Refine Neural Networks

Despite the tremendous success of deep neural networks (DNNs) across various fields, their susceptibility to potential backdoor attacks seriously threatens their application security, particularly in safety-critical or security-sensitive ones. Given this growing threat, there is a pressing need for research into purging backdoors from DNNs. However, prior efforts on erasing backdoor triggers not only failed to withstand increasingly powerful attacks but also resulted in reduced model performance. In this paper, we propose From Toxic to Trustworthy (FTT), an innovative approach to eliminate backdoor triggers while simultaneously enhancing model accuracy. Following the stringent and practical assumption of limited availability of clean data, we introduce a self-attention distillation (SAD) method to remove the backdoor by aligning the shallow and deep parts of the network. Furthermore, we first devise a semi-supervised learning (SSL) method that leverages ubiquitous and available poisoned data to further purify backdoors and improve accuracy. Extensive experiments on various attacks and models have shown that our FTT can reduce the attack success rate from 97% to 1% and improve the accuracy of 4% on average, demonstrating its effectiveness in mitigating backdoor attacks and improving model performance. Compared to state-of-the-art (SOTA) methods, our FTT can reduce the attack success rate by 2 times and improve the accuracy by 5%, shedding light on backdoor cleansing.

ANEDL: Adaptive Negative Evidential Deep Learning for Open-Set Semi-supervised Learning

Semi-supervised learning (SSL) methods assume that labeled data, unlabeled data and test data are from the same distribution. Open-set semi-supervised learning (Open-set SSL) con- siders a more practical scenario, where unlabeled data and test data contain new categories (outliers) not observed in labeled data (inliers). Most previous works focused on out- lier detection via binary classifiers, which suffer from insufficient scalability and inability to distinguish different types of uncertainty. In this paper, we propose a novel framework, Adaptive Negative Evidential Deep Learning (ANEDL) to tackle these limitations. Concretely, we first introduce evidential deep learning (EDL) as an outlier detector to quantify different types of uncertainty, and design different uncertainty metrics for self-training and inference. Furthermore, we propose a novel adaptive negative optimization strategy, making EDL more tailored to the unlabeled dataset containing both inliers and outliers. As demonstrated empirically, our proposed method outperforms existing state-of-the-art methods across four datasets.

A General Model for Aggregating Annotations AcrossSimple, Complex, and Multi-object Annotation Tasks (Abstract Reprint)

Human annotations are vital to supervised learning, yet annotators often disagree on the correct label, especially as annotation tasks increase in complexity. A common strategy to improve label quality is to ask multiple annotators to label the same item and then aggregate their labels. To date, many aggregation models have been proposed for simple categorical or numerical annotation tasks, but far less work has considered more complex annotation tasks, such as those involving open-ended, multivariate, or structured responses. Similarly, while a variety of bespoke models have been proposed for specific tasks, our work is the first we are aware of to introduce aggregation methods that generalize across many, diverse complex tasks, including sequence labeling, translation, syntactic parsing, ranking, bounding boxes, and keypoints. This generality is achieved by applying readily available task-specific distance functions, then devising a task-agnostic method to model these distances between labels, rather than the labels themselves. This article presents a unified treatment of our prior work on complex annotation modeling and extends that work with investigation of three new research questions. First, how do complex annotation task and dataset properties impact aggregation accuracy? Second, how should a task owner navigate the many modeling choices in order to maximize aggregation accuracy? Finally, what tests and diagnoses can verify that aggregation models are specified correctly for the given data? To understand how various factors impact accuracy and to inform model selection, we conduct large-scale simulation studies and broad experiments on real, complex datasets. Regarding testing, we introduce the concept of unit tests for aggregation models and present a suite of such tests to ensure that a given model is not mis-specified and exhibits expected behavior. Beyond investigating these research questions above, we discuss the foundational concept and nature of annotation complexity, present a new aggregation model as a conceptual bridge between traditional models and our own, and contribute a new general semisupervised learning method for complex label aggregation that outperforms prior work.

Unknown-Aware Graph Regularization for Robust Semi-supervised Learning from Uncurated Data

Recent advances in semi-supervised learning (SSL) have relied on the optimistic assumption that labeled and unlabeled data share the same class distribution. However, this assumption is often violated in real-world scenarios, where unlabeled data may contain out-of-class samples. SSL with such uncurated unlabeled data leads training models to be corrupted. In this paper, we propose a robust SSL method for learning from uncurated real-world data within the context of open-set semi-supervised learning (OSSL). Unlike previous works that rely on feature similarity distance, our method exploits uncertainty in logits. By leveraging task-dependent predictions of logits, our method is capable of robust learning even in the presence of highly correlated outliers. Our key contribution is to present an unknown-aware graph regularization (UAG), a novel technique that enhances the performance of uncertainty-based OSSL frameworks. The technique addresses not only the conflict between training objectives for inliers and outliers but also the limitation of applying the same training rule for all outlier classes, which are existed on previous uncertainty-based approaches. Extensive experiments demonstrate that UAG surpasses state-of-the-art OSSL methods by a large margin across various protocols. Codes are available at https://github.com/heejokong/UAGreg.

Linear semantic transformation for semi-supervised medical image segmentation

Medical image segmentation is a focus research and foundation in developing intelligent medical systems. Recently, deep learning for medical image segmentation has become a standard process and succeeded significantly, promoting the development of reconstruction, and surgical planning of disease diagnosis. However, semantic learning is often inefficient owing to the lack of supervision of feature maps, resulting in that high-quality segmentation models always rely on numerous and accurate data annotations. Learning robust semantic representation in latent spaces remains a challenge. In this paper, we propose a novel semi-supervised learning framework to learn vital attributes in medical images, which constructs generalized representation from diverse semantics to realize medical image segmentation. We first build a self-supervised learning part that achieves context recovery by reconstructing space and intensity of medical images, which conduct semantic representation for feature maps. Subsequently, we combine semantic-rich feature maps and utilize simple linear semantic transformation to convert them into image segmentation. The proposed framework was tested using five medical segmentation datasets. Quantitative assessments indicate the highest scores of our method on IXI (73.78%), ScaF (47.50%), COVID-19-Seg (50.72%), PC-Seg (65.06%), and Brain-MR (72.63%) datasets. Finally, we compared our method with the latest semi-supervised learning methods and obtained 77.15% and 75.22% DSC values, respectively, ranking first on two representative datasets. The experimental results not only proved that the proposed linear semantic transformation was effectively applied to medical image segmentation, but also presented its simplicity and ease-of-use to pursue robust segmentation in semi-supervised learning. Our code is now open at: https://github.com/QingYunA/Linear-Semantic-Transformation-for-Semi-Supervised-Medical-Image-Segmentation.

Neural collapse inspired semi-supervised learning with fixed classifier

Pseudo-labeling-based approaches are gaining prominence in Semi-Supervised Learning (SSL). Recent studies have identified that the key bottleneck in this methodology is addressing insufficient, incorrect, and imbalanced pseudo-labels. In this paper, we argue that the intrinsic problem behind this bottleneck is classifier bias, i.e., the classifier's prototypes suffer from poor uniformity. Further, inspired by neural collapse that reveals an optimal structure under supervised training scenarios, we address this classifier bias by utilizing an offline simplex Equiangular Tight Frame (ETF) classifier with maximally and equally separated prototypes. During the training phase, we maintain the prototypes of the classifier as fixed and concentrate on refining the feature encoder. Specifically, we integrate a straightforward clustering-based pseudo-labeling strategy with information maximization for feature learning. In practice, the fixed ETF classifier prevents the model from falling into a detrimental cycle, where a biased classifier exacerbates misaligned features, further perpetuating this bias. Furthermore, the clustering-based pseudo-labeling strategy reduces the dependency on complex threshold-adjusting mechanisms and effectively navigates the quantity-quality trade-off that plagues existing SSL methods. Leveraging these methodologies, we develop a simple yet powerful approach, termed ETF-SSL. Extensive experiments across Image, Text, and Audio datasets demonstrate that ETF-SSL can achieve competitive or superior performance compared to existing approaches. This success highlights the benefits of using a fixed ETF classifier in SSL and points to promising directions for future research in this area. The code is available at: https://github.com/yichenwang231/ETFSSL.

Semi-Supervised Machine Learning Method for Predicting Observed Individual Risk Preference Using Gallup Data

Risk and uncertainty play a vital role in almost every significant economic decision, and an individual’s propensity to make riskier decisions also depends on various circumstances. This article aims to investigate the effects of social and economic covariates on an individual’s willingness to take general risks and extends the scope of existing works by using quantitative measures of risk-taking from the GPS and Gallup datasets (in addition to the qualitative measures used in the literature). Based on the available observed risk-taking data for one year, this article proposes a semi-supervised machine learning-based approach that can efficiently predict the observed risk index for those countries/individuals for years when the observed risk-taking index was not collected. We find that linear models are insufficient to capture certain patterns among risk-taking factors, and non-linear models, such as random forest regression, can obtain better root mean squared values than those reported in past literature. In addition to finding factors that agree with past studies, we also find that subjective well-being influences risk-taking behavior.

Semi-supervised nonnegative matrix factorization with label propagation and constraint propagation

Semi-supervised nonnegative matrix factorization (SNMF) improves the clustering effect of nonnegative matrix factorization (NMF) by integrating label information. However, as one of the most commonly used semi-supervised learning methods, label propagation algorithm (LP) has some limitations due to its heavy dependence on the predefined similarity matrix. To address this deficiency and effectively use limited supervisory information, a structural NMF algorithm with label propagation and constraint propagation (PNLP-SCNMF) is proposed in this paper, which simultaneously performs data representation and classification in a joint manner. Both positive label information and negative label information are introduced into the proposed method, and the factor matrix is constrained by label information to construct the tri-decomposition form. Constraint propagation algorithm (CPA) is used to correct the constructed similarity matrix, so that the low-dimensional subspace obtained by matrix decomposition can retain the geometric structure characteristics of the original sample space. In addition, an efficient iterative update optimization scheme is proposed to solve this objective function, and its convergence is proved theoretically. A large number of comparative experiments on nine real datasets show that the proposed method can consistently achieve higher clustering accuracy in image clustering tasks.

A survey on semi-supervised graph clustering

Semi-Supervised Graph Clustering (SSGC) has emerged as a pivotal field at the intersection of graph clustering and semi-supervised learning (SSL), offering innovative solutions to intricate data analysis problems. However, despite its significance and wide-ranging applications, there exists a notable void in the literature—a comprehensive survey specifically dedicated to SSGC techniques and their diverse applications remains conspicuously absent. Addressing this gap, this paper presents a systematic and comprehensive review of SSGC methodologies, spanning from well-established approaches to cutting-edge developments. Through a meticulous categorization, critical examination, and insightful discussion of these techniques, this survey not only illuminates the current landscape of SSGC but also identifies unexplored avenues for exploration and innovation. In this paper we present a comprehensive survey of conventional graph construction, graph clustering, SSL methods, evaluation metrics and their primary development process. We then provide a taxonomy of SSGC techniques based on their structures and principles and thoroughly analyze and discuss the related models. Furthermore, we review the applications of SSGC in various fields. Lastly, we summarize the limitations of current SSGC algorithms and discuss the future directions of the field.

Online semi-supervised active learning ensemble classification for evolving imbalanced data streams

Concept drift is a core challenge in classification tasks of data streams. Although many drift adaptation methods have been presented, most of them assume that labels of all data are available, which is impractical in many real-world applications. Additionally, the absence of label makes the imbalance ratio of an imbalanced data stream difficultly being obtained in time, providing the inaccurate guidance for resampling and causing poor generalization. To tackle the joint challenges, an online semi-supervised active learning method is proposed to classifier imbalanced data streams with concept drift. A newly-arrived data is first added to the sliding window, and then assigned a pseudo label in terms of its nearest cluster. Meanwhile, semi-supervised clustering algorithm offers its predicted label. Based on the above two predictive labels, cluster-based query strategy provides the criteria for the evaluation and selection of representative instances. More especially, the uncertainty and importance of instances are defined to synthetically evaluate its representativeness. After obtaining true labels of typical ones, ensemble classifier is updated by all instances in current sliding window. Experimental results on 13 synthetic and real data streams indicate that the proposed method outperforms six comparative methods on both G-mean and Recall under various labeling budgets.

Semi-supervised adversarial deep learning for capacity estimation of battery energy storage systems

Battery energy storage systems (BESS) play a pivotal role in energy management, and the precise estimation of battery capacity is crucial for optimizing their performance and ensuring reliable power supply. Deep learning methodologies applied to battery capacity estimation have exhibited exemplary performance. However, deep learning methods necessitate supervised training with a significant volume of labeled data, presenting challenges for data collection in industrial scenarios. Moreover, a diverse range of battery types in industrial settings makes it difficult to develop capacity estimation models for different types of batteries from scratch. To address these issues, a semi-supervised adversarial deep learning (SADL) method is proposed for lithium-ion battery capacity estimation. Initially, a subset of labeled lithium-ion battery data, coupled with a subset of unlabeled data, is collected. Voltage and current data are then transformed into capacity increment features. Subsequently, an adversarial training strategy is employed, subjecting labeled and unlabeled data to adversarial training to enhance the performance of SADL. Finally, the effectiveness of the SADL method in estimating the capacity of other lithium-ion batteries is analysed. Experimental results demonstrate that the SADL method accurately estimates the capacity of various battery types, showcasing an RMSE error of approximately 2%, surpassing the performance of other methods. The proposed SADL method emerges as a promising solution for the precise estimation of lithium-ion battery capacity in BESS.

Semi-supervised deep learning for molecular clump verification

Context. A reliable molecular clump detection algorithm is essential for studying these clumps. Existing detection algorithms for molecular clumps still require that detected candidates be verified manually, which is impractical for large-scale data. Semi-supervised learning methods, especially those based on deep features, have the potential to accomplish the task of molecular clump verification thanks to the powerful feature extraction capability of deep networks. Aims. Our main objective is to develop an automated method for the verification of molecular clump candidates. This method utilises a 3D convolutional neural network (3D CNN) to extract features of molecular clumps and employs semi-supervised learning to train the model, with the aim being to improve its generalisation ability and data utilisation. It addresses the issue of insufficient labelled samples in traditional supervised learning and enables the model to better adapt to new, unlabelled samples, achieving high accuracy in the verification of molecular clumps. Methods. We propose SS-3D-Clump, a semi-supervised deep clustering method that jointly learns the parameters of a 3D CNN and the cluster assignments of the generated features for automatic verification of molecular clumps. SS-3D-Clump iteratively classifies the features with the Constrained-KMeans and uses these class labels as supervision to update the weights of the entire network. Results. We used CO data from the Milky Way Imaging Scroll Painting project covering 350 square degrees in the Milky Way’s first, second, and third quadrants. The ClumpFind algorithm was applied to extract molecular clump candidates in these regions, which were subsequently verified using SS-3D-Clump. The SS-3D-Clump model, trained on a dataset comprising three different density regions, achieved an accuracy of 0.933, a recall rate of 0.955, a precision rate of 0.945, and an F1 score of 0.950 on the corresponding test dataset. These results closely align with those obtained through manual verification. Conclusions. Our experiments demonstrate that the SS-3D-Clump model achieves high accuracy in the automated verification of molecular clumps. It effectively captures the essential features of the molecular clumps and overcomes the challenge of limited labelled samples in supervised learning by using unlabelled samples through semi-supervised learning. This enhancement significantly improves the generalisation capability of the SS-3D-Clump model, allowing it to adapt effectively to new and unlabelled samples. Consequently, SS-3D-Clump can be integrated with any detection algorithm to create a comprehensive framework for the automated detection and verification of molecular clumps.

Text Attribute Control via Closed-Loop Disentanglement

Abstract Changing an attribute of a text without changing the content usually requires first disentangling the text into irrelevant attributes and content representations. After that, in the inference phase, the representation of one attribute is tuned to a different value, expecting that the corresponding attribute of the text can also be changed accordingly. The usual way of disentanglement is to add some constraints on the latent space of an encoder-decoder architecture, including adversarial-based constraints and mutual-information-based constraints. However, previous semi-supervised processes of attribute change are usually not enough to guarantee the success of attribute change and content preservation. In this paper, we propose a novel approach to achieve a robust control of attributes while enhancing content preservation. In this approach, we use a semi-supervised contrastive learning method to encourage the disentanglement of attributes in latent spaces. Differently from previous works, we re-disentangle the reconstructed sentence and compare the re-disentangled latent space with the original latent space, which makes a closed-loop disentanglement process. This also helps content preservation. In addition, the contrastive learning method is also able to replace the role of minimizing mutual information and adversarial training in the disentanglement process, which alleviates the computation cost. We conducted experiments on three text datasets, including the Yelp Service review dataset, the Amazon Product review dataset, and the GoEmotions dataset. The experimental results show the effectiveness of our model.

Gearbox fault diagnosis method based on improved semi-supervised MTDL and GAF

Aiming at the problem that it takes a long time and high cost to obtain complete labeled data under intelligent fault diagnosis and unlabeled data is not used. This paper proposes an improved semi-supervised mean teacher deep learning (MTDL) and Gramian angle field (GAF) fusion diagnostic method. This method fully utilizes a small number of labeled samples and a large number of unlabeled samples to deeply mine invisible fault features and potential physical correlations. At the same time, it solves the problem of losing the inter-data correlation structure when one-dimensional time series signals are used as inputs for neural networks. The GAF-MTDL method uses consistency regularization and modifies the network structure in the mean teacher algorithm into a semi-supervised deep learning model enhanced by WideResNet. The experimental results show that the proposed GAF-MTDL method saves a lot of manual labeling costs, improves the recognition accuracy and generalization ability, and can achieve excellent prediction accuracy with very little labeled data. In the end, the accuracy of planetary gear fault identification reached 98.22% under the labeling rate of 20%, and the accuracy of fault identification reached 99.98% through the verification of the bearing data set of Case Western Reserve University. The value of this research is to bring an efficient and low-cost technology to the field of industrial intelligent fault diagnosis, which can significantly improve the accuracy of fault identification.