Semi-supervised Regression Research Articles

Semi-supervised regression based on Representation Learning for fermentation processes

Biofermentation faces challenges in obtaining real-time quality variables, making it necessary to predict these variables. However, the fermentation process data vary in length and lack sufficient labeled data for model establishment. To solve this problem, this study introduces a framework named RL-SSR(Representation Learning-based Semi-Supervised Regression). First, a data rotation mechanism is designed to address the issue of non-equal-length data. Second, representation learning pre-tasks containing contrastive learning and data reconstruction tasks are implemented to introduce a priori knowledge and numeric features. Finally, the pre-trained model will be fine-tuned with limited labeled data. Experimental results using an industrial-scale penicillin fermentation dataset reveal that RL-SSR outperforms other baseline models, particularly with a small number of labels, confirming the robustness and effectiveness of RL-SSR in the real-time prediction of quality variables in fermentation processes.

Semi-supervised regression with label-guided adaptive graph optimization

Semi-supervised regression with label-guided adaptive graph optimization

S-wave log construction through semi-supervised regression clustering using machine learning: A case study of North Sea fields

Accurate prediction of S-wave velocity from well logs is essential for understanding subsurface geological formations and hydrocarbon reservoirs. Machine learning techniques, including clustering and regression, have emerged as effective methods for indirectly estimating S-wave logs and other rock properties. In this study, we employed clustering algorithms to identify similarities among well log datasets, encompassing depth, sonic, porosity, neutron, and apparent density, facilitating the discovery of correlations among various wells. These identified correlations served as a foundation for predicting S-wave values using a novel semi-supervised approach. Our approach combined clustering, specifically k-means clustering, with different types of regressors, including Least Squares Regression (LSR), Support Vector Regression (SVR), and Multi-Layer Perceptron (MLP). Our results demonstrate the superior performance of this integrated approach compared to traditional regression methods. We validated our methodology using various parametric and non-parametric regression techniques, showcasing its effectiveness not only on wells within the training region but also on wells outside the study area. We achieved a significant improvement in the R2 score metric, ranging from 2.22% to 6.51%, and a reduction in Mean Square Error (MSE) of at least 31% when compared to predictions made without clustering. This study underscores the potential of machine learning techniques for accurate prediction of S-wave velocity and other rock properties, thereby enhancing our comprehension of subsurface geology and hydrocarbon reservoirs.

OSSEFS: An online semi-supervised ensemble fuzzy system for data streams learning with missing values

The incompleteness of data samples in data streams always affects the performance of learning model. In order to learn data streams with missing values from features and a large amount of unlabeled data, this paper proposes a semi-supervised ensemble algorithm based on evolving fuzzy systems (EFS) named OSSEFS. Aiming at missing values, a fuzzy rule based online imputation strategy is designed to improve the data feature integrity. Aiming at unlabeled data, a new ensemble fuzzy system architecture is proposed to fully utilize both labeled and unlabeled data implicit information to predict data labels accurately. In addition, an online rule parameters optimization strategy is proposed to achieve adaptive adjustment of rule distribution. Thanks to the above strategies, OSSEFS can perform well in data environments containing missing values and unlabeled data. Comparative experimental results on multiple datasets show that OSSEFS outperforms the classical semi-supervised regression algorithms.

Near-infrared spectral expansion method based on active semi-supervised regression

The utilization of near-infrared (NIR) spectroscopy, in conjunction with chemometric techniques, has been widely used in a variety of sectors, including agricultural production and pharmaceutical production. Nevertheless, the laborious and arduous procedure of gathering samples and evaluating their physicochemical properties leads to relatively limited training set sizes for modeling. This problem severely limits the optimization and practical application of NIR spectrum analysis models. The Safer Active Semi-Supervised Sample Augmentation Learning Model (Safer-AS3A) proposed in this paper tries to address the problem by incorporating active learning (AL) and semi-supervised learning (SSL) techniques. Experiments were conducted on two sets of publicly available NIR spectral datasets, and the Safer-AS3A model was compared to other models with similar characteristics. The experimental results indicate that the Safer-AS3A model proposed in our study outperforms comparable models in terms of accuracy and robustness when dealing with scenarios having a limited number of labeled samples. Furthermore, after the training set was expanded with the Safer-AS3A model, the Partial least squares regression (PLSR), Bayesian ridge regression (BRR), and Support vector regression (SVR) models on the Diesel dataset improved their R2 on the test set by 5.923%, 3.018%, and 7.331%, respectively, compared to the models using only the labeled sample set. On the other hand, the Ridge regression (RR), BRR, and SVR models on the test set on the Shoot dataset improved the R2 by 4.169%, 4.449%, and 11.597%, respectively. Overall, the Safer-AS3A model can effectively expand the NIR spectral dataset and considerably improve the performance of the NIR spectral analysis model. Using the AL method, the SSL method, and the co-training method together, a novel and effective method is presented for generating high-quality pseudo-labels. This method opens up new avenues for enhancing the efficiency and precision of NIR spectrum analysis. It also provides novel perspectives on sample diversification and prospective applications in other disciplines.

Deep semi-supervised regression via pseudo-label filtering and calibration

Semi-supervised learning (SSL) is a widely used model training paradigm that effectively utilizes a limited set of labeled data and a substantially larger pool of unlabeled data. Historically, the focus of SSL research has predominantly been on classification tasks, employing methods such as consistency regularization and pseudo-labeling. However, the direct application of these methods to regression tasks presents significant challenges, primarily due to the complexities associated with evaluating the reliability of pseudo-labels in a regression context. This paper introduces SimRegMatch, a novel semi-supervised regression (SSR) framework devised to overcome this specific challenge, by combining an uncertainty-based filtering mechanism with a similarity-based pseudo-label calibration approach. The former component is tasked with discerning which unlabeled examples possess pseudo-labels of sufficient reliability, achieved through the estimation of uncertainty levels. The latter component then refines these pseudo-labels by propagating information from labeled to unlabeled examples, thereby enhancing the overall quality of the pseudo-labels. The efficacy of SimRegMatch was rigorously tested through experiments conducted on the publicly available AgeDB dataset, which is centered around age prediction, as well as on a practical regression problem focused on the detection of interior noise levels in automobiles using accelerometer data. When benchmarked against current state-of-the-art methods in semi-supervised regression, SimRegMatch exhibited notable improvements in regression performance. Additionally, a series of ablation studies were carried out to dissect and understand the specific elements of the framework that were instrumental in achieving these performance enhancements. SimRegMatch addresses a pivotal issue in semi-supervised regression – the assessment of regression pseudo-label reliability – and substantially elevates model performance. By combining the strengths of uncertainty estimation and pseudo-label calibration, SimRegMatch emerges as a robust and versatile framework with significant potential for broad applicability in various SSR scenarios. A PyTorch implementation is publicly available at https://github.com/YongwonJo/SimRegMatch.

Automated spectral transfer learning strategy for semi-supervised regression on Chlorophyll-a retrievals with Sentinel-2 imagery

ABSTRACT Multispectral images make it possible to retrieve water quality parameters over a wide range and long time series by remote sensing. The robustness of traditional supervised machine learning models, which have been widely used, is affected in various regions, and these models are developed by in-situ measurements and limited satellite images. This study combined a spectral transfer learning strategy with a semi-supervised regression model to conduct data augmentation and developed a robust model for two inland lakes in Jilin Province, Northeast China, based on Sentinel-2 Multispectral Instruments images. Specifically, this study integrated an automated module for Balance Distribution Adaptation and a spectral features transfer method, and then applied it to Co-training Regressors for Chlorophyll-a retrievals to achieve the highest accuracy, which solved the selection of parameters and models. The Automated model was developed and achieved the best performance (R 2 is 0.91, the root mean square error is 2.62 μg L−1 and the mean absolute percentage error is 22.88%) compared with other models, and showed robustness in various lakes and months. Our results offer a reliable approach to provide an accurate Chlorophyll-a estimation of spatial–temporal variations.

A Quality Prediction Method Based on Tri-Training Weighted Ensemble Just-in-Time Learning–Relevance Vector Machine Model

The core quality data, such as interior ballistic performance, are seriously unbalanced in the plasticizing and molding process, which makes it difficult for traditional supervised learning methods to accurately predict this kind of index. A Tri-training weighted ensemble JITL-RVM model based on an integrated confidence evaluation strategy is proposed to solve the above problem. The method is based on Tri-training semi-supervised regression architecture and uses both labeled and unlabeled data for modeling. First of all, the traditional single similarity measure method is difficult to use to evaluate the real similarity between data samples reliably and stably. This method realizes diversity enhancement and data expansion of the data set for modelling through ensemble just-in-time modelling based on three homologous and heterogeneous mixed similarity measures. Secondly, a new integrated confidence evaluation strategy is used to select the unlabeled samples, and the pseudo-labeled data, which can improve the prediction performance of the model, can be selected. To improve the prediction effect of the model, the pseudo-label value of the data is revised continuously. The integrated confidence evaluation strategy can overcome many shortcomings of the traditional confidence evaluation method based on Co-training regression (Coreg). Finally, the final quality prediction value is obtained through weighted integration fusion, which reflects the difference between different models and further improves the prediction accuracy. The experimental results of interior ballistic performance prediction of single-base gun propellant show the effectiveness and superiority of the proposed method, and it can improve the RMSE, R2, and PHR to 0.8074, 0.9644, and 93.3%, respectively.

Semi-supervised cooperative regression model for small sample estimation of citrus leaf nitrogen content with UAV images

ABSTRACT Nitrogen is an essential nutrient element for the growth of citrus tree. The accurate estimation of leaf nitrogen content (LNC) is important to guarantee fruit quality and yield. The unmanned aerial vehicle (UAV) remote sensing has shown great potential for monitoring the LNC of crops. However, the number of training samples is always limited due to the high cost and time consumption of acquiring LNC samples. Obtaining satisfactory results is difficult for most machine-learning models with insufficient samples. Thus, a semi-supervised regression model is designed in this paper to improve the performance of estimating citrus LNC from UAV images under the condition of small samples. First, the local binary pattern (LBP) operator is employed to fully extract textural features in UAV images. Second, semi-supervised cooperative regression models based on ridge regression (Ridge), support vector regression (SVR), and random forest (RF) are constructed by combining spectral and textural features. Finally, the best learning model is used to realize the accurate limited sample LNC estimation from the available training samples. The experiments confirm that the semi-supervised cooperative regression model has better and promising results than other machine-learning methods under the condition of small samples. The RF-based cooperative regression model (CoRF) performs best among other models, with a determination coefficient (R2) of 0.716 and a root mean square error (RMSE) of 0.620 g·kg−1. The CoRF model also achieves superior performance when combining LBP textural features. The results estimated by the semi-supervised cooperative regression model is capable of providing instructions for the production and management of citrus orchard.

Safe co-training for semi-supervised regression

Co-training is a popular semi-supervised learning method. The learners exchange pseudo-labels obtained from different views to reduce the accumulation of errors. One of the key issues is how to ensure the quality of pseudo-labels. However, the pseudo-labels obtained during the co-training process may be inaccurate. In this paper, we propose a safe co-training (SaCo) algorithm for regression with two new characteristics. First, the safe labeling technique obtains pseudo-labels that are certified by both views to ensure their reliability. It differs from popular techniques of using two views to assign pseudo-labels to each other. Second, the label dynamic adjustment strategy updates the previous pseudo-labels to keep them up-to-date. These pseudo-labels are predicted using the augmented training data. Experiments are conducted on twelve datasets commonly used for regression testing. Results show that SaCo is superior to other co-training style regression algorithms and state-of-the-art semi-supervised regression algorithms.

Semi-Supervised Deep Regression with Uncertainty Consistency and Variational Model Ensembling via Bayesian Neural Networks

Deep regression is an important problem with numerous applications. These range from computer vision tasks such as age estimation from photographs, to medical tasks such as ejection fraction estimation from echocardiograms for disease tracking. Semi-supervised approaches for deep regression are notably under-explored compared to classification and segmentation tasks, however. Unlike classification tasks, which rely on thresholding functions for generating class pseudo-labels, regression tasks use real number target predictions directly as pseudo-labels, making them more sensitive to prediction quality. In this work, we propose a novel approach to semi-supervised regression, namely Uncertainty-Consistent Variational Model Ensembling (UCVME), which improves training by generating high-quality pseudo-labels and uncertainty estimates for heteroscedastic regression. Given that aleatoric uncertainty is only dependent on input data by definition and should be equal for the same inputs, we present a novel uncertainty consistency loss for co-trained models. Our consistency loss significantly improves uncertainty estimates and allows higher quality pseudo-labels to be assigned greater importance under heteroscedastic regression. Furthermore, we introduce a novel variational model ensembling approach to reduce prediction noise and generate more robust pseudo-labels. We analytically show our method generates higher quality targets for unlabeled data and further improves training. Experiments show that our method outperforms state-of-the-art alternatives on different tasks and can be competitive with supervised methods that use full labels. Code is available at https://github.com/xmed-lab/UCVME.

Predicting bite rate of grazing cattle from accelerometry data via semi-supervised regression

Accurate estimation of bite rate, i.e., the number of bites during a given grazing period, is important for the characterisation of cattle grazing behaviour and feed intake. In this paper, we estimate the bite rate of grazing cattle from accelerometer data using semi-supervised linear regression. We use a mixture of labelled and unlabelled accelerometer data collected during the grazing trial to train the semi-supervised model. The labelled data has associated ground-truth values for the bite rate and was recorded by an expert observer at intermittent times over the duration of the trial. The trial involved ten cattle, each fitted with a tri-axial accelerometer mounted on a neck collar, over a period of five weeks. We calculate various features from tri-axial accelerometer data and utilise several feature selection approaches and regression methods to fit bite-rate prediction models to our data and compare their performance. Our leave-one-animal-out cross-validated results show that a semi-supervised linear regression model with the specifically chosen feature set delivers good bite-rate prediction performance with a root mean squared error of 5.73 bites per minute and an R2 value of 0.73. Other regression models tested here (including orthogonal distance regression) yielded similar results.

Semi-Supervised Deep Kernel Active Learning for Material Removal Rate Prediction in Chemical Mechanical Planarization.

The material removal rate (MRR) is an important variable but difficult to measure in the chemical-mechanical planarization (CMP) process. Most data-based virtual metrology (VM) methods ignore the large number of unlabeled samples, resulting in a waste of information. In this paper, the semi-supervised deep kernel active learning (SSDKAL) model is proposed. Clustering-based phase partition and phase-matching algorithms are used for the initial feature extraction, and a deep network is used to replace the kernel of Gaussian process regression so as to extract hidden deep features. Semi-supervised regression and active learning sample selection strategies are applied to make full use of information on the unlabeled samples. The experimental results of the CMP process dataset validate the effectiveness of the proposed method. Compared with supervised regression and co-training-based semi-supervised regression algorithms, the proposed model has a lower mean square error with different labeled sample proportions. Compared with other frameworks proposed in the literature, such as physics-based VM models, Gaussian-process-based regression models, and stacking models, the proposed method achieves better prediction results without using all the labeled samples.

Orthogonal semi-supervised regression with adaptive label dragging for cross-session EEG emotion recognition

Owning to its merits of great temporal resolution, portability and low cost, electroencephalogram (EEG) signals have received increasing attention in emotion recognition. All the time, least square regression (LSR) has been widely employed in classification tasks. However, there exist two issues based on the LSR method, which limits its performance. The first problem is that the LSR method usually cannot make EEG data retain more discriminative information. The second is that it is improper for multiple emotion classification to use hard discrete labels as regression objectives. To address these issues, we develop orthogonal semi-supervised regression with adaptive label dragging model (OSRLD) to recognize emotions. OSRLD can get a closed-form solution with less computational costs. Furthermore, experimental findings using the open SEED-IV dataset show that 1) Compared to seven methods in terms of classification accuracy, OSRLD achieves the best average accuracy of 77.96%, 80.20% and 81.45%, 2) A technology of label dragging which modifies the label vector of each sample can effectively enlarge the distance between classes, and 3) Based on learned transformation matrix, the primary EEG frequency bands and brain areas are automatically identified, which lays theoretical basis for simplifying the hardware design of EEG acquisition devices.

Hybrid short-term load forecasting using CGAN with CNN and semi-supervised regression

Accurate short-term load forecasting (STLF) is essential to improve secure and economic operation of power systems. In this paper, a hybrid STLF model using the conditional generative adversarial network (CGAN) with convolutional neural network (CNN) and semi-supervised regression is proposed to improve the accuracy of STLF. Firstly, a conditional label matrix with relevant factors is constructed as the conditional labels of CGAN. The grey weighted correlation method is applied to generate high-quality similar days as one of the labels. The input data with conditional labels and load time series are decomposed into several sub-modes by the variational mode decomposition (VMD), which transforms the load forecasting into several sub-forecasting. Then, the CGAN generator is to capture the internal feature of each mode with the CNN and generate fake samples, while the CGAN discriminator is modified with a semi-supervised regression layer to extract the nonlinear and dynamic behaviors of the dataset and perform precise STLF. The final forecasting results are obtained by aggregating the results of all sub-mode. The generator and discriminator of the CGAN form a min–max game to improve the sample generation ability and reduce forecasting errors. The simulation results show that the STLF accuracy with the proposed model is significantly improved.

Diverse and consistent multi-view networks for semi-supervised regression

Diverse and consistent multi-view networks for semi-supervised regression

Correlated Bayesian Co-Training for Virtual Metrology

A rising challenge in manufacturing data analysis is training robust regression models using limited labeled data. In this work, we investigate a semi-supervised regression scenario, where a manufacturing process operates on multiple mutually correlated states. We exploit this inter-state correlation to improve regression accuracy by developing a novel co-training method, namely Correlated Bayesian Co-training (CBCT). CBCT adopts a block Sparse Bayesian Learning framework to enhance multiple individual regression models which share the same support. Additionally, CBCT casts a unified prior distribution on both the coefficient magnitude and the inter-state correlation. The model parameters are estimated using maximum-a-posteriori estimation (MAP), while hyper-parameters are estimated using the expectation-maximization (EM) algorithm. Experimental results from two industrial examples shows that CBCT successfully leverages inter-state correlation to reduce the modeling error by up to 79.40%, compared to other conventional approaches. This suggests that CBCT is of great value to multi-state manufacturing applications.

FuzzyGAN: Fuzzy generative adversarial networks for regression tasks

Generative Adversarial Networks (GANs) are well-known tools for data generation and semi-supervised classification. GANs, with less labeled data, outperform Deep Neural Networks (DNNs) and Convolutional Neural Networks (CNNs) in classification tasks. The success of GANs in classification tasks motivates the development of GAN-based techniques for semi-supervised regression tasks. However, developing GANs for regression introduces two major challenges: (1) inherent instability in the GAN formulation and (2) performing regression and achieving stability simultaneously. This paper introduces techniques that show improvement in the GANs’ regression capability. We bake a differentiable fuzzy logic system at multiple locations in a GAN. The fuzzy logic takes the output of either the generator or the discriminator to predict the output, y, and evaluate the generator’s performance. We outline the results of applying the fuzzy logic system across multiple GANs and summarize each approach’s efficacy. This paper shows that adding a fuzzy logic layer can enhance GAN’s ability to perform regression; the most desirable injection location is problem-specific, and we show this through experiments over various datasets. Besides, we demonstrate empirically that the fuzzy-infused GANs are competitive with the DNNs.

Semi-supervised regression with adaptive graph learning for EEG-based emotion recognition.

Electroencephalogram (EEG) signals are widely used in the field of emotion recognition since it is resistant to camouflage and contains abundant physiological information. However, EEG signals are non-stationary and have low signal-noise-ratio, making it more difficult to decode in comparison with data modalities such as facial expression and text. In this paper, we propose a model termed semi-supervised regression with adaptive graph learning (SRAGL) for cross-session EEG emotion recognition, which has two merits. On one hand, the emotional label information of unlabeled samples is jointly estimated with the other model variables by a semi-supervised regression in SRAGL. On the other hand, SRAGL adaptively learns a graph to depict the connections among EEG data samples which further facilitates the emotional label estimation process. From the experimental results on the SEED-IV data set, we have the following insights. 1) SRAGL achieves superior performance compared to some state-of-the-art algorithms. To be specific, the average accuracies are 78.18%, 80.55%, and 81.90% in the three cross-session emotion recognition tasks. 2) As the iteration number increases, SRAGL converges quickly and optimizes the emotion metric of EEG samples gradually, leading to a reliable similarity matrix finally. 3) Based on the learned regression projection matrix, we obtain the contribution of each EEG feature, which enables us to automatically identify critical frequency bands and brain regions in emotion recognition.

EEG-Based Emotion Recognition by Retargeted Semi-Supervised Regression with Robust Weights

The electroencephalogram (EEG) can objectively reflect the emotional state of human beings, and has attracted much attention in the academic circles in recent years. However, due to its weak, non-stationary, and low signal-to-noise properties, it is inclined to cause noise in the collected EEG data. In addition, EEG features extracted from different frequency bands and channels usually exhibit different levels of emotional expression abilities in emotion recognition tasks. In this paper, we fully consider the characteristics of EEG and propose a new model RSRRW (retargeted semi-supervised regression with robust weights). The advantages of the new model can be listed as follows. (1) The probability weight is added to each sample so that it could help effectively search noisy samples in the dataset, and lower the effect of them at the same time. (2) The distance between samples from different categories is much wider than before by extending the ϵ-dragging method to a semi-supervised paradigm. (3) Automatically discover the EEG emotional activation mode by adaptively measuring the contribution of sample features through feature weights. In the three cross-session emotion recognition tasks, the average accuracy of the RSRRW model is 81.51%, which can be seen in the experimental results on the SEED-IV dataset. In addition, with the support of the Friedman test and Nemenyi test, the classification of RSRRW model is much more accurate than that of other models.