Semi-supervised Support Vector Machines Research Articles

Mixed-integer quadratic optimization and iterative clustering techniques for semi-supervised support vector machines

AbstractAmong the most famous algorithms for solving classification problems are support vector machines (SVMs), which find a separating hyperplane for a set of labeled data points. In some applications, however, labels are only available for a subset of points. Furthermore, this subset can be non-representative, e.g., due to self-selection in a survey. Semi-supervised SVMs tackle the setting of labeled and unlabeled data and can often improve the reliability of the results. Moreover, additional information about the size of the classes can be available from undisclosed sources. We propose a mixed-integer quadratic optimization (MIQP) model that covers the setting of labeled and unlabeled data points as well as the overall number of points in each class. Since the MIQP’s solution time rapidly grows as the number of variables increases, we introduce an iterative clustering approach to reduce the model’s size. Moreover, we present an update rule for the required big-M values, prove the correctness of the iterative clustering method as well as derive tailored dimension-reduction and warm-starting techniques. Our numerical results show that our approach leads to a similar accuracy and precision than the MIQP formulation but at much lower computational cost. Thus, we can solve larger problems. With respect to the original SVM formulation, we observe that our approach has even better accuracy and precision for biased samples.

Reject inference in credit scoring based on cost-sensitive learning and joint distribution adaptation method

As traditional credit evaluation methods generally only use accepted sample modeling, the rejected data is omitted, which means the model's prediction of new customers is biased. However, reject inference can be used to solve this credit evaluation sample selection bias. This paper proposes a new reject inference method based on joint distribution adaptation (JDA) and cost-sensitive semi-supervised support vector machines (CS4VM). First, this method uses both accepted (labeled) samples and rejected (unlabeled) samples modeling, which overcomes the deviations in traditional credit evaluation methods. Second, as the accepted sample and the rejected sample distributions are different, this method reduces the distribution differences between the accepted and rejected sample sets, which ensures that the sample data conforms to the basic assumptions in the semi-supervised model, and improves the performance of the classification model. Third, this method reduces the overall cost in the actual credit business by considering both the traditional misclassification costs when mining the default samples and the different decision weights for the accepted and rejected samples. Finally, an empirical study verifies the excellent predictive performance of the proposed method and effectively reduces the total credit costs.

A classification method of fuzzy semi-supervised support vector machines for the problems of imbalance

Positive instances are often significantly less than negative instances in real-world classification problems. However, positive categories are typically more relevant to the primary focus of categorization tasks. Moreover, obtaining labeled data is often expensive, and the majority of real-life data is unlabeled. Therefore, semi-supervised learning has become a popular approach for addressing imbalanced problems. Traditional support vector machines (SVMs) treat all samples equally and are not suitable for semi-supervised learning. To address this issue, a semi-supervised model called the fuzzy semi-supervised SVM (FS3VM) has been proposed. The FS3VM model uses the degree of entropy-based fuzzy membership to ensure the materiality of positive classes by assigning positive instances to relatively large degrees of fuzzy membership. After introducing the mainstream FS3VM model, the fundamental theory and methods of the model are discussed and expanded upon, including the FS3VM algorithm, which applies the Sequential Minimal Optimization (SMO) algorithm to the dual problem. The proposed FS3VM model is a smooth and continuous optimization problem, and its dual is a standard quadratic programming. Experimental results demonstrate that the proposed FS3VM model outperforms other compared learning algorithms.

Semi-Supervised Ensemble Learning Framework for Accelerating Power System Transient Stability Knowledge Base Generation

Machine learning approaches have exhibited high potential in power system transient stability assessment (TSA), yet their initial preparation stages of stability knowledge base generation (SKBG) based on time-domain simulations often undergo high computational costs. In fact, how to ease the heavy computational burden of SKBG without sacrificing the reliability is still a significant challenge. To address this problem, this paper develops a semi-supervised ensemble learning (SSEL) framework for reliable SKBG acceleration, without additional need for computing hardware upgrading. In particular, it performs detailed simulations for a minority of cases and fast simulations for the majority ones to reduce the total computation time. Considering the absence of stability status information of those fast simulated cases, an SSEL scheme is systematically designed to reliably label their stability status. Before implementing SSEL, two concise feature descriptors are first introduced to efficiently extract transient features from multiplex system trajectories. Then, all the cases are characterized in a unified feature space, whereby a series of semi-supervised support vector machines are trained in randomly formed subspaces. Afterward, these single machines are systematically combined to derive an enhanced SSEL model, which is able to make reliable and robust labeling decisions. Further, a backtrace strategy is carefully devised for SSEL, so as to maintain the high reliability of SKBG. Test results on the IEEE 39-bus system and the realistic GD Power Grid in South China illustrate the excellent performances of the proposed framework on SKBG acceleration.

Comparison of general kernel, multiple kernel, infinite ensemble and semi-supervised support vector machines for landslide susceptibility prediction

Comparison of general kernel, multiple kernel, infinite ensemble and semi-supervised support vector machines for landslide susceptibility prediction

Applications of Semi-Supervised Support Vector Machines in Data Separation Methods in Structural Health Monitoring

Applications of Semi-Supervised Support Vector Machines in Data Separation Methods in Structural Health Monitoring

Safe transductive support vector machine

Since semi-supervised learning can use fewer labelled samples to train a better model, semi-supervised methods are becoming popular in data mining. As an important algorithm of semi-supervised support vector machines (SVM), transductive support vector machine (TSVM) sometimes may get worse models trained on both labelled samples and unlabelled samples than those trained only on labelled samples. To solve this problem, in this paper, we propose a safe TSVM (STSVM) based on the infinitesimal annealing algorithm. In the training of TSVM, we adopt the infinitesimal annealing and path following technology to approximate the step size of simulated annealing to balance the contradiction between annealing step and calculation time. During the annealing process, we call CP-step to update TSVM model with pseudo-labelled samples. If the current sample is on the boundary of combinatorial optimisation problem, SJ-step is called and a safety condition is designed to determine whether the sample needs to change its label or not, so as to ensure the TSVM model trained after changing is better than the model got before. The experimental results show that our STSVM algorithm can improve the accuracy of TSVM with a shorter running time, and is safer than the existing safe algorithms.

Semi-Supervised Support Vector Machine for Digital Twins Based Brain Image Fusion.

The purpose is to explore the feature recognition, diagnosis, and forecasting performances of Semi-Supervised Support Vector Machines (S3VMs) for brain image fusion Digital Twins (DTs). Both unlabeled and labeled data are used regarding many unlabeled data in brain images, and semi supervised support vector machine (SVM) is proposed. Meantime, the AlexNet model is improved, and the brain images in real space are mapped to virtual space by using digital twins. Moreover, a diagnosis and prediction model of brain image fusion digital twins based on semi supervised SVM and improved AlexNet is constructed. Magnetic Resonance Imaging (MRI) data from the Brain Tumor Department of a Hospital are collected to test the performance of the constructed model through simulation experiments. Some state-of-art models are included for performance comparison: Long Short-Term Memory (LSTM), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), AlexNet, and Multi-Layer Perceptron (MLP). Results demonstrate that the proposed model can provide a feature recognition and extraction accuracy of 92.52%, at least an improvement of 2.76% compared to other models. Its training lasts for about 100 s, and the test takes about 0.68 s. The Root Mean Square Error (RMSE) and Mean Absolute Error (MAE) of the proposed model are 4.91 and 5.59%, respectively. Regarding the assessment indicators of brain image segmentation and fusion, the proposed model can provide a 79.55% Jaccard coefficient, a 90.43% Positive Predictive Value (PPV), a 73.09% Sensitivity, and a 75.58% Dice Similarity Coefficient (DSC), remarkably better than other models. Acceleration efficiency analysis suggests that the improved AlexNet model is suitable for processing massive brain image data with a higher speedup indicator. To sum up, the constructed model can provide high accuracy, good acceleration efficiency, and excellent segmentation and recognition performances while ensuring low errors, which can provide an experimental basis for brain image feature recognition and digital diagnosis.

One novel class of Bézier smooth semi-supervised support vector machines for classification

The semi-supervised support vector machine (S3VM) for classification is introduced for dealing with quantities of unlabeled data in the real world. Labeled data are utilized to train the algorithm and then were adapted to classify the unlabeled data. However, this algorithm has several drawbacks, such as the non-smooth term of semi-supervised objective function negatively affects the classification precision. Moreover, it is required to endure heavy burden in solving two quadratic programming problems with inversion matrix operation. To cope with this problem, this article puts forward a novel class of Bezier smooth semi-supervised support vector machines (BS4VMs), based on the approximation property of Bezier function to the non-smooth term. Because of this approximation, a fast quasi-Newton method for solving BS4VMs can be used to decrease the calculating time scale. This new kind of algorithm enhances the generalization and robustness of S3VM for nonlinear case as well. Further, to show how the BS4VMs can be practically implemented, experiments on synthetic, UCI dataset, USPS dataset, and large-scale NDC database are offered. The theoretical analysis and experiments comparisons clearly confirm the superiority of BS4VMs in both classification accuracy and calculating time.

Enhancing SVM for survival data using local invariances and weighting

BackgroundThe necessity to analyze medium-throughput data in epidemiological studies with small sample size, particularly when studying biomedical data may hinder the use of classical statistical methods. Support vector machines (SVM) models can be successfully applied in this setting because they are a powerful tool to analyze data with large number of predictors and limited sample size, especially when handling binary outcomes. However, biomedical research often involves analysis of time-to-event outcomes and has to account for censoring. Methods to handle censored data in the SVM framework can be divided into two classes: those based on support vector regression (SVR) and those based on binary classification. Methods based on SVR seem to be suboptimal to handle sparse data and yield results comparable to Cox proportional hazards model and kernel Cox regression. The limited work dedicated to assess methods based on of SVM for binary classification has been based on SVM learning using privileged information and SVM with uncertain classes.ResultsThis paper proposes alternative methods and extensions within the binary classification framework, specifically, a conditional survival approach for weighting censored observations and a semi-supervised SVM with local invariances. Using simulation studies and some real datasets, we evaluate those two methods and compare them with a weighted SVM model, SVM extensions found in the literature, kernel Cox regression and Cox model.ConclusionsOur proposed methods perform generally better under a wide variety of realistic scenarios about the structure of biomedical data. Specifically, the local invariances method using the conditional survival approach is the most robust method under different scenarios and is a good approach to consider as an alternative to other time-to-event methods. When analysing real data is a method to be considered and recommended since outperforms other methods in proportional and non-proportional scenarios and sparse data, which is something usual in biomedical data and biomarkers analysis.

Multi-view semi-supervised least squares twin support vector machines with manifold-preserving graph reduction

Multi-view semi-supervised support vector machines consider learning with multi-view unlabeled data to boost the learning performance. However, they have several defects. They need to solve the quadratic programming problem and the time complexity is quite high. Moreover, when a large number of multi-view unlabeled examples exist, it can generate more outliers and noisy examples and influence the performance. Therefore, in this paper, we propose two novel multi-view semi-supervised support vector machines called multi-view Laplacian least squares twin support vector machine and its improved version with the manifold-preserving graph reduction which can enhance the robustness of the algorithm. They can reduce the time complexity by changing the constraints to a series of equality constraints and lead to a pair of linear equations. The linear multi-view Laplacian least squares twin support vector machine and its improved version with manifold-preserving graph reduction are further generalized to the nonlinear case via the kernel trick. Experimental results demonstrate that our proposed methods are effective.

Extreme semi-supervised learning for multiclass classification

Semi-Supervised Support Vector Machines (S3VMs) provide a powerful framework for Semi-Supervised Learning (SSL) tasks which leverage widely available unlabeled data to improve performance. However, there exist three issues in S3VMs: (i) S3VMs require concurrently training c one-against-all (OAA) classifiers (c is the number of classes) for multiclass classification, which is prohibitive for large c; (ii) S3VMs require huge computational time and large storage (because of the large kernel matrix) in large-scale training and testing; (iii) S3VMs require the balance constraint in the unlabeled data, which not only needs prior knowledge from the unlabeled data (the prior knowledge is unavailable in some applications), but also makes their nonconvex optimization problem more intractable. To address these issues, a novel method called Extreme Semi-Supervised Learning (ESSL) is proposed in this paper. First, the framework of Extreme Learning Machine (ELM) is adopted to handle both binary and multiclass classification problems in a unified model. Second, the hidden layer is encoded by an extremely small approximate empirical kernel map (AEKM) to greatly reduce the computational cost and the memory usage for training and testing. Third, the balance constraint (prior knowledge) in the unlabeled data is removed through the elaborative design of weighting function (which emphasizes the importance of labeled data and the minority pattern in the labeled data).By these three ways, ESSL can be solved effectively and efficiently based on alternating optimization (AO). More specifically, ESSL can be analytically and simply solved by generalized pseudoinverse and oneHotMap function (without any optimization solver and the OAA strategy) in the AO procedure, and consequently, better performance and much faster training speed are always achieved in ESSL. Our empirical study shows that ESSL significantly outperforms existing efficient SSL methods (e.g., meanS3VM and SS-ELM) in terms of accuracy, efficiency and memory, especially for large-scale multiclass problems. As an example, on the 20Newsgroups dataset, ESSL respectively runs 45 and 120 times faster than meanS3VM for training and testing with the improvement in accuracy of 3%, while the memory usage is reduced to 1/14. It is noteworthy that even though all the model parameters are with default values, ESSL already produces very excellent performance without fine-tuning parameters.

Semi-supervised modeling and compensation for the thermal error of precision feed axes

The data-driven modeling of thermal error-temperature relationship is key to achieve ideal compensation effect for precision machine tools. The improvements of the modeling quality are limited only depending on ameliorating the regression algorithm with same training data, and more information must be introduced for further improvements. The thermal error data, in particular for the feed axes, are usually high-cost and scarce, but the temperature data are usually readily available. Here, it is indicated that an extra information, the low-cost unlabeled temperature data which are easily accessible under various operation conditions, can be exploited to enrich the thermal error modeling data for the feed axes. Then the co-training semi-supervised support vector machines for regression (COSVR), which can include the pattern information of the unlabeled data in modeling, is employed to establish the thermal error-temperature model for feed axes. Thermal experiments were conducted on two cases of different axes, and the labeled data of temperature and thermal error and the unlabeled data of only temperature were obtained under different operating speeds. The linear thermal errors were modeled by COSVR using all the data, and by the genetic algorithm SVR (GA-SVR) using only the labeled data, respectively. Comparisons showed that the COSVR model outperformed the GA-SVR model by 11.45% and 34.14% in RMSE on the two axes, respectively, and by 53.03% of maximum thermal error reduction in the compensation.

Detection of Human Fall Using Floor Vibration and Multi-Features Semi-Supervised SVM.

Human falls are the premier cause of fatal and nonfatal injuries among older adults. The health outcome of a fall event is largely dependent on rapid response and rescue of the fallen elder. Being able to provide an accurate and fast fall detection will dramatically improve the health outcomes of the older population and reduce the associated healthcare cost after a fall. To achieve the goal, a multi-features semi-supervised support vector machines (MFSS-SVM) algorithm utilizing measurements from structural floor vibration obtained through accelerometers is proposed in this study to detect falling events with limited labeled samples. In this MFSS-SVM algorithm, the peak value, energy, and correlation coefficient of the accelerometer signal are used as classification features. The performance of the proposed algorithm was validated with laboratory experiments among activities including falling, walking, free jumping, rhythmic jumping, bag dropping, and ball dropping. To further illustrate the performance of the algorithm, a benchmark database was adopted and expanded to test its ability to accurately identify falling, compared with the algorithm used in the benchmark study. Results show that by using the proposed algorithm, the falling events can be identified with high accuracy and confidence, even with small training datasets and test nodes.

Seismic classification-based method for recognizing epicenter-neighboring orbits

From the point of view of the Fourth Paradigm, this paper attempts to find a recognizing method based on DEMETER (Detection of Electro-Magnetic Emissions Transmitted from Earthquake Regions) satellite data for epicenter-neighboring orbits during strong shocks. Detection points or small regions are used as research objects in numerous studies on seismic activities recognition. Due to the infrequency of strong shocks, the number of non-seismic data is far larger than the abnormal one, which results in the underfitting during the training of recognition model. Additionally, data located along the edge of seismic regions can hardly be classified into abnormal dataset or non-seismic one. A sloppy classification can badly reduce the accuracy of model. Hence, it is desired to put forward a more suitable approach to make better use of original data. In this paper, a seismic classification-based method for recognizing epicenter-neighboring orbit is proposed to address these problems. Unlike the existing approaches, our method regards the satellite orbits as the analyzing objects, which avoids the underfitting performance caused by the unbalanced data distribution. Moreover, error correcting output coding (ECOC) strategy is utilized to transform the recognizing problem into a series of binary classifications. By means of safe semi-supervised support vector machines (S4VMs) with kernel combination, the unlabeled orbits help obtain a better classification performance. Finally, three groups of comprehensive experiments are applied to validate the effectiveness of the method.

Reject inference in credit scoring using Semi-supervised Support Vector Machines

Credit scoring models are commonly built on a sample of accepted applicants whose repayment and behaviour information is observable once the loan has been issued. However in practice these models are regularly applied to new applicants, which may cause sample bias. This bias is even more pronounced in online lending, where over 90% of total loan requests are rejected. Reject inference is a technique to infer the outcomes for rejected applicants and incorporate them in the scoring system, with the expectation that predictive accuracy is improved. This paper extends previous studies in two main ways: firstly, we propose a new method involving machine learning to solve the reject inference problem; secondly, the Semi-supervised Support Vector Machines model is found to improve the performance of scoring models compared to the industrial benchmark of logistic regression, based on 56,626 accepted and 563,215 rejected online consumer loans.

Distributed semi-supervised support vector machines

The semi-supervised support vector machine (S3VM) is a well-known algorithm for performing semi-supervised inference under the large margin principle. In this paper, we are interested in the problem of training a S3VM when the labeled and unlabeled samples are distributed over a network of interconnected agents. In particular, the aim is to design a distributed training protocol over networks, where communication is restricted only to neighboring agents and no coordinating authority is present. Using a standard relaxation of the original S3VM, we formulate the training problem as the distributed minimization of a non-convex social cost function. To find a (stationary) solution in a distributed manner, we employ two different strategies: (i) a distributed gradient descent algorithm; (ii) a recently developed framework for In-Network Nonconvex Optimization (NEXT), which is based on successive convexifications of the original problem, interleaved by state diffusion steps. Our experimental results show that the proposed distributed algorithms have comparable performance with respect to a centralized implementation, while highlighting the pros and cons of the proposed solutions. To the date, this is the first work that paves the way toward the broad field of distributed semi-supervised learning over networks.

An aggregate and iterative disaggregate algorithm with proven optimality in machine learning

We propose a clustering-based iterative algorithm to solve certain optimization problems in machine learning, where we start the algorithm by aggregating the original data, solving the problem on aggregated data, and then in subsequent steps gradually disaggregate the aggregated data. We apply the algorithm to common machine learning problems such as the least absolute deviation regression problem, support vector machines, and semi-supervised support vector machines. We derive model-specific data aggregation and disaggregation procedures. We also show optimality, convergence, and the optimality gap of the approximated solution in each iteration. A computational study is provided.

A New Conic Approach to Semisupervised Support Vector Machines

We propose a completely positive programming reformulation of the 2-norm soft marginS3VMmodel. Then, we construct a sequence of computable cones of nonnegative quadratic forms over a union of second-order cones to approximate the underlying completely positive cone. Anϵ-optimal solution can be found in finite iterations using semidefinite programming techniques by our method. Moreover, in order to obtain a good lower bound efficiently, an adaptive scheme is adopted in our approximation algorithm. The numerical results show that the proposed algorithm can achieve more accurate classifications than other well-known conic relaxations of semisupervised support vector machine models in the literature.

Conic Relaxations for Semi-supervised Support Vector Machines

Semi-supervised support vector machines arise in machine learning as a model of mixed integer programming problem for classification. In this paper, we propose two convex conic relaxations for the original mixed integer programming problem. The first one is a new semi-definite relaxation, and its possibly maximal ratio of the optimal value is estimated approximately. The second one is a doubly nonnegative relaxation, which is relaxed from a well-known conic programming problem called completely positive programming problem that is equivalent to the original problem. Furthermore, we prove that the doubly nonnegative relaxation is tighter than the semi-definite relaxation. Finally, the numerical results show that two proposed relaxations not only generate proper classifiers but also outperform some existing methods in classification accuracy.