Twin Bounded Support Vector Research Articles

Robust twin bounded support vector machines for outliers and imbalanced data

Truncated loss functions are robust to class noise and outliers. A robust twin bounded support vector machine is proposed in this paper that truncates the growth of its loss functions at a pre-specified point, thus, flattens the function that pre-specified score afterwards. Moreover, to make the proposed method capable of handling datasets with different imbalance ratio, cost-sensitive learning is implemented by scaling the total error of the classes based on the number of samples of each class. However, the adopted loss functions take a non-convex structure which does not always assure global optimum. To handle this issue, we suggest concave-convex procedure (CCCP) to ensure global convergence by decomposing the cost functions into additions of one convex and one concave part. The behaviour of the proposed method for varied imbalance ratio is analysed experimentally and depicted graphically. Classification performance of the proposed method in terms of AUC, F-Measure and G-Mean is compared with other related methods, viz. Hinge loss support vector machine (SVM), Ramp loss SVM (RSVM), twin SVM (TWSVM), twin bounded SVM (TBSVM), Pinball loss SVM (pin-SVM), entropy-based fuzzy SVM (EFSVM), non-parallel hyperplane Universum SVM (U-NHSVM), stochastic gradient twin support vector machine (SGTSVM), k-nearest neighbor (KNN)-based maximum margin and minimum volume hyper-sphere machine (KNN-M3VHM) and affinity and class probability-based fuzzy SVM (ACFSVM) on several real-world datasets with imbalance ratio raging from low to high. Further, to establish the significance of the proposed method in pattern classification, pair-wise statistical comparison of the methods is performed based on their average ranks on AUC. Experimental results are convincing.

Decoding of voluntary and involuntary upper-limb motor imagery based on graph fourier transform and cross-frequency coupling coefficients

Objective.Brain-computer interface (BCI) technology based on motor imagery (MI) control has become a research hotspot but continues to encounter numerous challenges. BCI can assist in the recovery of stroke patients and serve as a key technology in robot control. Current research on MI almost exclusively focuses on the hands, feet, and tongue. Therefore, the purpose of this paper is to establish a four-class MI BCI system, in which the four types are the four articulations within the right upper limbs, involving the shoulder, elbow, wrist, and hand. Approach.Ten subjects were chosen to perform nine upper-limb analytic movements, after which the differences were compared in P300, movement-related potentials(MRPS), and event-related desynchronization/event-related synchronization under voluntary MI (V-MI) and involuntary MI (INV-MI). Next, the cross-frequency coupling (CFC) coefficient based on mutual information was extracted from the electrodes and frequency bands with interest. Combined with the image Fourier transform and twin bounded support vector machine classifier, four kinds of electroencephalography data were classified, and the classifier’s parameters were optimized using a genetic algorithm. Main results.The results were shown to be encouraging, with an average accuracy of 93.2% and 92.2% for V-MI and INV-MI, respectively, and over 95% for any three classes and any two classes. In most cases, the accuracy of feature extraction using the proximal articulations as the basis was found to be relatively high and had better performance. Significance.This paper discussed four types of MI according to three aspects under two modes and classed them by combining graph Fourier transform and CFC. Accordingly, the theoretical discussion and classification methods may provide a fundamental theoretical basis for BCI interface applications.

Smooth Twin Bounded Support Vector Machine with Pinball Loss

The twin support vector machine improves the classification performance of the support vector machine by solving two smaller quadratic programming problems. However, for the twin support vector machine and some of its variants, the constructed models are usually transformed from the original space into the dual space to obtain the solutions. Meanwhile, the hinge loss function used in above models is sensitive to noise and unstable in resampling. In order to further improve the performance of the twin support vector machine, the pinball loss function is introduced into the twin bounded support vector machine directly, and the non-differentiable problem of the pinball loss function at zero is solved by constructing a smooth approximation function. Based on this, a smooth twin bounded support vector machine model with pinball loss is obtained. The model is solved iteratively in the original space by using the Newton-Armijo method, then a smooth twin bounded support vector machine with pinball loss algorithm is proposed. In the experiments, the proposed twin support vector machine is validated on the UCI datasets, which shows the effectiveness of the proposed algorithm.

Capped [formula omitted]-norm distance metric-based fast robust twin bounded support vector machine

In this paper, to improve the performance of capped L1-norm twin support vector machine (CTSVM), we first propose a new robust twin bounded support vector machine (RTBSVM) by introducing the regularization term. The significant advantage of our RTBSVM over CTSVM is that the structural risk minimization principle is implemented. This embodies the marrow of statistical learning theory, so this modification can improve the performance of classification. Furthermore, to accelerate the computation of RTBSVM and simultaneously inherit the merit of robustness, we construct a least squares version of RTBSVM (called RTBSVM). This formulation leads to a simple and fast algorithm for binary classifiers by solving just two systems of linear equations. Finally, we derive two simple and effective iterative optimization algorithms for solving RTBSVM and FRTBSVM, respectively. Simultaneously, we theoretically rigorously analyze and prove the computational complexity, local optimality and convergence of the algorithms. Experimental results on one synthetic dataset and nine UCI datasets demonstrate that our methods are competitive with other methods. Additionally, the FRTBSVM is directly applied to recognize the purity of hybrid maize seeds using near-infrared spectral data. Experiments show that our method achieves better performance than the traditional methods in most spectral regions.

Medical data set classification using a new feature selection algorithm combined with twin-bounded support vector machine.

Early diagnosis and treatment are the most important strategies to prevent deaths from several diseases. In this regard, data mining and machine learning techniques have been useful tools to help minimize errors and to provide useful information for diagnosis. Our paper aims to present a new feature selection algorithm. In order to validate our study, we used eight benchmark data sets which are commonly used among researchers who developed machine learning methods for medical data classification. The experiment has shown that the performance of our proposed new feature selection method combined with twin-bounded support vector machine (FSTBSVM) is very efficient. The robustness of the FSTBSVM is examined using classification accuracy, analysis of sensitivity, and specificity. The proposed FSTBSVM is a very promising technique for classification, and the results show that the proposed method is capable of producing good results with fewer features than the original data sets. Graphical abstract Model using a new feature selection and grid search with 10-fold CV to optimize model parameters in our FSTBSVM.

Oblique Decision Tree Ensemble via Twin Bounded SVM

Ensemble methods with “perturb and combine” strategy have shown improved performance in the classification problems. Recently, random forest algorithm was ranked one among 179 classifiers evaluated on 121 UCI datasets. Motivated by this, we propose a new approach for the generation of oblique decision trees. At each non-leaf node, the training data samples are grouped in two categories based on the Bhattachrayya distance with randomly selected feature subset. Then, twin bounded support vector machine (TBSVM) is used to get two clustering hyperplanes such that each hyperplane is closer to data points of one group and as far as possible from the data points of other group. Based on these hyperplanes, each non-leaf node is splitted to generate the decision tree. In this paper, we used different base models like random forest (RaF), rotation forest (RoF), random sub rotation forest (RRoF) to generate the different oblique decision tree forests named as TBRaF, TBRoF and TBRRoF, respectively. In earlier oblique decision trees, like multisurface proximal support vector machine (MPSVM) based oblique decision trees, matrices are semi-positive definite and hence different regularization methods are required. However, no explicit regularization techniques need to be applied to the primal problems as the matrices in the proposed TBRaF, TBRoF and TBRRoF are positive definite. We evaluated the performance of the proposed models (TBRaF, TBRoF and TBRRoF) on 49 datasets taken from the UCI repository and on some real-world biological datasets (not in UCI). The experimental results and statistical tests conducted show that TBRaF and TBRRoF outperform other baseline methods.

Entropy-Based Fuzzy Twin Bounded Support Vector Machine for Binary Classification

Twin support vector machine (TWSVM) is a new machine learning method, as opposed to solving a single quadratic programming problem in support vector machine (SVM), which generates two nonparallel hyperplanes by solving two smaller size quadratic programming problems. However, the TWSVM obtains the final classifier by giving the same importance to all training samples which may be important for classification performance. In order to address this problem, in this paper, we propose a novel entropy-based fuzzy twin bounded support vector machine (EFTBSVM) for binary classification problems. By considering the fuzzy membership value for each sample and assigning it based on the entropy value, the samples with higher class certainty are assigned to relatively larger fuzzy membership. In addition, the proposed EFTBSVM not only maintains the superior characteristics of the TWSVM but also exploits the structural risk minimization principle by introducing a regularization term. The experimental results achieved on synthetic datasets and benchmark datasets illustrate the effectiveness of the proposed method.

Twin Bounded Weighted Relaxed Support Vector Machines

Data distribution has an important role in classification. The problem of imbalanced data has occurred when the distribution of one class, which usually attends more interest, is negligible compared with other class. Furthermore, by the existence of outliers and noise, the classification of these data confronts more challenges. Despite these challenges, doing fast classification with good performance is desired. One of the successful classifier methods for dealing with imbalanced data and outliers is weighted relaxed support vector machines (WRSVMs). In this paper, the improved twin version of this classifier, which is called twin-bounded weighted relaxed support vector machines, is introduced to confront the mentioned challenges; besides, it performs in a significant fast manner and it is more accurate in most cases. This method benefits from the fast classification manner of twin-bounded support vector machines and outlier robustness capability of WRSVM in the imbalanced problems. The experimentally, the proposed method is compared with the WRSVM and other standard SVM-based methods on the public benchmark datasets. The results confirm the efficiency of the proposed method.

Clustering by twin support vector machine and least square twin support vector classifier with uniform output coding

The recently proposed twin support vector clustering (TWSVC) is a powerful clustering method. However, TWSVC may encounter the singularity problem and it is time consuming in the learning stage. In this paper, we introduce some efficient techniques into TWSVC, and propose two clustering models, called twin bounded support vector clustering (TBSVC) and least square twin bounded support vector clustering (LSTBSVC), respectively. TBSVC introduces a maximum margin regularization term into TWSVC, which not only avoids its singularity but also significantly improves the performance. LSTBSVC introduces the least square formation into TBSVC to greatly accelerate its learning speed. Moreover, a uniform output coding for LSTBSVC is introduced to cope with the non-uniformed problem in the learning procedures. In addition, nonlinear clustering is also extended to the above clustering methods by using the kernel trick. Experimental results show the effectiveness and efficiency of our methods.

Alternating Relaxed Twin Bounded Support Vector Clustering

Maximum margin clustering (MMC) and its improved version are based on the spirit of support vector machine, which will inevitably lead to prohibitively computational complexity when these learning models are encountered with an enormous amount of patterns. To accelerate the clustering efficiency, we propose alternating twin bounded support vector clustering to decompose the original large problem in MMC and its variants into two smaller sized ones, in which solving expensive semi-definite programming is avoided by performing alternating optimization between cluster-specific model parameters and instance-specific labeling assignments. Also the structural risk minimization principle is implemented to obtain good generalization. Additionally, in order to avoid premature convergence, a relaxed version of our algorithm is proposed, in which the hinge loss in the original twin bounded support vector machine is replaced with the Laplacian loss. These two versions can be easily extended to the nonlinear context via kernel tricks. To investigate the efficacy of our clustering algorithm, several experiments are conducted on a number of synthetic and real-world datasets. Experimental results demonstrate that the proposed method has better performance than other existing clustering approaches in terms of clustering accuracy and time efficiency and also possesses the powerful ability to process larger-scaled datasets.

Least squares twin bounded support vector machines based on L1-norm distance metric for classification

In this paper, we construct a least squares version of the recently proposed twin bounded support vector machine (TBSVM) for binary classification. As a valid classification tool, TBSVM attempts to seek two non-parallel planes that can be produced by solving a pair of quadratic programming problems (QPPs), but this is time-consuming. Here, we solve two systems of linear equations rather than two QPPs to avoid this deficiency. Furthermore, the distance in least squares TBSVM (LSTBSVM) is measured by L2-norm, but L1-norm distance is usually regarded as an alternative to L2-norm to improve model robustness in the presence of outliers. Inspired by the advantages of least squares twin support vector machine (LSTWSVM), TBSVM and L1-norm distance, we propose a LSTBSVM based on L1-norm distance metric for binary classification, termed as L1-LSTBSVM, which is specially designed for suppressing the negative effect of outliers and improving computational efficiency in large datasets. Then, we design a powerful iterative algorithm to solve the L1-norm optimal problems, and it is easy to implement and its convergence to an optimum solution is theoretically ensured. Finally, the feasibility and effectiveness of L1-LSTBSVM are validated by extensive experimental results on both UCI datasets and artificial datasets.

Multiple TBSVM-RFE for the detection of architectural distortion in mammographic images

Breast cancer is a leading health threaten for women in the world. Among the several abnormalities observable on mammograms, architecture distortion is one of the most difficult to detect due to its subtlety. Computer-Aided Diagnosis (CAD) technology has been widely used for the detection and diagnosis of breast cancer. In this paper, a new automatic architectural distortion detection method for breast cancer in mammographic images is proposed. Firstly, Gabor filters and phase portrait analysis are used to locate the suspicious regions based on the image characteristic of architectural distortion. Twin bounded Support Vector Machine (TBSVM) is employed to reduce the large amounts of false positives. TBSVM is a kind of binary classifier, which has advantages in both computation efficiency and generalization when dealing with binary classification. For each suspicious region, several features are extracted. However, not every extracted feature contributes to the classification accuracy. We proposed a novel feature selection method for TBSVM and utilized it for the architectural distortion detection in mammograms, named Multiple Twin Bound Support Vector Machines Recursive Feature Elimination (MTBSVM-RFE). The results showed that our proposed method detect the region of architecture distortion with high accuracy.

A new fuzzy twin support vector machine for pattern classification

Fuzzy SVM is often used to solve the problem that patterns belonging to one class often play more significant roles in classification. In order to improve the efficiency and performance of fuzzy SVM, this paper proposes a new fuzzy twin support vector machine (NFTSVM) for binary classification, in which fuzzy neural networks and twin support vector machine (TWSVM) are incorporated. By design, the influence of the samples with high uncertainty can be mitigated by employing fuzzy membership to weigh the margin of each training sample, which improves the generalization ability. In addition, we show that the existing TWSVM and twin bounded support vector machines (TBSVM) are special cases of the proposed NFTSVM when the parameters of NFTSVM are appropriately selected. Moreover, the successive overrelaxation (SOR) technique is adopted to solve the quadratic programming problems (QPPs) in the proposed NFTSVM algorithm to speed up the training procedure. Experimental results obtained on several artificial and real-world datasets validate the feasibility and effectiveness of the proposed method.

Online Learning Algorithms for Double-Weighted Least Squares Twin Bounded Support Vector Machines

Twin support vector machine with two nonparallel classifying hyperplanes and its extensions have attracted much attention in machine learning and data mining. However, the prediction accuracy may be highly influenced when noise is involved. In particular, for the least squares case, the intractable computational burden may be incurred for large scale data. To address the above problems, we propose the double-weighted least squares twin bounded support vector machines and develop the online learning algorithms. By introducing the double-weighted mechanism, the linear and nonlinear double-weighted learning models are proposed to reduce the influence of noise. The online learning algorithms for solving the two models are developed, which can avoid computing the inverse of the large scale matrices. Furthermore, a new pruning mechanism which can avoid updating the kernel matrices in every iteration step for solving nonlinear model is also developed. Simulation results on three UCI data with noise demonstrate that the online learning algorithm for the linear double-weighted learning model can get least computation time as well considerable classification accuracy. Simulation results on UCI data and two-moons data with noise demonstrate that the nonlinear double-weighted learning model can be effectively solved by the online learning algorithm with the pruning mechanism.

A Novel Twin Support Vector Machine for Binary Classification Problems

Based on the recently proposed twin support vector machine and twin bounded support vector machine, in this paper, we propose a novel twin support vector machine (NTSVM) for binary classification p...

Twin Bounded Support Tensor Machine for Classification

The traditional vector-based classifiers, such as support vector machine (SVM) and twin support vector machine (TSVM), cannot handle tensor data directly and may not utilize the data informations effectively. In this paper, we propose a novel classifier based on tensor data, called twin bounded support tensor machine (TBSTM) which is an extension of twin bounded support vector machine (TBSVM). Similar to TBSVM, TBSTM gets two hyperplanes and obtains the solution by solving two quadratic programming problems (QPPs). The computational complexity of each QPPs is smaller than that of support tensor machine (STM). TBSTM not only retains the advantage of TBSVM, but also has its unique superior characteristics: (1) it makes full use of the structure information of data; (2) it has acceptable or better classification accuracy compared to STM, TBSVM and SVM; (3) the computational cost is basically less than STM; (4) it can deal with large data that TBSVM is not easy to achieve, especially for small-sample-size (S3) problems; (5) it adopts alternating successive over relaxation iteration (ASOR) method to solve optimization problems which accelerates the pace of training. Finally, we demonstrate the effectiveness and superiority by the experiments based on vector and tensor data.

Combined outputs framework for twin support vector machines

Twin support vector machine (TWSVM) is regarded as a milestone in the development of powerful SVMs. However, there are some inconsistencies with TWSVM that can lead to many reasonable modifications with different outputs. In order to obtain better performance, we propose a novel combined outputs framework that combines rational outputs. Based on this framework, an optimal output model, called the linearly combined twin bounded support vector machine (LCTBSVM), is presented. Our LCTBSVM is based on the outputs of several TWSVMs, and produces the optimal output by solving an optimization problem. Furthermore, two heuristic algorithms are suggested in order to solve the optimization problem. Our comprehensive experiments show the superior generalization performance of our LCTBSVM compared with SVM, PSVM, GEPSVM, and some current TWSVMs, thus confirming the value of our theoretical analysis approach.

Improvements on Twin Support Vector Machines

For classification problems, the generalized eigenvalue proximal support vector machine (GEPSVM) and twin support vector machine (TWSVM) are regarded as milestones in the development of the powerful SVMs, as they use the nonparallel hyperplane classifiers. In this brief, we propose an improved version, named twin bounded support vector machines (TBSVM), based on TWSVM. The significant advantage of our TBSVM over TWSVM is that the structural risk minimization principle is implemented by introducing the regularization term. This embodies the marrow of statistical learning theory, so this modification can improve the performance of classification. In addition, the successive overrelaxation technique is used to solve the optimization problems to speed up the training procedure. Experimental results show the effectiveness of our method in both computation time and classification accuracy, and therefore confirm the above conclusion further.