Decision Hyperplane Research Articles

Geometric relative margin machine for heterogeneous distribution and imbalanced classification

This paper introduces the geometric relative margin machine (GRMM) to address class imbalance and heterogeneous data distributions in classification tasks. GRMM innovatively incorporates boundary information as distribution information and implements differentiated misclassification cost strategies for each class, allowing adaptive adjustments to the decision hyperplane and enhancing robustness against outliers. The model employs a block iterative algorithm for optimization. Experimental results on artificial, benchmark, and credit risk assessment datasets demonstrate that GRMM significantly outperforms other benchmark methods in terms of classification accuracy and robustness. This research provides a comprehensive and effective solution to imbalanced classification problems, with potential applications in real-world scenarios, such as credit assessment.

Ensemble One-Class Support Vector Machine for Sea Surface Target Detection Based on k-Means Clustering

Sea surface target detection is a key stage in a typical target detection system and directly influences the performance of the whole system. As an effective discriminator, the one-class support vector machine (OCSVM) has been widely used in target detection. In OCSCM, training samples are first mapped to the hypersphere in the kernel space with the Gaussian kernel function, and then, a linear classification hyperplane is constructed in each cluster to separate target samples from other classes of samples. However, when the distribution of the original data is complex, the transformed data in the kernel space may be nonlinearly separable. In this situation, OCSVM cannot classify the data correctly, because only a linear hyperplane is constructed in the kernel space. To solve this problem, a novel one-class classification algorithm, referred to as ensemble one-class support vector machine (En-OCSVM), is proposed in this paper. En-OCSVM is a hybrid model based on k-means clustering and OCSVM. In En-OCSVM, training samples are clustered in the kernel space with the k-means clustering algorithm, while a linear decision hyperplane is constructed in each cluster. With the combination of multiple linear classification hyperplanes, a complex nonlinear classification boundary can be achieved in the kernel space. Moreover, the joint optimization of the k-means clustering model and OCSVM model is realized in the proposed method, which ensures the linear separability of each cluster. The experimental results based on the synthetic dataset, benchmark datasets, IPIX datasets, and SAR real data demonstrate the better performance of our method over other related methods.

Accelerated gradient descent using improved Selective Backpropagation

An improved version of the Selective Backpropagation (SBP+) is described which significantly reduces the training time and enhances the generalization. The algorithm eliminates correctly predicted instances from the Backpropagation (BP) in an iteration. The minimum useful content level to define a correct prediction is determined using the hyperparameter c. A new performance measure dm is introduced, which denotes the average of the fraction of remaining instances in the BP during an epoch. With a proper selection of c, a model can be adjusted to learn dominant patterns (mostly found in instances located further away from a decision hyperplane) and skip minor details (the very details which are due to the random error and can be found in instances close to a decision hyperplane). Since the SBP+ assigns a lower correct softmax score to instances located closer to a decision hyperplane, it is shown that — under certain conditions — a Neural Networks (NN) standard crisp output can be converted to a fuzzy membership scores set. The regulating effect of the content level c is compared with other standard techniques and its advantages are outlined. Lastly, the statistical process leading to minimization of the useful total error is explicated and the SBP+ ability to pinpoint outliers is illustrated with an example.

Research on roller bearing fault diagnosis method based on flexible dynamic adjustable strategy under data imbalance

In engineering practice, the collection of equipment vibration signals is prone to interference from the external environment, resulting in abnormal data and imbalanced data in different states. Traditional support vector machine, support matrix machine and other methods have advantages in balancing sample classification, but have limitations in obtaining low rank information, making it difficult to perform classification tasks under data imbalance. Therefore, a novel classification method that targets matrices as the input, called flexible dynamic matrix machine (FDMM), is proposed in this paper. First, FDMM establishes a regularization term using a flexible low-rank operator and sparse constrain, which can better take into account matrix structure information. Then, the upper bound of the loss function is truncated, reducing the impact of the loss on the construction of the decision hyperplane. Finally, the recognition performance of imbalanced data is improved by adjusting the game values of different categories of samples through dynamic adjustment function. Experimental results demonstrate that superior classification accuracy and generalization performance can be achieved with the FDMM method when applied to two roller bearing datasets.

Quantitative evaluation method of operation reliability of substation relay protection device based on improved neural network algorithm

With the increase in power demand and the increasing complexity of the power grid, the operation of relay protection devices in substations is often unreasonable. Therefore, a quantitative evaluation method of operation reliability of relay protection devices in substations based on an improved neural network algorithm is proposed. The time-varying probability of misoperation of relay protection in substation is calculated based on the improved neural network algorithm, identifying misoperation behavior of relay protection device, obtaining the optimal hyperplane decision function, selecting kernel function, and construct a quantitative evaluation function of operation reliability of relay protection device. In the evaluation process, two stages of the evaluation process are carried out, namely, reliability block diagram analysis, failure mode evaluation, and verification of relay protection device operation state, so as to complete the reliability quantitative evaluation. The experimental results show that this method can minimize the error of evaluation training and improve evaluation efficiency. At the same time, the error rate is small, which can be controlled within 0.5%.

Realtime Anomaly Detection in Healthcare IoT: A Machine Learning-Driven Security Framework

Healthcare IoT, a fast-growing field, could revolutionize patient monitoring and intervention. This interconnection raises new security concerns, requiring real-time anomaly detection to protect patient data and device integrity. This study presents a novel security framework that uses combination of Hidden Markov Models (HMM) and Support Vector Machine (SVM) to detect anomalies in real-time healthcare IoT environments with high accuracy. The framework prioritizes real-time strength and efficiency. Sensor data from wearables, medical devices, and other IoT devices is carefully segmented into time intervals. Features are carefully derived from each segment, including statistical summaries, patterns, and frequency domain characteristics. Feature engineering is essential for accurate anomaly detection. Integrating HMM and SVM capabilities is the framework’s core. HMM accurately represent concealed states within divided data sequences and analyze patterns of change over time. After that, an independent-trained SVM examines each state. Using proximity to a decision hyperplane in feature space, this SVM can classify data points as normal or anomalous. This method augments HMM temporal capabilities with SVM classification efficiency, increasing sensitivity to anomalous patterns and reducing false positives. The framework’s exceptional performance is shown by extensive evaluations on the PhysioNet Challenge 2017 dataset, which includes diverse ECG recordings with labeled anomalies. HMM-SVM outperforms Naive Bayes, LSTM, and Random Forest with 98.66% accuracy. The framework also has high precision, recall, and F1-score, indicating a refined ability to detect real anomalies and reduce false alarms. The framework prioritizes real-time understanding and application alongside its remarkable precision. HMM-SVMs reveal hidden data state changes, revealing context and possible causes of anomalies. Modular design and efficient algorithms enable seamless integration of real-time functionality in low-resource IoT devices, enabling quick and effective security responses. To conclude, this study introduces an HMM-SVM framework for fast Healthcare IoT abnormality detection. The framework emphasizes comprehensibility and real-time applicability while achieving high accuracy. This framework can protect patient data, improve device security, and create a more reliable healthcare IoT ecosystem.

SUPPORT VECTOR MACHINE MODELS FOR DIABETIC RETINOPATHY DIAGNOSIS AND GRADING

This work applies five variants of the support vector machine to the classification of the various stages of nonproliferative diabetic retinopathy. Four hundred eye fundus images from the Messidor repository are preprocessed and thirteen features extracted. The features best suited as inputs for support vector machine classification are identified. Initially, binary classification of severe nonproliferative diabetic retinopathy alone versus a normal eye is performed, achieving an accuracy of 97.44% using the standard support vector machine with Gaussian kernel and when optimized for accuracy. When optimized for sensitivity, the twin bounded support vector machine achieves the highest sensitivity of 99.06%. Then multiclass grading into all four stages of nonproliferative diabetic retinopathy is performed. Best performance with regards to four performance metrics, namely accuracy, sensitivity, specificity, and precision is achieved with the twin bounded support vector machine variant, when one-versus-one decision configuration is used in combination with a novel decision strategy that includes accumulated distances from the decision hyperplanes in the decision algorithm. The results compare favorably with data published in the literature.

HOMDA: High-Order Moment-Based Domain Alignment for unsupervised domain adaptation

Unsupervised domain adaptation aims to annotate unlabeled target domain samples by utilizing transferable knowledge learned from the source domain. Optimal transport (OT) has become a popular probability metric to measure the distribution discrepancy across domains. However, most OT-based methods inevitably suffer from false matching problem, resulting in poor domain alignment. Furthermore, because the domain shift cannot be completely eliminated, some target domain data distributed near cluster edges or far from their corresponding class centers are highly likely to be misclassified by the decision hyperplane learned from the source samples. We propose a High-Order Moment-Based Domain Alignment (HOMDA) method that uses an improved partial optimal transport (POT) strategy and a discriminative centroid-wise clustering regularization scheme to solve these issues. The improved POT strategy performs fine-grained domain alignment by limiting the amount of masses during mapping. The centroid-wise clustering scheme can promote the shared features with better intra-class compactness and inter-class separability, effectively reducing negative transfer caused by hard-aligned target samples. To better capture discriminative representations, HOMDA also includes a high-order statistics mixer. Extensive experiments show that our proposal can improve adaptation performance significantly and is competitive with state-of-the-art methods.

TSVM-M[formula omitted]: Twin support vector machine based on multi-order moment matching for large-scale multi-class classification

For multi-class classification, many existing methods, such as multiple weighted linear loss twin support vector machine (MWLTSVM), construct multiple decision hyperplanes by minimizing the positive points loss’s first-order moment (mean), which may lead to sensitivity to outliers. Also, when faced with a large-scale classification problem, how to speed up the process of solving the optimization model is also a challenge. An alternative is to use rectangular kernel technology (RKT) to reduce computational complexity. However, RKT is based on the uniform point selection method, which can be proven to be ineffective in improving classifier performance. To address these problems, a novel classifier under the structure of “one-versus-rest” for multi-class classification is proposed in this paper, named twin support vector machine based on multi-order moment matching (TSVM-M3). When constructing the decision hyperplanes, TSVM-M3 takes the first-order and second-order moments (mean and variance) of positive points loss into consideration and implements this by introducing an adjusting factor into the objective function. A theoretical analysis of the robustness of the proposed TSVM-M3 is also provided. Meanwhile, a novel RKT based on the density-dependent data selection method is proposed for large-scale classification. We demonstrate that the proposed RKT can benefit from reducing modeling error. Experimental results show the effectiveness of the proposed TSVM-M3.

(Retracted) Support vector machine parallelized remote sensing image classification algorithm based on big data

With the development of big data technology, machine learning classification methods have been widely used in the classification and recognition of remote sensing images. For remote sensing big data, how to quickly and efficiently use machine learning classification algorithms to classify remote sensing images is an urgent problem. It is a general term for the theory, method, technology, and activities of obtaining valuable information based on massive remote sensing data sets, synthesizing auxiliary data from other sources, and using big data thinking and means. The purpose of this paper is to study the support vector machine (SVM) parallelized remote sensing image classification algorithm based on big data. We propose a parallel nesting of GPU in MPI multiprocesses based on the big data framework, which can more effectively improve the calculation processing speed and build a high-performance SVM parallel computing framework based on the big data framework. The optimization problem of SVM considers both empirical risk and structural risk minimization and requires maximum edge distance when constructing hyperplane decision boundaries, so there is ample space between the interval boundaries to accommodate the test samples. Based on this framework, we improve the machine learning SVM algorithm and realize the high-performance parallel computing of SVM classification algorithm on this platform. It is an efficient hybrid parallel mode to nest GPU in MPI multiprocess in parallel. When the number of nodes is 2, 4, and 6, the speedup of the SVM classification algorithm is 1.52, 2.24, and 2.55.

Learning Pinball TWSVM efficiently using Privileged Information and its Applications

Expert knowledge plays a vital role in any learning framework. However, in the field of machine learning, an expert's knowledge is rarely utilised. Furthermore, most machine learning methods (support vector machine, SVM-based) use a hinge loss function that is sensitive to noise. Thus, in order to benefit from expert knowledge while reducing noise sensitivity, we propose in this paper a fast and novel Twin Support Vector Machine classifier based on privileged information with a pinball loss function, dubbed Pin-TWSVMPI, where expert knowledge is in the form of privileged information. The proposed Pin-TWSVMPI incorporates privileged information into two nonparallel decision hyperplanes by employing a correction function. Furthermore, we employ the Sequential Minimal Optimization (SMO) technique to get the classifier in order to make computations more efficient and faster, and we have demonstrated its applicability for pedestrian detection and handwritten digit recognition. Furthermore, for UCI datasets, we first construct a process that retrieves privileged information from the dataset's features, which is then used by Pin-TWSVMPI, resulting in improved classification accuracy with a reduced computing time.

Learning Pinball TWSVM efficiently using Privileged Information and their applications

In any learning framework, an expert knowledge always plays a crucial role. But, in the field of machine learning, the knowledge offered by an expert is rarely used. Moreover, machine learning algorithms (SVM based) generally use hinge loss function which is sensitive towards the noise. Thus, in order to get the advantage from an expert knowledge and to reduce the sensitivity towards the noise, in this paper, we propose a fast and novel Twin Support Vector Machine classifier based on privileged information with pinball loss function which has been termed as Pin-TWSVMPI where expert's knowledge is in the form of privileged information. The proposed Pin-TWSVMPI incorporates privileged information by using correcting function so as to obtain two nonparallel decision hyperplanes. Further, in order to make computations more efficient and fast, we use Sequential Minimal Optimization (SMO) technique for obtaining the classifier and have also shown its application for Pedestrian detection and Handwritten digit recognition. Further, for UCI datasets, we first implement a procedure which extracts privileged information from the features of the dataset which are then further utilized by Pin-TWSVMPI to which lead to enhancement in classification accuracy with comparatively lesser computational time.

Probability-driven scoring functions in combining linear classifiers

Although linear classifiers are one of the oldest methods in machine learning, they are still very popular in the machine learning community. This is due to their low computational complexity and robustness to overfitting. Consequently, linear classifiers are often used as base classifiers of multiple ensemble classification systems. This research is aimed at building a new fusion method dedicated to the ensemble of linear classifiers. The fusion scheme uses both measurement space and geometrical space. Namely, we proposed a probability-driven scoring function which shape depends on the orientation of the decision hyperplanes generated by the base classifiers. The proposed fusion method is compared with the reference method using multiple benchmark datasets taken from the KEEL repository. The comparison is done using multiple quality criteria. The statistical analysis of the obtained results is also performed. The experimental study shows that, under certain conditions, some improvement may be obtained.

Patch-based Medical Image Segmentation using Matrix Product State Tensor Networks

Tensor networks are efficient factorisations of high dimensional tensors into network of lower order tensors. They have been most commonly used to model entanglement in quantum many-body systems and more recently are witnessing increased applications in supervised machine learning. In this work, we formulate image segmentation in a supervised setting with tensor networks. The key idea is to first lift the pixels in image patches to exponentially high dimensional feature spaces and using a linear decision hyper-plane to classify the input pixels into foreground and background classes. The high dimensional linear model itself is approximated using the matrix product state (MPS) tensor network. The MPS is weight-shared between the non-overlapping image patches resulting in our <em>strided tensor network</em> model. The performance of the proposed model is evaluated on three three 2D- and one 3D- biomedical imaging datasets. The performance of the proposed tensor network segmentation model is compared with relevant baseline methods. In the 2D experiments, the tensor network model yeilds competitive performance compared to the baseline methods while being more resource efficient.

A minority oversampling approach for fault detection with heterogeneous imbalanced data

Between-class imbalance and feature heterogeneity commonly coexist in monitoring data collected from engineering systems. The decision hyperplanes of data-driven methods when adopted for fault detection with imbalanced data may be biased to the majority class, resulting in a low fault-detection rate. Various data- and algorithm-level methods have been proposed, with minority oversampling methods among the most popular and successful. However, state-of-the-art minority oversampling methods are unsuitable for imbalanced data with heterogeneous features, including both numeric and nominal variables. There are two main drawbacks: 1) taking a nominal variable as a numeric variable is not trivial, and synthetic minority samples may exceed the value range of nominal variables; 2) conventional distance measures, e.g., Euclidean distance, cannot properly measure the similarity of samples with heterogeneous features. For these considerations, this work proposes new fault-detection methods. The methodological contributions include: 1) two different distance measures adopted for heterogeneous features in the minority class; 2) a new method for coordinate calculation of synthetic samples considering feature heterogeneity; and 3) a new strategy to encode nominal variables into numeric data for data-driven models. Several public heterogeneous imbalanced datasets and a real case study considering fault detection in high-speed trains are considered to verify the effectiveness of the proposed methods. To the knowledge of the author, this is also the first time that the effectiveness of diverse oversampling methods on heterogeneous imbalance data are specifically discussed.

Porphyry Cu-Au prospectivity modelling using semi-supervised learning algorithm in Dehsalm district, eastern Iran

Porphyry Cu-Au prospectivity modelling using semi-supervised learning algorithm in Dehsalm district, eastern Iran

Can Boosted Randomness Mimic Learning Algorithms of Geometric Nature? Example of a Simple Algorithm That Converges in Probability to Hard-Margin SVM.

In the light of the general question posed in the title, we write down a very simple randomized learning algorithm, based on boosting, that can be seen as a nonstationary Markov random process. Surprisingly, the decision hyperplanes resulting from this algorithm converge in probability to the exact hard-margin solutions of support vector machines (SVMs). This fact is curious because the hard-margin hyperplane is not a statistical solution, but a purely geometric one-driven by margin maximization and strictly dependent on particular locations of some data points that are placed in the contact region of two classes, namely the support vectors. The proposed algorithm detects support vectors probabilistically, without being aware of their geometric definition. We give proofs of the main convergence theorem and several auxiliary lemmas. The analysis sheds new light on the relation between boosting and SVMs and also on the nature of SVM solutions since they can now be regarded equivalently as limits of certain random trajectories. In the experimental part, correctness of the proposed algorithm is verified against known SVM solvers: libsvm, liblinear, and also against optimization packages: cvxopt (Python) and Wolfram Mathematica.

Noniterative Sparse LS-SVM Based on Globally Representative Point Selection.

A least squares support vector machine (LS-SVM) offers performance comparable to that of SVMs for classification and regression. The main limitation of LS-SVM is that it lacks sparsity compared with SVMs, making LS-SVM unsuitable for handling large-scale data due to computation and memory costs. To obtain sparse LS-SVM, several pruning methods based on an iterative strategy were recently proposed but did not consider the quantity constraint on the number of reserved support vectors, as widely used in real-life applications. In this article, a noniterative algorithm is proposed based on the selection of globally representative points (global-representation-based sparse least squares support vector machine, GRS-LSSVM) to improve the performance of sparse LS-SVM. For the first time, we present a model of sparse LS-SVM with a quantity constraint. In solving the optimal solution of the model, we find that using globally representative points to construct the reserved support vector set produces a better solution than other methods. We design an indicator based on point density and point dispersion to evaluate the global representation of points in feature space. Using the indicator, the top globally representative points are selected in one step from all points to construct the reserved support vector set of sparse LS-SVM. After obtaining the set, the decision hyperplane of sparse LS-SVM is directly computed using an algebraic formula. This algorithm only consumes O(N2) in computational complexity and O(N) in memory cost which makes it suitable for large-scale data sets. The experimental results show that the proposed algorithm has higher sparsity, greater stability, and lower computational complexity than the traditional iterative algorithms.

A soft-margin convex polyhedron classifier for nonlinear task with noise tolerance

As a special form of piecewise linear classifier, the convex polyhedron classifier is simple to implement and achieves rapid response in real-time classification. However, it usually performs badly in the case of high noise where severe boundary intrusion exists. Inspired by the scheme of soft margin in support vector machine, in this paper we propose a soft-margin convex polyhedron classifier for nonlinear classification task. The base (linear) classifier is first generalized to its soft-margin version through kernelization process and slack variables. In each local region, the soft-margin base classifier learns a decision hyperplane with noise tolerance. Then, a series of learned hyperplanes are structurally integrated into a convex polyhedron classifier, which is essentially a convex polyhedron that encloses one class and excludes the other class outside. Experimental results on fifteen benchmark datasets show the proposed soft-margin convex polyhedron classifier is comparable to linear support vector machine and four piecewise linear classifiers, but does not perform as well as the support vector machine with radial basis function kernel in general. When random noises are added to datasets, the soft-margin convex polyhedron classifier achieves similar or better accuracies with the well-known classifiers used for comparison, implying its promising ability of noise tolerance.

Multiple Random Empirical Kernel Learning with Margin Reinforcement for imbalance problems

Imbalance problems arise in real-world applications when the number of negative samples far exceeds the number of positive samples, such as medical data. When solving the classification of imbalance problems, the samples located near the decision hyperplane contribute more to the decision hyperplane, and the samples far from the decision hyperplane contribute less to the decision hyperplane. So we can consider giving higher weights to the samples near the decision hyperplane, but they are sensitive to noise, and too much emphasis on them may lead to unstable performance. This paper proposes a Margin Reinforcement (MR) method to overcome the above dilemma. Because the imbalance problem is a cost-sensitive problem, MR gives positive samples a uniform high weight to improve the misclassification cost of the positive sample. For negative samples, according to their entropy, samples away from the decision surface and noise samples mixed in the positive samples are given a smaller weight, in order to improve the efficiency and robustness of the algorithm. Therefore, MR can emphasize the importance of samples located in overlapping regions of positive and negative classes and ignore the effects of noise samples to produce superior performance. Multiple Random Empirical Kernel Learning (MREKL) has proven to be effective and efficient in dealing with balance problems. In order to improve the performance of MREKL on imbalanced datasets, MR is introduced into MREKL to propose a novel Multiple Random Empirical Kernel Learning with Margin Reinforcement (MREKL-MR). MREKL-MR efficiently map the samples into low-dimensional feature spaces, then utilizes the MR approach to reenforce the importance of margin samples and decrease the effects of noise samples. Experimental results on 28 imbalanced datasets indicate that MREKL-MR is superior to comparison algorithms. Finally, the effectiveness of MREKL-MR in dealing with imbalance problems is verified on the Heart Failure dataset.