Margin Classifier Research Articles

A [formula omitted]-means triangular synthesis large margin classifier with unified pinball loss for imbalanced data

Large Margin Distribution Machine (LDM) improves the Support Vector Machine (SVM) by integrating the marginal distribution of samples into the objective function, which exhibits excellent classification and generalization performance. However, most of the existing LDM-based models exhibit inherent flaws: (1) The assumption of a category uniform distribution, resulting in an inability to effectively address the issue of class imbalance. (2) The inheritance of the hinge loss in SVM, leading to inferior anti-noise performance. Given the shortcomings above, this paper constructs a novel K-means Triangular Synthesis (KTS) method for imbalanced data classification and introduces Unified Pinball (UP) loss to ensure robust performance, demonstrated as a novel KTS large margin classifier with UP Loss (KTS-UPLMC) model. Specifically, the KTS method utilizes the triangular synthesis and generates samples based on the cluster density, effectively mitigating the problems of introducing noise samples and strip-shaped sample distribution in the traditional synthetic minority oversampling technique, further alleviating the LDM’s classification line offset. In addition, the UP Loss considers quantile distance, improving the anti-noise performance of the model. Comparative experiments on artificial datasets and benchmark datasets validate the effectiveness and noise resistance of the proposed KTS-UPLMC in imbalanced data classification problems. Furthermore, the stability of the KTS-UPLMC is substantiated by the parameter sensitivity analysis experiment. In conclusion, the KTS method effectively addresses category imbalance, while the integration of the UP Loss enhances noise resistance.

The misunderstanding of the R Classification—a survey amongst medical specialties treating breast cancer

Frequent discussions in the tumour board about the Residual tumour (R) Classification of the UICC’s “TNM Classification of Malignant Tumours”, especially in the case of breast surgery specimens, raised the question about differing interpretations amongst different medical specialties. Thus, we designed a survey about the R Classification with a special focus on breast cancer specimens. An online survey was conducted, where a web link to the survey was distributed via email to various medical professional societies dealing with breast cancer in Austria and Germany with the request to distribute the link to their members. The study population consisted of physicians of all educational levels of different medical professions, who deal with breast carcinomas in their daily routine. Two hundred two participants, of which 160 (79.2%) have more than 10 years’ professional experience, took part in the survey; 88 (43.6%) were surgeons/gynaecologists, 80 (39.6%) pathologists, 19 (9.4%) radiation oncologists/ therapists, 8 (4.0%) radiologists, and 7 (3.5%) oncologists. We show that the R Classification is not completely mastered by anyone and that there are significant differences in the interpretation of the R Classification between different medical specialties. For better differentiation between the residual tumour (R Classification) of the TNM and a pure resection margin assessment, we suggest the use of a Resection margin (Rm) Classification to avoid further misunderstandings. To assist better multidisciplinary cooperation and to ensure better patient care all medical disciplines should be educated about the actual meaning and correct application of the R Classification.

Using Support Vector Machines to Classify Road Surface Conditions to Promote Safe Driving.

Accurate detection of road surface conditions in adverse winter weather is essential for traffic safety. To promote safe driving and efficient road management, this study presents an accurate and generalizable data-driven learning model for the estimation of road surface conditions. The machine model was a support vector machine (SVM), which has been successfully applied in diverse fields, and kernel functions (linear, Gaussian, second-order polynomial) with a soft margin classification technique were also adopted. Two learner designs (one-vs-one, one-vs-all) extended their application to multi-class classification. In addition to this non-probabilistic classifier, this study calculated the posterior probability of belonging to each group by applying the sigmoid function to the classification scores obtained by the trained SVM. The results indicate that the classification errors of all the classifiers, excluding the one-vs-all linear learners, were below 3%, thereby accurately classifying road surface conditions, and that the generalization performance of all the one-vs-one learners was within an error rate of 4%. The results also showed that the posterior probabilities can analyze certain atmospheric and road surface conditions that correspond to a high probability of hazardous road surface conditions. Therefore, this study demonstrates the potential of data-driven learning models in classifying road surface conditions accurately.

Fuzzy Neighborhood-Based Manifold Learning and Feature Weight Matrix for Multilabel Feature Selection

In recent years, it has been difficult to determine neighborhood radii with almost all feature selection methodologies, and the existing multilabel feature selection approaches neglect these correlations among labels, which results in poor classification efficiency. To address these flaws, a fuzzy neighborhood-based manifold learning model and a feature weight matrix are developed to study multilabel feature selection in this article. First, the difference coefficient of the given samples and the new classification margin of each sample are defined to construct an adaptive fuzzy neighborhood radius. While considering the influences of features and labels on the similarity relationships of samples, a novel fuzzy neighborhood similarity relationship is developed by combining it with the label similarity approach that uses the Jaccard similarity coefficient. Second, the least-squares regression model is introduced to design a loss function for learning the mapping relationship between the feature and label spaces. To constrain the objective function, a regularization term based on the fuzzy neighborhood similarity relationship matrix is developed to establish a manifold learning model. To reduce the complexity of the loss function and to reduce the induced structural risk, the final objective function is constructed by incorporating two graph Laplacian matrices. Third, positive and negative confidence values are presented to incorporate the frequency parameter and to form the dependencies between labels into a new label correlation matrix. A new feature weight matrix integrated with the dependencies between labels is defined to learn the correlations between features and labels. Finally, an embedded feature selection method with an objective function is designed, where the objective function is optimized by the gradient descent scheme. Experiments conducted on 12 multilabel datasets reveal that the proposed approach yields better classification results than do other algorithms.

End-to-end model for automatic seizure detection using supervised contrastive learning

Epilepsy is a chronic neurological disorder characterized by recurrent and unpredictable seizures, caused by abnormal electrical activity in cerebral neurons. Given that it is one of the most common neurological disorders globally, the efficient and accurate automatic seizure detection is urgently needed in the diagnosis of epilepsy to reduce the workload of continuous electroencephalogram (EEG) monitoring. Current deep learning based seizure detection approaches usually employ cross-entropy loss as objective function, which generally suffer from inadequate utilization of sample labels and poor classification margins, resulting in decreased performance in seizure detection. In this study, we propose an end-to-end automatic seizure detection framework based on supervised contrastive learning, which effectively utilizes labeled EEG to cluster similar samples while separating dissimilar ones. A supervised contrastive learning loss is employed to optimize classification boundaries by making full use of EEG labels. We employ long-term continuous EEG for evaluation. Given the presence of various noise and interferences, assessment on long-term continuous EEG proves to be more challenging. Post-processing techniques such as smoothing filter, threshold judgment, and collar technique are further adopted to diminish the artifact impacts on seizure detection performance. The proposed method is evaluated on the publicly available Children's Hospital Boston and the Massachusetts Institute of Technology (CHB-MIT) scalp EEG dataset, achieving an event-based sensitivity of 99.71% and a false detection rate (FDR) of 0.35/h.

A novel cost-sensitive three-way intuitionistic fuzzy large margin classifier

Three-way decision (3WD) has been widely applied in diverse fields in tackling uncertainty, particularly in classification domain. As a discriminative learning algorithm, Large Margin Distribution Machine (LDM) aims to maximize the inter-class margin by leveraging the marginal distribution of samples, which captures the underlying structure and achieves superior classification performance. However, existing LDMs still suffer from some inherent flaws, including: i) the neglect of fuzzy decision items, leading to the inadequate handling of uncertainty; ii) the utilization of a cost-insensitive mechanism, resulting in high misclassification costs; and iii) the disregard of sample credibility, contributing to the noise susceptibility. To address these limitations, we introduce the intuitionistic fuzzy (IF) and cost-sensitive 3WD (CS3WD), presenting an innovative model known as the CS3W-IFLMC model. The IF theory is employed to quantify sample confidence levels, augmenting noise suppression in our proposed model. Moreover, the integration of the CS3WD method effectively reduces the overall decision cost and further improves the handling of uncertainty in datasets. Consequently, the CS3W-IFLMC model demonstrates superior noise resilience and generalization performance. Our comparative experiments validate the efficacy of the proposed CS3W-IFLMC model in achieving robustness to noise while upholding a competitive classification performance.

Attention-Enhanced Dual-Branch Residual Network with Adaptive L-Softmax Loss for Specific Emitter Identification under Low-Signal-to-Noise Ratio Conditions

To address the issue associated with poor accuracy rates for specific emitter identification (SEI) under low signal-to-noise ratio (SNR) conditions, where the single-dimension radar signal characteristics are severely affected by noise, we propose an attention-enhanced dual-branch residual network structure based on the adaptive large-margin Softmax (ALS). Initially, we designed a dual-branch network structure to extract features from one-dimensional intermediate frequency data and two-dimensional time–frequency images, respectively. By assigning different attention weights according to their importance, these features are fused into an enhanced joint feature for further training. This approach enables the model to extract distinctive features across multiple dimensions and achieve good recognition performance even when the signal is affected by noise. In addition, we have introduced L-Softmax to replace the original Softmax and propose the ALS. This approach adaptively calculates the classification margin decision parameter based on the angle between samples and the classification boundary and adjusts the margin values of the sample classification boundaries; it reduces the intra-class distance for the same class while increasing the inter-class distance between different classes without the need for cumbersome experiments to determine the optimal value of decision parameters. Our experimental findings revealed that, in comparison to alternative methods, our proposed approach markedly enhances the model’s capability to extract features from signals and classify them in low-SNR environments, thereby effectively diminishing the influence of noise. Notably, it achieves the highest recognition rate across a range of low-SNR conditions, registering an average increase in recognition rate of 4.8%.

Input Margins Can Predict Generalization Too

Understanding generalization in deep neural networks is an active area of research. A promising avenue of exploration has been that of margin measurements: the shortest distance to the decision boundary for a given sample or its representation internal to the network. While margins have been shown to be correlated with the generalization ability of a model when measured at its hidden representations (hidden margins), no such link between large margins and generalization has been established for input margins. We show that while input margins are not generally predictive of generalization, they can be if the search space is appropriately constrained. We develop such a measure based on input margins, which we refer to as 'constrained margins'. The predictive power of this new measure is demonstrated on the 'Predicting Generalization in Deep Learning' (PGDL) dataset and contrasted with hidden representation margins. We find that constrained margins achieve highly competitive scores and outperform other margin measurements in general. This provides a novel insight on the relationship between generalization and classification margins, and highlights the importance of considering the data manifold for investigations of generalization in DNNs.

Harnessing stochasticity for superconductive multi-layer spike-rate-coded neuromorphic networks

Conventional semiconductor-based integrated circuits are gradually approaching fundamental scaling limits. Many prospective solutions have recently emerged to supplement or replace both the technology on which basic devices are built and the architecture of data processing. Neuromorphic circuits are a promising approach to computing where techniques used by the brain to achieve high efficiency are exploited. Many existing neuromorphic circuits rely on unconventional and useful properties of novel technologies to better mimic the operation of the brain. One such technology is single flux quantum (SFQ) logic—a cryogenic superconductive technology in which the data are represented by quanta of magnetic flux (fluxons) produced and processed by Josephson junctions embedded within inductive loops. The movement of a fluxon within a circuit produces a quantized voltage pulse (SFQ pulse), resembling a neuronal spiking event. These circuits routinely operate at clock frequencies of tens to hundreds of gigahertz, making SFQ a natural technology for processing high frequency pulse trains. This work harnesses thermal stochasticity in superconducting synapses to emulate stochasticity in biological synapses in which the synapse probabilistically propagates or blocks incoming spikes. The authors also present neuronal, fan-in, and fan-out circuitry inspired by the literature that seamlessly cascade with the synapses for deep neural network construction. Synapse weights and neuron biases are set with bias current, and the authors propose multiple mechanisms for training the network and storing weights. The network primitives are successfully demonstrated in simulation in the context of a rate-coded multi-layer XOR neural network which achieves a wide classification margin. The proposed methodology is based solely on existing SFQ technology and does not employ unconventional superconductive devices or semiconductor transistors, making this proposed system an effective approach for scalable cryogenic neuromorphic computing.

SPGAN: Siamese projection Generative Adversarial Networks

Noise-to-image synthesis continues to be challenging, despite the application of the advanced loss functions in Generative Adversarial Networks (GANs). The main issue lies in the fact that discriminators employ hard margin classification, which is susceptible to misclassification. Moreover, the features of real image distribution learned by the generator are limited during adversarial training, thus the generated images are visually inferior to the real images. To tackle these challenges, a GAN method based on Siamese projection network (abbreviated as SPGAN) is proposed to learn a similarity measurement of features for image synthesis. Then, the similarity measurement is incorporated into the loss functions of the generator and discriminator, forming a similar feature adversarial learning. Through similar feature adversarial learning, SPGAN encourages the discriminator to maximize the dissimilarity between the features of real and generated images during recognition process. Simultaneously, it encourages the generator to synthesize images that contain more features resembling those of real images. Furthermore, we extend the SPGAN method by rewriting five representative loss functions, showcasing its compatibility with different loss functions. Experimental results demonstrate that the performance of SPGAN outperforms the advanced loss functions.

Regression-Based Hyperparameter Learning for Support Vector Machines.

Unification of classification and regression is a major challenge in machine learning and has attracted increasing attentions from researchers. In this article, we present a new idea for this challenge, where we convert the classification problem into a regression problem, and then use the methods in regression to solve the problem in classification. To this end, we leverage the widely used maximum margin classification algorithm and its typical representative, support vector machine (SVM). More specifically, we convert SVM into a piecewise linear regression task and propose a regression-based SVM (RBSVM) hyperparameter learning algorithm, where regression methods are used to solve several key problems in classification, such as learning of hyperparameters, calculation of prediction probabilities, and measurement of model uncertainty. To analyze the uncertainty of the model, we propose a new concept of model entropy, where the leave-one-out prediction probability of each sample is converted into entropy, and then used to quantify the uncertainty of the model. The model entropy is different from the classification margin, in the sense that it considers the distribution of all samples, not just the support vectors. Therefore, it can assess the uncertainty of the model more accurately than the classification margin. In the case of the same classification margin, the farther the sample distribution is from the classification hyperplane, the lower the model entropy. Experiments show that our algorithm (RBSVM) provides higher prediction accuracy and lower model uncertainty, when compared with state-of-the-art algorithms, such as Bayesian hyperparameter search and gradient-based hyperparameter learning algorithms.

Multiclass Graph-Based Large Margin Classifiers: Unified Approach for Support Vectors and Neural Networks.

While large margin classifiers are originally an outcome of an optimization framework, support vectors (SVs) can be obtained from geometric approaches. This article presents advances in the use of Gabriel graphs (GGs) in binary and mul-ticlass classification problems. For Chipclass, a hyperparameter-less and optimization-less GG-based binary classifier, we discuss how activation functions and support edge (SE)-centered neurons affect the classification, proposing smoother functions and structural SV (SSV)-centered neurons to achieve margins with low probabilities and smoother classification contours We extend the neural network architecture, which can be trained with backpropagation with a softmax function and a cross-entropy loss, or by solving a system of linear equations. A new subgraph-/distance-based membership function for graph regularization is also proposed, along with a new GG recomputation algorithm that is less computationally expensive than the standard approach. Experimental results with the Friedman test show that our method was better than previous GG-based classifiers and statistically equivalent to tree-based models.

Multiview Structural Large Margin Classifier and Its Safe Acceleration Strategy.

Multiview learning (MVL) concentrates on the problem where each instance is represented by multiple different feature sets. Efficiently exploring and exploiting the common and complementary information among different views remains challenging in MVL. Nevertheless, many existing algorithms deal with multiview problems via pairwise strategies, which limit the exploration of relationships among different views and dramatically increase the computational cost. In this article, we propose a multiview structural large margin classifier (MvSLMC) that simultaneously satisfies the consensus and complementarity principles in all views. Specifically, on the one hand, MvSLMC employs a structural regularization term to promote cohesion within-class and separability between-class in each view. On the other hand, different views provide extra structural information to each other, which favors the diversity of the classifier. Moreover, the introduction of hinge loss in MvSLMC results in sample sparsity, which we leverage to construct a safe screening rule (SSR) for accelerating MvSLMC. To the best of our knowledge, this is the first attempt at safe screening in MVL. Numerical experimental results demonstrate the effectiveness of MvSLMC and its safe acceleration method.

M2DC: A Meta-Learning Framework for Generalizable Diagnostic Classification of Major Depressive Disorder.

Psychiatric diseases are bringing heavy burdens for both individual health and social stability. The accurate and timely diagnosis of the diseases is essential for effective treatment and intervention. Thanks to the rapid development of brain imaging technology and machine learning algorithms, diagnostic classification of psychiatric diseases can be achieved based on brain images. However, due to divergences in scanning machines or parameters, the generalization capability of diagnostic classification models has always been an issue. We propose Meta-learning with Meta batch normalization and Distance Constraint (M2DC) for training diagnostic classification models. The framework can simulate the train-test domain shift situation and promote intra-class cohesion, as well as inter-class separation, which can lead to clearer classification margins and more generalizable models. To better encode dynamic brain graphs, we propose a concatenated spatiotemporal attention graph isomorphism network (CSTAGIN) as the backbone. The network is trained for the diagnostic classification of major depressive disorder (MDD) based on multi-site brain graphs. Extensive experiments on brain images from over 3261 subjects show that models trained by M2DC achieve the best performance on cross-site diagnostic classification tasks compared to various contemporary domain generalization methods and SOTA studies. The proposed M2DC is by far the first framework for multi-source closed-set domain generalizable training of diagnostic classification models for MDD and the trained models can be applied to reliable auxiliary diagnosis on novel data.

Dynamic Loss For Robust Learning.

Label noise and class imbalance are common challenges encountered in real-world datasets. Existing approaches for robust learning often focus on addressing either label noise or class imbalance individually, resulting in suboptimal performance when both biases are present. To bridge this gap, this work introduces a novel meta-learning-based dynamic loss that adapts the objective functions during the training process to effectively learn a classifier from long-tailed noisy data. Specifically, our dynamic loss consists of two components: a label corrector and a margin generator. The label corrector is responsible for correcting noisy labels, while the margin generator generates per-class classification margins by capturing the underlying data distribution and the learning state of the classifier. In addition, we employ a hierarchical sampling strategy that enriches a small amount of unbiased metadata with diverse and challenging samples. This enables the joint optimization of the two components in the dynamic loss through meta-learning, allowing the classifier to effectively adapt to clean and balanced test data. Extensive experiments conducted on multiple real-world and synthetic datasets with various types of data biases, including CIFAR-10/100, Animal-10N, ImageNet-LT, and Webvision, demonstrate that our method achieves state-of-the-art accuracy. The code for our approach will soon be made publicly available.

Boosting Few-shot Object Detection with Discriminative Representation and Class Margin

Classifying and accurately locating a visual category with few annotated training samples in computer vision has motivated the few-shot object detection technique, which exploits transfering the source-domain detection model to the target domain. Under this paradigm, however, such transferred source-domain detection model usually encounters difficulty in the classification of the target domain because of the low data diversity of novel training samples. To combat this, we present a simple yet effective few-shot detector, Transferable RCNN. To transfer general knowledge learned from data-abundant base classes to data-scarce novel classes, we propose a weight transfer strategy to promote model transferability and an attention-based feature enhancement mechanism to learn more robust object proposal feature representations. Further, we ensure strong discrimination by optimizing the contrastive objectives of feature maps via a supervised spatial contrastive loss. Meanwhile, we introduce an angle-guided additive margin classifier to augment instance-level inter-class difference and intra-class compactness, which is beneficial for improving the discriminative power of the few-shot classification head under a few supervisions. Our proposed framework outperforms the current works in various settings of PASCAL VOC and MSCOCO datasets; this demonstrates the effectiveness and generalization ability.

Maximum margin and global criterion based-recursive feature selection

In this research paper, we aim to investigate and address the limitations of recursive feature elimination (RFE) and its variants in high-dimensional feature selection tasks. We identify two main challenges associated with these methods. Firstly, the feature ranking criterion utilized in these approaches is inconsistent with the maximum-margin theory. Secondly, the computation of the criterion is performed locally, lacking the ability to measure the importance of features globally. To overcome these challenges, we propose a novel feature ranking criterion called Maximum Margin and Global (MMG) criterion. This criterion utilizes the classification margin to determine the importance of features and computes it globally, enabling a more accurate assessment of feature importance. Moreover, we introduce an optimal feature subset evaluation algorithm that leverages the MMG criterion to determine the best subset of features. To enhance the efficiency of the proposed algorithms, we provide two alpha seeding strategies that significantly reduce computational costs while maintaining high accuracy. These strategies offer a practical means to expedite the feature selection process. Through extensive experiments conducted on ten benchmark datasets, we demonstrate that our proposed algorithms outperform current state-of-the-art methods. Additionally, the alpha seeding strategies yield significant speedups, further enhancing the efficiency of the feature selection process.

Maximal Margin Support Vector Machine for Feature Representation and Classification.

High-dimensional small sample size data, which may lead to singularity in computation, are becoming increasingly common in the field of pattern recognition. Moreover, it is still an open problem how to extract the most suitable low-dimensional features for the support vector machine (SVM) and simultaneously avoid singularity so as to enhance the SVM's performance. To address these problems, this article designs a novel framework that integrates the discriminative feature extraction and sparse feature selection into the support vector framework to make full use of the classifiers' characteristics to find the optimal/maximal classification margin. As such, the extracted low-dimensional features from high-dimensional data are more suitable for SVM to obtain good performance. Thus, a novel algorithm, called the maximal margin SVM (MSVM), is proposed to achieve this goal. An alternatively iterative learning strategy is adopted in MSVM to learn the optimal discriminative sparse subspace and the corresponding support vectors. The mechanism and the essence of the designed MSVM are revealed. The computational complexity and convergence are also analyzed and validated. Experimental results on some well-known databases (including breastmnist, pneumoniamnist, colon-cancer, etc.) show the great potential of MSVM against classical discriminant analysis methods and SVM-related methods, and the codes can be available on https://www.scholat.com/laizhihui.

Improved large margin classifier via bounding hyperellipsoid

Support vector machine (SVM) is an excellent pattern recognition method. Many experiments have shown that SVM can achieve a generalization performance gain by carrying out it in the feature transformation space. Nevertheless, the theoretical foundation behind this phenomenon is presently lack of deep investigation. In the paper, we first give and prove a vital theoretical conclusion that SVM in the feature transformation space can obtain a lower radius-margin bound than one in the feature original space. This means that the performance of SVM can be improved by feature transformation since the radius-margin bound is directly associated with the generalization capacity. Based on this theoretical support, we further propose a novel method called covering-hyperellipsoid-constrained large margin classifier (CHC-LMC). The key characteristic of CHC-LMC is that it jointly learns the minimum bounding hyperellipse and the used classifier by directly minimizing the radius-margin bound in the transformation space, and so embodies the structural risk minimization principle. We develop the linear and nonlinear versions of CHC-LMC and employ an alternate optimization strategy to deal with the corresponding optimization problems. Finally, comprehensive experiments are conducted to verify the validity of CHC-LMC and evaluate the generalization performance by comparing it with the competing methods.

An unsupervised neural network for graphical health index construction and residual life prediction

To better characterize the health status and performing remaining useful life prediction, a composite health index is developed through the fusion of multi-channel signals. However, most of the existing literature limits the data fusion to be linear, which implies that the underlying degradation pattern must follow a linear form. This strong prerequisite of these approaches undermines the effectiveness of existing techniques for capturing the potential nonlinear nature of degradation process. In order to overcome this limitation as well as to improve the predictability, this paper proposes a nonlinear health index construction method achieving by an unsupervised neural network. Specifically, a neural network structure is introduced to approximate the highly nonlinear relationship between signals and health status. Furthermore, we consider the remaining useful life prediction as a binary classification problem, and then propose a maximal classification margin constraint, which is integrated with the monotonicity and minimal variability at the failure time to formulate the novel loss function. To estimate the model parameter, we developed a customized adaptive moment estimation algorithm (Adam). The comprehensive case study is performed based on the benchmark C-MAPSS dataset. As reported in the experiment, the constructed health index can better characterize the underlying degradation process.